SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines and implementation: Besides GIN, there are three lines of baseline methods: (1) self-supervised learning methods including EDGEPred, ATTRMASK, CONTEXTPred in (Hu et al., 2019), INFOMAX (Velickovic et al., 2019), JOAO (You et al., 2021), GRAPHLOG (Xu et al., 2021), MGSSL Zhang et al. (2021) and D-SLA (Kim et al., 2022), (2) semi-supervised learning methods including self-training with selected unlabeled graphs (ST-REAL) and generated graphs (ST-GEN) and INFOGRAPH (Sun et al., 2020), and (3) graph data augmentation (GDA) methods including FLAG (Kong et al., 2022), GREA (Liu et al., 2022a), and G-MIXUP (Han et al., 2022). For self-supervised pre-training, we follow their own settings and directly use their pre-trained models if available. For semi-supervised learning methods and DCT, we use 113K QM9 (Ramakrishnan et al., 2014) and 306K PPI graphs (Hu et al., 2019) as unlabeled data sources for the tasks on molecules/polymers and proteins, respectively. For DCT, we tune three major hyper-parameters: the number of perturbation steps  $D \in [1, 10]$ , the number of negative samples  $M \in [1, 10]$ , and top- $n\%$  labeled graphs of lowest property prediction loss selected for data augmentation.
4.2 Outstanding Property Prediction Performance
We report the model performance using mean and standard deviation over 10 runs Table 2. DCT is the best on all 15 tasks compared to the state-of-the-art baselines. Our observations are:
(1) GIN is the most competitive baseline and outperforms self-supervised learning methods. On 7 of 15 tasks, GIN outperforms all the 7 self-supervised learning methods. Because self-supervised pre-training imposes constraints on the model architecture, it undermines the true power of GNNs and under-performs the GNNs that are properly used.
(2) Self-training and GDA methods perform better than GIN but cannot effectively learn from unlabeled data. Self-training (ST-REAL and ST-GEN) is often the best baseline in regression tasks. GDA (GREA and G-MIXUP) methods outperform self-training in most classification tasks except ogbg-SIDER, because they are often designed to exploit categorical labeled data and remain underexplored for regression. Although self-training benefits from selecting unlabeled examples in some graph regression tasks, they are negatively affected by the unlabeled graphs in the classification tasks such as ogbg-ToxCast and ogbg-ClinTox. As indicated in Figure 3, it is inappropriate to pseudo-label unlabeled graphs in self-training due to the huge gap between the unlabeled data and target task.
(3) DCT transfers useful knowledge from unlabeled data by data augmentation. DCT outperforms the best baseline relatively by  $+3.9\%$ ,  $+13.4\%$ , and  $+10.2\%$  when there are only 1,210, 513, and 4,303 training graphs on ogbg-BACE, ogbg-FreeSolv, and GlassTemp, respectively. Compared to the self-supervised baselines, the improvement from DCT is more significant, so the knowledge transfer is more effective. For example, on ogbg-FreeSolv and  $\mathrm{O}_2\mathrm{Perm}$ , DCT performs better than the best self-supervised baselines relatively by  $+45.8\%$  and  $+8.0\%$ , respectively. On regression tasks that involve knowledge transfer across domains (e.g., from molecules to polymers), DCT reduces MAE relatively by  $1.9\% \sim 10.2\%$  compared to the best baseline. All these results demonstrate the outstanding performance of task-specific data augmentation in DCT.
Table 3: Comprehensive ablation studies for DCT on tasks ogbg-BACE, ogbg-SIDER, ogbg-FreeSolv, and  $\mathrm{O}_2\mathrm{Perm}$ . Objectives include minimizing  $\mathcal{I}_1(G', G)$  and/or maximizing  $\mathcal{I}_2(G', y)$ .
Objectives	Classification	Regression
I1(G',G)	I2(G',y)	BACE	SIDER	FreeSolv O2Perm
Top Baseline Method	82.4(2.4)	60.8(1.2)	1.547(0.082) 177.7(60.7)
Unlabeled Data Sources	X	X	84.4(2.6)	63.7(0.3)	1.473(0.192) 177.4(27.3)
✓	X	85.2(1.3)	63.7(0.2)	1.415(0.145) 171.4(14.0)
X	✓	84.7(1.8)	63.8(0.5)	1.344(0.096) 172.6(32.9)
✓	✓	85.6(0.6)	63.9(0.3)	1.339(0.075) 165.6(24.3)
ZINC	X	X	82.8(1.8)	63.5(0.7)	1.524(0.219) 175.5(11.9)
✓	X	83.3(2.2)	63.5(0.7)	1.455(0.207) 172.4(60.8)
X	✓	84.3(0.6)	63.5(0.6)	1.514(0.214) 171.5(26.0)
✓	✓	84.9(0.4)	63.7(0.7)	1.408(0.092) 169.3(15.3)
Figure 4: Effect of hyper-parameters, including the number of perturbation steps  $D \in [1,10]$ , the number of negative graphs  $M \in [1,10]$ , and top- $n\%$  labeled graphs whose labels are predicted the most accurately and that are selected for data augmentation, where  $n \in [10,100]$ .
4.3 Ablation Studies and Performance Analysis
Comprehensive ablation studies: In Table 3, we investigate how the task-related objectives in Eq. (5) impact the performance of DCT. First, DCT outperforms the top baseline even if the two task-related optimization objectives are disabled. This is because DCT generates new training examples based on original labeled graphs: the data augmentation has already improved the diversity of the training dataset a little bit. Second, adding the objective  $\mathcal{I}_1$  further improves the performance by encouraging the generation of diverse examples, because it minimizes the similarity between the original graph and augmented graph in the graph space. Third, we receive the best performance of DCT when it combines  $\mathcal{I}_1$  and  $\mathcal{I}_2$  objectives to generate task-related and diverse augmented graphs. When we change the unlabeled data source from QM9 to the ZINC dataset from (Jo et al., 2022), similar observations confirm the necessity of the task-related objectives.
Effect of hyper-parameters: The impacts of three hyper-parameters of DCT are studied: the number of perturbation steps  $D$ , the number of negative samples  $M$  in Eq. (6), and the number of augmented graphs in each iteration (i.e., top- $n\%$  selected graph for augmentation). Results from Figure 4 show that DCT is robust to a wide range of  $D$  and  $M$  valued from 0 to 10. They suggest that  $D$  and  $M$  can be set as 5 in most cases. As for the number of the augmented graphs in each iteration, results show that noisy graphs are often created when  $n$  is higher than  $30\%$ , because the predictor cannot effectively guide the data augmentation for those labeled graphs whose labels are hard to predict. So,  $10\%$  is suggested as the default of top- $n\%$ .
Iterative process: Figure 5 investigates the relationship between the quality of augmented graphs and the accuracy of property prediction models. We save a predictor checkpoint every 20 epochs
Figure 6: Case studies of augmented graphs. The green highlighted subgraphs are from GIN with top-k pooling. Examples show that the augmented graphs from DCT preserve the core structures of original graphs. Key concepts in the unlabeled graphs like chemical validity are transferred to downstream tasks. Domain knowledge such as the relationship between the permeability and the fluorine atom/methyl group is captured to guide task-specific generation.
Figure 5: Data augmentation and model training mutually enhance each other over epochs. The predictor is saved every 20 epochs to guide the generation of augmented graphs. The performance of GIN trained on these augmented graphs reflects the quality of the augmented data.
to guide the generation of the augmented examples. We evaluate the quality of augmented graphs by using them to train GIN and report AUC/MAE. The data augmentation gradually decreases the training loss of property prediction. On the other hand, the increased GIN performance indicates that the quality of augmented examples is also improved over epochs. The data augmentation and predictor training mutual enhance each other.
4.4 Interpretability of Visible Knowledge Transfer
Knowledge transfer by data augmentation gives visible examples, allowing us to study what is learned. We visualize a few augmented graphs in DCT using ogbg-BACE and  $\mathrm{O}_2\mathrm{Perm}$ . We adapt top-k pooling (Knyazev et al., 2019) to select the subgraphs that GIN used for prediction. The selected subgraphs are highlighted in green in Figure 6. The three examples show that the augmented graphs can identify and preserve the core structures that GIN uses to predict property values. These augmented graphs are chemically valid, showing that concepts such as some chemical rules from the unlabeled graphs are successfully transferred to downstream tasks. More results are in appendix D.2. Regarding task-specific knowledge, it is known that the fluorine atom and the methyl group are usually negatively and positively correlated to the permeability, respectively (Park et al., 2003; Corrado and Guo, 2020). The augmented examples show that DCT captures this domain knowledge with the task-related objectives. In example (b), DCT replaces most of the fluorine atoms with the methyl groups. It encourages GIN to learn the positive relationship between the methyl group and the permeability so that GIN predicts a high label value. In example (c), DCT replaces the methyl groups with fluorine atoms. It encourages GIN to learn the negative relationship between the fluorine atom and the permeability so that GIN predicts a low label value.
5.1 Graph Property Prediction
Graph neural networks (GNNs) (Kipf and Welling, 2017; Xu et al., 2019) are commonly used for graph property prediction in chemistry and polymer informatics tasks (Otsuka et al., 2011; Hu et al., 2020; Zhou et al., 2022). However, it is hard to annotate enough labels in these domains. Recent work used self-supervised tasks such as node attribute prediction and graph structure prediction (Hu et al., 2019; You et al., 2021; Kim et al., 2022) to pre-train architecture-fixed GNNs. Sun et al. (2022) observed that the existing methods might fail to transfer knowledge from unlabeled graph data. Flexible GNN architectures for downstream tasks would be desirable.
Graph data augmentation (GDA) methods do not restrict GNN architecture choices to improve prediction accuracy (Trivedi et al., 2022; Zhao et al., 2022, 2023; Ding et al., 2022). They learn to create new examples that preserve the properties of original graphs (Liu et al., 2022b,a; Kong et al., 2022; Han et al., 2022; Luo et al., 2022). However, they purely manipulate labeled examples and thus cannot utilize the knowledge in unlabeled graphs. Our DCT combines the knowledge from the unlabeled dataset and the labeled task dataset. It creates label-preserved graph examples with the knowledge transferred from the unlabeled data. It allows the GNN models to have flexible architectures.
5.2 Learning from Unlabeled Data
Pre-training on self-supervised tasks such as masked image modeling and autoregressive text generation is effective for large language and vision models (Brown et al., 2020; He et al., 2022). However, the hand-crafted self-supervised tasks could hardly help models learn useful knowledge from unlabeled graphs due to the gap between these label-agnostic tasks and the downstream prediction tasks towards drug discovery and material discovery (Sun et al., 2021; Kim et al., 2022; Inae et al., 2023). A universal self-supervised task to learn from the unlabeled graphs remains under-explored (Sun et al., 2022; Trivedi et al., 2022).
Semi-supervised learning assumes that unlabeled and labeled data are from the same source (Liu et al., 2023). The learning objective in the latent space is usually mutual information maximization that encourages similarity between the representations of unlabeled and labeled graphs (Sun et al., 2020). However, the distributions of the unlabeled and labeled data could be very different due to the different types of sources (Hu et al., 2019), leading to negative impacts on the property prediction on the labeled graphs. Self-training, as a specific type of semi-supervised learning method, selects the unlabeled graphs of confidently predictable labels and assigns pseudo-labels for them (Lee et al., 2013; Iscen et al., 2019). Many studies have explored improving uncertainty estimation (Gal and Ghahramani, 2016; Tagasovska and Lopez-Paz, 2019; Amini et al., 2020) to help the model filter out noise for reliable pseudo-labels. Recently, pseudo-labels have been applied in imbalanced learning (Liu et al., 2023) and representation learning (Ghiasi et al., 2021). However, self-training is restricted to confidently predictable labels and may ignore the huge number of any other unlabeled graphs (Huang et al., 2022). Therefore, it cannot fully utilize the knowledge in the unlabeled graphs.
In contrast, our DCT employs a diffusion model to extract knowledge (as the diffusion and reverse processes) from all the unlabeled graphs. DCT represents the knowledge as task-specific labeled examples to augment the target dataset, instead of uninterpretable pre-trained model parameters. We note that self- or semi-supervised learning does not conflict with DCT, and we leave their combinations for future work.
5.3 Diffusion Models on Graphs
Recent works have improved the diffusion models on graphs (Niu et al., 2020; Jo et al., 2022; Vignac et al., 2022; Kong et al., 2023; Chen et al., 2023). EDP-GNN (Niu et al., 2020) employed score matching for permutation-invariant graph data distribution. GDSS (Jo et al., 2022) extended the continuous-time framework [6] to model node-edge joint distribution. DiGress (Vignac et al., 2022) used the transition matrix to preserve the discrete natures of the graph structure. GraphARM (Kong et al., 2023) introduced a node-absorbing autoregressive diffusion process. EDGE (Chen et al., 2023) focused on efficiently generating larger graphs. Instead of improving the generation performance of the diffusion model, our model builds on score-based diffusion models (Jo et al., 2022; Song et al., 2021) for predictive tasks, i.e., graph classification and graph regression.
In this work, we made the first attempt to transfer minimal sufficient knowledge from unlabeled graphs by data augmentation. We proposed a data-centric framework to use the diffusion model trained on the unlabeled graphs and use two task-related objectives to generate task-specific augmented graphs. Experiments demonstrated the performance of the proposed framework through visible augmented examples. It is better than self-supervised learning, self-training, and graph data augmentation methods on as many as 15 tasks.
This work was supported by NSF IIS-2142827, IIS-2146761, IIS-2234058, CBET-2102592, and ONR N00014-22-1-2507.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33:14927-14937, 2020.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.04340, 2017.
Randall Balestriero, Leon Bottou, and Yann LeCun. The effects of regularization and data augmentation are class dependent. arXiv preprint arXiv:2204.03632, 2022.
Hans-Joachim Böhm, Alexander Flohr, and Martin Stahl. Scaffold hopping. *Drug discovery today: Technologies*, 1(3):217-224, 2004.
Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Hammers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, and Daniel Rueckert. Gan augmentation: Augmenting training data using generative adversarial networks. arXiv preprint arXiv:1810.10863, 2018.
Nathan Brown. Bioisosteres and scaffold hopping in medicinal chemistry. Molecular informatics, 33 (6-7):458-462, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Xiaohui Chen, Jiaxing He, Xu Han, and Li-Ping Liu. Efficient and degree-guided graph generation via discrete diffusion modeling. arXiv preprint arXiv:2305.04111, 2023.
Peter AG Cormack and Amaia Zurutuza Elorza. Molecularly imprinted polymers: synthesis and characterisation. Journal of chromatography B, 804(1):173-182, 2004.
Tanner Corrado and Ruilan Guo. Macromolecular design strategies toward tailoring free volume in glassy polymers for high performance gas separation membranes. Molecular Systems Design & Engineering, 5(1):22-48, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 113-123, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu. Data augmentation for deep graph learning: A survey. ACM SIGKDD Explorations Newsletter, 24(2):61-77, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoc V Le, and Tsung-Yi Lin. Multi-task self-training for learning general representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8856-8865, 2021.
Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. International Conference on Machine Learning, 2022.
A Additional Related Work on Data-Centric Approach
Data Augmentation Data augmentation creates new examples with preserved labels but uses no unlabeled data (Shorten and Khoshgoftaar, 2019; Kashefi and Hwa, 2020; Balestriero et al., 2022). Examples of heuristic data augmentation techniques include flipping, distorting, and rotating images (Shorten and Khoshgoftaar, 2019), using lexical substitution, inserting words, and shuffling sentences in texts (Kashefi and Hwa, 2020), and deleting nodes and dropping edges in graphs (Zhao et al., 2021, 2023). While human knowledge can be used to improve data diversity and reduce over-fitting in heuristic methods, it is difficult to use a single heuristic method to preserve the different labels for different tasks (Balestriero et al., 2022; Cubuk et al., 2019). So, automated augmentation (Cubuk et al., 2019) learned from data to search for the best policy to combine a bunch of predefined heuristic augmentations. Generation models (Antoniou et al., 2017; Bowles et al., 2018; Han et al., 2022) create in-class examples. Other learning ideas such as FATTEN (Liu et al., 2018) and GREA (Liu et al., 2022a) learned to split the latent space for data augmentation. However, learning and augmenting from insufficient labels at the same time may limit the diversity of new examples and cause over-fitting. DCT leverages unlabeled data to avoid them.
Relationship between Data-Centric Approaches As presented in Figure 7, perturb edges, delete nodes and mask attributes (Rong et al., 2019; Trivedi et al., 2022) for graphs are some heuristic ways for data augmentation. The augmented knowledge from them is mainly controlled by human prior knowledge on the perturbations and it often fails to be close to the task, i.e., random perturbations hardly preserve labels for the augmented graphs. The learning to augment approaches learn from labeled graphs to perturb graph structures (Luo et al., 2022), to estimate graphons for different classes (Han et al., 2022), or to split the latent space for augmentation (Liu et al., 2022a). Although these approaches could preserve labels for the augmented graphs, they introduce less extra knowledge to improve the model prediction. In summary, graph data augmentation is effective in expanding knowledge for limited labels, but it makes no use of unlabeled graphs. Besides,
Figure 7: Qualitative relationship of graphs from different data-centric approach on the task relatedness and contained knowledge.
the diversity and richness of the domain knowledge from augmented graphs are far from that contained in a large number of unlabeled graphs. To learn from unlabeled graphs, data-centric approaches like the self-training is assumed to be useful when the unlabeled and labeled data are from the same source. It is less studied when we have a single unified unlabeled source for different tasks.
B Additional Method Details
B.1 Upper bounding the mutual information
In Eq. (6), we use a leave-one-out variant of InfoNCE  $(\mathcal{I}_{\mathrm{bound}})$  to derive the upper bound of mutual information. We summarize the derivation (Poole et al., 2019) here.
$$
\begin{array}{l} \mathcal {I} _ {1} (G ^ {\prime}; G) = \mathbb {E} _ {p (G, G ^ {\prime})} \left[ \log \frac {p (G ^ {\prime} | G)}{p (G ^ {\prime})} \right] \\ = \mathbb {E} _ {p (G, G ^ {\prime})} \left[ \log \frac {p \left(G ^ {\prime} \mid G\right) q \left(G ^ {\prime}\right)}{q \left(G ^ {\prime}\right) p \left(G ^ {\prime}\right)} \right] \\ = \mathbb {E} _ {p (G, G ^ {\prime})} \left[ \log \frac {p \left(G ^ {\prime} \mid G\right)}{q \left(G ^ {\prime}\right)} \right] - \mathrm {K L} \left(p \left(G ^ {\prime}\right) \mid \mid q \left(G ^ {\prime}\right)\right) \tag {9} \\ \leq \mathbb {E} _ {p (G, G ^ {\prime})} \left[ \log \frac {p \left(G ^ {\prime} \mid G\right)}{q \left(G ^ {\prime}\right)} \right] \\ \end{array}
$$
The intractable upper bound is minimized when the variational approximation  $q(G')$  matches the true marginal  $p(G')$  (Poole et al., 2019). For each  $G_i$ , its augmented output  $G_i'$ , and  $M - 1$  negative
examples with different labels, we could approximate  $q(G_i') = \frac{1}{M - 1} \sum_{j \neq i} p(G_i'|G_j)$ . So, we have
$$
\begin{array}{l} \mathcal {I} _ {1} (G _ {i} ^ {\prime}, G _ {i}) \leq \log \frac {p (G _ {i} ^ {\prime} | G _ {i})}{\frac {1}{K - 1} \sum_ {j = 1 , j \neq i} ^ {M} p (G _ {i} ^ {\prime} | G _ {j})} \\ = \log \frac {p \left(G _ {i} ^ {\prime} \mid G _ {i}\right)}{\sum_ {j = 1 , j \neq i} ^ {M} p \left(G _ {i} ^ {\prime} \mid G _ {j}\right)} + \log (M - 1) \tag {10} \\ = \mathcal {I} _ {\text {b o u n d}} \left(G _ {i} ^ {\prime}; G _ {i}\right) + \text {c o n s t a n t} \\ \end{array}
$$
B.2 Extraction of Statistical Features on Graphs
For each molecule and polymer graph, we concatenate the following vectors or values for statistical feature extraction.
- the sum of the degree in the graph;
- the vector indicating the distribution of atom types;
- the vector containing the maximum, minimum and mean values of atoms weights in a molecule or polymer;
- the vector containing the maximum, minimum, and mean values of bond valence.
For each protein-protein interaction ego-graph in the biology field, we use the sorted vector of node degree distribution in the graph as the statistical features.
B.3 Technical Details for Graph Data Augmentation with Diffusion Model