Data-Centric Learning from Unlabeled Graphs with Diffusion Model
Authors: Gang Liu
Eric Inae
Tong Zhao
Jiaxin Xu
Tengfei Luo
Meng Jiang
Venue: NeurIPS 2023 poster (2023)
Keywords: Graph property prediction
Molecular property prediction
Diffusion model
Unlabeled data
Data augmentation
Transfer learning
TL;DR: We propose a data-centric approach that uses knowledge from unlabeled graphs more effectively and visibly than parameter-centric approaches such as graph self-supervised pre-training.

Abstract: Graph property prediction tasks are important and numerous. While each task offers a small size of labeled examples, unlabeled graphs have been collected from various sources and at a large scale. A conventional approach is training a model with the unlabeled graphs on self-supervised tasks and then fine-tuning the model on the prediction tasks. However, the self-supervised task knowledge could not be aligned or sometimes conflicted with what the predictions needed. In this paper, we propose to extract the knowledge underlying the large set of unlabeled graphs as a specific set of useful data points to augment each property prediction model. We use a diffusion model to fully utilize the unlabeled graphs and design two new objectives to guide the model's denoising process with each task's labeled data to generate task-specific graph examples and their labels. Experiments demonstrate that our data-centric approach performs significantly better than fifteen existing various methods on fifteen tasks. The performance improvement brought by unlabeled data is visible as the generated labeled examples unlike the self-supervised learning.

Data-Centric Learning from Unlabeled Graphs with Diffusion Model
Gang Liu University of Notre Dame gliu7@nd. edu
Eric Inae University of Notre Dame einaeênd. edu
Tong Zhao Snap Inc. tzhao@snap.. com
Jiaxin Xu
Tengfei Luo University of Notre Dame tluo@nd.
Meng Jiang University of Notre Dame mj i injanndndd.ccm edu
University of Notre Dame jxu240nd edu
Abstract
Graph property prediction tasks are important and numerous. While each task offers a small size of labeled examples, unlabeled graphs have been collected from various sources and at large scale. A conventional approach is training a model with the unlabeled graphs on self-superrised tasks and then tine-tuning the model on the prediction tasks. However, the sulf-ssde task knowledge could not be aligned or sometimes conflicted with what the predictions needed. In this paper, we propose extract the knowledge underlying the large set of unlabeled graphs set of useful data points to augment each property prediction model. We use a diffusion model to fully utilize the unlabeled graphs and design two new objectives to guide the model's s denoising process with each task's s labeled data to generate task specific graph examples and their labels. Experiments demonstrate that our data-centric approach performs significantly better than fifteen existing various methods on fifteen tasks. The performance improvement brought by unlabeled data is visible the generated labeled examples unlike the self-supervised learning.
specific
1 Introduction
Graph data such molecules and polymers are found to have attractive properties in drug and material discovery (Böhm et al., 2004; Huang et al., (202T). but annotating them requires specialized knowledge, as well as lengthy and costly experiments wet labs (Cormack and Elorza 2004). So, is important for graph property predictors to learn useful knowledge from unlabeled graphs.
Self-supervised learning (Hu al. 2019 Rong al. 2020 You et al. 202 Kim al. 2022 utilizes unlabeled graphs to learn through predictive tasks or contrastive lasks to represent and transfer the knowledge model parameters. Despite the empirical success in language and vision (Brown et al.) 2020: He et al. (022). their performance on graph data applications remains unsatisfactory because of the significant gap between the graph self-supervised task and the graph label prediction task. Models trained on node attribute prediction (Hu et al. 2019) as a simple predictive self-supervised task extract too limited knowledge from the graph structure, which has been observed after too fast convergence (Sun et al., 2022). More complex tasks like graph context prediction (Hu et al.] 2019 Zhang al. 2021) may transfer knowledge that conflicts with downstream tasks. Aromatic rings, for instance, are prevalent structure in molecules (Maziarka et al. 2020) and are considered valuable in context prediction tasks (Zhang et al., 2021). However, graph properties such as oxygen permeability can be more related to non-aromatic rings in some cases (L.iu et al. (022a). which is overlooked if not using tailored predictive tasks specifically for downstream tasks. As predictive tasks strive for
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Unlabeled graphs {G e glo y
Unlsbeled graphs
guido
Data-centric approach:
Existing approach:
denoising
Diffasion Model
Target knowledge in la- beled graphs guides the denoising process in dif- fusion model generate new labeled examples close to the target graph space instead of the un- labeled graph space. The augmented knowledge for the prediction model is visible as graphs.
Self-nupevisee
Knowledge from self- supervised tasks could not be aligned or even conflict with what predictions need. Parameter initialization could hardly interpret how unlabeled graphs were or
tooks
(Knowledge sbout g
Generated graph for tusk (G' E g yxii))
parameter
update
Labeled graphs in lask & (Koowldgg sbout gift yilh)
Labeled graphs task it {G, e giM,y, >xer)
would be able to improve the models, leading to high prediction errors.
Prediction
Prediction
Model
Model
Tasks
Tasks i = 1
Comparing the diagrams of the existing approach and the proposed approach to learning from unlabeled graphs for variety of graph property prediction tasks.
Figure 1:
universality, the transferred knowledge may force models to focus more on aromatic rings, leading to poor prediction.
On the other line, contrastive tasks Youu et al., 2021 Kim et al. 2022) aim learn the similarity between original and perturbed graphs. However, the learned similarity can hardly generalize across tasks (Kim et al.) 2022). First, perturbations without domain knowledge, e.g., bioisosteres, do not preserve broad biological properties (Sun et al. [021). Second, it is difficult, if not impossible, to find universal perturbations that generalize to diverse property prediction tasks. For example, bioisosteric (subgraph) replacements produce similar biological properties for molecules. And they may reduce toxicity (Brown) 2014). So, contrastive tasks with bioisosteric replacement enforce the similarity between toxic and non-toxic molecules. However, models pre-trained on such contrastive tasks hurt the performance on downstream tasks, e.g,, toxicity prediction.
Our data-centric idea avoids the use of self-supervised tasks that are not appropriate. We use diffusion probabilistic model (known as diffusion model) to capture the data distribution of unlabeled graphs. leveraging its capability of distribution coverage, stationarity, and scalability (Dhariwal and Nichol 202T). At the stage of performing a particular property prediction task, the reverse process, guided by novel task-related optimization objectives, generates new task-specific labeled examples. Minimal sufficient knowledge from the unlabeled data is transferred into these examples, instead of uninterpretable model parameters, and then to enhance the training of prediction models.
To implement our idea, we propose a Data- Centr-Ceiriic Transfer framework (DCT) based on diffusion model for graph data, as shown in Figure 1b It aims to transfer minimal sufficient knowledge from unlabeled graphs property predictors by data augmentation. The diffusion model gradually adds Gaussian noise to a graph from which score function (i.e. the gradient of the log probability density) then learned to estimate the noise step by step reverse the process. DCT trains the diffusion model on the unlabeled graphs to get ready to augment any labeled dataset. Given a labeled graph from a particular task (i.e., type of property), the diffusion model adds noise to perturb by a few steps and then generates new graph through the score function. The new graph could be close to the distribution of the unlabeled graphs for diversity, however, it would lose the relatedness to the target task. So, we add two task-related objectives into the score function to guide the reverse process. When a predictor model I has been trained on the task, given an original labeled graph G, the first objective is to optimize the new graph G' to sufficiently preserve the label of G with f. The second objective is to optimize G' to be very different from (i.e., minimally similar to) G. These two objectives ensure that G" carries minimal sufficient knowledge from the unlabeled graphs to be an augmentation of G. DCT iteratively generates new examples to augment the labeled dataset and progressively trains the prediction model with it.
We test DCT on fifteen graph property prediction datasets from three fields: chemistry (molecules), material science (polymers), and biology (protein-protein interaction graphs). DCT achieves the best performance over all these tasks. We find that the state-of-the-art self-supervised methods often struggle to transfer knowledge to regression tasks, etc. DCT reduces the mean absolute error relatively by 13.4% and 10.2% compared to the best baseline on the molecule and polymer graph regression tasks, respectively.
2

---

2 Problem Definition
Given K property prediction tasks, there are NTW) labeled graph examples for the k-th task. They {(G,: Mi) G2 E , E -yIii where Glk) is the graph space and yılki is the label space the task. The prediction model with parameters 0 is defined as f/ G + ylkl, f& consists of a GNN and a multi-layer perceptron (MLP). Without the loss of generality, we consider Graph Isomorphism Networks (GIN) (Xu et al., 2019) to encode graph structures. Given a graph G = (V, E Glki in the task k. GIN updates the representation vector of node v E V at 1-layer:
h = MLP' ((+te) (1+6)-h,-+ 1
E 4
)
(I)
aEA(e)
where e is learnable scalar and E [(v) is one of node v's neighbor nodes. After stacking L layers, the READOUT function (e.g., summation) gets the graph representation across all the nodes. The predicted label is:
J = MLP (READOUT ((h u / G)))
(2)
[ is hard to be well-trained because it is hard to collect graph labels at a large scale (NVlk is small).
Fortunately, regardless of the tasks, large number of unlabeled graphs are usually available from the same or similar domains. Self-supervised learning methods (Hu et al. 2019) rely on hand-crafted tasks to extract knowledge from the unlabeled examples {G, E G = 1, ...... N} as pre-trained model parameters f. The uninterpretable parameters are transferred to warm up the prediction models (.f(%) | on the downstream graph property prediction tasks. However, the gap and even conflict between the self supervised tasks and the property prediction tasks lead to suboptimal performance of the prediction models. In the next section, we present the DCT framework that transfers knowledge from the unlabeled graphs with data-centric approach.
3
The Data-Centric Transfer Framework
3.1 Overview of Proposed Framework
The goal of data-centric approaches is to augment training datasets by generating useful labeled data examples. Under that, the goal of our data-centric transfer (DCT) framework is transfer the knowledge from unlabeled data into the data augmentation. Specifically, for each graph-label pair (G E GlA), (| E yoA in the task ks the framework is expected to output a new example G with the label y //l such that (1) //Al = yil and (2) Gr(kl and Glll are from the same graph space glk), However, if the graph structures of G"(A] and GlM were too similar, the augmentation would duplicate the original data examples, become useless, and even cause over-fitting. So, the optimal graph data augmentation should enrich the training data with good diversity as well as preserve the labels of the original graphs. To achieve this, DCT utilizes diffusion probabilistic model to first learn the data distribution from unlabeled graphs (Section (.2)). Then DCT adapts the reverse process in the diffusion model to generate task-speciftt labeled graphs for data augmentation (Section 3.3) Thus, the augmented graphs will be derived from the distribution of a huge collection of unlabeled data for diversity. To preserve the labels, DCT controls the reverse process with two telkerelated optimization objectives to transfer minimal sufficient knowledge from the unlabeled data. The first objective minimizes upper bound of mutual information between the augmented and the original graphs in the graph space. The second objective maximizes the probability of the predicted label of augmented graphs being the same as the label of original graphs. The first is for minimal knowledge transfer, and the second is for sufficient knowledge transfer. DCT integrates the two objectives into the reverse process of the diffusion model to guide the generation of new labeled graphs. DCT iteratively trains the graph property predictor (used in the second objective) and creates the augmented training data. To simplify notations, we remove the task superscript [k] in the following sections.
3.2 Learning Data Distribution from Unlabeled Graphs
The diffusion process for graphs Figure appures applies to both graph structure and node features. The diffusion model slowly corrupts unlabeled graphs to standard normal distribution with noise. For
3

---

Figure 2: Diffusion model in DCT: It performs task-specific data augmen- tation using objectives Z1 and Iz in the reverse process. The model was trained unlabeled graphs to learn the general data distribution. Then it generates (G', y = y) based on (G.y) y) in the reverse process. It per- turbs G with D steps and optimizes G' to be minimally similar to G (Ob- jective I1) and sufficiently preserve the label of G (Objective I2).
Original graph:
G G
n T>
Output graph:
Labeled graph
fo
do
(G,y)
I
Self trainig w/ Real Unlabeled Graphs
Self trainig w/ Genersted Unlabeled Graphs
DCT w/ Augmented Labeled Graphs
25
+10.5%
150
Improvement %
3.75%
+1 0%
+0.22%
GIN
1. 885
9%
+17. 7%%
Molecule
Polymer
Polymer
ogigg-4lV boggg-iigggg-goCCaao ToxCast ogbg-Tox23 ogbg ogbg-BUBP BUBP ogig BACE ogbg. ClinTox Mistingeee
GlassTemp
CyPer
Figure 3: Relative improvement (increased AUC or reduced MAE) from three data-centric methods (over ten runs), compared to the basic GIN: Blue is for self-training with selected real unlabeled graphs. Green for self-training with graphs directly generated by a standard diffusion model. Red for DCT that generates task-specific labeled graphs. The first two often make little or negative impact. Our DCT has consistent and significant improvement shown as the percentages in red.
graph generation, the model samples noise from the normal distribution and learns score function to reverse the perturbed noise. Given an unlabeled graph G, we use continuous time / E [0, T] to index multiple diffusion steps [G(O]T= on the graph, such that (()) follows the original data distribution and (TT follows prior distribution like the normal distribution. The forward diffusion a stochastic differential equation (SDE) from the graph to the noise:
d = (G
.t) dt g(t) dw.
where w is the standard Wiener process, f(., G + g the drift coefficient and g()) - IR the diffusion coefficient. f(G(), t) and g(t) relate to the amount of noise added the graph at each infinitesimal step t. The reverse-time SDE uses gradient fields or scores of the perturbed graphs
V Veg log (G()) for denoising and graph generation from T to 0 (Song et al. 2021
d = (() t) = Geo log log (G)))) dt + g(t)dW,
(4)
where Pe(C PG((t) ) the marginal distribution at time & in forward diffusion. Wiener process. dt here is infinitesimal negative time step. The score Vate log (G is unknown in practice and it is approximated by the score function s(G(), t) with score matching techniques (Song et al. 202II. On graphs, Jo et al. (2022) used two GNNs to develop the score function (G t) to de-noise both node features and graph structures and details are in appendix B.3
reverse time standard
3.3 Generating Task-specific Labeled Graphs
Self-training approaches would propose to either (1) select unlabeled graphs by graph property predictor or (2) generate graphs directly from the standard diffusion model, and then use the predictor to assign them labels so that the training data could be enriched. However, we have observed that neither of them can guarantee positive impact on the prediction performance. In fact, as shown in Figure 3 they make very little or even negative impact. That is because the selected or directly- generated gruphs are too different from the labeled graph space of the target tasks, Task details of ten datasets are in appendix C

---

Given labeled graph (G, y) from the original dataset of a specific task, the new labeled graph (G' expected provide useful knowledge to augment the training set. We name it the augmented graph throughout this section. The augmented graph is desired to have the following two properties, as in Section 3.1 Task relatedness: As an effective training data point, G' E 9 and y' E y are from the graph/label spaces of the specific task where (G,y) come from and thus transfer sufficient task knowledge into the training set; Diversity: If G' was too similar to G, the new data point would cause severe over-fitting on the property prediction model. The augmentation aims to learn from unlabeled graph to create diverse data points, which should contain minimal task knowledge about G.
The selected unlabeled graphs used in self- trafiiraining have little task relatedness because the unlabeled data distribution might be too far from the one of the specific task. Existing graph data augmentation methods could not create diverse graph examples because they manipulated labeled graphs and did not learn from the unlabeled graphs. Our novel data-centric approach DCT works towards both desired properties by transferring minimally sufficient knowledge from the unlabeled graphs: Sufficiency achieved by maximizing the possibility for label preservation (i.e., y y). It ensures that the knowledge from unlabeled graphs is task-related; Minimality refers to the minimization of graph similarity between G' and to ensure that the augmentation introduces diversity. Both optimizations can be formulated using mutual information I(- --) to generate task-specific labeled data (G", y Definition 3.1 (Sufficiency for Data Augmentation), The augmented graph G' sufficiently preserves the label of the original graph G' if and only if I(G": y) v):
Definition 3.2 (Minimal Sufficiency for Data Augmentation). The Sufficiency is minimal for data augmentation if and only if [[G:: G) < I[G; G). YG represents any augmented graph that sufficiently preserves the original graph's 's label.
Self-supervised tasks applied similar philosophy in pre-training (Soatto and Chiuso 2016). however, they did not use labeled data from any specific tasks. So the optimizations were unable to extract useful knowledge and transfer it to the downstream (Tian et al. 2020).. In our DCT that performs task-specific data augmentation, the augmented graphs can be optimized toward the objectives using any labeled graph G and its label y:
min max EG I (G: IC v). Iz
For the first objective, we use the leave-one-one variant of InfoNCE (Poole et al. 2019 Oord et al. 2018 as the upper bound estimation. For the i-th labeled graph (G,,3.).
G.G
I S Tromsd (G;;; G.) = log Σj P(G
(6)
where G. is the augmented graph. When G", is optimized, G, makes a positive pair; [G3} (i + i)
are M - negative samples of labels that do not equal y/- (M is a hyperparameter) We use cosine exp(sim(G",.G) .G,) similarity and softmax function to calculate p(G]]G,) Σ exp(eim((,..)) In practice, extract statistical features of graphs to calculate their similarity. Details are in appendix B.2
For the second objective, we denote the predicted label of the augmented graph G' by (G').. We maximize the log likelihood log (vl Jo (G')) to maximize Iz(G"; y). Specifically, after the predictor fo trained for several epochs on the labeled data, we freeze its parameters and use it to optimize the augmented graphs so they are task-related:
C(G') Inound (G'; log log (v)
Framework details: As shown in Figure 2. after the diffusion model learns the data distribution from unlabeled graphs, given a labeled graph G from a specific task, DCT perturbs it for D (D < T) steps. The perturbed noisy graph. denoted by G(P), stays inside the task-specific graph and label space, rather than the noise distribution (at step T).. To reverse the noise in it and generate a task- specific augmented example G', DCT integrates the loss function in Eq. 7 into the score function s(, t) for minimal sufficient knowledge transfer:
a [(()(),)) ()) =t) (s(G'),) aV ((.)(((())) de + g())dw,
(
where a is a scalar for score alignment between and VL to avoid the dominance of any of them: 1s(G VV 4(G9))2 Because (+)) is intermediate state in the reverse process, the noise in it
5

---

Table 1: Statistics of datasets for graph property prediction in different domains.
Data Type
Dataset
# Graphs
Prediction Task
Task
Avg./Max Nodes
Avg.MMx Edges
ugbg-HIV
41,127
Classification
25.5/ 222
54.9/ 502
ogbg- Cast-
8,576
Classification
617
18.8/ 124
388
ogbg- ogbg-Tox21
7,831
Classification
12
18.6/ 132
38.6 38.6/290
ogbg-BBBP
2,039
Classification
24.1 132
51.9/290
Molecules
ogbg-BACE
1,513
Classification
34.1/97
73.7/202
ogbg- Tgbg-ClinToon Tox
1,477
Classification
26.2/ 136
/
ogbg-SIDER
1,427
Classification
27
33.6/492
70.7 1010
ughg-Lipp
4200
Regression
27/115
59 236
ugbg-ESSO
1128
Regression
13.3/55
27.4/124
ogbg-FreeSoly
642
Regression
8.7/24
/6
GlussTemp
7,174
Regression
36.7/ 166
79.3 362
MeltingTemp
3,651
Regression
26.9/ 102
55.4/212
Polymers
ThermCond
759
Regression
21.3 /71
42.3 162
OyPerm
595
Regression
37.3/ 103
82.1 234
Proteins
PPI
88000
Classification
49.4/ 111
890.8 11556
may fail the optimizations. So, we design a new sampling method named double-loop sampling for accurate loss calculation. It has inner-loop sampling using Eq. 44 to sample G(ey. as the denoised version of (+)) at the reverse time Then VoL(G(()) is calculated as an alternative for Vog (()()) Finally, outer-loop sampling takes one step guide denoising using Eq. (8)
DCT iteratively creates the augmented graphs (G', y'), updates the training dataset {(G,, vi)) and trains the graph property predictor fa. In each iteration, for task k < N labeled graphs of the lowest property prediction loss are selected to create the augmented graphs.The The predictor is better fitted to these graphs for more accurate sufficiency estimation of the augmentation.
Experiments
In this section, present and analyze experimental results demonstrate the outstanding perfor- mance of DCT, the usefulness of new optimization objectives, the effect of hyperparameters and iterative process, and the interpretability of "visible" knowledge transfer from unlabeled graphs.
4.1 Experimental Setup
Experiments are conducted on 15 graph property prediction tasks in chemistry, material science, and biology, including seven molecule classification, three molecule regression tasks from open graph benchmarks (Hu et al.) 2020). four polymer regression tasks, and protein function prediction (PPI) (Hu et al., 2019). Dataset statistics is presented in Table T We use the area under the ROC curve (AUC) to evaluate classifiers and mean absolute error (MAB) for regressors.
Tasks and metrics:
Baselines and implementation: Besides GIN, there are three lines of baseline methods: (1) self- supervised learning methods including EDGEPRED, ATTRMASK, CONTEXIPRED in (Hu al. 2019), INFOMAX (Velickovic et al. (019), JOAO (You et al. 2021), GRAPHLOG (Xu et al. 2021] MGSSL Zhang et a (2021) and D-SLA (Kim et al. (022). semi-supervised learning methods including self-training with selected unlabeled graphs (ST- (EAR) and generated graphs SENG and INFOGRAPH (Sun al. 2020), and (3) graph data augmentation (GDA) methods including
FLAG (Kngg et al.) 2022), GREA (Liu et al. 2022a), and G-MIXUP (Han et al.] 2022). For self-supervised pre-training. we follow their own settings and directly use their pre-trained models if available. For semi- supervised learning methods and DCT, we use 13K QM9 (Ramakrishnan 2014) and 306K PPI graphs (Hu al. 2019) as unlabeled data sources for the tasks on molecules/polymers and proteins, respectively. For DCT, we tune three major number of perturbation steps E [1, 10), the number of negative samples M E [1, 10), and top-n % labeled graphs of lowest property prediction loss selected for data augmentation.
the
4.2 Outstanding Property Prediction Performance
We report the model performance using mean and standard deviation over runs Table DCT the best on all 15 tasks compared to the state-of-the-att baselines. Our observations are:

---

Table 2: Mean(Sid) tasks from different fields. The best mean is bold. The best baseline is underlined. Results are highlighted if unlabeled graphs bring significant negative impacts compared to GIN. The MAE for ThermCond is scaled
100. G-MIXUP was proposed for classification. MGSSL was proposed for molecules.
Molecule Clussification: AUC (
ogbg-HIV 32,901
ogbg-ToxCast ToxCast
ogbg-Tox21 6,264
ogbg-BBBP 1,631
ogbg-BACE 1,210
ogbg-Clin ogbg-SIDDE 1,181
# Training Graphs
6,860
1,141
GIN
77.41.2)
66.9w2)
76,0ez)
67.5mm
77.52a)
88.83.)
58.1a
EDGEPRED
78.13.3
63,9k0.
75.5w.4
69.9ka.)
79.5
62.9k2.9
59, 59.7
Self-Superrisell
ATTRMASK
77.1
64.2m0.5
76,610.4)
63.9(12)
79.3m..)
70.40.1
60.7
CONTEXTPRED
78.4.011
63.7(..)
75.0ke.)
68.8c.9
75.7(1.)
63.2(65)
60.7
INFOMAX
75.4
61.7
75.5ke.4
69. 69.2
76.8
73.0k.a)
58 6..5)
JOAO
76.2.02)
64.8/0.3)
74.8(0.5)
69.3(2.5)
75.9
69.4(4.5
60.8m.))
GRAPHLOG
74.8(113
63.200.8
75.4.0.3
67.5c.3
80.4
69.0ks.6
57.010.9
MGSSL.
k
65.7e..4
77.2.0.3
66.9ta.9
81.3
69.8cs.)
63.6
D-SLA
76.9
60.80.2)
76.10..3>
62.6r.m
80.3w..)
78.3(2.4)
55,1(10)
GDA Somi-SZ
INFOGRAPH
73.3
61.50.
67.6m0.
61 664-4)
75.9(1.)
62.2(5.)
56.3g.
ST-REAL
78.3
64.5
76.2...)
66. 7
77.4(.))
82.2(2.4)
60.80.2)
ST--ENN
77.916
65.1
75.8
66.3)1
78.4
87.30.3
59.3(13
FLAG
74.6.17)
59,9118
76.90.7)
66.6x1.)
79.1
85.1
57.6(23)
GREA
79.3:a.
67 67.5w.7)
77
69. 7(1.)
82.4(.4)
87.9(.7)
60. (220)
G-MIXUP
77.1an
55.61.)
64.6w.4)
70.2
77.8
60.2
56.8(33)
DCT (Ours)
79.5.um
68.1m..)
78.2.0.)
70.8
85.60.6)
(2.1
63.9p
Molecule Regression: MAE
Polymer Regression: MAE
Bio: AUC (563
ogbg-Lipo 3,360
ogbg-ESOL 902
Sebg-FreeSolv 513
GlassTemp 4,303
MelungTomp 2,189
ThermCond
O.Perm 356
PPI
# Training Graphs
455
60,715
GIN
0.54Saam9
0.766j0c16)
16699o.44)
26.4.0.0
20.9
3.25/0.99
201.3:45.00
69.1m.
EDGEPRED
0.585ma000)
1.062a.06)
2.249%a.150
27. 6L.6).4
47.428
3.69
207.344.7
63.7(.D
Self SiipSipeeiisee
ATTRMASK
0.573ma..mm
m.041
1.952
27.7.0
45.822
3.17/033)
179.90008
64.10.
CONTEXTPRED
0.592mmmm:
0.97
m
2. 2.19330.111
27. 6x0.3
46.7
3,15:0
62.0x1
INFOMAX
0.581a.00m
0.935m.01s
2.197
27.5io.:
46.5c2s
3.31/0.55)
231.0:226)
63.3(12)
JOAO
0.596m.00
1.098mm77
2.465:0.095
27. S0.50.
46.0mm
3,55
763.7
61.5012)
GRAPHL.GG
m.577mam 1.)
2.373
29.5c13)
50.365
3.01camm
229.764k.)
62.1mm
MGSSL
0.569mmmm
0.998ma0u
1. 199560077)
26.9
4
3.10:a.)
201.1
N.A.
D-SLA
0.563m0004)
1.064m000)
2.190%.149)
27.5(Um)
51.7255
2.71/0005)
257.8/322
65.0t12)
GDA Somi St.
INFOGRAPH
D. (47
3.7 3.710
30.8
$1,2
2.75
207.2
67.7mm
ST-REAL
0.626
0.788mm00)
1.770z251)
26.66.31
42.3ua
2.64/00
256.0(1..)
68, 1)
ST-GEN
0.724
L.470002
26.8.0.1
42,0m²
2.70am0)
262.2/00.)
68.6mu.
FLAG
0.528
0.755
1.6650000
26.6x131
44 44.2
3.05ea.00
177.7
69.2m
GREA
0.586
0,805
26.7c.m
41.1m8
3,23
194.0c45.5)
68.8m..)
DCT (Ours)
0.516:001)
0.717
1.339mm
23.7mm
38.0mz
2.59
165.6043
69.5wm
(1) GIN is the most competitive baseline and outperforms self-supervised learning methods. On 7 of 15 tasks, GIN outperforms the self-supervised learning methods. Because self-supervised pre-training imposes constraints on the model architecture, it undermines the true power of GNNs and under-performs the GNNs that are properly used.
(2) Self-training and GDA methods perform better than GIN but cannot effectively learn from unlabeled data. Self-triiing (ST-REAL and ST ST-GEN) often the best baseline in regression tasks. GDA (GREA and G-MIXUP) methods outperform self- seff-training in most classification tasks except ogbg-Sh SIDER, because they are often designed to exploit categorical labeled data and remain under- explored for regression. Although self-training benefits from selecting unlabeled examples in some graph regression tasks, they are negatively affected by the unlabeled graphs in the classification tasks such as Tox and ogbg-ClinToox As indicated in Figure 3 it is inappropriate to pseudo-label unlabeled graphs in self-training due to the huge gap between the unlabeled data and target task.
(3) DCT transfers useful knowledge from unlabeled data by data augmentation. DCT outper- forms the best baseline relatively by +3.9% +13.4%, and + +10.2% when there are only 1,210, 513, ogbg-FreeSolv, and GlassTemp, respectively. Compared and 4,303 training graphs on improvement from DCT is more significant, so the knowledge to the self-supervised baselines, the transfer is more effective. For example, on ogbg- FreeSol and O,Perm, DCT performs better than the best self-supervised baselines relatively by +45.8% and +8.0%, respectively. On regression tasks that involve knowledge transfer across domains (e.g., from molecules to polymers). DCT reduces MAE relatively by 1.9% 10.2% compared to the best baseline. All these results demonstrate the outstanding performance of task-specific data augmentation in DCT.
7

---

Table 3: Comprehensive ablation studies for DCT tasks ogbg BACE, ogbg- SIDER, ogbg- Frbee-reeS,v, and OsPerm. Objectives include minimizing In (G', (G',G) and/or maximizing (2 (G'
Objectives
Classification
Regression
[G', G) Ia v)
BACE SIDER
FreeSolv
OyPerm
Top Buseline Method
82.4:2.: 60.8 1,547aa 177.7.00.7
x
84.4:2.: 63.7 1.473 177.4(273)
Unlabeled Data Sources
QM9
85.2 63.7 1.415(.34))
171.444.)
X
84.7c.8) 63.8(a.)
1.344:0.006)
172.6(229)
85.6cn.6) (3.9
1.33%)..055 165.6(24.3)
82.8(1.8 63.5
1.524x0.119
(75.5(11.9)
ZINC
V
x
83.3
I 1.455x0.207) 63.5
172.4(60.8)
84.3x..s
(3.5().))
1.51 2x.24) 171.5(26.0)
/
84.9to.; 63.7 1. 0.408 169.3(15.)
oglig BACE median:
ugbg. SIDER
aglg FroeSolv min: 1,274 mediam: 1,308
O.Perm
max: 64.1 mediinn; 63.5
min:: 158. median: 168.6
AUC
MAE
GRRA
82.4
10A0
2,105
1,943
2505
77.4
38.9
ST-Ge
1.047
67.6
10.11
Nag. Samppias
2.5
2,
Tub Purtmn. Stops
53.0
55.9
# Nop Samples
10.0
7.0 Pertudh sume
225
5 25 # Nop Samples
1000
7.0 Pertunt guop
149.2
120.6
255 # Now Sumple
10.
Portuch. Stap
7.0 1o
oPCp.
oiby,SIDER
aphy- FrroSSoly
OpPerr
253
1.500
305
$3.0
CREA
JONO
1.440
2000
75.0
38.0
12200
1320
173.0
150
PLAG
20
40
CO
all
025 Selected for
WN Selected Sot Ang.
Stacted Ang.
Selected
Aug.
Figure 4: Effect of hyper- parameters, including the number of perturbation steps D [1, 10), the number of negative graphs M E [1, 10), and top-n % labeled graphs whose labels are predicted the most accurately and that are selected for data augmentation, where n E [10, 100].
4.3 Ablation Studies and Performance Analysis
Comprehensive ablation studies: T we investigate how the task-related objectives in Eq. (5) impact the performance of DCT. First, DCT outperforms the top baseline even if the two task-related optimization objectives are disabled. This is because DCT generates new training examples based on original labeled graphs: the data augmentation has already improved the diversity of the training dataset a little bit. Second, adding the objective I₁ further improves the performance by encouraging the generation of diverse examples, because it minimizes the similarity between the original graph and augmented graph in the graph space. Third, we receive the best performance of DCT when combines I, and Iz objectives to generate task-related and diverse augmented graphs. When change the unlabeled data source from QM9 to the ZINC dataset from (Jo et al. 2022), similar observations confirm the necessity of the task-related objectives.
Effect of of perturbation steps D, the number of negative samples M in Eq. (6). and the number of augmented graphs in each iteration (i.e., top-n % selected graph for augmentation). Results from Figure 4 show that DCT is robust to wide range of D and M valued from 0 to 10. They suggest that D and M can be set as in most cases. As for the number the augmented graphs in each iteration, results often created when is higher than 30%, because the predictor cannot
The impacts of three
of DCT are studied: the number
show that noisy graphs effectively guide the data augmentation for those labeled graphs whose labels are hard to predict. So, 10% is suggested the default of top-n%.
Iterative process: Figure investigates the relationship between the quality of augmented graphs and the accuracy of property prediction models. We save predictor checkpoint every 20 epochs

---

Figure 5: Data augmentation and model training mutually enhance each other epochs. The predictor is saved every 20 epochs to guide the generation of augmented graphs. The performance GIN trained on these augmented graphs reflects the quality of the augmented data.
Train Tese
Porf. Aup. Date
oglygBBcC BACE
A11
OaPern
1.0
200
0.7
1.2
0.5
777
200
0.5
76
0.4
0.0
150
Fposhs
Firarh
Figure 6: Case studies of augmented graphs. The green highlighted sub- graphs are from GIN with top-k pool- ing. Examples show that the aug- mented graphs from DCT preserve the core structures of original graphs. Key concepts in the unlabeled graphs like chemical validity are transferred to downstream tasks. Domain knowl- edge such as the relationship be- tween the permeability and the fluo- rine atom/methyl group is captured to guide task-specific generation.
(4) Classification on BACE
Regression Task OsPern (c) Label 2.2
(b)
goo Preductio 23.0
Powckction: 70,1
M
Prediction 13.9
to guide the generation of the augmented examples. We evaluate the quality of augmented graphs by using them to train GIN and report AUC/MAE. The data augmentation gradually decreases the training loss of property prediction. On the other hand, the increased GIN performance indicates that the quality of augmented examples is also improved over epochs. The data augmentation and predictor training mutual enhance each other.
4.4 Interpretability of Visible Knowledge Transfer
Knowledge transfer by data augmentation gives visible examples, allowing us to study what is learned. We visualize a few augmented graphs DCT using ogbg-BACE and OyPerm. We adapt top-k pooling (Knyazev et al. 2019) to select the subgraphs that GIN used for prediction. The selected subgraphs are highlighted in green in Figure The three examples show that the augmented graphs can identify and preserve the core structures that GIN uses to predict property values. These augmented graphs are chemically valid, showing that concepts such as some chemical rules from the unlabeled graphs are successfully transferred to downstream tasks. More results are in appendix D.2) Regarding task-specift knowledge, it is known that the fluorine atom and the methyl group are usually negatively and positively correlated to the permeability, respectively (Park et al. 2003 Corrado and Guo 2020). The augmented examples show that DCT captures this domain knowledge with the task-related objectives. In example (b), DCT replaces most of the fluorine atoms with the methyl groups. It encourages GIN to learn the positive relationship between the methyl group and the permeability that GIN predicts high label value. In example (c), DCT replaces the methyl groups with fluorine atoms. It encourages GIN to learn the negative relationship between the fluorine atom and the permeability so that GIN predicts low label value.
Related Work
5.1 Graph Property Prediction
Graph neural networks (GNNs) (Kipf and Welling 2017 Xu et al. 2019 are commonly used for
graph property prediction in chemistry and polymer informatics tasks (Otsuka et al., 2011 I Hu
al.)
2020; Zhou et al., 2022). However, it is hard to annotate enough labels in these domains. Recent work used self-supervised tasky such as node attribute prediction and graph structure prediction (Hu et al., 2019 You et al.) 2021 Kim et al.) 2022) to pre-train architecture-fixe GNNs. Sun et al.
(2022) observed that the existing methods might fail to transfer knowledge from unlabeled graph data. Flexible GNN architectures for downstream tasks would be desirable.

---

Graph data augmentation (GDA) methods do not restrict GNN architecture choices to improve
prediction accuracy (Trivedi et al. 2022 Zhao al., 2022 2023 Ding et al. (022). They learn to create new examples that preserve the properties of original graphs Liiu et al. 2022b|a Kong al. 2022 Han et al., 2022 Luo et al. 2022). However, they purely manipulate labeled examples and thus cannot utilize the knowledge in unlabeled graphs. Our DCT combines the knowledge from the unlabeled dataset and the labeled task dataset. It creates tabel-preserved graph examples with the knowledge transferred from the unlabeled data. It allows the GNN models to have flexible
architectures.
5.2 Learning from Unlabeled Data
Pre-training on self-supervised tasks such as masked image modeling and autoregressive text genera- tion is effective for large language and vision models (Brown et al. [020 He et al., 2022). However, the hand-crafted self self-super tasks could hardly help models learn useful knowledge from unla-
beled graphs due to the gap between these label-agnostic tasks and the downstream prediction tasks towards drug discovery and material discovery (Sun et al. 2021 Kim et al. 2022) Inae et al. 2023).
A universal self-supervised task to learn from the unlabeled graphs remains under-explored (Sun
et al. 2022 Trivedi et al.) 2022).
Semi supirvipe learning assumes that unlabeled and labeled data are from the same source (Liu ct al. 2023). The learning objective in the latent space is usually mutual information maximization that encourages similarity between the representations of unlabeled and labeled graphs (Sun et al.,
2020). However, the distributions of the unlabeled and labeled data could be very different due to the different types of sources (Hu et al. 2019), leading negative impacts on the property prediction on the labeled graphs. Self-Iraining, as specific type of semi-supervised learning method, selects the unlabeled graphs of confidently predictable labels and assigns pseudo-labele labels for them (Lee let 20133 Iscen et al. 2019). Many studies have explored improving uncertainty estimation (Gal and Ghahramani 2016 TTaasovssaa and Lopez-Paz 2019) Amini al. 2020) to help the model filter out noise for reliable pseudo-labels. Recently, pscudo-labels have been applied in imbalanced learning (Liu et al. 2023) and representation learning (Ghiasi et al. (021). However, self-training is
restricted to confidently predictable labels and may ignore the huge number of any other unlabeled graphs (Huang et al. 2022). Therefore, cannot fully utilize the knowledge in the unlabeled graphs
In contrast, our DCT employs a diffusion model to extract knowledge (as the diffusion and reverse processes) from all the unlabeled graphs. DCT represents the knowledge task-specifit labeled examples to augment the target dataset, instead of uninterpretable pre-trained model parameters. We note that self- or semi-supervised learning does not conflict with DCT, and leave their combinations for future work.
5.3 Diffusion Models on Graphs
Recent works have improved the diffusion models on graphs (Niu et al. 2020 Jo et al 2022 Vignac
al. 2022 Kong al., 2023). Chen et al., 2023). EDP-GNN (Niu et al. (020) employed score matching for permutation-ine -invaaaaiioonnnniiaan graph data distribution. GDSS (Jo et al. 2022) extended the
continuous-tima framework [6] to model node-edge joint distribution. DiGress (Vignac et al. 2022)
used the transition matrix to preserve the discrete natures of the graph structure. GraphARM Kong
al., 2023) introduced node-absorbing autoregressive diffusion process. EDGE (Chen al. 2023)
focused on efficiently generating larger graphs. Instead of improving the generation performance of the diffusion model, our model builds on score-based diffusion models (Jo et al., 2022 2021) for predictive tasks, i.e., graph classification and graph regression.
Song et al.
Conclusion
In this work, we made the first attempt to transfer minimal sufficient knowledge from unlabeled graphs by data augmentation. We proposed data-centric framework to use the diffusion model trained on the unlabeled graphs and use two task-related objectives to generate task-specific augmented graphs. Experiments demonstrated the performance of the proposed framework through visible augmented examples. It is better than self-supervised learning, self-training, and graph data augmentation methods on many as 15 tasks.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by NSF IIS-2142827, IIS-2146761, 223428,, CBET-2102592, and ONR N00014-22-1-2507.
References
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33:14927-14937, 2020.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprini
2017.
Randall Balestriero, Leon Bottou, and Yann LeCun. The effects of regularization and data augmenta- tion class dependent. arXiv preprint arXiv:2204.03632, 2022.
Hans-Joachim Böhm, Alexander Flohr, and Martin Stahl. Scaffold hopping. Drug discovery today:
Technologies, (33):217-224) 2004.
Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Hammers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, and Daniel Rueckert. Gan augmentation: Augmenting training data using generative adversarial networks. arXiv preprint arXiv:1810. 100663 2018.
Nathan Brown. Bioisosteres and scaffold hopping in medicinal chemistry. Molecular informatics, 33 (6-7):458 462, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Xiaohui Chen, Jiaxing He, Xu Han, and Li-Ping Liu. Efficient and degree-guided graph generation via discrete diffusion modeling. arXiv preprint arXiv:2305.04111 2023.
Peter AG Cormack and Amaia Zurutuza Elorza. Molecularly imprinted polymers: synthesis and characterisation. Journal of chromatography B, 304 2004.
Tanner Corrado and Ruilan Guo. Macromolecular design strategies toward tailoring free volume in glassy polymers for high performance gas separation membranes. Molecular Systems Design & Engineering. 5(1):22 48, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 113-123, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794 2021.
Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu. Data augmentation for deep graph learning: A survey. ACM SIGKDD Explorations Newsletter, 24(2):61-77, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learming, pages 1050-1059, PMLR. 2016.
Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoe V Le, and Tsung- Yi Lin. Multi talki self-training for learning general representations. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1865,8865, 2021.
Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. International Conference Machine Learning, 2022.
11

---

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puitern Recognition, pages 16000-16009, 2022.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv: 1905. 12265, 2019.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22 18-22133, 2020.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf H Roohani, Jure Leskovec, Connor W. Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Kexin Huang, Vishnu Sresht, Brajesh Rai, and Mykola Bordyuh. Uncertainty-awan pseudo-labeling for quantum calculations. In The 38th Conference on Uncertainly in Artificial Intelligence, 2022.
Eric Inae, Gang Liu, and Meng Jiang. Motif-aware attribute masking for molecular graph pre-training. arXiv preprint arXiv:2309. 230.00399990 2023.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5070-5079, 2019.
Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International Conference on Machine Learning, volume 162, pages 10362-10383, PMLR, 2022.
Omid Kashefi and Rebecca Hwa. Quantifying the evaluation of heuristic methods for textual data augmentation. In Proceedings of the Sixth Workshop Noisy User-generated Text (W-NUT 2020), 2020.
Dongki Kim, Jinheon Baek, and Sung Ju Hwang. Graph self-supervised learning with accurate discrepancy learning. Advances in Neural Information Processing Systems, 2022.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Boris Knyazev, Graham W Taylor, and Mohamed Amer. Understanding attention and generalization in graph neural networks. Advances in neural information processing systems, 32, 2019.
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. Robust optimization as data augmentation for large-scale graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 60-69, 2022.
Lingkai Kong, Jiaming Cui, Haotian Sun, Yuchen Zhuang, B Aditya Prakash, and Chao Zhang. Autoregressive diffusion model for graph generation. International conference on machine learning, 2023.
Dong-Hyur Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. ICML, volume 3, page 896. Atlanta, 2013.
Bo Liu, Xudong Wang, Mandar Dixit, Roland Kwitt, and Nuno Vasconcelos. Feature space transfer for data augmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9090-9098, 2018
Gang Liu, Tong Zhao, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Graph rationalization with environment-based augmentations. Proceedings of the 28th ACM SIGKDD international confer- ence on knowledge discovery & data mining, 2022a.
Gang Liu, Tong Zhao, Erie Inae, Tengfei Luo, and Meng Jiang. Semi-supervised graph imbalanced regression. arxiv preprint arXiv:2305.1208/ 2023.
12

---

Songtao Liu, Rex Ying, Hanze Dong, Lanqing Li, Tingyang Xu, Yu Rong, Peilin Zhao, Junzhou Huang, and Dinghao Wu. Local augmentation for graph neural networks. International Conference on Machine Learning, pages 14054-14072. PMLR, 2022b.
Youzhi Luo, Michael McThrow, Wing Yee Au, Tao Komikado, Kanji Uchino, Koji Maruhash, and Shuiwang Ji. Automated data augmentations for graph classification. arXiv preprint arXiv:2202. 13248, 2022.
Ruimin Ma, Hanfeng Zhang, Jiaxin Xu, Luning Sun, Yoshihiro Hayashi, Ryo Yoshida, Junichiro Shiomi, Jian-xun Wang, and Tengfei Luo. Machine learning-assister exploration of thermally conductive polymers based on high-throughput molecular dynamics simulations. Materials Today Physics, 28: 100850, 2022.
Lukasz Maziarka, Agnieszka Pocha, Jan Kaczmarczyk, Krzysztof Rataj, Tomasz Danel, and Michal Warchot. Mol-cyclegan: a generative model for molecular optimization. Journal of Cheminformat- ics, 12(1))1-18, 2020.
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permu- tation invariant graph generation via score- score-based generative modeling. In International Conference on Artificial Intelligence and Statistics, pages 4474 4484. PMLR, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
Shingo Otsuka, Isao Kuwajima, Junko Hosoya, Yibin Xu, and Masayoshi Yamazaki. Polyinfo: Polymer database for polymeric materials design. In 2011 International Conference on Emerging Intelligent Data and Web Technologies, pages 29. IEEE, 2011.
Sang-Hee Park, Kwang-Je Kim, Wook So, Sang-Jin Moon, and Soo-Bok Lee. Gas separation properties of 6fda-basee polyimide membranes with polar group. Macromolecular Research, 11: 157-162, 2003.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171- 5180. PMLR, 2019.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. arXiv preprint arXiv: 19X7 2019.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances Neural Information Processing Systems, 33:12559-12577 2020.
Connor Shorten and Taghi M Khoshgoftaar. survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Stefano Soatto and Alessandro Chiuso. Visual representations:
Defining properties and deep approxi- mations. International Conference on Learning Representations, 2016.
Yang Song, Jascha Sohl- Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021.
Teague Sterling and John J Irwin. Zine 1i-lid discovery for everyone. Journal of chemical information and modeling, 55(1 1):2324-2337, 2015.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. International Conference on Learning Representations, 2020.
13

---

Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou. Moel: data-driven molecular fingerprint via knowledge- contrastive learning from molecular graph. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3585-3594, 2021.
Ruoxi Sun, Hamjun Dai, and Adams Wei Yu. Does gnn pretraining help molecular representation? Advances in Neural Information Processing Systems, 2022.
Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. Advances in Neural Information Processing Systems, 32, 2019.
A Thornton, L Robeson, B Freeman, and D Uhlmann. Polymer gas separation membrane database, 2012.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 0066877-689, 2020.
Puja Trivedi, Ekdeep Singh Lubana, Mark Heimann, Danai Koutra, and Jayaraman J Thiagarajan. Analyzing data-centric properties for graph contrastive learning. In Advances in Neural Information Processing Systems, 2022.
Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint 0rXiv:2209.14733 2022.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9():513-530, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? International Conference on Learning Representations, 2019.
Minghao Xu, Hang Wang, Binghing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning, pages 1548-11558, PMLR, 2021.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121-12132. PMLR, 2021.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self- supervised learning for molecular property prediction. Advances in Neural Information Processing Systems, 34:15870-1555 -18882 2021.
Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. In Proceedings of the caai conference on artificial intelligence, volume 35, pages 11015-11023, 2021.
Tong Zhao, Gang Liu, Daheng Wang, Wenhao Yu, and Meng Jiang. Learning from counterfactual links for link prediction. In International Conference on Machine Learning, pages 36911-26926 PMLR, 2022.
Tong Zhao, Wei Jin, Yozen Liu, Yingheng Wang, Gang Liu, Stephan Günneman, Neil Shah and Meng Jiang. Graph data augmentation for graph machine learning: A survey, IEEE Data Engineering Bulletin, 2023.
Kuangqi Zhou, Kaixin Wang, Jian Tang, Jiashi Feng. Bryan Hooi, Peilin Zhao, Tingyang Xu, and Xinchao Wang. Jointly modelling uncertainty and diversity for active molecular property prediction. In Learning on Graphs Conference, pages 29-1. PMLR, 2022.
14

---

A Additional Related Work on Data-Centric Approach
Data Augmentation Data augmentation creates new examples with preserved labels but uses no unlabeled data (Shorten and Khoshgoftaar 2019 Kashefi and Hwa 2020 Balestriero al.) 2022p. Examples of heuristic data augmentation techniques include flipping, distorting, and rotating images (Shorten and Khoshgoftaar, 2019), using lexical substitution, inserting words, and shuffling sentences in texts (Kashefi and Hwa 2020), and deleting nodes and dropping edges in graphs (Zhao) be used improve data diversity and reduce
et al., 2021 2023). While human knowledge over-fitting in heuristic methods, it is difficult to use a single heuristic method to preserve the
different labels for different tasks (Balestriero al.] 2022 Cubuk al.) 2019). So, automated augmentation (Cubuk et al. 2019) learned from data to search for the best policy to combine a bunch of predefined heuristic augmentations. Generation models (Antoniou et al. 2017 Bowles al. 2018 Han et al. (0222) create in-class examples. Other learning ideas such as FATTEN (Liu et al.
2018 and GREA (Liu al. 2022a) learned to split the latent space for data augmentation. However,
learning and augmenting from insufficient labels at the same time may limit the diversity of new examples and cause over-fitting DCT leverages unlabeled data to avoid them.
Relationship between Data-Centric Approaches As presented in Figure [7] perturb edges, delete nodes and mask attributes (Rong al. 2019 Trivedi al.) (2222) for graphs are some heuristic ways for data augmentation. The augmented knowledge from them is mainly controlled by human prior knowledge on the perturbations and it often
Figure 7: Qualitative relationship of graphs from different data-centric ap- proach on the task relatedness and con- tained knowledge.
fails be close to the task, i.e., random perturbations hardly preserve labels for the augmented graphs. The Iearn- ing to augment approaches learn from labeled graphs perturb graph structures (Luo et al.] [202)). to estimate graphons for different classes (Han et al. 2022), or split the latent space for augmentation (Liu et al. 2022a). Although these approaches could preserve labels for the augmented graphs, they introduce less extra knowledge to improve the model prediction. summary, graph data aug- mentation is effective in expanding knowledge for limited labels, but it makes no use unlabeled graphs. Besides, the diversity and richness of the domain knowledge from augmented graphs are far from that con- tained in a large number of unlabeled graphs. To learn from unlabeled graphs, data-centric approaches like the self-training is assumed to be useful when the unlabeled and labeled data are from the same source. less studied when we have a single unified unlabeled source for different tasks.
Tusk of Labels
Approach
Leam to Amamen Kong et al at.
Houristic
Augmentazion (Ron atall
Unlabeled Data
Self Training)
Consined Knowdolg
B
Additional Method Details
B.1
Upper bounding the mutual information
In Eq. 6 we use a leave-one-out variant InfoNCE (Tomna)) ) to derive the upper bound of mutual information. We summarize the derivation (Poole et al. 2019) here.
log G)
Ií GG Ex((,)))
p(G')
log p(G')G)a(G'))
Ep(c.o)
((G')((G))
(9)
log P(G'(G)
(I(p(G")ll(("")
(G'))
log G(G
Ep(c.or)
((G")
The intractable upper bound minimized when the variational approximation g(G") matches the
true marginal p(G") (Poole et al. 2019). For each G,, its augmented output G₁, and M 1 negative
15

---

examples with different labels, we could approximate q(G,) = M-1 So, we have
G.G
I (G', G;) I log
P(G)))
K
G
(10)
log
+ log( [ -)
j= Li LGA
Towand G G₂) + constant
B.2
Extraction of Statistical Features on Graphs
For each molecule and polymer graph, concatenate the following vectors values for statistical
feature extraction.
the sum of the degree in the graph;
the vector indicating the distribution of atom types;
the vector containing the maximum, minimum and mean values of atoms weights in
molecule or polymer;
the vector containing the maximum, minimum, and mean values of bond valence.
For each protein interaction ego-graph in the biology field, we use the sorted vector of node
degree distribution in the graph as the statistical features.
B.3 Technical Details for Graph Data Augmentation with Diffusion Model
The Lookup Table from Atom Type to Node Embedding Space Given graph G, we assume the node feature matrix on the graph is X E R where n is the number of nodes. The edge feature matrix is E E R mmx F., where the graph structure in practice. We can use either the dense adjacency matrix A E Rax" or sparse edge index I, E 22xm The diffusion model (Jo al. 2022) on graphs prefers the former, which is more straightforward for graph generations. The prediction model prefers the latter because of its flexibility, and less computational cost and time. The transformation between two types of graph structure representation takes additional time. Particularly for molecular graphs, the node features used for generation (one-hot encoding of the atom type) and for prediction (see the official package of OGBG for details) are different, which introduces extra time to process the graph data. For details, (I) first need to extract discrete node attributes given the atom type and its neighborhoods: (2) we then need to use an embedding table to embed node attributes in a continuous embedding space; (3) the embedding features of nodes with their graph structure are inputted into the graph neural networks to get the latent representation for nodes. The reverse process for data augmentation in DCT may need repeatedly process graph data with steps (1) and (2). It introduces additional time. To address these technical problems, we build
is the number of edges. There are two ways for G to represent
lookup table to directly map the atom type to the node embedding. We average the node attributes for the same type of node within the batch. We then use the continuous node attributes as weights to average the corresponding node embedding according to the table.
Instantiations of SDE on Graphs According to Song et al. (2021). we use the Variance Exploding (VE) SDE for the diffusion process. VE SDE is:
Given the minimal noise Tmia and the maximal noise Tmaxs the
maax
Tmax dw, t [0.1
dG = miin
2 log
(11)
amin
Tmin
The perturbation kernel is derived (Song al. 2021) as:
2r t (0,
Poc (O)(
G())
N
(()), (0))
o
Tmax
11)
(12)
Tmin
On graphs, we follow Jo et al. (002
to separate the perturbation of adjacency matrix and node
features:
Po (G
G 00) = Pox A (0)
A
(0)
Par
X (1)
X (0)).
(13)
16

---

The Sampling Algorithm in the Reverse Process for Graph Data Augmentation We adapt the Predictor-Correet Corrector (PC) samplers for the graph data augmentation in the reverse process. The algorithm is shown in Algorithm
Algorithm 1 Diffusion Based Graph Augmentation with PC Sampling
Input: Graph G with node feature X and adjacency matrix A, the denoising function for node feature sx and adjacency matrix SA. the fine-tune loss Lang. Lagcvnin MCMC step size S, scaling coefficient 61
(D) + A + ZA;
2, ~ NO I)
( + X + zx;
zx (O,T))
for f D -
do
GQ++; pou+1 (G++n|/ (inner-loop sampling with another PC sampler)
SA = { 1+ 1- -
L A(t) Lang (G(1-)))
Sx = +xx( - 2aVx x(o) Laug (
(G(+1))
A <- A + =(t)aA::
3 (0, T) (Predictor for adjacency matrix)
( 4- x(++) +g())xx gt)zzx:
zx ~ (O,T) (Predictor for node features)
A 4- A S tay
ZA (O,, [Corrector for adjacency matrix)
X + X
2
Sx +ay B2x;
zx (O, I) (Corrector for node features] ]
end for
return G' = (A x
The Algorithm of the Framework The algorithm of the proposed data-centric knowledge transfer framework is presented in Algorithm / and Algorithm 3
In detail, Algorithm 2 corresponds to
Section 3.2 and Algorithm corresponds to Section 3.3
Algorithm 2 The Data-Centric Knowledge Trans- Algorihmt The Data-Centric Knowledge Transfer fer Framework: Learning from Unlabeled Graphs Framework: Generating Task-specific Labeled Graphs
Given unlabeled graphs from the space glU1, randomly initialized score models sx and SA for node feature and graph adjacency matrix, respectively, the total diffusion time step T.
Input:
Given task & with the graph-label space (G.V), V), a randomly initialized prediction model fe. the well- trained score model s = (sx, Ss), the training data set {G,, 1 L.Nr total training epoch e, the hyper-paramtere n for current epoch e; from to do
Input:
while sx and SA not converged
Sample G (X, A) GlUI
Train fo on current training data (G., y. M
Sample Uniform (1, 2,...,
if e, is divisible by the augmentation interval then
Sample 2, ~ (0, I)
Select n graph-label pairs with the lowest training loss from {G., vs N.N.
Sample zx ~ (0,
Sample G with t, Z,, Zx and Eq. (13
Get the augmented examples {G', v/32 by Algo- rithm [oom[iiit with the selected examples
Optimize SA with the gradient:
ViIZa SA (G, 1)
Update {.,, with (G), e.g. add [G', v/} (G., M) No.
Optimize Sx with the gradient:
V 1x sx(G,
end if end for
end while
C Additional Experiments Set-ups
We perform experiments on 15 datasets, including eight classification and seven regression tasks from chemistry, material science, and biology. We use Area under the ROC curve (AUC) for classification performance and mean absolute error (MAE) for regression.
C.1 Molecule Classification and Regression Tasks
Seven molecule classification and three molecule regression tasks are from open graph benchmark (Hu) et al. 2020). They were originally collected by MoleculeNet (Wu et al.) (2018) and used to predict molecule properties. They include (1) inhibition to HIV virus replication in ogVg-HIV, (2) toxicologi- cal properties of 617 types in ogbg-ToxCast, (3) toxicity measurements such as nuclear receptors and stress response ogbg-Toxt Tox21, (4) blood-brain barrier permeability in ogbg-BBBP, (5) inhibition to
human S-secretase 1 in ogbg-BACE, BACE, (6) FDA approval status or failed clinical trial in Lebg-ClinTor.
17

---

(7) having drug side effects of 27 system organ classes in Debg-SIDER (8) predicting the property of lipophilicity in ogbg-Lipo. (9) predicting the water solubility (log solubility mols per litre) from chemical structures in ogbg-FSOI., (10) predicting the hydration free energy of molecules in water in ogbg-FreeSolv. For all molecule datasets, we use the scaffold splitiing procedure as the open graph benchmark adopted (Hu et al., 2020). It attempts to separate structurally different molecules into different subsets, which provides a more realistic estimate of model performance in experiments (Wu ct al. 2018).
C.2 Polymer Regression Tasks
Four polymer regression tasks include GlassTemp, MeltingTemp. ThermCond, and O2 They used predict different polymer properties such as glass transition lemperature ()C), melting temperature ("C), thermal conductivity (W/m)) and oxygen permeability (Barrer). GlassTemp and MeltingTemp are collected from PolyInfo, which is the largest web-based polymer database (Otsuka) et al. 2011). The ThermCond dataset is from molecular dynamics simulation and is an extension from the dataset used in (Ma et af., 2022). The OyPerm dataset created from the Membrane Society of Australasia portal, consisting of a variety of gas permeability data (Thortoo et al.) 2012). Since polymer is built from repeated units, researchers often use single unit graph with polymerization points polymer graphs to predict properties. Different from molecular graphs, two polymerization points are two special nodes (see "*" in Figure 2). indicating the polymerization of monomers (Cormack and Elorza, 2004). For all the polymer tasks, we randomly split by 60% 00%/10%%30% for training, validation, and test.
C.3 Protein Classification Task
An additional task is protein function prediction using protein-protein interaction graphs (Hu et al., 2019). A node is a protein without attributes, an edge is a relation type between two proteins such co-expression and co-occurrence. our DCT, treat all the relations the undirected edge without attributes.
C.4 Baselines and Implementation
When implementing GIN (Xu et al. 2019), we tune its
for different tasks with an early stop the validation set. We generally implement pre-training baselines following their own setting. For molecule and polymer property prediction and protein function prediction, the pre-trained GIN models with self-super tasks such EDGEPRED. ATTRMASK, CONTEXTPRED in (Hu ct al. 2019), INFOMAX (Velickovic et al. 2019) are available. So we directly use them. For other self-supervised methods, implement their codes with default hyper-parameters, parameters. Following their settings, we use 2M ZINC15 (Sterling and Irwin 2015) to pre-train GIN models for molecule and polymer property prediction. We use 306K unlabeled protein-protein interaction ego-networks (Hu et 2019) to pre-train the GIN for the downstream protein function property prediction. For self-training with real unlabeled graphs and INFOGRAPH (Sun al. 2020), we use 113K QM9 (Ra- makrishnan et al. 2014). For self-training with generated unlabeled graphs, we train the diffusion model (Jo et al. 2022) on the real QM9 dataset and then produce the same number of generated
unlabeled graphs. To train the diffusion model in our DCT, we also use QM9 (Ramakrishnan et al. 2014).
D Additional Experiment Analysis
D.1
The Power of Diffusion Model to Learn from Unlabeled Graphs
In T] when we replace the 133K QM9 with the 249K ZINC (Jo al. (022) to train the diffusion model, which nearly doubles the size of the unlabeled graphs and includes more atom types, we do not observe any additional improvement, and some cases, even worse performance. It is possible because of the constraint of the current diffusion model's capacity to model the data distribution for much larger number of more complex graphs. It encourages powerful generative models in the future, which could be directly used to benefit predictive models under the proposed framework.
18

---

D.2 Chemical Validity of the Generated Graphs in Downstream Tasks
In Figure 6 we show through some examples that concepts, such as certain chemical rules from the unlabeled graphs, are successfully transferred to downstream tasks. To further validate this point, we gathered 1,000 task-specific graphs generated in the intermediate steps on the tasks of ogbg-BAc BACE, ogbg-BBBP, gebg-FreeSolv, and OePer.. We then assessed the chemical validity of these graphs and observed that the validity 92.8%, 87.9%, 97.4%, and 62. I 1%, respectively. Results show that transferring knowledge from pre-trained molecular data to target molecules yields relatively high chemical validity. However, the validity drops to 62% when transferring knowledge from pre-trained molecular data to target polymer data. This finding indicates that the transferability of chemical rules becomes more challenging when the distribution gap between pre-training data and downstream task data is larger.
19


Reviews:
[{"review_id": "P60SEZWf0M", "reviewer": "Reviewer_6AKL", "summary": "This paper studies leveraging unlabeled graphs to aid the supervised graph learning tasks with limited labeled graphs. This paper starts with a motivation that previous self-supervised methods may conflict with the downstream task and lead to poor performance. This paper proposes to model the graph distribution with diffusion models and train the diffusion model jointly with the downstream prediction model. The proposed method optimizes the prediction loss of the augmented graphs and maximizes the difference between the augmented graph and the original labeled graph. This paper evaluates the proposed method on 15 graph property prediction tasks, including seven molecule classification, three molecule regression tasks, four polymer regression tasks, and one protein function prediction task. Compared to previous self-supervised learning, semi-supervised learning, and data augmentation approaches, the proposed method shows significant improvement over previous methods.", "questions": "- The proposed method uses a diffusion model to first learn graph distribution from unlabeled graphs. Is the diffusion model trained for each task? Or is a joint diffusion model trained and applied to every task? \n- What dataset is the diffusion model trained on in the experiments? \n- Is the prediction model trained on both originally labeled graphs and augmented graphs in the downstream tasks? How would the performance be if the method also trained on the original labeled graphs? \n- What would be the perturbation scale between originally labeled graphs and augmented graphs, which leads to the best performance? ", "limitations": "This work has discussed the limitations of the proposed method. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper leverage diffusion models to learn the distribution of unlabeled graph and use the model to augment labeled graph for downstream tasks. In contrast to previous self-supervised learning methods, the proposed method is more effective in terms of adapting to each downstream task. \n- Empirical evaluations are extensive. This paper provides a broad range of experiments that demonstrate the advantage of the proposed method over 15 previous methods on various datasets. This paper also presents ablation studies and case studies that show the working of the proposed method. ", "weaknesses": "Details of the proposed method need to be clarified, such as the training of the diffusion model and implementation details. Please see Questions for more details. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on the model details. Below, we offer detailed responses to each of your points.\n\n### To question 1 and 2 about the training of the diffusion model and dataset\n\nWe train one diffusion model for 14 molecule and polymer tasks on the unlabeled QM9 datasets. \n\nWe consider another task on protein-protein interaction (PPI) graphs. These graphs have proteins as nodes and protein-protein relations as edges, whereas molecular graphs have atoms as nodes and bonds as edges. It is not reasonable to transfer the knowledge about molecular graph structure to protein-protein interaction structure. Therefore, another diffusion model is trained on unlabeled PPI graphs for the downstream PPI tasks. \n\nMore details could be found in the baselines and implementation paragraph (lines 240-252). \n\n### To question 3 about the training of the downstream prediction model\n\nWe adopt an iterative approach to train the downstream prediction model and create augmented graphs. As a result, the prediction model is trained on both the original labeled graphs and the newly generated labeled graphs from the second iteration onwards. More details are in lines 212-227. The GIN model serves as the backbone, and we also report its performance in Table 1 as it is trained solely on the original labeled graphs. \n\n### To question 4 about the perturbation\n\nThank you for your insightful questions. We use the hyper-parameter, the perturbation step $D$, to represent the scale of perturbation applied to the augmented graphs. To analyze the sensitivity of this hyper-parameter, we conducted experiments with $D$ ranging from 1 to 10, as illustrated in the top figure of Figure 4. We observed that DCT remains robust across a broad range of $D$ values, and we recommend setting $D$ to 5 as a default value. Further details on this analysis can be found in Section 5.3, specifically in lines 300 to 301 of the paper.\n"}, {"review_id": "o4cGbtuvBJ", "reviewer": "Reviewer_T9FT", "summary": "The paper leverages diffusion SDE to approach the hidden chemical space and help the prediction tasks of molecular properties.", "questions": "See above", "limitations": "NA", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is a good idea to leverage mutual information for the alignment of the hidden chemical space within both labeled and unlabeled graphs.\n2. The results presented in this study seem to exhibit superior performance when compared to other methods examined in the paper, which is an encouraging observation.", "weaknesses": "1. The paper could benefit from greater originality. The primary concepts echo closely those found in references [1] and [2]. For instance, [1] utilized generative models to boost the prediction performance for an array of graph datasets, while [2] applied a score matching method to uncover the hidden chemical space, thereby facilitating active learning in the predictive task of molecular properties. I recommend an expanded discussion within the paper to clearly delineate the distinctions between the current study and the methods in references [1] and [2]. This would more effectively highlight the unique contributions of this work.\n\n2. The paper appears to have an insufficiency in its literature review section. It would greatly benefit from an evaluation of self-supervised learning-based molecular pre-training models as discussed in references [3], [4], and [5]. These studies could provide significant context and further inform the study's approach. \n\n3. The paper reports an incremental advancement in performance. Upon reviewing the OGB leaderboard [6], the results presented for ogbg-molhiv do not appear to supersede those achieved by SOTA methods currently leading on the leaderboard. Moreover, it seems that the performance does not even align favorably with the results presented in references [3], [4], and [5]. The authors are encouraged to provide an explanation or delve into the possible reasons behind this observed discrepancy in performance levels.\n\n[1] Liu, Songtao, et al. \"Local augmentation for graph neural networks.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Zhou, Kuangqi, et al. \"Jointly Modelling Uncertainty and Diversity for Active Molecular Property Prediction.\" Learning on Graphs Conference. PMLR, 2022.\n\n[3] Zhang, Zaixi, et al. \"Motif-based graph self-supervised learning for molecular property prediction.\" Advances in Neural Information Processing Systems 34 (2021): 15870-15882.\n\n[4] Rong, Yu, et al. \"Self-supervised graph transformer on large-scale molecular data.\" Advances in Neural Information Processing Systems 33 (2020): 12559-12571.\n\n[5] Xu, Minghao, et al. \"Self-supervised graph-level representation learning with local and global structure.\" International Conference on Machine Learning. PMLR, 2021.\n\n[6] https://ogb.stanford.edu/docs/leader_graphprop/#ogbg-molhiv", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely value your feedback. We respectfully and firmly defend the originality, the literature review, and a fair evaluation of the performance of our model. We begin by offering a concise overview of our rebuttal, followed by a comprehensive point-by-point elaboration.\n\n### Summary\n($\\text{The first six references strictly adhere to the order used by the reviewer}$.)\n\n1. Originality: Our novelties and motivations in lines 43-49: transfer learning by data augmentation, are essentially different from the reference [1,2]. Another clear difference is that we focused on graph property prediction with automatic knowledge transfer from unlabeled graphs, while [1] focused on node classification and [2] focused on active learning.\n\n2. Literature review: **We have already compared both methods** from the reference [3] (MGSSL) and from the reference [5] (GraphLoG) in our reported results in Table 1 (the 6th and 7th lines in the self-supervised part) and **particularly reviewed the disadvantages** of MGSSL [3] in lines 28-33. The reference [4] is found not as good as the MGSSL [3] in the study of the reference [3].\n\n3. Model performance: To ensure unbiased conclusions, we need (and have conducted) a fair evaluation, which should exclude complex training techniques like domain knowledge fingerprints and hybrid models. These techniques are broadly used in the ogbg-HIV leaderboard. \n\nWe give a detailed response to support our summary as follows.\n\n### To question 1 about originality:\n\nOur major novelties: transfer learning by data augmentation is essentially different from the reference [1,2]. \n\nWe focus graph-level knowledge transfer, whereas [1] concentrates more on node-level tasks where the local graph structure holds greater importance. As indicated in the reported performance from [1] on the graph-level task ogbg-HIV, which stands at 76.18, it appears to be lower than our GIN implementation which achieves 77.4. These results highlight the distinctions between node-level and graph-level approaches. For example, we find that local structures like aromatic rings could be misleading (see Lines 28-33 in our paper). Therefore, the gap between unlabeled distribution and task label distribution for graph data objectives is more important to address in our task.\n\nThe work [2] focused on: active learning for molecular property prediction. Active learning needs an oracle such as chemists, while we focus on automatically discovering useful knowledge from massive unlabeled graphs and transferring them to downstream tasks.\n\nIn summary, the work [1,2] is interesting. We use generative modeling techniques in common, but it is crucial and undeniable to emphasize that our main focus lies in different areas, as described earlier. We will add extended discussions for potential connections and applications about node-level tasks and active learning. However, as demonstrated above, our originality clearly distinguishes it from [1,2].\n\n### To question 2 about the lack of literature review and evaluation\n\nWe would like to politely clarify that \\textbf{we have already compared} both methods (MGSSL) [3] and (GraphLoG) [5] and particularly **reviewed** the disadvantages of MGSSL [3]. You can find our discussion on MGSSL [3] in lines 27-33 and a comparison with [3] and [5] in Table 1 (the 6th and 7th lines in the self-supervised part). The work [4] was proposed in 2020 and has been beaten by MGSSL, which was proposed in 2021. We would like to prefer more advanced techniques for comparison to leave more page space to comprehensively study the effectiveness of the proposed framework. However, we remain open to expanding our discussion for the reference [4], if it holds the potential to yield a broader impact on our research.\n\n### To question 3 about OGBG leaderboard\n\nIt is worth noting that the top-performing approaches on the ogbg-hiv leaderboard are the result of combining various training tricks and incorporating domain knowledge. Here are a few examples: (1) Fingerprint. It consists of domain knowledge rule extraction with expertise knowledge; (2) Advanced graph neural network (GNN) structures; (3) Hybrid models from different machine learning models and GNN models. Most of the existing graph pre-training models [5,7,9] do not include these tricks. We want to evaluate the model in line with these pre-training baselines. Besides, developing our method following the simplest setting helps avoid drawing spurious conclusions attributed to more complex models. In a fair evaluation, our method demonstrates notable improvements compared to 15 state-of-the-art baselines from self-supervised learning, semi-supervised learning, and graph data augmentation. Specifically, we achieve a reduction of 13.4\\% in mean absolute error for the molecule graph regression task and 10.2\\% for the polymer graph regression task.\n\n### To question 3 about the reported performance in [3,4,5]\n\nWe want to emphasize that direct comparisons between the reported numbers in our paper and those in papers [3,4,5] are not fair. The reason for the difference in reported numbers is due to different data splitting and underlying prediction models used in the evaluation process. While we generally follow the scaffold splitting method, the results may differ with different random seeds [7,8]. Additionally, in the work [4], three random-seeded scaffold splittings were used with the graph transformer as the backbone. These differences make direct comparisons with reported numbers in different papers impossible. In our paper, we ensure a unified and fair evaluation setting by implementing all baselines following [8].\n\n### Reference\n\n($\\text{The first six references strictly adhere to the order used by the reviewer}$.)\n\n[7] Strategies for pre-training graph neural networks. ICLR 2020.\n\n[8] Open graph benchmark: Datasets for machine learning on graphs. NeurIPS 2020.\n\n[9] Graph self-supervised learning with accurate discrepancy learning. NeurIPS 2022."}, {"review_id": "WLMDhD0eup", "reviewer": "Reviewer_trSH", "summary": "This paper proposes a new data augmentation method called DCT based on unlabeled graphs. The proposed DCT generates new augmented graphs using a diffusion model from unlabelled graphs and also with labeled graphs from down-stream tasks as guidance. Two specific objectives are proposed to guide the model's denoting process to generate task-specific graph examples and their corresponding labels. Experimental results demonstrate the superior performance of the proposed method.", "questions": "1. Does your method suffer from overfitting? In Figure 4, the performance of the proposed method seems to decrease as the number of selected labeled graphs for data augmentation increases. Although it is beneficial to consider downstream tasks when augmenting graphs, the potential overfitting problem should be discussed.\n2. How is the diffusion and reverse process applied to both graph structure and node features? Do you perturb different types of noise to graph structure/node features? More detailed discussions are necessary.\n3. In Definition 4.2, how to define \\bar{G}?", "limitations": "The authors have not discussed potential limitations including societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. A new graph augmentation method is proposed based on diffusion models developed in the computer vision field. The exploration of diffusion models on graphs is relatively new.\n2. The proposed method adopts two task-specific objectives, enabling to learn task-specific knowledge and generate task-related labeled graphs. This mechanism has been shown effective in addressing the shortcomings of self-supervised learning self-training methods. \n3. The experimental evaluation and ablation studies demonstrate the promising performance of the proposed method on graph property prediction tasks. \n", "weaknesses": "1. The idea of using diffusion models to generate graphs is borrowed from the field of computer vision, where the diffusion and reverse process is more intuitive. However, it remains unclear how it can be applied to both graph structure and node features. \n2. Some relevant literature review, e.g., on pseudo-labelling, is missing in related work.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback model details and literature review. We offer detailed responses to each of your points.\n\n### To weakness 1 and question 2 about the diffusion model on graphs\n\nWe follow [1,2] to define graph diffusion models. [2] unified score-based generative models and denoising diffusion probabilistic models, using stochastic differential equations (SDE) in continuous state spaces. [1] extended [2] to graphs. Besides [2], there are many recent work that has verified the effectiveness of the diffusion models on graphs [3,4,5]. We also want to point out that, differing from the existing work [1,3,4,5], which mainly focused on advancing the generative performance of the diffusion model, we propose to leverage the diffusion model for predictive tasks via data augmentation and knowledge transfer. Here, we present a more careful review to discuss recent advances for diffusion models on graphs.\n\nGiven a graph data point $G \\in \\mathcal{G}$ in the graph space with the adjacency matrix $\\mathbf{A} \\in \\mathbb{R}^{N \\times N}$ and the node feature matrix $\\mathbf{X} \\in \\mathbb{R}^{N\\times F}$, where $N$ is the number of nodes and $F$ is the dimension of node features, [1] extended the above SDE to the graph data $G \\in \\mathbb{R}^{N\\times N} \\times \\mathbb{R}^{N\\times F} $ (see our Eq. (3) and (4) in lines 152-167). Following [1], we perform functions on graphs, such as $\\mathbf{f}(\\cdot,t)$ in Eq. (3) and (4), separately on the node feature variable $\\mathbf{X}$ and adjacency matrix variable $\\mathbf{A}$. The marginal distribution of the graph probability $p_t(G^{(t)})$ at time step $t$ is also defined from two aspects: the node feature $\\mathbf{X}$ and the adjacency matrix $\\mathbf{A}$. The ground-truth gradient for the probability or $\\log p_t(G^{(t)})$ is estimated by two graph neural networks: one estimates $\\nabla_{\\mathbf{X}^{(t)}} \\log p_t (\\mathbf{X}^{(t)}, \\mathbf{A}^{(t)})$ for node features, and the other estimates $\\nabla_{\\mathbf{A}^{(t)}} \\log p_t (\\mathbf{X}^{(t)}, \\mathbf{A}^{(t)})$ for adjacency matrices. These neural networks are trained by score matching methods [2]. We kindly note that certain descriptions are available in our paper, particularly in Section 4.2. Unfortunately, due to the page limit, many details were moved to the appendix such as the B.3 section. For more details, including equations, instantiations, and algorithms, we kindly direct reviewers to the appendix. Additionally, we will add more details to the main text.\n\n### To weakness 2 about the literature review on pseudo-labeling\n\nThanks for your suggestion. Pseudo-labeling is a common technique for semi-supervised learning [6,7]. It first trains a model that iteratively assigns pseudo-labels to the set of unlabeled training examples. The pseudo-labeled examples are then used to enrich the labeled training set. And the model continues training with the updated training set. Many studies have explored improving uncertainty estimation [8,9,10] to help the model filter out noise for reliable pseudo-labels. Recently, pseudo-labels have been applied in imbalanced learning [11] and representation learning [12]. Pseudo-label methods are restricted to confidently predictable labels and may ignore the huge number of other unlabeled data.\n\n### To question 1 about performance drop in Figure 4 when increasing top-$n$\n\nThe performance drop from increasing augmented graphs per iteration could have multiple reasons. A key factor is the noisy graphs introduced with a high hyper-parameter $n$ in Figure 4 (e.g., > 30\\%). We use an iterative process for training and augmented graph creation, expecting mutual enhancement (confirmed in Figure 5). As the downstream prediction model isn't perfect during augmented graph creation, we select only top-$n$ lowest property prediction loss to prevent noisy graph generation. When the top-$n$ is very high, like larger than 30\\%, the noisy graphs would cause the performance drop as shown in Figure 4. We have discussed this phenomenon in lines 301-304 and lines 224-227.\n\nRegarding the concern of overfitting, we have briefly talked about it in lines 178-183. We discussed the balance between task-relatedness and diversity in generating augmented graphs. Limited diversity can lead to overfitting, thus we employ minimality (definition 4.2) to enhance augmented graph diversity.\n\n### To question 3 about the explanation of Definition 4.2\n\nThank you for your comments. Here we provide more details.$\\forall \\bar{G}$ represents any augmented graph that sufficiently preserves the original graph's label (as defined in Definition 4.1). We will add these details to clarify this definition in the paper.\n\n### Reference \n\n[1] Score-based generative modeling of graphs via the system of stochastic differential equations. ICML 2022.\n\n[2] Score-based generative modeling through stochastic differential equations. ICLR, 2021.\n\n[3] Autoregressive Diffusion Model for Graph Generation. ICML 2023.\n\n[4] Efficient and Degree-Guided Graph Generation via Discrete Diffusion Modeling. 2023 ICML.\n\n[5] Digress: Discrete denoising diffusion for graph generation. ICLR 2023.\n\n[6] Label propagation for deep semi-supervised learning. CVPR 2019.\n\n[7] Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop at ICML 2013.\n\n[8] Dropout as a bayesian approximation: Representing model uncertainty in deep learning. ICML 2016.\n\n[9] Single-model uncertainties for deep learning. NeurIPS 2019.\n\n[10] Deep evidential regression. NeurIPS 2020.\n\n[11] Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. CVPR 2021.\n\n[12] Multi-task self-training for learning general representations. CVPR 2021.\n"}, {"review_id": "fBqFB5jUJ8", "reviewer": "Reviewer_3gzg", "summary": "The paper discusses the problem of graph property prediction tasks, where there is a limited amount of labeled data and a large set of unlabeled graphs available. The conventional approach is to train a model using self-supervised tasks on the unlabeled graphs and then fine-tune it for prediction tasks. However, the knowledge gained from self-supervised tasks may not align with the requirements of the prediction tasks.\n\nTo address this issue, the authors propose a data-centric approach that extracts knowledge from the unlabeled graphs to enhance each property prediction model. They introduce a diffusion model to utilize the unlabeled graphs fully and two new objectives that guide the model's denoising process and generate task-specific graph examples and labels. Specifically, the first objective minimizes an upper bound of mutual information between the augmented and the original graphs in the graph space. The second objective maximizes the probability of the predicted label of augmented graphs being the same as the label of original graphs. \n\nExperimental results demonstrate that the proposed data-centric approach outperforms fifteen existing methods on various graph property prediction tasks. The improvement in performance is attributed to the generated labeled examples, which are more effective than self-supervised learning. The authors also discuss related work in graph property prediction and learning from unlabeled data, highlighting the limitations of existing methods and the need for a data-centric approach. They present their framework (DCT) in detail and provide results showing its superior performance compared to baseline methods used for comparison.", "questions": "1. In lines 46-48, the generation of labeled examples is part of the property prediction task pipeline. Please describe how to use these generated task-specific labeled examples to bridge the gap between the final prediction and the generated graph.\n\n2. In lines 60-61, it is described that the trained predictor model f is used in the first objective. How is the predictor trained? If it is trained on a small amount of labeled data, then why is the not-well-trained predictor (as stated in Line 119) able to help with graph generation?\n\n3. Can other graph diffusion model methods be applied to the graph property prediction task with trivial adaptation? If not, why?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method offers a novel approach to graph property prediction by employing a diffusion model, setting it apart from traditional techniques.\n\n2. Empirically, the introduced method surpasses other techniques in various areas including self-supervised learning, semi-supervised learning, and graph data augmentation.", "weaknesses": "1. The discussion of the graph diffusion model is insufficient. The proposed method relies on the graph diffusion model as its backbone technique. A clear discussion of the similarities and differences between this work and other graph diffusion model methods [1,2] can clarify the contribution of this work.\n\n2. The interpretability of \"visible\" knowledge transfer from unlabeled graphs is interesting, but only several case studies cannot support the claim that \"concepts such as some chemical rules from the unlabeled graphs are successfully transferred to downstream tasks\" (lines 318-319).\n\n\n\n[1] Jo, Jaehyeong, Seul Lee, and Sung Ju Hwang. \"Score-based generative modeling of graphs via the system of stochastic differential equations.\" In *International Conference on Machine Learning*, pp. 10362-10383. PMLR, 2022.\n\n[2] Vignac, Clement, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. \"DiGress: Discrete Denoising diffusion for graph generation.\" *arXiv preprint arXiv:2209.14734* (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### To weakness 1 about the literature review on graph diffusion model\n\nThanks for your comments. Here are discussions on recent advances for diffusion models on graphs.\n\nRecent works have improved the generative modeling of diffusion models on graphs [1,2,3,4,5]. EDP-GNN [1] employed score matching for permutation-invariant graph data distribution. GDSS [2] extended continuous-time framework [6] to model node-edge joint distribution. DiGress [3] used the transition matrix to preserve the discrete natures of the graph structure. GraphARM [4] introduced a node-absorbing autoregressive diffusion process. EDGE [5] focused on efficiently generating larger graphs. Instead of improving the generation capacity of the diffusion model, our model builds on [2,6] for predictive tasks, i.e., graph classification and graph regression. Our focus is on improving the transfer of knowledge from unlabeled graphs to various downstream tasks with the diffusion model.\n\nWe will add the above discussion to the main text.\n\n### To weakness 2 about lines 318-319\n\nThank you for your valuable suggestions. To validate our hypothesis regarding the transfer of chemical rules from unlabeled graphs to downstream tasks, we gathered 1,000 task-specific graphs generated in the intermediate steps. We then assessed the chemical validity of these graphs on two classification tasks (ogbg-BACE and ogbg-BBBP) and two regression tasks (ogbg-FreeSolv and O$_2$Perm). The outcomes of these evaluations are summarized in the table below.\n\n|              |  BACE   | BBBP    | FreeSolv | O$_2$Perm |\n|--------------|---------|---------|----------|--------|\n| Validity (\\%) | 0.928   | 0.879   | 0.974    | 0.621  |\n\nFirst, we observe that the validity is close to 90\\% in most cases, validating our claim. Furthermore, transferring knowledge from pre-trained molecular data to target molecules has relatively high chemical validity. However, the validity drops to 62\\% when transferring knowledge from pre-trained molecular data to target polymer data. This finding indicates that the transferability of chemical rules becomes more challenging when the distribution gap between pre-training data and downstream task data is larger. We will incorporate these findings into the main text accordingly.\n\n### To question 1 about lines 46-48\n\nThanks for your suggestions. We described our high-level ideas in lines 43--49. Technical details about the implementation of our idea could be found in lines 50-65 and all of Section 4. Specifically for your question for lines 46-48, we break the question into two parts: (1) how to generate graphs to close the gap between the unlabeled graphs and a specific task (2) how to use the generated task-specific graphs. A brief discussion on (1) could be found in lines 56-64 and details could be found in Section 4.3 (Lines 169-211). A brief discussion on (2) could be found in lines 64-65 and details could be found in Section 4.3 (Lines 212--226)\n\nHere we summarize the important points. For (1), we use two task-related losses to guide the generation of the diffusion model. So, the generated graphs are task-specific. For (2), we use an iterative process to alternatively generate task-specific graph data and train the downstream prediction model. The model is first trained on all the downstream labeled data and then the partially trained model is used to guide the generation of task-specific generated graphs. Then we start a new iteration and repeat training the model.\n\n### To question 2 about lines 60-61\n\nThanks for your question, as stated in lines 64-65:\n\n> DCT iteratively generates new examples to augment the labeled dataset and progressively trains the prediction model with it.\n>\n\nWe will iteratively update the downstream prediction model ($f$). We hypothesize that model training and task-specific data generation mutually enhance each other. This assumption is common in self-training [7,8,9,10]. We also empirically validated this assumption in Figure 5. Details are in lines 305-311.\n\n### To question 3 about the use of other diffusion models\n\nWe adopt the general framework [2,6] for continuous-time diffusion modeling via stochastic differential equations (SDE). Practical implementation requires discretizing SDE in both the diffusion and reverse process, yielding various model instances like SMLD [11] and DDPM [12]. Our framework could accommodate these diffusion models and their extensions with minimal adjustments. We note that the framework [2,6] uses Gaussian noise for data perturbation and noise estimation in reverse processes, while DiGress [3] employs discrete noise, predicting clean graphs from perturbed ones. Corresponding code adjustments are needed for different noise types and model outputs.\n\n### Reference\n\n[1] Permutation invariant graph generation via score-based generative modeling. AISTATS, 2020.\n\n[2] Score-based generative modeling of graphs via the system of stochastic differential equations. ICML 2022.\n\n[3] Digress: Discrete denoising diffusion for graph generation. ICLR 2023.\n\n[4] Autoregressive Diffusion Model for Graph Generation. ICML 2023.\n\n[5] Efficient and Degree-Guided Graph Generation via Discrete Diffusion Modeling. ICML 2023.\n\n[6] Score-based generative modeling through stochastic differential equations. ICLR 2021.\n\n[7] Label propagation for deep semi-supervised learning. CVPR 2019.\n\n[8] Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop at ICML 2013.\n\n[9] Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. CVPR 2021.\n\n[10] Multi-task self-training for learning general representations. CVPR 2021.\n\n[11] Generative modeling by estimating gradients of the data distribution. NeurIPS 2019.\n\n[12] Denoising diffusion probabilistic models. NeurIPS 2020."}]

Author rebuttal:

Meta-review: This paper leverage diffusion models to learn the distribution of unlabeled graph and use the model to augment labeled graph for downstream tasks. The reviewers acknowledged the completeness of the extensive experimental study, although showed slight concerns about originality and a more thorough literature review.
Decision: Accept (poster)