Iterative Graph Self-Distillation
Authors: Hanlin Zhang
Shuai Lin
Weiyang Liu
Pan Zhou
Jian Tang
Xiaodan Liang
Eric Xing
Venue:  (2021)
Keywords: graph-level representation learning
knowledge distillation

Abstract: How to discriminatively vectorize graphs is a fundamental challenge that attracts increasing attentions in recent years. Motivated by the recent success of unsupervised contrastive learning, we aim to learn graph-level representation in an unsupervised manner. Specifically, we propose a novel unsupervised graph learning paradigm called Iterative Graph Self-Distillation (IGSD) which iteratively performs the teacher-student distillation with graph augmentations. Different from conventional knowledge distillation, IGSD constructs the teacher with an exponential moving average of the student model and distills the knowledge of itself. The intuition behind IGSD is to predict the teacher network representation of the graph pairs under different augmented views. As a natural extension, we also apply IGSD to semi-supervised scenarios by jointly regularizing the network with both supervised and unsupervised contrastive loss. Finally, we show that finetuning the IGSD-trained models with self-training can further improve the graph representation power. Empirically, we achieve significant and consistent performance gain on various graph datasets in both unsupervised and semi-supervised settings, which well validates the superiority of IGSD.

Under review as a conference paper at ICLR 2021
ITERATIVE GRAPH SELF-DISTILLATION
Anonymous authors Paper under double-blind review
ABSTRACT
How discriminatively vectorize graphs is a fundamental challenge that attracts increasing attentions in recent years. Inspired by the recent success of unsupervised contrastive learning. we aim to learn graph-level representation in an unsupervised manner. Specifically, we propose a novel unsupervised graph learning paradigm called Iterative Graph Sift-Distillaiinn (IGSD) which iteratively performs the teacher-student distillation with graph augmentations. Different from conventional knowledge distillation, IGSD constructs the teacher with an exponential moving average of the student model and distills the knowledge of itself. The intuition behind IGSD to predict the teacher network representation of the graph pairs under different augmented views. As natural extension, we also apply IGSD semi-supervised scenarios by jointly regularizing the network with both supervised and unsupervised contrastive loss. Finally, we show that finetuning the IGSD- trained models with self-training can further improve the graph representation power. Empirically, we achieve significant and consistent performance gain on various graph datasets in both unsupervised and semi-supervised settings, which well validates the superiority of IGSD.
INTRODUCTION
Graphs are ubiquitous representations encoding relational structures across various domains. Learning low-dimensional vector representations of graphs is critical in various domains ranging from social science (Newman & Girvan 2004) to bioinformatics (Duvenaud et al. 2015 Zhou et al. 2020p. Many
graph neural networks (GNNs) (Gilmer al.] [017]] Kipf & Welling [016] Xu et al., 2018)) have been proposed to learn node and graph representations by aggregating information from every node's neighbors via non-linear transformation and aggregation functions. However, the key limitation of existing GNN architectures is that they often require huge amount of labeled data to be competitive but annotating graphs like drug-target interaction networks challenging since it needs domain- specific expertise. Therefore, unsupervised learning graphs has been long studied, such as graph kernels (Shervashidze et al. 2011) and faatiiiffctooiizaioon approaches (Belkin & Niyogi 2002).
Inspired by the recent success of unsupervised representation learning in various domains like images (Chen et al., [020bb He et al.] (2200) and texts (Radford et al.) 2018), most related works in the graph domain either follow the pipeline of unsupervised pretraining (followed by fine-tuning) InfoMax principle (Hjelm et al. 2018). The former often needs meticulous designs of pretext tasks (Hu al. 2019: You al. 2020) while the latter is dominant in unsupervised graph representation learning, which trains encoders to maximize the mutual information (MI) between the representations of the global graph and local patches (such as subgraphs) (Veličković al. 2018 Sun al. 2019)
Hassani & Khasahmadi 2020). However, MI-based approaches usually need to sample subgraphs as local views to contrast with global graphs. And they usually require an additional discriminator for scoring local-global pairs and negative samples, which is computationally prohibitive (Tschhnnen jet al. 2019). Besides, the performance also very sensitive to the choice of encoders and MI estimators (Tschannen et al. 2019). Moreover, MI-based approaches cannot be handily extended to the semi-supervised setting since local subgraphs lack labels that can be utilized for training. Therefore,
seeking an approach that learns the entire graph representation by contrasting the whole graph directly without the need of MI estimation, discriminator and subgraph sampling.
Motivated by recent progress contrastive learning, we propose the Iterative Graph Deff-Distillation (IGSD), teacher-student framework to learn graph representations by contrasting graph instances directly. The high-level idea of IGSD is based on graph contrastive learning where we pull sim-

---

Under review as a conference paper at ICLR 2021
ilar graphs together and push dissimilar graph away. However, the performance of conventional contrastive learning largely depends on how negative samples are selected. To learn discriminative representations and avoid collapsing trivial solutions, large set of negative samples (He et al.) 2020; Chen et al. 2020b) or a special mining strategy (Schroff et al., 2015 He et al., (020) are necessary. In order to alleviate the dependency negative samples mining and still be able to learn discriminative graph representations, we propose to use self-distillation as a strong regularization to guide the graph representation learning.
In the IGSD framework, graph instances are augmented as several views to be encoded and projected into a latent space where we define a similarity metrie for consistency -based training. The parameters of the teacher network are iteratively updated as an exponential moving average of the student network parameters, allowing the knowledge transfer hetween them, As merely small amount of labeled data is often available in many real-world applications, we further extend IGSD to the semi-supervised setting such that it can effectively utilize graph-level labels while considering arbitrary amounts of positive pairs belonging the same class. Moreover, in order to leverage the information from pseudo-labels with high confidence, we develop a self-training algorithm based on the supervised contrastive loss for fine-tuning:
We experiment with real-world datasets in various scales and compare the performance of IGSD with state-of- affthh--rt graph representation learning methods. Experimental results show that IGSD achieves competitive performance in both unsupervised and semi semi-supervised settings with different encoders and data augmentation choices. With the help of self-training, performance exceed state-of-the-arr baselines by a large margin.
To summarize, we make the following contributions in this paper:
We propose self-distillation framework called IGSD for unsupervised graph-level representation learning where the teacher-student distillation is performed for contrasting graph pairs under different augmented views.
We further extend IGSD to the supervupe scenario, where the labeled data are utilized effectively with the supervised contrastive loss and self-training.
We empirically show that IGSD surpasses state-of- of-te--f-te-art art methods in semi-supervised graph classification and molecular property prediction tasks and achieves performance competitive with state-of-the-art approaches in unsupervised graph classification tasks.
2
RELATED WORK
Contrastive Learning Modern unsupervised learning in the form of contrastive learning can be categorized into two types: context-instance contrast and context-contex. contrast (Liu et al.) 2020). The context-instance contrast, or so-called global-local contrast focuses on modeling the belonging relationship between the local feature of a sample and its global context representation. Most unsupervised learning models on graphs like DGI (Veličković et al. 2018). InfoGrapp (Sun et al. 2019). CMC-Graph (Hassani & Khasahmadi] 2020) fall into this category, following the InfoMax principle to maximize the the mutual information (MI) between the input and its representation. However, estimating MI is notoriously hard in MI-based contrastive learning and in practice tractable lower bound on this quantity is maximized instead. And maximizing tighter bounds on MI result in worse representations without stronger inductive biases in sampling strategies, encoder architecture and parametrization of MI estimators (Tschannen et al. 2019). Besides, the intricacies of negative sampling in MI-based approaches impose key research challenges like improper amount of negative samples or biased negative sampling (Tschannen et al. 2019 Chuang et al. 2020). Another line of contrastive learning approaches called context-context contrast directly study the relationships between the global representations of different samples as what metric learning does. For instance, recently proposed model BYOL (Grill et al. 2020) bootstraps the representations of the whole images directly. Focusing on global representations between samples and corresponding augmented views also allows instance-level supervision to be incorporated naturally like introducing supervised contrastive loss (Khosla et al., 2020) into the framework for learning powerful representations. Graph Contrastive Coding (GCC) (Qiu et al. 2020) is pioneer to leverage instance discrimination as the pretext task for structural information pre-training. However, our work is fundamentally different from theirs. GCC focuses on structural similarity to find common and transferable structural
2

---

Under review as a conference paper at ICLR 2021
patterns across different graph datasets and the contrastive scheme is done through subgraph instance discrimination. On the contrary, our model aims at learning graph-level representation by directly contrasting graph instances such that data augmentation strategies and graph labels can be utilized naturally and effectively.
Knowledge Distillation Knowledge distillation (Hinton al. 2013) is method for transferring knowledge from one architecture to another, allowing model compression and inductive biases transfer. Self-distillation (Furlanello et 2018) is special case when two architectures identical, which can iteratively modify regularization and reduce over-fitting if perform suitable rounds (Mobahi et al. 2020). However, they often focus on closing the gap between the predictive results of student and teacher rather than defining similarity loss in latent space for contrastive learning.
Semi-supervised Learning Modern semi-supervised learning can be categorized into two kinds: multi-task learning and consistency training between two separate networks. Most widely used semi- supervised learning methods take the form of multi-task learning: arg ming g (D,, 0) + (o,,,(,,,. 0) on labeled data Dj and unlabeled data D.. By regularizing the learning process with unlabeled data, the decision boundary becomes more plausible. Another mainstream of semi-supervised learning lies in introducing student network and teacher network and enforcing consistency between them (Tarvainen & Valpolal 2017] Miyato et al., 2019, Lee| 2013). It has been shown that semi- be greatly improved via unsupervised pre-training of (big)
supervised learning performance model, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge (Chen et al., 2020c). However, whether task-agnostic self-distillation would benefit semi-supervised learning is still underexplored.
3 PRELIMINARIES
3.1 FORMULATION
Unsupervised Graph Representation Learning Given a set of unlabeled graphs G {G aim at learning the dow-dimessional representation of every graph G, E 9 favorable for down- stream tasks like graph classification.
Semi-supervised Graph Representation Learning Consider whole dataset G 9r. U 90
composed by labeled data Gz = {G,,, 3:) and unlabeled data Gu [G (usually u >>>>>>>>>>>>/ 1). our goal is to learn model that can make predictions on graph labels for unseen graphs. And with K
augmentations, we get G/ [(G K and Gj = (G) k=1+1 [+++=
K((++)
our training data.
3.2 GRAPHI REPRESENTATION LEARNING
We represent graph instance as G(V, E) with the node set V and the edge set E. The dominant ways of graph representation learning are graph neural networks with neural message passing mechanisms (Hamilton al. 2017): 7 for every node E V. node representation hy is iteratively computed from the features of their neighbor nodes N(w) using differentiable aggregation function. Specifically, at the iteration we get the node embedding as:
h ( CONCAT (h; AGGREGATE, ((h Vu e (()))))
(1)
Then the graph-level representations can be attained by aggregating all node representations using a readout function like summation set2set pooling (Vinyals et al. 2015).
3.3 GRAPH DATA AUGMENTATION
It has been shown that the learning performance of GNNs can be improved via graph diffusion, which serves as a homophily-bar denoising filter on both features and edges real graphs (Klicpera et al.) 2019). The transformed graphs can also serve as effective augmented views in contrastive learning (Hassani & Khasahmadi 2020). Inspired by that, we transform a graph with transition matrix T via graph diffusion and sparsification S I -kzo T into
graph with adjacency matrix S as an augmented view in our framework. While there are many design choices in coefficients 0g like heat kernel, employ Personalized PageRank (PPR) with OPPR a(1 a)* due to its
3

---

Under review as a conference paper at ICLR 2021
superior empirical performance (Hassani & Khasahmadi 2020). As another augmentation choice, we randomly remove edges of graphs to attain corrupted graphs as augmented views to validate the robustness of models to different augmentation choices.
ITERATIVE GRAPH SELF-DISTILLATION
Intuitively, the goal of contrastive learning on graphs learn graph representations that are close in the metric space for positive pairs (graphs with the same labels) and far between negative pairs (graphs with different labels). To achieve this goal, IGSD employs the teacher-student distillation to iteratively refine representations by contrasting latent representations embedded by two networks and using additional predictor and EMA update to avoid collapsing to trivial solutions. Overall, IGSD encourages the closeness of augmented views from the same graph instances while pushing apart the representations from different ones.
4.1 ITERATIVE GRAPH SELF-DISTILLATIN FRAMEWORK
In IGSD, we introduce a teacher-student architec- ture comprises two networks in similar structure composed by encoder fe. projector ge and predic- tor ha. We denote the components of the teacher network and the student network as for go and fo. go. he respectively.
Augmentation
Contrast
G2
Projector
Ag Ltyh
Mini-Butch
Prodicter
G₂
G²'
fo
g
ha
G2
KolteX
G2
G2
Update
The overview of IGSD is illustrated in Figure D In IGSD, the procedure of contrastive learn- ing on negative pairs is described as follows: we first augment the original input graphs G, to get augmented view(s) Gj. Then G and different graph instance G, are fed respectively into two encoders fo. for for extracting graph represen- tations h, fo(G.) fo (G') with iterative message passing in Eqn. ① and readout func- tions. The following projectors go; for transform
G3
f
98
G3
(g(
62¹
Figure 1: Overview of IGSD. Illustration of our framework in the case where we augment input graphs G' once to get G' for only one forward pass. Blue and red arrows denote contrast on positive and negative pairs respectively.
graph representations to projections z, x via = (h) ((2) and x' go (h') h²), where a denotes a ReLU nonlinearity] To prevent collapsing into trivial solution (rril et al. 2020), a specialized predictor is used in the student network for attaining the prediction he(2) W (W( 2) of the projection z. For positive pairs, we follow the same procedure except feeding the original and augmented view of the same graph into two networks respectively.
To contrast latents (z) and z', we use L2 norm in the latent space approximate the semantic distance in the input space and the consistency loss can be defined as the mean square error between the normalized prediction he (z) and projection z'. By passing two graph instances G2 and G3 symmetrically, we can obtain the overall consistency loss:
2 (G;, G3) = 10 (z4) - 2/J 2/11 11 (2/) 23112
(2)
With the consistency loss, the teacher network provides a regression target to train the student network, and its parameters are updated as an exponential moving average (EMA) of the student parameters 0 after weights of the student network have been updated using gradient descent:
0g + 700-1 + (1 -),,
With the above iterative self-distillation procedure, we can aggregate information for averaging model weights over each training step instead using the final weights directly (Athiwaratkun et al.) 2018). It should be noted that maintaining slow-moving average network is also employed in some models like MoCo (He et al. 2020) with different motivations: MoCo uses an EMA of encoder
Although IGSD could directly predict the representations without projections, previous contrastive learning work (Chen et al. 2020b) in the image domain has shown that using projections improves performance empirically. We include the experimental results validate the effects of projectors in Appendix A.3
4

---

Under review as a conference paper at ICLR 2021
and momentum encoder to update the encoder, ensuring the consistency of dictionary keys in the memory bank. On the other hand, IGSD uses a moving average network to produce prediction targets, enforcing the consistency of teacher and student for training the student network.
4.2 UNSUPERVISED LEARNING WITH IGSD
In IGSD, to contrast the anchor G; with other graph instances Gj (i.e. negative samples), employ the following unsupervised InfoNCE objective (Oord et al. 2018):
exp (- (G; G:))
Cussu -BGing log exp (-Loon G.)) + N-1
(4)
j=1 Tizj exp L (- (G)
At the inference time, semantic interpolations samples, labels and latents result in better representations and can improve learning performance greatly (Zhang et al. 2017] Verma et al. 2019 Berthelot et al. 2019), we obtain the graph representation by interpolating the latent representations h= [G) and h for (G) with Mixup function Mix, (a, b) X x) b:
h Mixx (h, h')
(5)
4.3 SEMI-SUPERVISED LEARNING WITH IGSD
To bridge the gap between unsupervised pretraining and downstream tasks, we extend our model to the semi-supervised setting. In this scenario, it straightforward to plug in the unsupervised loss as regularizer for representation learning. However, the instance- supervision limited to standard supervised learning may lead to biased negative sampling problems (Chuang et al., 2020). To tackle
this challenge, we can use small amount of labeled data further to generalize the similarity loss to handle arbitrary number of positive samples belonging to the same class:
Kl
KI
Supcon
Σ
Tips Iy. (Gs, Gj)
(6)
where Ny denotes the total number of samples in the training set that have the same label y/ as anchor i. Thanks to the graph-level contrastive nature of IGSD, are able alleviate the biased negative sampling problems (Khosla et al.) 2020) with supervised contrastive loss, which is crucial (Chuang et al., 2020) but unachievable in most MI-based contrastive learning models since subgraphs are generally hard to assign labels to. Besides, with this loss we are able to fine-tune our model effectively using self-training where pseudo-labels are assigned iteratively to unlabeled data.
With the standard supervised loss like cross entropy or mean square error L(G 0), the overall objective can be summarized as:
semm L(Gr, + CL U Gu. 0) + w Supeon (GI,
Common semi- semi-supervised learning methods use consistency regularization to measure discrepancy between predictions made on perturbed unlabeled data points for better prediction stability and generalization (Oliver et al., 2018). By contrast, our methods enforce consistency constraints between latents from different views, which acts as regularizer for learning directly from labels.
Labeled data provides extra supervision about graph classes and alleviates biased negative sampling. However, they are costly to attain in many areas. Therefore, we develop a contrastive self-tre oolf-training algorithm to leverage label information more effectively than cross entropy in the semi-supervised scenario. In the algorithm, we train the model using a small amount of labeled data and then fine-tune it by iterating between assigning pacudo-labels unlabeled examples and training models using the augmented dataset. this way, we harvest massive pseudo-labels for unlabeled examples.
With increasing size of the augmented labeled dataset, the discriminative power of IGSD can be improved iteratively by contrasting more positive pairs belonging to the same class. In this way, we accumulate high-quality psuedo-labels after each iteration to compute the supervised contrastive loss in Eqn. (6) and make distinction from conventional self-training algorithms (Rosenberg et al., 2005). On the other hand, traditional self self-training can psuedo-labels for computing cross entropy only.
5

---

Under review as a conference paper at ICLR 2021
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Evaluation Tasks. We conduct experiments by comparing with state-of-the-art models on three tasks. In graph classification tasks, we experiment in both the unsupervised setting where we only have access to all unlabeled samples in the dataset and the semi-supervised setting where we use a small fraction of labeled examples and treat the rest unlabeled ones by ignoring their labels. In molecular property prediction tasks where labels are expensive to obtain, we only consider the semi-supervised setting.
Datasets. For graph classification tasks, we employ several widely-used graph kernel datasets (Kersting al. 2016) for learning and evaluation: 3 bioinformaties datasets (MUTAG, PTC, NCI1) and 3 social network datasets (COLLAB, IMDB-B, IMDB-M) with statistics summarized in Table I In the semi-supervised graph regression tasks, we use the QM9 dataset containing 134,000 drug-like organic molecules (Ramakrishnan et al. (014)) with 9 heavy atoms and select the first ten physicochemical properties as regression targets for training and evaluation. For detailed description of the properties in the QM9 dataset, see the Appendix C of (Sun et al. 2019).
Baselines. In the unsupervised graph classification, we compare with the following representative baselines: CCMC-Graphe (Hassani & Khasahmadi] (2200) InfoGraph (Sun et al., 2019). GCC (Qiu
al. 2020), Graph2Vec (Narayanan et al. 2017) and Graph Kernels including Random Walk Kernel (Gärtner et al., 2003),, Shortest Path Kernel (Kashima et al., 2003)) Graphlet Kernel (Shervashidze
ct al. (2009). Weisfeiler-Lehman Sub-tree Kernel (WL SubTree) (Shervashidze et al. [2011). Deep
Graph Kernels (Yanardag & Vishwanathan 2015). Multi-Scale Laplacian Kernel (MLG) (Kondor &
Pan 2016) and Graph Convolutional Kernel Network (GCKN) (Chen et al. 2020a).
For the semi-supervised graph classification, compare method with competitive baselines like InfoGraph, InfoGraph* and Mean Teachers. And the GIN bascline doesn't have access to the unlabeled data. In the semi-supervised molecular property prediction tasks, baselines include InfoGraph, InfoGraph* and Mean Teachers (Tarvainen & Valpola 2017).
Model Configuration. In our framework, We use GCNs (Kipf & Welling 2016) and GINs (Xu et al. 2018) as encoders to attain node representations for unsupervised and semi-supervised graph classification respectively. For semi-supervised molecular property prediction, we employ message passing neural networks (MPNNs) (Gilmer et al.) (017)) as backbone encoders to encode molecular graphs with rich edge attributes. All projectors and predictors are implemented as two-layer MLPs. For more details on param-prrrmeters selection, refer to appendix A.2
In semi-supervised molecular property prediction tasks, we generate multiple views based on edge attributes (bond types) of rich-annotated molecular graphs for improving performance. Specifically, we perform tubel-preserrve
augmentation attain multiple diffusion matrixes of every graph on different edge attributes while ignoring others respectively. The diffusion matrix gives denser graph based on each type of edges to leverage edge features better. We train our models using different numbers of augmented training data and select the amount using cross validation.
For unsupervised graph classification, we adopt LIB-SVM (Chang & Lin| (01T)) with parameter selected in {1c-3, 1c-2, .... lc2, 1e3} as downstream classifier. Then use 10-fold cross validation accuracy as the classification performance and repeat the experiments times to report the mean and standard deviation. For semi-supervised graph classification, we randomly select 5% of training data as labeled data while treat the rest as unlabeled one and report the best test set accuracy in 300 epochs. Following the experimental setup in (Sun et al., 2019), we randomly choose 5000, 10000, 10000 samples for training, validation and testing respectively and the rest are treated as unlabeled training data for the molecular property prediction tasks.
5.2 NUMERICAL RESULTS
Results on unsupervised graph classification. We first present the results of the unsupervised setting in Table I All graph kernels give inferior performance except in the PTC dataset. The Random Walk kernel runs out of memory and the Multi-Scale Laplacian Kernel suffers from a long running time (exceeds 24 hours) in two larger datasets. IGSD outperforms the baselines

---

Under review as a conference paper at ICLR 2021
Datusets N graphs
MUTAG
IMDB-B
IMDB-M
NCII
COLLAB
PTC
Datavets
188
1000
1500
4110
5000
344
# classes
3
2
Avg modes
Random Walk
1.5
14.7 0.2
Graph Kernel
SHORTEST PATH
85.2 2.4
55.6 0.2
3..
51.3 0.6
49.8 = 1.2
58.2
GRAPHLET KERNEL
81.7 2.1
65.9
4..9
5..9
56.3 - 0.6
57.3
WL SUBTREE
80.7 1.
72.31:34
47.0
55.1 15.1::10 1.6
50.2 1: 0.9
58.0 0.5
DEEP GRAPH
87.4
67.0
4.5
54.5
52.1
60.1 1 2.6
Day
GCKN
87.2 1 6.8
70.5 d: 3.1
50.8 1: 0.8
20.6
54.31100
58.4 1: 7.6
GRAPH2VEC
9.
71.1
0.9
7.
47.9
00.2 6.9
Unsiperited
INFOGRAPH
89.0
0.7
0.5
7.8
67.6
61.7 1.7
CMC-GRAPH
89.7
0.7
0.5
7..0
6,99
62.5
GCC
86.4 = 0.5
7.59
48.9
66.9
752
58.4 1.2
(RANDOM)
0.6
57.6
OURS
90.2 11 0.7
74.7
1.
0.6
51.5
1:
0.3
75.4
1.4400
0.3
70.4
10.11.1
61.4
Table 1: Graph classification accuracies (%) for kernels and unsupervised methods on datasets. We report the mean and standard deviation of final results with five runs. *>1 day" represents that the computation exceeds 24 hours. 'OMR' means out of memory error.
Datasets
IMDB-I B
IMDB M
COLLAB
NCI1
MEAN TEACHERS
69.0
49.3
72.5
71.1
71.1
GIN (SUPERVISED ONLY)
67.0
50.0
71.4
67.9
OURS (UNSUP)
72.0
50.0
72.6
70.6
OURS (SUPCON)
75.0
52.0
73.4
67.9
GIN (SUPERVISED ONLY)+SELF -RA-SEEL---RIINGG TRAINING
72.0
51.3
70.4
74.0
(UNSUP)+SELF-TRMM))))
72.5
OURS (SUPCON)+SELF TRAINNN
77.0
55.3
73.6
77.1
Table 2: Graph classification accuracies (%) of semi-supervised experiments on 4 datasets. We report the best results on test set in 300 epochs.
like InfoGraph, CMC-Graph and GCC in most datasets, showing that IGSD can learn expressive graph-level representations for downstream classifiers. Besides, our model still achieve competitive results in datasets like IMDB-M and NC11 with random dropping augmentation, which demonstrates the robustness of IGSD with different choices data augmentation strategies.
Results semi-supervised graph classification. We further apply model to semi-supervised graph classification tasks with results demonstrated in Table 4 where we set w and w' in Eqn. 7 be and 0 Ours (Unsup) while 0 and Ours (SupCon). In this setting, our model performs better than Mean Teachers and InfoGraph Both the unsupervised loss and supervised contrastive loss provide extra performance gain compared with GIN using supervised data only. Besides, both of their performance can be improved significantly combined using self-training especially with supervised contrastive loss. It makes empirical sense since self training iteratively assigns psuedo- labels with high confidence to unlabeled data, which provides extra supervision on their categories under contrastive learning framework.
Results on semi-supervised molecular property prediction. We present the regression perfor- mance our model measured in the QM9 dataset in Figure We display the performance of our square eπror ratio with respect supervised results and model
model and baselines outperforms all baselines in 9 out of 10 tasks compared with strong basclines InfoGraph, InfoGraph* and Mean Teachers. And in some tasks like R2 (5), UO (7) and U (8), IGSD achieves significant performance gains against its counterparts, which demonstrates the ability to transfer knowledge learned from unsupervised data for supervised tasks.
5.3 ABLATION STUDIES AND ANALYSIS
Effeets of self-training, We first investigate the effects of self-training for our model performance in table 4 Results show that self-training can improve the GIN baseline and models with unsupervised loss (Unsup) or supervised contrastive loss (SupCon). The improvement is even more significant combined with supervised contrastive loss since high-quality pseudo-labels provide

---

Under review as a conference paper at ICLR 2021
2.0
InfoGraph
InfoGraph*
Mean Teachers
Ours (IGSD)
1.5
Error Ratio
1.0
0.5
0.0
Mu (9)
Alpha (1) HOMO (2) LUMO (3)
Gap (4)
R2 (5)
ZPVE (6)
uo (7)
U (8)
H (9)
Molecular Properties
Figure 2: Semi-supervised molecular property prediction results in terms of mean absolute error (MAE) ratio. The histogram shows error ratio with respect to supervised results (1.0) every semi-supervised models. Lower scores are better and a model outperforms the supervised baseline when the score less than 1.0.
74.5
54
Ours
CMC-Graph
Ours
InfoGraph*
Accuracy (%)
74.0
Accuracyy (%)
52
50
48
25
50
75
100
125
Batch size (Negative Samples)
Fraction labeled data used (%)
(a)
Figure 3: Ablation studies. (a) Unsupervised performance with different batch size (number of negative pairs); (b) Semi-supervised graph classification accuracy with different proportion of labeled data.
additional information of graph categories. Moreover, our self-training algorithm consistently outperforms the traditional self-training baseline, which further validates the superiority of our model.
Effects of different amount of negative pairs. We then conduct ablation experiments on the amount of negative pairs by varying batch size over {16, 32, 64, 128} with results on IMDB-B dataset shown in Figure 3al Both methods contrast negative pairs batch-wise and increasing batch size improves performance of IGSD while degrades CMC-Graph. When batch size greater than 32, IGSD outperforms CMC-Grapp Graph and the performance gap becomes larger the batch size increases, which means IGSD is better at leveraging negative pairs for learning effective representations than CMC-Graph.
Effects of different proportion of labeled data. We also investigate the performance of different models with different proportion of labeled data with IMDB-B dataset. As illustrated in Figure 3b IGSD outperforms strong InfoGraph* baseline given different amount of labeled data consistently. And the performance gain is most significant when the fraction of labeled data is 10% since our models can leverage labels more effectively by regularizing original unsupervised learning objective when labels are scarce.
CONCLUSIONS
In this paper, we propose IGSD, novel unsupervised graph-level representation learning framework via self-distillation. Our framework iteratively performs teach-student distillation by contrasting augmented views of graph instances. Experimental results in both unsupervised and semi-supervised settings show that IGSD is not only able to learn effective graph representations competitive with state-of-the-an aat models but also robust with choices of encoders and augmentation strategies. In the future, we plan to apply our framework to other graph learning tasks and investigate the design of view generators generative effective views automatically.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ben Athiwaratkun, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many consis- tent explanations of unlabeled data: Why you should average. arXiv preprint arXiv: 1806.80066444 05594, 2018.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. In Advances in neural information processing systems, pp. 585-591, 2002.
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, pp. 5049-5059, 2019.
Karsten M Borgwardt and Hans-Peter Kriegel. Shortest-path kernels on graphs. Fifth IEEE international conference data mining (ICDM '05), pp. 8-pp. IEEE, 2005.
Chih-Chume Chang and Chih-Jen Lin. Libsvm: A library for support vector machines. ACM transactions on intelligent systems and technology (TIST), -3),
Dexiong Chen, Laurent Jacob, and Julien Mairal. Convolutional kernel networks for graph-structured data. arXiv preprint arXiv:2003.05189 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020b.
Ting Chen, Simon Kornblith. Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020c.
Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. Debi- ased contrastive learning, 2020.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks graphs for learning molecular fingerprints. In Advances in neural information processing systems, pp. 2224-2232, 2015.
Tommaso Furlanello. Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks, 2018.
Thomas Gärtner, Peter Flach, and Stefan Wrobel. On graph kernels: Hardness results and efficient alternatives. In Learning theory and kernel machines, pp. 129-143. Springer, 2003.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. arXiv prepriat arXiv: 170400/704 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch. Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach self-supervised learning, 2020.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, pp. 1024-1034, 2017.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. Proceedings of International Conference on Machine Learning, pp. 3451-3461. 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020. doi: 10. 1099cvp:42000. 20.0000000000000007.... URL http:
7/dx. doi. www.ao...gg..... /10 ..000.1100 wwwp/////////00000/ 2020. C0975
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint
2015.

---

Under review as a conference paper at ICLR 2021
R Devon Hjelm, Alex Fedorov, Samuel Lavoie Maroii--ar,,,,onn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808. 06670, 2018.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-traiee pre-training graph neural networks, 2019.
Hisashi Kashima, Koji Tsuda, and Akihiro Inokuchi. Marginalized kemnlss between labeled graphs. In Proceedings of the 20th international conference on machine learning (ICML-03), pp. 321-328, 2003.
Kristian Kersting, Nils M. Kriege, Christopher Morris, Petra Mutzel, and Marion Neumann. Bench- mark data sets for graph kernels, 2016. URL http::tt//ggppppreeeeeee cs. daa-dortuund
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning, 2020.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 160966699900777 2016.
Johannes Klicpera, Stefan Weißenberger, and Stephan Günnnemann. Diffusion improves graph learning, 2019.
Risi Kondor and Horace Pan. The multiscale laplacian graph kernel. In Advances in Neural Information Processing Systems, pp. 2990-2998, 2016.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-super learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, 2013.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang. Li Mian, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive, 2020.
Takeru Miyato, Shin-Ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 119)),9991119, Aug 2019. ISSN 1939-3539. doi: 10. 1009/ tpami
URL do1 org 10. 1109/TPAMII 2018 2858821
Hossein Mobahi, Mehrdad Farajtabar, and Peter I. Bartletr. Self-distillation amplifies regularization in hilbert space. arXiv preprint arXivv20002.07715 2020.
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. arXiv preprint 17Xiv:1707.05005. 2017.
Mark EJ Newman and Michelle Girvan. Finding and evaluating community structure in networks. Physical review E, 69(2):0261 13, 2004.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.088: 1107:0007.034, 2018.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang. Ming Ding, Kuansan Wang, and Jie Tang. Gec: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1150-1160, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
10

---

Under review as a conference paper at ICLR 2021
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. 2005.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten Borgwardt. Efficient graphlet kernels for large graph comparison. In Artificial Intelligence and Statistics, pp. 488-495, 2009.
Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeller-lemman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
Fan-Yur Sun, Jordan Hoffmman. Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv: 01000, 2019.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, 2017.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? arXiv preprint arXiv:2005.10243, 2020.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv: 13625, 2019.
Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint 18Xiv: 10341, 2018.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. (438-6447) PMLR, 2019.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets, 2015.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint
2018.
Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1374,, 2015.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? arXiv preprint arXiv:2006.09/.33 09136, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2017.
Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks, 2020.
Yadi Zhou, Fei Wang, Jian Tang, Ruth Nussinov, and Feixiong Cheng. Artificial intelligence in covid-19 drug repurposing. The Lancet Digital Health, 2020.
11

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1
RELATED WORK
Graph Representation Learning Traditionally, graph kernels are widely used for learning node and graph representations. This common process includes meticulous designs like decomposing graphs into substructures and using kernel functions like Weisfeiler-Leman graph kernel (Shervashidze cct al., 2011) to measure graph similarity between them. However, they usually require non-thr non-trivial hand-crafted substructures and domain-specifi kernel functions to measure the similarity while yields inferior performance on downstream tasks like node classification and graph classification. Moreover, they often suffer from poor scalability (Borgwardt & Kriegel 2005 and great memory consumption (Kondor & Pan 2016) due to some procedures like path extraction and recursive subgraph construction. Recently, there has been increasing interest in Graph Neural Network (GNN) approaches for graph representation learning and many GNN variants have been proposed (Ramakrishnan et al. [0014] Kipf
Welling 2016 Xu al. 2018). However, they mainly focus on supervised settings.
Data augmentation Data augmentation strategies on graphs are limited since defining views of graphs is non-trivial task. There two common choices of augmentations graphs (1) feature- space augmentation and (2) structure-space augmentation. A straightforward way is to corrupt the adjacency matrix which preserves the features but adds removes edges from the adjacency matrix with some probability distribution (Veličković et al., 2018). Zhao et al. (2020) improves performance in GNN-based semi-supervised node classification via edge prediction. Empirical results show that diffusion matrix can serve as a denoising filter to augment graph data for improving graph representation learning significantly both in supervised (Klicpera et al. 2019) and unsupervised
settings (Hassani & Khasahmadi] 2020p. Hassani & Khasahmadi (2020 shows the benefits of treating diffusion matrix as an augmented view of mutual information based contrastive graph representation learning. Attaining effective views is non-trivial since we need to consider factors like mutual information to preserve label information w.r.t the downstream task (Tian et al. 2020).
A.2 HYPER-PARAMETERS
For hyper-parameter tuning. we select number of GCN layers over (2, 8, 12), batch size over (16, 32, 64, 128, 256, 512), number of epochs over {20, 40, 100} and learning rate over {le-4, 1e-3) in unsupervised graph classification.
The hyper-parameters we tune for semi-supervised graph classification and molecular property prediction the same in (Xu al. 2018) and (Sun et al. 2019), respectively.
In all experiments, we fix the fixed a = 0.2 for PPR graph diffussion and set the weighting coefficient of Mixup function to be 0.5 and tune projection hidden size over { [1024, 2048} and projection size over (256, 512). We start self-training after 30 epochs and tune the number iterations over {20, 50), pseudo-labeling threshold over (0.9, 0.95).
A.3 EFFECT OF PROJECTORS
While we could directly predict the representation y and not projection z, previous contrastive learning works in the image domain like (Chen al. (020b) have empirically shown that using this projection improves performance. We also further investigate the performance with and without the projector on datasets:
Datasets
MUTAG
IMDB
IMDB M
PTC
IGSD
90.2
74.7
51.5 0.3
61.4 1.7
IGSD w/o PROJECTOR
87.7 + 0.9
74.2 + 0.6
51.1 + 0.6
56.7 +
Table 3: Effect of projectors on Graph classification accuracies ( (%).
Results above show that dropping the projector degrades the performance, which indicates the necessity of projector.
12

---

Under review as a conference paper at ICLR 2021
Meanwhile, to investigate the effect of projectors model performance, we fix the output size of layers in encoders so that their output size is always 512. Then we conducted ablation experiments on different size of the projection head on IMDB-B B with the following results:
218
IGSD
74.5
74.8 0.4 74.7 = 0.6 74.9
Table 4: Effect projector hidden size on Graph classification accuracies (%).
In general, the performance is insensitive to the projection size while larger projection size could slightly improve the unsupervised learning performance.
13


Reviews:
[{"review_id": "k5uTGNPlPFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a method for learning graph-level representation in an unsupervised contrastive way. Instead of contrasting between graph-level representation and patch representation like InfoGraph [1], they contrast graph-level representation of a graph to its augmented variation using a teacher-student framework.\n\n* why use InfoNCE objective instead of the Jensen-Shannon mutual information objective used in InfoGraph [1] ? \n\n* The major concern about this paer is that the proposed method encourages the closeness of augmented views from the same graph instances but provide no guarantee that the transformation used (graph diffusion and sparsification with PPR + random remove edges in this paper) would be label-preserving. For example, in molecular datasets, if we drop an edge and that edge happens to be in a structural motif, it will drastically change the attributes/labels of the molecule. \n\n* $v$ represents nodes in section 2.1 and 2.2 and it represents graph instances in section 3.1 and Figure 1. This can be confusing I suggest changing the notation in section 3.1 and Figure 1.1 to $G$\n\n[1] Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semisupervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000, 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "g0FAaD_-Pp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "------------------------------------\n\nupdate after reading the authors' response.\n\nThe authors didn't address my question \"Did the authors perform a significance study?\" A significance test such as double-sided t-test is needed to verifying whether the proposed method is significantly better than baselines.\n\n-------------------------------\n\nThis paper proposed a distillation approach for unsupervised graph representation learning. The approach partially builds upon contrastive self-supervised learning which contrasts pairs of augmented graphs. The approach is extended to the semi-supervised setting. The authors performed evaluation in graph classification and regression tasks. \n\nI recommend to reject this paper, due to the following major concerns: 1) experimental results are not strong; 2) important baselines were not compared; 3) important details such as optimal hyperparameter values are missing. \n\nMy major concerns of this paper include:\n1. The improvement of the proposed approach over baselines seem not significant. For example, in Table 1, comparing the mean and standard deviation of the proposed approach and CMC-GRAPH, it seems that the difference is not statistically significant. Did the authors perform a significance study?\n2. In the experiments, why the authors didn't compare with GCC, which is a contrastive self-supervised learning approach applied to graph classification?\n3. There are many other unsupervised graph representation learning methods. The authors need to compare with more to substantiate this work.\n4. In hyperparameter tuning, the authors gave the range of hyperparameters tuned, but didn't give the optimal value of the hyperparameters, which make the paper difficult to reproduce.\n5. In table 1, the authors excluded some results since they need more than 1 day to obtain. It is common for deep learning models to run several days to obtain results. I don't think it is proper to exclude these results simply because the runtime is more than 24 hours.\n\nHowever, the paper does have a few strong points.\n1. The ablation studies are well designed and the results are insightful.\n\n2. The paper is well-written and easy to follow, with a clear organization.\n\n3. The experiments were conducted on a rich collection of datasets. \n\nOther comments.\n1. In equation (3), the authors can draw a connection with MoCo.\n2. In Table, why didn't report mean and standard deviation of the results?\n3. For this result \"When batch size is\ngreater than 32, IGSD outperforms CMC-Graph and the performance gap becomes larger as the batch\nsize increases.\",  can the authors provide a reason that can possibly explain this phenomenon?\n4. The authors can add some statistics of the datasets used in Figure 2.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0KSolCXlVNq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall Comments:\nLearning graph-level representations with only labels has been explored by many works. However, it's not easy to annotate every graph. This paper applies the ideas from semi-supervised classification task to improve the representation quality learned by graph neural network. Specifically the proposed solution combines several kinds of existing techniques including diffusion graph augmentation, mean teacher consistency, debiased contrastive loss and pseudo class consistency. Finally they are combined together to act as a regularization term by utilizing the unlabelled data. From this point of view, the novelty of this work is incremental, but it's still an interesting work for improving graph-level representations.\n\nClarity:\nThe presentation is not clear enough. There exists many claims that are not clear, shown as follows:\n\n1. In the last sentence of 3rd paragraph in introduction section, it's difficult to get the connection between negative samples mining and self-distillation strategy. Why using the self-distillation can alleviate the dependency on negative samples mining? The unsupervised objective in Equation 4 still depends on negative samples. \n2. In Section 2.1, the notation for augmentations. Why are the graphs G_L attached without labels after being augmented?\n3. In Section 2.3, authors firstly use PPR to augment node features, then randomly remove edges to create a corrupted graphs. According to the description, the question is how many views that will be used in follow sections? I guess that the graph feature from original graph will be fed to student network, and the augmented corrupted graph will be fed to teacher network.\n\nQuestions for Rebuttal:\n1. Please clarify the mentioned questions above.\n\n2. The proposed method contains an encoder, projector, and predictor. The question is why we need a projector g to get a higher dimension z? Does it have a big influence on the performance? Could you please give the complete definition of function g and h?\n\n3. The definition of L^{con}  in Equation 2 is for positive sample extracted from the same graph G_i. However, the complete unsupervised loss needs negative samples G_j. Could you please also give the definition for L^{G_i, G_j}?\n\n4. The overall loss consists of supervised and unsupervised loss. The L^{sup} has conflict to the first term in Equation 7. Both of them use labels but it's difficult to tell which one should be aligned with the supervised loss shown in the ablation study (Table 2). The SupCon has never been shown in the main content before. Please pay attention to make it clear.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Cs2HNnWG0W", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a self-distillation based graph augmentation mechanism to alleviate the drawbacks of existing MI based models w.r.t. their high dependency towards negative sampling. Quantitatively the proposed model achieves encouraging results. However it would have been better if the system designs and significant difference of IGSD from existing work are discussed.\n\n\nStrength:\n\n- This work has clearly discussed a drawback of existing unsupervised MI based models which is the leading approach in graph classification\n- They propose a mechanism to address this issue with satisfiable quantitative results on unsupervised setting and extended semi-supervised setting with self-training also supported quantitatively.\n- Paper is clear in general, with a clear research problem, proposes mechanism for unsupervised/semi-supervised graph representation domain and encouraging quantitative results.\n\nWeakness:\n\n-There is a lack of qualitative analysis and discussion of the proposed method. \n-In Section 4.3 \"Performance with different amount of negative pairs\", it is not clear the reasoning of the provided observation from Figure 3a.\n-It is not clear the motivation behind selecting a teacher-student network for obtaining different views of the graph. These networks are normally used for knowledge transfer, but here used for contrastive learning. How is this more beneficial than an ensemble model w/o knowledge transfer step of Eq. 3. \n-The core difference of IGSD from CMC-graph is that CMC uses MI based contrastive between local patch representation and graph rep, while IGSD uses L2 based contrastive between 2 graph representations. Input, Encoders and projections are the same for both architectures. It could be useful to add some analysis to discuss these differences and their contributions to clearly understand the significance of IGSD.\n-This paper seems to have state-of-the-art results (although it is based on graph kernel). Why the results are not included?\n\nConvolutional Kernel Networks for Graph-Structured Data, ICML-2020\n\n\n\n=======================\n\nafter rebuttal:\n\n\nI thank the authors for the response. I still have concerns re their comparison with GCKN (ICML-2020). The reproduced results by the authors are different significantly from the published one in Table 1 of GCKN paper (~5%). E.g. MUTAG is 92.8 in original paper, but the authors report 87.2 for GCKN. The difference is significant.\n\nTherefore, I will keep my original rating.\n \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an unsupervised graph learning method [Iterative Graph Self-Distillation (IGSD)] by iteratively performing self-distillation to contrast graph pairs under different augmented views. This idea is then extended to semi-supervised setting where via a supervised contrastive loss and self-training. The method is empirically evaluated on some semi-supervised graph classification and molecular property prediction tasks, and has achieved promising results.

Reviewers agree that the method is interesting and the paper is well-written. The biggest concern from reviewers related to experimental evaluations of the method. The authors responded to this and included additional experiments. Although the reviewers appreciate the provided results and explanations, at the end they were not convinced about the empirical assessments. In particular, R1's post rebuttal comment indicates concerns about the reported performance of GCKN, which is different from the published one in Table 1 of GCKN paper. I encourage authors to improve on these experimental discrepancies and resubmit. 
Decision: Reject