Teacher-Guided Graph Contrastive Learning
Authors: Jay Nandy
Arnab Kumar Mondal
Manohar Kaul
Prathosh AP
Venue: Accepted by TMLR ()

Abstract: State-of-the-art self-supervised representation learning methods for Graphs are typically based on contrastive learning (CL) principles. 
These CL objective functions can be posed as a supervised discriminative task using *'hard'* labels that consider any minor augmented pairs of graphs as 'equally positive'. However, such a notion of 'equal' pairs is incorrect for graphs as even a smaller 'discrete' perturbation may lead to large semantic changes that should be carefully encapsulated within the learned representations. This paper proposes a novel CL framework for GNNs, called *Teacher-guided Graph Contrastive Learning (TGCL)*, that incorporates 'soft' pseudo-labels to facilitate a more regularized discrimination. In particular, we propose a teacher-student framework where the student learns the representation by distilling the teacher's perception. Our TGCL framework can be adapted to existing CL methods to enhance their performance. Our empirical findings validate these claims on both inductive and transductive settings across diverse downstream tasks, including molecular graphs and social networks. Our experiments on benchmark datasets demonstrate that our framework consistently improves the average AUROC scores for molecules' property prediction and social network link prediction. Our code is available at: https://github.com/jayjaynandy/TGCL.

Published in Transactions on Machine Learning Research (11/2024)
Teacher-Guided Graph Contrastive Learning
Jay Nandy"
juxjesmanndy@mmaillcom
Pujitsu Reseorch India Privote Limited
Arnab Kumar Mondal* Pujitsu Reseorch India Private Limited
Manohar Kauli
Monohar. Koad (3 Sujjitsu com
Pujitsu Reseorch India Private Limited
Prathosh AP
prathosh writhss@@iss..oo Wiisc. oc.in
Fujitsu Reseorch India Privote Limsted Indian Institute of Science Bengaluru
Reviewed on OpenReview: https: openreview. net/ fottm? id= 15tjpSHI15
Abstract
State-of-the-art self-supervised representation learning methods for Graphs are typically basod on contrastive learning (CL) principles. These CL. objective functions can be posed as supervised discrininative task using 'hard' labels that consider any minor augmented pairs of graphs as 'equally positive'. However, such notion of 'equal' pairs is incorrect for graphs as even smaller 'discrete' perturbation may lead to large semantic changes that should be carefully encapsulated within the learned representations. This paper pro- poses a novel CL framework for GNNs, called Teacher-guided Gruph Contrastive Learning (TGCL), that incorporates 'soft' pseudo-labels to facilitate a more regularized discrimi- nation. In particular, we propose a teacher-student framework where the student learns the representation by distilling the teacher's perception. Our TGCL framework can be adapted to existing CT. methods enhance their performance. Our empirical findings val- idate these claims on both inductive and transductive settings across diverse downstream tasks, including molecular graphs and social networks. Our experiments on benchmark datasets demonstrate that our framework consistently improves the average AUROC scores for molccules' property prediction and social network link prediction. Our code is available at https: http:////gguuuub.o/////aaaaaaaaa
1 Introduction
Graphs are versatile data structures representing relationships between entities various domains, such as social networks (Ohtsuki et al.] 2006 Fan et al. 2201)), bio-informatics (Muzio al. (021),, and knowledge Analyzing and understanding graph data is crucial in many
graphs (Wang et al. 2014 Back et al. 2020
real-world applications, including community detection [Foruunato] 2010), node classification (Bhagat et al.
2011), link prediction (Zhang & Chen 2018
Rossi el. al. 2021 recommendation ion (Wu et al. 2019), continual
learning (Mondal et al. [2024), and time-series analysis CCauhan et al. 2022). Graph representation learning can potentially make significant leaps in graph-based analysis and prediction.
Self-supervised Learning (SSL. for graphs has emerged as an important research area that. leverages the inherent structure or content of inputs learn informative representations without relying on explicit la-
bels (Hu et al. 2020a Hwang et al. 2020 Existing graph-SSL methods be broadly categorized as:
Equaal contribution.
1 Coorreppondigg author

---

Published in Transactions on Machine Learning Research (11/2024)
Edit Distance: Distilled Distance [Proposed]: 3.23
Edit Distance:
Edit Distance: 2,
Distilled Distance [Proposed]: 10.68
Distille Distanca [Proposed]: 7.87
(a)
(b)
(c)
Figure 1: Illustrating the shortcomings of existing CL methods: (a) Even a 'minor change,' i.e., removal of one edge can significantly change a graph's semantics, leading to disconnccted components that are not captured using edit-distancee-asedd discrepancy (Kim al. 2022). (b & c) more specific example of correlated-structureed molecules that either actively bind to a set of human 8-secretase inhibitors or inactive (Wu et al. 2018). (b) Molecules having dissimilar properties can have smaller edit distancos, while (c) molecules from the same class can have larger edit distances. In other words, edit distance remains ineffective in capturing chemical semanties. Our proposed distilled perception distance form pre-trained teacher incorporates 'soft' semantic distances for any arbitrary graphs to train a better student representation model.
(a) local Samilarity-based predictive learning & (b) global similarity-based contrastive learning. Predictive learning-band based methods (Hu et al. 2020a Kim & Oh 2021 Rong et al. 2020) produces artificial labels by capturing specific local contextual information of neighborhood sub-graphical features to produce the repre- sentations. However, it restricts them to capturing only the local graph semantics. Alternatively, contrastive learning (CL)-based models for graphs aim to maximize the agreements between instances perturbed by cemantic-invarian augmentations (positive views) while repelling the others (negative views) to capture global semantics. CL-based SSL models are extremely popular in the computer-vision community. For such applications, we can easily generate such semmatic-invariant perturbations using simple techniques e.g., rotation, flipping, and color jittering (Chen et al. 2020a Grill et al. 2020b).
Several graph contrastive learning methods are also proposed where the positive pairs are produced using transformations e.g., edge perturbation, attribute masking, and subgraph sampling. However, unlike con- tinuous computer vision domains, even 'minor' modifications in the graph structures, such removing edge node, can significantly change the properties of graphs due to their discrete nature (Figure La & lb Recently, (Kim et al., 2022) introduced discrepancy-basee self-superoised learning (D-SLA) by incorporating edil bissanccebassed discrepancy measures between two graphs to address these limitations. However, com- puting the edit distance between two arbitrary graphs is NP-hard (Sanfeliu & Fu 1983 Zeng et al. 2009). Further, it can only provide high-level structural information without capturing any semantic differences (Figure la & [b]. In this paper, we propose a graph representation learning framework by incorporating more semantically-rich soft-discriminative features using such an imperfect pre-trained teacher to regularize the learning.
1.1 Motivation & Contributions
The existing CL methods for graphs can be viewed under the same umbrella where these techniques learn representations by contrasting different views of the input graphs. Tn principle, their loss functions can be considered as superviscd classification objectives by creating pscudo-labels among different views of input graphs (Oord al.) 2018 Gutmann & Hyvärinen 2010). In contrast, in the supervised learning literature, it has been observed that incorporating 'soft labels", even from an imperfect teacher, in the form of Knowledge Distillation. (KD) leads to better generalization (Hinton et al. 2015 Menon et al. 2021 Kaplun et al. 2022). Given these prior results, we explore the following question: Can "soft" guidance from an imperfect teacher lead to a beller CL framework for gruphs?
The fundamental idca of KD is to use softened labels via a teacher network while minimizing the supervisod risk of student network by reducing the divergence between their logits (Hinton et al. 2015). Prior works have shown that freyes-distilled risk has lower variance compared to naive undistilled counterpart, which
2

---

Published in Transactions on Machine Learning Research (11/2024)
Perturbed from Original graph
Perturbed pains:
Perurbed from original Vp Graph
leacher
Fenurbed fon
Distance of regresentations
Positive pairs: Loarn Attract
Edit distance
Aezureen Portg
represinat ea
touttrat
popp
Schpanoptas Is rapel
Negative pairs: Learn to Repel
Nesative gairs: Learn to Repel
Different Up
graph
Different U/P
Graph
Toocher
Diferee (PPouph
(a) Contrastive Learning
(b) D-SLA
(c) Proposed TGCL
Figure 2: Comparing our proposed TGCL framework with the existing contrastive learning You et al. 2020 : Xu ct al. (2021), and D-SLA Kim et al. (2022). From the classification point of view, the standard CL melhods consider the similarity between the anchor and the perturbed graphs as "hard" positive pairs while the other graphs as "hard" negative pairs. D-SLA introduces "hard" discrepancies using edit distance between the anchor and the perturbed graphs, while the other graphs as "hard" negative pairs. Our proposed TGCL introduces a novel distilled perception distance for smooth discrimination between arbitrary graphs.
leads to better generalization (Menon et al. 2021 Motivated by these results, we propose a novel Teacher- guided Graph Contrastive Learning (TGCL) framework. We design distilled perception distance (or distilled dislance) between two arbitrary input graphs using their deep features obtained from a pre-trained "teacher" to define a softer notion of positive/negeee negative pairs. We train the student network by incorporating such 'soft labels' for each pair of graphs. We argue that by introducing distilled distance, we can introduce the regularized semantic difference between two arbilrary graphs, addressing the shortcomings of the existing CL frameworks for graphs. For example, Figure Tc demonstrates that our distilled distance obtained from the "teacher" can siguificantly differ among molecular graphs with correlated structures towards capturing the chemical semantic differences for graphs. Figure 1b shows that the distilled distance captures the chemical semantic difference of molecules with different chemical properties, however, with minor structural difference. The contributions of our work can be summarizod as follows:
1. we propose to obtain distilled perceptual distances by comparing the deep features from a pre-trained teacher, followed by injecting them "soft pseudo-labels" into the contrastive loss objective to appropri- ately capture the semantic differences between two arbitrary graphs in the student's representation space. Theoretically, by viewing the contrastive loss objective for graphs from supervised loss, incorporating such 'distilled perceptual distances' acts as soft pseudo-labels that reduce the variance of Bayes-distilled risk to provide better graph reprosentations. To the best of our knowledge, we are the first to propose such a Seache-ggidedd coff-disrrimination based contrastive learning framework for the discrete domain of graphs.
2. Our proposed concept of 'soft-labeled' pairs of graphs can be adapted to any contrastive learning frame- work. We demonstrated two variations of TGCL frameworks by modifying the well-known NT-Xent loss Chen et al. 2020a); You et al. (0200) and D-SLA method Kim et al. 2022 to incorporate smooth perception from a teacher network for training the student network. Notably, TGCL is specifically designed for graphs to appropriately incorporate the representational distance even when minor perturbations significantly change the input semantics.
3. Experiments on graph classification for molecular datasets and link prediction on social network datasets where our proposed framework consistently outperforms the existing methods by improving upon the teacher. we improve the average area under receiver operaling curve (AUROC) score by *x2.33% and 266% for molecules' property prediction and social network link prediction tasks respectively.
2 Related Work
2.1 Representation Learning on Graphs
Classical Approaches: One of the most straightforward representations for a graph is consider the 'bag of modes'. Weisfeiler-Lehman kernel (Shervashidze al. 2011) improves upon this idea by utilizing an iterative
3

---

Published in Transactions on Machine Learning Research (11/2024)
neighborhood aggregation strategy. One may also count the occurrence of small subgraph structures, called grophlets. However, it is combinatorially challenging problem, and approximate algorithms are required (Ahmed et al. 2015; Hočevar & Demsar 2014). A few other approaches enumerate different kinds of paths
in graphs Kashima et al. 2003 Borgwardt & Kricgel 2005).
Shallow Algorithms: WelpWalk (Perozzi et al. 2014) and LINE (Tang et al. 2015 are random walk-basod techniques using depth-first search (DFS) and breadth-first search (BFS), respectively. "node2vec' GGroerr & Leskovee 2016) combines both BFS and DFS to learn node embeddings by maximizing the likelihood of preserving node neighborhoods.
Predictive SSL for Graphs: These methods aim to predict specific graph properties, e.g., predicting the altributes of masked nodes/edges (Hu et al. 2020a), existence of an edge (Hwang et al. 2020) or contextual properties and presence motifs (Hu et al., 2020b Rong et al. 2020). These predictive tasks serve as self- supervisions, as they do not require explicit supervised labels. Instead, they rely on the local sub-structure of the graph for producing labels.
Contrastive SSL for Graphs: Deep Graph Infomax (DGI) (Veličković et al. 2019) maximizes the mutual information between graph representation and patch representation. InfoGraph (un et al. 2020) maximizes the mutual information between the graph-level representation and the representations of substructures of different scales, such as nodes, edges, and triangles. Several other works (You et al., 2020 2021 Zhu et al.
2021 Yin et al., 2022 Wang et al. 2022) employ contrastive learning by generating perturbed views of the original graph through attribute masking, edge perturbation, and subgraph sampling to obtain better
representations. Recent works (S et al. 2021: Yang et al. 2021) also explore adversarial augmentation strategies to further improve these frameworks.
2.2 Knowledge Distillation (KD)
KD Hintto al., 2015) was originally introduced transfer knowledge from complex 'teacher' model
with large capacity to efficient 'student' model with lower capacity while performing similarly to the teacher. Several works also focus on improving the student's performance on wide range of applications
(Heo et al. 2019 2021
Furlanello et al. 2018 Lopes et al. 2017 Li et al. 2021 Lee et al. 2018 Bhat et al.
Surpassing the Teacher's performance. KD allows the student to learn from both the raw data and distilled knowledge of the teacher, improving their generalized performance (Menon et al., 2021). Therefore, recent works successfully demonstrated that student with larger or the same capacity can consistently exceed the teacher's performance to produce more generalized model.
Existing Distillation-based SSL. Existing distillation-baseed SSL methods were mainly explored in the continuous domain (e.g., imaagss, video) to remove the requirement of negative sampling for contrastive learning frameworks Grill et al. (2020a); Caron et al.] (2021). They were also explored to reduce the size of the representation learning models (Abbasi Koohpayegani et al. 2020 Chen et al. 2020b). Many of these approaches combined KD with CL methods (Fang et al. [021] Gao et al.) 2022). SimCLR-v2 (Chen et al,, 2020b) applied larger teacher model, first trained using contrastive loss followed by supervised fine-tuning to distill smaller model using the teacher. (Xu et al. 2020) incorporates auxiliary contrastive loss to obtain richer knowledge from the teacher network. A few other approaches also explored transferring the
final embeddings of a self-supervised pre-trained teacher (Navaneet et al. 2022 Song et al. 2023).
2.3 Limitation of Existing Distillation- SSL methods for graphs
Since a minor perturbation does not change the input semantics in the continuous domain, existing KD- based SSL methods did not focus on learning any semantic distance in the representation space for a pair of "positive' samples. In rostr we should design a novel representation learning framework for graphs that appropriately incorporate semantic differences due to minor discrete perturbations.
Our proposed TGCL first aims to obtain the teacher's distilled perception to calculate the semantic difference for any pairs, followed by formulating soft. self-supervised losses to train the student. The notion of "distilled
4

---

Published in Transactions on Machine Learning Research (11/2024)
perception' was previously explored in computer vision, typically to access the difference between two images mitigating representations for semantically similar inputs. However, we introduce distilled distance for graphs to capture the semantic difference with subtle perturbations. Notably, such a notion is not necessary for computer vision applications as a minor perturbation does not change the semantics of an image.
Therefore, our proposed loss functions are specifically designed to accommodate the discrete nature of graphs and may not be appropriate for continuous domains (e.g,, computer vision). To the best of our knowledge, such teacher-guided distilled distance has not been explored far for representation learning (even in the vision domain) and is the key contribution of our paper.
3 Proposed Method
3.1
Preliminaries
Graph Neural Network (GNN). Let G= (V, XE) be an undirected graph in the space of graphs 9, where V, Xv, denote the set of nodes, edges, node attributes, and edge attributes respectively. GNN encodes a graph E G to a d-dimensional embedding vector: f: g + R.. f is often composed by stacking multiple message-passingg layers. Let A denote the representation of a node E V having a neighborhood N, in the 1th layer. 1 represents the attributes of edge (v,u) E E in the (1 1)th layer.
Then, (9) can be expressed as follows:
1 20 (1-1)
wht (1-1) " 1 "0 )))""))
(1)
MEN,
where o/ ( w/r (1-1) are the update and the message function of (1-1)th layer respectively. @ is a permutation invariant aggregator.
Global Representations using Contrastive Learning. Contrastive learning (CI. aims to learn mcan- ingful representations by altracting the positive pairs (i.e., similar instances, such two different perturba- tions of the same graph) while repelling negative pairs (i.e., dissimilar instances, such two different input graphs) in an unsupervised manner, shown in Figure 2a Formally, let Go denote the original graph and Gp denote a perturbed version (i.e., positive sample), and [Gn, } are other input graphs that are treated as negative sample. Then, the NT-Xent (normalized lemperature-scaledt cross-entropy) loss for CL objective
for Go can constructed as follows (Chen et al. 2020a You et al. 2020
exp (sim(
LCL = -log
(2)
Ecea, exp (an[J]((G--(())//
where f is a GNN and sim(..)) is a similarity mcasure for embeddings with temperaturc-scaling т.
Minimization of Equation 2 brings positive pairs closer and pushes negative pairs further apart the embedding space. However, unlike image augmentation schemes (e.g., scaling, rotation, color jitter), graph augmentation schemes (e.g., node/edge perturbations, subgraph sampling) may fail to preserve the graph semantics. For example, Figure [ illustrates that removing one edge leads to two disconnected graphs, significantly changing the original semantics. Recently, D-SLA incorporates edit distances to introduce representational discrepancy even between graphs with minor perturbations (Kim et al. 2022). However, it only partly solves the issuc.
3.2 Proposed TGCL Framework
This section presents our Teacher-guided Graph Contrastvv Learning (TGCL) framework. Our proposed TGCL is fundamentally based on the following theoretical propositions. While these two propositions were proposed in two different literature of unsupervisod contrastive learning and supervised distillation frame- works, they provide the perfect motivation to propose our teacher's distilled perception-guides contrastive learning framework for discrete domains of graphs.
5

---

Published in Transactions on Machine Learning Research (11/2024)
Proposition 1. Notse-contrastive estimation (NCE) to estimate the probability density function of a random variable is equivalent to training binary classifier to dislinguish between sumples drown from the true distribution and samples drawn from a noise distribution. The estimation of the true density function is derived from the learned binary classification function. (uutmann & Hyvärinen 2010
Proposition indicates that CL loss can be framed as supervised classi- fication loss where the network learns to separate posilive pairs from negative pairs by artificially assigning "hard pseudo-labels" for each pair. Specifically, in Eq. 2 for Les formulation, we com- pule the probability for any arbitrary graph, G' E Gp (G,, constructing a positive pair with the original graph Go softmax function i.e., Pr(Go, G") exp (iim (IGG)))))))))
Teacher GNN
Input Graph-1
harid Params.
peturitation
Shaned Panams.
Studien
Teacher GNN
Augmented
Shared P Params.
Shared Paramss.
Didlledditance betwee two arbttrry graphe
Studint
GNN
Teacher
Eocauuran exp (im((((s.)f(()))++ Hence, by labeling the ground truth for a positive pair as U(G),(Gp) = and negative pairs as, G(Go,Gn,) = 0 for all j, we can rewrite CL objective as: CCL = -(V(G..G,) log Pr(Gu, Gp) + Ea., G(G.,G.,)) log Pr(Go, Gny)l, which is precisely the eross-entropy loss for classification.
Input Graph-2
Graph Representations
GNN
(from student network)
Figure 3: Block diagram of our proposed TGCL framework. We obtain the representations from pre-trained teacher model and compute the distilled distance for each pair of inputs. These pairwise distances are employed to "soften' the loss functions to train the student.
construct CL objective by directly distinguishing scores for posilive pairs from negative pairs without computing explicit probabilities (You et al. 2020). Similarly, in D-SLA (Kim et al., 2022) both the graph discrimination loss and the odit-distancce-assel loss approach the task of distinguishing an anchor graph from a perturbed graph as a "hard" binary classification problem. Furthermore, the margin loss in D-SLA functions similarly to Las with a primary focus on separating positive and negative pairs.
Alternatively,
Proposition 2. In supervised learning, the variance of Bayesion- distilled risk, obtained using the soft labels, weighted by the likelihood from the Bayes teacher in the form of knowledge distillation (KD), remains lower than the variance of empirical risk, obtained using the 'hard' class-labels [Menon et al. (2021)) That is,
(R-g(R-(f.D)) S Va-olR((.
where, V() denotes the voriance of a random variable. |.1 is the cardinality. (G(G is the Bayes class- probability distribution, predicted using the Bayes teacher. (.( = -Geg (G)TE(((G) is the
Bayesian- Bayesian-distilled distilled risk. R(f _/b E is the empirical risk where eg. are the hard "class' labels for input graph G is the empirical loss on J (e.g., softmax cross-entropy). The equality holds iff VG e G, the loss vatues ((f(G)) are constant on the support of (G).
Proposition 2 suggests that irrespective of the size eeppaccty of a student model, it statistically produces a better generalization. In particular, it achieves better generalization by using 'soft and distilled' labels for existing CL method as the Bayes-distilled risk has lower variance compared to the naive un-distilled counterpart (Menon et al. 2021 Notably, this result does not depend on the capacity of the teacher or the student models.
Motivated by these results, we propose to obtain distilled perceptual distances, Dap by comparing the deep features from pro-trained teacher, followed by injecting them as "soft pscudo-labels' into the contrastive loss objective learn the semantic differences between two arbitrary graphs in student's representation space.

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 1: PROPOSED TGCL FRAMEWORK
Teacher model: fteacher: Unlabelled Training set: D Dirain Output: Student Model: f
Input:
for sampled mini-batch, 9B == (Gi): Dtrain do
/+ Create multiple perturbations for each anchor graph.
{Gpty) porturbed G;
eariations
/+ Compute Ddp using fteacher
Obtain (ap Gpyz) Vi,j between G; and Guy
Obtain Dap(Gr Gna)) V where Gngy GB G: if Framework: == TGCL- CT. then
6
mins, Ei L.TTCC-GGrpphCL (G+) (see Eq. clse if Framework TGCL-DSLA then
miny. f. Σ. LT Lo Gp.s) + E, pereeppp Gp;; X2 E, Margin (G Gpis: Gras)
(Eq.
Return: Student Model, fs
3.2.1 Distilled Perceptual Distance
Let Ga and G2 be two arbitrary graphs. Consider representation learning model with L message passing layers as the teacher. At each layer, 1, we obtain the node-embedding ( joev for graph G and apply a pooling operation (e.g., max-pool) to obtain fixed-length vector, h We extract such fixed-length features from each layer and concatenate them, i.e., fG.- = [th Ju] and hG, = [(h J1] for G, and G, respectively. The distilled perception distance (or distilled distance) Ddp then defined as the L2 distance between these concatenated features, as:
Dap Gb) llk.. hG.ll2
The distilled distance is similar to the "perceptual distance" in computer vision (Johnson et al. 2016). However, while we use distilled distance as pseudo-label to inject semantic differences into the representation space, perceptual distance is employed to reduce gaps between semanlically similar inpuls in computer vision.
3.3 Proposed Loss Functions
The concept of teacher-guided loss with "softer" positive/negative pairs to train the student network can introduced to any contrastive learning framework for graphs. To showcase the flexibility our proposed TGCL framework, we present two versions of our framework: (a) TGCI-GraphCL & (b) TGCL-DSLA.
3.3.1 TGCL-GraphCL: TGCL using GraphCL Loss.
NT-Xent (normalized lemperatures ure-scaled cross-entropy) is a well-known loss function for contrastive learning models that have been widely explored for different domains, including graphs (Chen al. 2020a You al. 2020).
Loss Objective. For GraphCL, the contrastive loss is obtained by applying the similarity function, sim (Eq. 2 using exponential of temperature scaled dot product of representations. In order to incorporate the "soft" distilled perception for pair of graphs, when the distilled perceptual distance, (a, G.) is smaller, we want the dot product of their representations, fGo) fG, to be higher. Therefore, we can balance their similarities by multiplying the teacher's distilled distance, (dp G,) with the normalized dot product, f.(Go) for each pair of graphs. It. leads to the following loss function:
[TA.GG]]l[[[A[G.]]]]]
f.(Ga)-fa(G.) exp (Go Gnd) + [[A.G.)I-III.CC..
CrGCL-GraphCL
log
(5)
Gny) Gnn) f. F.(o))-..(G.))
Gp: Eg., exp Dap
[T..(G))ll

---

Published in Transactions on Machine Learning Research (11/2024)
where, fs is the student network and f.) is the representations obtained from fe. I|- denotes the Lg distance. Go is the anchor sample, Gp. is the 5th perturbed sample. Gn is 5th negative sample for the anchor, Go- Notably, while we use dot-product-based similarity in Eq. any other similarity measures (e.g., Lg-basod) can also be used to design the loss function.
A Analysis. In the numerator, for the positive pairs with small distilled distance, Daps the student forced to produce larger (T
f.(GG))f.(G..) to minimize the overall loss. However, when Pap(Go, Gp:) is large, the
f.(Go)-f.(Cp.)) as before. The student would still model does not receive the same incentive to maximize [TA(G6]I-I1A[G..)
maximize [T f.(Go)-L.(Gg))) (at a smaller rate) to minimize the overall loss. Similarly, we can analyze the
negative pairs in the denominator.
Note that LrGCL-GraphCL loss learns to discriminate the global representations of the whole graph without capturing local structural changes. Therefore, the TGCI-CraphCT framework is more appropriate for tasks related to global representations such as graph classification. In the following, we present the TGCL-DSLA framework for other tasks (such as link prediction) where the representations should also capture the local structural changes.
3.3.2 TGCL-DSLA: TGCL framework using D-SLA.
Next, we present TGCL-DSLA by introducing macher-guided distilled perception distance for D-SLA. It consists of three components follows:
(a) Teacher-guided Soft Discrimination: We first discriminate the perturbed graphs from the original anchor by introducing Lr-Soft: It consists of two terms: The first one is a KD-based loss, LKD, while the second component is a weighted graph discrimination loss (LwaD). We first obtain the distilled distances: "Dap(Go. Go). (Dap(Go. (ep(G..P.))l Gp.) between the anchor, Go, with itself and the 7th perturbed variations, Gp. We obtain the similarities by taking reciprocals of the normalized distilled distance, followed by clipping ensure numerical stability:
1 80 clip Dap(Go, Go) ) and clip Dap Gp.))
G vi>O 0
(6)
Next, we compute a probability distribution (soft labels) using the softmax-activation with temperature, T, i.e,, softmax(so, 81,-.. ;T r). Similarly, we obtain score for each graph and compute probability distribution using temperature--caled softmax: softmax ( o fa =T). Now, we obtain the distillation loss, LKD by minimizing the entropy between these probability distributions:
CKD 72H (softmaxxss. s..... :7). softmax(X f f.(()))T))
where, Hly.0) = E,, -ylog is the cross-entropy function. V the scoring layer and fs the student network. V fg the composition of v and fs- The representations obtained from fa fed into the w layer to obtain the scores. Therefore, we incorporate the smoothened perception of the teacher in the score functions to learn the student's representations.
The second term, LOGD, is set of binary pross-entropy functions with Go is labeled as and Gp,s are (Go labeled as with the associated normalized soighweights, wi = Σ, (G,(G.
LOGD = oV fa(Go)) - Σ
(8)
Therefore, LOOD incorporates the teacher's soft label via w, Next, LT combines both components with a eyper-parameter a:
LT aLKD + (1 LDCD
(9)
8

---

Published in Transactions on Machine Learning Research (11/2024)
(b) Teacher-guided Perception Loss: Next, introduce perception loss, perprcepp- It ensures that the embedding-level difference between original and perturbed graphs is proportional to the teacher's perspective of their corresponding distilled distances.
- fa(Golll [[,, fo(Go)ll - 2 (Gp Go)
-perrroppt
(10)
Dap (Gp, Go)
(c) Teacher-guided Margin Loss: The third component, Cr- Margin is modified margin loss where the distilled distance acts as a regularizer, controlling the margin among the anchor, Go, its perturbed variations, Gp, and the negative sample, Gny²
Marging
Σ 2.j
max 0, + + (G.) f.(GG) - fa
where, the margin Big = max (3, Dep(Go- Gny) - Dap (Go, Gp.)) changes for each triplet based on teacher's perception.
(d) Overall Loss: Finally, obtain the overall loss by combining all three components follows:
CTGCL-DSLA CT-soft C Margin
(11)
Where \ and X2 are the hyper-paramcters for training the
4 Experiments
In this section, investigate the performance of TGCL for both TGCL-GraphCL and TGCL-DSLA frame- works two diverse sets of experiments: (i) Graph Classification task in the chemical and biological domain and (ii) Link prediction on social network datasets. The graph classification task needs to capture the global structural representation of the graphs to improve the performance. In contrast, the link prediction task relies on the quality of capturing the local structural information. Therefore, allows us to empirically compare our proposed TGCI-GraphCI. and TGCL-DSLA frameworks and understand their effectiveness for different scenarios.
4.1 Main Results
4.1.1 Graph Classification.
Datasets. Following the prior works (You et al. 2021 Xu et al. 2021 Kim et al.
2022), we utilize ZINC15 Next, we finetune the
(Sterling& & Irwin 2015) to train the self-supervised representation learning models.
models on eight different molecular benchmarks from MoleculeNet (Wu et al. 2018 We divide the datasets
based on the constituting molecules' scaffold (molecular substructure). In table
we evaluate models'
generalization ability on out-of-distribution test data samples (Wu et al. 2018
We also present results from biological domains where the datasets are produced by the sampled ego networks from the PPI networks Zitnik et al. (2019). We use the same experimental setup as You et. al. (2021) for predicting proteins' biological functions where we pre-train and fine-tune the model using the PPI network dataset Zitnik et al. (2019). In Table 4 provide the dataset statistics.
Evaluation Metric. We use the Area Under Receiver Operating Characteristic curve (AUROC) for benchnarking (Davis & Goadrich 2006). AUROC quantifies the overall discriminative power of the classifier across all possible classification thresholds where higher values indicating better discrimination ability of the model. We report mean + std with independent runs.
Performance Analysis. Table compares with several different existing models along with "no pretrain- ing" baselines. We can see that the "no pretraining" model achieves the least performance.

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: AUROC score (%) comparison for molecular property prediction (i.e., graph classification) task. For our TGCL models, we indicate the corresponding teacher models within brackets. These models are specifically designed for molecular graphs that incorporate an additional 50K 3D molecular graphs of GEOM for training their self-supervieed network.
Methods
BRRP
Coi
MUV
RACE
SIDER
Tox21
TexCaut
Arg
No Pretrain
63,8 1 4.5
56.0 1 4.4
7L8 2.5
18.3 L 1.9
70.1 3.4
57.3 11.311 1.6
14.3 1 0..8
65.4 1 11.6
066.8
Predictive
Edigepeed
(Hamiloo
67.4
8.7
76.4
76.0
64.1
70.25
ArtcMarkiig
74.7
77.2
79.3
0.7
76.7
64.2
ContextPred
68.0
2.0
63.9
9.8
75.8
77.3
1.0
70.6
1.2
00.9 0.6
15.7 1. 0.7
63.9
0.6
70.89
GraphMAE
Nou cl
70.1 = 0.5
80.8 1.2
74.5 +: 2.8
17.0 + 0.4
81.4 # 0.9
19.0 0.7
74.4 +: 0.5
68.9 + 01.4
72.84
Coutrastive
Infomax
2076
3.0
70.0
0.8
75.3
62.7
70.29
GraphCL
Fon 11
00.7
2.7
08.8
78.6
76.4
GULS 0.9
73.9
02.4
70.78
JOAO You & Bi Villa
7022
1.0
81.8
2.5
71.7 2 1.4
76.7 +: 1.2
77.8 21 0.5
60.3 4: 0.8
75.0 d: 00.1
622.9 2: 11.5
71.
JOA0v2
71.4 0.9
S1. 1.6
78.7 1: 1.0
17.7 1: 1.2
75.5 2: 1.3
00.5 2. 0.7
74.3 1: 0.6
163.2 1. 00.5
72.16
GraphLofr
72.5
63.5
95
BGRL
66.7
00.4
75.5
71.
1.4
74.8
11..2
SInGGL Yuro 02 Aloca
67.4 1 1.2
50.7 4.7
71.2 1: 1.8
75.0 11 0.9
74.1 2.7
ST.4 4. 1.7
74.4 J. 0.5
62.3
0..4
67.19
SimGRACE Du
2022
71.3 + 0.9
6122 4.5
71.2 344
T4.6 + 1.1
73.8 +: 1.4
GILTO + 12.
74.2 + 0.6
63.44 + 00.5
623.18
D-SLA
T8.6
74.13
Additionnl
4D-InfoKKare Stark 11
20123
67.9
80.7 0.5
76.7
73.4
78.9 0.9
0.7
75.3
66.6
73.4
Data (*)
GraphMyP-G. Lay 4:
2i22
70.1 = D.7
80.4 = 1.5
77.7 =
76.3 = 0.8
80.2 =: 1.5
61.0 +: 0.5
70.3 + 0.9
64.22 # 11.11
74.1
FragCT Km ml 2074
2074
71.4 1 0.4
95.2
1.0
77.6 1.0
76.3 + 0.4
62.5 1.6
61.0 +1.0++00 0.6
75.2 + 0.7
65.1 0.8
75.5
TGLL CraphCL [w/ GraphLoG)
16.7
75.79
TGCL-GmphCL (w/ D-SLA)
740 0.4
82.8
22
27.0 0.9
77.0 +:
64.3 1.0
64.2 0.3
26.6
64.7 0.8
75.19
TGCI-DSLA (w/ GraphtaG)
74.8 0.3
80.6
0.5
77.4
78.6 + 0.2
63.0 1.1
61.4 + 0.4
26.1 + 0.1
64.0 + 0.3
74.49
TGCL-DSLA (w/ D-SLA)
735
84.0
70.4
0.9
78.8 1
85.2
01.2 1 1.0
76.9 0.1
64.9
75.00
(a) Predictive vs. Contrastive Models. While predictive pretraining improves upon the no pertaining model, their performance remains worse than the CL models. This is because predictive methods primarily focus on the local structure, while molecular propertics depend on the global structure. In contrast, CL methods focus on the global structure by contrasting between original and perturbed graphs to achieve better performance. While few augmentation-free CT. (Yu et al. 2022 Xia al., 2022) methods are proposed, their performance remains significantly lower than the state-offthe-an GraphLoG achieves higher AUROC scores by exploring both global semantics and local substructures. However, D-SLA achieves the best performance among the existing models by exploring the local discrete graph structures.
(b) Performace of Proposed TGCL. For our TGCL, we report the results by using GraphLoG and D-SLA as teacher modules for both TGCL- GraphCL and TGCL-DSLA models, demonstrating the generalizability of our framework. We observe that our proposed method consistently boosts the performance of teachers irrespective of the teacher's training methodology for both TGCL models. Fur- thermore, we also purperoreedd the existing molec- ular graph-speeific representation learning models that incorporate additional 3D molccular graphs of GEOM for training their represcntation models.
Table 2: AUROC score(%) for PPT dataset. For TGCL models, indicate the corresponding teacher models within brackets.
Methods DSLA
AUROC
71.56 0.46
Graphlagg
66.92 1.58
TGCL-GraphCL (w/ GraphLoG)
71.96 0.77
TGCL-DSLA (w/ D-SLA)
71.63 0.96
Next, we observe that TGCI-GraphCT, (w/ GraphLoG) performs best. Also, TGCL-DSLA (w/ D-SLA) achieves comparable performance as TGCL-G (w/ GraphLoG). In particular, we do not observe an additional advantage of using more graph-specific D-SLA-based loss functions while learning global graph- level representations for molecular property prediction tasks.
Performance on PPI network. In table wc compare with GraphLoG and D-SLA and the corresponding student i.e., TGCL-GraphCL (w/ GraphLoG) and TGCL-DSLA (w/ D-SLA). The results show that the student models consistently outperform their corresponding teachers. However, the observed improvements are not substantial, and the teacher models have already approached their performance saturation point Furlanello et al. (2018). This phenomenon, known as knowledge saturation, is further explored in Appendix A.3.2
Visualizing the learned latent representation space. In Figure we visualize the Iearned latent representation space of D-SLA and our TGCL-DSLA (w/ DSLA) utilizing SNS (Van der Maaten & Hinton 2008). We observe that TGCL-DSLA segregates the positive and negative samples more successfully than
10

---

Published in Transactions on Machine Learning Research (11/2024)
DSLA. Therefore, it indicates that our TGCL-DSIA (w/ DSLA) produces better representation to easily predict the molecular properties.
4.1.2 Link Prediction.
Datascts. We consider COLLAB, IMDB-Binary, and IMDB-Multi from TU Benchmarks (Morris et al. 2020). We separate the dataset into four parts: pre- training. training, validation, and test sets in the ratio 5:1:1:3, as in Kim al. (2022). Additional details are provided in Table (Appendix).
30
-20
10
-30
Evaluation Metric. Table presents comparative results with scveral existing models and "No Pretrain' baselines. We compare average precision (as in (Kim et al., 2022)) where higher value in- dicates better performance. We report meantsst for 5 independent
-70
-20
40
(a) D-SLA
(b) TGCL-DSLA
Figure 4: I-SNE visualization of representations for BBBP
dataset using (a) D-SLA teacher and (b) TGCL-DSLA.
Performance Analysis.
(a) Predictive vs. Contrastive Models Unlike graph classification tasks, local context, plays a crucial role in link prediction. Therefore, the predictive models typically outperformed most of the CL methods. Among the existing CL methods, GraphLog performs similarly to Context as focuses on both local and global structures. D-SLA performs better by capturing local structures using edit-distance- based discriminations that standard CL models fail to distinguish.
(b) Performace of Proposed TGCL. In comparison, our pro- posed distilled distance from the teacher network integrates a regu- larized approach to both local and global semantics. Local semantics are captured from the initial la- tent features, while global seman- ties are derived from deeper global features. Therefore, we can sur- pass existing local and global rep- resentation learning-based models by visible margins for all three datasets. Interestingly, our TGCL- DSLA (w/ (w/rraphLog) performs bet- ter than TGCL-DLSA (w/D-SLA) even though D-SLA outperformed GraphLog. Therefore, better teacher does not necessarily produce better distillation for the student,
Table 3: Average precision score(%) comparison on link prediction task on social networks. For our TGCL models, we indicate the correspond- ing teacher models within brackets.
Methods
COLLAB
IMDB Binary
IMDB-
Avg.
Pretrain
80.01
2.58
64.93 + 1.92
71.22
AttrMasking HH
81.43 + 0.80
7062
3.68
03-37 + 2.15
71.81
ContextProd
$3.96
2.24
66.09
73.51
Infomax Velidione
50.53 1 0.62
07.25
1.87
61.95 1: 2.47
71.02
Graph You 2121
76.04 4: 1.01
03.71
2.98
62:
3.01
67.38
JOAO You
76.57 1: 1.54
65.37
3.28
62.76
1.52
68.23
Graph c
2121
82.95 4: 0.98
09.71
3.18
61.88
1.87
72.51
BGRL Thnloor
2119
76.79 1: 1.15
67.97
4.14
63.71
2.09
69.49
SinGCL Yiu xl 20222
77/46 4: 0.86
61.1
2.60
63.75
2.28
68.72
SimGRACE Nia
A022
74.51 1: 1.54
61.49
2.79
62.81
2.32
67.27
D-SLA (Kin et xl. W122
86.21 1 0.38
78.51 1: 2.79
09:15
2.29
78.07
CCCL-GGrahCL GraphLoG)
87.25 1
75,09
1.88
67.11
3.73
76.48
TCCL-CraphCI. (w/ D-SLA)
87.51 + 1.24
77.95
3.89
67.55 4: 2.20
77.78
TGCL-DSLA (w/ GraphLoG)
91.09 0.33
83.15
0.89 80.03
74.11 1: 1.44
82.78
TGCL-DSLA (w/ D-SLA)
87.51 4 0.59
4.13
70.97 4: 2.42
79.50
as previously observed and analyzed in supervised learning Menon et al. (2021) Kaplun et al. (0022) Zong
ct al. 2023).
4.1.3 TGCL-GraphCL vs. TGCL-DSLA: Choosing the correct framework for downstream tasks.
As we can see for the graph classification task (Table D. our TGCL-GraphCL framework achieves better performance while for link prediction tasks (Table 3). TGCL-DSLA produces better results. Therefore, these empirical results indicate that TGCL-GraphCL produces better global representations for graphs, allowing to easily distinguish two arbitrary graphs in inductive settings. Hence, this framework is better suited
11

---

Published in Transactions on Machine Learning Research (11/2024)
for graph classification tasks. In contrast, TGCL-DSLA also effective in capturing the local structural in- formation by explicitly learning to distinguish the anchor and augmented samples (using -o [Eq. 7 and -prreepp [Eq. 10 ). Hence, it leads to better performance for the link prediction task in transduc- tive settings. In summary, we should choose TGCI-GraphCL for inductive scttings and TGCL-DSLA for transductive settings.
5 Conclusion & Discussion
We utilize Knowledge distillation (KD) for graph representation learning, where a self-supervised pre-trained teacher is used to guide a student model to produce more generalized representations. Extensive experi- mentation demonstrates the effectiveness of the proposed method in improving the performance of graph classification and link prediction tasks. However, there are still many open challenges in graph representation learning, such as the efficient handling of large-scale graphs, the ability to handle heterogeneity and multi- modality, and the development of robust methods for noisy or incomplete data. Probing these challenges further and developing new graph representation learning techniques are in the scope of future research.
Discussion. Potential Advantages and Limitations of KD for CL Advantages. KD lypically enhances a CL framework to produce improved feature representations by transferring insights from a well-trained teacher, improving the downstream performance (Hinton et al, 2015 Furlancllo et al., 2018 Yim al.) 2017). Further, by utilizing the teacher's knowledge, up the training lime elliciency and reduce the sample requirement to learn the student model (Tian et al. 2019). A well-trained teacher can also improve the student model's robustness, leading to even better representations compared to the teacher. Finally, regularizing the student model with softer probability scores reduces the variance of Bayes-distille risk, therefore, making the model more generalizable and less prone to overfitting (Menon et al. 2021).
speed
The teacher-student framework for knowledge distillation may face several potential challenges- (a) KD with CL adds both computation overhead and training complexity (Hinton et al. 2015 Tian et al. 2019 (a) If the leacher model is flawed, the student may inherit its errors, leading to suboptimal performance (Hinton et al. (015)). (b) Compatiillity issues between teacher and student models and training inslabilily can hinder effective knowledge transfer. It requires intricate tuning of hyperparameters and designing the loss functions. (Mirzadeh et al.] 2020). (c) Further, the generalization may be restricted due to limited knowledge of the teacher with respect to the downstream tasks Yim et al.] (2017). (d) Finally, there risk of knowledge saturation, where the benefits of additional knowledge transfer may diminish after certain saturation point. (Furlanello et. al. 2018).
Limitations.
Appendix A.3 and
A.4 provide additional ablation studies to empirically investigate these challenges.
Broader Impact. KD can significantly impact graph representations, with broader implications for various fields, including bioinformatics, drug discovery, social network analysis, recommendation systems, etc. few potential impacts our work are as follows: (a) Improves the efficiency and scalability of graph representation learning by enabling 'soft' knowledge transfer from a pre-trained teacher model to a smaller, more efficient student network. (b) Improves the generalization performance of graph representation learning by leveraging the 'dark knowledge' encoded in pre-trained teacher model's representations.
References
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Compress: Self-supervised learning by compressing representations. NeurIPS, 2020.
Nesreen K Ahmed, Jennifer Neville, Ryan Rossi, and Nick Duffield. Efficient graphlet counting for large networks. Tn ICDM, 2015.
Jinheon Back, Dong Lee, and Sung Hwang. Learning to extrapolate knowledge: Transductive few-shot out-of-graph link prediction. NewrIPS, 2020.
Smriti Bhagat, Graham Cormode, and S Muthukrishnan. Node classification in social networks arXiv, 2011.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Prashant Bhat, Elahe Arani, and Bahram Zonooz. Distill the go: online knowledge distillation in self- supervised learning. In CVPR, 2021.
K.M. Borgwardt and H.P. Kriegel. Shortest-path kernels on graphs. In ICDM, 2005.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CV /EE international conference computer vision, pp. 9650-9660, 2021.
Jatin Chauhan, Aravindan Raghuveer, Rishi Saket, Jay Nandy, and Balaraman Ravindran. Multi-variate time series forecasting on variable subsets. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 76 86, 2022.
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICLR, 2020a.
Ting Chen, Simon Kornblith, et al. Big self-supervieed models are strong semi-supervised learners. NeurIPS, 2020b.
Jesse Davis and Mark Goadrich. The relationship between precision-recee and roc curves. In ICML, 2006.
Wenqi Fan, Yao Ma, Qing Lí, et al. Graph neural networks for social recommendation. In WWW, 2019.
Zhiyuan Fang, Jianfeng Wang, et al. Seed: Self-supervised distillation for visual representation. arXiv, 2021.
Matthias Fey and Jan E Lenssen. Fast graph representation learning with pytorch geometrie. arXiv, 2019.
Santo Fortunato. Community detection in graphs. Physics reports, 2010.
Tommaso Furlanello, Zachary Lipton, et al. Born again neural networks. In ICML, 2018.
Yuting Gao, Jia-Xin Zhuang, et al. Disco: Remedying self-supervised learning on lightweight models with distilled contrastive learning. In ECCV, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach sele-supervieed learning. NewrIPS, 2020a.
Jean-Bastien Grill, Florian Strub, et al. Booostrap your own latent-a approach to self-supervised lcarning. NewrIPS, 2020b.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In SIGKDD, 2016.
Michael Gutmann and Aapo Hyvärinen. Noiso-contrastive estimation: A new estimation principle for un- normalized statistical models. In AISTATS, 2010.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. NeurIPS, 2017.
Byeongho Heo, Minsik Lee, et al. Knowledge transfer via distillation of activation boundaries formed by hidden ncurons. In AAAI, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXív, 2015.
Tomaš Hoževar and Janez Demšar. A combinatorial approach to graphlet counting. Bioinformatics, 2014.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjic Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. Tn SIGKDD, 2022.
Weihua Hu, Bowen Liu, et al. Strategies for pre-training graph neural networks. In ICL.R 2020a.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Ziniu Hu, Yuxiao Dong, et al. Gpt-gun: Generative pre-training of graph neural networks. In SIGKDD, 2020b.
Dasol Hwang, Jinyoung Park, et al. Self-super auxiliary learning with meta-paths for heterogeneous graphs. NeurIPS, 2020.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In ECCV, 2016.
Gal Kaplun, Eran Malach, et al. Knowledge distillation: Bad models can be good role models. In NearIPS, 2022.
Hisashi Kashima, Koji Tsuda, and Akihiro Inokuchi. Marginalized kernels between labeled graphs. In JCML, 2003.
Dongki Kim, Jinhcon Back, and Sung Ju Hwang. Graph self-supervised learning with accurate discrepancy learning. In NewrIPS, 2022.
Dongkwan Kim and Alice Oh. How to find your friendly neighborhood: Graph attention design with self- supervision. In ICLR, 2021.
Seojin Kim, Jaehyun Nam, Junsu Kim, Hankook Lee, Sungsoo Ahn, and Jinwoo Shin. Fragment-based multi-view molecular contrastive learning. In Workshop on" MaMManee Learning for Materials "ICLR 2023, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv, 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Scung Hyun Lee, Dac Ha Kim, and Byung Cheol Song. Self-supervised knowledge distillation using singular value decomposition. In ECCV, 2018.
Xingjian Li, Haoyi Xiong, Knowledge distillation with attention for deep transfer learning of convolu- tional networks. TKDD, 2021.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. ICLR, 2022.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arviv, 2017.
Aditya Menon, Ankit Rawat, al. A statistical perspective on distillation. In JCML, 2021.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAT, 2020.
Arnab Kumar Mondal, Jay Nandy, Manohar Kaul, and Mahesh Chandran. Stochastic experience-replay for graph continual learning. In The Third Learning Graphs Conference, 2024.
Christopher Morris, Nils M Kriege, Franka Bause, et al. Tudataset: A collection of benchmark datasets for learning with graphs. Xiv, 2020.
Giulia Muzio, Leslic O'Bray, and Karsten Borgwardt. Biological network analysis with deep learning. Brief- ings in bioinformatics, 2021.
KL Navaneet, Soroush A Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Simreg: Regression as simple yet effective tool for supervipe knowledge distillation. arXiv, 2022.
Hisashi Ohtsuki, Christoph Hauert, Erez Lieberman, and Martin Nowak. A simple rule for the evolution of cooperation on graphs and social networks. Nature, 2006.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arxiu, 2018.
Adam Paszke, Sam Gross, al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. SIGKDD, 2014.
Yu Rong, Yatao Bian, et al. Self-supervised graph transformer on large-scale molecular data. NeurIPS, 2020.
Andrea Rossi, Denilson Barbosa, et al. Knowledge graph embedding for link prediction: A comparative analysis. ACM TKDD, 2021.
Susheel S, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation improve graph contrastive learning. In NewrIPS, 2021.
Alberto Sanfeliu and King-Sun Fu. distance measure between attributed relational graphs for pattern recognition. IEEE Transactions on Systems, Man, and Cybernetics, 1983.
Nino Shervashidze, Pascal Schweitzer, et al. Weisfeiler-ehmman graph kernels. JMLR, 2011.
Kaiyou Song, Jin Xie, Shan Zhang, and Zimeng Luo. Multi-mode online knowledge distillation for self- supervised visual representation learning. In CVPR, 2023.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günne- mann, and Pictro Liò. 3d infomax improves gnns for molccular property prodiction. In ICML, 2022.
Teague Sterling and John J Irwin. Zinc 15 ligand discovery for everyone. Journal of chemical information and modeling, 2015.
Fan- Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maxinization. In ICLR, 2020.
Jian Tang, Meng Qu, et al. Line: Large-scale information network embedding. In WWW, 2015.
Shantamu Thakoor, Corentin Tallec, et al. Large-scale reprosentation lcarning on graphs via bootstrapping. ICLR, 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910.1069: 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-SNE. JMLR, 2008.
Petar Veličković, William Fedus, et al. Deep graph infomax. ICLR, 2019.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir B Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 2022.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating hyperplanes. In AAAI, 2014.
Shu Wu, Yuyuan Tang, et al. Session-based recommendation with graph neural networks. In AAAI, 2019.
Zhenqin Wu, Bharath Ramsundar, et al. Moleculenet: a benchmark for molecular machine learning. Chemical science, 2018.
Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. SimGRACE: A simple framework for graph contrastive learning without data augmentation. In WWW, 2022.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In ECCV, 2020.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
Minghao Xu, Hang Wang, et al. Self-supervised graph-level representation learning with local and global structure. In TCML, 2021.
Longqi Yang, Liangliang Zhang, and Wenjing Yang. Graph adversarial self-supervised learning. NewrIPS, 2021.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimiza- tion, network minimization and transfer learning. In Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 4133 4141, 2017.
Yihang Yin, Qingzhong Wang, et al. AutoGCL: Automated graph contrastive learning via learnable view generators. In AAAT, 2022.
Yuning You, Tianlong Chen, el al. Graph contrastive learning with augmentations. NewrIPS, 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In ICML, 2021.
Junliang Yu, Hongzhi Yin, Xin Xia, et al. Are graph augmentations necessary? simple graph contrastive learning for recommendation. In SIGIR, 2022.
Zhiping Zeng, Anthony KH Tung, Jianyong Wang, Jianhua Feng, and Lizhu Zhou. Comparing stars: On approximating graph edit distance. VLDB, 2009.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. NeurIPS, 2018.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW, 2021.
Marinka Zitnik, Rok Sosič, Marcus W Feldman, and Jure Leskovee. Evolution of resilience in protein interactomes across the tree of life. Proceedings of the National Academy of Sciences, 2019.
Martin Zong, Zengyu Qiu, Xinzhu Ma, et al. Better teacher better student: Dynamic prior knowledge for knowledge distillation, In ICLR, 2023.
16

---

Published in Transactions on Machine Learning Research (11/2024)
A Experimental Setup
A.1 Molecular graph classification
This section presents the implementation details and dataset descriptions of our experiments on molecular graph classification and link prodiction tasks. For all experiments, we use PyTorch (Paszke et al.) 2019) and PyTorch Geometric libraries (Fey Lenssen 2019 with a single NVIDIA A30 Tensor Core GPU for all of our experiments.
Datasets. For our first experiments on molecular graph classification,
the ZINC dataset (Sterling & Irwin 2015), large corpus of 2 million unlabelled molecules for pretraining the teacher and student network. For the downstream tasks, we experimented with 8 labeled molecular datasets from MolecularNet (Wu et al. 2018 The molecule classes are determined using the biophysical and physiological properties.
We also present results from biological domains where the datasets are produced by the sampled ego networks from the PPI networks Zitnik et al.] (2019). We use the same experimental setup as You et al. (2021) for predicting proteins' biological functions where we pre-train and fine-tune the model using the PPI network provide the statistics of these datasets.
dataset Zitnik et al. (2019). In Table 4
Table 4: Descriptions of Molecular and PPI network datasets.
Chemical Datasets
# Grapphs
Avg.
Nodes
Avg.
Edges
Tasks
ZINC15
2,000,000
26.62
28.86
BBBP
2,039
24.06
25.95
ClinTox
1,478
26.16
27.88
MUV
93,087
24.23
26.28
17
HTV
41,127
25.51
27.47
BACE
1,513
34.09
36.86
1
SIDER
1,427
33.64
35.36
27
Tox21
7,831
18.57
19.29
12
ToxCast
8,575
18.78
19.26
617
Biological Datasets
#Graph
Avg. Nodes
Avg. Edges
Tasks
PPT (Pre-training)
306,925
39.83
364.82
PPI (Finetune)
88,000
49.35
445.39
40
Implementation details. For our proposed framework, use the same network architecture for both the teacher and the student model. In particular, we use Graph Isomorphism Networks (GINs) (Xu et al., 2019) as applied in the previous works Hu et al. (2020a); Xu et al. [2021] Kim et al. (202)). These networks consist of 5 layers with 300 dimensional embeddings for nodes and edges along with average pooling stralegies for obtaining the graph representations. To obtain distilled perception distance from the teacher network,
we use global average pooling to extract the fixed-length fcatures from each layer.
We use the official D-SLA coded provided by Kim et al. 20022 as the backbone for our experiments and apply the same perturbation strategies as used in (Kim et al. 2022). In particular, their perturbation strategy aims to minimize the risk of creating unreasonable cycles, reducing the chance of significant change in the chemical properties. For our experiments, we use three perturbations for each input sample.
We report results using two different teacher modules, trained using existing GraphLog (Xu ct al. 2021 and D-SLA (Kim et al. 2022) while training the following student network using the loss functions as proposed
in Section 3.3] We divide the perceptual distances by 4 and as we use GraphLog (Xu et al. 2021 and
D-SLA (Kim et al.
2022) the teacher, respectively. For TGCI-GraphCI, For TGCL-DSLA,
10 in Equation
use X and X₂ to 1.0 and 0.5 respectively for the student model. For L1 -o loss, we
set the temperature,
10 (Equation 7 and 0.95 (Equation Dh. For LT Lm-mangin: set B = 5. Both
teacher and student models are trained using batch-size of 256 and for 25 epochs with learning rate 1e - 3 and Adam optimizer (Kingma & Ba 2014
http:://ghhab..mmm////gginnnntn
17

---

Published in Transactions on Machine Learning Research (11/2024)
A.2 Link Prediction
Datasets. For this task, select three datasets i... COLLAB, IMDB-Binary, and IMDB-Multi from the TU dataset benchmark Morris et al. (2020).
Table 5: Statistics of social net work datasets for link prediction.
Datasets COLLAB
#Graphs
Avg. Nodes
Avg. Edges
4320
2331.4 20.13
76.12
IMDB-B
2039
85.5 16.64
IMDB-M
1478
77.9
COLLAB a dataset scientific collaboration networks. It contains 4, 320 graphs where the researcher and their collaborators are nodes and an odge indicates collaboration between two rescarchers. A rescarcher's ego network has three possible labels: High Energy Physics, Condensed Maller Physics, and Astro Physics, representing the field of the researchers.
IMDB-Binary is movic collaboration dataset. It consists of the ego networks of actors/actresser from the movies in IMDB. It consists of 2, 039 graphs For each graph, the nodes are actors /attresses,, with an edge between them if they appear in the same movie. These graphs are derived from the Action and Romance genres.
IMDB-Multi is a relational dataset that consists of network of actors or actresses, played in movies in TMDB. It contains 1,478 graphs. As before, node represents actor actress, and an edge connects two nodes when they appear in the same movie. The edges are collected from three different genres: Comedy, Romance, and Sci-Fi.
Implementation details. For our experiments, we use Graph Convolutional Network (GCN) Kipl & Welling (2017) for both teacher and student models. These networks consist of three layers with 300 dimensions for embeddings. As before, we use the same perturbation strategy as applied in Kim et al. 20022). We have also experimented with two different teachers, i.e., D-SLA Kim et al. (2022) and GraphLog Xu et al. (2021). We use a batch size of 32 and a learning rate 0.001 for training the student representation learning models. For PGCL-GraphhCi we set T 10 (Eq. 5). For TGCL-DSLA, we use \₁ and X2 0.7 and 0.0, respectively. For -o-soft loss, select the temperature, T from three different values i.c., {5, 10, 20} (Equation 7 and set a 0.95 (Eq. 9
A.3 Ablation Study on Teacher-Student Framework for Graph COntrastive Learning
In this section, we evaluate the performance of TGCL, models with various teacher models and model capacities. We also analyze the trade-off between computational overhead and convergence by applying undertrained teacher models vs. well-trained teacher model.
A.3.1 Student Network with reduced Capacity
The original goal of Knowledge Distillation (KD) (Hinton et al. 2015) was to transfer knowledge from a complex 'teacher' model to a simpler 'student' model, achieving comparable performance with reduced computational cost. However, subsequent studies demonstrate that using a complex student model, similar to the tcacher, can also enhance model robustness, efficiency, and generalization across varions applications (Furlanello al. 2018 Tian al., 2019). Although our main experiments utilize identical architectures for both teacher and student models, this section explores the impact of employing smaller student network.
Table 6 demonstrates the performance of student TGCL-DSLA model on the downstream molocular property prediction lask. We can see that, with the same capacity (i.e., 5 layers of GNN) as the teacher module of D-SLA, our proposed student network consistently outperformed the teacher. As we decrease the capacity of our student network by reducing the number of layers, the overall performance reduces. However, we observe that even with 3 layers of GNN, our student module outperforms the teacher D-SLA model. Therefore, these results demonstrate that our proposed TGCL. framework can compress the student
18

---

Published in Transactions on Machine Learning Research (11/2024)
representation network by enabling smoothened knowledge transfer from pre-trained teacher to the student representation learning model.
Table 6: Impact of the capacity of the student TGCL-DSLA models (mean + std) for graph classification. "Fuul-capacity" denotes the same capacity as the teacher.
ВВBP
ClinTTx
MUV
BACE
ToxCast
Avg
D.SLA (Kin al. 2012
72.6 1 0.8
80.2 L 1.5
76.6
0.9
78.6 1 0.4
83.8 1 1.0
60.2 1 1.1
76.8 0.5
64.2 0.5
74.13
w/ full-capacity
13.5 1: 0.9
84.9 à: 1.3
79.4
4 0.9
78.8 1: 0.5
85.2 1 0.4
61.2
1.0
76.9
4: 0.1
64.9 4: 0.2
75.60
w/
3-layer Student
74.8 = 0.4
84.6
: 1.4
76.4
= 1.0
77.9
4: 0.1
82.7 4: 1.1
61.0 1 0.3
75.0
0.1
63.6
0.4
74.48
2-layer Student
0.5
814 0.4
77.3
1.5
77.6
0.2
+ 0.4
+ 0.4
74.7
0.1
63.0
73.50
A.3.2 TGCL with multi-level teachers & Knowledge Saturation
Proposition 2 suggests that irrespective of the size/capacity of student model, statistically produces a better generalization. Hence, raises a natural question: can we further improve the performance by iteratively using the student models as the teacher to train another collow-up student network? In Table 7 investigate this by training a 2-level iterative teacher model for TGCL-DSLA framework. In other words, we use the TGCL-DSLA (w/ D-SLA) student model to train another 2nd-level student, denoted -SCL2-DSLA (w/ D-SLA).
We can see that the performance of TGCL2-DSLA (w/ D-SLA) saturates and does not improve the overall performance than the original TGCL-DSLA (w/ D-SLA). These results indicate that the TGCL-DSLA al- ready receives sufficient probability calibrations from the first-level teacher model. Hence, their performance improvement converges after the first-level teacher. This phenomenon known "knowledge saturation" in the KD literature Furlanello et al. 2018
Table Performance comparison of TGCL models with two-level teachers.
ВВBP
ClinTox
MUV
HIV
BACE
SIDER
Tox21
ToxCast
Avg.
72.6 0.8
80.2
76.6 0.9
838
60.2
0.5
74.1
TCCL-DSLA (w/ D-SLA)
73.5 0.9
84.9
79.4 0.9
78.8 0.5
85.2
0.4
61.2 1 1.0
76.9 0.1
64.9 0.2
75.6
TGCI2-DSLA (w/ D-SLA)
74.15 +0.9
84.65 +2.0
76.43 +1.3
78.67
404
84.07 0.8
62.73 0.8
76.15 0.4
64.32
0.4
75.2
A.3.3 Choice of teacher models: Computational overhead and convergence
KD enhances a CL. framework by improving feature representations but roduces additional computational overhead and training complexity. In Table S we examine the performance of the TGCL student model using an undertrained teacher model with early stopping.
We observe that the student model consistently outperforms the teacher models, even when the teacher models are undertrained. For instance, achieves an average AUROC score of 74.4%, surpassing the performance of D-SLA model trained for 100 epochs. These results suggest that the TGCL student model can achieve comparable performance without requiring fully trained teacher model. Consequently, is possible to enhance downstream performance with only a modest increase in computational overhead.
TGCL-GraphCI. model with D-SLA (trained for 20 epochs)
Table 8: Performance comparison of TGCL students using an undertrained teacher model with early stop- ping.
BBBP
ClinTox
MUV
HIV
BACE
SIDER
Tox21
ToxCast
Avg.
DSLA [100 epochs]
72.6 = US
80.2 + 1.5 76.6 = 0.9
78.6 2 0.1
83.8 # 1.0
60.2 1.1
76.8 = U.5
61.2 = U.5
74.1
Trap-GraphCL
(w/ D-SLA [20 epochs])
73.3
82.7 1 0.8 76.0 1.1
78.5 0.8
82.8 1. 0.8
62.4 1 0.2
75.1 1.
64.2 0.4
74.4
CCaI-GraphCL
(w/ D-SLA [60 opochs])
73.4 40.6
84.3 +1.3
=0.8
79.3 +006
84.9 0.6
+ 0.6
75.4
63.7 0.2
74.8
TGCI-tiraphCE
(w/ D-SLA [100 epochs])
74.0 10.4
82.8 12.2
77.0 :1.9
77.9 1033
8433 #1.0
64.2 0.3
78.6 1 0.1
64.7 :: 0.4
73.2
19

---

Published in Transactions on Machine Learning Research (11/2024)
A.4 Ablation Studies ppper-parameters and loss components
In this section, we investigate the effect of different hyperparameteess and contributions of different loss components for our TGCL framework and the scnsitivity of their hyperparameters.
A.4.1 Impact of different loss components.
In Table we first demonstrate the impact of different loss components. The first three rows demonstrate the performance of individual loss components. We observe that Lsoft is the most essential component, providing the maximum performance boost for the downstream molecular prediction tasks. The other two loss components, i.c. perperept and Lm-mariin act regularizer. While, individually, they do not perform well, incorporating them with Lsoft in Loverall, observe significant boost in the overall performance.
Table 9: Impact of individual loss components. (at = 0.95, 10)
CT/t perust
margin
BВВР
Cliu
MUV
HIV
ВАCE
SIDER
Tox21
ToxCasl
72.9 1.4
79.8 +
79.1 2 0.7
77.7 1 0.6
81.9 1 0.3
62.1 0.7
76.9 1: 0.2
64.1 = 0.3
74.31
76.6
61.7
72.90
72.7 0.06
77.9 + 2.0
74.1 0.9
76.6 +..6+00.
82. + 0.6
62.8
+ 0.5
74.2
61.9 + 0.8
72.891
73.6 + 0.5
8122
73.7 04
773
832
62.8 0.8
15.2
63.3 0.5
74.04
72.8 0.1
816
79.2
78.8
81.1
59.7 19.7003
76.3
63.8 0.1
71.20
/
72.1 1 0.5
84.0 1: 23
76.7 2 1.3
1.7
82.5 1: 0.5
61.4
76.3
64.3 = 0.7
74.40
/
A.4.2 Impact of hyper-parameters of LT-soft-
Since Cr-soft is the most important loss component, we further analyze Syper-parameters associated with it. We in Eq. g Ls is similar to the distillation loss for classification tasks, consisting of two loss components, f.e. LKD (Eq. 7 and LOGD (Eq. g Here, analyze the temperature term, for LKD. followed by the weights of these components,
Table 10: Impact of the temperature, т. (at a 0.95)
BBBP
ClinTTx
MUV
HIV
BACE
SIDER
Tox21
ToxCast
Aug
75.1 + 0.4
86.7 + 1.6
74.4 + 0.1
77.5 + 0.6
83.3 + 1.0
61.2 + 0.4
75.6 + 0.1
63.4 + 0.4
74.65
0.2
1.4
0.4
73.5
= 0.9
81.9
= 13
79.1 = 0.9
78.8 0.5
85.2 = 0.4
61.2 = 1.0
76.9 = 0.1
61.9 = 0.2
75.60
20
72.9
+ 0.6
$3.9
2.6
77.1 + 0.5
78.3 + 0.7
84.0 + 0.8
61.8 + 0.4
76.2 + 0.4
64.6 + 0.5
74.85
2.8
0.5
0.3
In Table 10 we demonstrate the results by varying the temperature, 5 to (1,5, 1,5,10 10, 20, 100). We observe that
lower temperature of 1, we achieve the best performance for BBBP and Clintox datasets, while the performance remains lower for the other datasets. On the other hand, at 100, achieve the best performance for SIDER. Finally, we obtain the most consistent result as we select 10 and set it to report our results.
Table 11: Impact of a. (at T = 10)
BBBP
Clin
MUV
HIV
BACE
SIDER
Tox21
ToxCast
Aug
73.2 + 0.7
80.2
1.8
7644 0.7
78.1 0.6
81.
0.9
62.3 = 0.5
76.5 + 0.3
63.8 = 0.3
74.20
0.5
73.6
0.7
82.5
1.2
75.0 +
78.4 + 0.6
85.6 + 0.5
62.4 + 0.1
75.8 + 0.1
64.5 + 0.3
74.73
1.3
1.0
72.8 0.6
83.6
1.2
77.6 + 166
78.8 0.4
834
61.3 = 0.6
76.6
+ 0.3
611 = 0.2
74.78
Next, in Table II we analyze the impact of a with fixed r = 10. A larger value of T provides more weight to CKD. We can see that incrcasing a a non-zero value improves the model's overall performance. However, performance tends to reduce as we choose a remove LOOD entirely. We achieve the best performance at 0.95.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Table 12: Sensitivity analysis X and A2 for TGCL - DSLA(w/ DSLA) model.
A =.=
Ag = 0.5
= 0.7
1.0
BBBP
12.55 + 0.32
72.03
= 0.50
74.46
= 1.54
73.5 = 0.9
Tlxx
BACE
83.64
+ 0.71
83.18
1.02
83.38
+ 0.33
86.2
0.4
MUV
76.28
0.91
76.00
0.37
77.08
1.45
79.4
0.9
HIV
78.61
0.62
78.93
78.64
0.46
0.5
Au 0.3
Au 0.5
Ap 0.7
1.0
BBBP
Clix
83.53
1.51
84.9
1: 1.3
81.95
1: 0.93
80.83 1: 1.68
BACE
53,33
0.31
85.2
0.4
83.60
1 0.90
83.53 = 0.72
MUV
75.79
0.78
79.4
0.9
76.75
1.54
76.02
0.11
HIV
78.68 + 0.81
78.8
0.5
78.17
+ 0.62
79.49 = 0.84
A.4.3 Sensitivity of Å and X2
Table 12 we present the performance -TCCL-DLA(w///
model as we vary A1 and X2 in Eq. We first vary d1 to (0.3, 0.5, 0.7, 1.0] as we fix X2 0.5. We observe that the performance improves
choose larger values, i.e., when set Aι to 0.7 1.0. Nexl, vary A2 to {0.3, 0.5, 0.7, 1.03 as fix A 1.0. Here, we observe that achieve the average performance set X2 = 0.5
21


Reviews:
[{"review_id": "CftQq06LW1", "reviewer": "Reviewer_N9yF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper correctly points out the limitations of using hard labels for positive and negative pairs in graph contrastive learning, which provides a strong motivation for the methodology. The experiments seem to be carefully designed and conducted.\n\nWeaknesses:\n\nAlthough Section 3.2 provides some justifications for the ideas behind the proposed method, direct theoretical guarantees are still lacking. In particular,\n1) The statement of Proposition 1 is vague. What does the equivalence exactly mean? I cannot find the relevant discussion in Section 2.1 of Gutman and Hyvarinen (2010).\n2) Proposition 2 justifies the superiority of Bayes-distilled risk in the context of supervised learning. However, without a precise equivalence between graph contrastive learning and supervised learning, it does not lead to a formal theoretical guarantee for the proposed framework.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a teacher-student framework for improving the existing contrastive learning methods for graphs. The proposed method leverages a teacher model to produce soft pseudo-labels for pairs of graphs, which help the student model to achieve better generalization via knowledge distillation. Experiments on biological and social network datasets demonstrate performance improvement over some state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Proposition 1 should be rewritten to be more precise. Is it possible to develop a stronger and more direct theoretical guarantee for the proposed method?\n\nMinor issue: The AUROC scores reported in all tables seem to have been multiplied by 100. This should be mentioned somewhere in the text.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WXlJZxxUYm", "reviewer": "Reviewer_QeW5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The idea is novel. Authors incorporate the knowledge distillation method into graph contrastive learning by injecting soft pseudo-labels into contrastive loss objective.\n2. The general KD idea is flexible to applied into any CL methods including NTXent and DSLA.\n\nWeaknesses:\n1. It is not clear why authors combine the student distance and teacher's distilled distance by D_dp(G_0, G_i) f(G_0) f(G_i). Why not D_dp(G_0, G_i) || f(G_0) - f(G_i)|| _2?\n2. The TGCL performance does not have a clear improvement for DSLA method shown in Table 2.\n3. More interpretation on how KD help those generalization in applications are of interest, while we only see that in Figure 1", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel contrastive learning framework for GNNs by incorporating more semantically-rich soft-discriminative features using pretrained teacher. Specifically it proposes teacher-guided distilled distance which is designed to accommodate the discrete nature of graphs via knowledge distillation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Authors may add more interpretation insight on how the KD helps the CL.\n2. It seems that in this paper, the teacher model is not nesseary to be larger than the student models. It can be pretrained peer GNN as well. Can you elaborate more on that?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QetiDQUSmw", "reviewer": "Reviewer_ydtR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Empirical results: The proposed methods show improvements over baselines on multiple benchmark datasets. The paper provides ablation studies and analysis of different components.\n\nWeaknesses:\n\n1. Computational overhead: The teacher-student framework introduces additional computational cost compared to single-model approaches. A detailed account of this overhead is lacking.\n2. Dependence on teacher quality: The performance gains rely on having a good pre-trained teacher model. The paper lacks a thorough analysis of how student network performance varies with teacher capacity.\n3. Limited novelty: While the teacher-student framework is applied to graphs, the core idea is borrowed from existing knowledge distillation literature.\n\nDespite presenting an interesting approach to graph representation learning, this paper falls short of making a significant contribution. The marginal performance gains do not justify the added complexity and computational cost of the teacher-student framework. The lack of thorough analysis on computational overhead and teacher-student dynamics, combined with limited novelty, weakens the paper's impact. Unless these issues are substantially addressed, rejection is recommended.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Teacher-Guided Graph Contrastive Learning (TGCL), a framework for self-supervised representation learning on graphs. TGCL uses a pre-trained \"teacher\" model to guide a \"student\" model's training by providing soft pseudo-labels based on perceptual distances between graph samples. Two variations are presented: one based on NT-Xent loss and another on the D-SLA method. Experiments on molecular property prediction and social network link prediction tasks are conducted to evaluate the approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Comments and Suggestions:\n\n- The computational overhead of the teacher-student framework should be quantified and discussed.\n- A detailed analysis of how student network performance varies with teacher capacity is required.\n- The paper should discuss potential limitations and failure cases of the proposed approach more thoroughly.\n\nMinor comments:\n\n- Figure 2 panel b D-SLA uses the same schema for attraction and repulsion\n- In section 3.2.1, \"pseudo-level\" is used - should this be \"pseudo-label\"?\n- Equation 5 has extra brackets", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a teacher-student framework for graph contrastive learning, leveraging a pre-trained teacher to generate soft pseudo-labels for improved generalization. Empirical results across benchmark datasets support the approach's effectiveness, showing notable improvements over traditional hard label-based methods. Reviewers highlighted the paper's robust empirical performance, practical value, and the flexibility to integrate with existing contrastive methods. Concerns were raised about computational overhead, reliance on teacher quality, and limited theoretical justification, particularly regarding the clarity of Propositions 1 and 2. However, the authors addressed these concerns during the response period, clarifying aspects of the computational overhead and enhancing the justification of their approach. Given its innovative contributions and practical relevance, I recommend acceptance.
Decision: Accept (poster)