Link Prediction with Non-Contrastive Learning
Authors: William Shiao
Zhichun Guo
Tong Zhao
Evangelos E. Papalexakis
Yozen Liu
Neil Shah
Venue: ICLR 2023 poster (2023)
Keywords: graph learning
graph neural networks
non-contrastive learning
link prediction

Abstract: Graph neural networks (GNNs) are prominent in the graph machine learning domain, owing to their strong performance across various tasks. A recent focal area is the space of graph self-supervised learning (SSL), which aims to derive useful node representations without labeled data. Notably, many state-of-the-art graph SSL methods are contrastive methods, which use a combination of positive and negative samples to learn node representations. Owing to challenges in negative sampling (slowness and model sensitivity), recent literature introduced non-contrastive methods, which instead only use positive samples. Though such methods have shown promising performance in node-level tasks, their suitability for link prediction tasks, which are concerned with predicting link existence between pairs of nodes (and have broad applicability to recommendation systems contexts) is yet unexplored. In this work, we extensively evaluate the performance of existing non-contrastive methods for link prediction in both transductive and inductive settings. While most existing non-contrastive methods perform poorly overall, we find that, surprisingly, BGRL generally performs well in transductive settings. However, it performs poorly in the more realistic inductive settings where the model has to generalize to links to/from unseen nodes. We find that non-contrastive models tend to overfit to the training graph and use this analysis to propose T-BGRL, a novel non-contrastive framework that incorporates cheap corruptions to improve the generalization ability of the model. This simple modification strongly improves inductive performance in 5/6 of our datasets, with up to a 120% improvement in Hits@50 - all with comparable speed to other non-contrastive baselines, and up to $14\times$ faster than the best-performing contrastive baseline. Our work imparts interesting findings about non-contrastive learning for link prediction and paves the way for future researchers to further expand upon this area.

Published as a conference paper at ICLR 2023
LINK PREDICTION WITH NON-CONTRASTIVE LEARNING
William Shiao'; Zhichun ; Zhao?, Evangelos E. Papalexakis', Yozen Liu', Neil Shah Uniiveriity of California, Riverside University of Notre Dame SSnap Inc.
[wshiaa02, epapalex)ouer edu, 2zguo50nd. edu, "[zzaa, ,onao,,yluu yliu2, nshah) nshah) com
ABSTRACT
Graph neural networks (GNNs) are prominent in the graph machine learning domain, owing to their strong performance across various tasks. A recent focal area is the space of graph self-supervised learning (SSL), which aims derive useful node representations without labeled data. Notably, many state-of-the- art graph SSL approaches are contrastive methods, which use a combination of positive and negative samples to learn node representations. Owing to challenges negative sampling (slowness and model sensitivity). recent literature introduced non-contraxtive methods, which instead only use positive samples. Though such methods have shown promising performance in node-level tasks, their suitability for link prediction tasks, which are concerned with predicting link existence between pairs of nodes, and have broad applicability to recommendation systems contexts, is yet unexplored. In this work, we extensively evaluate the performance of existing non-contrastive methods for link prediction both transductive and inductive settings. While most existing non-contrastive methods perform poorly overall, we find that, surprisingly. BGRL generally performs well in transductive settings. However, it performs poorly in the more realistic inductive settings where the model has to generalize to links to/from unseen nodes. We find that non-contrastive models tend to overfit to the training graph and use this analysis to propose T-BGRL, a novel non-contrastive framework that incorporates cheap corruptions to improve the generalization ability of the model. This simple modification strongly improves inductive performance in 5/6 of our datasets, with up to 120% improvement in Hits 50-all with comparable speed to other non-contrastive baselines, and up to 14x faster than the best-performing contrastive baseline. Our work imparts interesting findings about non-contrastive learning for link prediction and paves the way for future researchers to further expand upon this area.
INTRODUCTION
Graph neural networks (GNNs) are ubiquitously used modeling tools for relational graph data, with widespread applications in chemistry (Chen et al., 2019; Guo et al., 2021; 2022a; Liu et al., 2022), forecasting and traffic prediction (Derrow-Pinion et al., 2021; Tang et al., 2020), recommendation
systems (Ying et al., 2018b; He et al., 2020; Sankar et al., 2021; Tang et al., 2022; Fan et al., 2022), graph generation (You et al., 2018; Fan & Huang, 2019; Shiao & Papalexakis, 2021), and more. Given significant challenges obtaining labeled data, one particularly exciting recent direction is the advent of graph self- supervisdd learning (SSL), which aims to learn representations useful for various downstream tasks without using explicit supervision besides available graph structure and node features (Zhu et al., 2020; Jin et al., 2021; Thakoor et al., 2022; Bielak et al., 2022).
One prominent class of graph SSL approaches are contrastive methods (Jin et al., 2020). These methods typically utilize contrastive losses such as InfoNCE (Oord et al., 2018) or margin-base based losses (Ying et al., 2018h) 2018b) between node and negative sample representations. However, such methods usually require either many negative samples (Hassani & Ahmadi, 2020) carefully chosen ones (Ying et al., 2018b; 8b; Yang et al., 2020), where the first one results with quadratic number of Work done while interning Snap Inc.

---

Published as a conference paper at ICLR 2023
in- in-batch comparisons, and the latter is especially expensive on graphs since often store the sparse adjacency matrix instead of its dense complement (Thakoor et al., 2022; Bielak et al., 2022). These drawbacks motivated the development of con-contraetive methods (Thakoor et al., 2022; Bielak et al., 2022; Zhang et al., 2021; Kefato & Girdzijauskss, 2021), based on advances in the image domain (Grill et al., 2020; Chen & He, 2021; Chen al., 2020), which do not require negative samples and solely rely on augmentations. This allows for a large speedup compared to their contrastive counterparts with strong performance (Bielak et al., 2022: Zhang et al., 2021).
However, non-contrastive SSL methods are typically evaluated on node-level tasks, which is a more direct analog of image classification in the graph domain. In comparison, the link-level task (link prediction), which focuses on predicting link existence between pairs of nodes, is largely overlooked. This presents critical gap in understanding: Are non-contrastive methods suitable for link prediction tasks? When do they (not) work, and why? This gap presents huge opportunity, since link prediction a cornerstone in the recommendation systems community (He al., 2020; Zhang & Chen, 2019; Berg et al., 2017).
Present Work. To this end, our work first performs an extensive evaluation of non-contrastive SSL methods in link prediction contexts to discover the impact of different augmentations, architectures, and non-contrastive losses. We evaluate all of the (to the best of our knowledge) currently existing non-contrastive methods: CCA-SSG (Zhang et al., 2021), Graph Barlow Twins (GBT) (Bielak et al., 2022), and Bootstrapped Graph Latents (BGRL) (Thakoor et al., 2022) (which has the same design as the independently proposed SelfGNN (Kefato & Girdzijauskas, 2021)). )). We also compare these methods against baseline end-to-end GCN (Kipf& & Welling, 2017) with cross-entropy loss, and two contrastive baselines: GRACE (Zhu et al., 2020), and GCN trained with max-margin loss (Ying et al., 2018a). We evaluate the methods in the transductive setting and find that BGRL (Thakoor al., 2022) greatly outperforms not only the other contrasste methods, but also GRACE strong augmentation based contrastive model for node classification. Surprisingly, BGRL even performs on-par with a margin-loss GCN (with the exception of 2/6 datasets). However, in the more realistic inductive setting, which considers prediction between new edges and nodes at inference time, we observe a huge gap in performance between BGRL and a margin- GCN (ML-GCN). Upon investigation, we find that BGRL is unable to sufficiently push apart the representations of negative links from positive links when new nodes are introduced, owing to a form of overfitting. To address this, we propose T-BGRL, novel non-contrastive method which uses corruption function to generate cheap "negative" samples- -without performing the expensive negative sampling step of contrastive methods. We show that it greatly reduces overfitting tendencies, and outperforms existing non-contrastive methods across 5/6 datasets on the inductive setting. We also show that maintains comparable speed with BGRL, and is 14x faster than the margin-loss GCN on the Southoor-Physiee dataset.
Main Contributions. In short, our main contributions are as follows:
To the best of our knowledge, this is the first work to explore link prediction with non-contrastive
SSL methods.
We show that, perhaps surprisingly. BGRL (an existing non-contrastive model) works well in the transductive link prediction, with performance par with contrastive baselines, implicitly behaving similarly to other contrastive models in pushing apart positive and negative node pairs.
We show that non-contrastive SSL models underperform their contrastive counterparts in the inductive setting, and notice that they generalize poorly due to lack of negative examples.
Equipped with this understanding, we propose T-BGRL, novel non-contrastive method that uses cheap "negative" samples to improve generalization. T-BGRL is simple to implement, very efficient when compared to contrastive methods, and improves on BGRL's inductive performance in 5/6 datasets, making it at above par with the best contrastive baselines.
PRELIMINARIES
Notation. We denote a graph = (V,E), where V is the set of nodes (i.e., [VD and C V V be the set of edges. Let the node-wise feature matrix be denoted by X E R=*f, where is the number of raw features, and its i-th row as is the feature vector for the i-th node.
2

---

Published as a conference paper at ICLR 2023
Let A {0, 10 denote the binary adjaceney matrix. We denote the graph's learned node representations as H E R=xd where is the size of latent dimension, and h; is the representation for the i-th node. Let Y E {0.1]=xm be the desired output for link prediction. as E and A may have validation and test edges masked off. Similarly, let Y E {0, 1 be the output predicted by the decoder for link prediction. Let ORC be perfect oracle function for our link prediction task, i.e., ORC(A, X) Y. Let NEIGH(u) {v (u, v) E & V (u, u) 8). Note that we use the terms "embedding" and "representation" interchangeably in this work.
GNNs for Link Prediction. Many new approaches have also been developed with the recent advent of graph neural networks (GNNs). A predominant paradigm is the use of node-embedding- based methods (Hamilton et al., 2017: Berg et al., 2017; Ying et al., 2018b; Zhao et al., 2022b). Node- embedding- based methods typically consist of an encoder H ENC(A, X) and a decoder DEC(H)). The encoder model is typically a message-passingg passing based Graph Neural Network (GNN) (Kipf & Welling, 2017; Hamilton et al., 2017; Zhang et al., 2020). The mnessage-passingg iterations of a GNN for a node u can be described as follows:
h (1++) UPDATE (k) (n AGGREGATE (4) ((h Vu NEIGH(W)))
(I)
where UPDATE and AGGREGATE are differentiable functions, and h. = zu. The decoder model is usually an inner product MLP applied on concatenation of Hadamard product of the source and target learned node representations (Rendle et al., 2020; Wang et al., 2021).
Graph SSL. Below, we define a few terms used throughout our work which helps set the context for our discussion.
Definition 2.1 (Augmentation). An augmentation AuG is a label-preserving random transfor- mation function AUG : (A, (A, that does not change the oracle's expected value: B[ORC(AUG" (ORRCAAU"(AAXXX (A,X)) Y
Definition 2.2 (Corruption). A corruption AUG is a label-altering random transformation AUG X) + (Á, X) that changes the oracle's 's expected value: E|ORC(AUG (A, x))] + Y.
Definition 2.3 (Contrastive Learning). Contrastive methods select anchor samples (e.g. nodes) and then compare those samples to both positive samples (e.g. neighbors) and negative samples (e.g. non-neighbors) relative to those anchor samples.
Definition 2.4 (Non-Contrastive Learning). Non-contrastive methods select anchor samples, but only compare those samples variants of themselves, without leveraging other samples in the dataset.
BGRL. While examine the performance of all of the non-contrastive graph models, focus our detailed analysis exclusively on BGRL.2 (Thakoor et al., 2022) due to its superior performance in link prediction when compared GBT (Bielak al., 2022) and CCA-SSG (Zhang et al., 2021).
BGRL consists of two encoders, one of which is referred to as the online encoder ENC₃; the other referred to as the target encoder ENC₁- BGRL also incorporates predictor PRED (typically
MLP) and two sets of augmentations: AA,t AA. A single training step for BGRL is as follows: (a)
we apply these augmentations: (Ā X (1) AUG XA; (Ā (2)) AUG, (A, X). (b)
we perform forward propagation H = ( ()) H2 = ())( (2) (2) (c) we pass the
the mean pairwise cosine distance of Z and H2 as the loss (see Eqn. 2). (e) ENC, is updated via backpropagation and ENC, is updated via exponential moving average (EMA) from ENCg. The BGRL. loss is as follows:
output through the predictor Z PRED(H,), (d)
2 n-
1 h
LBGRL =
(2)
n
In the next section, we evaluate BGRL and other non-contrastive link prediction methods against contrastive baselines.
Note that the definition of these functions are different from the corruption functions in Zhu al. (2020) (which we define as augmentations) and are instead similar to the conruption functions Veličković et al. (2018). 2Self-GNN (Kefato & Girdzijauskas, 2021), which was published independently, also shares the same architecture. As such, we refer to these methods as BGRI.
3

---

Published as a conference paper at ICLR 2023
3 Do NON- NON-CONTRASTIVE LEARNING METHODS PERFORM WELL ON LINK PREDICTION TASKS?
Several non-contrastive methods have been proposed and have shown effectiveness in node classifica- tion (Kefato & Girdzijauskas, 2021; Thakoor et al., 2022; Zhang et al., 2021; Bielak et al., 2022). However, none of these methods evaluate or target link prediction tasks. We thus aim to answer the following questions: First, how well do these methods work for link prediction compared to existing contrastive/end-to-end baselines? Second, do they work equally well in both transductive and inductive settings? Finally, if they do work, why; if not, why not?
Differences from Node Classification. Link prediction differs from node classification in several key aspects. First, we must consider the embedding of both the source and destination nodes. Second, we have a much larger set of candidates for the same (raph-O(n²)) instead of O(n). Finally, real applications, link prediction is usually treated as ranking problem, where we want positive links to be ranked higher than negative links, rather than as a classification problem, c.g. in recommendation systems, where we want retrieve the top-k most likely links (Cremonesi et al., 2010; Hubert et al., 2022). We discuss this in more detail in Section 3. 1 below. Given these differences, it is unclear methods performing well node classification naturally perform well on link prediction tasks.
Ideal Link Prediction. What does it mean to perform well on link prediction? We clarify this point here. For some nodes u, v, E V, let (u, v) & and (u, w) E E. Then, ideal encoder for link prediction would have DiST(hu: h.) < Dhs hw) for some distance function DIST. This idea the core motivation behind margin-loss-based models (Ying al., 2018a; 8a; Hamilton et al., 2017).
3.1 EVALUATION
Datasets. We use datasets from three different domains: citation networks, co-authorship networks, and co-purchase networks. We use the Cora and Citeseer citation networks (Sen et al., 2008), the Coauthor-CS and Coauthor-Physicce co-authorship networks, and the Amazon crazzonCommpters and Shaazon-Photon co-purchase networks (Shchur et al., 2018). We include dataset statistics in Appendix A.1.
Metric. Following work in the heterogeneous information network (Chen et al., 2018), knowledge- graph (Lin et al., 2015), and recommendation systems (Cremonesi et al., 2010; Hubert et al., 2022) communities, we choose to use Hits@k k over AUC-ROC metrics, since we often empirically prioritize ranking candidate links from a selected node context (e.g. ranking the probability that user A will buy item B, C, or D), as opposed to arbitrarily ranking a randomly chosen positive over negative link (e.g. ranking whether the probability that user A buys item B is more likely than user C' does not buy item D). We report Hits (k- = 50) to strike a balance between the smaller datasets like Cora and the larger datasets like Coauthor -hhyiirs However, for completeness of the evaluation, we also include AUC-ROC results in Appendix A.8.
Since goal is to evaluate the performance of the encoder, we the same decoder for all of our experiments across all of the methods. The choice of decoder has also been previously we use the best-performing decoder- a Hadamard product MLP.
Decoder.
studied (Wang al., 2021; 2022), For candidate link (u, F). we have Y DEc(h, * h.)) where * represents the Hadamard product, and DEC is a two-layer MLP (with 256 hidden units) followed by sigmoid. For the self-supervieed methods, we first train the encoder and freeze its weights before training the decoder. As a contextual baseline, we also report results on an end-to-end GCN (E2E-GCN) for which we train the encoder and decoder jointly, backpropagating a binary cross-entropy loss on link existence.
3.1.1 TRANSDUCTIVE EVALUATION
Transductive Setting, We first evaluate the performance of the methods in the transductive setting, where we train on Gtrain = (V, Etrain) for Etrain C E, validate our method on Gyal = (V, Evat) for Eval (E Etrain), and test on Gtest (V, Etest) for Etest ع Etrain - Eval- Note that the same nodes are present in training, validation, and testing. We also do not introduce any new edges during inference iimee-iffeeene is performed on Errain
Results. The results of our evaluation are shown in Table 1. As expected, the end-to-end GCN generally performs the best across all of the datasets. We also find that CCA-SSG and GBT similarly
4

---

Published as a conference paper at ICLR 2023
Table 1: Transductive performance of different link prediction methods. We bold the best- pert-performing method and underline the second-best method for each dataset. BGRL consistently outperforms other non-contrastive methods and GRACE, and also outperforms ML-GCN, on 3/6 datasets.
End-To-End
Contrastive
Con-Contrastive
Dataset
E2F-GGN
MI.-GCN
GRACE
CCA-SSG
GBT
BGRL.
Cora
0.816
0.815=a0c
0.686+0006
0.348 +0.01
0.460=a.14
0.792-0.05
Citeseer
0.822a.om
0.77 1 2:00200
0.707
0.249.amm
0.472.20. 196
0.858 20000
Amazon-Photos
0.430
0.486
0.369 iams
0.434_omx
0.562
Amazon-Computera
0.426=00606
0.320=a000
0.240
0.201
0.258=0000
0.346-0.01
Coauthor-CS
0.762
0.787 =0.011
0,456
0.229 L0.DI
0,298
0.003
0.515
Poauthor-Phyeics
0.798
0.810
OOM
0.157 =0.00
0.187
0.476
BORL NogiliveeNeallvee Link Scores
ML-GON PosiliveNNegalive Link Scores
3.
Pootev Unns Negstive Links
11
Negative Links
30
30
Recant
Percent
1.5
0.3
0.0
-00
Similaity
0.5 1.0 -0.2
0.0
Simlerit
Figure 1: These plots show similarities between node embecdings. Left: distribution of positive/nega- tive link similarities for BGRL. Right: distribution of positive/negative link similarities for ML-GCN. We can see that while they behave similarly, the ML-GCN does a better job of ensuring that posi- tive/negative links well separated. These scores are computed on mmazon-Photonn
perform poorly relative to the other methods. This is intuitive, as neither method was designed for link prediction and were only evaluated for node classification in their respective papers. Surprisingly, however, BGRL outperforms the ML-GCN (the strongest contrastive baseline) on 3/6 of the datasets and performs similarly on other (Cora). It also outperforms GRACE across all of the datasets.
Understanding BGRL Performance. Interestingly, we find that BGRL exhibits similar behavior to the MI-GGNN on many datasets, despite the BGRL. loss function (see Equation (2)) not explicitly optimizing for this. Relative to an anchor node we can express the max-margin loss of the MI-GGN as follows:
L(u) (Nsca(((( [ (TunnENNNGH(Ll) J (u, v, w)]
(3)
where J (u, U, is the margin ranking loss for an anchor u. positive sample v, and negative w:
J (u, v, w) max h. h, h. hw A}
(4)
and A is hyperparameter for the size of the margin. This seemingly explicitly optimizes for the aforementioned ideal link prediction behavior (anchor-aware ranking positive over negative links). Despite these circumstances, Figure shows that both BGRL and ML-GCN both clearly separate positive and negative samples, although ML-GCN pushes them further apart. We provide some intuition why this may occur in Appendix A.10 below.
Why Does BGRL Not Collapse? The loss function for BGRL (see Equation (2)) is 0 when h(?) 0 or 2, 0, i.e., the loss is minimized when the model produces all-zero outputs. While theoretically possible, this is clearly undesirable behavior since this does not result in useful embeddings. We refer to this case model collapse. It is not fully understood why non-contrastive models do not collapse, but there have been several reasons proposed in the image domain with both theoretical and empirical grounding. We discuss this more in Appendix A.9. Consistent with the findings from Thakoor et al. (2022), we find that collapse does not occur in practice (with reasonable hyperparameter selection).
Conclusion. We find that CCA-SSG and GBT generally perform poorly compared to contrastive baselines. Surprisingly, we find that BGRL generally performs well in the transductive setting by successfully separating positive and negative link distance distributions. However, this setting may
5

---

Published as a conference paper at ICLR 2023
Table 2: Performance of various methods in the inductive setting. See Section 3.1.2 for an explanation of our inductive setting. Although we do not introduce T-BGRL until Section 4, we include the results here to save space.
End-To-End
Contrastive
Non-Contrastive
Dataset
E2E-GCN
MI.GGCN
GRACE
GBT
CCA-SSG
BGRL
T-BGRL
Overall
Cora
0.523+ams
0.490=0
0.448
0.135
0.120=am1
0.324=0.154
0.568
Citeseer
0.621
0.66 gamms
0,514
2.305 200006
0.170gmmm
0.526
0.727
Coauthor-Ca
0,484
0.572
0.313
0.1 0.1825000
0.176
0.438=0.025
0.534
Phyytic
0.386220s
0.550
QOM
0.11220014
0.037.00m:
0.439_o00s
0.463.10.0
Amazon-Conputenn Computtrr
0.179
0.279
0.212
20011
g.155
0.270
0.312
Amazon Photos
0.420
0.478 2008
0.262
0.289 20052
0.182
0.460
0.450
Performance on Observed-Observell Node Edges
Cora
0.574gmi0
0,557 0.490
0.149
0.124
0.345=001mm
0.624
Citeseer
0.610aam
0.6022a0.00
0.358=00031
0,197
0.605=0045
0.768
Coauthor-Ca
0.5042000
0.591 2:0044
0.33210008
0.177
0.462..0.m
0.535
Coauthor- ca
0.390
0.566
0OM
2.017
0.039gu0ss
0.445
2.469
Amazon Computerr
0.177:000
0.278
20
0.212
0.169:0..0
0.155
0.270
0.313
Amazon-Phone Photos
0.418gmmm
0.483 2o0
0.265 200m
0.295 2000
0.185 g0oo
0.467 2000s
0.457.90s
Performance on
Node Edges
Cora
0.4622mm2
0.487:2mm1
0.367:2004
0.128ggo.rs
0.115
0.30920.115
0.528
Citeseer
0.645200s
0.70520ms
0.4581000
0.280
0.148
0.487_20044
0.708
Coauthor-Cs
0.459
0.545
0.284
0.175
0.177
0,402220025
0.536
Coauthor-Physte Physice
0.379
0.525 2nss
OOM
0.1061
0.035
0.429=000s
0.455
Amazon- Computtee
0.183
0.28 ganees
12000
0.177
0.155gnmm
0. 0.270_0001
0.312.90m
Aaszon-Photos
0.424
0.470=0007
0.258
0.279
0.178:
0.4499:00002
0.439
Performance Unobserved-Unobsett Node Edges
Cora
0.2392mmm
0.507
0.252.0006
0.100
0.125
0.287_20.1
0.463
Citeseer
0.595
0.681 ::
0.287
0.137:2009
0.126;naa
00.08
0.595
Coauthor-Ga
0.3722000
0.483:2004
0.230.2019
0.159
0.157
0.341
0.517:1000
Coauthor-Physi cs
0.3655.mm
0.505 gowws
OOM
0.098
0.034gnmm
0.424_0001
0.445
Amazon- Anszou-Computere
0.183:2000
0.275
0.214
0.181
0.155
0.265
=0.002
0.005
Anazon-Photos
0.41920.2
0.461:00:4
0.251 10000
0,265-2004
0.172an164
0.442
0.4161000
not be representative of real-world problems. In the next section, evaluate the methods in the more realistic inductive setting to see if this performance holds.
3.1.2 INDUCTIVE EVALUATION
Inductive Setting. While we observe some promising results in favor of non-contrastive methods (namely, BGRL) in the transductive setting, we note that this setting is not entirely realistic. In practice, we often have both new nodes and edges introduced at inference time after our model is trained. For example, consider a social network upon which a model trained at some time t but is used for inference (for a GNN, this refers to the message-passing step) at time f2, where new users and friendships have been added to the network in the interim. Then, the goal of a model run at time would be to predict any new links new network state l3 (although we assume there no new nodes introduced at that step since we cannot compute the embedding of nodes without performing inference on them first). To simulate this setting, we first partition the graph into two sets of nodes: "observed" nodes (that we see during training) and "unobserved nodes" (that are only used for inference and testing). We then withhold portion of the edges at each of the time steps ts, t2, t to serve as testing-only. inference-only, and training-only edges, respectively. We describe this process more detail in Appendix A.4.
Results. Table 2 shows that in the inductive setting, BGRL is outperformed by the contrastive MI-GCN on all datasets. It still outperforms CCA-SSG and GBT, but it is much less competitive in the inductive setting. We next ask: what accounts for this large difference in performance?
Why Does BGRL Not Work Well in the Inductive Setting? One possible reason for the poor performance of BGRL in the inductive setting is that it is unable to correctly differentiate unseen positives from unseen negatives, i.e., it is overfitting on the training graph. Intuitively, this could happen due to a lack of negative samples- BGRL never pushes samples away from each other. We show that this is indeed the case in Figure 2, where BGRL's negative link score distribution has heavy overlap with its positive link score distribution. We can also this behavior in Figure 1 where the

---

Published as a conference paper at ICLR 2023
Citssoer Negalive Inductive Link Scores
Citssoor Positive inductive Link Scores
10a
T-RGI BCPL
BGAIL
Cola
Peront
Parcent
00
0.0 08.2
085 1.0 6.0 0.25 0.11 0.5 0..5 1.0
Figure 2: These plots show similarities between node embeddings on Citeseer. Left: distribution of similarity non-neighbors for T-BGAL and BGRL. Right: distribution of similarity neighbors for T-BGRL and BGRL.. Note that the y-axis is on a logarithmic scale. T-BGRL clearly does a better job of ensuring that negative link representations are pushed far apart from those of positive links.
ENCp
PRED
(A.X)
Auoi
(APS 1R9
Auo
ENC,
AuG
Figure 3: T-BGRL architecture diagram. The loss function is also shown in Equation (5).
ML-GCN does a clearly better job of pushing positive/negative samples far apart, despite BGRL's surprising success. Naturally, improving the separation between these distributions increases the chance of a correct prediction. We investigate this hypothesis Section below and propose T-BGRL (Figure 3), novel method to help alleviate this issue.
4 IMPROVING INDUCTIVE PERFORMANCE IN A NON-CONTRASTIN FRAMEWORK
In order reduce this systematic gap in performance between ML-GCN (the bert-performing contrastive model) and BGRL (the best- Dest-performing non-contrastive model), we observe that we need push negative and positive node pair representations further apart. This way, pairs between new nodes-introducc at inference time-have a higher chance of being classified correctly. Contrastive methods utilize negative sampling for this purpose, but we wish avoid negative sampling owing to high computational cost. In lieu of this, we propose simple, yet powerfully effective idea below.
To emulate the effect of negative sampling without actually performing it, we propose BGRLL (T-BGRL). In addition to the two augmentations performed during standard non-contrastive SSL training, we add a corruption to function as a cheap negative sample. For each node, like BGRL, minimize the distance between its representations across two augmentations. However, taking inspiration from triplet-style losses (Hoffer & Ailon, 2014), we also maximize the distance between the augmentation and corruption representations.
Model Intuition.
Model Design. Ideally, this model should not only perform better than BGRL in the inductive setting, but should also have the same time complexity BGRL. In order meet these expectations, design efficient, linear- time corruptions (same asymptotic runtime as the augmentations). We also choose to the online encoder ENC, to generate embeddings for the corrupted graph so that

---

Published as a conference paper at ICLR 2023
T-BGRL does not have any additional parameters. Figure illustrates the overall architecture of the proposed T-BGRL, and Algorithm 1 presents PyTorch-style pseudocode. Our new proposed loss function is as follows:
h4
2
h (2)
LTGRL
(5)
(2)
1=0
1=0
T-BGRL Loas Term
BGRI.
where À is a hyperparameter controlling the repulsive forces between augmentation and corruption.
Corruption Choice. We experiment with sev-
Algorithm 1: PyTorch-style pseudocode for
eral different corruptions methods, but limit ourselves linear-time corruptions in order
T-BGRL
to maintain the efficiency of BGRL. We find
Enc_o: online encoder network
# tnc_t: target encoder network
that SHUPLEFFATTANDDMEDDEEC X), where {~ 1 and X SHUFFLEROWS( X) works the best. We de- scribe each of the different corruptions we ex- perimented with in Appendix A.7.
X)
Pred: predictor netuork
u lam: trade-off
decay: EMA decay paraneter
# g: input graph
feat: node features
gl, feat = augnent (g, feat) # augmentation #1 g2, feat2 augnent (g, feat) augnentation #2 c_g, c_feat corrupt(g, feat) # corruption
Inductive Results. Table 2 shows that T-BGRL
improves inductive performance over BGRL in
hi Enc_a(gt, feat1)
5/6 datasets, with very large improvements in
h2 = Enc_t inc_t(g2, feat2)
c_z Enc_t(c_E. c_feat)
the Cora and Citeseer datasets. The only
21 - Pred(z1)
dataset where BGRL outperformed T-BGRL is the Amazon-Phote is much smaller (0.01 difference in Hits @50) than the improvements the other datasets. We plot the scores output by the decoder for unseen negative pairs compared to those for unseen pos- itive pairs in Figure 2. We can see that T-BGRL pushes apart unseen negative and positive pairs much better than BGRL.
loss lam+cosine, seeeeosine_similene similarity(11 c_z) \
dataset. However, this gap
(1-1am)+co6ine. wwweooiineeemmllaeeeeeeeeeeeeaee similarity(1,,, h2)
loas. backward() backprop
Update Enc_t with EMA
Enc_t.pa params dacay Enc Enc_t.pan papamss
(1-decay) Enc p.prams
Transductive Results. We also evaluate the performance of T-BGRL in the transductive set- ting to ensure that it does not significantly re- duce performance when compared BGRL.
Table 3: Transductive performance of T-BGRL compared to ML-GCN and BGRL (same numbers as Table above; full figure in Table 5).
Dataset
ML-GCN
BGRL
T-BGRL
See Table 3 on the right for the results.
Cora
0.815
0.792
0.773 mooo
Difference from Contrastive Methods. While our method shares some similarities with con- trastive methods, we believe T-BGRL strictly non-contrastive because it does not require the O(n²) sampling from the complement of the edge index used by contrastive methods. This is clearly shown in Figure where T-BGRL and BGRL. have similar runtimes and are much faster than GRACE and ML-GCN. The corruption can be viewed as a "negative" augmentation-wiin for each link. In fact, one of the corruptions that we consider, PERRIIFFFATSPARSSIE EEAASIIFEEDEEE, is essentially the same the augmentations using by BGRL (except with much higher drop probability). We discuss other corruptions below in Appendix A.7.
Citeseer
0.771
0.858
0.868 =0c23
Coauthor-Cs
0.787
0.515
0.555 suoce
Poauchor-Physics
0.810
0.476
0.471
Amazon-Computers
0.320
0.346
0.315
Amazon-Photos
0.430
0.562
0.517
the only difference being that it changes the expected label
Scalability, We evaluate the runtime of our model on different datasets. Figure shows the running times to fully train model for different contrastive and con-contrastive methods over 5 different runs. Note that we use fixed 10,000 epochs for GRACE CCA-SSG, GBT, BGRL, and T-BGRL, but use early stopping on the ML-GGN with maximum of 1,000 epochs. We find that (i) T-BGRL is comparable to BGRL in runtime owing to efficient choices of corruptions, (ii) is about 4.3x faster than GRACE on Amazon-Computere (the largest dataset which GRACE can run on), and (ii) it is 14x faster than ML-GCN. CCA-SSC SSG is the fastest of all the methods but performs the worst. As mentioned above, methods due to how slow they are during inference. SUREL (Yin et al., 2022) is -250x slower, and SEAL (Zhang& & Chen, 2018) is about ~3900x slower according Yin et al. (2022). In conclusion,
do not compare with SEAL (Zhang & Chen, 2018) or other subgraph-bast based
8

---

Published as a conference paper at ICLR 2023
we find that T-BGRL is roughly scalable as other non-contrastive methods, and much more scalable than existing contrastive methods.
OTHER RELATED WORK
Link prediction is long- standing graph machine learning task. Some traditional methods include (i) matrix (Menon & Elkan, 2011; Wang et al., 2020) or tensor factor- ization (Acar et al., 2009; Dunlavy et al., 2011) methods which factor the adjacency and/or fea- ture matrices to derive node representations which can predict links equipped with inner products, and (ii) heuristic methods which score node pairs based
Link Prediction.
Amazon-Computerr
10*
Coauthor-CS
Coauthor-Physics
Toral Runlime
1454
10a
neighborhood and over- lap (Yu et al., 2017; Zareic& & Sakellariou, 2020; Philip al., 2010). Several shallow graph em- bedding methods (Grover & Leskovec, 2016; Perozzi al., 2014) which train node embed- dings by random-walk strategies have also been used for link prediction. In addition to the node- embedding- based GNN methods mentioned in Section 2, several works (Yin et al., 2022; Zhang & Chen, 2018: Hao al., 2020) propose subgraph-bas beased methods for this task, which aim classify subgraphs around each candidate link. Few works focus on scalable link prediction with distillation (Guo et al., 2022b), decoder (Wang et al., 2022), and sketching designs (Chamberlain et al., 2022).
ML-GCN GRACE CCA-SSG GBT
BGRL
T-BGRL
Model
Figure 4: Total runtime comparison of different contrastive and non-contrastive methods. T-BGRL and BGRL have relatively similar runtimes and are significantly faster than the contrastive methods (GRACE and ML-GCN).
Graph SSL Methods. Most graph SSL methods can be put categorized into contrastive and non- contrastive methods. Contrastive learning has been applied to link prediction with margin-loss-base methods such as PinSAGE (Ying et al., 201 (a), and GraphSAGE (Hamilton et al., 2017), where negative sample representations are pushed apart from positive sample representations. GRACE (Zhu et al., 2020) uses augmentation (Zhao et al., 2022a) during this negative sampling process to further increase the performance of the model. DGI (Veličković et al., 2018) leverages mutual information maximization between local patch and global graph representations. Some works (Ju et al., 2022; Jin et al., 2021) also explore using multiple contrastive pretext tasks for SSL. Several works (You et al., 2020; Lin et al., 2022) also focus on graph-level contrastive learning, via graph-level augmentations and careful negative selection. Recently, non-contrastive methods have been applied to graph representation learning. Self-GNN (Kefato & Girdzijauskas, 2021) and BGRL (Thakoor et al., 2022) use ideas from BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021) to propose a graph framework that does not require negative sampling. We describe BGRL depth in Section 2 above. Graph Barlow Twins (GBT) (Bielak et al., 2022) is adapted from the Barlow Twins model in the image domain (Zbontar et al., 2021) and uses cross-correlation to learn node representations with shared encoder. CCA-SSG (Zhang et al., 2021) uses ideas from Canonical Correlation Analysis (CCA) (Hotelling, 1992) and Deep CCA (Andrew et al., 2013) for their loss function. These models somewhat similar in that it has also been shown that Barlow Twins is equivalent to Kernel CCA (Balestriero & LeCun, 2022).
CONCLUSION
To our knowledge, this is the first work study non-contrastive SSL methods and their performance on link prediction. We first evaluate several contrastive and non-contrastive graph SSL methods on link prediction tasks, and find that surprisingly, one popular non-contrastive method (BGRL) is able to perform well in the transductive setting. We also observe that BGRL struggles in the inductive setting, and identify that it has tendency to overfit the training graph, indicating it fails to push positive and negative node pair representations far apart from each other. Armed with these insights, we propose T-BGRL, a simple but effective non-contrastive strategy which works by generating extremely cheap "negatives" by corrupting the original inputs. T-BGRL sidesteps the expensive negative sampling step evident in contrastive learning, while enjoying strong performance benefits. T-BGAL improves on BGRL's inductive performance in 5/6 datasets while achieving similar transductive performance, making it comparable to the best contrastive baselines, but with a 14x speedup over the best contrastive methods.
9

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To ensure reproducibility. our source code is available online at https://github: com/ resp-reeearchhoeeee contrastive -iiii---ddiiiion... The hyperparameters and instructions for reproducing all experiments are provided in the README md file.
ACKNOWLEDGMENTS
UCR coauthors were partly supported by the National Science Foundation under CAREER grant no. IIS 2046086 and were also sponsored by the Combat Capabilitties Development Command Army Research Laboratory under Cooperative Agreement Number 1911NF-13-200045 (ARL Cyber Security CRA). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding parties.
REFERENCES
Evrim Acar, Daniel Dunlavy, and Tamara G. Kolda. Link prediction on evolving data using matrix and tensor factorizations. In 2009 IEEE International Conference on Data Mining Workshops, pp- 262-269, 2009. doi: 10.1 N.1109/CDDW.2009.55
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pp. 1247-1255, PMLR, 2013.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint arXiv:2205.I/500, 2022.
Rianne van den Berg, Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. arXiv preprint arXiv: (706.022700222 2017.
Piotr Bielak, Tomasz Kajdanowicz, and Nitesh V. Chawla. Graph barlow twins: A self-supervised
representation learning framework for graphs. Knowledge-Bused Systems, pp. 109631, 2022. ISSN 0950-7051. doi:
URL https://www.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL httph:: setps://wwy wwdd/bbwwww..b......../mm... wandb. com/ Software available from wandb.com.
Lei Cai, Jundong Li, Jie Wang, and Shuiwang Line graph neural networks for link prediction, 2020. URL http:///axvvvvv/aavaee
10046.
Benjamin Paul Chamberlain, Sergey Shirobokov, Emanuele Rossi, Fabrizio Frasca, Thomas Markovich, Nils Hammerla, Michael M Bronstein, and Max Hansmire. Graph neural networks for link prediction with subgraph sketching. arXiv preprint arXiv:2209. 200.22299944466 15486,
2022.
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks universal machine learning framework for molecules and crystals. Chemistry of Materials, 31(9):3564-33 35722, 2019.
Hongxu Chen, Hongzhi Yin, Weiging Wang, Hao Wang, Quoe Viet Hung Nguyen, and Xue Li. Pme: Projected metric embedding on heterogeneous networks for link prediction. Proceedings the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '18, pp. 1177-1186, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450355520. doi: 10.1 /0.1145/3219889..221 3219986.
URL https:
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ICML, Proceedings of Machine Learning Research, pp. 1597-1607, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the JEEF/CVF Conference Computer Vision and Patern Recognition, pp. 15750-15758, 2021.
10

---

Published as a conference paper at ICLR 2023
Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. Performance of recommender algorithms top-n recommendation tasks. In Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys 10, pp. 39-46, New York. NY, USA, 2010. Association for Computing Machinery. ISBN 9781 605589060. doi: 10.1 00.115/18647088 1864721. URL https: /doi. 1///10.11453
1864708. 1864721
Austin Berrow-Pinion, Jennifer She, David Wong, Oliver Lange, Todd Hester, Luis Perez, Marc Nunkesser, Seongjae Lee, Xueying Guo, Brett Wiltshire, et al. Eta prediction with graph neural networks in google maps. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 3767-3776, 2021.
Daniel M Dunlavy, Tamara G Kolda, and Evrim Acar. Temporal link prediction using matrix and tensor factorizations. ACM Transactions on Knowledge Discovery from Data (TKDD), 5(2):1-27, 2011.
Shuangfei Fan and Bert Huang. Labeled graph generative adversarial networks. arXiv preprint arXiv:1966.03220 2019.
Wenqi Fan, Xiaorui Liu, Wei Jin, Xiangyu Zhao, Jiliang Tang, and Qing Li. Graph trend filtering networks for recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 112-121, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-super learning. Advances in neural information processing systems, 13:21271-21231 2020.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In KDD, pp. 855-864 864. ACM, 2016.
Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh V Chawla. Few-shot graph learning for molecular property prediction. In Proceedings of the Web Conference 2021, pp. 2559-2567, 2021.
Zhichun Guo, Bozhao Nan, Yijun Tian, Olaf Wiest, Chuxu Zhang, and Nitesh V Chawla. Graph-based molecular representation learning. arXiv preprint arXiv:2207.04869 2022a.
Zhichun Guo, William Shiao, Shichang Zhang, Yozen Liu, Nitesh Chawla, Neil Shah, and Tong Zhao.
Linkless link prediction via relational distillation. arXiv preprint arXiv:2210.05801, 2022b.
William L. Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. In NIPS. pp. 1024-1034, 2017.
Yu Hao, Xin Cao, Yixiang Fang, Xike Xie, and Sibo Wang. Inductive link prediction for nodes having only attribute information. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, jul 2020. doi: 10. (0.22463/ijjcii..o 2020/168.. 2F168.
URL https:///oi....... org/ 10. 24963/2F1jcaa 2020%
Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-view representation learning on graphs. In ICML,, volume 119 of Proceedings of Machine Learning Research, pp. 4116 PMLR, 2020.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp. 639-648, 2020.
Flad Hoffer and Nir Ailon. Deep metric learning using triplet network, 2014. URL httthtttp///aiiv org/abs/ 2221.66622
Harold Hotelling. Relations between two sets of variates. In Breakthroughs in statistics, pp. 162-190. Springer, 1992.
11

---

Published as a conference paper at ICLR 2023
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint wrXiv:2005.00687, 2020.
Nicolas Hubert, Pierre Monnin, Armelle Brun, and Davy Monticolo. New Strategies for Learning Knowledge Graph Embeddings: the Recommendation Case, In EKAW 2022 23rd International Conference on Knowledge Engineering and Knowledge Management, Bolzano, Italy, September 2022. URL /tttss:/hha...
ST/bal-0372288
Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Self- supervised learning on graphs: Deep insights and new direction. arXiv preprint arXiv:2006.1011 2020.
Wei Jin, Xiaorui Liu, Xiangyu Zhao, Yao Ma, Neil Shah, and Jiliang Tang. Automated self-supervised learning for graphs. arXiv preprint arXiv22006005770, 2021.
Mingxuan Ju, Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye, and Chuxu Zhang. Multi-task self-supervised graph neural networks enable stronger task generalization. arXiv preprint arXivv2210.020166 2022.
Zekarias T. Kefato and Sarunas Girdzijauskas. Self-supervised graph neural networks without explicit negative sampling. CoRR, abs/2103. 14958, 2021. URL htttt:t/////tttii hts//bb//110 14958.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Shuai Lin, Chen Liu, Pan Zhou, Zi- Yuan Hu, Shuojia Wang, Ruihui Zhao, Yefeng Zheng, Liang Lin, Eric Xing, and Xiaodan Liang. Prototypical graph contrastive learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI 15, pp. 2181-2187. AAAI Press, 2015. ISBN 026251 1290.
Gang Liu, Tong Zhao, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Graph rationalization with environment-based augmentations. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1069-1078, 2022.
Aditya Krishna Menon and Charles Elkan. Link prediction via matrix factorization. In Joint european conference machine learning and knowledge discovery in databases, pp. 437-452. Springer, 2011.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.0377 2018
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. DeepWalk. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, aug 2014. doi: 10.1145722233002222777 URL https: / http:/iii.oo////////// wtt.:/..ggg/....1122222.100/ 114572226622330 2623732.
S Yu Philip, Jiawei Han, and Christos Faloutsos. Link mining: Models, algorithuss, and applications. Springer, 2010.
Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. Neural collaborative filtering vs. matrix factorization revisited. In Fourteenth ACM conference on recommender systems, pp. 240.248, 2020.
Aravind Sankar, Yozen Liu, Jun Yu, and Neil Shah. Graph neural networks for friend ranking in large-scale social platforms. In Proceedings of the Web Conference 2021, pp. 2535-2546, 2021.
Prithviraj Sen, Galilco Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Chassi-RRd Collective classification in network data. Al magazine, 2008
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Gunnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: /8/ 1058688, 2018
12

---

Published as a conference paper at ICLR 2023
William Shiao and Evangelos E Papalexakis. Adversarially generating rank-constrained graphs. In 2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA), pp- 1-8. IEEE, 2021.
Xianfeng Tang, Yozen Liu, Neil Shah, Xiaolin Shi, Prasenjit Mitra, and Suhang Wang. Knowing your fate: Friendship, action and temporal explanations for user engagement prediction on social apps. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2269-22799 2020.
Xianfeng Tang, Yozen Liu, Xinran He, Suhang Wang, and Neil Shah. Friend story ranking with edge-contextual local graph convolutions. In Proceedings of the Fifieenth ACM International Conference Web Search and Data Mining, pp. 1007-1015, 2022.
Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L. Dyer, Rémi Munos, Petar Velickovic, and Michal Valko. Large-scale representation learning graphs via bootstrapping. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https:// net/fforuniid-0XX6PPP..
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs, 2021. URL
http//aaxvvv...gaa/// /ttp///.s////..e//// 06810.
Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax, 2018. URL https: hwtpg//bb//8800. 10341.
Shijie Wang, Guiling Sun, and Yangyang Li. Svd++ recommendation algorithm based on backtracking. Information, 11(7):369, 2020.
Yiwei Wang, Bryan Hooi, Yozen Liu, Tong Zhao, Zhichun Guo, and Neil Shah. Flashlight: Scalable link prediction with effective decoders. arXiv preprint arXiv:2209.10100 2022.
Zhitao Wang, Yong Zhou, Litao Hong, Yuanhang Zou, and Hanjing Su. Pairwise learning for neural link prediction. arXiv preprint vrXiv:21/2.029955 2021.
Zixin Wen and Yuanzhi Li. The mechanism of prediction head in non-contrastive self-supervised learning. arXiv preprint 22ti::2220000222.. 2022.
Zhen Yang, Ming Ding, Chang Zhou, Hongxia Yang. Jingren Zhou, and Jie Tang. Understanding negative sampling in graph representation learning, 2020. URL htttttts////axii wwwp:/iwwwrg////////ooo 09863.
Haoteng Yin, Muhan Zhang, Yanbang Wang, Jianguo Wang, and Pan Li. Algorithm and system co-design for efficient subgraph-based graph representation learning. Proceedings of the VLDB Endowment, 15(1) (00278--2999 2022.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, jul 20. doi: 10.1145/321981192229889 URL https://doi: weeedi..orgg// wrw....../............. 11551,2211222291811 3219890.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 974-983, 2018b.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphran: Generating realistic graphs with deep regr-sseee models. In International conference on machine learning. pp. 5708-5717. PMLR, 2018.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33: 5812-5823, 2020.
13

---

Published as a conference paper at ICLR 2023
Chuanming Yu, Xiaoli Zhao, Lu An, and Xia Lin. Similarity-base based link prediction in social networks: A path and node combined approach. Journal of Information Science, 43(5):683 695, 2017.
Ahmad Zareie and Rizos Sakellariou. Similarity-based link prediction in social networks using latent relationships between the users. Scientific Reports, 10(1):1-11, 2020.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction, 2021. URL https: http://wwwwvv..aaaaaa///. atrgg/ab///133... 00202200100000..0..
Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip Yu. From canonical correlation analysis to self-supervised graph neural networks. Advances Neural Information Processing Systems, 34:76-89, 00776689,700 2021.
Muhan Zhang and Yixin Chen. Link prediction based graph neural networks. In Advances in Neural Information Processing Systems, pp. 5165-5175, 2018.
Muhan Zhang and Yixin Chen. Inductive matrix completion based on graph neural networks. arXiv preprint arXiv: 1904. 12058, 2019.
Ziwei Zhang, Peng Cui, and Wenwu Zhu Deep learning on graphs: A survey, IEEE Transactions on Knowledge and Data Engineering, 2020.
Tong Zhao, Wei Jin, Yozen Liu, Yingheng Wang, Gang Liu, Stephan Günneman, Neil Shah, and Meng Jiang. Graph data augmentation for graph machine learning; A survey. arXiv preprint 0888: 2022a.
Tong Zhao, Gang Liu, Daheng Wang, Wenhao Yu, and Meng Jiang. Learning from counterfactual links for link prediction. In International Conference on Machine Learning, pp. 26911-2699 PMLR, 2022b.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint 2006v:2006.041 2020.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DATASET STATISTICS
Table 4: Statistics for the datasets used in our work.
Dataset
Nodes
Edges
Features
Cora
2,708
5,278
1,433
Citeseer
3,327
4,552
3,703
Coauthor-Cs
18,333
163,788
6,805
Coauthor-Physics
34,493
495,924
8,415
Amazon-Computers
13,752
491,722
767
Amazon-Photos
7,650
238,162
745
A.2 MACHINE DETAILS
We run all of experiments either NVIDIA P100 VI00 GPUs. We use machines with 12 virtual CPU cores and 24 GB RAM for the majority of our experiments. We exclusively use V100s for our timing experiments. We ran our experiments on Google Cloud Platform.
A.3 TRANSDUCTIVE SETTING DETAILS
We use an 85/5/10 split for training/validdttionkesinngg data follooolloiing Zhang & Chen (2018); Cai al. (2020).
A.4 INDUCTIVE SETTING DETAILS
The inductive setting represents more realistic setting than the transductive setting. For example, consider social network upon which model is trained some time 1 but is used for inference (for a GNN, this refers to the message-passingg step) at time f2, where new users and friendships have been added the network the interim. Then, the goal of a model run time 12 would be predict any new links at new network state to (although we assume there are no new nodes introduced at that step since we cannot compute the embedding of nodes without performing inference on them first). To simulate this setting, we first perform the following steps:
We withhold a portion the edges (and the same number of disconnected node pairs) to use testing-only edges.
2. We partition the graph into two sets of nodes: "observed" nodes (that we see during training) and "unobserved nodes" (that can only be seen during testing).
3. We mask out some edges to use as testing-only edges.
4. We mask some edges use as inference-only edges.
5. We mask out some edges to use as validation-on only edges.
6. We mask out some edges to use as training-only edges.
As the test edges are sampled before the node split, there will be three kinds of them after the splitting. Specifically: edges within observed nodes, edges between observed nodes and unobserved nodes, and edges within unobserved nodes. For ease of data preparation, we use the same percentages for the test edge splitting, unobserved node splitting, and validation edge splitting. Specifically, we mask out 30% of the edges (at each of the above stages) the small datasets (Cora and Citeseer), and 10% on all the other datasets. We use a 30% split on the small datasets to ensure that we have a sufficient number of edges for testing and validation purposes.
A.5 EXPERIMENTAL SETUP
To ensure that we fairly evaluate each model, we run a Bayesian hyperparameter sweep for 25 runs across each model-dataset combination with the target metric being the validation Hits 50. Each run
15

---

Published as a conference paper at ICLR 2023
Table 6: Area under the ROC curve for the methods in the transductive setting.
End-To-End
Contrastive
Non-Contrastive
Dataset
E2E-GCN
ML-GCN
GRACE
CCA-SSG
GBT
BGRL
T-BGRL
Cora
0.911
2.893
0.883
0.647
0.736
0.911
0.91020ces
Citeseer
0,922++00s
0.89 #0006
0,863+0022
0.66] +0050
0.755+a.15
0.934
0,953+n00
Coauthor-Cs
0.964110ms
0.966_nncn
0.961 1000
0.758gaom
0.894
0.959
gome
0.956gncom
Sauuthor-Physiie
0.978
0.986=nc00
OOM
0.82 =005)
0.834
0,961 coc
0.963
Amazon Anazon-Computtere
0,985 Loor
D. 0.983 Luoos
0.951 Toor:
0.907 Lovazs
0,94611om
0,969 Linee:
0.9761mco
Anazon- Anazon-Photos 0.989 100
0.983
0.981 1o:m
0.939
0.9561omx
0.980
0.982_1n00
is the result of the mean averaged over 5 runs (retraining both the encoder and decoder). We used the Weights and Biases (Biewald, 2020) Bayesian optimizer for our experiments. We provide sample configuration file to reproduce our sweeps, as well as the exact parameters used for the top T-BGRL shown in tables.
We used the reference GRACE implementation and BGRL implementation but modified them for link prediction instead of node classification. We based our E2E-GCH off of the OGB (Hu al., 2020) implementation. We re-implemented CCA-SSG and GBT. The code for all of our implementations and modifications be found in the link in our paper above.
A.6 FULL RESULTS
Table 5 shows the results of all the methods (including T-BGRL) on transductive setting.
Table 5: Full transductive performance table (combination of Tables and 3).
End-To-End
Contrastive
Non-Contrastive
Dataset
E2E-GCN
ML-GCN
GRACE
CCA-SSG
GBT
BGRL.
T-BGRL
Cora
0,816ioms
i.815
0.686
0.348:toum
0.460
0.792
0.773
Citeseer
0,822=0017
0.771 =002
0.707=0m
0.249
0,472 106
0.858=0020
0.868
Seazon-Photor
0,642++002
0.430=aoz2
0,486-oom
0.369+cm1
0.434=ams
0.562+0a13
0.517+00us
ComputtCop
0.4262:0m
0.320=a0se
0.240
0.201 aom
0.258
0.346go01
0.315
Coauthor-Cs
0.762:20010
0.787=5011
0,456=00m
0.229
0.298
0.515
0.555
PhauherrPPyesice
0.798_ooms
0.810
0OM
0.157.omm
0.187_nmm
0.476
0.471 Inom
A.7 CORRUPTIONS
In this work, we experiment with the following corruptions:
1. RANDOMFFAARR.NDOMEDDD
Randomly generate an adjacency matrix À and X with the same sizes as A and X, respectively. Note that A and A also have the same number of non-zero entries, i.e., the same number of edges.
2.
Randomly shuffle the rows of X, and generate a random À with the same size as A. Note that À and A also have the same number non-zero entries, i.e., the same number of edges.
3.
Mask out a large percentage (we chose 95%) of the entries
in X and A.
Of these corruptions, we find that SHUFFLEFEATRANDOMEEDOR works the best across our experi- ments.
A.8 AUC-ROC RESULTS
Here include the under the ROC curve for each of the different models under both the inductive and transductive settings. Note that we perform early stopping on the validation 50 when training the link prediction model, not the validation AUC-ROC.
16

---

Published as a conference paper at ICLR 2023
Table AUC-ROC of various methods in the inductive setting. See Section 3.1.2 for an explanation of our inductive setting.
End-To-Ena
Contrastive
Non-Contrastive
Dataset
E2E-GCN
ML-GCN
GRACE
GBT
CCA-SSG
BGRL
T-BGRL
Overall
Cora
0.788
0.842:200s
0.858
0.704
0,595
0.814
0.920
Citeseer
0.810=ams
0.873+0004
0.886
0.691
0.621
0. 0.891 -00000
0.954
Coauthor-Ca
0.881 110000
0.956.20m
0.944g0001
0.875
0.831 au0oo
0.968_0900
0.958.400
Coauthor-Physics
0.957=0004
0.976
OOM
0.614 0.818=0092
0.974=00001
0.976:2000
Amazon- Conzon-Computera
0.97441008
0,98 0:0000
0,972.0002
0.919
0.910
0. 0.980_0.00
0.982
Anazon-Photos
0.976.m000
0.982
1.000
0.962
0.885 gm087
0. 0.984
0.981 100000
Performance Observed- Node Edges
Cora
0.827=amm
0.834=a01
0.883+3610
0.714
0.584
0.800=0.:05
0.929:0006
Citeseer
0.79220004
0.84020013
20002
0,705 29001
0.635
0.875
0.956.1005
Coauthor-Ca
0.886:4467
=
0.947
0.874
0.828
0.967=00001
0.955 2.0002
Soauthor-Physicee
0.9599000
0.9762000
OOM
0.819
0.615
0.974_0000
0.975
mmazon-Computers
0.974gmmo
0.981 nm
0.971
0.918
0.910amoo
0.979_00000
0.981
Amazon Photos
0.9762000
0.98 1:000 001
0.977
0.96226001
0.000000000
0.98 10000
Performance on Observed-Unobeerved Node Edges
Cora
2.74
0.844
0.840
0.696_00020
0.6022.004
0.818=090t
0.912
Citeseer
0.841=a009
0,901 =0000
0.877=0002
0.687=6016
0.610
0. 0.904=0.006
0.955
Coauthor-Ca
0.877.004
0.9642200
0.940
0.876200m6
0.836gam0
0.969
0.964
Coauthor-Physicom
0.953amm0
0.975
OOM
0.817
0.612anme
0.975
0.976
Amazon Computerr
0.974=.00
0 1:0000
0.973
0.921 210022
0.909
0. 0.980=00000
0.982..000
Amazon-Phoon Phottss
0.977
g.98
0.977
0.963
0.884
0. 0.986_0000
0.981 10000
Performance on Unobscrved-Unobserved Node Edges
Cora
2000
0.8792nms
0.81020019
0.693
0.626gn040
0.866_0.01
0.911 100m
Citeseer
0.8522000
0.917.2m
0.827 1:000
0.637
0.599gmm
0.916_2001
0.941:100
Coauthor-Cs
0.850
0.964
0.928
0.877=0004
0.839
0.967
0.966
Coauthor-Phyee Physice
0.9999.0ms
0.978
0OM
0.8 0.818:10.99
0.61 3.0ss
0.978=0.001
0.981 10000
Amazon- Mazzon-Computers
0,970gemm
gmom
0.9691001
0.914 0.914
0.899
-.977=0000
0.979 2000
Amazon
Daszon-Photoss Photos
0.978:
0.982:0002
0.977:2002
0.965
0.886
0.986-00001
0.982::000
A.9
WHY DOES BGRL NOT COLLAPSE?
The loss function for BGRL (see Equation (2)) is when h or ž; 0. While theoretically possible, this is clearly undesirable behavior since this does not result in useful embeddings. We refer to this case as the model collapsing. It is not fully understood why non-contrastive models do not collapse, but there have been several reasons proposed in the image domain with both theoretical and empirical grounding. Chen & He (2021) showed that the SimSiam architecture requires both the predictor and the stop gradient. This has also been shown to be true for BGRL. Tian et al. (2021) claim that the eigenspace of predictor weights will align with the correlation matrix of the online network under the assumption of one-layer linear encoder and one-layer linear predictor. Wen & Li (2022) looked at the case of a two-layer non-linear encoder with output normalization and found that the predictor is often only useful during the learning process, and often converges to the identity function. We did not observe this behavior on BGRL-the predictor is usually significantly different from that of the identity function.
A. HOW DOES BGRL PULL REPRESENTATIONS CLOSER TOGETHER?
Here we clarify the intuition behind BGRL pulling similar points together. To simplify this analysis, assume that the predictor is the identity function, which Wen & Li (2022) found is true in the image representation learning setting. Although we have not observed this the graph setting, this assumption greatly simplifies analysis and we argue it is sufficient for understanding why BGRL works.
Suppose have three nodes: anchor node u, a neighbor v, and neighbo That is, have (u, v) E E, (u, w) & E, and (v, w) & E. Let u, v, w be the embeddings for u, v w. respectively (e.g. ENC(u))
17

---

Published as a conference paper at ICLR 2023
Assuming homophily between the nodes, have For of visualization, let us project the points in 2D space. Then, we have the following:
20
u
v
We then apply the two augmentations sake of simplicity, let us assume that we perform edge dropping and feature dropping with the same probability p (in practice, they may be different from each other). We represent the space of possible values for ūj and 22 as circle with radius r centered at u, where is controlled by p.
producing ú₁ AUG1 (u) and ü2 AUG2(u). For the
1
AUG
u
P
AuG
i2
The BGRL loss is stated in Equation (2) above, but rewrite relative our anchor u and with our assumption about the predictor:
U2
C.
(6)
[
Minimizing this loss pushes ūi and ú closer. Let us denote the encoder after one round of optimization as ENc'. Then:
[|ENC (NC((AU(( < E | [|EC(AAU(()) ENC(AUG(u)III (7)
Note that v in this example lies within the space of possible augmentations that is, A, where A the set of all possible values of AuG(u). This means, as we repeat this process, we implicitly push and closer together- leading to distributions like those shown in Figure 1.
18

---

Published as a conference paper at ICLR 2023
A.11
ADDITIONAL PLOTS
Coauthor Negative Link Embedding Similarity
Coauthor-Co Positive Link Embedding Similariy
F
BQRL
83RL
Ronoort
107
0.0
Simiarity
Simlintty
Figure 5: These plots show similarities between node embeddings on Coauthor-Cs. Left: distribu- tion of similarity to non-neighbors for T-BGRL and BGRL (closer to 0 is better). Right: distribution of similarity to neighbors for T-BGRL and BGRL (closer to is better). Note that the y-axis is on logarithmic scale. T-BGRL clearly does better job of ensuring that negative link representations pushed far apart from those of positive links.
Cora Negative Link Node Embodding Siilarty
Cora Positive Link Node Emboddingg Similarit
Trplet
er
Poccont
107
107
0.2
11.6
0.5
11.2 0:4
0.6
Simiianty
Siilintty
Figure 6: These plots show similarities between node embeddings Cora. Left: distribution of similarity to non-neighbors for T-BGRL and BGRL (closer to is better). Right: distribution of similarity neighbors for T-BGRL and BGRL (closer to 1 is better). Note that the y-axis is logarithmic scale. T-BGRL clearly does a better job ensuring that negative link representations are pushed far apart from those of positive links, but does not do well at differentiating between positive links.
19


Reviews:
[{"review_id": "wJPs1FKv6G", "reviewer": "Reviewer_N6ym", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first compares the performance of contrastive and non-contrastive methods on the graph link prediction task. Second, a novel model i.e. Triplet-BGRL (T-BGRL) is proposed based on Bootstrapped Graph Latents (BGRL). The proposed T-BGRL is categorised as a non-contrastive method and it relies on an efficient corruption method to generate cheap negative samples. Extensive experiments demonstrate that T-BGRL can achieve competitive performance for both the transductive and inductive settings with a good efficiency. ", "summary_of_the_review": "This paper needs to be polished, and some experimental setup needs to be justified to support the proposed claims. ", "strength_and_weaknesses": "### Strength:\n\n1. The discussion of contrastive and non-contrastive methods for the link prediction task is comprehensive. \n\n### Weakness:\n\n1. In Section 3.1, it’s unclear why the experiments only focus on the performance of the encoder and adapt the same decoder for all evaluated models. Moreover, it’s unclear why the encoder and decoder are not trained together. If this is following some existing works, please cite them and discuss the reason(s).\n2. ML-GCN is only loosely defined but used as an important baseline. Is ML-GCN just a GCN model using the margin-loss? Also, what is ‘the supervised GCN’ (Section 3.1.1)? Is it just an end-to-end GCN?\n3. Figure 1 does not have an x-axis. \n4. ‘collapse’ is used without only once in Section 3.1.1 without definition and further discussion or claim. \n5. Why ML-GCN and BGRL are both claimed as ‘our’ best-performing contrastive/non-contrastive models?\n6. Is the SHUFFLEFEATRANDEDGE the same as the SHUFFLEFEATRANDOMEDGE method described in A.6? It’s unclear why a randomly generated adjacency matrix can help the effectiveness of BGRL. In fact, if the proposed method only generates a random matrix, then there is no corruption. \n7. It’s unclear why some arguments/conclusions are stated in the caption of the figures e.g. figures 1&3. \n8. It’s unclear why the early-stop is only used by ML-GCN.\n9. This paper is poorly organised. Usually, there is no connection between two sections. Furthermore, many sections (e.g. the last subsection of Section 3.1.1) end without a solid conclusion or constructive advice. \n\n### Typos:\n1. On page 3, ‘none evaluate or target link prediction tasks.’  ’none evaluates or targets the link prediction task.’\n2. On page 5, ‘which relative to an anchor node u is expressed as follows’. This sentence needs to be rewritten.\n3. On page 5, ‘wen & Li (2022) looks at xxx’ ‘wen & Li (2022) looked at xxx’\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "pKvJsG5SR-", "reviewer": "Reviewer_myZB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on investigating the performance of existing \nnon-contrastive methods for link prediction in both transductive and inductive \nsettings. In their experiments, they find that BGRL generally performs well in transductive settings, but poorly in the more realistic inductive settings, which motivates them to develop an extension of BGRL named T-BGRL to alleviate its overfitting phenomenon.", "summary_of_the_review": "This paper is more like a technical report of these non-contrastive learning methods to me, rather than a paper focusing on a new developed method.", "strength_and_weaknesses": "Strength\n\n1.The writing of this paper is good and easy to follow.\n2.The experimental results on non-contrastive learning for link prediction is convinced and comprehensive.\n\nWeakness\n1.The novelty and theoretical analysis of the developed method is not enough", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "od4Td8RfuK7", "reviewer": "Reviewer_Qz7f", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is trying to tackle the area of graph self-supervised learning, which aims to derive useful node representations without labeled data. The paper proposes T-BGRL, a novel non-constrastive model to solve the link prediction problem. ", "summary_of_the_review": "Overall, the paper needs some improvement on explaining the significance of the work for it to be accepted.", "strength_and_weaknesses": "Strength:\n1. The problem the paper is trying to solve is important research problem.\n2. The method proposed is easy to implement and the experiments show that the performance did improve on most datasets.\n\nWeakness:\n1. I do not quite understand why we want to apply non-contrastive learning methods for link prediction. What is wrong with those contrastive learning methods?\n2. Moreover, I do not understand why the proposed method falls in the category of non-contrastive learning methods. So it is very hard for me to judge the significance of the method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2FA7n_pIgGl", "reviewer": "Reviewer_tANx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper follows BGRL, and notices that it generalizes poorly due to a lack of negative examples.\nThey propose a non-contrastive framework for link prediction named T-BGRL that uses cheap “negative” samples to improve generalization. \nExperiments show that T-BGRL improves BGRL’s inductive performance in 5/6 datasets and is more efficient than contrastive methods.", "summary_of_the_review": "The analysis of BGRL makes sense and the proposed T-BGRL performs well in the inductive setting.\nI am open to raising my score if the authors can conduct experiments on ogbl-collab or ogbl-ppa.", "strength_and_weaknesses": "Strength:\n\n[1] This paper finds that BGRL tends to overfit the training graph, and thus can only perform well in the transductive setting. They introduce cheap “negative” samples to improve generalization on the indictive setting.\n\nWeaknesses:\n\n[1] They said it's the first work to explore link prediction with non-contrastive SSL methods. However, BGRL is a non-contrastive model and can be applied to link prediction tasks easily.\n\n[2] These papers didn't compare with subgraph-based methods due to they are slow during inference. however, these subgraph-based methods perform well on large datasets. How about the performance of T-RGRL on large datasets (i.e. ogbl-ppa and ogbl-collab)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster