Graph Barlow Twins: A self-supervised representation learning framework for graphs
Authors: Piotr Bielak
Tomasz Jan Kajdanowicz
Nitesh Chawla
Venue: ICLR 2022 Submitted (2022)
Keywords: graph representation learning
self-supervised learning

Abstract: The self-supervised learning (SSL) paradigm is an essential exploration area, which tries to eliminate the need for expensive data labeling. Despite the great success of SSL methods in computer vision and natural language processing, most of them employ contrastive learning objectives that require negative samples, which are hard to define. This becomes even more challenging in the case of graphs and is a bottleneck for achieving robust representations. To overcome such limitations, we propose a framework for self-supervised graph representation learning - Graph Barlow Twins, which utilizes a cross-correlation-based loss function instead of negative samples. Moreover, it does not rely on non-symmetric neural network architectures - in contrast to state-of-the-art self-supervised graph representation learning method BGRL. We show that our method achieves as competitive results as the best self-supervised methods and fully supervised ones while requiring fewer hyperparameters and substantially shorter computation time (ca. 30 times faster than BGRL).

Under review as a conference paper at ICLR 2022
GRAPH BARLOW TWINS: A SELF-SUPERVISED REPRE- SENTATION LEARNING FRAMEWORK FOR GRAPHS
Anonymous authors Paper under double-blind review
ABSTRACT
The self-supervised learning (SSL) paradigm is an essential exploration area, which tries eliminate the need for expensive data labeling. Despite the great success of SSL methods in computer vision and natural language processing, most of them employ contrastive learning objectives that require negative samples, which are hard to define. This becomes even more challenging in the case of graphs and is a bottleneck for achieving robust representations. To overcome such limitations, propose a framework for self-supervised graph representation learning Graph Barlow Twins, which utilizes a cross-correlation-tee negative samples. Moreover, it does not rely non-symmetric neural network architectures - in contrast to state-of-the-art self-supervised graph representation learning method BGRL. We show that our method achieves competitive results as the best self-supervised methods and fully supervised ones while requiring fewer hyperparameters and substantially shorter computation time (ca. 30 times faster than BGRL).
loss function instead
INTRODUCTION
Graph representation learning has been intensively studied for the last few years, having proposed various architectures and layers, like GCN (Kipf & Welling, 2017), GAT (Veličković al., 2018), GraphSAGE (Hamilton et al., 2017) etc. A substantial part of these methods was introduced in the supervipe learning paradigm, which requires the existence of expensive labeled data (e.g. node labels or whole graph labels). To overcome this, the research community has been exploring unsupervised learning methods for graphs. This resulted in variety of different approaches including: shallow ones (DeepWalk (Perozzi et al., 2014), Node2vec (Grover & Leskovec, 2016), LINE (Tang et al., 2015)) that ignore the feature attribute richness, focusing only on the structural graph information; and graph neural network methods (DGI (Veličković et al., 2019), GAE, VGAE (Kipf & Welling, 2016)) that build representations upon node or graph features, achieving state-of- the-art performance in those days.
Recently self-supervised paradigm is the most emerging branch of unsupervised graph representation learning and gathers current interest and strenuous research effort towards better results. The most prominent methods were developed around the contrastive learning approach, such as GCA (Zhu et al., 2020b), GraphCL (You al., 2020), GRACE (Zhu et al., 2020a) DGI (Veličković al., 2019). Although contrastive methods are popular in many machine learning areas, including computer vision and natural language processing, their fundamental limitation is the need for negative samples. Consequently, the sampling procedure for negative examples highly affects the overall quality of the embeddings. In terms images or texts, the definition of negative samples might seem not that problematic, but in the case of graphs there is no clear intuition. For instance, what is the negative counterpart for particular node in the graph, should it be node that is not direct neighbor, or a node that is in different graph component? There are multiple options available, but the right choice strictly dependent on the downstream task.
Researchers have already tackled the problem of building so-called negative-sample-fre methods. In research being conducted in computer vision they obtained successful results with methods such as BYOL (Grill et al., 2020), SimSiam (Chen & He, 2020) or Barlow Twins (Zbontar et al., 2021). These models utilize siamese network architectures with various techniques, like gradient stopping, asymmetry batch and layer normalizations, to prevent collapsing trivial solutions. Based on

---

Under review as a conference paper at ICLR 2022
BYOL, Thakoor et al. (2021) proposed the Bootstrapped Representation Learning on Graphs (BGRL) framework. It utilizes two graph encoders: an online and a target one. The former one passes the embedding vectors to predictor network, which tries predict the embeddings from the target encoder. The loss is measured as the cosine similarity and the gradient is backpropagated only through the online network (gradient stopping mechanism). The target encoder is updated using an exponential moving average of the online encoder weights. Such setup has been shown to produce graph representation vectors that achieve state-of-the-an performance in node classification using various benchmark datasets. Notwithstanding, assuming asymmetry between the network twins (such as the predictor network, gradient stopping, and moving average on the weight updates) the method is conceptually complex.
Employing a symmetric network architecture would seem more intuitive and reasonable, hence in this paper, we propose Graph Barlow Twins (G-BT), self-supervised graph representation learning framework, which computes the embeddings cross-correlation matrix of two distorted views of single graph. The approach was firstly introduced in image representation learning as the Barlow Twins model (Zbontar et al., 2021) but was not able to handle graphs. The utilized network architecture is fully symmetric and does not need any special techniques to build non trivial embedding vectors. The distorted graph views are passed through the same encoder which is trained using the backpropagated gradients (in a symmetrical manner).
Our main contributions can be summarized as follows: (I) We propose a self-supervised graph representation learning framework Graph Barlow Twins. It is built upon the recently proposed Barlow Twins loss, which utilizes the embedding cross-correlation matrix of two distorted views of a graph to optimize the representation vectors. Our framework neither requires using negative samples (opposed to most other self-supervised approaches) nor it introduces any kind of asymmetry in the network architecture (like sfate-of-the-ar BGRL). Moreover, our architecture is converges substantially faster than all other state-of- sfute-of-the-att methods. (II) We evaluate our framework in node classification tasks: (1) for 5 smaller benchmark datasets in a transductive setting, (2) using the ogb-arxiv dataset from the Open Graph Benchmark (also in the transductive setting), (3) for multiple graphs in the inductive setting using the PPI (Protein-Proter Interaction) dataset, and finally (4) for the large-scale graph dataset ogb-products in the inductive setting. We use both GCN-based encoders as well as a GAT-based one. We observe that our method achieves analogous results compared to ktte-of-tte of-the-art methods. (III) We ensure reproducibility by making the code of both models as well as experimental pipeline available (currently attached in the supplementary materials).
2 RELATED WORKS
Self-supervised learning The idea of self-supervised learning (SSL) has long history. Introduced in the early work of Schnidhuber (Schmidhuber, 1990) has more than 30 years of exploration and research now. Recently self-supervised learning was again rediscovered and found a broad interest, especially in computer vision and natural language processing. One of the most prominent SSL methods for image representation learning, Bootstrap Your Own Latent, BYOL (Grill et al., 2020), performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. It relies on two neural networks that interact and learn from each other. From an augmented view of an image, it trains the first one to predict the target network representation of the same image under different view. At the same time, it updates the second network with slow-moving average of the first network. Another approach to image representation SSL implements simple siamese networks, namely SimSiam (Chen & He, 2020). It achieves comparative results while not demanding negative samples, large batches, nor momentum encoders. Authors emphasize collapsing solutions for the loss and structure but show how a poopggradeett operation plays an essential role in preventing it. Recent method, Barlow Twins (Zbontar et al., 2021), advances the SSL field with a new objective function that naturally avoids collapses by measuring the cross-correlation matrix between the outputs of two twin, identical networks fed with distorted versions of sample, and makes it as close to the identity matrix as possible. Representations of distorted versions of samples are then expected to be similar, reducing the redundancy between them. What differentiates the method is that it does not require large batches or asymmetry between the network twins. outperforms previous methods on ImageNet for semi-supervised classification.
2

---

Under review as a conference paper at ICLR 2022
Graph representation learning Learning the representation also spreads to other domains. The graph embedding problem has also attracted much attention from the research community worldwide in recent years, Plenty of methods have been developed, each focused on different aspect of network embeddings, such as proximity, structure, attributes, learning paradigm or scalability. There exist plenty of shallow methods, among others DeepWalk (Perozzi et al., 2014), Node2vec (Grover & Leskovec, 2016) or LINE (Tang et al., 2015), that use simple notion of graph coding through random walks or on encoder-decoder objectives that optimize first and second-order node similarity. More complex graph neural networks, such as GCN(Kipf & Welling, 2017) or GraphSAGE (Hamilton et al., 2017) implements the basic encoder algorithm with various neighborhood aggregation. Following the extension, graph attention network GAT (Veličković et al., 2018) 8) leverages masked self-attentional layers to address the shortcomings of graph convolutions and their troublesome approximations.
Self-supervised graph representation learning Inspired by the success of contrastive methods in vision and NLP, the procedures were also adapted to graphs. Early DGI (Veličković et al., 2019) employs GNN to learn node embeddings and obtains the graph embedding via readout function and maximizes the mutual information between node embeddings and the graph embedding by discriminating nodes in the original graph from nodes in corrupted graph. GCA (Zhu et al., 2020b) studied various augmentation procedures. GRACE (Zhu et al., 2020a) creates two augmented versions of graph, pulls together the representation of the same node in both graphs, and pushes apart representations of every other node. Recent GraphCI. (You et al., 2020) method another example of representative approach using contrastive learning. All the previous methods use negative sampling approaches for the embedding optimization, yet such setting has a high complexity. To overcome this, BGRL (Thakoor et al., 2021) proposed use an approach that does not rely on negative samples. It uses two kinds of encoder networks (online and target), introducing a non- intuitive asymmetric pipeline architecture, but provides ofate-of-the-ar SSL results. Moreover, relies on several techniques to prevent trivial solutions (gradient stopping, momentum encoder). A concurrent approach to BGRL DGB (Che et al., 2020).
PROPOSED FRAMEWORK
Motivated by the emerging self-supervised learning paradigm and its recent applications in graph representation learning (BGRI. (Thakoor et al., 2021)), we propose Graph Barlow Twins a framework that builds node embeddings using symmetric network architecture and empirical cross-correlation based loss function. The overall pipeline of our framework shown in Figure 1.
The consecutive processing steps can be described follows:
Original graph
Augmented graph views (edge dropping, node feature masking)
GNN encoder
Embeddings Z E R [V|dd
Empirical
cross-
(())
(X(1) A
correlation matrix
(
C E .ddd
Apr.FA9 (g)
g = AX
Same weights
2(2)
(2)
(2))
Aprips (9)
(
Barlow Twins loss LB
W
fo
Figure 1: Overview of our proposed Graph Barlow Twins framework. We transform an input graph G using an augmentation function and obtain two views: g(1) and ((2). We pass both of them through the same GNN encoder fo to compute two embedding matrices z(1, Zt2), We build loss function such that the embeddings" empirical cross-correlation matrix C is optimized into the identity matrix.
3

---

Under review as a conference paper at ICLR 2022
Graph data We represent graph 9 with nodes V and edges & as the tuple: (X, A), where X E R is the node feature matrix and k is the feature dimensionality; A E {0, 1| is the adjacency matrix, such that A, = 1 iff (i,j) E E. In the general case, a graph could also have associated edge features graph level features, but for simplicity omit those here. Nevertheless, these could also be used in our framework, as long as the encoder can make use of such features.
Generating graph views via augmentation Following other works (Thakoor et al., 2021; Zhu et al., 2020b; You al., 2020; Zhu et al., 2020a), we select two kinds of augmentations edge dropping and node feature masking and generate two views of the input graph g()) and g In the edge dropping case, we remove edges according to a generated mask of size |E| (number of edges in the graph) with elements sampled from the Bernoulli distribution B pa). When it comes to masking node features, we employ a similar scheme and generate a mask of size also sampled from the Bernoulli distribution B1 px)- Note that mask node features at the scale of the whole graph, i.e. the same features are masked for each node. Other works apply different augmentation parameters PX.PA for each generated view, but as our framework is fully symmetrical, we postulate that it is enough to use the same parameters to generate both augmentations (see Section 4.4).
Encoder network The main component of the proposed framework is the encoder network fu 9 + R Vixd It takes an augmented graph as the input and computes (in our case) a d- dimensional representation vector for each node in the graph. Note that we do not specify any particular encoder network and one may use even encoders that construct embeddings for edges or whole graphs. In our experiments, we will show the application GCN (Kipf & Welling, 2017) and GAT (Veličković et al., 2018) based encoder networks. Both augmented graph views g(1), g(2) passed through the same encoder, resulting in two embedding matrices ( and Z respectively. The original Barlow Twins method specified also a projector network (implemented as an MLP) to reduce high embedding dimensionality (of the ResNet encoder). Our approach eliminates that step as it uses GNNs with low dimensional embeddings.
Loss function In our work, we propose to use a negative-sumple-fres loss function to train the encoder network. We first normalize the embedding matrices Z(1) and (2) along the batch dimension (a mean of zero and standard deviation equal to one), and then we compute the empirical cross- correlation matrix C e Rdxd,
(1) 20s 2..
Cjj V D2 EZ
(1)
where b are the batch indexes and i are the indexes of embeddings. Such setting was originally proposed under the name Barlow Twins. Neuroscientist H. Barlow's 's redundancy-reuuctio redductio principle has motivated many methods both in supervised and unsupervised learning (Deco & Parra, 1997; Schmidhuber et al., 1996; Ballé et al., 2017). Recently, Zbontar et al. (2021) has employed this principle to build a self-supervised image representation learning algorithm (we bring this idea to the domain of graph- structurd data).
The cross-correlation matrix is optimized by the Barlow Twins loss function LBT (see Equation 2) to be equal to the identity matrix. The loss is composed of two parts: (I) the invariance term and (II) the redundancy reduction term. The first one forces the on diagonal elements C2 to be equal to one, hence making the embeddings invariant the applied augmentations. The second term optimizes the off-diagonal elements Cij to be equal to zero this results in decorrelated components of the embedding vectors.
LBT = - Ca)2 + + ΣΣ cz² J+L
(2)
The À > 0 parameter defines the trade-off between the invariance and redundancy reduction terms when optimizing the overall loss function. In Tsai et al. (2021), the authors proposed use À = 1 which we employ in our experimental setting. Otherwise, one can perform a simple grid search to find the best À value in particular experiment.
4

---

Under review as a conference paper at ICLR 2022
Please note, that in such setting the gradient is symmetrically backpropagated through the encoder network. We do not rely on any special techniques, like momentum encoders, gradient stopping, or predictor networks. In preliminary experiments, also investigated the filleet-Schhid Indepen- dence Criterion (due to its relation to the Barlow Twins objective (Tsai et al., 2021)), but we did not observe any performance gain.
EXPERIMENTS
We evaluate the performance of our model using variety of popular benchmark datasets, including smaller ones, such as WikiCS, Amazon-Phote or Coauthor-CS, as well as larger ones, such as ogb-arxiv, ogb-products, provided by the Open Graph Benchmark (Hu et al., 2020). We describe In this section, we will provide an overview experimental scenario details, and the discussion of the results. Overall, we setup, as the state-of-the-and self-superrised graph representation learning method BGRL (Thakoor al., 2021), we can perform fair comparison this method. To track experiments and provide a simple way for reproduction, we employ the Data Version Control tool (DVC; Kuprieiev al. (2021)). We perform all experiments on TITAN RTX GPU with 24GB RAM.
all datasets along with their statistics in Appendix
similar experimental
4.1 EVALUATION PROTOCOL
Self-supervised framework training We start the evaluation procedure by training the encoder networks using our proposed Graph Barlow Twins framework. In all scenarios, we use the W optimizer (Gugger & Howard, 2018) with weight decay equal to 10-5, The learning rate is updated using a cosine annealing strategy with linear warmup period. Our framework uses single set of augmentation parameters for both graph views. Therefore we do not use reported values of these parameters from other publications that use two different sets. Instead we perform a grid search over the range: PA- {0, 0.1...... 0.5} for 500 epochs with a warmup time of 50 epochs. We implement our experiment using the PyTorch Geometric (Fey & Lenssen, 2019) library. All datasets are available in this library as well. The details about the used augmentation hyperparameters, node embedding dimensions and the encoder architecture are given in Appendix B.
Node embedding evaluation We follow the linear evaluation protocol proposed by Veličković et al. (2019). We use the trained encoder network, freeze the weights and extract the node embeddings for the original graph data without any augmentations. Next, regression classifier from the Scikit learn (Pedregosa et al., 2011) 1) library. We also perform grid search over the regularization strength using following values: {2-10, 2 20,2-12 29, "...,,,," 20,2.. In the case
train L2 Iz-regularized logistic
of the larger ogb-arxiv, ogb-products and the PPI datasets, the Scikit implementation takes too long converge. Hence,
implement the logistic regression classifier in Py and optimize it for 1000 steps using the AdamW optimizer. We check various weight decay values using smaller grid search: (2 10 21.2- 2- .,...., 28, 22"). We use these classifiers to compute the classification accuracy and report mean and standard deviations over 20 model initializations and splits, except for the ogb-arxiv. agb-products and PPI datasets, where we there is only one data split provided we only re-initialize the model weights 20 times (5 times for ogb-products due to long training time).
4.2 TRANSDUCTIVE EXPERIMENTS
We evaluate and compare framework to other graph representation learning approaches on 6 real-world datasets using the transductive setting. The whole graph including all the node features observed during the encoder training. The node labels (classes) are hidden at that moment (unsupervised learning). Next, we use the frozen embeddings and labels of training nodes to train the logistic regression classifier.
4.2. SMALL AND MEDIUM SIZES BENCHMARK DATASETS
Our first experiment uses small and medium sized popular benchmark datasets, namely: WikiCS, Amazon Computers, Amazon Photos, Coauthor CS and Coauthor Physics.
5

---

Under review as a conference paper at ICLR 2022
Encoder model Similarly to (Thakoor et al., 2021), we build our encoder fo 2-layer GCN Kipf & Welling (2017) network. After the first GCN layer we apply a batch normalization layer (with momentum equal to 0.01) and the PReLU activation function. Accordingly to the original Barlow Twins method, we do not apply any normalization or activation to the final layer. A graph convolution layer (GCN) uses the diagonal degree matrix D to apply symmetrical normalization to the adjacency matrix with added self-loops À A + I. Hence the propagation rule of such layer is defined as follows:
GCN A)
D =
(3)
Note that we do not include the activation (.) in this definition, first apply the batch normal- ization and then the activation function.
Results and discussion We train our framework for total of 1000 epochs, but we observe that our model converges earlier at about 500-900 epochs (depending on the dataset; see Appendix B). This significantly faster than the state-of-the- of--te-rtttttttt method BGRL, which converges and reports results for 10 000 epochs. Additionally, we reproduce the results of BGRL and provide values for BGRL at 1000 epochs. In Table we report the mean node classification accuracy along with the standard deviations. As our experimental scenario was aligned with BGRL. one, we re-use their reported scores and compare them to our results. We observe that our proposed method outperforms other baselines and achieves comparable results to state-of-the-ee methods. Moreover, our G-BT model outperforms BGRL at 1000 epochs.
Table 1: Mean and std accuracy of transductive node classification over 20 data splits and initial- izations obtained in BGRL paper (Thakoor et al., 2021) and experiment (+) within the same experimental setup. OOM denotes running out of memory on a 16GB V100 GPU.
WikiCS
Am-CS
Am-Photo
Co-CS
Co-Physies
Raw features
71.98 0.00
73.81
0.00
78.53 + 0.00
90.37 + 0,00
93.58 + 0.00
DeepWalk
74.35 + 0.06
85.68 + 0.06
89.44 + 0.11
84.61 + 0.22
91.77 : 0.15
Weep + fts
77.21 0.03
86.28 0.07
90.05 + 0.08
87.70 = 0.04
94.90 = 0.09
DGI MVGRL
75.35 + 0.14
83.95 0.47
91.61 + 0.22
92.15 = 0.63
94.51 + 0.52
77.52 0.08
0.11
91.74 + 0.07
92.11 = 0.12
95.33 = 0.03
GRACE (l0k epochs)
80.14 + 0.48
89.53 0.35
92.78 + 0.45
91.12 = 0.20
OOM
BGRL (10k opachs)
79.36 + 0.53
89.68 + 0.31
92.87 + 0.27
93.21 = 0.18
95.56 + 0.12
G-BT * (S
spuchs)
76.65 0.62
88.14 +
92.63 + 0.44
92.95 + 0.17
95.07 + 0.17 OOM*
BGRL* epochs)
73.24 + 0.62
87.37 + 0.40
91.77 + 0.57
92.07 = 0.06
GCA Supervised GCN
78.35 + 0.05
88.94 + 0.15
92.53 # 0.16
93.10
95.73 + 0.03
77.19 + 0.12
86.51 0.54
92.42 + 0.22
93.03 = 0.31
95.65 + 0.16
4.2.2 OGBIVRXIV DATASET
In the next experiment, we use ogb-arxiv - a larger graph from the Open Graph Benchmark (Hu et al., 2020) with about 1 170 thousand nodes and about 1.1 million edges.
Encoder model Due to the larger size of the graph, we extend the encoder fe to a 3-layer GCN model. We employ batch normalization and PReLU activations after the first and second layer, leaving the final layer as is (i.e. without any activation of normalization). In the BGRL paper, the authors suggested use layer normalization together with weight standardization (Qiao et al., 2019), yet we did not observe any performance gain, but more importantly the training procedure was unstable, with many peaks in the loss function.
Results and discussion In Table 2 we report the mean classification accuracy along with the standard deviations. Note that we provide values for both validation and test splits, as the provided data splits are build according to chronological order. Hence, any model will be more affected with the out-tf-distributiin eтror on further (in time) away data samples. We evaluate our model for 500 epochs but it converges as fast as about 300-400 epochs (further training did not give any

---

Under review as a conference paper at ICLR 2022
improvements). The model achieves results which are only 1.5 pp off to the state-of-the- art method BGRL, which turn takes 10 000 epochs to converge to such solution.
4.3 INDUCTIVE EXPERIMENTS
We evaluate our proposed G-BT framework in inductive tasks over single and multiple graphs.
Table 2: Mcan and std accuracy of transductive node classification the ogb-arxiv dataset over 20 data splits and initializations obtained in BGRL paper (Thakoor et al., 2021) and our experiment
4.3.1 PPI
For the inductive learning case with multiple graphs, we employ the Protein-Protein Inter- action (PPI) dataset (Zitnik & Leskovec, 2017). Aligned with other methodss we provide results for multilabel node classification in terms of the Micro-F1 score.
(*) within the same experimental setup.
Validation
Test
MLP
57.65 + 0.12
55.50 + 0.23 70.07 0.13
node2vec
71.29 0.13
DGI
71.26 0.11
70.34 + 0.16
GR ARACE (10K epachs)
72.61 + 0.15
71.51 + 0.11
Encoder model We employ a Graph Atten- tion (GAT) Veličković et al. (2018) based en- coder model, as previous works have shown bet- ter results of such network compared to standard GCN layers on PPI. Specifically, we build our encoder fe 3-layer GAT network with skip connections. The first and second layer uses 4 attention heads of size 256 which are concate- nated, and the final layer uses 6 attention heads of size 512, whose outputs attention mechanism learns the weights that are used to aggregate information from neighboring nodes. The attention weights Ozj computed according the following equation:
BGRL (l0k spachs)
72.53 + 0.09
71.64 + 0.12
G-BT epoch))
71.16+ 0.14
70.12 + 0.18
Supervised GCN
73.00 + 0.17
71.74 + 0.29
averaged instead of applying concatenation. In the GAT model, an
exp (LeakyReLU (a" [Wh,]] (h,l))
(4)
EkENS exp (LeakyReLU (a7 [Wh.]] Wh./Wha.]) Whal))
where Mf are the neighbors of node i, W is trainable matrix to transform node attributes, a the trainable attention matrix, and || denotes the concatenation operation.
We use the ELU activation for the first and second layer, leaving the last layer without any activation function. We do not apply any normalization techniques in the model as preliminary experiments showed no performance improvement.
Results and discussion We train our framework using a batch size of 1 graph for total of 500 epochs, which turned out to be enough for the model to converge (we conducted some preliminary experiments). In Table 3, we report the mean Micro-F1 score along with the standard deviations over 20 model initialization, as this dataset provided only one data split. Training for only 500 epochs provided results on par with SOTA method BGRI. our model achieves 70.49 using a GAT encoder.
4.3.2 DGB--RODUEE
We study the applicability of proposed model in the case of large-scale graphs. We select the ogb-products dataset, which has about 2.5 million nodes and 61 million edges.
Encoder model and setup We utilize the same GAT-based encoder as for PPI. Due to the size of this dataset and the resulting training time, we decide to perform inductive node classification, i.e., during training we use only the nodes from the training set and edges among them. Moreover, as this graph does not fit into GPU memory, selected batched setting with neighbor sampling (as proposed in Hamilton et al. (2017)) instead of the full-batch scenario. We train our model with a batch size of 512 for 100 epochs.
7

---

Under review as a conference paper at ICLR 2022
Results and discussion BGRL does not report results for this dataset, so modify the implemen- tation of the BGRL method to accept batches instead whole graphs and evaluate it on this dataset. We also include results from the OGB leaderboard, but note that virtually all methods reported there are trained in a semi-supervised setting, contrary to our approach in the self-supervised setting. Therefore, may expect worse results. We summarize the mean and std node classification accuracy values in Table 4. We observe that G-BT highly outperforms BGRL on both validation and test sets.
Table 3: Mean and std Micro-F1 multilabel node classification the PPI dataset over 20 model initializations obtained BGRL paper (Thakoor et al., 2021) and our experiment (*) within the same experimental setup.
Table 4: Mean and std aceuracy of inductive node classification on the ogh-products dataset over 5 model initializations obtained in the OGB leader- board and our experiment (x) within the same ex- perimental setup.
PPI (test set)
Validation
Test
Raw features
42.20
Features*
63.18 + 0.01
50.93 + 0.01
DeepWalk>
87.42 + 0.09
73.11 + 0.44
DGI
63.80
0.20
DeepWalk ftss*
87.84 + 0.09
73.38 +: 0.11
GMI
65,00 = 0.02
GRACE
66.20
0.10
BGRL* (100
78.06 2.12
63.97 + 1.62
GRACE GAT encoder (Ik epochs)
69.71
0.17
G-BT* * (100 epo0h)))h)
85.04 + 0.23
70.46 + 0.38
BGRL GAT encoder (Ik epechs)
70.49
0.05
Supervised GCN
92.00 + 0.03
75.64 + 0.21
G-BT (500 spochs)
70.49 0.19
Supervised GAT
97.30 0.20
4.4 AUGMENTATION HYPERPARAMENT SETS
In our model, we postulate use the same augmentation function hyperparameters to generate both graph views. This is motivated by the symmetrical architecture of our model, and hyperparameter search complexity. Performing simple grid search over the value space yields in our case total number of 6 = 36 evaluated combinations (values: {0, 00.......... 0.5)). In contrary. usage of different parameter sets for both graph views, would generate (6 1296 combinations, which can be further reduced by exploiting the symmetrical architecture, yielding final value of 630 (2) combinations to evaluate. To demonstrate the impact of using both the same and different augmentation hyperparameter value sets we provide the results in Figure There no substantial difference in terms of test accuracy.
WikiCS
Amazon-CS
Amazon-Phoer Photo
Coauthor-CS
Poasssor-Physics
Test accuracy [%)]
78
72.90
78,48
89.0
88,57
92.97
92.94
93,36 93,23
94.94
95,22
86.5
88.24
92
94
74
87.5
87.0
93
86.5
70
Same Different
Same Different
Same Different
Same Different
Same Different
Figure 2: Comparison of using the "Same" and "Different" augmentation hyperparameter sets.
4.5 TRAINING TIME COMPARISON
We compare the training time of all considered models by the duration of single epoch (the evaluation phase the same in all models). We run each model for 10 training epochs and report the mean and standard deviation of the time measurements (Table 5). In virtually all cases model takes the least
8

---

Under review as a conference paper at ICLR 2022
time for single training iteration due to the simple symmetrical architecture. Compared to BGRL our method speeds up computations about 17-42 times.
Table 5: Single epoch running time (in seconds) averaged over 10 training epochs.
WikiCS
Am-CS
Am-Photo
Co-CS
Co-Phy
WeepWalk
0.83 0.02
0.96 = 0.02
0.62 = 0.02
1.22 0.03
2.25 0.03
DGI
0.06 + 0.03
0.09 + 0.00
0.05 + 0.00
0.19 0.00
0.50 + 0.21
MVGRL.
00M
00M
00M
OOM
OOM
GRACE
0.33 + 0.10
0.43 + 0.03
0.15 + 0.01
OOM
OOM
BGRL
0.12 0.01
0.18 + 0.01
0.08 + 0.00
0.35 0.01
OOM
Epochs I training time [s]:
10 000 1 200
10 000 800
10 000 800
10 000 500
10 000 1-
G-BT
0.05 0.00
0.07 + 0.00
0.04 + 0.00
0.22
0.05
0.44 + 0.01 9001
Epochs training time [s]:
900 45
600 42
500 20
900 198
Speedup (vs BGRL):
26x
42x
40x
17x
4.6 BATCHED PROCESSING
Our proposed method allows working in both full-batch and mini-batch settings. For most considered datasets, splitting them up into batches is not required these fit completely into the GPU's memory. Nevertheless, we run additional experiments where we train our G-BT model on these datasets in a batched manner. Batches created using neighbor sampling. i.e. for k-layer encoder model, sample the k-hop neighhorhood of a node. More specifically, we first subsample the direct neighbors, then we sample neighbors of those, augmentation hyperparamter values and number of epochs found in the full-batch case and retrain the G-BT model for each batch size 5 times (Table 6). We observe an expected decrease in performance when using the batched scenario (subject to further finetuning).
(as proposed in Hamilton et al. (2017)). We re-use the
Table 6: Evaluation of G-BT model in batched setting.
Batch size
WikiCS
Am-CS
Am-Photo
Co-CS
Co-Phy
Full-batch
76.65 + 0.62
88.14 = 0.33
92.63 + 0.44
92.95 + 0.17
95.07 + 0.17
256
75.69 + 1.02
87.93 = 0.39
91.24 = 0.46
91.82 0.22
94.98 = 0.14
512
75.83 0.64
88.21 = 0.44
91.21 0.44
91.62 0.22
94.95 = 0.12
1024
75.79 + 0.77
87.94 + 0.50
91.24 + 0.47
91.54
94.91
0.12
2048
75.58 + 0.52
87.92 + 0.29
91.21 + 0.40
91.43 + 0.28
94.84 0.11
CONCLUSIONS
In this work we presented Graph Barlow Twins, self-supervised graph representation learning framework, which utilizes the embeddings' cross-correlation matrix computed from two distorted views of a particular graph. The framework is fully symmetric and does not need any special techniques to build non trivial embedding vectors. It builds representations that are invariant to the applied augmentations and reduces the redundancy in the representation vectors by enforcing the cross-correlation matrix to be equal to the identity matrix (Barlow Twins loss). Using 8 real-world datasets we evaluate our model in node classification tasks, both transductive and inductive, and achieve results that
par better than SOTA methods in SSL graph representation learning. We also show that our model converges an order of magnitude faster than other approaches.
Our method allows to reduce the computation cost (faster convergence) keeping a decent performance in downstream tasks. Consequently, it can be used to process larger graph datasets and efficiently perform tasks such node classification, link prediction or graph classification. These tasks have crucial impact on various machine learning areas where graph structured data is used, e.g. detection of bots hate speech in social networks, or building graph based recommendation engines.

---

Under review as a conference paper at ICLR 2022
REFERENCES
J Ballé, V Laparra, and E Simoncelli. End-to-end optimized image compression. In Int'l Conf on Learning Representations (ICLR), Toulon, France, April 2017. URL https: /ttp:://xxivvvooo abs/1611 611.1.1 01704. Available at http://aniivoor/ab///61..100
Feihu Che, Guohua Yang, Dawei Zhang, Jianhua Tao. Pengpeng Shao, and Tong Liu. sule-supervieed graph representation learning via bootstrapping. arXiv preprint arXiv:201 1,05/26, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2020.
G. Deco and L. Parra. Non-linear feature extraction by redundancy reduction in unsupervised stochastic neural network. Neural Netw., 10(4))683-69) 691, June 1997. ISSN 0893-6080. doi: 10. 12.1016/50893--0800990000 10-4. URL https: //t::///oooo////ooo www.ddo....oooooooooo 101 6 008//0000030000 6080 (96) 00110-4.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Reprexentation Learming on Graphs and Manifolds, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to sulf--uper learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284. Curran Associates, Inc., 2020. URL https: httthhttss////rceeeeeee http///ooeeeeeegeeeeeeee www.iiinn....u..uneee.e www.ieerss..... cc/ paper 22000//220///0 file///l//// 8
42b1 7b81 92b 92b2958e-Paper. pdf.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In KDD, 2016.
Gugger and Howard. Adamw and super-convergence is now the fastest way to train neural nets., 2018. URL https://// / www. fast. htt.taaa/////////// 8/0//02/////////// /adammdamwwee addm-weight-deee seeeihhhtdcyy/////
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning large graphs In NIPS, 2017.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovee. Open graph benchmark: Datasets for machine learning on graphs. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 22118-22133. Curran As- sociates, Inc., 2020. URL /ttpssss//// neurips. ca/ happpppppp0000 file/ pdf.
Thomas N Kipf and Max Welling. Variational graph auto-encoders. NIPS Workshop on Bayesian Deep Learning, 2016.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks.
In International Conference on Learning Representations (ICLR), 2017.
Ruslan Kuprieiev, Dmitry Petrov, Saugat Pachhai, Pawet Redzyáski, Casper da Costa-Luis, Alexander Schepanovski, Peter Rowlands, Ivan Shcheklein, Jorge Orpinel, Fábio Santos, Batuhan Taskaya, Aman Sharma, Zhanibek, Gao, Dani Hodovic, Andrew Grigorev, Earl, ni, Nabanita Dash, George Vyshnya, maykulkarni, Max Hora, Vera, Sanidhya Mangal, Wojciech Baranowski, Clemens Wolff, Alex Maslakov, Alex Khamutov, and Kurian Benoy. Dve: Data version control git for data & models, May 2021. URL ht:ps: /dto:.////iiiie/ org 10. /2w////eooo...oo/ooooooooooooooe 4733984.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Image-based rec- ommendations styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15, pp. 43-52, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 978145033622 10. 10.1145/276666222777555 URL https:tpp///cc htt./dd..orrg///// 10..00..1152227666 27 67755.
doi:
Péter Mernyei and Cätälina Cangea. Wiki-cs: A wikipedia-bused benchmark for graph neural networks. arXiv preprint arXiv:2007.02991, 2020.
10

---

Under review as a conference paper at ICLR 2022
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 12:2825 2830, 2011.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '4, pp. 701-710, New York, NY, USA, 2014, ACM. ISBN 978- 1-4503-2956-9. doi: 10. 10.1145/222333022222213 2623330. 2623732.
URL http:t////oooooott cm,
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. arXiv preprint arXiv: 190.0 2019.
J. Schmidhuber. Making the world differentiable: On using fully recurrent self supervvee neural net- works for dynamic reinforcement learning and planning in non-stationary environments. Technical Report FKI-126-90 (revised), Institut für Informatik, Technische Universität München, November 1990.
Jürgen Schmidhuber, Martin Eldracher, and Bernhard Foltin. Semilinear predictability minimization produces well-known feature detectors. Neural Computation, 8(4):773-786, 1996. doi: 10.1162/ 1e96..996...4477777
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June (Paul) Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th International Conference on World Wide Web, WWW "15 Companion, pp. 243-246, New York, NY. USA, 2015. Association for Computing Machinery. ISBN 9781453334730. doi: 10.1 N0.1145/2740088.22444 2742839. URL /nttts:///ol.e htt.//oi.ogggg///////oo///////oe /10. 1145/ 2740 0000444008..00 2742833
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In WWW. ACM, 2015.
Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Remi Munos, Petar Veličković, and Michal Valko. Bootstrapped representation learning on graphs. In ICLR 202/ Workshop on Geometrical and Topological Representation Learning, 2021. URL httpps://peerrevier net/ net/forum?ide www.uooiorrurrraeee -itidii--rrrRAAA 9Ud.
Yao-Hung Hubert Tsai, Shaojie Bai, Louis-Philippe Morency, and Ruslan Salakhutdinov. A note on connecting barlow twins with Srgative-sample-iree contrastive learning, 2021.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
URL. https::tpp...... /tttl://pppeeee /openrevieweeeeeeeoeoee net/ forum? dd= kCZ. accepted as poster.
Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep Graph Infomax. International Conference on Learning Representations, 2019.
URL https:ttp//cccc http:://ppeeevvveeeee
net forum? www.ummidiri22iieee..
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5812- 5823. Curran Associates, Inc., 2020. URL https: htttt::/tpppceeeeeee http:////ccceeeeeeeeeeee neurips cc/paper/ 2020/
Paper .df.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction, 2021.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep Graph Contrastive Representation Learning. In ICML. Workshop on Graph Representation Learning and Beyond, 2020a. URL http:///////
/aaaiiiii.. org/ abs abs/2006.... 04131.
Yanqiao Zhu, Yichen Xu, Feng Yu. Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. CoRR, abs 0. 4.45.
2020b. URL https: / // /awxivivvvo///// orgg/
11

---

Under review as a conference paper at ICLR 2022
Marinka Zimikk and Jure Leskovec. Predicting multicellular function through multi-layer tissue net- works. Bioinformatics, 33(14)x110--1198 07 2017. ISSN 1367-4803, doi: 10.109//bioinformatie btx252. Utthttss:
A DATASET DESCRIPTIONS
We provide brief descriptions for each dataset, including the basic statistics (see Table 7) and the employed dataset split type for the node classification downstream task:
WikiCS (Memyye & Cangea, 2020) is network of Computer Science related Wikipedia articles with edges denoting references between those articles. Each article belongs to one of 10 subfields (classes) and has features computed averaged Glo embeddings of the article content. We use the provided 20 train/val/test data splits without any modifications.
Amazon Computers, Amazon Photos (McAuley et al., 2015) are two networks extracted from Amazon's ' co-purchase data. Nodes are products and edges denote that these products were often bought together. Each product is described using a Bag-of- (node features) based on the reviews. There are 10 and 8 product categories (node classes), respectively. For these datasets there are no data splits available, so similar to BGRI,, we generate 20 random train/val/test splits (10% (10%/10%//80%)
representation
Coauthor CS, Coauthor Physics
networks extracted from the Microsoft Academic Graph (Sinha et al., 2015). Node are authors and edges denote a collaboration of two authors. Each author is described by the keywords used in their articles (Bag-of-Words representation; node features). There are 15 and 5 author research fields (node classes), respectively. Similarly the Amazon datasets there are no data splits provided, so we generate 20 random train/val/test splits (10%//0%%/80%)
ogb-arxiv is a larger graph from the Open Graph Benchmark (Hu et al., 2020) with about 170 thousand nodes and about 1.1 million edges. The graph was extracted from the Microsoft Academic Graph (Sinha et al., 2015), where nodes represents a Computer Science article on the arXiv platform and edges denote citations across papers. The node features are build as word2vec embeddings of the whole article content. There are 40 subject areas a node can be classified into (node label/class). The ogb-arxiv dataset provides a single train/val/test split, we use it without any modifications, but we retrain the whole framework 20 times.
Protein-Protein Interaction (PPI) (Zitnik & Leskovee, 2017) consists of 24 separate graphs. Each node in single graph represents a protein, described by 50 biological features, and edges denote interactions among those proteins. There are 121 node labels, but note that contrary to other cases, PPI uses multilabel classification, i.e. single protein can be assigned with multiple labels. Aligned with other methods, we provide results in terms of the Micro-Fl score. For PPI, there exists a predefined data split, where 20 graphs are used for training, graphs for validation and 2 graphs for testing. Note that the validation and test graphs are completely unobserved during training time, hence the model is more challenged during inference time.
ogn-products is large-scale graph from the Open Graph Benchmark (Hu et al., 2020) with about 2.4 million nodes and 62 million edges. The graph was extracted from the Amazon product co- po-purcchsiin network. Nodes represent products from the Amazon store and edges denote whether two products were bought together. There are 100 node features, which are obtained from bag-of-we words products descriptions reduced using PCA. Each product (node) can be classified into one of 47 categories (node labels). This dataset comes with a predefined data split, so use is.
B EXPERIMENTAL DETAILS
Augmentation hyperparameters Our proposed framework uses a single pair of augmentation hyperparameters P.A E Px R compared to other methods that use different values to generate both graph views. We show that single set is enough to achieve a decent performance a symmetrical network architecture like ours. Therefore, we cannot the reported values of other
12

---

Under review as a conference paper at ICLR 2022
Table 7: Dataset statistics. We small to medium sized standard datasets together with the larger ogb-arxiv dataset in the transductive setting, We also evaluate the inductive setting using the ogb-products and PPI (multiple graphs) dataset.
Name
Nodes
Edges
Features
Classes
WikiCS
11,701
216,123
300
10
Amazon Computers
13,752
245,861
767
10
Amazon Photos
7,650
119,081
745
8
Coauthor CS
18,333
81,894
6,805
15
Coauthor Physics
34,493
247,962
8,415
5
ogb-arxiv
169,343
1,166,243
128
40
PPI (24 graphs) ogb-products
56,944
818,716
50
121 (multilabel)
2.449,029
61,859,140
100
47
works. We instead perform grid search over these hyperparameters and use those where the model performs the best (in terms of classification accuracy or Micro-F1 score, for PPI). We do not evaluate the model during training and just use the final version after training. We use the following setting:
the framework is trained to 500 epochs,
the learning rate warmup time 50 epochs.
for both hyperparameters PA and px we check following values: (0,0.1,.... 0.53.
For values greater than 0.5 the augmentation removes too much information from the graph. In the case of the ogb-products dataset, due to large size, we trained our model only for 10 epochs with warmup period of epochs, but we evaluated the same augmentation hyperparameter values. We summarize the augmentation hyperparameters of the best performing models Table
Table 8: Augmentation hyperparameters. Ogb-products trained in the batched setting with batch size of 512.
G-BT
PA
Px
WikiCS
0.2
0.1
Amazon-CS
0.4
0.1
Amazon-Photo
0.0
0.5
Coauthor-CS
0.5
0.1
Coauthor-Physics
0.1
0.4
ogb-arxiv
0.2
0.0
PPI ogb-products+
0.1
0.1
0.2
0.1
Training setup For all datasets. we train our framework using the AdamW (Gugger Howard, 2018) optimizer with a weight decay of 10 50-5, The learning rate is adjusted using cosine annealing strategy with linear warmup period up to the base learning rate. During training we set a total number of epochs and an evaluation interval, after which the frozen embeddings are evaluated in downstream tasks (using either the la regularized logistic regression from Scikit learn (Pedregosa et al., 2011) with liblinear solver, or the custom PyTorch version with AdamW for ogb-arxiv and PPI). For instance, if we set the total number of epochs to 1000 and the evaluation interval to 500, the model will be evaluated at epochs: 0, 500 and 1000 (three times in total). We report the values for the best performing model found during those evaluations. We summarize these training statistics in Table 9.
13

---

Under review as a conference paper at ICLR 2022
Table 9: Training hyperparameters.
G-BT
total epochs
warmup
evaluation interval
base
best model found
learning rate
WikiCS
000
100
100
10-4
900
Amazon-CS
1 000
100
100
5 10
600
Amazon-Photo
1 000
100
100
1 10-4
500
Coauthor-CS
000
100
100
10-5
900
Soauthor-Physice
1 000
100
100
1 10-5
900
ogb-arxiv
500
100
100
10-3
300
PPI
500
50
100
5 10
500
ogb-products
100
10
10
10
100
Encoder architecture We compare our framework against the state-of-the-art self-supervised graph representation learning method BGRL (Thakoor et al., 2021). To provide fair comparison, we use similar encoder architectures to the ones presented in their paper. We do not use any predictor networks in our framework, loss function (as given in the Barlow Twins paper (Zbontar et al., 2021)), i.e. we do not apply any normalization (like batch layer normalization) or activation function in the final layers of the encoder. Note that the lack of predictor network and batch normalization in the final layer, reduces the overall number of trainable network parameters. all cases, we use batch normalization with the momentum equal to 0.01 (as in BGRL (Thakoor et al., 2021), where they use the equivalent weight decay equal to 0.99).
need to slightly modify the encoders be better suited for the
For the small up to medium sized datasets, i.e. WikiCS, Amazon-CS, Amazon-Phoo Coauthor-CSS CS, Coauthor Coauthor-Physics, we use 2-layer GCN (Kipf Welling, 2017) based encoder with the following architecture:
GCN(K: 2d),
BatchNorm(2dd),
PReLUO,
GCN(2d, d),
where k is the number of node features and d is the embedding vector size.
For the ogb-arxiv dataset, we use slightly larger model - a 3-layer GCN (Kipf & Welling, 2017) based encoder. We tried utilize weight standarization (Qiao al., 2019) and layer normalization, but our model did not benefit from those techniques (as helped in BGRL (Thakoor et al., 202 I)). The training procedure under this setting unstable with various fluctuations and peaking of the loss function. The final architecture is summarized as follows:
GCN(K, d),
BatchNorm(d).
PReLUO,
GCN(d, d),
BatchNorm(d).
PReLUO,
GCN(d, d).
In the inductive experiment with the PPI dataset, we 3-layer GAT (Veličković et al., 2018) based encoder. Graph Attention network are known to perform better on this dataset compared to GCNs. This was also showed in BGRL (Thakoor al., 2021), where their approach with GAT layers provided ttate-of-the-at performance in self-supervised graph representation learning for PPI. Our architecture can be summarized follows:
14

---

Under review as a conference paper at ICLR 2022
GAT(k; 256, heads=4) + Linear(k, 4 256)
ELUO,
GAT(4 256, 256, heads=4) + Linear(4 256, 4 256)
ELUO.
GAT(4 256, d. heads=6) + Linear(4 * 256, d)
The outputs of the attention heads in the first and second layer concatenated and for the last GAT layer, the attention heads outputs are averaged. In every layer, we utilize skip connections using linear layers to project the outputs of the previous layer (features in the case of the first layer) the desired dimensionality.
The exact values for the input feature dimension & and the embedding dimension d are given in Table 10.
Table 10: Encoder layer size parameters.
G-BT
number of node features
embedding dimensionality
WikiCS
300
256
Amazon-CS
767
128
Amazon-Phote
745
256
Coauthor-CS
805
256
Coauthor-Physies
8 415
128
ogb-arxiv
128
256
PPI
50
512
ogh-products
100
128
Code and reproducibility We implement all our models using the PyTorch-Geometrie library (Fey Lenssen, 2019). The experimental pipeline is built using the Data Version Control tool (DVC (Kuprieiev et al., 2021)). It enables to run all experiments in single command and ensure better reproducibility. We attach the code in the supplementary material. To reproduce the whole pipeline run: dve repro and to execute single stage use: dve repro <stage name>, There are following stages:
preprocess _atrtsss__ttteeeeeeeeee <dattset name> downloads the edataset_name> dataset; applicable, generates the node splits for train/val/test,
the augmentation hyperparameter search for given dataset in the full-batch case,
full 1_bbbbthhhhhm Bacchhhhh@s____
full full_bat.ch_te tra Eedaaasee_naaeee batched_tree (rraannddattttt_nneee name trains and evaluates the G-BT model for given dataset in the full-batch case and the batched scenario, respectively,
batched seched_hps_ogbrn poggoonooduucttsssppoooos
the augmentation hyperparameter search for the ogb-products dataset in the batched scenario,
batched ra ctacheeddtrrinoonnn____oo lrrii_nooonnooouuceee
trains and evaluated the G-BT model for the ogb-products dataset in the batched scenario,
compareeaaumentat" sit_tiinnehheeee pwwypperaammeeer... raweter_settettee - - loads all full-batch augmen- tation hyperparameter results, compares the when using the same different sets of hyperparameters to generate both graph views (outputs Figure 2).
compare, pempare_rrus www__rnnnnngg. mes computes the average running time of a training epoch for the following methods: DeepWalk, DGI, MVGRI, GRACE, BGRL and G-BT,
train _____or_______ full fuul_bathhheee <daatett.t > name> trains and evaluates the BGRL model in the full-batch case for WikiCS, Amazon-CS. Amazon-Photo, and Coauthor-CS.
15

---

Under review as a conference paper at ICLR 2022
bgrl hps batcheddoob--producess runs the augmentation hyperparameter search for BGRL using the ogb-products dataset,
beteeeeddtrrinnoorrrre saaainnooon-prouucttt products trains and evaluates the BGRL model for the ogb-products dataset,
eva wrr_luatt__eeaurrs__proode - evaluates the performance of ogb-products raw node features,
evaluate evaluate_deepwalkk wwwpweeleperrdeeeeeeeeee products evaluates the performance of DeepWalk on the ogb-products dataset; additionally the case of DeepWalk features concatenated with raw node features is also evaluated.
All hyperparametens described in this Appendix are stored in configuration files in the experimentssconfigge directory, whereas the experimental Python scripts are placed in the experiments/script.. directory.
16


Reviews:
[{"review_id": "kReNgHYO49b", "reviewer": "Reviewer_bHed", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is easy to follow, well-written and organized. This paper is a heuristic attempt to apply Barlow Twins in graph domain. Both transductive and inductive experiments are done to evaluate the performance. The proposed method achieved results on par with the SOTA methods while the time and space complexity are lower.\n\nI have some minor concerns below for the authors to address.\n\n1. The novelty of this paper is limited. The challenges to be tackle of the application about Barlow Twins in graph domain is unclear to me. \n\n2. Data augmentation in vision tasks comes from strong human prior, e.g., random resize, cropping and horizontal flipping would not change the semantic of an image. While the graph data augmentation methods used in this paper is borrowed from previous literatures, it makes no sense to me. For example, applying edge dropping to a protein would obviously lead to different bio-molecules.\n\n3. The experimental results are on par with baseline methods on the most tasks. Considering that the low time and space complexity is coming from previous literature, i.e., Barlow Twins, the experimental contribution is limited.\n\n4. In terms of the encoder network and augmentation hyperparameter design, the paper did not provide comprehensive analysis or ablation studies.\n\n5. The authors carefully describe the downstream datasets. Maybe I miss it, but I don't find the pre-trained dataset used in the experiment.", "summary_of_the_paper": "This paper proposed a self-supervised learning framework for graph representation learning based on a cross-correlation-based loss function. In the proposed framework, two views of the input graph obtained by augmentation methods are passed through the same encoder to compute two embedding matrices, then Barlow Twins loss is used to compute the loss according to the embedding matrices.\nThe main contribution of this paper lies in that it adapted Barlow Twins from vision to graph representation learning field and evaluated the performance of this self-supervised framework in multiple node classification tasks. The proposed method achieved analogous results compared to SOTA methods with lower time and space complexity.\n", "summary_of_the_review": "This paper adapted the recent Barlow Twins to self-supervised graph representation learning and provided some informative empirical experiment results. With such interesting trials, the reviewer expected to see the concerns are well addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "cR3Xg4dkEVl", "reviewer": "Reviewer_Bwpq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenghts:\n\n1. Paper is clearly written and easy to follow.\n2. The experimental evaluation is robust and experimental details are clearly stated.\n\nWeakness:\n\nThe paper is the straightforward extension of previous work Barlow-Twins for graph structured data. The paper does not has any technical novelty in my opinion. I am willing to increase the score of the paper if in the rebuttal, the authors can clearly state the novelty of the paper w.r.t to the Barlow- Twin paper. ", "summary_of_the_paper": "The paper applies the recently proposed self-supervised learning method Barlow-Twins to graph structured data. For constructing the augmented version of a graph, previous methods such as edge-dropping or feature masking are used. The paper conducts experimental evaluation on datasets of various scales on both transductive and inductive setting.  ", "summary_of_the_review": "Although the paper is clearly written and many experiments are presented, the paper does not meet the bar of the top conference such as ICLR because of it being a very direct application of a previous paper.\n\n\nMy review is rather short for this paper because based based on the lack of novelty of this paper, I do not have many questions to ask or suggestions to make.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "aMkmg59WBT-", "reviewer": "Reviewer_H4MK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Disclosure: I have reviewed a previous version of this paper. The authors have included thorough full-batch experiments as well as larger scale datasets such as ogbn-products, which is much appreciated.\n\nThe paper is clear, well-written and easy to follow. The authors propose a simple but meaningful extension of the Barlow Twins idea to the graph domain, and demonstrate its effectiveness on relevant experiments. I think allowing for symmetric loss is a very important direction for graph representation learning, and the proposed solution is an elegant way of achieving that.\n\nMy main concern would be with the reported BGRL results on ogbn-products. I understand that the authors have ran BGRL under the same computational budget as G-BT, but it appears clear that BGRL needs more time to reach peak performance. Would it be possible, just to avoid muddying the waters for future work, to run BGRL for longer and report how the performance is affected? It is OK if this number is higher than G-BT's reported performance -- the authors are optimising for a different metric.", "summary_of_the_paper": "The authors study symmetric self-supervised graph representation learning without negative samples, inspired by the Barlow Twins method previously proposed in the image domain. They illustrate that using their method, it is possible to achieve competitive performance to state-of-the-art methods such as BGRL, at a fraction of the training cost.", "summary_of_the_review": "I think that sufficiently many of my previous concerns have been addressed, and I am now leaning on the side of acceptance. The authors have presented a useful extension of Barlow Twins into the graph domain, and now have experiments in support of the industrial relevance of their method. The novelty is somewhat limited (as is the case for most of the recent graph SSL papers that adapt image domain techniques) but it is useful in and of itself that the gains observed in images transfer well to the irregular domains.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use the recently introduced "Barlow-twins" contrastive learning objective, to the case of graph networks. The main concern raised by reviewers was the limited novelty of this work, which they argued mostly combines existing lines of work, and does not introduce sufficiently new concepts. This was also discussed between the authors and the reviewers.
Having read the paper and the reviews, I tend to agree with the reviewers that this paper is more of a combination of existing works, and their relatively straightforward application to the graph network domain. Thus, although the empirical results are encouraging, I agree the paper has limited novelty, and falls below the ICLR acceptance bar.
Decision: Reject