Graph Barlow Twins: A self-supervised representation learning framework for graphs

1 INTRODUCTION . Graph representation learning has been intensively studied for the last few years , having proposed various architectures and layers , like GCN ( Kipf & Welling , 2017 ) , GAT ( Veličković et al. , 2018 ) , GraphSAGE ( Hamilton et al. , 2017 ) etc . A substantial part of these methods was introduced in the semi-supervised learning paradigm , which requires the existence of expensive labeled data ( e.g . node labels or whole graph labels ) . To overcome this , the research community has been exploring unsupervised learning methods for graphs . This resulted in a variety of different approaches including : shallow ones ( DeepWalk ( Perozzi et al. , 2014 ) , Node2vec ( Grover & Leskovec , 2016 ) , LINE ( Tang et al. , 2015 ) ) that ignore the feature attribute richness , focusing only on the structural graph information ; and graph neural network methods ( DGI ( Veličković et al. , 2019 ) , GAE , VGAE ( Kipf & Welling , 2016 ) ) that build representations upon node or graph features , achieving state-ofthe-art performance in those days . Recently self-supervised paradigm is the most emerging branch of unsupervised graph representation learning and gathers current interest and strenuous research effort towards better results . The most prominent methods were developed around the contrastive learning approach , such as GCA ( Zhu et al. , 2020b ) , GraphCL ( You et al. , 2020 ) , GRACE ( Zhu et al. , 2020a ) or DGI ( Veličković et al. , 2019 ) . Although contrastive methods are popular in many machine learning areas , including computer vision and natural language processing , their fundamental limitation is the need for negative samples . Consequently , the sampling procedure for negative examples highly affects the overall quality of the embeddings . In terms of images or texts , the definition of negative samples might seem not that problematic , but in the case of graphs there is no clear intuition . For instance , what is the negative counterpart for a particular node in the graph , should it be a node that is not a direct neighbor , or a node that is in a different graph component ? There are multiple options available , but the right choice strictly dependent on the downstream task . Researchers have already tackled the problem of building so-called negative-sample-free methods . In research being conducted in computer vision they obtained successful results with methods such as BYOL ( Grill et al. , 2020 ) , SimSiam ( Chen & He , 2020 ) or Barlow Twins ( Zbontar et al. , 2021 ) . These models utilize siamese network architectures with various techniques , like gradient stopping , asymmetry or batch and layer normalizations , to prevent collapsing to trivial solutions . Based on BYOL , Thakoor et al . ( 2021 ) proposed the Bootstrapped Representation Learning on Graphs ( BGRL ) framework . It utilizes two graph encoders : an online and a target one . The former one passes the embedding vectors to a predictor network , which tries to predict the embeddings from the target encoder . The loss is measured as the cosine similarity and the gradient is backpropagated only through the online network ( gradient stopping mechanism ) . The target encoder is updated using an exponential moving average of the online encoder weights . Such setup has been shown to produce graph representation vectors that achieve state-of-the-art performance in node classification using various benchmark datasets . Notwithstanding , assuming asymmetry between the network twins ( such as the predictor network , gradient stopping , and a moving average on the weight updates ) the method is conceptually complex . Employing a symmetric network architecture would seem more intuitive and reasonable , hence in this paper , we propose Graph Barlow Twins ( G-BT ) , a self-supervised graph representation learning framework , which computes the embeddings cross-correlation matrix of two distorted views of a single graph . The approach was firstly introduced in image representation learning as the Barlow Twins model ( Zbontar et al. , 2021 ) but was not able to handle graphs . The utilized network architecture is fully symmetric and does not need any special techniques to build non trivial embedding vectors . The distorted graph views are passed through the same encoder which is trained using the backpropagated gradients ( in a symmetrical manner ) . Our main contributions can be summarized as follows : ( I ) We propose a self-supervised graph representation learning framework Graph Barlow Twins . It is built upon the recently proposed Barlow Twins loss , which utilizes the embedding cross-correlation matrix of two distorted views of a graph to optimize the representation vectors . Our framework neither requires using negative samples ( opposed to most other self-supervised approaches ) nor it introduces any kind of asymmetry in the network architecture ( like state-of-the-art BGRL ) . Moreover , our architecture is converges substantially faster than all other state-of-the-art methods . ( II ) We evaluate our framework in node classification tasks : ( 1 ) for 5 smaller benchmark datasets in a transductive setting , ( 2 ) using the ogb-arxiv dataset from the Open Graph Benchmark ( also in the transductive setting ) , ( 3 ) for multiple graphs in the inductive setting using the PPI ( Protein-Protein Interaction ) dataset , and finally ( 4 ) for the large-scale graph dataset ogb-products in the inductive setting . We use both GCN-based encoders as well as a GAT-based one . We observe that our method achieves analogous results compared to state-of-the-art methods . ( III ) We ensure reproducibility by making the code of both our models as well as experimental pipeline available ( currently attached in the supplementary materials ) . 2 RELATED WORKS . Self-supervised learning The idea of self-supervised learning ( SSL ) has a long history . Introduced in the early work of Schmidhuber ( Schmidhuber , 1990 ) has more than 30 years of exploration and research now . Recently self-supervised learning was again rediscovered and found a broad interest , especially in computer vision and natural language processing . One of the most prominent SSL methods for image representation learning , Bootstrap Your Own Latent , BYOL ( Grill et al. , 2020 ) , performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks . It relies on two neural networks that interact and learn from each other . From an augmented view of an image , it trains the first one to predict the target network representation of the same image under a different view . At the same time , it updates the second network with a slow-moving average of the first network . Another approach to image representation SSL implements simple siamese networks , namely SimSiam ( Chen & He , 2020 ) . It achieves comparative results while not demanding negative samples , large batches , nor momentum encoders . Authors emphasize collapsing solutions for the loss and structure but show how a stop-gradient operation plays an essential role in preventing it . Recent method , Barlow Twins ( Zbontar et al. , 2021 ) , advances the SSL field with a new objective function that naturally avoids collapses by measuring the cross-correlation matrix between the outputs of two twin , identical networks fed with distorted versions of a sample , and makes it as close to the identity matrix as possible . Representations of distorted versions of samples are then expected to be similar , reducing the redundancy between them . What differentiates the method is that it does not require large batches or asymmetry between the network twins . It outperforms previous methods on ImageNet for semi-supervised classification . Graph representation learning Learning the representation also spreads to other domains . The graph embedding problem has also attracted much attention from the research community worldwide in recent years . Plenty of methods have been developed , each focused on a different aspect of network embeddings , such as proximity , structure , attributes , learning paradigm or scalability . There exist plenty of shallow methods , among others DeepWalk ( Perozzi et al. , 2014 ) , Node2vec ( Grover & Leskovec , 2016 ) or LINE ( Tang et al. , 2015 ) , that use a simple notion of graph coding through random walks or on encoder-decoder objectives that optimize first and second-order node similarity . More complex graph neural networks , such as GCN ( Kipf & Welling , 2017 ) or GraphSAGE ( Hamilton et al. , 2017 ) implements the basic encoder algorithm with various neighborhood aggregation . Following the extension , graph attention network GAT ( Veličković et al. , 2018 ) leverages masked self-attentional layers to address the shortcomings of graph convolutions and their troublesome approximations . Self-supervised graph representation learning Inspired by the success of contrastive methods in vision and NLP , the procedures were also adapted to graphs . Early DGI ( Veličković et al. , 2019 ) employs GNN to learn node embeddings and obtains the graph embedding via a readout function and maximizes the mutual information between node embeddings and the graph embedding by discriminating nodes in the original graph from nodes in a corrupted graph . GCA ( Zhu et al. , 2020b ) studied various augmentation procedures . GRACE ( Zhu et al. , 2020a ) creates two augmented versions of a graph , pulls together the representation of the same node in both graphs , and pushes apart representations of every other node . Recent GraphCL ( You et al. , 2020 ) method is another example of representative approach using contrastive learning . All the previous methods use negative sampling approaches for the embedding optimization , yet such setting has a high complexity . To overcome this , BGRL ( Thakoor et al. , 2021 ) proposed to use an approach that does not rely on negative samples . It uses two kinds of encoder networks ( online and target ) , introducing a nonintuitive asymmetric pipeline architecture , but provides state-of-the-art SSL results . Moreover , it relies on several techniques to prevent trivial solutions ( gradient stopping , momentum encoder ) . A concurrent approach to BGRL is DGB ( Che et al. , 2020 ) . 3 PROPOSED FRAMEWORK . Motivated by the emerging self-supervised learning paradigm and its recent applications in graph representation learning ( BGRL ( Thakoor et al. , 2021 ) ) , we propose Graph Barlow Twins – a framework that builds node embeddings using a symmetric network architecture and an empirical cross-correlation based loss function . The overall pipeline of our framework is shown in Figure 1 . The consecutive processing steps can be described as follows : Graph data We represent a graph G with nodes V and edges E as the tuple : ( X , A ) , where X ∈ R|V|×k is the node feature matrix and k is the feature dimensionality ; A ∈ { 0 , 1 } |V|×|V| is the adjacency matrix , such that Ai , j = 1 iff ( i , j ) ∈ E . In the general case , a graph could also have associated edge features or graph level features , but for simplicity we omit those here . Nevertheless , these could also be used in our framework , as long as the encoder can make use of such features . Generating graph views via augmentation Following other works ( Thakoor et al. , 2021 ; Zhu et al. , 2020b ; You et al. , 2020 ; Zhu et al. , 2020a ) , we select two kinds of augmentations – edge dropping and node feature masking – and generate two views of the input graph G ( 1 ) and G ( 2 ) . In the edge dropping case , we remove edges according to a generated mask of size |E| ( number of edges in the graph ) with elements sampled from the Bernoulli distribution B ( 1− pA ) . When it comes to masking node features , we employ a similar scheme and generate a mask of size k also sampled from the Bernoulli distribution B ( 1 − pX ) . Note that we mask node features at the scale of the whole graph , i.e . the same features are masked for each node . Other works apply different augmentation parameters pX , pA for each generated view , but as our framework is fully symmetrical , we postulate that it is enough to use the same parameters to generate both augmentations ( see Section 4.4 ) . Encoder network The main component of the proposed framework is the encoder network fθ : G → R|V|×d . It takes an augmented graph as the input and computes ( in our case ) a ddimensional representation vector for each node in the graph . Note that we do not specify any particular encoder network and one may use even encoders that construct embeddings for edges or whole graphs . In our experiments , we will show the application of GCN ( Kipf & Welling , 2017 ) and GAT ( Veličković et al. , 2018 ) based encoder networks . Both augmented graph views G ( 1 ) , G ( 2 ) are passed through the same encoder , resulting in two embedding matrices Z ( 1 ) and Z ( 2 ) , respectively . The original Barlow Twins method specified also a projector network ( implemented as an MLP ) to reduce high embedding dimensionality ( of the ResNet encoder ) . Our approach eliminates that step as it uses GNNs with low dimensional embeddings . Loss function In our work , we propose to use a negative-sample-free loss function to train the encoder network . We first normalize the embedding matricesZ ( 1 ) andZ ( 2 ) along the batch dimension ( a mean of zero and a standard deviation equal to one ) , and then we compute the empirical crosscorrelation matrix C ∈ Rd×d : Cij = ∑ b Z ( 1 ) b , i Z ( 2 ) b , j√∑ b ( Z ( 1 ) b , i ) 2 √∑ b ( Z ( 2 ) b , j ) 2 , ( 1 ) where b are the batch indexes and i , j are the indexes of embeddings . Such setting was originally proposed under the name Barlow Twins . Neuroscientist H. Barlow ’ s redundancy-reduction principle has motivated many methods both in supervised and unsupervised learning ( Deco & Parra , 1997 ; Schmidhuber et al. , 1996 ; Ballé et al. , 2017 ) . Recently , Zbontar et al . ( 2021 ) has employed this principle to build a self-supervised image representation learning algorithm ( we bring this idea to the domain of graph-structured data ) . The cross-correlation matrix C is optimized by the Barlow Twins loss function LBT ( see Equation 2 ) to be equal to the identity matrix . The loss is composed of two parts : ( I ) the invariance term and ( II ) the redundancy reduction term . The first one forces the on diagonal elements Cii to be equal to one , hence making the embeddings invariant to the applied augmentations . The second term optimizes the off-diagonal elements Cij to be equal to zero – this results in decorrelated components of the embedding vectors . LBT = ∑ i ( 1− Cii ) 2 + λ ∑ i ∑ j 6=i Cij2 ( 2 ) The λ > 0 parameter defines the trade-off between the invariance and redundancy reduction terms when optimizing the overall loss function . In Tsai et al . ( 2021 ) , the authors proposed to use λ = 1d , which we employ in our experimental setting . Otherwise , one can perform a simple grid search to find the best λ value in a particular experiment . Please note , that in such setting the gradient is symmetrically backpropagated through the encoder network . We do not rely on any special techniques , like momentum encoders , gradient stopping , or predictor networks . In preliminary experiments , we also investigated the Hilbert-Schmidt Independence Criterion ( due to its relation to the Barlow Twins objective ( Tsai et al. , 2021 ) ) , but we did not observe any performance gain .

Summary: This paper proposed a self-supervised learning framework for graph representation learning based on a cross-correlation-based loss function. In the proposed framework, two views of the input graph obtained by augmentation methods are passed through the same encoder to compute two embedding matrices, then Barlow Twins loss is used to compute the loss according to the embedding matrices. The main contribution of this paper lies in that it adapted Barlow Twins from vision to graph representation learning field and evaluated the performance of this self-supervised framework in multiple node classification tasks. The proposed method achieved analogous results compared to SOTA methods with lower time and space complexity.