Graph Barlow Twins: A self-supervised representation learning framework for graphs

1 INTRODUCTION . Graph representation learning has been intensively studied for the last few years , having proposed various architectures and layers , like GCN ( Kipf & Welling , 2017 ) , GAT ( Veličković et al. , 2018 ) , GraphSAGE ( Hamilton et al. , 2017 ) etc . A substantial part of these methods was introduced in the semi-supervised learning paradigm , which requires the existence of expensive labeled data ( e.g . node labels or whole graph labels ) . To overcome this , the research community has been exploring unsupervised learning methods for graphs . This resulted in a variety of different approaches including : shallow ones ( DeepWalk ( Perozzi et al. , 2014 ) , Node2vec ( Grover & Leskovec , 2016 ) , LINE ( Tang et al. , 2015 ) ) that ignore the feature attribute richness , focusing only on the structural graph information ; and graph neural network methods ( DGI ( Veličković et al. , 2019 ) , GAE , VGAE ( Kipf & Welling , 2016 ) ) that build representations upon node or graph features , achieving state-ofthe-art performance in those days . Recently self-supervised paradigm is the most emerging branch of unsupervised graph representation learning and gathers current interest and strenuous research effort towards better results . The most prominent methods were developed around the contrastive learning approach , such as GCA ( Zhu et al. , 2020b ) , GraphCL ( You et al. , 2020 ) , GRACE ( Zhu et al. , 2020a ) or DGI ( Veličković et al. , 2019 ) . Although contrastive methods are popular in many machine learning areas , including computer vision and natural language processing , their fundamental limitation is the need for negative samples . Consequently , the sampling procedure for negative examples highly affects the overall quality of the embeddings . In terms of images or texts , the definition of negative samples might seem not that problematic , but in the case of graphs there is no clear intuition . For instance , what is the negative counterpart for a particular node in the graph , should it be a node that is not a direct neighbor , or a node that is in a different graph component ? There are multiple options available , but the right choice strictly dependent on the downstream task . Researchers have already tackled the problem of building so-called negative-sample-free methods . In research being conducted in computer vision they obtained successful results with methods such as BYOL ( Grill et al. , 2020 ) , SimSiam ( Chen & He , 2020 ) or Barlow Twins ( Zbontar et al. , 2021 ) . These models utilize siamese network architectures with various techniques , like gradient stopping , asymmetry or batch and layer normalizations , to prevent collapsing to trivial solutions . Based on BYOL , Thakoor et al . ( 2021 ) proposed the Bootstrapped Representation Learning on Graphs ( BGRL ) framework . It utilizes two graph encoders : an online and a target one . The former one passes the embedding vectors to a predictor network , which tries to predict the embeddings from the target encoder . The loss is measured as the cosine similarity and the gradient is backpropagated only through the online network ( gradient stopping mechanism ) . The target encoder is updated using an exponential moving average of the online encoder weights . Such setup has been shown to produce graph representation vectors that achieve state-of-the-art performance in node classification using various benchmark datasets . Notwithstanding , assuming asymmetry between the network twins ( such as the predictor network , gradient stopping , and a moving average on the weight updates ) the method is conceptually complex . Employing a symmetric network architecture would seem more intuitive and reasonable , hence in this paper , we propose Graph Barlow Twins ( G-BT ) , a self-supervised graph representation learning framework , which computes the embeddings cross-correlation matrix of two distorted views of a single graph . The approach was firstly introduced in image representation learning as the Barlow Twins model ( Zbontar et al. , 2021 ) but was not able to handle graphs . The utilized network architecture is fully symmetric and does not need any special techniques to build non trivial embedding vectors . The distorted graph views are passed through the same encoder which is trained using the backpropagated gradients ( in a symmetrical manner ) . Our main contributions can be summarized as follows : ( I ) We propose a self-supervised graph representation learning framework Graph Barlow Twins . It is built upon the recently proposed Barlow Twins loss , which utilizes the embedding cross-correlation matrix of two distorted views of a graph to optimize the representation vectors . Our framework neither requires using negative samples ( opposed to most other self-supervised approaches ) nor it introduces any kind of asymmetry in the network architecture ( like state-of-the-art BGRL ) . Moreover , our architecture is converges substantially faster than all other state-of-the-art methods . ( II ) We evaluate our framework in node classification tasks : ( 1 ) for 5 smaller benchmark datasets in a transductive setting , ( 2 ) using the ogb-arxiv dataset from the Open Graph Benchmark ( also in the transductive setting ) , ( 3 ) for multiple graphs in the inductive setting using the PPI ( Protein-Protein Interaction ) dataset , and finally ( 4 ) for the large-scale graph dataset ogb-products in the inductive setting . We use both GCN-based encoders as well as a GAT-based one . We observe that our method achieves analogous results compared to state-of-the-art methods . ( III ) We ensure reproducibility by making the code of both our models as well as experimental pipeline available ( currently attached in the supplementary materials ) . 2 RELATED WORKS . Self-supervised learning The idea of self-supervised learning ( SSL ) has a long history . Introduced in the early work of Schmidhuber ( Schmidhuber , 1990 ) has more than 30 years of exploration and research now . Recently self-supervised learning was again rediscovered and found a broad interest , especially in computer vision and natural language processing . One of the most prominent SSL methods for image representation learning , Bootstrap Your Own Latent , BYOL ( Grill et al. , 2020 ) , performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks . It relies on two neural networks that interact and learn from each other . From an augmented view of an image , it trains the first one to predict the target network representation of the same image under a different view . At the same time , it updates the second network with a slow-moving average of the first network . Another approach to image representation SSL implements simple siamese networks , namely SimSiam ( Chen & He , 2020 ) . It achieves comparative results while not demanding negative samples , large batches , nor momentum encoders . Authors emphasize collapsing solutions for the loss and structure but show how a stop-gradient operation plays an essential role in preventing it . Recent method , Barlow Twins ( Zbontar et al. , 2021 ) , advances the SSL field with a new objective function that naturally avoids collapses by measuring the cross-correlation matrix between the outputs of two twin , identical networks fed with distorted versions of a sample , and makes it as close to the identity matrix as possible . Representations of distorted versions of samples are then expected to be similar , reducing the redundancy between them . What differentiates the method is that it does not require large batches or asymmetry between the network twins . It outperforms previous methods on ImageNet for semi-supervised classification . Graph representation learning Learning the representation also spreads to other domains . The graph embedding problem has also attracted much attention from the research community worldwide in recent years . Plenty of methods have been developed , each focused on a different aspect of network embeddings , such as proximity , structure , attributes , learning paradigm or scalability . There exist plenty of shallow methods , among others DeepWalk ( Perozzi et al. , 2014 ) , Node2vec ( Grover & Leskovec , 2016 ) or LINE ( Tang et al. , 2015 ) , that use a simple notion of graph coding through random walks or on encoder-decoder objectives that optimize first and second-order node similarity . More complex graph neural networks , such as GCN ( Kipf & Welling , 2017 ) or GraphSAGE ( Hamilton et al. , 2017 ) implements the basic encoder algorithm with various neighborhood aggregation . Following the extension , graph attention network GAT ( Veličković et al. , 2018 ) leverages masked self-attentional layers to address the shortcomings of graph convolutions and their troublesome approximations . Self-supervised graph representation learning Inspired by the success of contrastive methods in vision and NLP , the procedures were also adapted to graphs . Early DGI ( Veličković et al. , 2019 ) employs GNN to learn node embeddings and obtains the graph embedding via a readout function and maximizes the mutual information between node embeddings and the graph embedding by discriminating nodes in the original graph from nodes in a corrupted graph . GCA ( Zhu et al. , 2020b ) studied various augmentation procedures . GRACE ( Zhu et al. , 2020a ) creates two augmented versions of a graph , pulls together the representation of the same node in both graphs , and pushes apart representations of every other node . Recent GraphCL ( You et al. , 2020 ) method is another example of representative approach using contrastive learning . All the previous methods use negative sampling approaches for the embedding optimization , yet such setting has a high complexity . To overcome this , BGRL ( Thakoor et al. , 2021 ) proposed to use an approach that does not rely on negative samples . It uses two kinds of encoder networks ( online and target ) , introducing a nonintuitive asymmetric pipeline architecture , but provides state-of-the-art SSL results . Moreover , it relies on several techniques to prevent trivial solutions ( gradient stopping , momentum encoder ) . A concurrent approach to BGRL is DGB ( Che et al. , 2020 ) . 3 PROPOSED FRAMEWORK . Motivated by the emerging self-supervised learning paradigm and its recent applications in graph representation learning ( BGRL ( Thakoor et al. , 2021 ) ) , we propose Graph Barlow Twins – a framework that builds node embeddings using a symmetric network architecture and an empirical cross-correlation based loss function . The overall pipeline of our framework is shown in Figure 1 . The consecutive processing steps can be described as follows : Graph data We represent a graph G with nodes V and edges E as the tuple : ( X , A ) , where X ∈ R|V|×k is the node feature matrix and k is the feature dimensionality ; A ∈ { 0 , 1 } |V|×|V| is the adjacency matrix , such that Ai , j = 1 iff ( i , j ) ∈ E . In the general case , a graph could also have associated edge features or graph level features , but for simplicity we omit those here . Nevertheless , these could also be used in our framework , as long as the encoder can make use of such features . Generating graph views via augmentation Following other works ( Thakoor et al. , 2021 ; Zhu et al. , 2020b ; You et al. , 2020 ; Zhu et al. , 2020a ) , we select two kinds of augmentations – edge dropping and node feature masking – and generate two views of the input graph G ( 1 ) and G ( 2 ) . In the edge dropping case , we remove edges according to a generated mask of size |E| ( number of edges in the graph ) with elements sampled from the Bernoulli distribution B ( 1− pA ) . When it comes to masking node features , we employ a similar scheme and generate a mask of size k also sampled from the Bernoulli distribution B ( 1 − pX ) . Note that we mask node features at the scale of the whole graph , i.e . the same features are masked for each node . Other works apply different augmentation parameters pX , pA for each generated view , but as our framework is fully symmetrical , we postulate that it is enough to use the same parameters to generate both augmentations ( see Section 4.4 ) . Encoder network The main component of the proposed framework is the encoder network fθ : G → R|V|×d . It takes an augmented graph as the input and computes ( in our case ) a ddimensional representation vector for each node in the graph . Note that we do not specify any particular encoder network and one may use even encoders that construct embeddings for edges or whole graphs . In our experiments , we will show the application of GCN ( Kipf & Welling , 2017 ) and GAT ( Veličković et al. , 2018 ) based encoder networks . Both augmented graph views G ( 1 ) , G ( 2 ) are passed through the same encoder , resulting in two embedding matrices Z ( 1 ) and Z ( 2 ) , respectively . The original Barlow Twins method specified also a projector network ( implemented as an MLP ) to reduce high embedding dimensionality ( of the ResNet encoder ) . Our approach eliminates that step as it uses GNNs with low dimensional embeddings . Loss function In our work , we propose to use a negative-sample-free loss function to train the encoder network . We first normalize the embedding matricesZ ( 1 ) andZ ( 2 ) along the batch dimension ( a mean of zero and a standard deviation equal to one ) , and then we compute the empirical crosscorrelation matrix C ∈ Rd×d : Cij = ∑ b Z ( 1 ) b , i Z ( 2 ) b , j√∑ b ( Z ( 1 ) b , i ) 2 √∑ b ( Z ( 2 ) b , j ) 2 , ( 1 ) where b are the batch indexes and i , j are the indexes of embeddings . Such setting was originally proposed under the name Barlow Twins . Neuroscientist H. Barlow ’ s redundancy-reduction principle has motivated many methods both in supervised and unsupervised learning ( Deco & Parra , 1997 ; Schmidhuber et al. , 1996 ; Ballé et al. , 2017 ) . Recently , Zbontar et al . ( 2021 ) has employed this principle to build a self-supervised image representation learning algorithm ( we bring this idea to the domain of graph-structured data ) . The cross-correlation matrix C is optimized by the Barlow Twins loss function LBT ( see Equation 2 ) to be equal to the identity matrix . The loss is composed of two parts : ( I ) the invariance term and ( II ) the redundancy reduction term . The first one forces the on diagonal elements Cii to be equal to one , hence making the embeddings invariant to the applied augmentations . The second term optimizes the off-diagonal elements Cij to be equal to zero – this results in decorrelated components of the embedding vectors . LBT = ∑ i ( 1− Cii ) 2 + λ ∑ i ∑ j 6=i Cij2 ( 2 ) The λ > 0 parameter defines the trade-off between the invariance and redundancy reduction terms when optimizing the overall loss function . In Tsai et al . ( 2021 ) , the authors proposed to use λ = 1d , which we employ in our experimental setting . Otherwise , one can perform a simple grid search to find the best λ value in a particular experiment . Please note , that in such setting the gradient is symmetrically backpropagated through the encoder network . We do not rely on any special techniques , like momentum encoders , gradient stopping , or predictor networks . In preliminary experiments , we also investigated the Hilbert-Schmidt Independence Criterion ( due to its relation to the Barlow Twins objective ( Tsai et al. , 2021 ) ) , but we did not observe any performance gain .

Summary: The paper applies the recently proposed self-supervised learning method Barlow-Twins to graph structured data. For constructing the augmented version of a graph, previous methods such as edge-dropping or feature masking are used. The paper conducts experimental evaluation on datasets of various scales on both transductive and inductive setting.