SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks

1 Introduction . Graph representation learning has grown rapidly and found applications in domains where a natural graph of the data points is available [ 4 , 25 ] . Graph neural networks ( GNNs ) [ 40 ] have been a key component to the success of the research in this area . Specifically , GNNs have shown promising results for semi-supervised classification when the available graph structure exhibits a high degree of homophily ( i.e . connected nodes often belong to the same class ) [ 57 ] . We study the applicability of GNNs to ( semi-supervised ) classification problems where a graph structure is not readily available . The existing approaches for this problem either fix a similarity graph between the nodes or learn the GNN parameters and a graph structure simultaneously ( see Related Work ) . In both cases , one main goal is to construct or learn a graph structure with a high degree of homophily with respect to the labels to aid the GNN classification . The latter approach is sometimes called latent graph learning and often results in higher predictive performance compared to the former approach ( see , e.g. , [ 12 ] ) . We identify a supervision starvation problem in latent graph learning approaches in which the edges between pairs of nodes that are far from labeled nodes receive insufficient supervision ; this results in learning poor structures away from labeled nodes and hence poor generalization . We propose a solution for this problem by adopting a multi-task learning framework in which we supplement the classification task with a self-supervised task . The self-supervised task is based on the hypothesis ∗Work was done when authors were at Borealis AI . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . that a graph structure that is suitable for predicting the node features is also suitable for predicting the node labels . It works by masking some input features ( or adding noise to them ) and training a separate GNN aiming at updating the adjacency matrix in such a way that it can recover the masked ( or noisy ) features . The task is generic and can be combined with several existing latent graph learning approaches . We develop a latent graph learning model , dubbed SLAPS , that adopts the proposed self-supervised task . We provide a comprehensive experimental study on nine datasets ( thirteen variations ) of various sizes and from various domains and perform thorough analyses to show the merit of SLAPS . Our main contributions include : 1 ) identifying a supervision starvation problem for latent graph learning , 2 ) proposing a solution for the identified problem through self-supervision , 3 ) developing SLAPS , a latent graph learning model that adopts the self-supervised solution , 4 ) providing comprehensive experimental results showing SLAPS substantially outperforms existing latent graph learning baselines from various categories on various benchmarks , and 5 ) providing an implementation for latent graph learning that scales to graphs with hundreds of thousands of nodes . 2 Related work . Existing methods that relate to this work can be grouped into the following categories . We discuss selected work from each category and refer the reader to [ 60 ] for a full survey . Similarity graph : One approach for inferring a graph structure is to select a similarity metric and set the edge weight between two nodes to be their similarity [ 39 , 44 , 3 ] . To obtain a sparse structure , one may create a kNN similarity graph , only connect pairs of nodes whose similarity surpasses some predefined threshold , or do sampling . As an example , in [ 14 ] a ( fixed ) kNN graph using the cosine similarity of the node features is created . In [ 47 ] , this idea is extended by creating a fresh graph in each layer of the GNN based on the node embedding similarities in that layer . Instead of choosing a single similarity metric , in [ 15 ] several ( potentially weak ) measures of similarity are fused . The quality of the predictions of these methods depends heavily on the choice of the similarity metric ( s ) . Fully connected graph : Another approach is to start with a fully connected graph and assign edge weights using the available meta-data or employ the GNN variants that provide weights for each edge via an attention mechanism [ 45 , 53 ] . This approach has been used in computer vision [ e.g. , 43 ] , natural language processing [ e.g. , 56 ] , and few-shot learning [ e.g. , 13 ] . The complexity of this approach grows rapidly making it applicable only to small-sized graphs . Zhang et al . [ 54 ] propose to define local neighborhoods for each node and only assume that these local neighborhoods are fully connected . Their approach relies on an initial graph structure to define the local neighborhoods . Latent graph learning : Instead of a similarity graph based on the initial features , one may use a graph generator with learnable parameters . In [ 30 ] , a fully connected graph is created based on a bilinear similarity function with learnable parameters . In [ 12 ] , a Bernoulli distribution is learned for each possible edge and graph structures are created through sampling from these distributions . In [ 49 ] , the input structure is updated to increase homophily based on the labels and model predictions . In [ 6 ] , an iterative approach is proposed that iterates over projecting the nodes to a latent space and constructing an adjacency matrix from the latent representations multiple times . A common approach in this category is to learn a projection of the nodes to a latent space where node similarities correspond to edge weights or edge probabilities . In [ 48 ] , the nodes are projected to a latent space by learning weights for each of the input features . In [ 38 , 21 , 8 ] , a multi-layer perceptron is used for projection . In [ 52 , 55 ] , a GNN is used for projection ; it uses the node features and an initial graph structure . In [ 26 ] , different graph structures are created in different layers by using separate GNN projectors , where the input to the GNN projector in a layer is the projected values and the generated graph structure from the previous layer . In our experiments , we compare with several approaches from this category . Leveraging domain knowledge : In some applications , one may leverage domain knowledge to guide the model toward learning specific structures . For example , in [ 24 ] , abstract syntax trees and regular languages are leveraged in learning graph structures of Python programs that aid reasoning for downstream tasks . In [ 23 ] , the structure learning is guided for robustness to adversarial attacks through the domain knowledge that clean adjacency matrices are often sparse and low-rank and exhibit feature smoothness along the connected nodes . Other examples in this category include [ 19 , 38 ] . In our paper , we experiment with general-purpose datasets without access to domain knowledge . Proposed method : Our model falls within the latent graph learning category . We supplement the training with a self-supervised objective to increase the amount of supervision in learning a structure . Our self-supervised task is inspired by , and similar to , the pre-training strategies for GNNs [ 17 , 18 , 22 , 51 , 58 ] ( specifically , we adopt the multi-task learning framework of You et al . [ 51 ] ) , but it differs from this line of work as we use self-supervision for learning a graph structure whereas the above methods use it to learn better ( and , in some cases , transferable ) GNN parameters . 3 Background and notation . We use lowercase letters to denote scalars , bold lowercase letters to denote vectors and bold uppercase letters to denote matrices . I represents an identity matrix . For a vector v , we represent its ith element as vi . For a matrixM , we represent the ith row asMi and the element at the ith row and jth column as Mij . For an attributed graph , we use n , m and f to represent the number of nodes , edges , and features respectively , and denote the graph as G = { V , A , X } where V = { v1 , . . . , vn } is a set of nodes , A ∈ Rn×n is an adjacency matrix withAij indicating the weight of the edge from vi to vj ( Aij = 0 implies no edge ) , andX ∈ Rn×f is a matrix whose rows correspond to node features . Graph convolutional networks ( GCNs ) [ 27 ] are a powerful variant of GNNs . For a graph G = { V , A , X } with a degree matrix D , layer l of the GCN architecture can be defined as H ( l ) = σ ( ÂH ( l−1 ) W ( l ) ) where Â represents a normalized adjacency matrix , H ( l−1 ) ∈ Rn×dl−1 represents the node representations in layer l-1 ( H ( 0 ) = X ) , W ( l ) ∈ Rdl−1×dl is a weight matrix , σ is an activation function such as ReLU [ 34 ] , and H ( l ) ∈ Rn×dl is the updated node embeddings . For undirected graphs where the adjacency is symmetric , Â = D− 1 2 ( A + I ) D− 1 2 corresponds to a row-and-column normalized adjacency with self-loops , and for directed graphs where the adjacency is not necessarily symmetric , Â =D−1 ( A+ I ) corresponds to a row normalized adjacency matrix with self-loops . Here , D is a ( diagonal ) degree matrix for ( A+ I ) defined asDii = 1 + ∑ jAij . 4 Proposed method : SLAPS . SLAPS consists of four components : 1 ) generator , 2 ) adjacency processor , 3 ) classifier , and 4 ) self-supervision . Figure 1 illustrates these components . In the next three subsections , we explain the first three components . Then , we point out a supervision starvation problem for a model based only on these components . Then we describe the self-supervision component as a solution to the supervision starvation problem and the full SLAPS model . 4.1 Generator . The generator is a function G : Rn×f → Rn×n with parameters θG which takes the node features X ∈ Rn×f as input and produces a matrix Ã ∈ Rn×n as output . We consider the following two generators and leave experimenting with more sophisticated graph generators ( e.g. , [ 50 , 32 , 31 ] ) and models with tractable adjacency computations ( e.g. , [ 7 ] ) as future work . Full parameterization ( FP ) : For this generator , θG ∈ Rn×n and the generator function is defined as Ã = GFP ( X ; θG ) = θG . That is , the generator ignores the input node features and directly optimizes the adjacency matrix . FP is similar to the generator in LDS [ 12 ] except that the generator of LDS treats each element of Ã as the parameter of a Bernoulli distribution and samples graph structures from these distributions . FP is simple and flexible for learning any adjacency matrix but adds n2 parameters which limits scalability and makes the model susceptible to overfitting . MLP-kNN : Here , θG corresponds to the weights of a multi-layer perceptron ( MLP ) and Ã = GMLP ( X ; θG ) = kNN ( MLP ( X ) ) , where MLP : Rn×f → Rn×f ′ is an MLP that produces a matrix with updated node representations X ′ ; kNN : Rn×f ′ → Rn×n produces a sparse matrix . The implementation details for the kNN operation is provided in the supplementary material . Initialization and variants of MLP-kNN : Let AkNN represent an adjacency matrix created by applying a kNN function on the initial node features . One smart initialization for θG is to initialize it in a way that the generator initially generatesAkNN ( i.e . Ã = AkNN before training starts ) . This can be trivially done for the FP generator by initializing θG toAkNN . For MLP-kNN , we consider two variants . In one , hereafter referred to simply as MLP , we keep the input dimension the same throughout the layers . In the other , hereafter referred to as MLP-D , we consider MLPs with diagonal weight matrices ( i.e. , except the main diagonal , all other parameters in the weight matrices are zero ) . For both variants , we initialize the weight matrices in θG with the identity matrix to ensure that the output of the MLP is initially the same as its input and the kNN graph created on these outputs is equivalent to AkNN ( alternatively , one may use other MLP variants but pre-train the weights to output AkNN before the main training starts. ) . MLP-D can be thought of as assigning different weights to different features and then computing node similarities .

Summary: The authors propose SLAPS, a methodology for deriving latent k-NN graphs for usage as edges of a GNN. They argue theoretically and empirically that it is important to combine self-supervised learning with graph structure learning, to avoid the prevalent edge starvation problem. They validate an approach consisting of a denoising self-supervised objective and various graph derivation strategies against relevant latent graph inference benchmarks, achieving impressive results.