Learning Parametrised Graph Shift Operators

1 INTRODUCTION . Real-world data and applications often involve significant structural complexity and as a consequence graph representation learning attracts great research interest ( Hamilton et al. , 2017b ; Wu et al. , 2020 ) . The topology of the observations plays a central role when performing machine learning tasks on graph structured data . A variety of supervised , semi-supervised or unsupervised graph learning algorithms employ different forms of operators that encode the topology of these observations . The most commonly used operators are the adjacency matrix , the Laplacian matrix and their normalised variants . All of these matrices belong to a general set of linear operators , the Graph Shift Operators ( GSOs ) ( Sandryhaila & Moura , 2013 ; Mateos et al. , 2019 ) . Graph Neural Networks ( GNNs ) , the main application domain in this paper , are representative cases of algorithms that use chosen GSOs to encode the graph structure , i.e. , to encode neighbourhoods used in the aggregation operators . Several GNN models ( Kipf & Welling , 2017 ; Hamilton et al. , 2017a ; Xu et al. , 2019 ) choose different variants of normalised adjacency matrices as GSOs . Interestingly , in a variety of tasks and datasets , the incorporation of explicit structural information of neighbourhoods into the model is found to improve results ( Pei et al. , 2020 ; Zhang & Chen , 2018 ; You et al. , 2019 ) , leading us to conclude that the chosen GSO is not entirely capturing the information of the data topology . In most of these approaches , the GSO is chosen without an analysis of the impact of this choice of representation . From this observation arise our two research questions . Question 1 : Is there a single optimal representation to encode graph structures or is the optimal representation task- and data-dependent ? On different tasks and datasets , the choice between the different representations encoded by the different graph shift operator matrices has shown to be a consequential decision . Due to the past ∗Equal contribution . successful approaches that use different GSOs for different tasks and datasets , it is natural to assume that there is no single optimal representation for all scenarios . Finding an optimal representation of network data could contribute positively to a range of learning tasks such as node and graph classification or community detection . Fundamental to this search is an answer to Question 1 . In addition , we pose the following second research question . Question 2 : Can we learn such an optimal representation to encode graph structure in a numerically stable and computationally efficient way ? The utilisation of a GSO as a topology representation is currently a hand-engineered choice of normalised variants of the adjacency matrix . Thus , the learnable representation of node interactions is transferred into either convolutional filters ( Kipf & Welling , 2017 ; Hamilton et al. , 2017a ) or attention weights ( Veličković et al. , 2018 ) , keeping the used GSO constant . In this work , we suggest a parametrisation of the GSO . Specific parameter values in our proposed parametrised ( and differentiable ) GSO result in the most commonly used GSOs , namely the adjacency , unnormalised Laplacian and both normalised Laplacian matrices , and GNN aggregation functions , e.g. , the averaging and summation message passing operations . The beauty of this innovation is that it can be seamlessly included in both message passing and convolutional GNN architectures . Optimising the operator parameters will allow us to find answers to our two research questions . The remainder of this paper is organised as follows . In Section 2 , we give an overview of related work in the literature . Then in Section 3 , we define our parametrised graph shift operator ( PGSO ) and discuss how it can be incorporated into many state-of-the-art GNN architectures . This is followed by a spectral analysis of our PGSO in Section 4 , where we observe good numerical stability in practice . In Section 5 , we analyse the performance of GNN architectures augmented by the PGSO in a node classification task on a set of stochastic blockmodel graphs with varying sparsity and on learning tasks performed on several real-world datasets . 2 RELATED WORK . GSOs emerge in different research fields such as in physics , network science , computer science and mathematics , taking usually the form of either graph Laplacian normalisations or variants of the adjacency matrix . In an abundant number of machine learning applications the expressivity of GSOs is exploited , e.g. , in unsupervised learning ( von Luxburg , 2007 ; Kim et al. , 2008 ) , semisupervised node classification on graph-structured data ( Kipf & Welling , 2017 ; Schlichtkrull et al. , 2018 ) and supervised learning on computer vision tasks ( Chang & Yeung , 2006 ) . The majority of these works assumes a specified normalised version of the Laplacian that encodes the structural information of the problem and usually these versions differ depending on the analysed dataset and the end-user task . Recently , new findings on the impact of the chosen Laplacian representation have emerged that highlight the contribution of Laplacian regularisation ( Dall ’ Amico et al. , 2020 ; Saade et al. , 2014 ; Dall ’ Amico et al. , 2019 ) . The different GSO choices in different tasks indicate a data-dependent relation between the structure of the data and its optimal GSO representation . This observation motivates us to investigate how beneficial a well-chosen GSO can be for a learning task on structured data . GNNs use a variety of GSOs to encode neighbourhood topologies , either normalisations of the adjacency matrix ( Xu et al. , 2019 ; Hamilton et al. , 2017a ) or normalisations of the graph Laplacian ( Kipf & Welling , 2017 ; Wu et al. , 2019 ) . Due to the efficiency and the predictive performance of GNNs , a research interest has recently emerged in their expressive power . One of the examined aspects is that of the equivalence of the GNNs ’ expressive power with that of the Weisfeiler-Lehman graph isomorphism test ( Dasoulas et al. , 2020 ; Maron et al. , 2019 ; Morris et al. , 2019 ; Xu et al. , 2019 ) . Another research direction is that of analysing the depth and the width of GNNs , moving one step forward to the design of deep GNNs ( Loukas , 2020 ; Li et al. , 2018 ; Liu et al. , 2020 ; Alon & Yahav , 2020 ) . In this analysis , the authors study phenomena of Laplacian oversmoothing and combinatorial oversquashing , that harm the expressiveness of GNNs . In most of these approaches , however , the used GSO is fixed without a motivation of the choice . We hope that the parametrised GSO that is presented in this work can contribute positively to the expressivity analysis of GNNs . We will now delineate our approach and that of a closely related work by Klicpera et al . ( 2019 ) . Klicpera et al . ( 2019 ) demonstrate that varying the choice of the GSO in the message passing step of GNNs can lead to significant performance gains . In Klicpera et al . ( 2019 ) two fixed diffusion operators with a much larger receptive field than the 1-hop neighbourhood convolutions , are inserted into the architectures , leading to a significant improvement of the GNNs ’ performance . In our work here we replace the GSOs in GNN frameworks with the PGSO , which has a receptive field equal to the 1-hop neighbourhood of the nodes . We find that parameter values of our PGSO can be trained in a numerically stable fashion , which allows us to chose a parametric form unifying the most common GSOs and aggregation functions . As with standard GNN architectures the receptive field of the convolutions is increased in our architectures by stacking additional layers . Klicpera et al . ( 2019 ) increase the size of the receptive field and keep the neighbourhood representation fixed , while we keep the size of the receptive field fixed and learn the neighbourhood representation . 3 PARAMETRISED GRAPH SHIFT OPERATORS . We define notation and fundamental concepts in Section 3.1 and introduce our proposed parametrised graph shift operator γ ( A , S ) in Section 3.2 . In Section 3.3 , we provide a detailed discussion of how γ ( A , S ) can be applied to the message-passing operation in GNNs and we demonstrate use cases of the incorporation of γ ( A , S ) in different GNN architectures . 3.1 PRELIMINARIES . Let a graph G be a tuple , G = ( V , E ) , where V and E are the sets of nodes and edges and let |V | = n. We assume the graph G to be attributed with attribute matrix X ∈ Rn×d , where the ith row of X contains the d-dimensional attribute vector corresponding to node vi . We denote the n×n identity matrix by In and the n-dimensional column vector of all ones by 1n . Given the node and edge sets , one can define the adjacency matrix , denoted A ∈ [ 0 , 1 ] n×n , where Aij 6= 0 if and only if ( i , j ) ∈ E , and the degree matrix of A as D = Diag ( A1n ) . Recently the notion of a GSO has been defined as a general family of operators which enable the propagation of signals over graph structures ( Sandryhaila & Moura , 2013 ; Shuman et al. , 2013 ) . Definition 1 . Graph Shift Operator A matrix S ∈ Rn×n is called a Graph Shift Operator ( GSO ) if it satisfies Sij = 0 for i 6= j and ( i , j ) /∈ E ( Mateos et al. , 2019 ; Gama et al. , 2020 ) . This general definition includes the adjacency and Laplacian matrices as instances of its class . Remark 1 . According to Definition 1 , the existence of an edge ( i , j ) ∈ E does not imply a nonzero entry in the GSO , Sij 6= 0 . Hence , the correspondence between a GSO and a graph is not bijective in general . 3.2 PARAMETRISED GSO . We begin by defining our parametrised graph shift operator . Definition 2 . We define the parametrised graph shift operator ( PGSO ) , denoted by γ ( A , S ) , as γ ( A , S ) = m1De1a +m2De2a AaDe3a +m3In , ( 1 ) where Aa = A+ aIn and Da = Diag ( Aa1n ) is the degree matrix of Aa . We denote the parameter tuple corresponding the γ ( A , S ) by S = ( m1 , m2 , m3 , e1 , e2 , e3 , a ) consisting of scalar multiplicative parameters m1 , m2 , m3 , scalar exponential parameters e1 , e2 , e3 and an additive parameter a . The main motivation of the parametrised form in Equation ( 1 ) is to span the space of commonly used GSOs and indeed we are able to generate a wide range of different graph shift operators by choosing different values for the parameter set S. In Table 1 , we give examples of parameter values in γ ( A , S ) which result in the most commonly used GSOs and message-passing operators in GNNs . Unlike the GSO , the PGSO uniquely identifies the graph it corresponds to , i.e. , the GSO and PGSO do not share the property discussed in Remark 1 . Although we base the definition of γ ( A , S ) on the adjacency matrix , we can define the PGSO using other graph representation matrices , such as the non-backtracking operator B , γ ( B , S ) , ( Krzakala et al. , 2013 ; Bordenave et al. , 2015 ) or the diffusion matrix S , γ ( S , S ) ( Klicpera et al. , 2019 ) . 3.3 SUGGESTED METHOD : GNN-PGSO AND GNN-mPGSO Next , we formally discuss how γ ( A , S ) is incorporated in GNN models . Let a GNN model be denoted by M ( φ ( A ) , X ) , taking as input a non-parametrised function of the adjacency matrix φ ( A ) : [ 0 , 1 ] n×n → Rn×n and an attribute matrix ( in case of an attributed graph ) X ∈ Rn×d . Further , let K denote the number of aggregation layers thatM consists of . The Parametrised Graph Shift Operator ( PGSO ) formulation transforms the GNN modelM ( φ ( A ) , X ) into the GNN-PGSO model M′ ( γ ( A , S ) , X ) . Moreover , we define the GNN-mPGSO model M′′ ( γ [ K ] ( A , S [ K ] ) , X ) , where γ [ K ] ( A , S [ K ] ) = [ γ ( A , S1 ) , ... , γ ( A , SK ) ] , i.e. , we assign each GNN layer a different parameter tuple Sl for l ∈ { 1 , . . . , K } . Message-passing steps and convolutions In a spectral-based GNN ( Wu et al. , 2020 ) , where the GSO is explicitly multiplied by the model parameters , it is straightforward to replace the GSO with γ ( A , S ) . However , with some further analysis γ ( A , S ) can also be incorporated in spatial-based GNNs , where the node update equation is defined as a message-passing step . Here we illustrate the required analysis on the sum-based aggregation operator , where we sum the feature vectors hj ∈ Rd in the neighborhood of a given node vi , denoted N ( vi ) . The sum operator of the neighborhood representations can be reexpressed as : ∑ j : vj∈N ( vi ) hj = ∑n j=1Aijhj . Using this observation we can derive the application of γ ( A , S ) in a message-passing step to be , ( γ ( A , S ) h ) i = m1 ( Da ) e1i hi +m2 n∑ j=1 ( Da ) e2 i ( Aa ) ij ( Da ) e3 j hj +m3hi . ( 2 ) Examples The following examples highlight the usage of the γ ( A , S ) operator : 1 . In the standard GCN ( Kipf & Welling , 2017 ) the propagation rule of the node representation H ( l ) ∈ Rn×d in a computation layer l is , H ( l+1 ) = σ ( D − 12 1 A1D − 12 1 H ( l ) W ( l ) ) , where W ( l ) is the trainable weight matrix and σ denotes a non-linear activation function . The GCN-PGSO and GCN-mPGSO models , respectively , perform the following propagation rules , H ( l+1 ) = σ ( γ ( A , S ) H ( l ) W ( l ) ) and H ( l+1 ) = σ ( γ ( A , Sl ) H ( l ) W ( l ) ) . 2 . The Graph Isomorphism Network ( GIN ) ( Xu et al. , 2019 ) consists of the following propagation rule for a node representation h ( l ) i ∈ Rd of node vi in the computation layer l , h ( l+1 ) i = σ ( h ( l ) i W ( l ) + ∑ j : vj∈N ( vi ) h ( l ) j W ( l ) ) . Using the Equation ( 2 ) the propagation rule is transformed into the GIN-PGSO formulation as , h ( l+1 ) i = σ ( ( m1 ( Da ) e1 i +m3 ) h ( l ) i W ( l ) +m2 ∑ j : vj∈N ( vi ) ijh ( l ) j W ( l ) ) , where uv are edge weights defined as ij = ( Da ) e2 i ( Da ) e3 j . Analogously , we can construct the GIN-mPGSO formulation by superscripting every parameter in S by ( l ) . Computational Cost Since in ( 1 ) the exponential parameters are applied only to diagonal matrices the PGSO and mPGSO are efficiently computable and optimisable . γ ( A , S ) can be extended by using vector instead of scalar parameters . Although this extension leads to better expressivity , the computational cost is increased , as the number of parameters then depends on the size of the graph . 4 SPECTRAL ANALYSIS OF γ ( A , S ) In this section , we study spectral properties of γ ( A , S ) in theoretical analysis in Section 4.1 and through empirical observation in Section 4.2 . The obtained theoretical results provide a foundation for further analysis of methodology involving the PGSO and allow an efficient observation of spectral support bounds of commonly used GSOs , that are instances of γ ( A , S ) .

Summary: The authors consider the problem of learning a parametrized graph shift operator (or message passing operator) in the context of graph neural networks. They consider a family of GSO (that they name PGSO) based on seven scalar parameters, and show that it includes most commonly used operators such as the adjacency matrix or the laplacian. The spectral properties of the PGSO are analyzed. Finally, some empirical results are provided demonstrating the PGSO as a drop-in replacement for standard GSO in GNN architectures.