Learning Parametrised Graph Shift Operators

1 INTRODUCTION . Real-world data and applications often involve significant structural complexity and as a consequence graph representation learning attracts great research interest ( Hamilton et al. , 2017b ; Wu et al. , 2020 ) . The topology of the observations plays a central role when performing machine learning tasks on graph structured data . A variety of supervised , semi-supervised or unsupervised graph learning algorithms employ different forms of operators that encode the topology of these observations . The most commonly used operators are the adjacency matrix , the Laplacian matrix and their normalised variants . All of these matrices belong to a general set of linear operators , the Graph Shift Operators ( GSOs ) ( Sandryhaila & Moura , 2013 ; Mateos et al. , 2019 ) . Graph Neural Networks ( GNNs ) , the main application domain in this paper , are representative cases of algorithms that use chosen GSOs to encode the graph structure , i.e. , to encode neighbourhoods used in the aggregation operators . Several GNN models ( Kipf & Welling , 2017 ; Hamilton et al. , 2017a ; Xu et al. , 2019 ) choose different variants of normalised adjacency matrices as GSOs . Interestingly , in a variety of tasks and datasets , the incorporation of explicit structural information of neighbourhoods into the model is found to improve results ( Pei et al. , 2020 ; Zhang & Chen , 2018 ; You et al. , 2019 ) , leading us to conclude that the chosen GSO is not entirely capturing the information of the data topology . In most of these approaches , the GSO is chosen without an analysis of the impact of this choice of representation . From this observation arise our two research questions . Question 1 : Is there a single optimal representation to encode graph structures or is the optimal representation task- and data-dependent ? On different tasks and datasets , the choice between the different representations encoded by the different graph shift operator matrices has shown to be a consequential decision . Due to the past ∗Equal contribution . successful approaches that use different GSOs for different tasks and datasets , it is natural to assume that there is no single optimal representation for all scenarios . Finding an optimal representation of network data could contribute positively to a range of learning tasks such as node and graph classification or community detection . Fundamental to this search is an answer to Question 1 . In addition , we pose the following second research question . Question 2 : Can we learn such an optimal representation to encode graph structure in a numerically stable and computationally efficient way ? The utilisation of a GSO as a topology representation is currently a hand-engineered choice of normalised variants of the adjacency matrix . Thus , the learnable representation of node interactions is transferred into either convolutional filters ( Kipf & Welling , 2017 ; Hamilton et al. , 2017a ) or attention weights ( Veličković et al. , 2018 ) , keeping the used GSO constant . In this work , we suggest a parametrisation of the GSO . Specific parameter values in our proposed parametrised ( and differentiable ) GSO result in the most commonly used GSOs , namely the adjacency , unnormalised Laplacian and both normalised Laplacian matrices , and GNN aggregation functions , e.g. , the averaging and summation message passing operations . The beauty of this innovation is that it can be seamlessly included in both message passing and convolutional GNN architectures . Optimising the operator parameters will allow us to find answers to our two research questions . The remainder of this paper is organised as follows . In Section 2 , we give an overview of related work in the literature . Then in Section 3 , we define our parametrised graph shift operator ( PGSO ) and discuss how it can be incorporated into many state-of-the-art GNN architectures . This is followed by a spectral analysis of our PGSO in Section 4 , where we observe good numerical stability in practice . In Section 5 , we analyse the performance of GNN architectures augmented by the PGSO in a node classification task on a set of stochastic blockmodel graphs with varying sparsity and on learning tasks performed on several real-world datasets . 2 RELATED WORK . GSOs emerge in different research fields such as in physics , network science , computer science and mathematics , taking usually the form of either graph Laplacian normalisations or variants of the adjacency matrix . In an abundant number of machine learning applications the expressivity of GSOs is exploited , e.g. , in unsupervised learning ( von Luxburg , 2007 ; Kim et al. , 2008 ) , semisupervised node classification on graph-structured data ( Kipf & Welling , 2017 ; Schlichtkrull et al. , 2018 ) and supervised learning on computer vision tasks ( Chang & Yeung , 2006 ) . The majority of these works assumes a specified normalised version of the Laplacian that encodes the structural information of the problem and usually these versions differ depending on the analysed dataset and the end-user task . Recently , new findings on the impact of the chosen Laplacian representation have emerged that highlight the contribution of Laplacian regularisation ( Dall ’ Amico et al. , 2020 ; Saade et al. , 2014 ; Dall ’ Amico et al. , 2019 ) . The different GSO choices in different tasks indicate a data-dependent relation between the structure of the data and its optimal GSO representation . This observation motivates us to investigate how beneficial a well-chosen GSO can be for a learning task on structured data . GNNs use a variety of GSOs to encode neighbourhood topologies , either normalisations of the adjacency matrix ( Xu et al. , 2019 ; Hamilton et al. , 2017a ) or normalisations of the graph Laplacian ( Kipf & Welling , 2017 ; Wu et al. , 2019 ) . Due to the efficiency and the predictive performance of GNNs , a research interest has recently emerged in their expressive power . One of the examined aspects is that of the equivalence of the GNNs ’ expressive power with that of the Weisfeiler-Lehman graph isomorphism test ( Dasoulas et al. , 2020 ; Maron et al. , 2019 ; Morris et al. , 2019 ; Xu et al. , 2019 ) . Another research direction is that of analysing the depth and the width of GNNs , moving one step forward to the design of deep GNNs ( Loukas , 2020 ; Li et al. , 2018 ; Liu et al. , 2020 ; Alon & Yahav , 2020 ) . In this analysis , the authors study phenomena of Laplacian oversmoothing and combinatorial oversquashing , that harm the expressiveness of GNNs . In most of these approaches , however , the used GSO is fixed without a motivation of the choice . We hope that the parametrised GSO that is presented in this work can contribute positively to the expressivity analysis of GNNs . We will now delineate our approach and that of a closely related work by Klicpera et al . ( 2019 ) . Klicpera et al . ( 2019 ) demonstrate that varying the choice of the GSO in the message passing step of GNNs can lead to significant performance gains . In Klicpera et al . ( 2019 ) two fixed diffusion operators with a much larger receptive field than the 1-hop neighbourhood convolutions , are inserted into the architectures , leading to a significant improvement of the GNNs ’ performance . In our work here we replace the GSOs in GNN frameworks with the PGSO , which has a receptive field equal to the 1-hop neighbourhood of the nodes . We find that parameter values of our PGSO can be trained in a numerically stable fashion , which allows us to chose a parametric form unifying the most common GSOs and aggregation functions . As with standard GNN architectures the receptive field of the convolutions is increased in our architectures by stacking additional layers . Klicpera et al . ( 2019 ) increase the size of the receptive field and keep the neighbourhood representation fixed , while we keep the size of the receptive field fixed and learn the neighbourhood representation . 3 PARAMETRISED GRAPH SHIFT OPERATORS . We define notation and fundamental concepts in Section 3.1 and introduce our proposed parametrised graph shift operator γ ( A , S ) in Section 3.2 . In Section 3.3 , we provide a detailed discussion of how γ ( A , S ) can be applied to the message-passing operation in GNNs and we demonstrate use cases of the incorporation of γ ( A , S ) in different GNN architectures . 3.1 PRELIMINARIES . Let a graph G be a tuple , G = ( V , E ) , where V and E are the sets of nodes and edges and let |V | = n. We assume the graph G to be attributed with attribute matrix X ∈ Rn×d , where the ith row of X contains the d-dimensional attribute vector corresponding to node vi . We denote the n×n identity matrix by In and the n-dimensional column vector of all ones by 1n . Given the node and edge sets , one can define the adjacency matrix , denoted A ∈ [ 0 , 1 ] n×n , where Aij 6= 0 if and only if ( i , j ) ∈ E , and the degree matrix of A as D = Diag ( A1n ) . Recently the notion of a GSO has been defined as a general family of operators which enable the propagation of signals over graph structures ( Sandryhaila & Moura , 2013 ; Shuman et al. , 2013 ) . Definition 1 . Graph Shift Operator A matrix S ∈ Rn×n is called a Graph Shift Operator ( GSO ) if it satisfies Sij = 0 for i 6= j and ( i , j ) /∈ E ( Mateos et al. , 2019 ; Gama et al. , 2020 ) . This general definition includes the adjacency and Laplacian matrices as instances of its class . Remark 1 . According to Definition 1 , the existence of an edge ( i , j ) ∈ E does not imply a nonzero entry in the GSO , Sij 6= 0 . Hence , the correspondence between a GSO and a graph is not bijective in general . 3.2 PARAMETRISED GSO . We begin by defining our parametrised graph shift operator . Definition 2 . We define the parametrised graph shift operator ( PGSO ) , denoted by γ ( A , S ) , as γ ( A , S ) = m1De1a +m2De2a AaDe3a +m3In , ( 1 ) where Aa = A+ aIn and Da = Diag ( Aa1n ) is the degree matrix of Aa . We denote the parameter tuple corresponding the γ ( A , S ) by S = ( m1 , m2 , m3 , e1 , e2 , e3 , a ) consisting of scalar multiplicative parameters m1 , m2 , m3 , scalar exponential parameters e1 , e2 , e3 and an additive parameter a . The main motivation of the parametrised form in Equation ( 1 ) is to span the space of commonly used GSOs and indeed we are able to generate a wide range of different graph shift operators by choosing different values for the parameter set S. In Table 1 , we give examples of parameter values in γ ( A , S ) which result in the most commonly used GSOs and message-passing operators in GNNs . Unlike the GSO , the PGSO uniquely identifies the graph it corresponds to , i.e. , the GSO and PGSO do not share the property discussed in Remark 1 . Although we base the definition of γ ( A , S ) on the adjacency matrix , we can define the PGSO using other graph representation matrices , such as the non-backtracking operator B , γ ( B , S ) , ( Krzakala et al. , 2013 ; Bordenave et al. , 2015 ) or the diffusion matrix S , γ ( S , S ) ( Klicpera et al. , 2019 ) . 3.3 SUGGESTED METHOD : GNN-PGSO AND GNN-mPGSO Next , we formally discuss how γ ( A , S ) is incorporated in GNN models . Let a GNN model be denoted by M ( φ ( A ) , X ) , taking as input a non-parametrised function of the adjacency matrix φ ( A ) : [ 0 , 1 ] n×n → Rn×n and an attribute matrix ( in case of an attributed graph ) X ∈ Rn×d . Further , let K denote the number of aggregation layers thatM consists of . The Parametrised Graph Shift Operator ( PGSO ) formulation transforms the GNN modelM ( φ ( A ) , X ) into the GNN-PGSO model M′ ( γ ( A , S ) , X ) . Moreover , we define the GNN-mPGSO model M′′ ( γ [ K ] ( A , S [ K ] ) , X ) , where γ [ K ] ( A , S [ K ] ) = [ γ ( A , S1 ) , ... , γ ( A , SK ) ] , i.e. , we assign each GNN layer a different parameter tuple Sl for l ∈ { 1 , . . . , K } . Message-passing steps and convolutions In a spectral-based GNN ( Wu et al. , 2020 ) , where the GSO is explicitly multiplied by the model parameters , it is straightforward to replace the GSO with γ ( A , S ) . However , with some further analysis γ ( A , S ) can also be incorporated in spatial-based GNNs , where the node update equation is defined as a message-passing step . Here we illustrate the required analysis on the sum-based aggregation operator , where we sum the feature vectors hj ∈ Rd in the neighborhood of a given node vi , denoted N ( vi ) . The sum operator of the neighborhood representations can be reexpressed as : ∑ j : vj∈N ( vi ) hj = ∑n j=1Aijhj . Using this observation we can derive the application of γ ( A , S ) in a message-passing step to be , ( γ ( A , S ) h ) i = m1 ( Da ) e1i hi +m2 n∑ j=1 ( Da ) e2 i ( Aa ) ij ( Da ) e3 j hj +m3hi . ( 2 ) Examples The following examples highlight the usage of the γ ( A , S ) operator : 1 . In the standard GCN ( Kipf & Welling , 2017 ) the propagation rule of the node representation H ( l ) ∈ Rn×d in a computation layer l is , H ( l+1 ) = σ ( D − 12 1 A1D − 12 1 H ( l ) W ( l ) ) , where W ( l ) is the trainable weight matrix and σ denotes a non-linear activation function . The GCN-PGSO and GCN-mPGSO models , respectively , perform the following propagation rules , H ( l+1 ) = σ ( γ ( A , S ) H ( l ) W ( l ) ) and H ( l+1 ) = σ ( γ ( A , Sl ) H ( l ) W ( l ) ) . 2 . The Graph Isomorphism Network ( GIN ) ( Xu et al. , 2019 ) consists of the following propagation rule for a node representation h ( l ) i ∈ Rd of node vi in the computation layer l , h ( l+1 ) i = σ ( h ( l ) i W ( l ) + ∑ j : vj∈N ( vi ) h ( l ) j W ( l ) ) . Using the Equation ( 2 ) the propagation rule is transformed into the GIN-PGSO formulation as , h ( l+1 ) i = σ ( ( m1 ( Da ) e1 i +m3 ) h ( l ) i W ( l ) +m2 ∑ j : vj∈N ( vi ) ijh ( l ) j W ( l ) ) , where uv are edge weights defined as ij = ( Da ) e2 i ( Da ) e3 j . Analogously , we can construct the GIN-mPGSO formulation by superscripting every parameter in S by ( l ) . Computational Cost Since in ( 1 ) the exponential parameters are applied only to diagonal matrices the PGSO and mPGSO are efficiently computable and optimisable . γ ( A , S ) can be extended by using vector instead of scalar parameters . Although this extension leads to better expressivity , the computational cost is increased , as the number of parameters then depends on the size of the graph . 4 SPECTRAL ANALYSIS OF γ ( A , S ) In this section , we study spectral properties of γ ( A , S ) in theoretical analysis in Section 4.1 and through empirical observation in Section 4.2 . The obtained theoretical results provide a foundation for further analysis of methodology involving the PGSO and allow an efficient observation of spectral support bounds of commonly used GSOs , that are instances of γ ( A , S ) .

Summary: The paper proposes a parametric form for a matrix representation of a graph to be used as a building block within graph neural networks (GNNs). In essence, people use different normalized versions of the adjacency and Laplacian matrices within GNNs. The authors, in turn, propose a generic parametrized version that encompasses those normalizations and that can be learned from data.