Invariant and Equivariant Graph Networks

1 INTRODUCTION . We consider the problem of graph learning , namely finding a functional relation between input graphs ( more generally , hyper-graphs ) G ` and corresponding targets T ` , e.g. , labels . As graphs are common data representations , this task received quite a bit of recent attention in the machine learning community Bruna et al . ( 2013 ) ; Henaff et al . ( 2015 ) ; Monti et al . ( 2017 ) ; Ying et al . ( 2018 ) . More specifically , a ( hyper- ) graph data point G = ( V , A ) consists of a set of n nodes V , and values A attached to its hyper-edges1 . These values are encoded in a tensor A . The order of the tensor A , or equivalently , the number of indices used to represent its elements , indicates the type of data it represents , as follows : First order tensor represents node-values where Ai is the value of the i-th node ; Second order tensor represents edge-values , where Aij is the value attached to the ( i , j ) edge ; in general , k-th order tensor encodes hyper-edge-values , where Ai1 , ... , ik represents the value of the hyper-edge represented by ( i1 , . . . , ik ) . For example , it is customary to represent a graph using a binary adjacency matrix A , where Aij equals one if vertex i is connected to vertex j and zero otherwise . We denote the set of order-k tensors by Rnk . The task at hand is constructing a functional relation f ( A ` ) ≈ T ` , where f is a neural network . If T ` = t ` is a single output response then it is natural to ask that f is order invariant , namely it should produce the same output regardless of the node numbering used to encode A . For example , if we represent a graph using an adjacency matrix A = A ∈ Rn×n , then for an arbitrary permutation matrix P and an arbitrary adjacency matrix A , the function f is order invariant if it satisfies f ( P TAP ) = f ( A ) . If the targets T ` specify output response in a form of a tensor , T ` = T ` , then it is natural to ask that f is order equivariant , that is , f commutes with the renumbering of nodes operator acting on tensors . Using the above adjacency matrix example , for every adjacency matrix A and 1A hyper-edge is an ordered subset of the nodes , V every permutation matrix P , the function f is equivariant if it satisfies f ( P TAP ) = P T f ( A ) P . To define invariance and equivariance for functions acting on general tensors A ∈ Rnk we use the reordering operator : P ? A is defined to be the tensor that results from renumbering the nodes V according to the permutation defined by P . Invariance now reads as f ( P ? A ) = f ( A ) ; while equivariance means f ( P ? A ) = P ? f ( A ) . Note that the latter equivariance definition also holds for functions between different order tensors , f : Rnk → Rnl . Following the standard paradigm of neural-networks where a network f is defined by alternating compositions of linear layers and non-linear activations , we set as a goal to characterize all linear invariant and equivariant layers . The case of node-value input A = a ∈ Rn was treated in the pioneering works of Zaheer et al . ( 2017 ) ; Qi et al . ( 2017 ) . These works characterize all linear permutation invariant and equivariant operators acting on node-value ( i.e. , first order ) tensors , Rn . In particular it it shown that the linear space of invariant linear operatorsL : Rn → R is of dimension one , containing essentially only the sum operator , L ( a ) = α1Ta . The space of equivariant linear operators L : Rn → Rn is of dimension two , L ( a ) = [ αI + β ( 11T − I ) ] a . The general equivariant tensor case was partially treated in Kondor et al . ( 2018 ) where the authors make the observation that the set of standard tensor operators : product , element-wise product , summation , and contraction are all equivariant , and due to linearity the same applies to their linear combinations . However , these do not exhaust nor provide a full and complete basis for all possible tensor equivariant linear layers . In this paper we provide a full characterization of permutation invariant and equivariant linear layers for general tensor input and output data . We show that the space of invariant linear layers L : Rnk → R is of dimension b ( k ) , where b ( k ) is the k-th Bell number . The k-th Bell number is the number of possible partitions of a set of size k ; see inset for the case k = 3 . Furthermore , the space of equivariant linear layers L : Rnk → Rnl is of dimension b ( k + l ) . Remarkably , this dimension is independent of the size n of the node set V. This allows applying the same network on graphs of different sizes . For both types of layers we provide a general formula for an orthogonal basis that can be readily used to build linear invariant or equivariant layers with maximal expressive power . Going back to the example of a graph represented by an adjacency matrix A ∈ Rn×n we have k = 2 and the linear invariant layers L : Rn×n → R have dimension b ( 2 ) = 2 , while linear equivariant layers L : Rn×n → Rn×n have dimension b ( 4 ) = 15 . Figure 1 shows visualization of the basis to the linear equivariant layers acting on edge-value data such as adjacency matrices . In Hartford et al . ( 2018 ) the authors provide an impressive generalization of the case of node-value data to several node sets , V1 , V2 , . . . , Vm of sizes n1 , n2 , . . . , nm . Their goal is to learn interactions across sets . That is , an input data point is a tensor A ∈ Rn1×n2×···×nm that assigns a value to each element in the cartesian product V1 × V2 × · · · × Vm . Renumbering the nodes in each node set using permutation matrices P1 , . . . , Pm ( resp . ) results in a new tensor we denote by P1 : m ? A . Order invariance means f ( P1 : m ? A ) = f ( A ) and order equivariance is f ( P1 : m ? A ) = P1 : m ? f ( A ) . Hartford et al . ( 2018 ) introduce bases for linear invariant and equivariant layers . Although the layers in Hartford et al . ( 2018 ) satisfy the order invariance and equivariance , they do not exhaust all possible such layers in case some node sets coincide . For example , if V1 = V2 they have 4 independent learnable parameters where our model has the maximal number of 15 parameters . Our analysis allows generalizing the multi-node set case to arbitrary tensor data over V1 × V2 × · · · × Vm . Namely , for data points in the form of a tensor A ∈ Rn k1 1 ×n k2 2 ×···×n km m . The tensor A attaches a value to every element of the Cartesian product Vk11 × · · · × V k2 2 , that is , k1-tuple from V1 , k2-tuple from V2 and so forth . We show that the linear space of invariant linear layers L : Rn k1 1 ×n k2 2 ×···×n km m → R is of dimension ∏m i=1 b ( ki ) , while the equivariant linear layers L : Rn k1 1 ×n k2 2 ×···×n km m → Rn l1 1 ×n l2 2 ×···×n lm m has dimension ∏m i=1 b ( ki + li ) . We also provide orthogonal bases for these spaces . Note that , for clarity , the discussion above disregards biases and features ; we detail these in the paper . In appendix C we show that our model is capable of approximating any message-passing neural network as defined in Gilmer et al . ( 2017 ) which encapsulate several popular graph learning models . One immediate corollary is that the universal approximation power of our model is not lower than message passing neural nets . In the experimental part of the paper we concentrated on possibly the most popular instantiation of graph learning , namely that of a single node set and edge-value data , e.g. , with adjacency matrices . We created simple networks by composing our invariant or equivariant linear layers in standard ways and tested the networks in learning invariant and equivariant graph functions : ( i ) We compared identical networks with our basis and the basis of Hartford et al . ( 2018 ) and showed we can learn graph functions like trace , diagonal , and maximal singular vector . The basis in Hartford et al . ( 2018 ) , tailored to the multi-set setting , can not learn these functions demonstrating it is not maximal in the graph-learning ( i.e. , multi-set with repetitions ) scenario . We also demonstrate our representation allows extrapolation : learning on one size graphs and testing on another size ; ( ii ) We also tested our networks on a collection of graph learning datasets , achieving results that are comparable to the state-of-the-art in 3 social network datasets . 2 PREVIOUS WORK . Our work builds on two main sub-fields of deep learning : group invariant or equivariant networks , and deep learning on graphs . Here we briefly review the relevant works . Invariance and equivariance in deep learning . In many learning tasks the functions that we want to learn are invariant or equivariant to certain symmetries of the input object description . Maybe the first example is the celebrated translation invariance of Convolutional Neural Networks ( CNNs ) ( LeCun et al. , 1989 ; Krizhevsky et al. , 2012 ) ; in this case , the image label is invariant to a translation of the input image . In recent years this idea was generalized to other types of symmetries such as rotational symmetries ( Cohen & Welling , 2016a ; b ; Weiler et al. , 2018 ; Cohen et al. , 2018 ) . Cohen & Welling ( 2016a ) introduced Group Equivariant Neural Networks that use a generalization of the convolution operator to groups of rotations and reflections ; Weiler et al . ( 2018 ) ; Cohen et al . ( 2018 ) also considered rotational symmetries but in the case of 3D shapes and spherical functions . Ravanbakhsh et al . ( 2017 ) showed that any equivariant layer is equivalent to a certain parameter sharing scheme . If we adopt this point of view , our work reveals the structure of the parameter sharing in the case of graphs and hyper-graphs . In another work , Kondor & Trivedi ( 2018 ) show that a neural network layer is equivariant to the action of some compact group iff it implements a generalized form of the convolution operator . Yarotsky ( 2018 ) suggested certain group invariant/equivariant models and proved their universality . To the best of our knowledge these models were not implemented . Learning of graphs . Learning of graphs is of huge interest in machine learning and we restrict our attention to recent advancements in deep learning on graphs . Gori et al . ( 2005 ) ; Scarselli et al . ( 2009 ) introduced Graph Neural Networks ( GNN ) : GNNs hold a state ( a real valued vector ) for each node in the graph , and propagate these states according to the graph structure and learned parametric functions . This idea was further developed in Li et al . ( 2015 ) that use gated recurrent units . Following the success of CNNs , numerous works suggested ways to define convolution operator on graphs . One promising approach is to define convolution by imitating its spectral properties using the Laplacian operator to define generalized Fourier basis on graphs ( Bruna et al. , 2013 ) . Multiple follow-up works ( Henaff et al. , 2015 ; Defferrard et al. , 2016 ; Kipf & Welling , 2016 ; Levie et al. , 2017 ) suggest more efficient and spatially localized filters . The main drawback of spectral approaches is that the generalized Fourier basis is graph-dependent and applying the same network to different graphs can be challenging . Another popular way to generalize the convolution operator to graphs is learning stationary functions that operate on neighbors of each node and update its current state ( Atwood & Towsley , 2016 ; Duvenaud et al. , 2015 ; Hamilton et al. , 2017 ; Niepert et al. , 2016 ; Veličković et al. , 2017 ; Monti et al. , 2017 ; Simonovsky & Komodakis , 2017 ) . This idea generalizes the locality and weight sharing properties of the standard convolution operators on regular grids . As shown in the important work of Gilmer et al . ( 2017 ) , most of the the above mentioned methods ( including the spectral methods ) can be seen as instances of the general class of Message Passing Neural Networks .

Summary: The paper presents a maximally expressive parameter-sharing scheme for hypergraphs, and in general when modeling the high order interactions between elements of a set. This setting is further generalized to multiple sets. The paper shows that the number of free parameters in invariant and equivariant layers corresponds to the different partitioning of the index-set of input and output tensors. Experimental results suggest that the proposed layer can outperform existing methods in supervised learning with graphs.