Lossless Compression of Structured Convolutional Models via Lifting

1 INTRODUCTION . Lifted , often referred to as templated , models use highly expressive representation languages , typically based in weighted predicate logic , to capture symmetries in relational learning problems ( Koller et al. , 2007 ) . This includes learning from data such as chemical , biological , social , or traffic networks , and various knowledge graphs , relational databases and ontologies . The idea has been studied extensively in probabilistic settings under the notion of lifted graphical models ( Kimmig et al. , 2015 ) , with instances such as Markov Logic Networks ( MLNs ) ( Richardson & Domingos , 2006 ) or Bayesian Logic Programs ( BLPs ) ( Kersting & De Raedt , 2001 ) . In a wider view , convolutions can be seen as instances of the templating idea in neural models , where the same parameterized pattern is being carried around to exploit the underlying symmetries , i.e . some forms of shared correlations in the data . In this analogy , the popular Convolutional Neural Networks ( CNN ) ( Krizhevsky et al. , 2012 ) themselves can be seen as a simple form of a templated model , where the template corresponds to the convolutional filters , unfolded over regular spatial grids of pixels . But the symmetries are further even more noticeable in structured , relational domains with discrete element types . With convolutional templates for regular trees , the analogy covers Recursive Neural Networks ( Socher et al. , 2013 ) , popular in natural language processing . Extending to arbitrary graphs , the same notion covers works such as Graph Convolutional Networks ( Kipf & Welling , 2016 ) and their variants ( Wu et al. , 2019 ) , as well as various Knowledge-Base Embedding methods ( Wang et al. , 2017 ) . Extending even further to relational structures , there are works integrating parameterized relational logic templates with neural networks ( Sourek et al. , 2018 ; Rocktäschel & Riedel , 2017 ; Marra & Kuželka , 2019 ; Manhaeve et al. , 2018 ) . The common underlying principle of templated models is a joint parameterization of the symmetries , allowing for better generalization . However , standard lifted models , such as MLNs , provide another key advantage that , under certain conditions , the model computations can be efficiently carried out without complete template unfolding , often leading to even exponential speedups ( Kimmig et al. , 2015 ) . This is known as “ lifted inference ” ( Kersting , 2012 ) and is utilized heavily in lifted graphical models as well as database query engines ( Suciu et al. , 2011 ) . However , to our best knowledge , this idea has been so far unexploited in the neural ( convolutional ) models . The main contribution of this paper is thus a “ lifting ” technique to compress symmetries in convolutional models applied to structured data , which we refer to generically as “ structured convolutional models ” . 1.1 RELATED WORK . The idea for the compression is inspired by lifted inference ( Kersting , 2012 ) used in templated graphical models . The core principle is that all equivalent sub-computations can be effectively carried out in a single instance and broadcasted into successive operations together with their respective multiplicities , potentially leading to significant speedups . While the corresponding “ liftable ” template formulae ( or database queries ) generating the isomorphisms are typically assumed to be given ( Kimmig et al. , 2015 ) , we explore the symmetries from the unfolded ground structures , similarly to the approximate methods based on graph bisimulation ( Sen et al. , 2012 ) . All the lifting techniques are then based in some form of first-order variable elimination ( summation ) , and are inherently designed to explore structural symmetries in graphical models . In contrast , we aim to additionally explore functional symmetries , motivated by the fact that even structurally different neural computation graphs may effectively perform identical function . The learning in neural networks is also principally different from the model counting-based computations in lifted graphical models in that it requires many consecutive evaluations of the models as part of the encompassing iterative training routine . Consequently , even though we assume to unfold a complete computation graph before it is compressed with the proposed technique , the resulting speedup due to the subsequent training is still substantial . From the deep learning perspective , there have been various model compression techniques proposed to speedup the training , such as pruning , decreasing precision , and low-rank factorization ( Cheng et al. , 2017 ) . However , to our best knowledge , the existing techniques are lossy in nature , with a recent exception of compressing ReLU networks based on identifying neurons with linear behavior ( Serra et al. , 2020 ) . None of these works exploit the model computation symmetries . The most relevant line of work here are Lifted Relational Neural Networks ( LRNNs ) ( Sourek et al. , 2018 ) which however , despite the name , provide only templating capabilities without lifted inference , i.e . with complete , uncompressed ground computation graphs . 2 BACKGROUND . The compression technique described in this paper is applicable to a number of structured convolutional models , ranging from simple recursive ( Socher et al. , 2013 ) to fully relational neural models ( Sourek et al. , 2018 ) . The common characteristic of the targeted learners is the utilization of convolution ( templating ) , where the same parameterized pattern is carried over different subparts of the data ( representation ) with the same local structure , effectively introducing repetitive sub-computations in the resulting computation graphs , which we exploit in this work . 2.1 GRAPH NEURAL NETWORKS . Graph neural networks ( GNNs ) are currently the most prominent representatives of structured convolutional models , which is why we choose them for brevity of demonstration of the proposed compression technique . GNNs can be seen as an extension of the common CNN principles to completely irregular graph structures . Given a particularly structured input sample graph Sj , they dynamically unfold a multi-layered computation graph Gj , where the structure of each layer i follows the structure of the whole input graph Sj . For computation of the next layer i+1 values , each node v from the input graph Sj calculates its own value h ( v ) by aggregating A ( “ pooling ” ) the values of the adjacent nodes u : edge ( u , v ) , transformed by some parametric function CW1 ( “ convolution ” ) , which is being reused with the same parameterization W1 within each layer i as : h̃ ( v ) ( i ) = A ( i ) ( { C ( i ) W i1 ( h ( u ) ( i−1 ) ) |u : edge ( u , v ) } ) ( 1 ) The h̃ ( i ) ( v ) can be further combined through another CW2 with the central node ’ s representation from the previous layer to obtain the final updated value h ( i ) ( v ) for layer i as : h ( v ) ( i ) = C ( i ) W i2 ( h ( v ) ( i−1 ) , h̃ ( v ) ( i ) ) ( 2 ) This general principle covers a wide variety of GNN models , such the popular GCNs ( Kipf & Welling , 2016 ) , graph-SAGE ( Hamilton et al. , 2017 ) , GIN ( Xu et al. , 2018a ) , and others ( Xu et al. , 2018b ; Gilmer et al. , 2017 ) , which then reduces to the respective choices of particular aggregations A and transformations CW . An example computation graph of a generic GNN unfolded over an example molecule of methane is shown in Fig . 2 . 2.2 COMPUTATION GRAPHS . For the sake of this paper , let us now define the common notion of a computation graph more formally . A computation graph is a tuple G = ( N , E , F ) , where N = ( 1 , 2 , . . . , n ) is a list of nodes and E ⊆ N 2 × N is a list of directed labeled edges . Each labeled edge is a triple of integers ( n1 , n2 , l ) where n1 and n2 are nodes of the computation graph and l is the label . The labels are used to assign weights to the edges in the computation graph . Note this allows to define the weight sharing scheme as part of the graph ( cf Example 1 below ) . Finally , F = { f1 , f2 , . . . , fn } is the list of activation functions , one for each node from N . As usual , the graph is assumed to be acyclic . Children of a node N are naturally defined as all those nodes M such that ( M , N , L ) ∈ E , and analogically for parents . Note that since E is a list , edges contained in it are ordered , and the same edge may appear multiple times ( which will be useful later ) . Children of each node are also ordered – given two children C and C ′ of a node N , C precedes C ′ iff ( C , N , L ) precedes ( C ′ , N , L′ ) in the list of edges E . We denote the lists of children and parents of a given node N by Children ( N ) and Parents ( N ) , respectively . Computation graphs are then evaluated bottom up from the leaves of the graph ( nodes with no children ) to the roots of the graph ( nodes with no parents ) . Given a list of weightsW , we can now define the value of a node N ∈ N recursively as : value ( N ; W ) = fN ( WL1 · value ( M1 ; W ) , . . . , WLm · value ( Mm ; W ) ) , where ( M1 , . . . , Mm ) ≡ Children ( N ) is the ( ordered ) list of children of the nodeN , and L1 , . . . , Lm are the labels of the respective edges ( M1 , N , L1 ) , . . . , ( Mm , N , Lm ) ∈ E , andWLi is the Li-th component of the list W . Note that with the structured convolutional models , such as GNNs , we assume dynamic computation graphs where each learning sample Sj generates a separate Gj . Consequently , we can associate the leaf nodes in each Gj with constant functions1 , outputting the corresponding node ( feature ) values from the corresponding structured input sample Sj . 3 PROBLEM DEFINITION . The problem of detecting the symmetries in computation graphs can then be formalized as follows . Definition 1 ( Problem Definition ) . Let G = ( N , E , F ) be a computation graph . We say that two nodes N1 , N2 are equivalent if , for any W , it holds that value ( N1 ; W ) = value ( N2 ; W ) . The problem of detecting symmetries in computation graphs asks to partition the nodes of the computation graph into equivalence classes of mutually equivalent nodes . Example 1 . Consider the computation graph G = ( N , E , F ) , depicted in Fig . 1 , where N = { 0 , 1 , 2 , 3 , 4 } , E = ( ( 0 , 2 , 1 ) , ( 1 , 3 , 1 ) , ( 2 , 4 , 2 ) , ( 3 , 4 , 2 ) ) , F = { f0 = f1 = 1 , f2 ( x ) = f3 ( x ) = x , f4 ( x , y ) = x · cos ( y ) } . 1in contrast to static computation graphs where these functions are identities requiring the features at input . LetW = ( w1 , w2 ) be the weight list . The computation graph then computes the function ( w1w2 ) · cos ( w1w2 ) . It is not difficult to verify that the nodes { 0 , 1 } , and { 2 , 3 } are functionally equivalent . This also means , as we discuss in more detail in the next section , that we can “ merge ” them without changing the function that the graph computes . The resulting reduced graph then has the form N = { 1 , 3 , 4 } , E = { ( 1 , 3 , 1 ) , ( 3 , 4 , 2 ) , ( 3 , 4 , 2 ) } , F = { f1 = 1 , f3 ( x ) = x , f4 ( x , y ) = x · cos ( y ) } . In the example above , the nodes { 0 , 1 } and { 2 , 3 } are in fact also isomorphic in the sense that there exists an automorphism ( preserving weights and activation functions ) of the computation graph that swaps the nodes . Note that our definition is less strict : all we want the nodes to satisfy is functional equivalence , meaning that they should evaluate to the same values for any initialization ofW . We will also use the notion of structural-equivalence of nodes in computational graphs . Two nodes are structurally equivalent if they have the same outputs for any assignment of weightsW and for any replacement of any of the activation functions in the graph.2 That is if two nodes are structurally equivalent then they are also functionally equivalent but not vice versa . Importantly , the two nodes do not need to be automorphic3 in the graph-theoretical sense while being structurally equivalent , which also makes detecting structural equivalence easier from the computational point of view . In particular , we describe a simple polynomial-time algorithm in Section 4.2 .

Summary: The paper provides an interesting work in the scale/speed up of structured convolutional models. In particular, it proposes an idea using a technique named lifting which is used in scaling up of graphical models to detect the symmetries and compress the neural model such as Graph Neural Network. Authors show that this compression can lead to speedups of the models in many tasks.