Adaptive Filters for Low-Latency and Memory-Efficient Graph Neural Networks

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have emerged as an effective way to build models over arbitrarily structured data . For example , they have successfully been applied to computer vision tasks : GNNs can deliver high performance on point cloud data ( Qi et al. , 2017 ) and for feature matching across images ( Sarlin et al. , 2020 ) . Recent work has also shown that they can be applied to physical simulations ( Pfaff et al. , 2020 ; Sanchez-Gonzalez et al. , 2020 ) . Code analysis is another application domain where GNNs have found success ( Guo et al. , 2020 ; Allamanis et al. , 2017 ) . In recent years , the research community has devoted significant attention to building more expressive , and better performing , models to process graphs . Efforts to benchmark GNN models , such as Open Graph Benchmark ( Hu et al. , 2020 ) , or the work by Dwivedi et al . ( 2020 ) , have attempted to more rigorously quantify the relative performance of different proposed architectures . One common conclusion—explicitly stated by Dwivedi et al . ( 2020 ) —is that anisotropic models , in which messages sent between nodes are a function of both the source and target node , are the best performing models . By comparison , isotropic models , where messages are a function of the source node only , achieve lower accuracy , even if they have efficiency benefits over comparable anisotropic models . Intuitively , this conclusion is satisfying : anisotropic models are inherently more expressive , hence we would expect them to perform better in most situations . Our work provides a surprising challenge to this wisdom by providing an isotropic model , called Efficient Graph Convolution ( EGC ) , ∗Corresponding author . Contact at sat62 @ cam.ac.uk that outperforms comparable anisotropic approaches , including the popular GAT ( Veličković et al. , 2018 ) and PNA ( Corso et al. , 2020 ) architectures . In addition to providing a surprising empirical result for the community , our work has significant practical implications for efficiency , as shown in Figure 1 . As EGC is an isotropic model achieving high accuracy , we can take advantage of the efficiency benefits offered by isotropic models without having to compromise on model accuracy . We have seen memory consumption and latency for state-of-the-art GNN architectures increase to O ( E ) in recent years , due to state-of-the-art models incorporating anisotropic mechanisms to boost accuracy . EGC reduces the complexity to O ( V ) , delivering substantial real-world benefits , albeit with the precise benefit being dependent on the topology of the graphs the model is applied to . The reader should note that our approach can also be combined with other approaches for improving the efficiency of GNNs . For example , common hardware-software co-design techniques include quantization and pruning ( Sze et al. , 2020 ) could be combined with this work , which proposes an orthogonal approach of improving model efficiency by improving the underlying architecture design . We also note that our approach can be combined with graph sampling techniques ( Zeng et al. , 2019 ; Hamilton et al. , 2017 ; Chen et al. , 2018a ) to improve scalability further when training on graphs with millions , or billions , of nodes . Contributions ( 1 ) We propose a new GNN architecture , Efficient Graph Convolution ( EGC ) , and provide both spatial and spectral interpretations for it . ( 2 ) We provide a rigorous evaluation of our architecture across 6 large graph datasets covering both transductive and inductive use-cases , and demonstrate that EGC consistently achieves better results than strong baselines . ( 3 ) We provide several ablation studies to motivate the selection of the hyperparameters in our model . ( 4 ) We demonstrate that our model simultaneously achieves better parameter efficiency , latency and memory consumption than competing approaches . Code and pre-trained models for our experiments ( including baselines ) can be found at https : //github.com/shyam196/egc . At time of publication , EGC has also been upstreamed to PyTorch Geometric ( Fey & Lenssen , 2019 ) . 2 BACKGROUND . 2.1 HARDWARE-SOFTWARE CO-DESIGN FOR DEEP LEARNING . Several of the popular approaches for co-design have already been described in the introduction : quantization , pruning , and careful architecture design are all common for CNNs and Transformers ( Vaswani et al. , 2017 ) . In addition to enabling better performance to be obtained from general purpose processors such as CPUs and GPUs , these techniques are also essential for maximizing the return from specialized accelerators ; while it may be possible to improve performance over time due to improvements in CMOS technology , further improvements plateau without innovation at the algorithmic level ( Fuchs & Wentzlaff , 2019 ) . As neural network architecture designers , we can not simply rely on improvements in hardware to make our proposals viable for real-world deployment . 2.2 GRAPH NEURAL NETWORKS . Many GNN architectures can be viewed as a generalization of CNN architectures to the irregular domain : as in CNNs , representations at each node are built based on the local neighborhood using parameters that are shared across the graph . GNNs differ as we can not make assumptions about the the size of the neighborhood , or the ordering . One common framework used to define GNNs is the message passing neural network ( MPNN ) paradigm ( Gilmer et al. , 2017 ) . A graph G = ( V , E ) has node features X ∈ RN×F , adjacency matrix A ∈ RN×N and optionally D-dimensional edge features E ∈ RE×D . We define a function φ that calculates messages from node u to node v , a differentiable and permutation-invariant aggregator ⊕ , and an update function γ to calculate representations at layer l + 1 : h ( i ) l+1 = γ ( h ( i ) l , ⊕j∈N ( i ) [ φ ( h ( i ) l , h ( j ) l , eij ) ] ) . Propagation rules for baseline architecture are provided in Table 1 , with further details supplied in Table 5 in the Appendix . Relative Expressivity of GNNs Common wisdom in the research community states that isotropic GNNs are less expressive than anisotropic GNNs ; empirically this is well supported by benchmarks . Brody et al . ( 2022 ) prove that GAT models can be strictly more expressive than isotropic models . Bronstein et al . ( 2021 ) also discuss the relative expressivity of different classes of GNN layer , and argue that convolutional ( also known as isotropic ) models are well suited to problems leveraging homophily1 in the input graph . They further argue that attentional , or full message passing , models are suited to handling heterophilous problems , but they acknowledge the resource consumption and trainability of these architectures may be prohibitive—especially in the case of full message passing . Scaling and Deploying GNNs While GNNs have seen success across a range of domains , there remain challenges associated with scaling and deploying them . Graph sampling is one approach to scaling training for large graphs or models which will not fit in memory . Rather than training over the full graph , each iteration is run over a sampled sub-graph ; approaches vary in whether they sample node-wise ( Hamilton et al. , 2017 ) , layer-wise ( Chen et al. , 2018a ; Huang et al. , 2018 ) , or subgraphs ( Zeng et al. , 2019 ; Chiang et al. , 2019 ) . Alternatively , systems for distributed GNN training have been proposed ( Jia et al. , 2020 ) to scale training beyond the limits of a single accelerator . Some works have proposed architectures that are designed to accommodate scaling : graph-augmented MLPs , such as SIGN ( Rossi et al. , 2020 ) , are explicitly designed as a shallow architecture , as all the graph operations are done as a pre-processing step . Other work includes applying neural architecture search ( NAS ) to arrange existing GNN layers ( Zhao et al. , 2020 ) , or building quantization techniques for GNNs ( Tailor et al. , 2021 ) . Finally , a recent work has shown that using memoryefficient reversible residuals ( Gomez et al. , 2017 ) for GNNs ( Li et al. , 2021 ) enables us to train far deeper and larger GNN models than before , thereby progressing the state-of-the-art accuracy . Why Are Existing Approaches Not Sufficient ? It is worth noting that many of these approaches have significant limitations that we aim to address with our work . Sampling methods are often ineffective when applied to many problems which involve model generalization to unseen graphs—a common use-case for GNNs . We evaluated a variety of sampling approaches and observed that even modest sampling levels , which provide little benefit to memory or latency , cause model performance to decline noticeably . In addition , these methods do not accelerate the underlying GNN , hence they may not provide any overall benefit to inference latency . There is also no evidence that we are aware of that graph-augmented MLPs perform adequately when generalizing to unseen graphs ; indeed , they are known to be theoretically less expressive than standard GNNs ( Chen et al. , 2021 ) . We also investigated this setup , and found that these approaches do not offer competitive accuracy with state-of-the-art approaches . Experiment details and results , along with further discussion of the limitations of existing work , is provided in Appendix B . In summary , our work on efficient GNN architecture design is of interest to the community for two reasons : firstly , it raises questions about common assumptions , and how we design and evaluate GNN models ; secondly , our work may enable us to scale our models further , potentially yielding improvements in accuracy . In addition , for tasks where we need to generalize to unseen graphs , such as code analysis or point cloud processing , we reduce memory consumption and latency , thereby enabling us to deploy our models to more resource-constrained devices than before . We note that efficient architecture design can be usefully combined with other approaches including sampling , quantization , and pruning , where appropriate . 1Homophily means that if two nodes are connected , then they have high similarity 3 OUR ARCHITECTURE : EFFICIENT GRAPH CONVOLUTION ( EGC ) . In this section we describe our approach , and delay theoretical analysis to the next section . We present two versions : EGC-S ( ingle ) , using a single aggregator , and EGC-M ( ulti ) which generalizes our approach by incorporating multiple aggregators . Our approach is visualized in Figure 2 . 3.1 ARCHITECTURE DESCRIPTION . For a layer with in-dimension of F and out-dimension of F ′ we use B basis weights Θb ∈ RF ′×F . We compute the output for node i by calculating combination weighting coefficients w ( i ) ∈ RB per node , and weighting the results of each aggregation using the different basis weights Θb . We calculate w ( i ) = Φx ( i ) + b , where Φ ∈ RB×F and b ∈ RB are weight and bias parameters associated with calculating the combination weighting coefficients . The output for node i is computed in three steps . First , we perform the aggregation with each set of basis weights Θb . Second , we compute the weighting coefficients w ( i ) = Φx ( i ) +b ∈ RB for each node i , where Φ ∈ RB×F and b ∈ RB are weight and bias parameters for calculating the combination weighting coefficients . Third , the layer output for node i is the weighted combination of aggregation outputs : y ( i ) = B∑ b=1 w ( i ) b ∑ j∈N ( i ) α ( i , j ) Θbx ( j ) ( 1 ) where α ( i , j ) is some function of nodes i and j , andN ( i ) denotes the in-neighbours of i . A popular method pioneered by GAT ( Veličković et al. , 2018 ) to boost representational power is to represent α using a learned function of the two nodes ’ representations . While this enables anisotropic treatment of neighbors , and can boost performance , it necessarily results in memory consumption of O ( |E| ) due to messages needing to be explicitly materialized , and complicates hardware implementation for accelerators . If we choose a representation for α that is not a function of the node representations— such as α ( i , j ) = 1 to recover the add aggregator used by GIN ( Xu et al. , 2019 ) , or α ( i , j ) = 1/ √ deg ( i ) deg ( j ) to recover symmetric normalization used by GCN ( Kipf & Welling , 2017 ) — then we can implement our message propagation phase using sparse matrix multiplication ( SpMM ) , and avoid explicitly materializing each message , even for the backwards pass . In this work , we assume α ( i , j ) to be symmetric normalization as used by GCN unless otherwise stated ; we use this normalization as it is known to offer strong results across a variety of tasks ; more formal justification is provided in section 4.2 . Adding Heads as a Regularizer We can extend our layer through the addition of heads , as used in architectures such as GAT or Transformers ( Vaswani et al. , 2017 ) . These heads share the basis weights , but apply different weighting coefficients per head . We find that adding this degree of freedom aids regularization when the number of heads ( H ) is larger thanB , as bases are discouraged from specializing ( see section 5.3 ) , without requiring the integration of additional loss terms into the optimization—hence requiring no changes to code for downstream users . To normalize the output dimension , we change the basis weight matrices dimensions to F ′ H ×F . Using ‖ as the concatenation operator , and making the use of symmetric normalization explicit , we obtain the EGC-S layer : y ( i ) = H ‖ h=1 B∑ b=1 w ( i ) h , b ∑ j∈N ( i ) ∪ { i } 1√ deg ( i ) deg ( j ) Θbx ( j ) ( 2 ) EGC works by combining basis matrices . This idea was proposed in R-GCN ( Schlichtkrull et al. , 2018 ) to handle multiple edge types ; Xu et al . ( 2021 ) can be viewed as a generalization of this approach to point cloud analysis . In this work we are solving a different problem to these works : we are interested in designing efficient architectures , rather than new ways to handle edge information .

Summary: This paper introduces Adaptive Filters that enable some of the benefits of Message Passing architectures, but while maintaining a memory consumption that scales with the number of nodes.  The authors claim that this architecture is not only better performing, and use less memory, but more efficient for GPUs through the use of sparse matrix multiplication.  The idea of the model is that you have a number of filters (MLPs or linear layers) applied to each sending node latent, a "filter". These are then weighted and summed by a weighting vector calculated as a function of the receiving node. Since there are no functions that take as input both sending, receiving or edge inputs, the memory will scale with the number of nodes. You, in essence, get an efficient pseudo-attention mechanism.