Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have recently become very popular for graph machine learning ( GML ) research and have shown great results on node classification tasks ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Veličković et al. , 2017 ) like product prediction on co-purchasing graphs and paper category prediction on citation graphs . However , for large-scale industrial applications , MLPs remain the major workhorse , despite common ( implicit ) underlying graphs and suitability for GML formalisms . One reason for this academic-industrial gap is the challenges in scalability and deployment brought by data dependency in GNNs ( Zhang et al. , 2020 ; Jia et al. , 2020 ) , which makes GNNs hard to deploy for latency-constraint applications that require fast inference . Neighborhood fetching caused by graph dependency is one of the major sources of GNN latency . Inference on a target node necessitates fetching topology and features of many neighbor nodes , especially on small-world graphs ( detailed discussion in Section 4 ) . Common inference acceleration techniques like pruning ( Zhou et al. , 2021 ) and quantization ( Tailor et al. , 2021 ; Zhao et al. , 2020 ) can speed up GNNs to some extent by reducing Multiplication-and-ACcumulation ( MAC ) operations . However , their improvements are limited given the graph dependency is not resolved . Unlike GNNs , MLPs have no dependency on graph data and are easier to deploy than GNNs . They also enjoy the auxiliary benefit of sidestepping the cold-start problem that often happens during the online prediction of relational data ( Wei et al. , 2020 ) , meaning MLPs can infer reasonably even when neighbor information of a new encountered node is not immediately available . On the other hand , this lack of graph dependency typically hurts for relational learning tasks , limiting MLP performance on GML tasks compared to GNNs . We thus ask : can we bridge the two worlds , enjoy the low-latency and dependency-free nature of MLPs and the graph context awareness of GNNs at the same time ? Present work . Our key finding is that it is possible to distill knowledge from GNNs to MLPs without losing significant performance , but reducing the inference time drastically for node classification . The knowledge distillation ( KD ) can be done offline , coupled with model training . In other words , we can shift considerable work from the latency-constrained inference step , where time reduction in milliseconds makes a huge difference , to the less time-sensitive training step , where time cost in hours or days is often tolerable . We call our approach Graph-less Neural Network ( GLNN ) . Specifically , GLNN is a modeling paradigm involving KD from a GNN teacher to a student MLP ; the resulting GLNN is an MLP optimized through KD , so it enjoys the benefits of graph contextawareness in training but has no graph dependency in inference . Regarding speed , GLNNs have superior efficiency and are 146×-273× faster than GNNs and 14×-27× faster than other inference acceleration methods . Regarding performance , under a production setting involving both transductive and inductive predictions on 7 datasets , GLNN accuracies improve over MLPs by 12.36 % on average and match GNNs on 6/7 datasets . We comprehensively study when and why GLNNs can achieve competitive results as GNNs . Our analysis suggests the critical factors for such great performance are large MLP sizes and high mutual information between node features and labels . Our observations align with recent results in vision and language , which posit that large enough ( or slightly modified ) MLPs can achieve similar results as CNNs and Transformers ( Liu et al. , 2021 ; Tolstikhin et al. , 2021 ; Melas-Kyriazi , 2021 ; Touvron et al. , 2021 ; Ding et al. , 2021 ) . Our core contributions are as follows : • We propose GLNN , which eliminates neighbor-fetching latency in GNN inference via KD to MLP . • We show GLNNs has competitive performance as GNNs , while enjoying 239×-758× faster inference than vanilla GNNs and 27×-32× faster inference than other inference acceleration methods . • We study GLNN properties comprehensively by investigating their performance under different settings , how they work as regularizers , their inductive bias , expressiveness , and limitations . 2 RELATED WORK . Graph Neural Networks . The idea of GNNs started with generalizing convolution to graphs ( Bruna et al. , 2014 ; Defferrard et al. , 2017 ) and later simplified the to a message-passing neural network ( MPNN ) framework by GCN ( Kipf & Welling , 2016 ) . The majority of GNNs after are build on GCN and can be put under MPNN . For example , GAT employs attention ( Veličković et al. , 2017 ) , PPNP employs personalized PageRank ( Klicpera et al. , 2019 ) , GCNII and DeeperGCN employ residual connections and dense connections ( Chen et al. , 2020 ; Li et al. , 2019 ) . Inference Acceleration . Inference acceleration have been proposed by hardware improvements ( Chen et al. , 2016 ; Judd et al. , 2016 ) and algorithmic improvements through pruning ( Han et al. , 2015 ) , quantization ( Gupta et al. , 2015 ) . For GNNs , pruning ( Zhou et al. , 2021 ) and quantizing GNN parameters ( Zhao et al. , 2020 ) have been studied . These approaches speed up GNN inference to a certain extent but do not eliminate the neighbor-fetching latency . In contrast , our cross-model KD solves this issue . Graph-MLP also tries to bypass GNN neighbor fetching ( Hu et al. , 2021 ) . It trains an MLP with a neighbor contrastive loss to push together embeddings for neighboring nodes and pulls apart embeddings for far away nodes . Some sampling works focus on speed up GNN training ( Zou et al. , 2019 ; Chen et al. , 2018 ) , which are complementary to our goal on inference acceleration . GNN distillation . Some works tried to distill large GNNs to smaller GNNs . LSP ( Yang et al. , 2021b ) and TinyGNN ( Yan et al. , 2020 ) both do KD while preserving local information . Their student is a GNN with fewer parameters , but not necessarily fewer layers than the teacher GNN , which makes both designs still graph-dependent and require latency-inducing fetching . GFKD ( Deng & Zhang , 2021 ) focuses on graph-level KD via graph generation . In GFKD , data instances are independent graphs , whereas we focus on dependent nodes within a graph . GraphSAIL ( Xu et al. , 2020 ) uses KD to train a student that works well on new data while preserving its performance on old data . CPF ( Yang et al. , 2021a ) distills GNNs to a student with label propagation ( LP ) to improve model performance . The student in CPF is not a GNN , but it is still heavily graph-dependent as it uses LP . 3 PRELIMINARIES . Notations . For GML tasks , the input is usually a graph and its node features , which we write as G = ( V , E ) , with V stands for all nodes , and E stands for all edges . Let N denote the total number of nodes . We use X ∈ RN×D to represent node features , with row xv being the D-dimensional feature of node v ∈ V . We represent edges with an adjacency matrix A , with Au , v = 1 if edge ( u , v ) ∈ E , and 0 otherwise . For node classification , one of the most important GML applications , the prediction targets are Y ∈ RN×K , where row yv is a K-dim one-hot vector for node v. For a given G , usually a small portion of nodes will be labeled , which we mark using superscript L , i.e . VL , XL , and Y L. The majority of nodes will be unlabeled , and we mark using the superscript U , i.e . VU , XU , and Y U . Graph Neural Networks . Most GNNs fit under the message-passing framework , where the representation hv of each node v is updated iteratively in each layer by collecting messages from its neighbors denoted asN ( v ) . For the l-th layer , h ( l ) v is obtained from the previous layer representation h ( l−1 ) u ( h ( 0 ) u = xu ) via an aggregation operation AGGR followed by an UPDATE operation as h ( l ) N ( v ) = AGGR ( { h ( l−1 ) u : u ∈ N ( v ) } ) and h ( l ) v = UPDATE ( h ( l ) N ( v ) , h ( l−1 ) v ) 4 MOTIVATION . As mentioned in Sec 1 , GNNs can have considerable inference latency due to graph dependency . One more GNN layer means fetching one more hop of neighbor nodes . To infer a single node with a L-layer GNN on a graph with average degree R will require O ( RL ) fetches . R can be large for real-world graphs , e.g . 208 for the Twitter ( Ching et al. , 2015 ) . Also , as fetching for successive layers must be done sequentially , the total latency explodes quickly as L increases . Figure 1 illustrates the dependency added by each GNN layer and the exponential explosion in the number of fetches and inference time . In contrast , the inference time of an MLP with the same number of layers is much smaller and only grows linearly . This marked gap contributes greatly to the practicality of MLPs in industrial applications over GNNs . The node-fetching latency is exacerbated by two factors : firstly , newer GNN architectures are getting deeper , e.g . 64 layers ( Chen et al. , 2020 ) , 112 layers ( Li et al. , 2019 ) , and even 1001 layers ( Li et al. , 2021 ) . Secondly , industrial-scale graphs are frequently too large to fit into the memory of a single machine , necessitating sharding of the graph out of the main memory . For example , Twitter has 288M monthly active users ( nodes ) and an estimated 60B followers ( edges ) as of 3/2015 . Facebook has 1.39B active users with more than 400B edges as of 12/2014 ( Ching et al. , 2015 ) . Even when stored in a sparse-matrix-friendly format ( often COO or CSR ) , these graphs are on the order of TBs and are constantly growing . Moving away from in-memory storage results in even slower neighbor-fetching . MLPs , on the other hand , lack the means to exploit graph topology , which hurts their performance for node classification . For example , test accuracy on Products is 78.61 for GraphSAGE compared to 62.47 for an equal-sized MLP . Nonetheless , recent results in vision and language posit that large ( or slightly modified ) MLPs can achieve similar results as CNNs and Transformers ( Liu et al. , 2021 ) . We thus also ask : Can we bridge the best of GNNs and MLPs to get high-accuracy and low-latency models ? This motivates us to do cross-model KD from GNNs to MLPs . 5 GRAPH-LESS NEURAL NETWORKS . We introduce GLNN and answer exploration questions of its properties : 1 ) How do GLNNs compare to MLPs and GNNs ? 2 ) Can GLNNs work well under both transductive and inductive settings ? 3 ) How do GLNNs compare to other inference acceleration methods ? 4 ) How do GLNNs benefit from KD ? 5 ) Do GLNNs have sufficient model expressiveness ? 6 ) When will GLNNs fail to work ? 5.1 THE GLNN FRAMEWORK The idea of GLNN is straightforward , yet as we will see , extremely effective . In short , we train a “ boosted ” MLP via KD from a teacher GNN . KD was introduced in Hinton et al . ( 2015 ) , where knowledge was transferred from a cumbersome teacher to a simpler student . In our case , we generate soft targets zv for each node v with a teacher GNN . Then we train a student MLP with both true labels yv and zv . The objective is as Equation 1 , with λ being a weight parameter , Llabel being the cross-entropy between yv and student predictions ŷv , Lteacher being the KL-divergence . L = λΣv∈VLLlabel ( ŷv , yv ) + ( 1− λ ) Σv∈VLteacher ( ŷv , zv ) ( 1 ) The model after KD , i.e . GLNN , is essentially a MLP . Therefore , GLNNs have no graph dependency during inference and are as fast as MLPs . On the other hand , through offline KD , GLNN parameters are optimized to predict and generalize as well as GNNs , with the added benefit of faster inference and easier deployment . In Figure 2 , we show the offline KD and online inference steps of GLNNs .

Summary: This paper applies KD in the context of the graph. It aims to distill the teacher output of a GNN model into a simple MLP model. Empirically, they show that this simple KD design is able to improve the student MLP model by a large margin and can match the results coming from a teacher GNN model. Besides, it shows empathically, the inference time can be greatly improved.