Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have recently become very popular for graph machine learning ( GML ) research and have shown great results on node classification tasks ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Veličković et al. , 2017 ) like product prediction on co-purchasing graphs and paper category prediction on citation graphs . However , for large-scale industrial applications , MLPs remain the major workhorse , despite common ( implicit ) underlying graphs and suitability for GML formalisms . One reason for this academic-industrial gap is the challenges in scalability and deployment brought by data dependency in GNNs ( Zhang et al. , 2020 ; Jia et al. , 2020 ) , which makes GNNs hard to deploy for latency-constraint applications that require fast inference . Neighborhood fetching caused by graph dependency is one of the major sources of GNN latency . Inference on a target node necessitates fetching topology and features of many neighbor nodes , especially on small-world graphs ( detailed discussion in Section 4 ) . Common inference acceleration techniques like pruning ( Zhou et al. , 2021 ) and quantization ( Tailor et al. , 2021 ; Zhao et al. , 2020 ) can speed up GNNs to some extent by reducing Multiplication-and-ACcumulation ( MAC ) operations . However , their improvements are limited given the graph dependency is not resolved . Unlike GNNs , MLPs have no dependency on graph data and are easier to deploy than GNNs . They also enjoy the auxiliary benefit of sidestepping the cold-start problem that often happens during the online prediction of relational data ( Wei et al. , 2020 ) , meaning MLPs can infer reasonably even when neighbor information of a new encountered node is not immediately available . On the other hand , this lack of graph dependency typically hurts for relational learning tasks , limiting MLP performance on GML tasks compared to GNNs . We thus ask : can we bridge the two worlds , enjoy the low-latency and dependency-free nature of MLPs and the graph context awareness of GNNs at the same time ? Present work . Our key finding is that it is possible to distill knowledge from GNNs to MLPs without losing significant performance , but reducing the inference time drastically for node classification . The knowledge distillation ( KD ) can be done offline , coupled with model training . In other words , we can shift considerable work from the latency-constrained inference step , where time reduction in milliseconds makes a huge difference , to the less time-sensitive training step , where time cost in hours or days is often tolerable . We call our approach Graph-less Neural Network ( GLNN ) . Specifically , GLNN is a modeling paradigm involving KD from a GNN teacher to a student MLP ; the resulting GLNN is an MLP optimized through KD , so it enjoys the benefits of graph contextawareness in training but has no graph dependency in inference . Regarding speed , GLNNs have superior efficiency and are 146×-273× faster than GNNs and 14×-27× faster than other inference acceleration methods . Regarding performance , under a production setting involving both transductive and inductive predictions on 7 datasets , GLNN accuracies improve over MLPs by 12.36 % on average and match GNNs on 6/7 datasets . We comprehensively study when and why GLNNs can achieve competitive results as GNNs . Our analysis suggests the critical factors for such great performance are large MLP sizes and high mutual information between node features and labels . Our observations align with recent results in vision and language , which posit that large enough ( or slightly modified ) MLPs can achieve similar results as CNNs and Transformers ( Liu et al. , 2021 ; Tolstikhin et al. , 2021 ; Melas-Kyriazi , 2021 ; Touvron et al. , 2021 ; Ding et al. , 2021 ) . Our core contributions are as follows : • We propose GLNN , which eliminates neighbor-fetching latency in GNN inference via KD to MLP . • We show GLNNs has competitive performance as GNNs , while enjoying 239×-758× faster inference than vanilla GNNs and 27×-32× faster inference than other inference acceleration methods . • We study GLNN properties comprehensively by investigating their performance under different settings , how they work as regularizers , their inductive bias , expressiveness , and limitations . 2 RELATED WORK . Graph Neural Networks . The idea of GNNs started with generalizing convolution to graphs ( Bruna et al. , 2014 ; Defferrard et al. , 2017 ) and later simplified the to a message-passing neural network ( MPNN ) framework by GCN ( Kipf & Welling , 2016 ) . The majority of GNNs after are build on GCN and can be put under MPNN . For example , GAT employs attention ( Veličković et al. , 2017 ) , PPNP employs personalized PageRank ( Klicpera et al. , 2019 ) , GCNII and DeeperGCN employ residual connections and dense connections ( Chen et al. , 2020 ; Li et al. , 2019 ) . Inference Acceleration . Inference acceleration have been proposed by hardware improvements ( Chen et al. , 2016 ; Judd et al. , 2016 ) and algorithmic improvements through pruning ( Han et al. , 2015 ) , quantization ( Gupta et al. , 2015 ) . For GNNs , pruning ( Zhou et al. , 2021 ) and quantizing GNN parameters ( Zhao et al. , 2020 ) have been studied . These approaches speed up GNN inference to a certain extent but do not eliminate the neighbor-fetching latency . In contrast , our cross-model KD solves this issue . Graph-MLP also tries to bypass GNN neighbor fetching ( Hu et al. , 2021 ) . It trains an MLP with a neighbor contrastive loss to push together embeddings for neighboring nodes and pulls apart embeddings for far away nodes . Some sampling works focus on speed up GNN training ( Zou et al. , 2019 ; Chen et al. , 2018 ) , which are complementary to our goal on inference acceleration . GNN distillation . Some works tried to distill large GNNs to smaller GNNs . LSP ( Yang et al. , 2021b ) and TinyGNN ( Yan et al. , 2020 ) both do KD while preserving local information . Their student is a GNN with fewer parameters , but not necessarily fewer layers than the teacher GNN , which makes both designs still graph-dependent and require latency-inducing fetching . GFKD ( Deng & Zhang , 2021 ) focuses on graph-level KD via graph generation . In GFKD , data instances are independent graphs , whereas we focus on dependent nodes within a graph . GraphSAIL ( Xu et al. , 2020 ) uses KD to train a student that works well on new data while preserving its performance on old data . CPF ( Yang et al. , 2021a ) distills GNNs to a student with label propagation ( LP ) to improve model performance . The student in CPF is not a GNN , but it is still heavily graph-dependent as it uses LP . 3 PRELIMINARIES . Notations . For GML tasks , the input is usually a graph and its node features , which we write as G = ( V , E ) , with V stands for all nodes , and E stands for all edges . Let N denote the total number of nodes . We use X ∈ RN×D to represent node features , with row xv being the D-dimensional feature of node v ∈ V . We represent edges with an adjacency matrix A , with Au , v = 1 if edge ( u , v ) ∈ E , and 0 otherwise . For node classification , one of the most important GML applications , the prediction targets are Y ∈ RN×K , where row yv is a K-dim one-hot vector for node v. For a given G , usually a small portion of nodes will be labeled , which we mark using superscript L , i.e . VL , XL , and Y L. The majority of nodes will be unlabeled , and we mark using the superscript U , i.e . VU , XU , and Y U . Graph Neural Networks . Most GNNs fit under the message-passing framework , where the representation hv of each node v is updated iteratively in each layer by collecting messages from its neighbors denoted asN ( v ) . For the l-th layer , h ( l ) v is obtained from the previous layer representation h ( l−1 ) u ( h ( 0 ) u = xu ) via an aggregation operation AGGR followed by an UPDATE operation as h ( l ) N ( v ) = AGGR ( { h ( l−1 ) u : u ∈ N ( v ) } ) and h ( l ) v = UPDATE ( h ( l ) N ( v ) , h ( l−1 ) v ) 4 MOTIVATION . As mentioned in Sec 1 , GNNs can have considerable inference latency due to graph dependency . One more GNN layer means fetching one more hop of neighbor nodes . To infer a single node with a L-layer GNN on a graph with average degree R will require O ( RL ) fetches . R can be large for real-world graphs , e.g . 208 for the Twitter ( Ching et al. , 2015 ) . Also , as fetching for successive layers must be done sequentially , the total latency explodes quickly as L increases . Figure 1 illustrates the dependency added by each GNN layer and the exponential explosion in the number of fetches and inference time . In contrast , the inference time of an MLP with the same number of layers is much smaller and only grows linearly . This marked gap contributes greatly to the practicality of MLPs in industrial applications over GNNs . The node-fetching latency is exacerbated by two factors : firstly , newer GNN architectures are getting deeper , e.g . 64 layers ( Chen et al. , 2020 ) , 112 layers ( Li et al. , 2019 ) , and even 1001 layers ( Li et al. , 2021 ) . Secondly , industrial-scale graphs are frequently too large to fit into the memory of a single machine , necessitating sharding of the graph out of the main memory . For example , Twitter has 288M monthly active users ( nodes ) and an estimated 60B followers ( edges ) as of 3/2015 . Facebook has 1.39B active users with more than 400B edges as of 12/2014 ( Ching et al. , 2015 ) . Even when stored in a sparse-matrix-friendly format ( often COO or CSR ) , these graphs are on the order of TBs and are constantly growing . Moving away from in-memory storage results in even slower neighbor-fetching . MLPs , on the other hand , lack the means to exploit graph topology , which hurts their performance for node classification . For example , test accuracy on Products is 78.61 for GraphSAGE compared to 62.47 for an equal-sized MLP . Nonetheless , recent results in vision and language posit that large ( or slightly modified ) MLPs can achieve similar results as CNNs and Transformers ( Liu et al. , 2021 ) . We thus also ask : Can we bridge the best of GNNs and MLPs to get high-accuracy and low-latency models ? This motivates us to do cross-model KD from GNNs to MLPs . 5 GRAPH-LESS NEURAL NETWORKS . We introduce GLNN and answer exploration questions of its properties : 1 ) How do GLNNs compare to MLPs and GNNs ? 2 ) Can GLNNs work well under both transductive and inductive settings ? 3 ) How do GLNNs compare to other inference acceleration methods ? 4 ) How do GLNNs benefit from KD ? 5 ) Do GLNNs have sufficient model expressiveness ? 6 ) When will GLNNs fail to work ? 5.1 THE GLNN FRAMEWORK The idea of GLNN is straightforward , yet as we will see , extremely effective . In short , we train a “ boosted ” MLP via KD from a teacher GNN . KD was introduced in Hinton et al . ( 2015 ) , where knowledge was transferred from a cumbersome teacher to a simpler student . In our case , we generate soft targets zv for each node v with a teacher GNN . Then we train a student MLP with both true labels yv and zv . The objective is as Equation 1 , with λ being a weight parameter , Llabel being the cross-entropy between yv and student predictions ŷv , Lteacher being the KL-divergence . L = λΣv∈VLLlabel ( ŷv , yv ) + ( 1− λ ) Σv∈VLteacher ( ŷv , zv ) ( 1 ) The model after KD , i.e . GLNN , is essentially a MLP . Therefore , GLNNs have no graph dependency during inference and are as fast as MLPs . On the other hand , through offline KD , GLNN parameters are optimized to predict and generalize as well as GNNs , with the added benefit of faster inference and easier deployment . In Figure 2 , we show the offline KD and online inference steps of GLNNs .

Summary: The paper proposes a Knowledge Distillation (KD) based approach for producing MLPs able to achieve comparable performance to Graph Neural Networks (GNNs) on node-wise classification tasks. The MLPs receive as input only the features of each target node v and it is trained to imitate the behavior of a pre-trained GNN on an extended version of the training set consisting of:  1) the original labeled set where the GNN was trained; 2) an additional set of unlabeled nodes which are equipped with soft labels produced by the GNN.  As the GNN (which operates as a teacher in this scenario) has a good inductive bias, it can learn a robust decision boundary even from a rather small training set. Such a decision boundary can then be taught (at least partially) to the simpler MLP (which operates as a student) by training the MLP on a larger training set where it learns to capture patterns in the target nodes features that correlate well with the soft labels produced by the GNN. If the GNN shows good performance over unseen condition, the MLP trained in this way ends up being more robust wrt a MLP trained only on the original labeled set as it can observe a potentially significantly wider variety of conditions at training time.    As in this scenario the MLP doesn’t receive any descriptor representing the behavior of the neighborhood of the target nodes, the overall approach doesn’t work in all possible settings (e.g. in situations where the label of a node depends exclusively on the behavior of the neighbors). Nonetheless, the authors showed in the paper that in classic scenarios (i.e. node classification tasks on citation or product networks) it is possible to effectively train a MLP able to achieve comparable performance to a GNN, while requiring a fraction of the computational cost as no convolution is realized on the given graph.