Parameter Efficient Node Classification on Homophilic Graphs
Authors: Lucas Prieto
Jeroen Den Boef
Paul Groth
Joran Cornelisse
Venue: Accepted by TMLR ()

Abstract: Deep Learning on Graphs was recently made possible with the introduction of Graph Neural Networks (GNNs). GNNs use learnable diffusion processes to propagate information through the graph and improve performance on downstream tasks. However, learning this diffusion process can be expensive in terms of memory and computation. While a lot of research has gone into making these models more expressive and able to capture more complex patterns, in practice, edges in common benchmarking datasets often encode similarity of nodes with respect to the downstream task. This property is called homophily. We argue that for these homophilic graphs, learnable diffusion processes and large receptive fields are not required to achieve competitive performance. We propose Graph Non-Parametric Diffusion (GNPD) a method that outperforms traditional GNNs using only 2 linear models and non-parameteric diffusion. Our method takes ideas from Correct & Smooth (C&S) and the Scalable Inception Graph Network (SIGN) and combines them to create a simpler model that outperforms both of them on several datasets. Our method achieves an unmatched parameter efficiency, competing with models with two orders of magnitude more parameters. Additionally GNPD can also forego spectral embeddings which are the computational bottleneck of the C&S method.

Parameter Efficient Graphs
Node
Classification on Homophilic
Lucas Prieto Socialdatabase
com
University of Amsterdam
Jeroen Den Bocf Socialdatabase
jeroendenbbee119998@mmal
Paul Groth
p.t. p.f.groth@hwvo.rl
University of Amsterdam
Joran Cornelissc Socialdatabase
joren.coruulisse wwwp://iiiissmmmmicmmmmm com
Reviewed OpenReview: https: https://openrevvaau
openreview. net/ forum? id= XXXX
Abstract
Deep Learning Graphs recently made possible with the introduction of Graph Neural Networks (GNNs). GNNs use learnable diffusion processes to propagate information through the graph and improve performance on downstream tasks. However, learning this diffusion process can be expensive in terms of memory and computation. While much research has gone into making these models more expressive and able to capture more complex patterns, in practice, edges in common benchmarking datasets often encode similarity of nodes with respect to the downstream task. This property is called homophily. We argue that for these homophilic graphs, lcarnable diffusion processos and large receptive fields not required to achieve competitive performance. We propose Graph Porammrrcct Diffusion (GNPD) hethod that perforrmss traditional GNNs using only 2 linear models and non-parameteric diffusion. Our method takes ideas from Correct & Smooth (C&S) and the Scalable Inception Graph Network (SIGN) and combines them to create a simpler model that outperforms both of them on several datasets. Our method achieves unmatched parameter efficiency, competing with models with two orders of magnitude more parameters. Additionally GNPD can also forego spectral embeddings which are the computational bottleneck of the C&S method.
1 Introduction
Pairwise relationships between entitics arise many settings in the real world. From people following cach- other on social networks, to citations between papers (Rossi & Ahmed 20155) (Bhattacharya & Getoor 2007). links between atoms in molecule (Hu al. 2020). These relationships structure which is mathematically referred to as graph with nodes connected by edges. Since graphs can be used to reflect the structure of real world data, they are increasingly used to make predictions and perform Machine Learning (MI.) in settings where relat ionships between entities are important predictors (Hu et al. 2020). This has led to a rising interest finding the best way to use this graph structure to make predictions.
be described by network
Harnessing the structure of data find local patterns has been key to the success of Deep Learning (DL) tasks images (LeCun al. 2015 (Szegedy et al. 2014] and text (Devlin et al.] 2018). However, the input

---

spaces for these tasks have clear and rigid structure. Sentences are sequences of words, and images are grids of pixels. This is not the case for graphs, where each node can have a different number of neighbors which are not ordered in any particular way. Despite the challenges that come with a more complex structure, several methods have successfully implemented deep learning over graphs with Graph Neural Networks (GNNs). Common GNN methods include Graph Convolutional Networks (GCN) (Kipf & Welling) 2016). Graph Attention Networks (Velićković et al., 2018) and Message Passing GNNs (MPGNNs) (Gilmer et al. 2017). These GNNs outperformed preexisting methods on most benchmarking datasets for machine learning graphs (Hu et al. 2020).
Despite the success of these methods, recent papers have pointed out that GNNs may have inherited some assumptions from Deep Learning, which do not hold on graphs. For example, Simplified GCN (SGCN) (Wul et al.) 2019h and SIGN (Rossi et al. 2020) showed that network depth is not needed to achieve state of the art (SOTA) performance on graphs. By using shallow networks, these methods become substantially faster while requiring less memory. Similarly, the Correct & Smooth (C&S) (Huang et al.) 2020) method outperformed many GNNs by using a graph agnostic Multi Layer Percept (MLP) combined with residual and label propagation post-processing through the graph structure. This makes it orders of magnitude more scalable than most GNN methods since the graph structure is only used once during training.
While more scalable than traditional GNNs, these methods also have shortcomings. For optimal performance C&S requires the calculation of spectral embeddings to encode the graph structure, which can be expensive for large graphs. On the other hand, SIGN trains several models for every task, resulting in millions of parameters for most datasets (Hu et al., 2020). Finally, SIGN and SGCN were the SOTA at the time of their publication but have since been left behind by newer methods.
To address these limitations we make the following contributions: 1) Re-framing scalable node classification methods as examples of non-maraettric diffusion and highlighting their similarities; 2) Introducing Graph Non-Parametric Diffusion (GNPD), a simple method which outperforms every model with a comparable number of parameters; 3) Highlighting that non-parametric diffusion methods can be competitive in the context. of node classification on homophilic graphs.
2 Related Work
2.1 Graphs
A graph g = E), is a collection of nodes vi E V and edges ea E C V x V between pairs of nodes. The degree of a node is its number of neighbors. In directed graphs nodes have an in-degree and an out-dogree corresponding to the edges going in and out of the node respectively. described by an Adjacency Matrix A E _ax= with Ag E {0, where Agg = =1 if es E E and 0 else. It is important to note that there is canonical ordering of the nodes in a graph but they are given an arbitrary order create adjacency matrix. The same ordering of nodes is used to create feature matrix X E Boxd with a vector of dimension d representing the features of each node. In ML tasks, self-loops connecting each node to itself are often added A = A + In and the Degree Matrix D with D, Ais is often used to create the normalized adjacency matrix S D -
graph with n = |V] nodes can be
2.2 Graph Neural Networks
Intuitively, GNNs aggregate information from local neighborhoods to produce more accurate node property predictions. This is done by multiplying the messages M E R of size m, to be sent from each node by diffusion operator. If this operator is the normalized adjacency matrix S, each node will receive weighted average of the messages from neighboring nodes. This process can be viewed as an aggregation with cach node receiving messages from it's neighbours, or a diffusion process with each node diffusing its message lo the rest of the network. By multiplying the message vectors by the t power of the normalized adjacency matrix M' = S the message from each node will reach any other node within & connections or less, this is called the k-hop neighborhood of a node. GNNs split this diffusion process into & GNN layers separated by non-linearities. A k layer GNN is then able to aggregate information from it's k-hop neighborhood. Tn
2

---

practice, GNNs face diminishing returns from additional layers the mumber of layers goes past 3 or 4 [Kipf & Welling 20166. Beyond the reduced benefit of adding more layers, stacking them is also expensive in terms of computation and memory which is why network depth has not been as successful for GNNs as it has in traditional deep learning methods.
The main distinction between different GNN architectures the kind of message each node sends to its neighbors. The most general approach to design a Message Passing GNN that can learn the most relevant messages to send between two nodes given their features. In practice, however, learning message vectors be expensive in terms of memory (Bronstein et al., 2021 , so the message sent by each node can be restricted to weighted copy of the features of the node. The weight of these restricted messages can be learned by gradient descent as in GAT, or lixed by some heuristic as in GCN.
GCNs use the node degrees du and dy to determine the importance of node for node u, with Cav vd While each layer of GCN is non parametric, the update function between layers is learned. This means that for GCNs of more than one layer, the diffusion process has to be repeated for every training step. The update function of a GCN layer can be written in vectorized form as Equation | for H = X and W a learnable, layer specific weight matrix.
H ((+1) "SS
(1)
For GCN with & layers, the prodicted labels are the softmax of the output from the last layer denoted
in Equation 2
Y GCN = softmar (SH (9) W*)
(2)
2.2.1 Inductive Bias In GNNs
An inductive bias allows us to encode assumptions into our models which can be useful to generalize to unseen data (Battaglia et al. 2018). Using the graph structure to change the representations of the nodes in order to get belter predictions can be viewed as the inductive bias underpiuning GNNs. The GNNs explored in this section add an increasingly strong inductive bias to the Message Passing framework. Attentional GNNs assume that each node should send weighted copy of it's current features as message and convolutional GNNs use heuristic to decide these attention weights. Choosing Cue di in Kipf & Welling (2016) presents an inductive bias which gives more importance to messages coming from nodes with fewer neighbors, while other choices for the value of Cun result in other kinds of inductive bias. Different choices of diffusson operators like triangle based adjacency matrix (Rossi et al. 2020) can also bias the predictions in different directions.
2.3 Non Non-parametric Diffusion
For all the methods described in subsection 2.2 some parts of the aggregation process must be learned through gradient descent with back-propagation. This means the aggregation process is repeated for every training step, leading to slower training and the need for graph sampling when the graph does not fit inlo memory. To avoid this, Wu et al. (2019) proposed Simplified GCN (SGCN), where the -ineliitaittes in the aggregation function are removed. This allows us to collapse the learnable matrices (' ((1) ...., ((k) into a single matrix W. So the prediction of a SGCN which aggregates information from its k-hop neighborhood becomes:
Y SGON coft SXW (w()). (w))) SGON = (offmarrS''xi
(3)
(4)
Here, the matrix S can be pro-computed, so the classifier becomes simple logistic regression model which can ignore the graph structure. This means we can minibatches to parallelize the training process
3

---

without any sampling issues. Figure 1 depicts the contrast between this approach and that of traditional GNNs.
Aggrogation
Traditional GNNs
inmmatle Upcade Functiis
-
Difusion
Repeated every training step
Prediciors
SGCN
Logase
Graph
Non-parametic
Difluiion
training
Pred
Figure 1: Schematic overview of traditional GNN methods and SGCN. In both cases the part of the method that is repeated for cach training step is highlighted by a box. SGCN omits propagation from this highlighted section, resulting in classical, graph-agnostic ML classification.
Diffusion through the graph can be identified the bottleneck of most GNN methods. Therefore, making the diffusion process non parametric to avoid having to repeat it during training has been at the core of recent attempts to make graph methods scalable (Wu et al., 2019 (Rossi et al. 2020 (Huang et al., 2020). While SGCN managed to achieve comparable performance to GCN, both methods have fallen behind the state of the art on most benchmarks. However, newer non parametric methods like C&S, SIGN, and their variants (SAGN (Sun & Wu, 2021)) have outperformed the top Iearnable diffusion methods like GAT, and currently hold top spots on node classification leaderboards (Hu al. 2020).
SIGN uses the same idea as SGCN and eliminates the lineeriiiis from the diffusion process, making it non parametric. Morcover, SIGN adds exprossivity to its method by performing diffusion with different diffusion operators and feeding a sequence of powers of these operators through different models. The outputs of these models are then aggregated in final dense layer that makes the final prediction.
Correct & Smooth takes a different approach to non-parametric diffusion, instcad using it as a pre- processing step to create features for a traditional classifier, takes the predictions from a traditional classifier and performs diffusson through the graph as a post processing step. schematic overview of both SIGN and C&S is shown in Figure These methods will be explored in more detail in subsections 2.4 and 25
2.4 Correct & Smooth
Correct & Smooth (C&S) (uang et al. 2020 uses graph agnostic classifier, like an MLP linear model to make initial prediction based the features of each node. Assuming that the classifier makes similar mistakes on connected nodes, it uses residual propagation to address this. It further assumes that connected nodes are more likely to have the same labels and uses label propagation to enforce this. This method outperforms many GNNs with a fraction of the parameters. While C&SS accurate and fast, spectral embeddings of the graph are required as feature augmentation to achieve optimal performance.
Spectral embeddings designed to be similar for connected nodes, therefore partially encoding the structure of the graph. They be obtained by performing eigendecomposition of the Laplacian of the graph, which can be computationally prohibitive for large graphs. Even with the approximation methods described in Kipf & Welling (2016), calculating the spectral embeddings is still the computational bottleneck of the C&S method.
The residuals the correction step are scaled by a cocfficient which can have a significant impact on performance for some datasets. Huang al. (2020) propose two scaling coefficients with the choice between the two hyperparameter: autoscale and fixed diffusion scaling. Autoscale scales the diffused errors to
4

---

Hi We
SIGN
H W
Hy =
Graph
Non paramenni Difusion
Repeated every training
Fredictions
S
C&S
Legaeic
Hy
Residual Propegation
Label Propegation
Prediciora
Graph
Figure 2: Schematic overview of SIGN and C&S. In both cases the part of the method that is repeated for each training step is highlighted by a box. the case of SIGN, a non-parametric diffusion is performed before learning the matrices Wo to W& and the classifier S. C&S trains a graph agnostic classifier and only then sends information across the graph, but the graph is still not used during training.
the magnitude of the average rosiduals in the training set. Fixed diffusion scaling fixes the residuals of the training nodes so they are not updated during diffusion. Fixed scale diffusion leads label leakage which is problem wants perform learnable downstream tasks after label propagation, this will be elaborated in Section 3
2.5
SIGN
SIGN can also be seen set of SGCNNs with different operators and receptive fields, and an additional dense layer to aggregate the outputs of all the SGCNs. The diffusson operators can be the normalized adjacency matrix S, as in other methods, but Rossi et al. (2020) propose additional operators based on personalized page rank triangle based adjacency matrix. The set of linear diffusion operators is given by A (!) for E (1,..,k] The set of predictions from each SGCN is described in Equation 5
p() - A"
With W the learnable weight matrix, specific to each operator. final dense layer is then learned to aggregate the outputs the different SGCN layers afler applying a non-linearity a and the concatenation operator "
Y SIGN = (P (())))
(6)
3 Methodology
Building upon the aforementioned frameworks, we propose GNPD, novel non parameetit diffusion model for learning on graphs. GNPD employs residuals and predictions akin C&S with an aggrogation model that combines inputs from the provided diffusion operators, similar to SIGN.
3.1 Our Approach
In designing approach we aim to address the following limitations of existing non- non-par parametric diffusion methods:
SGCN is scalable but has fallen behind other models in terms of accuracy (Hu al. 2020

---

2. C&S relies heavily feature augmentation with spectral embeddings which computationally expensive for large graphs.
SIGN trains a model for every power each diffusion operator, resulting millions of learnable parameters for most graphs (Hu al. 2020). It's accuracy has also fallen behind that of other methods.
To address these issues, we perform non- paraaaaetttic diffusion in the prediction space, as in C&S, but use an aggregation model to combine different diffusion processes as in SIGN. By working in the prediction space we are able to encode prior knowledge in our diffusion process through Diffusion Bias. The result is a non-parametric diffusion method which outperforms SIGN, C&S and SGCN without the need for spectral embeddings and with orders of magnitude fewer parameters than SIGN.
3.2 Architecture
GNPD consists of 4 steps described this section and depicted in Figure 3 An initial prediction made, then used perform residual and prediction diffusion. The outputs of the previous steps are then aggregated by a final model and smoothed with the labels from the training data. In this section we explain each of these steps in detail.
C1
C2
S Logatic Regression
LighIGBM
Graph
Sofl Correct Smooth
smoothing create prediclian
W
C1
Logistic Reg Reggns
Graph
Soft Correct Smooth
Final smoothing to prediction
retraining
Figure 3: Schematic overview of the two versions of our method. Both present two classifiers which are trained independently, with non-parametric diffusion of residuals and predictions in between the two. The parts the cach mothod that are repeated during training are shown in boxes. The symbol "I" represents the concatenation operator and W is a learnable weight matrix.
3.2.1 Graph Agnostic Classifier
An initial prediction is made by a simple graph agnostic classifier using only the features of each node. In this paper we us logistic regression model make these predictions. Contrary C&S, this model does not use additional spectral embeddings. The predictions therefore given by P = softmar(XW) using the previously established notation.
3.2.2 Soft Non-parametric Diffusion
The predictions from this linear model are then corrected and smoothed using our proposed Soft. Correct Smooth (SoftC&:S). This version of C&S propagates predictions in the smoothing step, instead of propagating the real labels. This allows us to train downstream classifier, the training loss longer goes zero in the

---

smoothing process. Additionally, bring C&S closer other GNN methods by reducing its receptive field. The original C&S method uses 100 diffusion steps in total, resulting in each node potentially aggregating information from 100 hops away. We found that this is only viable if the spectral embeddings are included. Excluding these, a receptive field of to 6 hops seems to be more adequate. This in line with the findings of Kipl & Welling (2016) and Wu et al. 2019) which reported similar optimal receptive fields.
While the label smoothing can be replaced by prediction smoothing to avoid using the labels, the residuals cannot be computed without accessing the true labels. This is not problematic when using autoscale, since the residuals of the training nodes also updated at every step without preserving their values from the previous step. This means the diffused residuals in the training set are representative of the residuals on the unlabeled nodes as they both arise from the same diffusion process. However, as fixed diffusion scaling treats training nodes differently from the rest, patterns found in the training data will no longer generalize the validation and test sets. This makes fixed scaling incompatible with our method in its current form.
Sof can be writlen as correction step shown in Equation followed by smoothing step shown in Equation
E = Y softmar p)
(7)
Correct = softmax (WW,) +S
(8)
Y Smooth = S Correct
(9)
3.2.3 Aggregation
We hypothesized that, while C&S fixes errors at a local level in the graph, this process must also be introducing errors, which could be addressed by looking at the dataset as a whole. We propose using an additional classifier after SoftC&:S which will be able to address the global patterns in the errors introduced by C&S. This final aggregation can be performed by any traditional classifier, in this work we propose two approaches which are depicted in Figure 3 The first is to add a gradient boasted model (LightGBM [ee et al.) 2017)) to make the final predictions, this allows us to capture complex patterns while enabling parallelization during training. We also propose a lightweight version of our method which reuses the initial text based linear model without retraining it. We change its input with another linear model which takes the output of the diffusion steps input. These two approaches are shown in Figure 3
We design the aggregation classifier Ω (3c+2)x + Rxx to take in concatenation of the original prediction P, the prediction after the correct step and the prediction after smoothing, along with node features like the in-degree din and out-degree dout of the nodes. This results in the feature matrix H shown in Equation 10 and an aggregator prediction depicted in Equation 11
H correct Y Smooth ddi (dout)
(10)
Aggregator (HH)
(11)
For the second version of our method shown in Figure B we reuse the initial classifier + along with the initial node features X. The predictions from this version our method are described in Equation 12
Y Aggregator (H)(H) + X)
(12)
3.2.4 Final Smoothing
Finally we perform additional smoothing steps, this time including the real labels for the training set, as no further classifiers need to be trained on this data. For the data from the validation and test sets, the true labels are not available the predictions P are used.
7

---

3.3 Diffusion Bias
An advantage of performing diffusion in the prediction space is that predictions can be easily interpreted as the probabilities assigned by the model of each node belonging to each class. Each node is represented by a probability distribution and aggregates information from the classes assigned to it's neighbors. The interpretability these representations allows us to introduce our background knowledge into the diffusion process what we call Diffusion Bias. We do this one specific way which is relevant for most real world applications, but this can be extended by using dataset specific insights.
3.3.1 Class Specific Homophily
We observe that, for most graphs, homophily is not homogeneous across classes. For example, in graph citation networks, some paper topics can lend themselves to citations from diverse set of rescarch areas, while other research areas predominantly cite papers with the same topic. To reflect this, we add class specific homophily coefficient based on the External-Interaa (EI) homophily index (Coleman 1964). This index looks at Internal edges which connect nodes of the same class, and External edges which connect nodes of different classes, the formula for the index is shown in Equation
13
The effect of adding class specific
homophily is explored in Section
E1 External Internal
(13)
External + Internal
We use the training set to calculate the ET. index of each class c, an index of -1 corresponds to complete homophily while an index of 1 corresponds to the opposite (heterophily). We then set the importance of messages to be proportional to - AEI. with À E [0, as a hyperparameter. As described in Subsection 2.2.1 GCNs and SGCNs use heuristic to set the importance e of node u for node u, Cuv = Vd.d," Class specific homophily modifies this heuristic with a different inductive bias using Cuo 1-XEL, instead. vded,
3.4
Relationship to SGCN
We argue that the smoothing used in our method and in C&S (Equation 15 has common features with SGCN (Equation 14]. The main difference between the two is whether the weight matrix W is learned before or after the diffusion process. SGCN aggregates node features and then makes a prediction with a logistic regression model whereas our model, like Lincar C&S aggregates the predictions of a logistic regression model.
Y SGON = softmar( XxXW)
(14)
Ý Smooth = Sh softmar XW
(15)
3.5
Relationship to SIGN
A SIGN model with & distinct operators also requires l models like the one described in Equation 2 with k distinct learnable W matrices. By using the kind of smoothing described in Equation 15 the W, matrix is learned before diffusion in GNPD, meaning the same matrix can be used for every diffusion operator. This allows our model to combine different kinds of diffusion operators without increasing the mumber of parameters. In our current method, we perform two kinds of diffusion, prediction and residual diffusion. However, our approach could be extended to include other diffusson operators, like the ones used in SIGN [Rssii et al. 2020), while keeping the same amount of learnable parameters.
3.6 Relationship C&S
Our method uses the same kind of residual and prediction diffusion as C&S, with some significant changes. By smoothing predictions instead of the true labels, stop label leakage and allow for training of

---

downstream model. This allows us to treat correction and smoothing different diffusion operators for which we concatenate the outpuis as input for an aggregator which makes the final prediction. Once this final prediction is made, we can make use of the training labels in a last smoothing step. Our method is also different in the lack of spectral emboddings and the inclusion of class specific homophily.
4 Experimental Setup
4.1 Datasets
Dataset arxiv
Nodes
Edges
Classes
Train Val Test
169,343
1,166,243
40
54% 10%% 18% 28%
Products
2,449,029
61,859,140
47
10% 00/6/2%/88/ 88%
Pubmed 19,717
44,338
92% 92%/3%/52 1096//55//300
Citeseer 3,327
4,732
6
Table Summary statistios for the datasets used in this paper.
The statistics of the datasets used in this paper are described in Table !
4.2 Additional Text Features
As described in Subsection 4.1 most graph benchmarking datasets come with text embeddings to represent the node features instcad of using the text corresponding to each node. This equalizes the information that each method can extract from the text and puts the focus learning from the graph structure. However, recent methods (Chien et al. (021) have started making use of additional text features to learn better node representations, outperforming any models that do not use these additional features {Hu et al. 2020). In this work we evaluate our models on all datasets in the originally intended setting, without additional text features. However, we also use the text of the papers generate better text encodings for ogbn-arxiv using sentence transformer (Reimers & Gurevych] 2019). This allows us evaluate how much our mothod can benefit from additional lext features and how it compares to state of the art ensemble models based on GIANT (Chien et al. 2021) which are designed to make use of these additional features.
5 Results
Performance metrics and model parameter counts for experiments on ogbn-arxiv without additional text features are denoted in Table 2 Our method is compared to the SOTA and the architectures it draws inspiration from, SIGN and C&SS. Notably, our method outperforms SIGN, all iterations of C&S and achieves close to SOTA performance with only 1% of the parameters. This parameter elficiency is highlighted in Figure
4a
Table compares GNDP performance on ogbn-arxiv using additional text features to that of the overall SOTA ensemble using this feature augmentation. This inclusion boosts our test. accuracy to 74.76 + 0.03%. depicts the contrast in paramcter efficiency between GNDP and the overall best performing architectures on ogbn-arxiv.
Figure
4b
In Table evaluate our method on three additional datasets. We see improvement over baselines in Pubmed and Citeseer, but not on ogbn-products. While our method outperforms the autoscale version of C&S along with the other baselines, it does not outperform C&S with Fixed Diffusion (FixedDifl) Scaling on ogbn-prooe products...s. FixedDiff makes a minor difference on every dataset reported in Huang et al. 2020 except for ogbn-products where it leads to an increase in accuracy of 5.58%. Since our method incompatible with FixedDiff Scaling, it cannot benefit from this boost in performance and therefore only outperforms the autoscale version of C&S ogbn-produets.
9

---

Method
Validation Accuracy
Test Accuracy
Parameters
MLP + C&S
72.42% #0.06
71.04% #0.09
85,323
Linear + C&S
73.00% #0.01
71.18% #0.01
15,400
GCN
73.00% #0.17
71.74% #0.29
110,120
SIGN
73.23 + 0.06
71.95% #0.11
3,566,128
Linear + C&S + embeddings
73.68% +0.04
72.22% #0.02
15,400
MLP + C&S embeddings
73.91% 10.15
73.12 #0.12
175,656
GNPD (Linear)
74.49% #0.06
73.51% #0.04
15,400
GNPD (LM)
74.68 +0.08
73.72% +0.04%
AGDN+BoT=self-KDD
75.19 +0.09
74.31 #0.14
1,513,294
Validation and test metries ogbn-arxiv. Reported accuracy scores averages over 10 runs along with the corresponding standard deviations. Best performing results are depicted in bold, while the best results out of our basclines are underlined.
Table 2:
Method GLANT-XRI-DRGATE
Additional text
Validation Accuracy
Test Accuracy
Parameters
Yes
77.25% #0.08
76.33 #0.06
2,685,527
GNDP (Linear)
Yes
76.20%
:0.04
74.76% 10.03
15,400
Yes
75.95% #0.11
74.35% #0.14
546,344
No
75.19 +0.09
74.31 #0.14
1,513,294
Table 3: Accuracies over 10 runs and number of parameters on ogbn-arxiv using additional text features. We also include the SOTA model without text features to highlight the importance of better node features over complex aggregation methods in homophilic graphs.
For ogbn-products, the SOTA model listed in Table corresponds to CAMLP-RLU-SCR+CC ensem- ble model with 3,335,831 parameters. For Citeseer and Pubmed, the SOTA models are currently versions of Snowball (Luan et al. 2021
With respect to model runtimes, avoiding the use of spectral embeddings results in a substantial increase in efficiency, with our model scaling belter larger datasets than methods using spectral embeddings, like C&S. The benefits of spectral embedding removal due increasing computational cost for larger datasets is highlighted in Figure 5a
6 Analysis
Our results show that on common benchmarking datasets, complex diffusion processes with millions of learned parameters are not needed achieve competitive performance. In fact, we achieve this without any learnable parameters in the diffusion process. This suggests that in practice, for many tasks in the real world, edges encode similarity, making smoothing in the prediction space very effective, even without learnable weights. We show that in this setting, pairing a simple method like GNPD with improved node features has a bigger impact on accuracy than using complex models to find patterns the graph (Table Even without additional text features, our method competes with models that have orders of magnitude more parameters. By making diffusion non-parametire, the number of parameters in our method does not depend on the size of the graph, but only on the size of the node features.
6.1 Ablation study
In this section we measure the importance of the different components of our method. While most of the hyper-parameters in this method inherited from C&S, introduced the X parameter to regulate
10

---

Classification Accuracy (%)
BG
Classification Accuracy (%)
77.0
GIANT-XRT-
GAT+l reuse
*DD
76.0
DIANT-XRT RevGAT
GNPD (Linear)
73.5
73.0
75.0
GNPD (Linear)
74.5
GIANT-XRT+ GraphSGEE
GIANT GATV2
72.5
Unear+C&5
73.5
72.0
SIGN
GIANT-XRT
10a
10a
10'
Number learnable parameters (Log)
Number of learnable parameters (Log)
(a) Parameter elficiency of top methods without
(b) Parameter efficiency of top methods using
additional texl. features.
additional lext features.
Figure 4: Parameter efficiency of the top methods on the ogbn-arxiv leaderboard with and without using additional text features. A regression line added show the exponential relationship (linear log scale) between parameters, and accuracy.
Method SGCN
Pubmed
Citeseer
ogbn-products
84.04
72.04
GCN
88.13
73.68
75.64
Linear+C&S autoscale
89.99%
76.31%
80.20%
Linear+C& FifedDiff
89.74%
76.22%
82.54%
MLP+ +C&S fembedding autoscale
89.33%
76.31%
78.60%
MLP C FixedDiff
89.23%
76.42%
84.18%
GNPD (Linear) autoscale
90.02%
+0.04
79.32%
+0.21
82.80%
#0.08
SOTA
91.44% +0.59
82.07% +1.04
85.20%
#0.08
Table 4: Test accuracies averaged over 10 runs with the corresponding standard deviation when available. SOTA results are shown in bold, while the best results out of our baselines are underlined.
class specific homophily; we show the sensitivity of our method to this hyper-parameter and the number of diffusion steps & in Table 5
Our method can be split in four parts, described in Scction 3. show how much each of these parts contribute to the final accuracy in Table G We see non-parametric diffusion drastically increases the accuracy of the initial Linear model, from 52.32% to over 70%, which higlights the homophily of the graph. Interostingly, we see accuracy goes down after the aggregation step. We hypothesize that this is due to the graph agnostic nature of the aggregator which makes it unable to fully preserve the smoothing of the previous steps. Therefore, the improvements made by the aggregator are initially outweighed by giving up on the smoothness of the prediction. However, after reintroducing this smoothness with the final smoothing with labels, we gel an accuracy of 73.50%. By comparison, if skip the aggregator and go directly to the final smoothing with labels, the final accuracy drops to 69.09%. This happens because the output of the "Correction and Smoothing' step is alrcady smooth, and smoothing it further leads to oversmoothing, while the output of the aggregator is not smooth and therefore does benefit from smoothing.
6.2 Limitations
While GNPD is very simple model with unmatched parameter efficiency, the implications of this for scala- bility and speed are not thoroughly explored in this paper. We show that, by foregoing spectral embeddings,
11

---

Execution times for ogbn-arxiv
Execution times for ogbn-products
160
Diffusion time
Diffusion time
Classifier Training
4000
Classifier Training
140
Spectral embeddings
Spectral embeddings
120
3000
100
Time (s)
Time
80
2000
40
1000
GNPD (Linear)
C&S embeddings
GNPD (Linear)
C&S + embeddings
(a) Compared run-times on ogbn-arxiv.
(b) Compared runtimes on ogbn-products
Figure Compared runtimes of GNPD (Linear) and the closest version of C&S in terms of accuracy SULL-+CC&Seemmedddinni averaged over 10 runs on datasets. We see the spectral embeddings are the most computationally expensive part of C&S and that by removing them, our model divides the training time by in the case of ogbn-arxiv and by 10 in the case of ogbn- produutrodutt....
Method
X
Test Accuracy on arxiv
k
Test Accuracy arxiv
C&S +Lnnear
71.26 #0.01
50
71.26 =0.01
C&S + Linear embeddings
72.22 +0.02
50
72.22 #0.02
72.65% +0.08
51.95 +0.6
0.1
72.87% +0.7
72.52 +0.7
0.2
73.19% +0.08
72.13 +0.7
GNDP (Linear)
0.3
73.46% +0.06
73.50 +0.7
0.4
73.50% +0.07
4
72.98 +0.8
0.5
73.46% +0.14
73.20 +0.6
0.6
73.32% 10.08
6
73.02 10.7
Table 5: Sensitivity analysis with respect to À and k (the number of diffusion steps). The results averaged over 10 runs and shown with standard deviation.
our model is substantially faster than C&S, particularly large graphs. However, more comprehensive comparison to the run-times, time and memory complexities of other methods would be required to make confident claims about the scalability of this method. We also want to emphasize that this method is only tested and expected work in homophilic graphs.
7 Conclusion
We introduced GNPD, a novel method for non-parametric diffussio,, designed to enable parameter efficient node classification while improving accuracy over existing non-parametrie diffusion models. In doing this, grounded the C&SS method in the same theoretical framework other GNNs, allowing us to draw from insights in the GNN literature to design our method and forego the computational bottleneck of spectral embeddings. Our work shows that. graph agnostic linear models can be alternated with non-paramelric diffusion to achieve competitive results with unmatched parameter efficiency on homophilic graphs. Further work could explore the possibility of adding different diffusion operators to our method as SIGN (Rossi) et al.) 2020), since our method makes possible to do this without multiplying the number of learnable parameters.
12

---

Test Accuracy arxiv
GNPD
Linear
Linear + C&S embeddings
Linear model output
52.32%
52.32%
52.32%
Correction and Smoothing
71.5%
71.26%
72.22%
Aggregation output
69.74%
Final Smoothing with labels
73.50%
Table 6: Ablation study of the different parts of our method compared to C&S + Linear with and without embeddings. Note that the Smoothing step uses predictions in the case of GNPD and labels the case of C&S.
References
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzaler, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, abs/ 06,661 2018. URL http:://ar::ti wttp:///.wrrrr////////////////// 01261
Indrajit Bhattacharya and Lise Getoor. Collective entity resolution in relational data. ACM Trans. Knowl. Discov. Dala, 1(1):5 es, mar 2007. ISSN 1556-4681. doi: 10. 10.1145/1217290.127773 URL https://dol
org/ 10. /114/1211/199 1217304
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. CoRR, abs abs/2104.13448, 2021. URL httpss://axiii http://xxvv..or/abbaae/ /
2104. 13478
Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovic, and Inderjit S Dhillon. Node feature extraction by self-supervised multi-scale neighborhood prediction. arXiv preprint arXiv:211.00006, 2021.
James Samuel Coleman. Introduction to mathematicnl sociology [by] James S. Coleman. Free Press of Glencoe [New York), 1964.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirec- tional transformers for language understanding. CoRR, abs abs/1110.04805, 2018. URL http:///rxii orrg/
abs/1810 04800
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. CoRR, abs/1774.01122,, 2017. URL http:://txxii....t//// wwwpoo/////.......//0 01212.
Weihua Hu, Matthias Foy, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.000m 2020.
Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R. Benson. Combining label propagation 2020. URL https:
and simple models out-porforms graph neural networks. CoRR,
//axxiv.or. attgaaas//0...1000/00
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yar Liu. Lightgbm: highly efficient gradient boosting decision tree. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpp::/pproceedings www.ine.ucccccoooo
oar775/a4444paa0022224444e222222 pdd
13

---

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. CoRR, abs/1600.029900 02907, 2016. URL
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, (21(653)) 2015.
Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. Is heterophily A real mirhtmare for graph neural networks to do node classification? CoRR, abs/ abss22109005411, 2021. URL
05641
Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistios, 11 2019. URL
10084
Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael M. Bronstein, and Federico Monti. SIGN: scalable inception graph neural networks. CoRR, abs/2004.11198,, 2020. URL https:
//axxiv.org/abb/200..1 11198
Ryan A. Rossi and Nesreen K. Ahmed. The network data repository with interactive graph analytics and visualization. In AAAI, 2015. URL httpppss/haatworrrrepssss.o/.. com
Chuxiong Sun and Guoshi Wu. Scalable and adaptive graph neural networks with self-labbeeehhaeeed train-
ing. CoRR, 0is/2104.09377 2021. URL
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanct, Scott E. Reed, Dragomir Anguclov, Dumitru Er- han, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR, abs/ hbs/1409.4442 2014. URL
org/ herg/abs/100 1409.4109.40 4842
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liø, and Yoshua Bengio. Graph attention networks, 2018.
Felix Wu, Tianyi Zhang, Amauri Н. Souza Jr., Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. CoRR, abs/1192.07113, 2019. URL http://arxw..org//////// 1902. 07153
14


Reviews:
[{"review_id": "xh5aqbfTlF", "reviewer": "Reviewer_xc7N", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The biggest strength of the method is its parameter-efficiency which is most clearly evident in Figure 4 where GNDP can achieve satisfactory accuracy (but lower than SOTA) with orders of magnitude fewer parameters.\n- The simplicity of the approach is another strength. \n\n- A big weakness of the proposed approach is the lack of an ablation study. For example the authors introduce several additional modifications such as correcting for class-specific homophily (3.3.1) and confidence (3.3.2). It is not clear how important these additions are. Moreover, would adding these modification to the baselines also improve their performance?\n- Similarly, the final classifier concatenates several different sources of information (Eq. 12) and again it is not clear which of these is crucial for obtaining good performance.\n- Another weakness is the lack of a sensitivity analysis. and how sensitive is the method to the hyperaparameters, e.g. $\\lambda, s, k$, etc.\n- The paper lacks a discussion of the limitations of the proposed approach. For example, similar to the methods it is inspired by, its good performance is likely highly dependent on the degree of homophily.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Inspired by the success of C&S, SIGN, and SGC the authors propose GNDP that aims to integrate the stregths of the aforementioned approaches while avoiding their weakness. Similar to other \"decoupled\" GNNs, the diffusion and feature processing steps are separate from each other which allows for efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I propose that the authors include a detailed ablation study as well as sensitivity analysis w.r.t. the hyperparameters.\n\nIn addition I think the paper will benefit from a rewrite to reduce the amount of expository explanation. Currently a large portion is dedicated to explaining the background and the baseline methods. These sections can be written significantly more succinctly, which will help better highlight the contribution of this paper.\n\nOverall, while I believe that the simplicity of the approach is a strength, given the lack of technical contributions, I suggest the empirical analysis to be straightened by including the above suggestions, as well as more detailed analysis of the limitations and performance of the method on different types of graphs (e.g. graphs with heterophily). ", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tNCuaNi8ny", "reviewer": "Reviewer_RaZ3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n(1) The main idea is clearly written. \n(2) Faster runtime of the proposed method compared to existing methods. \n(3) Performance boost is okayish compared to C&S and SIGN. \n\nWeaknesses: \n(1) The proposed method is conceptually very similar to the Correct and Smooth(C&S) approach by Huang et al. Pretty much the only differences being the altered sequence of weight learning and smoothing procedure (proposed method also don't use spectral embedding).\n\n(2) The results section is unstructured. For example, in table 4 the comparisons with C&S+ Linear is missing which is very relevant. See requested changes section. \n\n(3) Effects of steps like 3.2.3 and 3.2.4 are not studied properly. See below.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new aggregation scheme aimed towards decreasing number of learnable parameters in GNNs. Towards this they propose a framework which roughly works in four stages: (1) Initial prediction using a logistic regression model (2) Label correction and smoothing using averaging over neighbors (3) Re-prediction using smoothened predictions using another model (4) Final smoothing using neighborhood averaging. Because smoothing steps are decoupled from weights learning steps, number of learnable parameters doesn't depend on the diameter of averaging and thus results in significantly fewer number of parameters compared to traditional GNNs. \nThe proposed method achieves close to SOTA results on 4 datasets. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(1) The authors should conduct ablation studies to measure impact of the third step (referred to as the aggregation in 3.2.3) and final smoothing (section 3.2.4). \n\n(2) Comparison with other relevant methods for example: C&S+ Linear  is only reported in Table 2 ogbn-arxiv, while missing from Table 4. Please include them for other datasets. \n\n(3) If the authors can demonstrate the effectiveness of the proposed approach on other modality dataset/task (visual, audio) that will greatly strengthen the experiment section. \n\n(4) Corresponding to figure 5, also include a table that compares total runtime of different methods. ", "broader_impact_concerns": "None. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EXCmubEVMY", "reviewer": "Reviewer_MXvx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### strengths\n\n1. The task to use low amount of parameters in node classification models is interesting.\n2. The proposed method mitigates the high cost of spectral embedding in C&S.\n\n### weaknesses\n\n1. The experiments of the proposed **scalable** node classification method on large-scale graph datasets are insufficient.\n\t1. There are some large graph datasets, e.g., ogbn-papers100m, ogbn-mag240m can be considered to show the scalability of the proposed method\n\t2. Though the number of model parameters is reported, the time complexity, running time, memory consumption are also expected to support the claim of **scalable**.\n2. The design of combination of C&S, SIGN, SGCN seems quite ad-hoc and straighforward. More theoretical analysis or insights of the proposed method are expected, e.g., why the non-parametric methods can achieve comparable performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies an interesting problem, i.e., using non-parametric diffusion in graph node classification tasks, to make the model scalable. \n\nThe proposed model shows better accuracy performance than SGCN and less computation-consuming than SIGN and C&S.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Experiments\n\t1. Experiments on larger graph datasets are expected to show the scalability.\n\t2. The computation complexity/memory consumption/running time are expected. (Only comparing the running time with C&S is insufficient)\n2. Methods\n\t1. More theoretical analysis or insights of the proposed method are expected, e.g., why the non-parametric methods can achieve comparable performance.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As noted in my comments on claims and evidence, the key claims of the paper, #1 and #2, are well supported.  There were concerns about claim #3 and concerns about the claim of “scalability”, which I believe could be addressed by the authors with a new revision.

All reviewers lean on the rejection side for this paper, potentially due to limited novelty.  But I found the paper hard to reject according to the TMLR acceptance criteria, as it passes both the claims and evidence threshold and the audience threshold.  Therefore I’m recommending an acceptance but request the authors to revise the paper further, in particular addressing the concerns highlighted in the claims and evidence comment.

I’d also encourage the authors to consider one reviewer’s suggestion about the limitation of the proposed approach, which would help make the paper better.
Decision: Accept (poster)