Personalized Layer Selection for Graph Neural Networks
Authors: Kartik Sharma
Vineeth Rakesh
Yingtong Dou
Srijan Kumar
Mahashweta Das
Venue: Accepted by TMLR ()

Abstract: Graph Neural Networks (GNNs) combine node attributes over a fixed granularity of the local graph structure around a node to predict its label. However, different nodes may relate to a node-level property with a different granularity of its local neighborhood, and using the same level of smoothing for all nodes can be detrimental to their classification. In this work, we challenge the common fact that a single GNN layer can classify all nodes of a graph by training GNNs with a distinct personalized layer for each node. Inspired by metric learning, we propose a novel algorithm, MetSelect, to select the optimal representation layer to classify each node. In particular, we identify a prototype representation of each class in a transformed GNN layer and then, classify using the layer where the distance is smallest to a class prototype after normalizing with that layer’s variance. Results on 10 datasets and 3 different GNNs show that we significantly improve the node classification accuracy of GNNs in a plug-and-play manner. We also find that using variable layers for prediction enables GNNs to be deeper and more robust to poisoning attacks. We hope this work can inspire future works to learn more adaptive and personalized graph representations.

Published in Transactions on Machine Learning Research (04/2025)
Personalized Layer Selection for Graph Neural Networks
Kartik Sharma
ksartik@gatech.com edu
Georgia Institute of Technology
Vineeth Rakesh Visa Research
vinmohan@rrisa. com
Yingtong Dou Visa Research
yidou@erisa com
Srijan Kumar
srijon@gatech.. edu
Georyia Instilute of Technology
Mahashweta Das
mahdas@vias com
Visa Research
Reviewed on OpenReview: https: // openreview. net/ forum? id- JyJTJA992
Abstract
Graph Neural Networks (GNNs) combine node attributes over a fixed granularity of the local graph structure around node predict its label. However, different nodes may relate to node-level property with different granularity of its local neighborhood, and using the same level of smoothing for all nodes can be detrimental to their classification. In this work, we challenge the common fact that single GNN layer can classify all nodes of graph by training GNNs with a distinct personalized layer for each node. Inspired by metric learning, we propose a novel algorithm, MetSelect, to select the optimal representation layer to classify each node. In particular, we identify a prototype representation of each class in a transformed GNN layer and then classify each node using the layer where the distance smallest. to class prototype after normalizing with that. layer's variance. Results 10 datasets and different GNNs show that significantly improve the node classification accuracy of GNNs in plug-and-play manner. We also find that using variable layers for prediction enables GNNs to be deeper and more robust to poisoning attacks. We hope this work can inspire future works to learn more adaptive and personalized graph representations.
1 Introduction
Graph Neural Networks (GNNs) (Hamilton et al. 2017 Kipl & Welling 2016) show superior performance various node-level and graph-level tasks for real-world applications, including fraud detection (Tang et al.) 2022) and online content recommendation (Ying et al. 2018) A GNN of depth L iterrtively learns L different representations by combining a node's attributes with its local neighborhood Gilmer et al. (0117) and then use the final layer representation and their underlying property may correlate with distinct granularity of each node's neighborhood. For instance, consider the task of predicting the political ideologies (Ée., conservative or liberal) of users in social network. Users tend to form echo chambers by interacting with people of similar ideologies (Cinelli al. 2021), whereas some doubtful users may interact with diverse idcologies while lcaning towards one (Boutyline & Willer 2017). While the neighbor information may be useful to accurately classify the users within an echo chamber, it can confuse a classifier for the users that interact with both chambers where using just the
classify each node. However, nodes in graph differ from each other
Work done during internship Visa Research

---

Published in Transactions on Machine Learning Research (04/2025)
Existing GNNs (Fixed Prediction Layer)
GNNs with Personalized Prediction Layer
Heterophily performance
Depth stability
100
Frund
Lager
40
Lagar
32
Predics
Depch
Robustness
Node-leve personalization
a5
Layer
Fentraatia
20
10
Layar
Figure 1: GNNs with personalized prediction layer can empower existing GNNs for more accurate, personalized, robust, and deeper node classification (e.g., classifying unlabeled users (gray) as conservative (purple) or liberal (green)). For example, while some nodes benefit from their neighbors, others may be better off by predicting using only their attributes.
users' traits may be better. Figure ! highlights the importance of using a personalized layer instead of a fixed layer of a GNN with this example.
We hypothesize that selecting personalized GNN layer has wide-ranging implications. Existing GNNs are known to suffer from the problem of oversmoothing, where node representations become indistinguishable due to convolution NNt & Machara 2019). This leads to poor performance on datasets where labels are typically associated with lower-level features. Existing works have focused on handling this issue by either enhancing the representations with higher frequency channels (Luan et al.) 2022: Chien et al., 2020 Nt & Machara
2019 Eliasof et al.) 2023 Zhao Akoglu 2019), continuous neural diffusion processes (Bodnar et al. 2022
Di Giovanni ct al.
2022
by rewiring the underlying graph (Rong et al. 2019 Gutteridge et al.) 2023
However, we note that one can enhance the accuracy of existing GNNs by simply using mode-personalizge representation for prediction instead of the final representation. We can also circunvent the problem of loss in performance in deeper GNNs (Li et al., (0118) by identifying the optimal layer for each node and thus, ignoring the node reprosentations at higher depths if they don't help to classify that node. Using personalizod GNN layer also boosts their robustness to untargeted training-time perturbations (Zügner el. al., 2018 Sun al. 2020 Xu et al. 2019). These perturbations are carefully crafted to reduce the overall accuracy of the GNNs over given dataset but their effect will be limited since can adaptively select the layer that is useful for classification and ignore any redundant information that may be perturbed. Finally, these improvements can be provided to any GNN in a plug-and-play manner while preserving their analytic simplicity by personalizing the selection of these representations.
In this work, we first formalize the problem of finding the optimal GNN layer for each node to solve the transductive node classification with a given GNN. Then, we propose a novel method, called MetSelect that leverages Metric learning to select the optimal layer in an efficient manner. Here, we find a prototype representation of each class in transformations of each GNN layer and then compare the metric distance of a node to these prototypes across layers to find the optimal layer. To train the models, we minimize the cross entropy loss of the decoded probabilities from the optimal layers for each training node while minimizing their distance to the true class protolype in the transformed space. Through experiments on 10 datasets and GNNs, we show that MetSelect boosts the node classification performance by up to 20%, especially on heterophilic datasots. We also find that MetSelect enables deeper GNNs by preserving the accuracics for as high as 10 layers. GNNs trained using MetSeloct are also found to be robust against untargeted poisoning attacks (Zügner al. 2018), giving up to 100% enhancement, for different graph neural networks.
2

---

Published in Transactions on Machine Learning Research (04/2025)
2 Related Work
Layer aggregation. Layer aggregation methods such as JK-net, try to learn personalized representations by aggregating different representations through pooling operations and/or skip connections for node classifi- cation (Xu et al., 2018b Dwivedi et al.) 2023; Rusch et al.) 2022). However, they do not adaptively select a single layer for classifying a given node but rather merge the representations for all nodes the same way. This reduces the interpretability and increases the complexity of the underlying network by predicting through non-linear pooled transformation of the GNN representations. On the other hand, the simpler layer selection of Met preserves the interpretability and the analytic simplicity of the underlying GNN as the predictions are still made through the GNN embeddings. Furthermore, layer aggregation methods require all representations to lie in the same space and to have the same dimension while selecting optimal layers through MetSelect completely relaxes such assumptions.
Alternative GNN architectures. Based on the observations that a GCN can only approximate a low-pass filter (Nt & Maehara 2019; Di Giovanni al. 2022 Yan et al., 2022), new models have been proposed to
address the heterophilic (Pei et al.) 2020 Luan et al. 2022; Eliasof et al. 2023 Chien et al. 2020: Rossi
ct al. 2024) and closely-related oversmoothing (Bodnar et al. 2022 Yan et al. 2022 challenge. However,
these models achieve these enhancements by modifying the underlying message- paatnnggpassingggggg architecture, while we present a plug-and-play approach to show that these issues can possibly be addressed by adaptive layer selection. NDLS (Zhang et al. 22021] also finds node-specific smoothing level, their method is based on a simple feature smoothing model and is agnostic of the underlying GNN and the label distribution. Furthermore, while they hypothesize that the optimal layer is the one at which the influence of other nodes is similar to the stationary influence, we hypothesize that given GNN, it is the one that minimizes the distance from the class protolype in the corresponding embedding space.
Alternative training of GNNs. Alternative training strategics have been proposed to support larger
graphs (Gandhi & Iyer| 2021 faster training (Cai el al.] 2021), ,, longer distances (Li el al. 2021 Gutteridge
jtt al.] 2023 Rong et al. 2019), robustness (Gosch et al.j 2023), and a joint structure and embedding learning (Jin et al.) 2020 Zeng et al., [021]]. : However, none of these approaches focus on our problem of selecting the optimal personalized layer and instead focus on increasing either robustness faster convergence.
Zeng et al. (2021) decouples the depth and the scope of the GNNs to consider a neighbor multiple times in the message passing by using GNN of depth greater than the neighborhood scope of that node, On the other hand, our objective here is to find an optimal representation depth L' strietly less than the full scope L that best predicts the node's class.
Neighbor importance. Certain methods aim to identify useful neighbors for each node separately to better classify them. Self-interpetablee (Dai & Wang, 2021 Zhang et al., 2022 Feng et al., 2022) and attention-basse (Veličković et al., 2017 Shi et al. 2020 Brody et al. 2021 graph neural networks weigh nodes based on the similarity between different nodes in the whole graph or the local graph structure. While this implicitly weighs each neighbor differently, the representations of all nodes are still formed from a fixed number of gressage-passing steps. Thus, these are still susceptible to oversmoothing and are not scalable as opposed to their graph neural network counterparts. Instead, we provide alug-and--lay improvements to the-offthe-shelf GNNs by simply presenting a way to select the optimal smoothing level for each node.
Metric Learning. Our strategy to choose the optimal layer is inspired by prior works in the image domain
to learn embedding spaces by forming multiple clusters per class (Rippel et al. 2015 Deng et al.] 2020).. While it is prevalent to use distance- baasance-basedd classifiers in images (Deng et al.) 2019 Salakhutdinov & Hinton 2007
for learning robust embeddings, they have not been explored in the context of graph neural networks before. In this work, we use the embedding distance from class clusters to do personalized node-level smoothing.
3 Background
In this work, we study the problem of transductive node classifieation, where we are given a partially labeled graph and we want to learn the labels of the remaining nodes. More formally,
3

---

Published in Transactions on Machine Learning Research (04/2025)
Problem 1. (Transductive Node Classification) Given graph G= (X, (V,E)) with node attributes X, modes V, edges &, and set of labeled nodes (w, y) E Ver V,y E ))) the objective is to learn a function fw(() to predict the label y E y of each node u E V.
We then consider encoder-decoder architecture of fw(() := g+ o h-)+) E [0, 10 that predicts the probability of belonging to each class using a (trainable) decoder function ge. The encoder he is message- passing Graph Neural Network (GNN) Kipf & Welling (2016); Gilmer et al. (2017) of depth L that encodes
each node vs as (w:) hn()) E Rd defined recursively as:
(1+1)_ Update(h) (0) Agg( (msg(h) (1) nl"): (uy, v.) E E))),
h
(1)
where Update(-), Agg(-), msg(-) are learnable functions, h X, and the 1-th layer h is node represen- fation formed using the neighbors at distance of at most. I hops from that node, i.e., (v:).
4 Problem
GNNs aggregate the L-hop information of each node to learn their label-separating representations. However, a major limitation stems from the fact that the same neighborhood scope L is used to classify all the nodes. As shown in Figure D different nodes may require a different granularity of the local information to be classified properly, and providing more information could unnecessarily smooth out the classifying signal. To predict the user ideologies in a social network, nodes within an echo chamber may benefit from their 1-hop neighbors. However, classifier may be better off ignoring the neighbors of the nodes that interact with both chambers and just using the attributes (0-hop). To capture such intraclass differences, we propose novel problem that aims to classify a node using the optimal layer for each node. In other words,
Problem 2. dosdea-ersonalizeed layer selection in GNN) Given graph G = (V,)) and GNN encoder he()) of depth I., the objective is to identify a personalized layer 1" (v) E [0, L.] for each node v E V such that h( (v)) (v) best predicts the true label y(v).
As detailed in Section 1 this problem of selecting representations is important to study since it allows us to
1. Increased accuracy through personalized node-speeific smoothing level.
2. Enables deeper architectures by disregarding over-smoothened representations.
3. More robustness against adversarial attacks as it ignores suboptimal layers.
4. Plug-and-play improvement irrespective of the underlying GNN architecture.
5 Methodology
While the original GNN learns decoder (+()) over h(t) to decode the labels, we compare the L + 1
representations formed by different layers of the encoder, i.e., (h (0) (1) .... h (n( and pick the one most suitable to classify given node. We call our method MetSelec as we employ Metric learning to select
the node-optimal personalized layer to classify nodes. Figure illustrates method with an example of the transductive classification of graph.
As noted in Equation each layer of GNN h() represents the nodes in distinct ways (typically, using the I-hop neighborhood). Thus, one can separate the nodes based on their labels in each of these spaces. Then, we propose to find the best layer 1' (v( for a node based on how confidently that layer predicts its class. To compare the confidence of the prediction from different representation spaces, we adopt distance-basod class decoding and use the distance as the notion of the confidence (Salakhutdinov & Hinton 2007). However, distance-based decoding can distort the softmax decoding that we do to predict the labels from these layers. Thus, we propose to use a transformed representation h (1) = wh() + b to find the best layer l".
Code will be open-sourced after publication.
4

---

Published in Transactions on Machine Learning Research (04/2025)
U Pn
Training (()) arg m
h
Normalized distance
Optimal layer /* for each node
(h 81
Pe
J (0.) )
h())
B
Inference
KDj
(1a) arg min mwofttnn 2
Graph
GNN
X
K(0
Figure 2: GNN with node-optimal prediction layer trained using MetSelect. Labeled nodes are denoted in purple and green while gray denotes unlabeled nodes.
Distance-based classifiers Rippel et al. (2015); ; Deng et al. 20199 2020 Salakhutdinov & Hinton (0077)
typically find a prolotype embedding of each class and then classify the input the class whose prolotype is the closest in the embedding space, For instance, for layer I, the prototype of each class can simply be given the mean embedding of the training nodes belonging to that class, i.e.,
Me
Enjev, [y:=c] Emer.. Ily, = cch
(2)
where is the label of the node vy and [y = cj = 1f y. and 0 otherwise. However, we cannot simply compare the distances [lh{(0 poy 122 and [i('') phg (1 where l' * l since different spaces can have different. class densitics. Inspired by Rippel et al. (2015). we normalize the distance by the variance to discriminato densities in different spaces. In particular, we find the variance for each space (1) as simply the empirical variance given the means,
K2 |.. Exer, (h P (E (
M.n E EV. (I Σ. 1. =
(3)
Then, propose MetSSelect that selects the personalized layer 1" (v) such that it minimizes the normalized distance or the Mahlanabois distance, (!)) s.c = (h.))
(1) K, (h ()) In particular,
(x:) = arg min mingen.pm (exp( (20-449) / Em exp(- [0)]
(4)
For training, since we know the true label of each node in the training set, we find the optimal layer as 1f (valo) arg mi [esp( d/ D/E expl
Training. Existing GNNs decode the labels from final layer representations by training an additional neural network g+ to minimize LCE(v, yi) ) = log(gæ h (L) )o.. where is the actual label of v;. Since we want to decode from different layers, we employ L 1 different decoders g0 for I L] specialized for each layer.
Thus, train this using a personalized Cross-Entropy loss
LoE(U, vk) log (go (1 (1((1)))
h ((((l))))))
Since learning one decoder per layer can be hard especially for higher number of layers, we also propose directly leverage the class moments training node from the protolype of the correct class in the closest layer I and maximize the distance from the prototypes of other classes in all layers. Assuming a = 1, we propose
distanco-based classification. Here, minimize the distance of
(muld))) log Elso E,ty, exp( exp(-d())) Latatanoe (V,, v)) max(0, a dig.
5

---

Published in Transactions on Machine Learning Research (04/2025)
However, only use Lce in the experiments unless otherwise mentioned.
Algorithm I describes the training steps in more detail. In particular, first find the class means and variance for each layer and then use these valuos to minimize the loss L for each training pair (vi, Wi) : vs E Ver- Also, note that we treat (!) K₁ as constants in the loss function so the two moments pe"
Algorithm 1: Training MctSelect
Require: GNN of depth L, Training nodes Ver C VVV Number of epochs Ne, Learning
rate ")
that no gradients pass through them for the parameters W to simplify the gradient deescent.
No Σ, 1ly c] for all classes c. 2: for == to N₂ do
Time Complexity. The total training time taken per epoch by Algorithm | includes the time for (1) Finding the moments of each class cluster, (2) Finding the optimal layer for each node; (3) Calculating the loss over the training nodes, and (4) Updating the parameters using gradient descent. One can find the means ue in time ((rl))))r)d)) and the variances K in time (d)),, where d denotes the embodding dimension. The optimal layers can be found using [([V)rL[[)](d)) One can then decode the probabilities and accumulate the loss in time ((IV)r/DD/(d)) This makes up ((v (())) while the forward pass through the GNN takes at least O(La) (due to multiplication with d x d weight matrices). Thus, MetSelect. remains efficient as long as |vrl and | are small.
3: Initialize C 0, ftc +0, 0, Kr.0 + 0.
4: for U, E Ver do
5: for 1 [0, I.] do
My +- My + (
6:
Ki,0 Ki,0 + h (h())..
end for
9:
L. t- C ((N,,ue!")) Eqns. 5 6
10:
end for
11:
[ 4b W < 0 $ W] V.V..
12:
pe P /Ne.
(1)
(1)
13: K, 1 P (KID E. (M(
14: end for
While we empirically find the running time to be similar, specific sampling techniques to reduce the graph size can also be used to scale to larger graphs (Chiang el al. 2019
Space Complexity. As shown in Algorithm | we calculate the means and variance by updating them in an online manner over cach node. Thus, the space overhcad of MctSclect does not depend on the number of nodes as it only involves (LL))d to store the means and O(Ld²) to store the variance.
6 Experimental Setup
Datasets. We consider 4 standard homophilic co-citation network datasets Cora, Citeseer, Pubmed (Kipf & Welling 2016) and ogbn-arxiv (ogba) (Hu et al. 2020), where each node represents a paper that is classified based on its topic area. We also used 6 heterophilic datasets Actor, Chameleon, Squirrel, Coruell, Wisconsin, Texas (Pei let al.] 2020). Following Pei et al. (2020), we evaluate the models 10 different random train-val-test splits for all the datasets except ogba, where we used the standard OGB split. Table I notes the statistics of these datasets along with label homophily, which measures the proportion of nodes with the same labels with an edge in common.
Table 1: Statistics of the datasets used in this work. Label Homophily measures the proportion of nodes with same labels that share an edge.
Dataset Cora
Nodes
# Edges
# Festures
Label
Label Homophllly
[X]
2708
5429
1433
0188
Ciseseer
3327
4732
3703
0.71
Pulumed
19717
44338
500
0179
Actor
7600
33544
931
0.24
Chamelcon
2277
36101
2525
0.25
Squitrel
5201
217073
2089
0.22
Cornell
235
1703
Wisconsin
251
499
1703
0.16
Texas
1703
0.06
Ogho-arsiv
169343
1168240
128
40
0.65
Graph Neural Networks. We use 3 representative base GNNs to assess the plug-and-play improvement of our method - Graph Convolutional Network (GCN) (Kipf & Welling 2016), Graph Atlention Network (GAT) (Veličković et al. 2017], and Graph Isomrrphism Network (GIN) (Xu et al. 2018a). All models have depth L == 2 unless otherwisc mentioned.
Baselines. While baseline exists that directly finds the optimal layer for each node, we extend existing methods for our task to compare them. We use (NN+(name) to denote method "name" and just GNN to denote GNN trained with FinalSclect.
6

---

Published in Transactions on Machine Learning Research (04/2025)
FinalSelect: Here, simply use existing GNN implementations that consider the final layer representation for all nodes to predict their labels.
NDLSelect (Zhagg et al., 2021 NDLS a node-dependent local smoothing method that trains a simplified GCN (Wu et al. 2019) with varying smoothing levels for each node. We use these node-specific layers ((;;) to train GNNs using Equation and classify each node accordingly. Note that these personalized layers are agnostic of models and label distribution.
AttnSelect (Xu et al. 2018a): We replace the softmax in JKNet-L.STM to max such that it selects a fixed layer for each node and train using Equation 6 for these node-specific layers.
Implementation. In GNNs, we assume h (0) = X but the feature dimension can be very high. Since in MetSelect, we find distances in the embedding space h(), a high dimension is not suitable due to the curse of dimensionality (Jordan Mitchell 2015). Thus, we used a trainable linear layer to transform the features into d-dimensional space, i.e., h [0] WoX + bo Rd. We also found other dimensionality reduction methods such PCA and random transformation to perform similarly. All the models were trained using an Adam optimizer for 500 epochs with the initial learning rale luned between {0.01, 0.001). The best-trained model was chosen using the validation accuracy and the case of multiple splits, the mean validation accuracy across splits. All the experiments were conducted on Python 3.8.12 on a Ubuntu 18.04 PC with an Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz processor, 512 GB RAM, and Tesla V100-SXMP 32 GB GPUs.
Metrics. Following existing works, use the micro-F1 accuracy score across different classes for the test set (Test ACC), validation set (Val ACC), and train set (Train ACC) to assess the performance. We found that the trends are similar for other aggregations of F1 across different classes such as class-weighted weighted-F1 and macro-F1 scores.
7 Results
Here, we show empirically that while solving Problem 2 one can make GNNs more accurate, deeper, and robust in a plug-and-play manner.
7.1 Can MetSelect be used to enhance the performance of GNNs in a plug-and-play manner?
Here, we test the impact of using a personalized prediction layer to classify the nodes from different GNN representations. Table ( reports the performance of different layer selection methods on different GNNs and datasots. One can note that our proposed MetSelect method almost always gives the best accuracy among the baselines. This can be observed from the lowest average difference from the maximum accuracy (S 0.015). Standard GNNs that only use the final layer to classify all nodes are known obtain good performance in homophilic datascts (Hu et al.] 2020). In these cases, the three personalized layer selection methods (NDLSelect, AttnSelect, and MetSelect) preserve the performance of the test set across different GNNs for these datasets, with MetSelect being the most consistent among the three.
GNNs trained using MetSelect can ignore a node's neighbors at a scope that is not useful for its classification. This explains the overall boost in performance on heterophilic datasets. Table 2 shows that MetSelect can indeed improve/man match the performance of the GNNs with positive increase the average performance for GCN, GAT, and GIN, In particular, we find a constant improvement in the extremely label-heterophilic datasets such as Cornell, Texas, and Wisconsin. This is particularly because MetSelect learns to ignore the graph structure when predicting labels for many nodes since the graph structure is not aligned with the label distribution for these datascts. Note that this is identified by MetSclect in an automated manner without specifically encoding this preference in the method anywhere. We also note that performance on homophilic datasets does not show a significant increase, which can be explained due to the fact that MetSelect does not enhance the representation power of these models and selecting the most representative final layer tends be the most powerful for these datasets. Overall, we find that MetSelect provides the best overall performance, being at least 4 times closer to the best performance across datasets.
7

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: Test Fl-score of the GNN models averaged over different splits (except Ogba) in the datasets with standard deviation in the subscript. MetSelect- inverts the proposed MetSelect method by selecting the layer that maximizes the distance from the class prototype. A Max denotes the average difference from the mean maximum value across datasets.
Cora
Citsser
Pubmed
Actor
Chameloon
Squirrel
Carnell
Wistonsin
Texas
Ogba
A Max
FinalSelect
84,99,.4
74,46
87. 17.44
30.40,.or
68 08.36.as
62.78
58.65g 20
G1 11.57
67.03
67.75
2.77
NDLSeloct
85.15) as
75.98
28.53, 86.57e.41
63.75; oc
51.14
43.24
50.987
58.386.14
63.15
7.37
GCN
AttuSelect
81887:15
73.23 32
88.28
29.68, 75
58. 38.20,
31.35,
53.78m.m1
61. 61.18p.x
61.89k.c
67.29
7.31
MetSelect (ours)
81,23,.s ns
86 66.60l..c
30. 10 TM
66 28
74
68 92s.4
97
64.05
66.88 35
MetSoleet-max
76.742.1
70.48p.18
84.466.00
20.39..40
53.63
37.70,.s
50.27
58.03,
66.35%
55.28
9.79
FindlSelect
81.611.72
74.23,
86 58
39.13..9
61.69,..
46. 16.751.75
51.05ga7
60.00k
62. 62.43
68.58
5.83
NDLSee
83.48,44
74.611.6
85 85.60u.s
28 68
02.392.42
44.842.00
42. 12.00..0
53.92a.a a1
60.27s..1
05.58
8.79
GAT
AttuSeloct
71.253 94
73.99 3s
85.750.42
32.30g
49.7
36.69
70.54
80.784
77.03g..0
58.84
5.26
MetSelect (ours)
82.80,
74.53
85,94
28. 99,
62.27
48.49g
70.27
81 81.96i
73.24
66.25
1.47
MetSelect-max
70.162.06 80
81.72p.00
29.76, 20
37.37291
41.60, se
46,22 am
53.533 ati
61.22
58.26
10.32
FindSddect
81.63, 56
69.55g 5t
87.19p.5a
27.64 22
66.952m7
49.58
51.085.20 .9
61.18p. 1a
65.149.20
68.37
NDLSelect
80.72 az
71.79, as
86,36p
26.55, .s
67 67.262.41
53. 33
30.27
51.12s
60.544.14
66.18
6.95
GIN
AttnSelect
81 31.29,.0
71.64g.u7
86.57e.54
27. 27.62.9s
G1 364.0
41 41.58
60.00
66.08
70.27
64.51
5.58
MetSolect (ours)
80.97
72.682..7
86.730.43
30.36, 21
63. 20.202.7
18.562.4
60 06
70.022.or 07
75.684.41 41
76
1.33
MetSeleut-max
76.29, a1
68.97g.u 4a
85.01p.63
27.59, a6
36.13
41.37
49.86p..
59.12,
62.97s.00
55.83
10.36
Metteett
GCN
Text ACC
0.7
TestACC
0.7
0.0
0.5
0.4
50
100
50 160
50
100
Number of Layers (L)
Number of Layers (L)
Numbar of Layers (L)
(a) Cora
(b) Citeseer
(c) Pubmed
Figure 3: Test Accuracy of GCN and GCNeMet with varying depths for different datasets.
7.2 Does MetSelect enable deeper GCN models?
To test MetSelect's effect at different depths, use the distance-basee loss function in Equation 6 to efficiently train the model at larger depths but find similar results to hold for the linear loss (i.e., Equation G till depth
= 8. Since we optimally choose the prediction layer to classify each node, we expect the performance of
-et models to preserve the test performance even when we increase the number of GNN layers.
Figure g demonstrates this effect on Cora, Citeseer, and Pubmed datasets. We find that the accuracy of GCN + MetSeloct drops by only up to 10 accuracy points even for depth as high as 128, while the standard GCN model drops by at least. 50 points with just 32 layers. This shows that MelSelect can indeed enable much deeper GCNs, thus increasing its representative power. For ogbn-arxiv, we found that while GCN's accuracy reduces from 0.67 to 0.22 and 0.06 at depth of 16 and 32, while MetSelect preserves the accuracy with a slight reduction from 0.66 to 0.65 and 0.62.
7.3 Are the GNNs trained using MetSelect more robust to poisoning attacks?
Untargeted poisoning attacks Zügner et al. 2018 Sun et al. 2020 have demonstrated that the mean GNN performance can suffer heavily from change in the graph structure during training, In Section 4 we have highlighted that MetSelect can potentially make the models more robust to such untargeted attacks by ignoring the neighborhood scope not useful for a node's prediction and, thus, potentially the perturbations. To test this, we use Mettack (Zügner et al., 2018) to perturb our training graph that is used to train a GCN model using the standard and the proposed MetSelect loss. In particular, we use the black-box version of Mettack, which finds the attacks using a surrogate GCN model trained on the same dataset. These attacks are structural and represent the flip of link, i.e, add the link if it doesn't 't exist and delete it otherwise. The
8

---

Published in Transactions on Machine Learning Research (04/2025)
MetSelec
GCN
Tran MCO
0.7
0.0
Tain ACC
TestACC
0.6
0.83
0.5
0.4
0.90
0.4
0.73
0.2
0.6
0.0
0.2
0.6
0.0
0.2
0.0
0.2
Mettack (%)
Mettack Link (V%)
Mettack Link (1%)
Mettack Link (0%)
(a) Cora. Train
(b) Cora Test
(c) Citesoer Train
(d) Citoscer Tost
Figure 4: Accuracy of GCN and +cttMetSll after different strengths of Mettack perturbations.
Cheseer
Citeser
Puemodd
ogtn ancv
ogbn andv
Debratt
ogbn
Actor
Chamelcon
Chameleon
Squiral
Squirel
Squirel
Comoll
Comdl
Wiscomsnn
Wiscorsin
Viscons in
Teaws
Txxos
0.6
1
1.0
Proportion rodes
nodes
GCN
(b) GAT
(c) GIN
Figure 5: Proportion of test nodes with a particular personalized layer f" E [0, 1, 2] as identified by MetSeloct.
total number of links flipped, or the budgel is fixed as a parameter that we set to be p x [8] where we vary p E [0, 0.5] denoted by Mettack Link (%).
Figure shows the robustness of the GCN model for Cora and Citeseer different budgets using just the final layer (denoted as just GCN) and using personalized layer trained using Equation (denoted GCN+ MetSelect). We find that the GCN trained using MetSelect is significantly more robust than the standard GCN with the training accuracy of Meteett remaining unaffected by the attacks introduced during training. For instance, while the final layer fails to learn an accurate classifier even for the training set at higher budgets 50,, Met.Seloct is able to still learn the representations to classify the training set. This is also reflected in the improved generalization of these classifiers as the test accuracy of the Meteeet is also significantly higher than GCN after the altacks. In particular, we see an improvement of up to points in the test accuracy of GCN trained using MetSelect over the standard GCN. This establishes the robustness of the MetSelect loss owing to its variable prediction layer for nodes.
7.4 Analysis on MetSelect
Ablation. Here, we assess how much the node classification performance depends on the prediction layer identified by MetSelect. To quantify this, compare it with method that picks the polar opposite layer of what MetSelect would select. In particular, we consider the one that maximizes the normalized distance from the class center. Table 2 shows the effectiveness of (w) chosen by the MetSelect model. One can observe that MetSelect almost. always outperforms the MetSelect-max (only exception being a little reduction in Actor for GAT) with an average positive gain in accuracy of over Fl points.
Self-loops. We further study the effect of including the feature information in the GNN and see if the gains remain consistent even in this setting. Particularly, we include the node features by adding a self-loop on each node which ensures that the node information accounted for all the aggregation layers (Hamilton) et al., 2017). Table | shows that the performance remains consistently higher than the final layer selection baseline while highlighting the plug-and-play benefits of our method.
9

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4: Comparison of FinalSelect and MetSelect for GCN with self-loops.
Cora
Citecer
Pubmed
Actor
Chameleon
Squirce)
Cornell
Wisconsin
Texas
Ogba
A Max 4
VinnlSelect 86. 18..88.. MetSelect
87 a2.
30.800
68.62
55.99h m
58. 58.11 56
61.96
67.84, mi
69.72 66.77
2.68
83.72
73.05
86.03c.0z
30.99
63.75g
51.22
66.22
76.875.0
71.63
1.77
Table 5: Comparison of MetSelect and layer aggregation methods in GCN using F1/100.
Cora
Citavur
Pubmed
Actor
Chamaleen
Squirrel
Coruell
W Wisconsin
Texas
Ogla
A Max
JKNet-Max 0.86
0.76
0.88
0.36
0.62
0.40
0.62
0.75
0.67
0.68
0.023
JKNet-Cal 0.86
0.76
0.42
0.71
MSNe
0.71
0.72
0.88
0.35
0.50
0.29
0.70
0.79
0.75
0.68
0.046
MKNNet-MetSin
0.84
0.74
0.88
0.36
0.58
0.35
0.67
0.79
0.75
0.67
0.009
Rumning Time. We theoretically found the running time complexity of MetSelect to be higher than existing GNNs. However, we find empirically that it remains within an acceptable factor of < 2.5 across different datasets to the original method. Table X shows the mean per-epoch time on the largest. dataset (ogbn- arxiv) and the remaining small datasets. As compared to NDLSelect, we remain within reasonable bounds of < 1.5, while sometimes doing even better. Thus, our method is equivalent to the one-time cost of training proxy model as is done in NDLSelect.
Table Mean time taken by different algorithms for Ogba and other small datasets.
FinalSclect
NDLSclect
MctScloct
Small Ogba
0.15
0.37
0.43
GCN
0.59
1.07
1.41
Small
0.17
0.41
0.48
GAT
Ogba
0.81
1.22
1.05
Small Ogba
0.07
0.26
0.20
GIN
0.59
0.98
1.50
We argue that this does not lead to scaling cost as the running time is reasonably small even for the largest OGB dataset with over 100k nodes.
Optimal layer distribution. To further analyze MetSelect, Figure shows the proportion of the nodes that can be effectively classified by GCN, GIN, GAT using its 1" layer where 1" E (0, 1,2.. One can note that datasets like Cora, Citeseer, Pubmed, and ogbn-arxiv use higher-order graph structure for prediction, as noted by the high proportion of nodes with 1" = 2. On the other hand, datasets like Cornell, Wisconsin, and Texas benefit from a lower-order prediction (i.e., layer 1" = 0). This shows that different datasets may have a different layer distribution that idcal for prodicting their node labels and MctSelect can adaptively find this underlying preference to certain representations in an automated manner.
7.5 How does MetSelect compare with layer aggregation methods?
Since layer aggregation methods (Xu et al. 2018b make the iondictions from an aggregated/poolei layer using LSTM, max, or concatenation operators, they cannot solve Problem 2 In other words, they cannot select one of the existing GNN representations as a personalized layer to predict the node class. Thus, we avoid such a comparison for fairness since unlike layer aggregation methods, we do not enhance existing models' representation power using non-linear activations.
For fair comparison of the two methods, we create setup where MetSelect also sees the of pooled representations. Thus, consider different layer aggregation strategies as multiple representations and select. the best aggregation strategy among (LSTM) (CAT), (MAX)), We find in Table 5 that MetSelect can be used to select personalized aggregation layers for improved performance than using any of the aggregation. This shows complementary gains of our method in addition to simple GNN layer selection.
8 Conclusion
Our work has shown that graph neural networks are further empowered by exploiting a personalized prediction layer for each node instead of using the same layer for all nodes. This challenges the common notion that the final layer representation of GNNs should fit. all nodes. After formalizing the problem of finding the
10

---

Published in Transactions on Machine Learning Research (04/2025)
node-optimal GNN layer, proposed a novel method called MetSelect, to find the optimal layer for given node in any given GNN. In particular, we use the layer that minimizes the distance of that node's representation from the class prototype in that layer. Results show that such a simple strategy can handle oversmoothing, boost performance, and robustncss to attacks. We hope our work inspires further investigations into pode-personalieed training of GNNs, including regularization techniques for our method learn more robust personalized representations. While it extremely hard to prove that particular layer optimal for prediction, new benchmarks can be specifically designed for this task either synthetically or by sampling from real-world distributions. We also believe that theoretical investigations into how personalized decoding interacts with the training loss function can help gain useful insights into the convergence specificity of node classification of different nodes. Our work can also be extended to identifying personalized arbitrary motifs for each node for more fine-grained node classification. Aulomated layer selection can also inspire sparse architectures of the mixture of experts for graph neural networks (Shazeer et al. 2017
11

---

Published in Transactions on Machine Learning Research (04/2025)
References
Cristian Bodnar, Francesco Di Giovanni, Benjamin Chamberlain, Pictro Liö, and Michael Bronstein. Neural sheaf diffussin: A topological perspective on heterophily and oversmoothing in gnns. Advances in Neural Information Processing Systems, 35:18527-18541, 2022.
Andrei Boutyline and Robb Willer. The social structure of political ccho chambers: Variation in idcological homophily in online networks. Political psychology, 38(3):551-500 2017.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? or Xiv preprinl arX4v:2105.14491 2021.
Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Tie-yan Liu, and Liwei Wang. Graphnorm: A principled approach to accelerating graph neural network training. In International Conference on Machine Learning, pp. 1204 1215. PMLR, 2021.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gent An efficient algorithm for training deep and large graph convolutional networks In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 257 266, 2019.
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovie. Adaptive universal generalized pagerank graph neural network. arXiv preprint arXiv:2006. 07988, 2020.
Matteo Cinelli, Gianmarco De Francisci Morales, Alessandro Galcazzi, Walter Quattrociocchi, and Michele Starnini. The echo chamber effect social media. Proceedings of the National Academy of Sciences, 118 (9):3303331 2021.
Enyan Dai and Suhang Wang. Towards self celf-expla graph neural network. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 302-311, 2021.
Jiankang Deng, Jia Guo, Niannan Xue, and Stelanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVE /CVF conference computer vision and pottern recognition, pp. 4690-4699, 2019.
Jiankang Deng, Jia Guo, Tongliang Liu, Mingming Gong, and Stefanos Zafeirion. Sub-center arcface: Boosting face recognition by large-scale noisy web faces. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 741-757. Springer, 2020.
Francesco Di Giovanni, James Rowbottom, Benjamin P Chamberlain, Thomas Markovich, and Michael M Bronstein. Graph neural networks as gradient flows. arXiv preprind urXiv:2206. 10991, 2022.
Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 24(43):1-48, 2023.
Moshe Eliasof, Lars Ruthotto, and Eran Treister. Improving graph noural networks with Icarnable propagation operators. In International Conference on Machine Learning, pp. 9224 9245. PMLR, 2023.
Aosong Feng, Chenyu You, Shiqiang Wang, and Leandros Tassiulas. Kergnns: Interpretable graph neural networks with graph kernels. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6614-6622, 2022.
Swapnil Gandhi and Anand Padmanabha Tyer. P3: Distributed deep graph learning at scale. In 15th { [USENIX} Sympostun on Operating Systems Design and Implementation (L (LODDI) 21), pp. 551 568, 2021.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Lukas Gosch, Simon Geisler, Daniel Sturm, Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Adversarial training for graph neural networks: Pitfalls, solutions, and new directions. In Thirty-seventh Conference on Newral Information Processing Systems, 2023.
12

---

Published in Transactions on Machine Learning Research (04/2025)
Benjamin Gutteridge, Xiaowen Dong, Michael M Bronstein, and Francesco Di Giovanni. Drew: Dynamically rewired message passing with delay. In International Conference on Machine Learning, pp. 12252- 12267. PMLR, 2023.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advonces in neural information processing systems, 33:22118 121133,, 2020.
Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 66-74, 2020.
Michael I Jordan and Tom M Mitchell. Machine learning: Trends, perspectives, and prospects. Science 349 (6245):255-2000) 2015.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint ar xrx:::1009 02907,, 2016.
Guohao Li, Matthias Mülerr, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In International conference on machine learning, pp. 6437 6449. PMLR, 2021.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi- supervised learning. In Proceedings of the A AAT conference on artificial intelligence, volume 32, 2018.
Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao Wen Chang, and Doina Precup. Revisiting heterophily for graph neural networks. Advances in neural in information processing systems, 35:1362-1375, 2022.
Hoang Nt and Takanori Machara. Revisiting graph neural networks: All we have low-pass filters. arXiv preprinl ar Xix: 09550,, 2019.
Hongbin Pei, Bingzhe Wei, Kevin Chen- Chen- Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. arXiv preprint arXi::2002 052277, 2020.
Oren Rippel, Manohar Paluri, Piotr Dollar, and Lubomir Bourdev. Metric learning with adaptive density discrimination. arXie preprint orXio:1511.0593, 2015.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. arXiv preprint Xix:;;1907.
2019.
Emanuele Rossi, Bertrand Charpentier, Francesco Di Giovanni, Fabrizio Frasca, Stephan Günnemann, and Michael M Bronstein. Edge directionality improves learning on heterophilic graphs. In Learning on graphs conference, pp. 25-1. PMLR, 2024.
T Konstantin Rusch, Benjamin P Chamberlain, Michael W Mahoncy, Michael M Bronstein, and Siddhartha Mishra. Gradient gating for deep multi-rate learning on graphs. orXiv preprinl ar arXiv:2210.005: 2022.
Ruslan Salakhutdinov and Gcoff Hinton. Learning nonlinear embedding by preserving class neighbourhood structure. In Artificiol intelligence and statistics, pp. 412 419. PMLR, 2007.
Noam Shazeer, Azalia. Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXio preprint arXiv:1701 701.06538. 2017.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint Xixv:20009 Xix:20009.0300, 2020.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Yiwei Sun, Suhang Wang, Xianfeng Tang, Tsung- Yu Hsich, and Vasant Honavar. Non-target-speciffe node injection attacks on graph neural networks: A hierarchical reinforcement learning approach. In Proc. WWW, volume 3, 2020.
Jianheng Tang, Jiajin Li, Ziqi Gao, and Jia. Li. Rethinking graph neural networks for anomaly detection. Tn International Conference on Machine Learning, pp. 21076-21080 PMLR, 2022.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 2017.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning, pp. 6861 6871. PMLR, 2019.
Kaidi Xu, Hongge Chen, Sijia Liu, Pin-Yu Chen, Tsui Wei Weng, Mingyi Hong, and Xue Lin. Topology attack and defense for graph neural networks: An optimization perspective. ar preprinl arXiv: 1506:990.000 04214, 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks?
arXiv preprint Xrv: 810. 00826, 2018a.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi,, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning, pp. 5453 5462. PMLR, 2018b.
Yujun Yan, Milad Hashemi, Kevin Swersky, Yaoqing Yang, and Danai Koutra. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. In 2022 IEEE International Conference on Data Mining (ICDM), pp. 1287 1292. IEEE, 2022.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William I. Hamilton, and Jure Leskovee. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 174-983, 2018
Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, and Ren Chen. Decoupling the depth and scope of graph neural networks. Advances in Neural Information Processing Systems, 34:19665 19679, 2021.
Wentao Zhang, Mingyu Yang, Zeang Sheng, Yang Li, Wen Ouyang, Yangyu Tao, Zhi Yang, and Bin Cui. Node dependent local smoothing for scalable graph learning. Advances in Neural Information Processing Systems, 34:20321 -00322, 2021.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Checkong Lec. Protgmn: Towards self-cxplaining graph neural networks. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 36, pp- 9127-9135, 2022.
Lingxiao Zhao and Leman Akoglu. arXiv:1909. 12223, 2019.
Pairnorm:
Tackling oversmoothing in gnns. arXie preprint
Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann. Adversarial attacks on neural networks for graph data. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery 0 data mining, pp. 2847-2856, 2018
14


Reviews:
[{"review_id": "qWoiSyJtDS", "reviewer": "Reviewer_5yj7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The manuscript focuses on an important research problem which is inherently related to over-smoothing and robustness of GNNs.\n- The proposed idea is intuitive and easy to follow.\n\nWeaknesses:\n\n- Although I can see the merits of an adaptive GNN layer selection for different nodes, I am not convinced about the proposed design. I believe using information coming from different layers by learning importance scores for the differerent layers can make a better use for the available information (e.g., using an LSTM-attention like strategy in JK-Net without changing the softmax to max).\n\n- The proposed method relies on a manual distance-based layer selection strategy. I am curious about the performance of applying a self-attention mechanism on the representations of different layers to create a final representation for classification, which can also scale better than the proposed method.\n\n- The baselines for the experiments should be improved. There are a large number of works dealing with heterophilic graphs or the over-smoothing issue, e.g., [1], [2]. These studies should be used as baselines for utility and scalability to justify the proposed method.\n\n- For the homophilic graphs, the performance gain of the proposed method does not justify its additional computational cost. Thus, more thorough experiments (with better baselines) for the heterophilic graphs are needed. \n\n\n[1] Yu, Zhizhi, et al. \"LG-GNN: local-global adaptive graph neural network for modeling both homophily and heterophily.\" Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. 2024.\n\n[2] Wang, Chenhao, et al. \"HeterGCL: graph contrastive learning framework on heterophilic graph.\" Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an adaptive GNN layer selection mechanism to mitigate the limitations sourced from using a fixed-level of aggregation for every node. To achieve this, the paper presents a manual layer selection mechanism based on the embedding distances to class prototype representations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- JK-Net (LSTM-attention) without changing the softmax nonlinearity to max should be employed as another baseline.\n\n- An attention based aggregation of different layers' representations should be used as a baseline to justify the cost incurred from the manual selection.\n\n- Better baselines (state-of-the-art works designed specifically for the over-smoothing issue) should be selected to improve the experimental setup.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F3aoc2hN65", "reviewer": "Reviewer_VKiv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The idea of **personalized layer selection** for each node is **novel** and addresses a significant limitation of existing GNNs.  \n- The method is **theoretically well-founded** and leverages **metric learning** effectively to identify the optimal layer.\n- Extensive experiments on **diverse datasets** show **consistent improvements** in accuracy, especially on **heterophilic datasets**.\n- The results indicate that **MetSelect enables deeper GNNs** and enhances **robustness against adversarial attacks**.\n- The proposed method can be applied to **various GNN architectures** without modifying their core structure, making it **versatile and practical**.\n- The findings that **MetSelect improves robustness to adversarial attacks** and supports **deeper GNNs** are particularly valuable, as these are active research areas in GNNs.\n\n### Weaknesses\n\n- The method introduces **additional computational overhead** due to the need to evaluate **multiple layers** and compute distances to class prototypes.\n- The paper mentions using **sampling techniques** to scale to larger graphs, but **detailed experiments on very large datasets** (e.g., with **millions of nodes**) are **missing**.\n- The selection of the optimal layer is based on **minimizing distance to class prototypes**, which may **not always be intuitive or interpretable**.\n- Providing more **insights into why certain layers are preferred** for specific nodes could enhance the **understanding of the method**.\n- The paper compares **MetSelect** with a few existing methods, but a **more comprehensive comparison** with **other state-of-the-art techniques** that address **oversmoothing or heterophily in GNNs** would provide a **clearer picture of its advantages**.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel method called **MetSelect** to address the limitation of traditional Graph Neural Networks (GNNs) that use a fixed layer for classifying all nodes. The authors argue that different nodes may benefit from different layers of the GNN based on the granularity of their local neighborhood. **MetSelect** leverages metric learning to identify the optimal layer for each node by minimizing the distance to the class prototype in the transformed embedding space. The method is evaluated on **10 datasets** and **3 different GNN architectures**, demonstrating **significant improvements** in node classification accuracy, robustness to attacks, and the ability to support deeper GNNs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Conduct a **more comprehensive comparison** with other **state-of-the-art techniques** that address **oversmoothing or heterophily in GNNs**. This should include methods like **Heterophily Graph Neural Networks** and **PairNorm**.This will provide a **clearer picture** of the advantages of **MetSelect** over existing methods and **strengthen the claims** of **novelty and effectiveness**. This adjustment is **essential** to **secure a recommendation for acceptance**, as it directly addresses the **novelty and superiority** of the proposed method.\n- Include experiments on **very large datasets** (e.g., with **millions of nodes**) to demonstrate the **scalability** of **MetSelect**. Specifically, evaluate the performance on datasets like **ogbn-papers100M**. Scalability is a **crucial aspect** for practical applications, and demonstrating that **MetSelect** can handle **large-scale graphs** will **significantly enhance its applicability**. Scalability is a **key concern** for many applications, and addressing it will be **critical for acceptance**.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Y6DImLw37", "reviewer": "Reviewer_D3Uc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The paper is well-written and easy to follow. \n2. The paper challenges the basic understanding of GNNs that all nodes use a single set of parameters for predictions, and proposes an elegant solution.\n3. The paper achieves good performance on both homophily and heterophily graphs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper challenges the idea that a single GNN layer can classify all nodes and introduces the concept of personalized layers for each node. Thus, the authors propose MetSelect which selects the best representation layer for each node by identifying class prototypes in a transformed GNN layer and choosing the layer where the distance to the class prototype is smallest. The method both improves the accuracy and model robustness under distribution shift.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Scalability: The proposed method may face challenges when applied to very large graphs due to computational and memory constraints. Thus, I encourage the authors to experiment with larger graphs like obgn-products. \n2. The baselines in Table 2 are limited, including GIN, GAT, and GCN. The authors should include state-of-the-art GNN baselines.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9ULewLkoTx", "reviewer": "Reviewer_LWxs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Choosing representations from potentially different layers for the different nodes of a graph is an interesting research direction which can enable the development of models that preserve information from different granularities for each node. \n\n- The proposed method significantly outperforms the baseline methods on some heterophilic datasets. Furthermore, while the performance of base GNNs significantly degrades as the number of layers increases, this is not the case for the proposed method. This makes MetSelect very attractive for practitioners who do not need to tune the number of GNN layers for a specific application.\n\nWeaknesses\n\n- The proposed method outperforms the baseline methods only on three heterophilic dataset (Cornell, Wisconsin and Texas). On the rest of the datasets, the method does not provide any consistent improvements. For example, on the Ogba dataset, MetSelect is outperformed by FinalSelect no matter what the base GNN model is.\n\n- The proposed method is only evaluated on real-world datasets. It would strengthen the paper if the authors could construct some synthetic dataset to validate the paper's claims. For example, they could construct a node classification dataset where for some of the nodes the representations of the first GNN layer are the most informative, while for the rest of the nodes the representations of the third GNN layer are the most informative. Such an experiment would validate that the proposed method can indeed choose the most informative representation.\n\n- In the experimental evaluation, the proposed method is compared against three baseline approaches (FinalSelect, NDLSelect and AttnSelect). One of my major concerns with this paper is that some baselines, which would give us more information about the significance of the reported results, are missing from Table 2. The authors should at least compare MetSelect against a model which concatenates the initial node features with the representations from the different message passing layers and feeds the concatenated vectors to an MLP to make predictions.\n\n- The MetSelect would fail on datasets where the output depends on the representations of more than one layers since it cannot combine information from the representations of multiple layers. Could the proposed method somehow be modified to handle such scenarios?\n\n- The proposed method introduces complexity to the different GNN models. Such personalized node representations could also produced by an attention mechanism that computes a weighted sum (with different coefficients for each node) of the initial node features and the representations that emerge at the different message passing layers. Such a mechanism could focus on the representation that provides the most useful information and would be more efficient than MetSelect. What would be the advantages of MetSelect over such a model?\n\n- The presentation of the paper leaves the reader with a feeling that this was a draft of the work. There are several typos and the writing should be improved for clarity. This is a non-exhaustive list of typos/grammatical errors:\\\np.4: \"neighborhood) Thus\" -> \"neighborhood). Thus\"\\\np.4: \"Distance-based classifiers Rippel et al. (2015); Deng et al. (2019; 2020); Salakhutdinov & Hinton (2007) typically finds\" -> \"Distance-based classifiers (Rippel et al., 2015; Deng et al., 2019; 2020; Salakhutdinov & Hinton, 2007) typically find\"\\\np.8: \"in Equation ??\" -> Equation number is missing\\\np.9: \"Table 7.4\" -> There is no Table 7.4 in the manuscript", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes MetSelect, a method which can be applied to any GNN model and which selects for each node one of the representations produced at the different layers of the GNN (or its initial features) as its final representation. MetSelect is inspired by distance-based classifiers in computer vision, and selects for each node the layer that minimizes the normalized distance from the prototype embedding of some class. The proposed method is evaluated on 10 homophilic and heterophilic node classification datasets. MetSelect outperforms the baselines on a subset of the heterophilic datasets, while it performs on par with them on the rest of the datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see the weaknesses mentioned above addressed in the revised version of the manuscript.\n\n1. Construct some synthetic dataset and evaluate MetSelect on that dataset to validate the paper's claims (that the method can indeed select the most informative representation from the representations of the different layers).\n2. Compare MetSelect against other baseline methods such as a model which concatenates the initial node features with the representations from the different message passing layers and feeds the concatenated vectors to an MLP.\n3. Explain why MetSelect does not outperform the baselines on the homophilic datasets.\n3. Fix typos and improve clarity of writing.", "broader_impact_concerns": "--", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers recognized the technical contribution of this work and raised some concerns on methodology and experiments. Most of them have been well addressed. However, there are still a few remaining concerns: (1) the optimality of the selected layer for each node; and (2) additional baselines of GNNs that "concatenate the initial node features with the representations from the different message passing layers". The authors are highly encouraged to address these concerns in the final version.
Decision: Accept (poster)