Ordered GNN: Ordering Message Passing to Deal with Heterophily and Over-smoothing
Authors: Yunchong Song
Chenghu Zhou
Xinbing Wang
Zhouhan Lin
Venue: ICLR 2023 poster (2023)
Keywords: GNN
heterophily
over-smoothing

Abstract: Most graph neural networks follow the message passing mechanism. However, it faces the over-smoothing problem when multiple times of message passing is applied to a graph, causing indistinguishable node representations and prevents the model to effectively learn dependencies between farther-away nodes. On the other hand, features of neighboring nodes with different labels are likely to be falsely mixed, resulting in the heterophily problem. In this work, we propose to order the messages passing into the node representation, with specific blocks of neurons targeted for message passing within specific hops. This is achieved by aligning the hierarchy of the rooted-tree of a central node with the ordered neurons in its node representation. Experimental results on an extensive set of datasets show that our model can simultaneously achieve the state-of-the-art in both homophily and heterophily settings, without any targeted design. Moreover, its performance maintains pretty well while the model becomes really deep, effectively preventing the over-smoothing problem. Finally, visualizing the gating vectors shows that our model learns to behave differently between homophily and heterophily settings, providing an explainable graph neural model.

Published as a conference paper at ICLR 2023
ORDERED GNN: ORDERING MESSAGE PASSING TO DEAL WITH HETEROPHILY AND OVER-SMOOTHING
Yunchong Song', Chenghu Zhou², Xinbing Wang', Zhouhan Lin Shanghai Jiao Tong University, "Chinese Academy of Sciences {ycsong, xwang8Jes edu. en, 1in. zhouhan& gmail com
ABSTRACT
Most graph neural networks follow the message passing mechanism. However, it faces the over-smoothing problem when multiple times of message passing is applicd to a graph, causing indistinguishable node representations and prevents the model to effectivvlly learn dependencies between farther-away nodes. On the other hand, features of neighboring nodes with different labels are likely to be falsely mixed, resulting in the heterophily problem. In this work, propose order the messages passing into the node representation, with specific blocks of neurons targeted for message passing within specific hops. This achieved by aligning the hierarchy of the rooted-tree of a central node with the ordered neurons in its node representation. Experimental results on an extensive set of datasets show that our model can simultaneously achieve the sfute-of-thh-art in both homophily and heterophily settings, without any targeted design. Moreover, its performance maintains pretty well while the model becomes really deep, effectively preventing the over-smoothing problem. Finally, visualizing the gating vectors shows that our model learns to behave differently between homophily and heterophily settings, providing an explainable graph neural model
INTRODUCTION
Graph neural network (GNN) (Scareelii et al. 2008 Bruna et al. [0113] Defferrard et al. 2016 Kipf
& Welling 2016 Veličković et al. [017] Hamilton et al. 2017 Gilmer et |017
al. 2017 Xu et al. [201 8a)
has become the prominent approach to learn representations for graphs, such social networks (Li
& Goldwasser] 2019). biomedical information networks Yann et al. (019), communication networks (Suárez-Varela al. (02I), n-body systems (Kipf et al. 2018), etc. Most GNNs rely on the message passing mechanism (Gimmer et al. 2017) to implement the interactions between neighbouring nodes. Despite its huge success, message passing GNNs still faces two fundamental but fatal drawbacks: fails to generalize to heterophily where neighboring nodes share dissimilar features or labels, and simple multilayer perceptron can outperform many GNNs (Zhu et al., 2020b), this limit GNNs ex- tending to many real-world networks with heterophily; it is also observed the node representations become indistinguishable when stacking multiple layers, and suffers sharply performance drop, re- sulting in the so-called "over-smoothing" problem (Li al. 2018), which prevent GNNs to utilize high-order neighborhood information effectively.
To address these two drawbacks, numerous approaches have been proposed. Most of them con- centrate on the aggregation stage of message passing. Some design signed messages to distinguish neighbors belong different classes (Yang al. 202T Bo et al. 2021 Luan et al. 2021 Yan al. 2021], allowing GNNs to capture high-frequency signals; Min et al. (020)) design specific filters capture band-pass signals; some apply personalized aggregation with reinforcement learning (Lai eet al. (2020) or neural architecture search (Wang et al. (022b); others attempt to aggregate mes- sages not only from the direct neighbors, but also from the embedding space (Pei et al. (0220) or higher-order neighbors (Zhu et al.) 2020b). These aggregator designs have achieved good perfor- mance, however, they primarily focus on the single-round message passing process and ignore the integration of messages from multiple hops. Another line of works focus on the effective utilization
"Zhouhan Lin is the corresponding author.
Code is available at atthtpss:/ http::/.....///..aaaaaaaa/////ea

---

Published as a conference paper at ICLR 2023
of multiple hops information, which is mainly accomplished by designing various skip-connections. Klicpera et al. ; Chen et al. (2020b) propose initial connection to prevent ego or local infor- mation from being "washed out" by stacking multiple GNN layers; inspired by ResNet (He et al.
2016), some works (Li et al. 2019 Chen et al. 2020b Cong et al. 2021) explores the application of residual connection on GNNs to improve the gradients; others combined the output of intermediate
GNN layers with well-designed components. such as concat (Xu et al. 20118)] Zhu et al. 2020b),
learnable weights (Zhu et al. 2020b Abu-EE-Haaja et al. 2019 Liu et al. (020), signed weights
(Chien et al. (020), or RNN-like architectures (Xu et al. [2118b] Sun et al. (019). These works are
simple yet effective, however, they can only model the information within few hops, but unable to
model the information exactly at some orders, this lead to a mixing of features at different orders;
besides, many of these approaches (Li et al. 2019; Abu-El-Haija et al.) 2019 Chen et al. 2020b] Zhu et al. 2020b] Chien et al.) [020] Cong et al.) 2021) are unable to make personalized decisions for each node. These deficiencies result in suboptimal performance. In addition to caring about the model side, other approaches focus on how to modify the graph structure. These methods are called "graph rewiring". including randomly removing edges (Rong et al. 2019) or nodes (Feng et al. 2020), or computing new graph with heuristic algorithms (Suresh al. [022]] Zeng et al. 2021 In general, these algorithms are not learnable and thus only applicable to certain graphs
Unlike the previous works, address both problems by designing the combine stage of message passing and em- phasize the importance of it. The key idea is to integrate an inductive bias from rooted-tree hierarchy, let GNN encode the neighborhood information exactly some orders and avoid feature mixing within hops. The com- bine stage has been rarely focused before, most works simply implement it as a self-loop. This would result in an unreasonable mixing of node features (Zhu et al. 2020b). To avoid this "mixing", Hamilltn et al. (2017); at. (b)
AP
1
et al. (2020b) concat the node rep- resentation and the aggregated message, which has been identified as an effective design to deal with heterophily (Zhu et al.. 2020b). However, keeping the embedding dimension constant across layers, the local information will be squeezed an exponential rate. The most related work to ours is Gated GNN (Li et al., 2015),, it applys a GRU in the combine stage and strengthen the expres- sivity, but fails to prevent feature mixing, limiting the performance.
Figure 1: Aligning the hierarchy of rooted-tree Te underlying the graph with the node embedding of the root node U. Neighboring nodes within hops of edges naturally form depth l subtree. Messages passed to v from nodes within this subtree restricted to the first p(() neurons in the node embed- ding of v.
In this paper, we present the message passing mechanism in an ordered form. That is, the neurons in the node embedding of a certain node is aligned with the hierarchies in the rooted-tree of the node. Here, by rooted-tree of a node we refer to the tree with the node itself as the root and its neighbors as its children. Recursively, for each child, its children are again the neighboring nodes of the child. (c.f. Figure [). We achieve the alignment by proposing novel ordered gating mechanism, which controls the assignment of neurons to encode subtrees with different depth. Experimental results on an extensive set of datasets show that our model could alleviate the heterophily and over-smoothing problem at the same time. Our model provides the following practical benefits:
We design the combine stage guided by the rooted-tree hierarchy, a very general topological inductive bias with least assumption about the neighbors' distribution, this allows a flexible integration of information different orders, and leading superior performance in both heterophily and homophily.
The ordered gating mechanism prevent the mixing of node features within hops, enable us to model information at different orders. This open a door to extract similar neighborhood patterns for each node under heterophily; this also make it easy to preserve ego and local information, then effectively alleviating over-smoothing.
Our model aligns neighboring structures with blocks in node embeddings through plicit gating mechanism, thus the gating mechanism could provide visualizations to reveal the connecting type of the data and offer explainability.
2

---

Published as a conference paper at ICLR 2023
2 OUR APPROACH
In this section, we will first present a general formalization of GNNs. Then we show how to align the hierarchy of the rooted-tree to the node embeddings, the intuitions behind it, well the two necessary components to imeplement it. Finally, our model is presented in Section 2.5 with few cases to explain it's practical benefits.
2.1 GRAPH NEURAL NETWORKS
Consider unweighted and self-loop-free graph 9 = (V,E), with V and & denoting its nodes and edges, respectively. Let v E V being one of the nodes with its label yo E V, and assume that there N nodes in the graph. The adjacency matrix for graph G is denoted as A E R Assume that the input features for each node form a matrix X E R NXP, where F is the dimension of input features, with X, E IR-V being the features of node v. Before feeding it into the GNN, an input projection function fo(.) is optionally incorporated to nonlinearly project the features into a space mathing the node embedding dimensions, i.c..
= fo
where D is the dimension of node embeddings, and Zy E TRP is feed to the first layer of GNN, i.e., (0) Ze. In our work, simply parameterize fo(-) with an MLP.
Generally, a GNN layer with message passing can be divided into two stages: the aggregate stage and the combine stage. For the node e at the k-th layer, it first aggregates information from its neighbors, forming the message vector m*) Then in the second stage (h) is combined with v's ego representation he (k-1) from the last layer. Formally,
m. (k) = AGGREGATE( (( ( n1 = COMBINE(A) ((6-"),m" (2)
where N(v) is the set of node v's direct neighbors. The AGGREGATEC) operator denotes a permutation-inven invariant function on all the neighboring nodes of v, such as addition, mean pooling or max pooling, optionally followed by transformation function. The transformation function could be an MLP (Xu al., 2018a) or transformation matrix (Kipf Welling 2016). The COMBINE(.) operator denotes combine function that fuses the ego information and the aggrated message.
Our model only modifies the combine stage, which means it could be seamlessly extended to many popular GNNs, such as GCN (Kipf & Welling, 2016), GAT (Veličkovćć et al.) (017)). GIN (Xu et al.) 2018a), etc., by changing the aggregate stage correspondingly. In this paper, we simply implement the aggregate stage computing the mean value of neighboring vectors.
2.2 ALIGNING ROOTED- TRREE WITH NODE EMBEDDING
From the viewpoint of a node, the neighboring nodes within multiple hops around it can be naturally presented as rooted-tree (Xu et al. 20188 b. Liu al. 2020)). For node v, its k-th order rooted-tree (.(*)) is composed by putting its first order neighbors as its children, and then recursively for each child, putting the neighboring nodes of the child its children (except the child that is also the ancester of the node), unitl its k-th order neighbors (Figure
Since there is only one round of message passing in each GNN layer, stacking multiple layers of GNNs corresponds to doing multiple rounds of message passing. And since for each single round of message passing, only direct neighbors exchange their information, the node representation of in the k- layer (R) is only depending on the nodes within its k-th order rooted-tree (T(*) Our goal is to properly encode the information within the k-th order rooted-tre trreddteeeeie into the fixed size vector h
Note that there is a nested structure between cooted-trees of v, and we are going to utilize this structure as anchors to align the rooted-trees with the node embedding of v. It is obvious that the k 1-th rooted-tree (((-1) is subtree of the k-th rooted-tree (((*), with the same root e and the same tree structure up to depth A: 1. Thus we have (assuming the model has K layers)
T C (((1) C T(()) .... E...C75 T.KK)
(3)

---

Published as a conference paper at ICLR 2023
where %8 corresponds to the neurons used represent the node's ego information. As k grows, Thth becomes larger and more complex, requiring more neurons to encode its information. As a result, it is natural to assume that the neurons used to represent message passing within Te(k-1) should be subset of those for T,%h) Correspondingly, we are going to implement the same nested structure in the node embedding of v. We first order all the neurons in ha into a sequence, thus we can index them with numbers. For information within ( we allow be encoded within the first p(*-1) neurons. As for the next level rooted-tree Te()) more neurons are involved, corresponding to the first p(() neurons. Apparently here we have (( < ((
There is critical property that comes with this way of ordering the neurons. Consider the neurons between the two split points p)*-1) and (( They reflect the delta between k: - and rounds of message passing, encoding the neighbor information exactly at k-th order. Intuitively, it organizes the message passing in an ordered form, avoid the mixing of neighbor features at different orders within hop. We implement the ordered neurons with the gating mechanism (which will be intro- duced in the following sections in detail), thus it's natural to visualize the gates as way to probe the message passing of the model.
Now we have the node embedding of v split by a set of spliting points
p(O) < p(1) S .... S p(k) < ... < p(K)
(4)
Each of them is predicted by GNN layer, with (K() = D pointing to the end of the node embed- ding. If we can learn the positions of these spliting points and ensure their relative magnitudes, then the alignment could be realized. We are going to show how we predict the spliting points through gating mechanism, and how we lock their relative magnitudes with the differentiable OR operator. Our model naturally emerges by putting everything together in Section 2.5
2.3 THE SPLIT POINTS
Consider the split point p(*) that split up the ordered node embedding into two blocks, with the left block containing neurons indexed between [0, p(A) 1] and the right block [P(()] D). We can represent the split by D-dimensional gating vector go with its left p!*) entries are 1s and the rest are Os. This gating vector could then be used in the combine stage control the integration between
the ego representation as input to the k-th layer (k-1) and the aggregated context mk)
(b) =9 (k-1) + (1 (1-11) glx)) m(()
(5)
Where o denotes element-wise multiplication. Intuitively, for the left pb() neurons in (x) since their corresponding gating vectors in ge are 1s, they suppress the newly aggregated context my and only inherit the last layer's output n.( (k-1) And vice versa for the right block of neurons with gating vectors of O.
Ideally, we would prefer crispy split, with clear boundary P. and binary gating vectors in (k) However, this will result discretized operations, rendering the model non-differentiable. In this work, we resort to "soften" the gates by predicting the expectations of them, thus keeping the whole model differentiable. The expectation vector ge presented as the cumulative sum of the probability of each position in the node embedding being the split point ((k) Concretely, (k) is parameterized as
go
g(9h) = cumax, (5(
(K) (h(t-1), m)(+) cumaxx (W/(A) [ (k-1), ((4m);mm))) + +b())
(6)
where cumax,- (.) = cumsum, (sofumax(...) is the composite operator of cumulative sum and softmax, with the subscript < indicating that the direction of cumulation is from right to left. f(() denotes function fusing the two vectors
h (k-1) and m Here we simply put it as linear projec-
tion with bias ((k), The he (k-1), m m indicates the concatenation of the two vectors 1 (k-1) and (k) m.
(b)
We refer the readers to Appendix A.1 for proof of g( being the expectation of the gating vector go
4

---

Published as a conference paper at ICLR 2023
Case 2
Case 3
Case
Figure 2: Some rooted-tree tree cases, each rooted-tree is constructed from root node by message pass- ing, node's 's color indicate it's label. Case I and Case 2 show two cases under heterophily; Case is under homophily, node 1 are at the center and the border of a community, respectively.
2.4 THE DIFFERENTIABLE OR OPERATOR
In the above section we provide differentiable way to predict the position of the split points, as well as corresponding gates that implement the split operation. However, there's no guarantee that the predicted split points will comply with the restrictions in Equation Not ensuring Equation would break the alignment between the rooted-trees and node embeddings.
To ensure such relative magnitude, we conduct bitwise OR operator between the newly computed gating vector g(a) and its predecessor. We change the notation to go to discriminate from the originally computed gating vector gb. Again, the OR operator is not differentiable in nature. We instead implement the softened version of by
ge SOFTOR ,g(h)) = g(*-1) + (1 g( 1)))0a)))))) ((k)
We call it the SOFTOR(-) operator.
2.5 PUTTING IT ALL TOGETHER
With the above discussion, we can get the updating rule for the k--th GNN layer by putting everything together, leading to the Ordered GNN model:
m MEAN
:.r:D:EENGG :NE
(4) cumax ( (1.. (1(R-11,, ml)
(8)
g. = SOFTOR
Ir () ",90 (0))
((t=1),900))
(+) = (b) o (1--)) + (1 g(*)) (k)
The first equation corresponds to the aggregate stage, while the latter three correspond to the com- bine stage. In practice we can decrease the parameters the gating network (parameters in the second equation) by applying chunking trick, i.c., set Dm D/C the size of the gating vector, where the dimension of the GNN's hidden state, and the chunking size factor. In this setting, each gate controls C neurons instead, thus drastically reducing the number of gates to be predicted.
In order to explain the practical benefits model provided for heterophily and over-smoothing problems, we illustrate three cases in the Figure 2 and analyse them with the insights from related theoretical works. [uan et al. (2021) Ma al. (222)) notice that heterophily is not always harmful for GNNs, and Ma et al. (202T) points out that a simple GCN can work well if similar neighbor patterns are shared across nodes with the same label, they called "good heterophily". In Case 1, two root nodes share the same first-order neighborhood pattern, one GNN layer can capture this "good heterophily" and obtain good performance, however, multi-layer GNN will mix the second-order neighbor features that have different pattern, the captured "good heterophily" would be destroyed. Our model can protect the captured "good heterophily" and reject that "bad heteropihly" by learning same split points for layer & 2. In Case 2, two root nodes' first-order neighbor patterns differ, but they share the same second-orde neighborhood pattern. Since an ordinary GNN can only capture information within k-hops, they not capture this "good heterophily" at second-on order, whereas our model could do so with the help split points prediction. Furthermore, we can filter the first-order "bad heterophily".
5

---

Published as a conference paper at ICLR 2023
Some works et al. 2018 Chen et al. 2020a Ma et al. 2021) show that neighbor feature smooth- ing is the key reason why GNN works, but too much smoothing lead to the mixing of nodes from different classes. Case 3 shows that while GNN allows node 1 to smooth neighbor features within it's class, it also mixing node features belong to different classes for node 2. With the fine-grained ordered gating mechanism, our model allow node to aggregate similar features while filtering out information beyond node 2's own, thus helpful for homophily and over-smoothing.
The analysis of these cases shows that it critical to predict a set of flexible split points. By analyzing the convergence rate of the gates, we prove that our model only constrain the relative position of the split points, the delta between split points is very flexible, i... the number of neurons assigned to encode the information at different orders is flexible. We provide the proof in Appendix A.2
EXPERIMENTS
In this section, test model on 11 datasets. We mainly evaluate our model in 3 aspects: performances on homophily data, on heterophily data, and the robustness towards over-smoothing. if our model could scale well w.r.t. the size of the dataset,
We further use larger datasets these two datasets are from large-scale homophily and heterophily benchmarks, respectively.
Datasets & Experiment Setup For homophily data, tation network datasets, i.e., Cora, CiteSeer, and PubMed (Sen al. 2008). While six heterophilous web network datasets (Pei al., [2200] Rozemberczki et 2001)) i.c., Ac- tor, Texas, Cornell, Wisconsin, Squir- rel and Chameleon, are used to eval- uate our model on the heterophily setting. In citation networks, nodes and edges correspond to documents and citations, respectively. Node fea- tures are bag-of-worte representation
three ci-
Table 1:
Dataset statistics.
Datasct
Classes
Nodes
Edges
Features
H(G)
Cora
2,708
5,429
1,433
0.81
PubMed
19,717
44,338
500
0.80
CiteSeer
6
3,327
4,732
3,703
0.74
Coreell
183
280
1,703
0.30
Chameleon
2,277
31,421
2.325
0.23
Squiree
5.201
198.493
2,089
0.22
Actor
7,600
26.752
931
0.22
Texas
183
295
1,703
0.21
Wisconsin
251
466
1,703
0.11
ogbn-arxiv
40
169.343
1.166.243
128
0.66
arXiv-year
169,343
1.166.243
128
0.22
of the documents. In web networks, nodes and edges represent web pages and hyper-links, with node features being bag-of-words representation of web pages. For the over-smoothing problem, following Rong et al. (2019). we use Cora, CiteSeer, and PubMed testbeds. For the two larger datasets, we select ogbn-arxiv from Hu et al. (2020) and arXiv-year from Lim et al. (202I). These two dataset have same nodes and edges that represent papers and citations. In ogbn-arxiv, node labels represent the papers' academic topics, which tend to be a homophily setting, while in arXiv- year, the node labels represent the publishing year, thus exhibiting more heterophily label patterns. We include the edge homophily score H(G) (Yan et al.) 2021) for each dataset, which represents the homophily level, together with the statisties of the datasets summarized in Table [ We use the
Adam optimizer (Kingma& & Ba 2014), and apply dropout and L2 regularization for fo and (kk) We also insert a LayerNorm (Ba et al. 2016) layer after every other layers. We set the hidden dimension as 256 and chunk size factor as 4. For each dataset, unless otherwise noted, layer model used (which is already deeper than most popular GNN models). We perform a grid search to tune the hyper-parameter for all the models. More details are listed in Appendix
3.1 HOMOPHILY AND HETEROPHILY
We evaluate our model's performance on aforementioned datasets, following the same data splits as Pei et al.] (2020). Results are shown in Table 2 We report the mean classification accuracy with the standard deviation on the test nodes over random data splits. For baselines, we include classic GNN models such as vanilla GCN (Kipf & Welling 2016). GAT (Vliikkvčo et al.) (017), and BaaphsSAGEHamillle
et al. 2017): ofate-of-the-art the- methods that are specifically designed for heterophily: H2GCN (Zhu et al. 2020b), GPRGNN (Chien et al. 2020), and GGCN (Yan et al. 2021I: state-of-the the-hhe-ar art models on three homophily citation network datasets: GCNII (Chen et al, 2020b), Geom-GCN (Pei et al. 2020). We also include models that utilize the information from

---

Published as a conference paper at ICLR 2023
Table 2: Experimental results on homophily and heterophily datasets, for each dataset, we bold the model with the best performance.
Texas
Wisconsin
Actor
Squirel
Chameloon
Comeel
CiteSeer
PubMed
Cora
Ordered GNN
86.22
88.04.se
37.9911.0
62.44
72.28
87.03
77.31
90.15
18.37
H12GCN
87.851..58 1.22
35.7011.00
16.48.1.86
60.1112:5
82, 26
77.11
80.491aas
87.8711.29
GPRGNN
78.36 38.38-44.0
22.94
31.63e1.22
31 61.611122
46.38e1771
77.19e1.7 20.27
17.95 17.5
GGCN
81.86=4.56
86.86es.29
37.51=1150 56
35.17-1.58
71.14=184
68
77.14
89. 15
87.95+11.06
GCNII*
77 57.5788
80 30
37 1:31
38 1.51.58
63. 03.86x3004
77. 8..8665. 79
77 77.33+1144
90.15=0.45
88.37+1.25
Geom-GCN
66. mm
31.50a1 g3.5
60.00ga:81
00.5413.07
78.02 15
80.96ga-47
85.36:11.55
MixHop
77.6 84
75. 580
32. 22+2.54 54
43 8..80=144
73.51+6.34 60).50e2.58
16.26
85.31=0061
+0.6
JK-Ner
83. 33.78=2.a1
82 56
35.14=1.39
63. 79
68.6844000 08
76.06
88. 88.41=0045
85.96
GCN
56. 14
51. 76
17.32
53 13.43=9000
64.822+2:24
00.54455.0 3o
76. 1r.50011.00 a6
88. 38.422a.50
86,9 17
GAT
52. 62.16
27.44
40.72.1.30
00.26.225
61.8915.
76.5511.95
86.3311-48
87.3011
GraphSAGE
82.43...34
81.18
34.23
41.61:0074
58.73:
76.04:1.30
a8.45
86.90111us
MLP
80.81
s5.29
36.5320.7
08..7:11.00
46.21
81.891s.40
74.02a1.0
87.16
CiteSeer
PubMed
10-
Accuracy
40-
DCM DAT
GAT
GNT
GOUTYAAGE
EXCESCN
ROSULH- D10
ROSOERH Dro
Layer
Layer
Layer
Figure 3: Node classification accuracy results for ovor-smoothing problem with various depths.
different hops: MixHop (Abu-El-Haija et al. 2019), JK-Net (Xu et al., 2018b); and finally a 2- layer MLP. For the baselines that have multiple architectural variants, we choose the best one for each dataset and denote the model with an asteriod "*" All the baseline results are from Yan et al. (2021), except for those H2GCN, JK-Net and MixHop are from Zhu et al. (2020b). All of them were evaluated on the same set of data splits as ours.
Our Ordered GNN model consistently achieves SOTA both homophilous and heterophilous datasets. On homophilous datasets, we match the performance of the SOTA model GCNII. How- ever, GCNII performs worse heterophilous datasets, leaving gap of more than 20 percents Squirrel dataset. The amazing performance on this dataset might related to the "over-squashing" problem (Alon & Yahav 2020: Topping et al., 202T) (see Appendix [B.3). We also achieve SOTA on the heterophilous datasets and consistently outperform basclines across all datasets. The superior
performances on both homophily and heterophily data show the generality of our model.
3.2 OVER-SMOOTHING
We test our model's performance with depth 2, 4, 8, 16, 32 to evaluate the ability in alleviating the over-smoothing problem, we include following models for comparison: classical GNN models GCN (Kipf & Welling 2016). GAT (Veličković et al.) 2017) and GraphSAGE (Hamilton et al. 2017), models that modily the architecture to alleviate over-smoothing: ResGCN (Li et al. 2019). Je- et al. 2018b). and IncepGCN (Rong et al. 2019). We also include three state-of of- art methods ResGCN+Drop. JK-Net+Drop. and IncepGCN+Drop that is equipped with the advanced training trick DropEdge (Rong et al., 2019) which is developed for alleviating over- sworssnoo,,ing. We reuse the results from Rong et al. (2019) for all baselines except GAT. For GAT, conduct
7

---

Published as a conference paper at ICLR 2023
Lear
Lear
Levi
Love
(a) Homophily
(b) Heterophily
(c) Over-smoothing
Figure Visualization of gating vectors across layers in our models on (a) homophily, (b) heterophily and (c) over-smoothing settings. Vertical and horizontal axes are gating vectors and chunk channels, respectively. Each box in the boxplot refect the distribution of the value of the corresponding gate across the whole dataset.
separate grid search for each depth to tune it's hyperparameters, with the search space the same to ours for fair comparison. Detailed configurations can be found Appendix We follow the same data split from Rong et al. (2019) used in baselines.
Performances are shown Figure 3 Compared with the baseline models, our model consistently achieve the best performance across all datasets and all layers. Furthermore, for each dataset, our model's performance doesn't 't drop significantly as some baselines. Note that this is achieved without using training tricks such as DropEdge (Rong al. 2019). This shows that our model can well alleviate the over-smoothing problem.
3.3 VISUALIZING THE GATING VECTORS
We checkout the gating vectors across all layers in our model, plotted in Figure We can find that the gates automatically learns to behave differently for homophilous and heterophilous data, and leamss to suppress message passing when needed to prevent over- wworshmoo
On homophilous data the split points distributes more uniformly first layer, with a gradually increasing trend of gating vectors. As the layer gets deeper, gates start to saturate. After layer 4, most of the positions are saturated, leaving the last very few positions open for update. This coinsides with the fact that 4 layers are the depth when many GNNs starts to show the over-smoothing phenomenon.
On heterophilous data, large block of gates become saturated even in the first layer, emphasizes the importance of node's ego information by suppressing most of the incoming messages. This observation consistent with the amazing performance Texas dataset the saturated gates keep the same for layer
simple MIPP model when deal with
heterophily (see Table 2). Morcover, and 2, however enlarge on layer 3, reflect the flexibility our gating mechanism and the ability to capture "good heterophily" for high-order neighbors as described in Section 2.5
We also plot the gating vectors of our model on Cora dataset in Figure H()) all the gates closed when layers become very deep. This phenomenon reveals Ordered GNN's superior performance on deep models: manages to keep node representations distinguishable by learning to reject incoming messages from farther-away neighbors.
We can see that almost
8

---

Published as a conference paper at ICLR 2023
Table 3: Summary of classification accuracy results larger benchmarks, we bold the model with the best performance.
GCN
SGC
C&S 72.62
JK-Net 72.19a0.01
APPNP 00:38
GCNII
UniMP
Ordered GNN 72.81
ogbn-arxiv
71.74
00.39
68.50
12.7430.16
73.1
GCN
soC
C&S
JetNet
APPNP
LINK
H2GCN
Ordered GNN
arXiv-year
26
32 33
16.07e0.16
42.17
16, 28,000
38.15e0..5
53.97+0.18
19.09
54.4920.9
3.4 SCALING TO LARGER DATASETS
The larger ogbn-arxiv and aiv-v-year datasets contain much more nodes, and present a complex linking pattern. We compared our model with these models: baseline graph neural models SGC [Wu et al. 2019), GCN (Kipf & Welling, 2016), label propagation based methods LP (Zhou et al.;
2003). C&S (Huang al.J 2020), strong baselines: JK-Net (Xu et al. (018b), APPNP (Klicperal et al. 2018). and state-of-the art model UniMP (Shi et al. 2020), GCNII (Chen et al. (020b) for ogbn-arxiv, and LINK (Lim et al. 2021). H2GCN (Zhu al. 2020b) for arXiv-year Features, links and labels are inherited from Hu et al. 22220), all the models follow the same data split from Hu
al. (2200) and Lim al. (021))
1). The baselines results are from Lim et al. 2021
al. (020aa)
and Huang et al. (2020).
For homophily dataset agxii, our model beats all the baseline methods except UniMP. This is understandable since UniMP based on the graph Transformer much larger model size, and uses the extra node label information. For heterophily dataset, our method outperform the SOTA models. The performance on these two large dataset demonstrate the scalability and expressivity of our model to capture the complex patterns in large networks.
3.5 ABLATION STUDY
We perform an ablation study to evaluate our proposed gating mechanism, Table 4 shows the re- sults. "Bare GNN" denotes a simple GNN model with mean aggregator; "+simple gating" denotes adding a gating module where each gate is predicted independently by the gating network, this is equivalent to the Gated GNN (Li et al. 2015) with mean aggregator; "++ordered gating" denotes an ordered gating module as we introduced in Equation 6 "*++SOFTOR" indicates our proposed model Ordered GNN. We test on three settings: homophily, heterophily, and over-smoothing. We use CiteSeer for both homophily and over-smoothing settings and Wisconsin for heterophily setting. We set model layers
for the first two settings and 32 for the last one.
As we can see, the "Bare GNN" model performs poorly in all three settings, espe- cially in the heterophily setting, revealing the challenge of these settings; the "+sim- ple gating" model and the "++ordered gat- ing" model perform closely, both provid- ing performance improvements, and let the model comparable to strong baselines; with the help of our proposed differentiable OR mod- ule, the "+++SFFOOR" model obtains SOTA performance, demonstrating the effectiveness of our proposed methods. Overall, the ablation study verified each component we designed for combine stage of message passing is useful and necessary.
Table 4: Ablation study
Model
Homophily
Heterophily
Over-smoothing
Bare GNN
73.79
53.92=6.79
74.70
+simple gating
86.27
79.30
++ordered gating
76.45
85.69
79.40
++4SOPOOR
77.3
88.042a.63
$0.50
CONCLUSION
We propose Ordered GNN, novel GNN model that aligns the hierarchy of the rooted-tree of a central node with the ordered neurons in its node representation. This alignment implements more organized message passing mechanism, by ordering the passed information into the node embedding w.r.t. their distance towards the node. This model results to be both more explainable and stronger in performance. achieves the ofate-of-the-an on various datasets, ranging across both homophily and heterophily settings. Moreover, we show that the model could effectively prevent over-smoothing, when the model gets very deep. We discuss the connections with related works, the limitation of our model and broader impact building more powerful GNNs in Appendix B due to the space limit.

---

Published as a conference paper at ICLR 2023
5 ETHICS STATEMENT
In this work, we propose a novel GNN model to deal heterophily and over-smoothing problem. Our model integrates inductive bias from graph topology and does not contain explicit assumptions about the labels of nodes on graph, thus avoiding the occurrence of (social) bias. However, the gating mechanism of our model needs learn from samples with labels, and is possible be misleading by targeting corrupted graph structure or features, which can result in negative social impact, because both internet data and social networks can expressed with graphs. We can develop corresponding ethical guidelines, and constraint the usage of our model to avoid such problems.
6 REPRODUCIBILITY STATEMENT
We provide the introduction for our used datasets and model configuration in Section the space of grid search to tune our model and the hardware & software experimental environment are introduced in Appendix E.] we also post every set of hyper-parameters to reproduce results in Table 7 Table 8 and Table 9
including the baseline model's. The download, load, and pre-process steps introduced in Appendix E We've test our model on settings, the setting for heterophily & homophily are introduced in Section [.1] the setting for over-smoothing is introduced in Section 3.2 and the setting for larger benchmarks are introduced Section In our released source code, we list the steps needed to reproduce results, and also list all the param-erraeeters with yaml files.
ACKNOWLEDGEMENT
This work was sponsored by the National Natural Science Foundation of 62106143), and Shanghai Pujiang Program (No. 2IPJ1405700).
China (NSFC) grant (No.
REFERENCES
Sami Abu-EI-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learn- ing. pp- 21-29, PMLR, 2019.
Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and practical implications. In International Conference on Learning Representations, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 160.006400 2016.
Deyu Bo, Xiao Wang. Chuan Shi, and Huawei Shen. Beyond frequeeyyccy information in graph convolutional networks. arXiv preprint arXiv:2101.00799 2021.
Giorgos Bouritsas, Fabrizio Frasca, Stefanos P 7afeiriou, and Michael Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203. 2013.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 3438-3445, 2020a.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph con- volutional networks. In International Conference on Machine Learning, pp. 1725-1735. PMLR, 2020b.
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. arXiv preprint arXiv:2006.07988. 2020.
10

---

Published as a conference paper at ICLR 2023
Weilin Cong, Morteza Ramezani, and Mehrdad Mahdavi. On provable benefits of depth in training graph convolutional networks. Advances in Neural Information Processing Systems, 34:9936- 9949, 2021.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, 2016.
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. arXiv preprint arXiv:21 10.07875. 2021.
Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Evgeny Kharlamov, and Jíe Tang. Graph random neural networks for semi-supervised learning graphs. Advances in neural information processing systems, 33:22092-22103, 2020.
Justin Gilmer, Samuel Schoenholz, Patrick F Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs Advances in neural information processing systems, 30, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 18 22133, 2020.
Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R Benson. Combining label propagation and simple models out-performs graph neural networks. arXiv preprint arXiv:2010. 13993, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint 1rXiv:1412.6980 2014.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pp. 2688- 2697. PMLR, 2018.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: NXXi:1669000007, 2016.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:8810.05997, 2018.
Kwei-Hemng Lai, Daochen Zha, Kaixiong Zhou, and Xia Hu. Policy-gnn: Aggregation optimization for graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 461-471, 2020.
Chang Li and Dan Goldwasser. Encoding social information with graph convolutional networks forpolitical perspective detection in news media. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp.
2019.
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go deep as cnns? In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9267-9276. 2019.
Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergen: All you need to train deeper gens, arXiv preprint 07X77.2006.07739 2020a.

---

Published as a conference paper at ICLR 2023
Pan Li, Yanbang Wang. Hongwei Wang. and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. Advances in Neural Information Processing Systems, 13:4465-44777 2020b.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference artificial intelligence, 2018.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprini arXiv: 1.05493,
Derek Lim, Xiuyu Li, Felix Hohne, and Ser-Nam Lim. New benchmarks for learning on non- homophilous graphs. arXiv preprint arXiv:2 2001:22004000 (10.001144,4 2021.
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 338-348, 2020.
Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-War Chang, and Doina Precup- Is heterophily a real nightmare for graph neural networks to do node classification? arXiv preprint (09.v:2009.05644 2021.
Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily necessity for graph neural networks? arXiv preprint 206.v:2/660061 34, 2021.
Yimeng Min, Frederik Wenkel, and Guy Wolf. Scattering gen: Overcoming oversmoothness in graph convolutional networks. Advances in Neural Information Processing Systems, 33:14498- 14508, 2020.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. arXiv preprint 2XXi:2002.052887 2020.
Yu Rong. Wenbing Huang. Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. arXiv preprint arXi: 2019.
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-Scale Attributed Node Embedding. Journal of Complex Networks, 9(2), 2021.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.
The graph neural network model. IEEE transactions on neural networks, 20(1))61-80, 2008
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Cliassi-Rad Collective classification in network data. Al magazine, 29(3):93-93, 2008
Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron Courville, and Yoshua Bengio. Straight to the tree: Constituency parsing with neural syntactic distance. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1171- 2018a
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. arXiv preprint arXiv:1810.09536 2018b.
Yunsheng Shi, Zhengjie Huang, Shikun Feng. Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint 0:X20: 2020.
José Varela, Paul Almasan, Miquel Ferriol-Galmés, Krzysztof Rusek, Fabien Geyer, Xiangle Cheng, Xiang Shi, Shihan Xiao, Franco Scarselli, Albert Cabellos-Aparicha et al. Graph neural networks for communication networks: Context, use cases and opportunities. arXiv preprint 22Xiv:21 2021.
Ke Sun. Zhanxing Zhu, and Zhouchen Lin. Adagen: Adaboosting graph convolutional networks into deep models. arXiv preprint arXiv: arXiv: 000000 2019.
12

---

Published as a conference paper at ICLR 2023
Susheel Suresh, Vinith Budde, Jennifer Neville, Pan Li, and Jianzhu Ma. Breaking the limit of graph neural networks by improving the assortativity of graphs with local mixing patterns, arXiv preprint arXiv:2106.0655 2021.
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. arXiv preprint arXiv:2111.14522, 2021.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks, arXiv preprint arXiv: 2017.
Haorui Wang. Haoteng Yin, Muhan Zhang, and Pan Li. Equivariant and stable positional encoding for more powerful graph neural networks. arXiv preprint ar arXiv:2203.00119, 2022a.
Zhen Wang, Zhewei Wei, Yaliang Li, Weirui Kuang, and Bolin Ding. Graph neural networks with node-wise architecture. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1949-1958, 2022b.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference on machine learning, pp- 6861-6871. PMLR, 2019.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful graph neural networks? arXiv preprint arXiv: 180.0008, 2018a.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning, pp. 5453-5462. PMLR, 2018b.
Yujun Yan, Jiong Zhu, Marlena Duda, Eric Solarz, Chandra Sripada, and Danai Koutra. Groupinn: Grouping-basee interpretable neural network for classification of limited, noisy brain data. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 772-782, 2019.
Yujun Yan, Milad Hashemi, Kevin Swersky, Yaoqing Yang, and Danai Koutra. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. arXiv preprint arXiv:2102.06462 2021.
Liang Yang, Mengzhe Li, Liyang Liu, Chuan Wang, Xiaochun Cao, Yuanfang Guo, et al. Diverse message passing for attribute with heterophily, Advances in Neural Information Processing Sys- tems, 34:4751-4763. 2021.
Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. In International conference on machine learning, pp. 7134-7143. PMLR, 2019.
Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kan- nan, Viktor Prasanna, Long Jin, and Ren Chen. Decoupling the depth and scope of graph neural networks. Advances in Neural Information Processing Systems, 34:19665- 2021.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems, 16, 2003.
Jiong Zhu, Ryan A Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K Ahmed, and Danai Koutra. Graph neural networks with heterophily. arXiv preprint arXiv: 13566, pp. 11168 11176, 2020a.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33:7793-7804, 2020b.
13

---

Published as a conference paper at ICLR 2023
A PROOF
A.1 SOFT BINARY GATE
Ideally, we want obtain binary gate vector ged (1, .1..... 1,0, ..,,,,.. 0), and first p(() gate is all 1s. We denote p(() as a categorical random variable: p( p!!t) = softmax(....) Then for each position / in the gate vector, the probability of I-th gate gr
(k) U| being could be calculated with the
disjunction of p! the gate ge [X] being holds when (M 2 I, as the categories are mutually
exclusive, we get the probability of 1-th gate being in gate vector: p(ge (k) [1] 1) p(P
Ep(p! = i), this can be implement with cumulative sum. Now we get the probability of gate
being 1, the gate vector can be treat as it's distribution: g cunsum_(softmax( As go
is a binary vector, this is equivalent to computing the expectation of it.
A.2 CONVERGENCE RATE OF GATES
We analyze the convergence trend of gating vectors follows: from Equation have g(
(k-1) 1) + (1 g0th ((--1) )) Dr then the finite difference for g(0h is Age (R)
(k+FT)
Jo
go
go (1
Po ) o gl since every gate's 's value is between 0 and 1, we have Age (R) 0, thus the gate vectors has the trend to saturate as the layer goes deeper.
(1)
However, the rate of the gate saturation can be flexible, in order explain this more formally, we further compare the finite difference of gates at different positions: for the gate with index
i, we assume i j. we ignore node notation for simplicity, then for gate g. 19 P (1-1) Em= (((
=(k)
have
E E (k+ Σm= 9n
(k++) 1(++1) + 9) (k+1)
2 e(k+1) E 1),
1(k+1) the "raw gate" output from the linear layer of the gating network, and it's
here denote gn
index is n, then we have:
Ag(h) (k) (R)
=(1 9. (k) (0: (k+l)) - (1- -1(a)) (0)
(9)
= (1 g. (R+ + 1 - (1 9j (0) (k+1))
0 1k+ (0) (k) ) + e(++1) (1 - g (k)
here 0/50) g. < 0 and gi > 0, and و3
(k+11 2(b+1) = E (k++) since (k+1) En 9j
every gn .(k+1) is generated by a linear layer and don't rely on historical values, then the relative size of g k-1)) and 1(+++1 is not restricted, of*h (
/2(k+1) 1) (0, +0o), then the relative size of (g(h) and is not restricted, which means the convergence rate of gates different positions can be de- termined by gating network and thus leaving flexible mechanism predict P!!) pP!)) ,... (())
Ag
k))
we only ensure the relative magnitude of split points, but not restrict ( between GNN layers, this make it possible to assign flexible block of neurons for ego information and neighborhood information at different orders, theoretically.
B DISCUSSION
B. CONNECTIONS WITH RELATED WORKS
Heterophily Heterophily is knotty problem for GNNs, widely accepted method to deal with it is to allow the coefficient of message to be negative (Chien et al. 2020 Yang et al. 202 Bo et al. 2021: Luan al.) 202 I: Yan al. 2021), this signed message let model distinguish harmful neighbor information bad heterophily" and reject them, we can also do this with our proposed gating mechanism; some recent works (Luan al. 2021
Ma al. 202T) shows that there also
14

---

Published as a conference paper at ICLR 2023
"good heterophily" that are friendly to GNNs, but didn't show way capture it actively rather then passively accept it, however, our model can extract it from high-order neighbors as we described in Section 2.3] It is worth noting that model only design the combine stage, thus we can incorporate the signed message into aggregate stage to further boost the performance.
Over-smoothing Over-smoothing a very active area in graph learning, although our model focus on the combine stage of message passing mechanism, we can link it to previous successful methods and provide insight about why our model works. Similar to JK-Net (Xu et al. (2018b) combination of different scale information, our model allows different blocks of neurons to represent different hop information; our model locks the ego information of nodes at hop=0 in a specified block of neurons, which well preserved it and is similar to APPNP (Klicpera et al.) 2018) and GCNII (Chen) itt al. 2020b)'s ss initial connection. While these useful skip connections seem to be ad-hoc, our ordered gating mechanism can be treated as a unified way to incorporate them; beyond just combine contexts within different hops, we further model the internal relationship (tree hierarchy) between contexts.
Ordered Neurons Ordered Neurons (Shen et al. 2018b) is originally proposed to induce the syn- tactic tree structure from LSTMMbased language models. They apply two master gating networks to control the update frequency of neurons at different positions, and employed the syntactic distance (Shen et al., 2018a) convert the gating result into syntactic trees. Our method also relies on or- dering the neurons, but the induction is in an opposite direction: the rooted-tre theeettree structure is known a priori, while the gating results are learned based on the tree structure instead.
B.2 LIMITATIONS
As for limitations, for scale-free graphs that contain both large hubs low-degree nodes, the rooted- tree rollout for certain nodes would vary considerably, thus our model's performance could depend on whether the data distribution is balanced, in some extreme cases, might need to resort to the few-shot learning technique to deal with them; secondly, since our model only modifies the combined stage, we may need to modify the aggregate stage when modeling complex dependencies between nodes (e.g. knowledge graphs), we leave it for the future work.
B.3 BROADER IMPACT
Over-squashing Our model achieves amazing performance on the Squirrel datasets, and we be- lieve that related to the "over-squashing" problem (Alon & Yahav 2020 Topping et al.] [202T). From the dataset statistics listed Table we can see that the Squirrel dataset the densest dataset we'r ve tested, with the ratio of edges over nodes being 40, which is 2.6 times larger than the second densest dataset Chameleon. Thus we suspect that the baselines' lousy performance is caused by the "over- squashing" problem (Alon & Yahav 2020 Topping et al. 2222), i.e., the number of nodes contained in each node's receptive field will grow exponentially with GNN layer's growth, thus making the local information squeezed. This problem will be much more severe in more densely connected graphs. Since the baselines mainly focus on the design of the aggregate stage of the weessage-passing mechanism and ignore the combine stage, they cannot prevent squashing. On the other hand, our model effectively preserves the local and ego information with the ordered gating mechanism. We believe studying the over-squashing problem with our ordered form of message-passingg would be an excellent future direction.
Powerful GNNs
Although in this work we mainly focus on our model's performance for node- level tasks, if we put it into a broader context of building more powerful GNNs (Xu et al. 201 Ba) that have stronger structural distinguish power, the integration of rooted-tree hierarchy can serve as implicit learned positional encoding: with an embedding structure that forcing the order of neurons to represent the distance and importance to the central node, the central node can be aware of precise neighborhood structure, instead of blurred impression about it's neighbors within few hops.
This learned positional encoding is different from previous injected positional encoding ZZhag & Chen 2018) You et al., 2019; Li et al., 2020b] Bouritsas et al., (2022) that inject pre-computed graph heuristics into node embeddings and recent learnable positional encoding (Dwivedi et al., 2021
15

---

Published as a conference paper at ICLR 2023
Table 5: Averaged per epoch running time different settings.
Model
Homophily
Heterophily
Over-smoothing
Large-scale
Ordered GNN
86.52ms 102.6lms
52.96ms
243.97ma 245.2
255.28ms
GAT
66.18ms
232.13ms
Table 6: Model's performance 64 layers when dealing with the over-smoothing.
Methods
Cora
Citeseer
Pubmed
Ordered GNN
89.10
80.50
91.60
GCN
52.00
44.60
79.70
ResGCN
79.80
21.20
87.90
JKNet
86.30
76.70
90.60
IncepGCN
85,30
79.00
OOM
GraphSAGE
31.90
16.90
40.70
GCN+DrooEEde
53.20
45.60
79.00
ResGCN+DropEdge
84.80
75.30
90.20
Drodee
87.90
80.00
91.60
IncepGCN+DropEdge
88.20
79.90
90.00
GraphSAGE+DrooEddge
31,90
25.10
63.20
Wang et al. 2022a) that provide individual learning channel to extract the injected positional features, we actually open a door for learning positional encoding without external source. Exploring this new (implicit) positional encoding method graph-level and link-level tasks, which are more dependent on positional and structural features, will be a very interesting future direction.
C RUNNING TIME
We report the averaged per epoch running times of the models 4 settings (homophily, heterophily, over-smoothing and large-scale settings) in Table we use relatively larger datasets PubMed, Actor, PubMed and ogbn-arxiv for these settings, apply 8, 8, 32, 4 GNN layers for models respectively. We use GAT as the baseline for comparison, we implement GAT and our model with the same PyTorch Geometric framework, and share the training/evaluation code, running the same hardware. The results shows that our model consume similar training time compared to the popular model GAT.
D FURTHER INCREASING THE GNN LAYERS
Although our Ordered GNN model shows excellent performance in Table 3 when dealing with over-smoothing problem, one may still be curious about how the model will perform at a deeper level. We showed our model's performance at 64 layers Table 6 We can see that our model could still consistently outperform other methods on 3 datasets. Although some baselines perform well, they enjoyed the extra benefits from the advanced training trick DropEdge. Without the help of DropEdge (see ResGCN and GraphSAGE), datasets and much worse than ours; if we further compare to GCN without advanced architecture modification (c.g. residual connection in ResGCN, dense connection in JKNet), its performance deteriorated sharply.
that their performance is volatile across
E EXPERIMENTAL DETAILS
We perform grid search to tune the hyper- hyper-parameters for models, including dropout rate, learning rate, weight decay, and MLP layers for fo. The baseline model GAT on the over-smoothing problem shares the same search space model for the same configurable items. We tune dropout and
16

---

Published as a conference paper at ICLR 2023
Table 7: Best hyper-parameters for homophily and heterophily setting.
Dataset
Hyper-parameters
Cora
dropoutg:0.1. d0opoutt::.2,, dropouts:0.4. dropoute;0. 055 1r:0.001, MLP layers:2
:5e-06, L2, -5c
:500 1r:00005, MLP layers:
CiteSeer
:5e-08,
PubMed
dropoutg:0.44 dropouts :0 1.,,0000055L2..5500061000000,,,0 layees::
Actor
dropouts:0,, dropoute :0, L2, :0.000000, 42, 10.05, ML layers:2
Texas
dropouts:0.33 dropout 10 L2, :5c000 L2, 00.05, Ir:0.005, MLP layers:
Wisconsin
d dropout, 0.22, 1,, 1r:0.005, MLP layers:1
Cornell
10g 1r:00005, MLP layers:
Chameleon
dropouls:0.1. h/tp::/hg...uooou.........///000 10,0000000,,00000000,,0,,,,,,,,, 10:0000 Ir:0.005, MLP layers:
Squirrel
droponls:0.3, dropoul. =0.0005 L,2 :0.0005, lr:0.005, MLP layers:1
weight decay for So and _e separately as they represent different functionalities. We run up to 2000 epochs, select the best model and apply an early stop with 200 epochs based on the accuracy of the validation set. We use NVIDIA GeForce RTX 3090 and NVIDIA GeForce RTX 2080 as the hardware environment, and use PyTorch and Py PoTch Geometric as our software environment. The datasets are downloaded from PyTorch Geometric, we also dataloader provided by PyTorch Geometric to load and pre-process datasets. We fix the random seed for reproducibility. We report detailed search space as follows:
Search space for homophily and heterophily experiments; dropout (0.0..10.2,0.33004)) weight de- cay (5e-2.5e-4,5e-655--)) learning rate (0.01.0.005,00001)) MLP layers (1,2,3), on PubMed and Texas datasets, we tied parameters for (() across layers; we applied empty features (Zhu et al, 2020a) and reversed edges on Chameleon and Squirrel datasets. We report the best parameters for each model-dataset combination in Table 7 Search space for over-smoothing experiments: dropout (0.00.1002.03333044000.... weight decay 1,5555--2.5e44,5--6,5----
learning rate 10.01,0.005,0.001
MLP layers (1,2). We tied parameters for ((k) across layers for all models except models with 32 layers. We report the best parameters for each model-dataset combination in Table g Search space for expressivity experiments: dropout (0.05,0.1,0.1555
weight decay (0,5e-4,5e-8), we fix
the MLP layers as 3, and learning rate as 0.005, we tied parameters for (k) across layers ogbn- arxiv dataset. We report the best parameters for each model-datasee combination in Table
17

---

Published as a conference paper at ICLR 2023
Table 8: Best hyper-parametters for over-smoothing problems.
Dataset
Method
Layers
Hyper-parameteen
droponlo10.4. dr. L2, :5e-08, Lz, 000005 Ir:0.01, MLP layers:
droponla::... d.oposll:0.2. L2, :5e-06, 15-555008. Ir:0.001, MLP layers:2
Ordered GNN
dropouta;0... dropoulg:0.3. 15e-08, 20.000 Ir:0.01, MLP layers:2
dropoute:0.2. dropout;:0011 Lz, :5c-06. L2, 30,05, Ir:0.005, MLP layers:2
dropouts:0.5, dropout; L2, :0.05, 20.05, 1r:0.001, MLP layers:2
Cora
dropoul:0.4,
1,200,0005, Ir:0.005
dropoul:0,5. L-255e088 Ir:0.005
GAT
dropord:30.4. Z.2:5c=06, Ir:0.01
droport:00.3, 1,220, 1r:0.001
Ir:0.001
dropouts:0.. dropoulg:0.3, L2g L2, :3e-06, 20055, Ir:00005, MLP layers:
dropouts:0.. :.33, dropouts:0.22 :5e 1.0000. Ir:0.01. MLP layers:
Ordered GNN
dropouts:0.44 dropout, :0, L2, :5e-06.
:5- 1r:00001, MLP layers:2
dropouto:0.4, dropout,: L2, :0.05. L2, 53-06, Ir:0.001, MLP layers:2
dropoutt:0.5. 0.5,, dropouf, 2..00.000055 L2, :5c-08, 1r:0.001.
MLP layers:2
CiteSeer
dropout:0.5, Lc:3c-06, Ir:0.005
dropout:0.5,,
L2:5c-06, Ir:0.001
GAT
dropouf:0,
L2:5e-06, Ir:0,005
dropoul: 0022, Lg255-06, lr:0.001
32
dropoul:0.1,
L2:3e-08, lr:0.001
dropouty;0, 30.3, 1.2, 1,z, :5c-06, = 20,0005, I10001, MLP layers:2
dropoutt dropouuf:3..1 1.z, 30,00015, 1.2,, 50,0005, 1r:001 MLP layers:2
Ordered GNN
8
droponto:..2. droponts :0.1, 1.2, :5c-088 7,2, :5e-06, 1E:1005, MTP layers:2
dropoutg::.22 dropout; 1.2, -0.05, 1,2, 3005, Ir:0,0, MLP layerss::
PubMed
32
d.0 dropoule:0. L2, 30.5, Lz, 00,005, 1r:00005, MLP layerss:
dropout:0.2. L2:3e-08, Ir:0.01
4
dropout::.1, 1.230, 1r:0.01
GAT
8
2.. L2:5e-08, 1r:0.01
16
dropout:0, L2:5e-06, Ir:0.005
32
dropouf:0, L2:5e-08, lr:0.001
Table 9; Best hyper-parameters on larger datasets.
Dataset
Hyper-parameters
ogbn-arxiv
dropoutg:0.33 0.00ott L2, 0.0000, L2, :5c008
arXiv-year
dropouts:0.1, dropott 0005, L2, :5c008, L2,
18


Reviews:
[{"review_id": "wNCUu7fZQl", "reviewer": "Reviewer_gn7g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new message passing method for GNN. It orders the message passing into the node representation with specific blocks of neurons targeted for message passing within specific hops. Extensive experiments show the effectiveness of the proposed method on homophily and heterophily data. It also alleviates the over-smoothing problem. ", "summary_of_the_review": "My worry about this paper is on the experimental part. Most of the comparison methods are published in or before 2020. Only GGCN, which was published in 2021, is compared. Much progress has been achieved in 2021 and 2022. Therefore, it is fairer to compare some of them. Furthermore, the improvement is also not significant in most cases.", "strength_and_weaknesses": "The strength of this paper includes:\n1. The paper is well-motivated. \n2. The experiments are extensive.\n3. The analysis is sufficient.\n The weaknesses are:\n1. The experimental results are not impressive. The improvement is not significant.\n2. The comparison methods are not STOA. Some recent works are ignored.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "uBOOmldJOP", "reviewer": "Reviewer_rLLb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an ordering mechanism for the messages passing in GNN for node representation learning. In particular they are focused on the combining stage in the GNN procedure. The ordering mechanism is achieved by the rooted-tree hierarchy and a curated gating function. which can iteratively preserve the ego information and gradually absorb k-hop neighboring features. The authors claimed to address the problem of “over-smoothing” and “heterophily”. The experiment results show the performance boost in both settings above. And the large scale dataset experiment demonstrated the scalability of the proposed method.", "summary_of_the_review": "The proposed Ordered GNN approach is clear and the experiments and the ablation studies demonstrated the effectiveness of each part of the design. The algorithm is tested in both small and large scale dataset. I recommend for accept.", "strength_and_weaknesses": "Strength:\n1. Intuitive and effective design of the gating function to iteratively update the feature while preserving the ego node info by inductive bias using ordered neurons.\n2. Sufficient number of experiment in both small and large scale dataset.\n3. Good presentation, easy to follow.\n\nTo be clear:\n1. For the experiment result in the Squirrel dataset, the performance surpass the baselines by a quite large margin. I'm wondering if there is an underline reason for such improvement.\n2. Fig 3 shows some baselines which previously designed to deal with the over-smoothing problem are actually pretty good. Even though there overall performance doesn't beat the proposed Ordered GNN. A good question is that if the number of layers is further increased, what would happen? In another word, is such carefully designed gating procedure actually needed for solving the over-smoothing problem?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TTEbag9ntov", "reviewer": "Reviewer_UBsM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an Ordered GNN for handling the heterophily problem, which attempts to improve the updating process in the message passing scheme. The paper studies an important problem, i.e., how to extend current GNNs to heterophilic graphs.", "summary_of_the_review": "The paper studies an important problem, i.e., how to extend current GNNs to heterophilic graphs. Unfortunately, there are many confusions and presentation problems in this paper, which makes it hard to follow. Besides, I also have several concerns about its motivation, detailed method, and evaluations.\n", "strength_and_weaknesses": "This paper presents an Ordered GNN for handling the heterophily problem, which attempts to improve the updating process in the message passing scheme. The paper studies an important problem, i.e., how to extend current GNNs to heterophilic graphs. Unfortunately, there are many confusions and presentation problems in this paper, which makes it hard to follow. Besides, I also have several concerns about its motivation, detailed method, and evaluations.\n\nComments\n1. I am afraid that there are many confusions and grammar errors in the paper, which makes it hard to follow. For example, in Sec. 2.2, I am wondering what \"one round of message passing\", \"the neurons used to represent the nodes’ ego information\", \"the neurons used to represent message passing within Tao^{(k-1)} should be a subset of those for Tao^{(k)}\", and \"the delta between k-1 and k\" mean. Besides, the paper may benefit from revising certain presentations, e.g., \"its k-th order rooted tree\", \"And since …\", \"Apparently here we have…\", \". where Tao^{(0)} corresponds to…\". I am afraid that the paper needs to be carefully revised.\n2. I am afraid that some utilized mathematic symbols lack formal definitions and thus induce unnecessary confusions. For example, Tao^{(k)}_{v} is defined as the \"k-th order rooted tree\" of node v. However, it is not clear what its subset operation is (as shown in Eq. (3)) and why \"Tao^{(0)}_v corresponds to the neurons used to represent the ego representations\".\n3. I am afraid that the reviews of related work in Sec. 1 are confusing. This paper focuses on two drawbacks of the message passing scheme, i.e., over-smoothing and heterophily. It is not clear which problems these reviewed literatures were designed to solve. It actually seems that this paper considers these two problems as the same problem.\n4. In Sec. 2.5, I am afraid that the intuitive analysis for Fig. 2 is not convincing. The paper states \"In Case 1, two root nodes share the same first-order neighborhood pattern, one GNN layer can capture this good heterophily and obtain good performance\". This analysis only considers node 1 and node 2. However, the children of these two nodes possess different situations which are not considered in this analysis. The analysis for Case 2 also has a similar problem.\n5. I am afraid that the results of the employed baselines in Table 2 are most borrowed from a preprint paper, which is not convincing. For example, I am just wondering why the results of GRP-GNN are much lower than its reported ones.\n6. I am just wondering how the proposed method can specifically solve the over-smoothing and heterophily problems.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EPfWzplm0Rk", "reviewer": "Reviewer_44qL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed one ordered GNNs to mix the features calculated from different hops. This method can achieve the SOTA performance on both homophily and heterophily graphs, and could prevent the over-smoothing problem.", "summary_of_the_review": "It is interesting to employ the ordered neurons in the graph representation learning and the experiments can demonstrate the effectiveness. However, the proposed method is not well motivated and some baselines are lacked. ", "strength_and_weaknesses": "### Strengths:\nIt is interesting to introduce the ordered neurons into graph representation learning, and it is one straightforward solution for the over-smoothing problem. The experimental results demonstrate the effectiveness in preventing this problem.\n\n### Weakness:\n\n1. The motivation for separating the different hops is not clear. As mentioned in the introduction, what are the advantages of “model the information exactly at some orders”? \n\n2. The evaluations on the node-wise split points are lacking. Designing the personalized operation for each node helps to improve the model performance [1,2]. It seems that different nodes have different split points as mentioned in Section 2.3. This aspect is not evaluated in the experiments, either by comparing with some node-wise GNNs or showing the different split points. Besides, there lacks a detailed discussion about the ordered neuron methods to verify the contributions of this paper.\n\n3. The information extraction on the ego features. The self-loop-free graphs are utilized in this paper. How to incorporate the node features from itself? \n\n### Refs:\n[1] Graph Neural Networks with Node-wise Architecture. KDD 2022\n\n[2] Policy-GNN: Aggregation Optimization for Graph Neural Networks. KDD 2020\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster