A Practical, Progressively-Expressive GNN
Authors: Lingxiao Zhao
Neil Shah
Leman Akoglu
Venue: NeurIPS 2022 Accept (2022)
Keywords: GNN
k-WL
expressiveness

Abstract: Message passing neural networks (MPNNs) have become a dominant flavor of graph neural networks (GNNs) in recent years. Yet, MPNNs come with notable limitations; namely, they are at most as powerful as the 1-dimensional Weisfeiler-Leman (1-WL) test in distinguishing graphs in a graph isomorphism testing frame-work. To this end, researchers have drawn inspiration from the k-WL hierarchy to develop more expressive GNNs. However, current k-WL-equivalent GNNs are not practical for even small values of k, as k-WL becomes combinatorially more complex as k grows. At the same time, several works have found great empirical success in graph learning tasks without highly expressive models, implying that chasing expressiveness with a “coarse-grained ruler” of expressivity like k-WL is often unneeded in practical tasks. To truly understand the expressiveness-complexity tradeoff, one desires a more “fine-grained ruler,” which can more gradually increase expressiveness. Our work puts forth such a proposal: Namely, we first propose the (k, c)(≤)-SETWL hierarchy with greatly reduced complexity from k-WL, achieved by moving from k-tuples of nodes to sets with ≤k nodes defined over ≤c connected components in the induced original graph. We show favorable theoretical results for this model in relation to k-WL, and concretize it via (k, c)(≤)-SETGNN, which is as expressive as (k, c)(≤)-SETWL. Our model is practical and progressively-expressive, increasing in power with k and c. We demonstrate effectiveness on several benchmark datasets, achieving several state-of-the-art results with runtime and memory usage applicable to practical graphs. We open source our implementation at https://github.com/LingxiaoShawn/KCSetGNN.


A Practical, Progressivey--xxpeesiie GNN
Lingxiao Zhao Carnegie Mellon University lingxiao@emu. edu
Louis Härtel
RWTH Aachen University
haertel @etheeelliiffffmmaiii wtt.mmmaa.....ttaaaaaaaaaaaa www.wwwchhh...eeem
Neil Shah
Leman Akoglu
Snap Inc. nshah@snap. com
Carnegie Mellon University
lakoglu@andrev. cmu. edu
Abstract
Message passing neural networks (MPNNs) have become dominant flavor of graph neural networks (GNNs) in recent years. Yet, MPNNs come with no- table limitations; namely, they are most as powerful as the 1-dimensional Weisfeiler-Leman (1-WL) test in distinguishing graphs in graph isomorphism testing framework. To this end, researchers have drawn inspiration from the k-WL hierarchy to develop more expressive GNNs. However, current k-WL- equivalent GNNs are not practical for even small values of k, as k-WL becomes combinatorially more complex as k grows. At the same time, several works have found great empirical success in graph learning tasks without highly expressive models, implying that chasing expressiveness with "coarse-graing ruler" of expressivity like &- often unneeded in practical tasks. To truly under- stand the
tradeoff, desires more "fine-grained ruler," which can more gradually increase expressiveness. Our work puts forth such proposal: Namely, we first propose the (k, (S)-EEET hierarchy with greatly reduced complexity from k-WL, achieved by moving from k-tuples of nodes to sets with <k nodes defined over Sc connected components in the in- duced original graph. We show favorable theoretical results for this model in relation to k- WL, and concretize it via (k. (EES)) which as expres- sive as (k, ( increasing in power with and We demonstrate effectiveness on several bench- mark datasets, achieving several state-of-the-art results with runtime and mem- ory usage applicable to practical graphs. We open source our implementation http://giibb..om//iiiiiiiaaaaaae
Our model is practical and progressively-expeees
Introduction
In recent years, graph neural networks (GNNs) have gained considerable attention [58, 60] for their ability tackle various node-level [35, 56), link- [51, 66) and graph-level [9, 40] learning tasks, given their ability to learn rich representations for complex graph-structured data. The common tem- plate for designing GNNs follows the message passing paradigm; these so-called message-passing neural networks (MPNNs) are built by stacking layers which encompass feature transformation and aggregation operations over the input graph [25, 39]. Despite their advantages, MPNNs have sev- eral limitations including oversmoothing [12, 47, 67). oversquashing [2, 55], inability to distinguish node identities [63] and positions [62], and expressive power [61].
Since Xu et al.'s [61] seminal work showed that MPNNs are at most as powerful as the first-order Weisfeiler-Leman (1-WL) test in the graph isomorphism (GI) testing framework, there have been several follow-up works on improving the understanding of GNN expressiveness [3, 15]. In re-
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

0-1 sots
super edgne sual supergraph
B-Bpertile Super-graphi
cud sets
super edges 13, and) super grach
2-sets
3-sets
K-Bpanile GNN
Crd sots
eiqur only in 65, 3) super straph
13
145
Super-raphh
Build Super-graph
14
124
Backward Forward
23 Sels Singraphs
Sat "Caler
15
134
Bidirectional Message Passing
Bidirectional Message Passing
3(s)-Sets
All
GNN
24
135
Colors (Emb.)
Input Graph
D) GNN
1-sets
123
Pooling
<>> GNN
23
125
<>
GNN 045
25
234
Graph Emb.
GNN
145
Figure 1: Main steps of (k; (s)) Given a graph and (k, c), we build the (k; e)-bipartite super-graph (in middle) containing sets with up to k nodes and e connected components in the induced subgraph, on which a base GNN assigns initial "colors". Bidirectional bipartite GNN layers with frw.-bck. message passing learn set embeddings, pooled into graph embedding. The size of super-graph, and accordingly its expressiveness, grows progressively with increasing k and c. The 2- bipartite message passing generalizes normal GNN, edge GNN, and line graph (see Appendix.A.12)
sponse, the community proposed many GNN models to overcome such limitations [4, 52, 68). Several of these aim to design powerful higher-order GNNs which increasingly expressive [9, 33, 41, 43] by inspiration from the k-WL hierarchy [54].
A key limitation of such higher-order models that reach beyond 1-WL is their poor scalability; in fact, these models can only be applied to small graphs with small k in practice [40, 41, 43] due to combinatorial growth in complexity. On the other hand lies an open question on whether practical graph learning tasks indeed need such complex, and extremely expressive GNN models. Historically, Babai et al. [5] showed that almost all graphs on nodes can be distinguished by 1- WL. In other contexts like node classification, researchers have encountered superior generalization performance with graph-augmented multi-layer perceptron (GA-MLP) models [37, 50] compared to MPNNs, despite the former being strictly less expressive than the latter [13]. Considering that increased model complexity has negative implications in terms of overfitting and generalization [29], it is worth re-evaluating continued efforts to pursue maximally expressive GNNs in practice. Ideally, one could study these models' generalization performance (expressiveness) in the k- LL hierarchy. Yet, given the impracticality of this too-coarse "ruler" of expressiveness, which becomes infeasible beyond k>>3, one desires more ine-grained "ruler", that is, new hierarchy whose expressiveness grows more gradually. Such a hierarchy could enable us gradually build more expressive models which admit improved scaling, and avoid unnecessary leaps in model complexity for tasks which do not require them, guarding against overfitting.
various real-world tasks by increasing
Present Work. In this paper, we propose such hierarchy, and an associated practical progressively- whose expressiveness can be modulated through
expressive GNN model, called (k, k and In a nutshell, we take inspiration from k-WL, yet achieve practicality through three design ideas which simplify key bottlenecks of scaling &-WL: First, we move away from k-tuples of the original k- to k-multisets (unordered), and then k<<)-setss We demonstrate that these steps drastically reduce the number nodes in the k-WL graph while retaining significant expressiveness. (Sec.s 3.2-3.3) Second, by considering the underlying sparsity of an input graph, we reduce scope to k; d(s)-sett that consist of <k nodes whose induced subgraph is comprised of SeSk connected components. This also yields massive reduction in the number of nodes while improving practical- ity: i.e. small values of e on sparse graphs can allow one to increase well beyond 3 in practice. (Sec. 3.4) Third, we design super-graph architecture that consists of sequence of k-1 bi- partite graphs over which we learn embeddings for our k; c( -ssstts using bidirectional message passing. These embeddings can be pooled to yield final graph embedding... (Sec.s 4.1-4.2) We also speed up initializing "colors" for the k, csst-sssss for tation and memory. (Sec. 4.3) Fig. 1 overviews of our proposed framework. Experimentally, outperforms existing state-of-the
1, substantially reducing compu-
our (k, ()) well as real-world graph-level tasks, achieving new bests on substructure counting and ZINC-12K, respectively. We show that generalization performance reflects increasing expressiveness by k and Our proposed scalable designs allow us to train models with e.g. 1=10 or c=3 with practical running time and memory requirements.
GNNs on simulated expressiveness as

---

2
Related Work
MPNN limitations. Given the understanding that MPNNS have expressiveness upper-bounded by 1-WL [61], several researchers investigated what else MPNNS cannot learn. To this end, Chen et al. [15] showed that MPNNs cannot count induced connected substructures of 3 or more nodes, while along with [15], Arvind et al. [3] showed that MPNNs can only count star-shaped patterns. Loukas [38] further proved several results regarding decision problems on graphs (e.g. subgraph verification, cycle detection), finding that MPNNs cannot solve these problems unless strict conditions of depth and width are met. Moreover, Garg et al. [22] showed that many standard MPNNs cannot compute properties such as longest or shortest cycles, diameters or existence of cliques.
Improving expressivity. Several works aim to improve expressiveness limitations of MPNNs. One approach is to inject features into the MPNN aggregation, motivated by Loukas [38] who showed that MPNNs can be universal approximators when nodes are sufficiently distinguishable. Sato al. [53] show that injecting random features can better enable MPNNs to solve problems like minimum dominating set and maximum matching. You et al. [63] inject cycle counts as node features, moti- vated by the limitation of MPNNs not being able to count cycles. Others proposed utilizing subgraph isomorphism counting to empower MPNNs with substructure information they cannot learn [9, 10]. Earlier work by You et al. [62] adds positional features to distinguish node which naive MPNN embeddings would not. Recently, several works also propose utilizing subgraph aggregation; Zhang et al. [65]. Zhao et al. [68] and Bevilacqua et al. [8] propose subgraph variants of the WL test which are no less powerful than 3-WL. Recently a following up work [21] shows that rooted subgraph based extension with 1-WL as kernel is bounded by 3-WL. The community is yet exploring several distinct avenues in overcoming limitations: Murphy et al. [46] propose a relational pooling mecha- nism which sums over all permutations of a permutation--snnsitve function to achieve above- expressiveness. Balcilar et al. [6] generalizes spatial and spectral MPNNs and shows that instances of spectral MPNNs are more powerful than 1-WL. Azizian et al. [4] and Geerts et al. [24] unify expressivity and approximation ability results for existing GNNs.
k ---iipispieed GNNs. The k-WL test captures higher-order interactions in graph data by consider- ing all k-tuples, i.e., size ordered multisets defined over the set of nodes. While highly expressive, it does not scale to practical graphs beyond a very small i (k = 3 pushes the practical limit even for small graphs). However, several works propose designs which can achieve k-WL in theory and strive to make them practical. Maron et al. [41] proposes general class of invariant graph networks k-IGNs having exact k- expressivity [23], while being not scalable. Morris et al. have several work on WI and its variants like k-L.WI [42], k-GNN [43], and --W-NNGGN [44]. Both k-LWI. and G-GNN use a variant of 6- that only considers k-sets and are strictly weaker than kL but much more practical. In our paper we claim that k(S)-sets should be used with additional designs to keep the best expressivity while remaining practical. The --L-WL-NNN works with k-tuples but sparsifies the connections among tuples by considering locality. Qian et al. [48] extends the k-tuple to subgraph network but has same scalability bottleneck as k- A recent concurrent work Spe- qNet [45] proposes to reduce number of tuples by restricting the number of connected components of tuples' induced subgraphs. The idea is independently explored in our paper in Sec.3.4. All four variants proposed by Morris et al. do not have realizations beyond k > 3 in their experiments while we manage to reach k = 10. Interestingly, a recent work [34] links graph transformer with k-IGN that having better (or equal) expressivity, however is still not scalable.
3 A practical progressivellyeepprssss
isomorphism test: (K:C)())-SETWL
We first motivate our method (k: (k, ()s))-TTTTWL We first introduce notation and background of k-WL (Sec. 3.1). Next, we show how to increase the practicality of k-WL without reducing much of its expressivity. by removing node ordering (Sec. 3.2), node repetitions (Sec. 3.3), and leveraging graph sparsity (Sec. 3.4). We then give the complexity analysis (Sec. 3.5). We close the section with extending the idea to k-FWL which is as expressive as (k+1)-WL (Sec. 3.6). All proofs are included in Appendix A.4.
from the GI testing perspective by introducing
Notation: Let G = (V(G), E(G), (c) be an undirected, colored graph with nodes V(G), edges E(G), and color-labeling function lo V(G) + C' where C' = (cI....,Ca Ca} denotes a set of distinct colors. Let [n] = {1, 3, ..., nI. Let (-) denote a tuple (ordered multiset), {(-)) de- note a multiset (set which allows repetition). and {.) denote set. We define o (u1..., wk) as a k-tuple, : = Kui,.., 0% B as a k-multiset, and = {rI,.., 2k) as a k-set. Let क. [z/i]
3

---

(Ui...., .... 1;-1, 2, 1it,...., Ui8) denote replacing the i-th element in with and analogously for 0|//j] and [// (assume mutlisets and sets are represented with the canonical ordering). When E (G)), let G J. G( G( denote the induced subgraph G with nodes inside V.0, ü, re- spectively (keeping repetitions). An isomorphism between two graphs G and G' is bijective map- ping p V(G) + ((G") which satisfies Vu, E V(G), (u, u) E E(G) < (o(u).p())) E E(G') and Vu E V(G), (u)) = lc (p(u).. Two graphs G, G' are isomorphic if there exists an isomor- phism between them, which we denote G = G'.
3.1 Preliminaries: the :: Leman (k-WL) Graph Isomorphism Test
The I-dimensional Weisfeiler- (1-WL) test, also known as color refinement [49] algorithm, is a widely celebrated approach to (approximately) test GI. Although extremely fast and effective for most graphs (1-WL can provide canonical forms for all but n 117 fraction of n-vertex graphs [5]) it fails to distinguish members of many important graph families, such regular graphs. The more powerful k-WL algorithm first appeared in [57], and extends coloring of vertices (or 1-tuples) to that of k-tuples. k-WL is progressively expressive with increasing k, and can distinguish any finite set of graphs given a sufficiently large k. k- has many interesting connections to logic, games, and linear equations [11, 28]. Another variant of k-WL is called k-FWL (Folklore WL), such that (A + 1)-WL is equally expressive as k-FWL [11, 26]. Our work focuses on k- k-WL.
k- &-WL iteratively recolors all n² k-tuples defined on a graph with n nodes. At iteration O, each k-tuple रे = (vi, ................................ ux) E V(G)* is initialized with a color as its atomic type al&(G, Φ). Assume G has d colors, then ata
E {0, 1)2(2))) (b)++ is an ordered vector encoding. The first (%) entries indicate whether 8; Uj. Vi, j. 1si<jSk which is the node repetition information. The second (9) entries indicate whether (ve, U2) E E(G).. The last kd entries one-hot encode the initial color of Uy Vi, S sis k. Importantly, at& (G, 8) = at& o') if and only if 0 + of is an isomorphism from G[ G. Let ! (G, Φ) denote the color of k-tuple 전 graph G at I-th iteration of k- WL, where colors are initialized with w0 (G, d¹ = atx(G, )).
At the 1-th iteration, k- WL updates the color of each E V(G)* according to wt w (G, HHASH wt("(c, (c, t) ff ( v/./// DEvIGI) }......,.. ..... (WN)) (I)
Let g (')())))) (G)) denote the encoding of G at t-th iteration of k-WL. Then,
(G, gunl (G) HASH { m0 (G, क) /) P v((cr+ }}
Two k-tuples Je and से are connected by an edge if Hi [K]],; w.Yl k k- 1, informally they share (k-1) entries. Then, k- WL defines a super-graph Sk-w with its nodes being all k-tuples in G. and edges defined above. Eq. (1) defines the rule of color refinement on St-wi(G) (G). Intuitively, k- WL is akin to (but more powerful as it orders subgroups of neighbors) running 1-WL. algorithm the supergraph Wl.w(()), As l + xc, the color wit (G, δ) converges to stable value, denoted as (x)()) (G, f) and the corresponding stable graph color denoted as gwol (G). k gurl! (20) (G'). The expressivity of wi! (t) can be exactly characterized by first-order logic with count- k
For two non-isomorphic graphs G, G" k-WL can successfully distinguish them if gwt (20) (G) +
ing quantifiers. Let Cj denote all first-order formulas with at most A variables and t-depth counting
quantifiers, then wil (G, - wi(h (G', d²) = Vg E Ct (GG F) = "(G", w²). Additionally, there is t-step bijective pebble game that are equivalent to t-iteration k-WL in expressivity. See Appendix.A.. A.2 for the pebble game characterization of k-WL.
Despite its power, k- uses all n tuples and has Okr)*) complexity at each iteration.
3.2 From WL to M-MULTISET WL: Removing Ordering
Our first proposed adaptation to k-WL is to remove ordering in each k-tuple, i.e. changing k-tuples to k-multisets. This greatly reduces the number of supernodes to consider by O(k!)) times.
Let e Vo,..., 2 be the corresponding multiset of tuple 3 (V1. vx). We introduce canonical order function on G. V(G) + [m]] and corresponding indexing function og - i),

---

( which returns the i-th element of V1, ... Ug sorted according to og- Let murl (G, 6) denote the color of the m-multiset ú at t-th iteration K-MULTISETWL, formally defined next.
At / = O, we initialize the color of e as muok 50 (G, 6) - HASH( (atx:G,,pp())) where perm[k] denotes the set of all permutation mappings of l elements. It can be shown that mww (0) (G, ú) maol (0) (G', .0') if and only if Gl and G'( are isomorphic.
E perm[k]])
At (-th iteration, k K-MULTISETWL updates the color of every k-multiset by
murt m 6) HASH mwl (G, (). { (mml [ (C, (6.. V(C)
(3)
.... Amall (6, E
Where 0/x//0" V(G). Let (G) denote the super-graph defined by B-MULTISETWL. Similar to Eq. (2), the graph level encoding is gmwll (1) (G) = HASH( m (G, d) Vū E m)))).
(8, i)] denotes replacing the i-th (ordered by 06) element of the multiset with
Interestingly, although k- 6-MULTISET WL has significantly fewer number of node groups than k-WL, we show it is no less powerful than k- W-L in terms of distinguishing graphs, while being upper bounded by k- WL in distingushing both node groups and graphs.
(9 (G, ö) == !wll (())))) E perm[k]]. For all t E N and all graphs G, G": k- 6-MULTISETWL is upper bounded by k- in distinguishing multisets G, ū and G², d' at t-th iteration, i.e. w (G, ú) = x (G', i') = mwu (G,ö) " (G',ö')
Theorem 1. Let & 2 1 and wt
Theorem 2. k-MULTISETWL is no less powerful than (k-1)-WL in distinguishing graphs: for any 12 there exists graphs that can be distinguished by k-MULTISETWL but not by (k-1)-WL.
Theorem is proved by using a variant of a series of CFI [11] graphs which cannot be distinguished by k-WL. This theorem shows that M-LLLTISETWL is indeed very powerful and finding counter examples of k- B- distinguishable graphs that cannot be distinguished by k k-MULTISETWL. is very hard. Hence conjecture that k M-MULTISETW tinguishing undirected graphs, with an attempt of proving it in Appendix.A Arrendix.A.44111.. 10. Additionally, the theorem also implies that MUULLTISTTWL is strictly more powerful than (k- 1)-MULTISETWI.
may have the same expressivity as k- in dis-
We next give a pebble game characterization of k- M-MULTISET WI., which named doubly bijective k-pebble game presented in Appendix.A.3. The game is used in the proof of Theorem 2.
Theorem 3. A-MULTISETWL. has the same expressivity as the doubly bijective k-pebble game.
3.3 From h- MULLISETT WL to (<<)-SETWL: Removing Repetition
Next, we propose further removing repetition inside any k-multiset, i.e. transforming k-multisets k(s)-sets. We assume elements of k-multiset ö and k(<)-set in G are sorted based on the ordering function oc, and omit og for clarity. Let s(-) transform a multiset to set by removing repeats, and let r(-) return tuple with the number of repeats for each distinct element in a multiset. Specifically, let è s(Ø) and n = r(0), then m := [6] distinct elements
|n| E [k]] denotes the number of k-multiset ú, and Vi E [m]. è; is the i-th distinct element with á; repetitions. Clearly there is an injective mapping between é and (ő. n): let f be the inverse mapping such that = f(s(ū), r())). Equivalently, each m-set è can be mapped with a multiset of k-multisets: ò <> i = -( n) Li=1 â. k, Vi á. 2 18. Based on this relationship, we extend the connection among k-multisets to k(S)-sets: given mi m2 E [k), a mse is connected to a m22sstt ú if and only if 3 nu. f(ū, f.) is connected with f(u, n2.) in k-MULTISETWI. Let (k-sm(G)) denote the defined guper-graph on G by k(5)-SETWI, SL. It can be shown that this is equivalent to either (1) (lm1 mal = 1) A (l n ûl min(m,, (2)) or (2) (mi m2) A (l6 n âl mi 1) is true. Notice that Sx swi (G) contains a sequence of k-1 bipartite graphs with each reflecting the connections among the (m-1)-sets and the m-sets. It also contains k-1 subgraphs, i.e. the connections among the m-sets for m = 2..... k. Later on we will show that these k-1 subgraphs can be ignored without affecting k(<) (K))-SETWL.
m
'This function should also consider repeated elements; omit this for clarity of presentation.
5

---

Let s k (G, 0) denote the color of m-set é at I-th iteration of k(S)-SETWL. Now we formally define -SETWLL At = 0, we initialize the color of a m-set e (m E (k:) as:
swi 00) (C, 6) HASH (m (0 (G, f(0, +)) 0i +... 7m k Vi n 2 1B)
(4)
Clearly swl, (0) (G, 0) swl (G', 6') if and only if G[ and G²[6'] are isomorphic. At t-th iteration,
updates the color of every m-set é by
swlk (G, HASH s 6), !swl ( ( (z)) x E EVC)) \ 6B. ( (=,
(G, (6. 1))) x V(G) \ oB.... [ (G, //z//0 (0, m)]) (0, =VV(()) b13))
(5)
Notice that when m = and m = k, the third and second part of the hashing input is an empty multiset, respectively. Similar to Eq. (2), we formulate the graph level encoding as gswl (G) HASH((swll (G)
6) Vb E (G))B.
To characterize the relationship of their expressivity, we first extend K-MULTISETWL. on sets by defining the color of a m-set on k-MULTISETWL as mwrk (1) (G. é) ==
( kmoll (0)
[(, (G, i)) -i=1 6; Vi n. 2 1B. We prove that K-MULTISETWL is at least in terms of separating node sets and graphs.
as expressive as
Theorem 4. Let k 21 1, then Vt E N and all graphs G,G': G¹: murt (G, 6) mwrk ( (G', 0') =
swo (2) (G, swi, (G", o'). We also conjecture that k-MULTISETWL and ((S)-SETWL could be equally expressive, and leave it to future work. As a single m-set corresponds to (t-)) k-multisets, moving from k- MULTISETWL to k(S)-SETWL further reduces the computational cost greatly.
3.4 From (S)-SETWLL to (k:, (SS)-SETW.. Accounting for Sparsity
Notice that for two arbitrary graphs G and G' with equal number of nodes, the number of k(S)- sets and the connections among all k(<)-sets in k(5)-SETWL are exactly the same, regardless of whether they are dense or sparse. We next propose to account for the sparsity of graph G to further reduce the complexity of ((S)-SETWL. As the graph structure is encoded inside every m- set, when the graph becomes sparser, there would be more sparse m-sets with potentially large number of disconnected components. Based on the hypothesis that the induced subgraph over a set (of nodes) with fewer disconnected components naturally contains more structural information, we propose to restrict the (<$)) )-sets to be (k, (=)-sset::: all sets with at most i nodes and at most connected components in its induced subgraph. Let Sk. denote the super-graph defined by (k, ()) then (G))-sw(G) = Sk-soi ( HCop@o/Hcmmmmetsssccii induced subgraph on the super-graph defined by ((S)-SE Fortunately, Sk.o-si can be efficiently and recursively constructed based on S(x-1.c)-wi(G)) and we include the construction algorithm in the Appendix A.5. (E) SL can be defined similarly to k(S)-SETW) WL (Eq. (4) and Eq. (5)), however while removing all colors of sets that do not exist on Sk.
< c)]. which is the
(ks, (((S)SSTWT
progressively expressive with increasing i: and e, and when h. (k, c))
SETWL becomes the same as k(5) as all k(S)-sets are then considered. Let swl (1)
k.c (G, 0)
denote the color of (s) e t-th iteration of (s) then gswl k.c (G)
(1)
WASH((swll) (3) k.c (G, b) VO (G.)sw(()) B.
Theorem 5. Let 2 1, then Vt E N and all graphs G,G': G":
(1) when 1 S Ci < 02 S k, if G, G cannot be distinguished by (K)) they cannot be distinguished by (5)e)))
(2) when kn < liz, Ve S kb, ifG,G' cannot be distinguished by (k2, (55)SSWW) they cannot be distinguished by (k1, (5))))(W)
3.5 Complexity Analysis
All color refinement algorithms described above run on a super-graph; thus, their complexity at each iteration is linear to the number of supernoees and number of edges of the super-graph. Instead

---

of using bigO notation that ignores constant factors, we compare the exact number of supernodes and edges. Let G be the input graph with n nodes and average degree d. For k-WL, there are n supernodes and each has n= k number of neighbors, hence wI)) has n² supernodes and km kn*+1/ /2 edges. For m E [k:. there are (m) m--sets and each connects to m number of (m 1)-sets. So Sk-swl has Σ
( m-k+1 supernodes and Σ (({) = n E (*7)) S n((=l) a-k+1 edges 2k+1
n-2k+2
(derivation in Appendix). Here we ignore edges within m-sset for any m E [K] as they can be reconstructed from the bipartite connections among (m s)-ttts and m-sett, described in detail in
Sec. 4.1. Consider e.g., n = 30, = 5; we get [V(Sz=-w
[()) 2182. Directly [V)) 139, (G))
analyzing the savings by restricting number of components is not possible without assuming a graph family. In Sec. 5.3 measured the scalability for (k: ()) -ETGNNN with different number of components directly on sparse graphs, where (k) WL. shares similar scalability.
3.6 Set version of k-FWL
k-FWL is a stronger GI algorithm and it has the same expressivity as k+1-WL [26], in this section we also demonstrate how extend the set k-FWL to get ((S)-SETWLL. Let fwl (G, d) denote the color of k-tuple &े at t-th iteration of k-FWL. Then k-FWL is initialized the same the k-W i.. fwlt (G, ) = wt (0) (G, d). ) At t-th iteration, k-FWL updates colors with
- swl (G. (). . (swlf (G. (a/l)///.......... oo
(G.T///4) sE sEv(Ca)))
(6)
Let f (G, d) denote the color of m-set e at f-th iteration of ((T)-S Then at t-th iteration it updates with
(t) &
(G (G.0)=HASH( (fftt (f { (G.Gle//))........... (...s
sevIci)) (evvoc))))))))))))
of wf
(7)
The k(s) SETFWL should have better expressivity than k(S)-SETWL. We show in the next sec- tion that k( ((S)-SETWL can be further improved with less computation through an intermediate step while this is nontrivial for ((S)-SETFWL. We leave it to future work of studying -(E)-SL
4 A practical progressively-expressis GNN: (s)c)))))SETGN))
In this section we transform (k:, (SSSTT to GNN model by replacing the HASH function in Eq. (5) with combination of ML.P and DeepSet [64], given they are universal function approxima- tors for vectors and sets, respectively [31, 64]. After the transformation. we propose two additional improvements (Sec. 4.2 and Sec. 4.3) to further improve scalability. We work on vector-attributed graphs. Let G = (V(G), B(G), X) be an undirected graph with node features E Rd Vi E (G).
4.1 From (ks (SeeSWL to (k, -SSSSGNN
(k, (-SSEE defines super-graph Sk.c-swt which aligns with Eq. (5). We first rewrite Eq. (5) to reflect its connection to Sk.eswl. For a supernode in Sk.oowl, let M/ (0) = {ú E Sk.o,wl; <>> and ú = |o] - 1}, and A
right (0) = {ūr â E Sk.cswi, ú +> è and Í = |8] 1).
Then we can rewrite Eq. (5) for (k, c) (=)SE-E as
swih (G, (swt( (G, 0), swi (+ (G. 0), (aw! (G, | e (oi Lawl (G M (
where swl (1+ st s.c (G, = := (
E EVG
nighl H.) Notice that we omit HASH and apply
it implicitly. Eq. (8) essentially splits the computation of Eq. (5) into two steps and avoids repeated computation via caching the explicit t+ step. It also implies that the connection among m-sets for any m E [k] can be reconstructed from the bipartite graph among m-sett and (m 1)-sets.
Next formulate (k: (-)e)))))TTGNN formally. Let 10 (6) Rd denote the vector representa-
tion of supernode è on the t-th iteration of (k:, (SEEGENNN For any input graph G, it initializes representations of supernodes by
h [0) (6) BaseGNN(G(o))
where the BaseGNN can be any GNN model. Theoretically the BaseGNN should be chosen to encode non-isomorphic induced subgraphs distinctly, mimicking HASH. Based on empirical tests
7

---

of several GNNs on all (11,117) possible 8-node non-isomorphic graphs [6], and given GIN [61] is simple and fast with nearly 100% separation rate, we use GIN as our BaseGNN in experiments. (ks, ())))))))TTNN SETGNN iteratively updates representations of all supernodes by (() =
E
MLP 2) (t))
(10)
(e (+)
(0) MLP (10)
Σ (E
MLP
Σ
MLP (1(+)())))
(11)
GEMM)
Then after T iterations, we compute the graph level encoding
(G) POOL ( (T) ê E V(Sk.oow))
(12)
where POOL can chosen as summation. We visualize the steps in Figure 1. Under mild condi- tions, (ks, (B..))))SSTGN))) has the same expressivity as (k, (
Theorem 6. When (i) BaseGNN can distinguish any non-isomorhpic graphs with most l nodes, (ii) all MLPs have sufficient depth and width, and (iii) POOL is an injective function, then for any t N, t-layer (ki ()))))(()
is expressive as (ke, (5,c))))-TTWW the t-th iteration.
The following facts can be derived easily from Theorem and Theorem 6.
Corollary 6.1. (k, (SSSTTNN is progresiively-eepessive with increasing is and c, that is, is more expressive than (k, c2) (5) -SEGGN,, and SETGNN is more expressive than (k22, (52,c)(()-STT))) -SETGNN.
(1) when 01
(2) when h > 12.
4.2 Bidirectional Sequential Message Passing
The f-th layer of - (Eq. (10) and Eq. (11) essentially propagating informa- tion back and forth on the super-graph Sw.e-w((G), which is a sequence of k-1 bipartite graphs (see the middle of Figure 1), in parallel for all supernodes. We propose to change it to bidirectional blockwise sequential message passing, which we call (k: -))=S)SETGGNN: defined as follows.
m = k - to 1, Vm-set i, b0+ = MLP m.
(1
Σ
MLP
(13)
GENE
m to k, Ve--set é, (0) MLP((+)) (1 (6).
Σ
MLP 1) mL
(1ttt (i))
(14)
REA/C Mn
Notice that (k, (SEEGNN) has lower memory usage, (k, (K.c)( plete supergraph (k- bipartites) while (ks ()) loads 1 out of k bipartites at time, which is beneficial for limited-size GPUs. What is more, for small, finite t it is even more expres- sive than (k, <) (S)-SETGNN We provide both implementation parallel and sequential message passing in the official github repository, while only report the performance of (E:))(S)-SETGNN- given its efficiency and better expressivity.
load the com-
Theorem 7. For any t E N, the t-layer (k,, -ESTGNN is more expressive than the t-layer (EEGGNN As lim, (EECGN is as expressive as (K..)(S)-SETGNN.
4.3 Improving Supernode Initialization
Next we describe how to improve the supernode initialization (Eq. (9) and extensively reduce computational and memory overhead for without losing any expressivity. We achieve this by the fact that graph with components can be viewed as a set of connected components. Formally,
Theorem 8. Let G be a graph with connected components C1,..., Co and G' be a graph also with e connected components Ci,,..,C', C.,, then G and G' are isomorphic if and only if 3p : [c] + [c], s.1.
Vi [c], C, and p(t) are isomorphic.
The theorem implies that we only need to apply Eq. (9) to all supernodes with a single com- ponent, and for any c-components i with components i...... úe, can get the encoding by 1,00) ( DeepSet((1*)) (1))) ..., (0) ((.))))) without passing its induced subgraph to BaseGNN. This eliminates the heavy computation of passing a large number of induced subgraphs. We give the algorithm of building connection between é to its single components in Appendix A.6.

---

Table 1: Simulation data performances. For (k. (=) --EEEGNN" (k: c) values that achieve re-
ported performance in parenthesis. (ACC: accuracy, MA[S]E: mean abs.[sq.] error)
Method
EXP (ACC)
SR25
Counting Suhstructurrss (MAF)
Graph Properties (MSE))
(ACC)
Triangic
Tailed Tri.
Star
4-Cyele
IsConnected
Diameter
Radius
GCN
50%
6.67%
0.4186
0.3248
0.1798
0.2822
-1.7057
-2.4705
-3,9316
GIN
50%
6.67%
0.3569
0.2373
0.0224
0.2185
-1.9239
-3.3079
-,7584
PNA
50%
6.67%
0.3532
0.2648
0.1278
0.2430
-1.9395
-3,4382
4,9470
PPGN
100 %
6.67%
0.0089
0.0096
0.0148
0.0090
-19804
-3.6147
-5.0878
GIN-AK+
100%
6.67%
0.0123
0.0112
0.0150
0.0126
-2.7513
-3,9687
-5.1846
PNA*-AK*
100%
6.67%
0,0118
0.0138
0.0166
0.0132
-2.6199
-3,9011
55,0026
(ke, e)(
100%
100%
0.0073
0.0075
0.0134
0.0075
-5.4667
4,0800
-5.1603
23. 2)
214
(3. 2)
14
(3.2)
(4, 1)
(4.
(4, 1)
(2, 2)
Table 2: Train and Test performances on substructure counting tasks by varying & and c. Notice the orders of magnitude drop in Test MAF between bolded entries per task.
Counting Substractures (MAF)
Trinngle
Tailed Tri.
4-Cylle
Train
Test
Train
Test
Train
Train
Test
0.994] a
a
1.1506 0:2542
a 0.0292
a
0,0311 + 0.0025
0.0088 a 0.0001
0.0303 + 0.0108
0.00KS a 0.0018
0.0559 e 0.0019
0.0151 à 0.0906
0.1351 + 0.0058
0.18V3 a 0,083)
0.032] 1 0.0008
0.0151 2 0.0074
0.0307 1: 0.0085
0.0075 2 0.0012
0.0687 :: 0.00001
0.0339 2 0.0008
0.03:9 1 0.000
0.0075 2: 0.0002
0.8902 + 0.0070
0.0208 + 0.0042
0,0853 + 0.0009
0.0189+ 0.0024
0.0565 + 0.0007
0.0266 + 0.0023
0.0377 + +.0007
D.0115 + 0.0036
0.0344: a 0.0024
0.0247 a 0.0085
0.0357 a 0.0017
0.0171 a 0.0000
0,0560 : 0.0000
0.0168 a 0.0022
0,0356 e 0.0014
0.0163 a 0.0064
0,3452 à 0.0329
0.4029
0.0053
0,2723 a 0.0157
0,2898 à 0.0055
0,0466 2: 0.0025
0,0242 à: 0.0006
0,2369 a: 0.0123
0,2512 à 0.0029
0.0234 a 0.0030
0.0073 0.0009
0.0296 à 0.0074
0.0100 a 0.0009
0.0640 000000
0.0131 à 0.0006
0.0181
a 0.0135
0.0194 à 0.0065
0,0587
0,0438 000011
0,0094 0.0002
0,0488 0.0088
0,0209 0:003
0.0464 0:0057
0.0110 0:000
5
Experiments
We design experiments to answer the following questions. Q1. Performance: How does (ke (S)- SETGNN compare SOTA expressive GNNs? Q2. Varying k and c: How does the progressively increasing expressiveness reflect on generalization performance? Q3. Computational require- ments: Is (.))))))STTGN) feasible on practical graphs w.r.t. running time and memory usage?
5.1 Setup
To inspect the expressive power, we use four different types of simulation datasets: 1) EXP [1] contains 600 pairs of 1&2-WL failed graphs which we split into two where graphs in each pair assigned to two different classes; 2) SR25 [6] has 15 strongly regular graphs (3- WI. failed) with 25 nodes each, which we transform to 15-way classification task; 3) Substructure counting (i.e. triangle, tailed triangle, star and 4-cycle) tasks on random graph dataset [15]; 4) Graph property regression (i.e. connectedness, diameter, radius) tasks on random graph dataset [17]. We also evaluate performance on two real world graph learning tasks: 5) ZINC-12K [19], and 6) QM9 [59] for molecular property regression. See Table 6 in Appendix A.7 for detailed dataset statistics.
Datasets.
Baselines. We use GCN [36], GIN [61], PNA" * [17], PPGN [40], PF-GNN [18], and GNN-AK [68] as baselines on the simulation datasets. On ZINC-12K we also reference CIN [9] directly from literature. Most baselines results are taken from [68]. Finally, we compare to GINE [32] on QM9. Hyperparameter and model configurations are described in Appendix A.8.
5.2 Results
Theorem 2 shows that a M-MULTISETWL is able to distinguish CFI(k) graphs. It also holds for k({)-
SETWL. as the proof doesn't ' use repetitions. We implemented the construction of CFI(k) graphs for any k. Empirically (k>6 out of memory). This empirically verifies its theoretical expressivity.
found that (k, ()S--SETGNN is able distinguish CFI(k) for k = 3, 4,
Table shows the performance results all the simulation datasets. The low expressive models such as GCN, GIN and PNA* underperform on EXP and SR25, while 3-WL equivalent PPGN excels only on EXP. Notably, than 2 and 3, and any e larger than and O, resp. for EXP and SR25. (k:, SEETNNN also outperforms the baselines all substructure counting tasks, as well property prediction tasks (except Radius) with significant gap, for relatively small values of k and c.
achieves 100% discriminating power with any l larger
two out of three graph
In Table 2, we show the train and test MAES on substructure counting tasks for individual values of k and c. As expected, performances improve for increasing when e fixed, and vice versa. is notable that orders of magnitude improvements on test error occur moving from 2=2 to 3 for the
9

---

triangle tasks as well as the star task, while similarly large magnitude drop is obtained at k=4 for the 4-cycle task, which is expected as triangle and 4-cycle have 3 and 4 nodes respectively. Similar observations hold for graph property tasks as well. (See Appendix A.9)
In addition to simulation datasets, we evaluate our (5)) SETGNN on real-world data; Table 3 shows our performance on ZINC-12K. Our method achieves new state-of-the-art performance, with a mean absolute error (MAE) of 0.0750, using k=5 and c-2.
Table 3: SeIGNN achieves
SOTA on 1INC-12K.
Method
MAE
GaledGCN
0.363 1: 0,009
+ 0.009
In Table 4 we show the test and validation MAE along with training loss for varying k and for ZINC- For a fixed c, validation MAE and training loss both follow first decaying and later increasing trend with increasing k, potentially owing to the difficulty in fitting with too many sets (i.e. supernodes) and edges in the super-graph.
PNA
0.188 + 0.004
DGN
0.168 +: 0,003
GIN
4: 0.004
GINE
0.157 + 0.004
HIMP
0.151 + 0,006
1: 0.006
GSN
0.115 + 0.012
PF-GNN
0.122 + 0.010
Similar results are shown for QM9 in Table 5. For comparison we also show GINE performances using both 4 and 6 layers, both of which are significantly lower than (k, ()SETGNN
GIN-AK
0.080 1 0.001
CIN
0.079 + 0.006
(s,)(()
0.0750 1: 0.0027
Table 4: (Ii, (B)
performances
Table 5: (ki, (S-SEGGN performances on
on ZINC-12K by varying (k,c). Test MAE at lowest Val. MAE, and lowest Test MAE
QM9 by varying (ki,). Test MAE at lowest Val.
MAE, and lowest Test MAE. All variances are
Train loss
Val. MAB
Test MAE
<0.002 and thus omitted.
1
0.1381 = 0.0240
0.2429 :: 0.0071
0,2345 = 0.0131
c
Train loss
Val. MAB
Test MAE
0.1172 0.0063
0.2298= 0.0060
0,2252 0.0030
0.0693 + 0.01
0.1645 + 0.0052
0.1636 0.0052
2
0,0376 0.0005 0.0005
0.0387 0.0007
0,0389 1 0.0008
0.0643 0.0019
0.1593 0.0051
0.1447 0.0013
0,0308 0.0010 0.0386 0.0017
0.0379 0.0010
0.0519 = 0.0064
0.0994 + 0.0093
0.0843 0.0048
4
0,0338 0.0003
0,0371 = 0.0005
0.0370 : 0.0006
0.0543 0.0048
0.0965 0.0061
0.0747 0,0022
5
0.0299 = 0.0017
0.0343 = 0.0008
0.0341 + 0.0009
0.0564 0.0152
0.0961 1 0.0043
0.0732 0,0037
6
0,0226 0.0004
0.0296 0.0007
0,0293 = 0.0007
I
0,0817 0.0274
0.0909 2 0.0094
0.0056 0,0550
0,0824
0.0208 0.0005
0.0289
0.0007
0.0269 + 0.0003
10
1
0.0894 0.0266
0,1050 A 0.0157
0.0102
2
0,1783 + 0,0602
0.2913 + 0.0102
0,2948
0,0210
0.0367 0.0007
0.0398
0.0004
0.0398 0.0004
2
0.0640 0.0072
0.1668 # 0.0078
0.1391
0.0102
3
0,0282 0.0013
0.0358
0.0009
0.0356 : 0.0007
2
0,0499 0,0043
0.1029 0.0033
0.0836 0,0010
2
0.0219 = 0.0004
0.0280 + 0.0008
0.0278 + 0 0.0008
2
0.0483 0.0017
0.0899 # 0.0056
0.0750
0.0027
0.0175 0.0003
0.0267= 0.0005
0.0251= 0.0006
2
0,0530 0,0064
0,0927 0.0050
0.0737
0,0006
2
0.0547 0.0036
0.0984 = 0.0047
0,0784
0.0043
0.0391 0.0107
0.0428
0.0057
0.0425 à 0.0052
0,0219 0.0011
0.0301
0.0010
0.0286 0.0044
3
0.0798 0,0062
0.1881 a: 0.0076
0.1722 0,0086
3
0.0565 + 0.0059
0.1121 + 0.0066
0.0869
0.0026
GINE (L=1)
0.0507 = 0.0014
0.0478 = 0.0003
0.0479 + 0.0004
3
0.0671 0.0156
0.1091 0.0097
0.0920 0,0054
GINE (1==6) 0.0440
0.0009
0.0440
0.0009
0.0451
0.0009
5.3
Computational requirements
35K
Runtime (sec: eguch)
190
We next investigate how increasing k and e change the computational footprint of (ks (FFSS- in prac- tice. Fig. 2 shows that increasing these parameters ex- pectedly increases both memory consumption (in MB in (a)) as well as runtime (in seconds per epoch in (b)). No- tably, since larger and e increases our model's s expres- sivity, we observe that suitable choices for k and al- low us to practically realize these increasingly expressive models commodity hardware. With conservative val- ues of e (e.g. c=1), we are able to consider passing mes- sages between sets of & (e.g. 10) nodes far larger than k- L-yyeet higher order models can achieve (E3).
Memory (M43
30K
35k
80
20k
50
15K
10k
20
Nodes (k)
6
8
10
48 of Nodes (k)
(a) Memory usage
(b) Training time
Figure 2: (S.C)(S)-SEGGNNN' foot- print scales practically with both k and c in memory (a) and running time (b) results on ZINC-12K.
Conclusion
Our work is motivated by the impracticality of higher-order GNN models based on the k-Wl hierar- chy, which make it challenging to study how much expressiveness real-world tasks truly necessitate. To this end, we proposed (k,, c)( )-eetL, a more practical and mrogressively-express with theoretical connections k- WL and drastically lowered complexity. We also designed and im- plemented practical model (k, c) (SSSGNNN) expressiveness of which is gradually increased by larger k and c. Our model achieves strong performance, including several new best results on graph-level tasks like ZINC-12K and expressiveness-rlevann tasks like substructure counting, while being practically trainable on commodity hardware.
hierarchy
10

---

References
[1] R. Abboud, i. i. Ceylan, M. Grohe, and T. Lukasiewicz. The surprising power of graph neural networks with random node initialization. In Proceedings of the Thirtieth International Joint Conference on Artifical Intelligence (IJCAI), 2021.
[2] U. Alon and E. Yahav. On the bottleneck of graph neural networks and its practical im- plications. In International Conference on Learning Representations, 2021. URL https:
[3] V. Arvind, F. Fuhlbrück, Köbler, and O. Verbitsky. On weisfeiler-leman invariance: Sub- graph counts and related graph properties. Journal of Computer and System Sciences, 113: 42-59, 2020.
[4] W. Azizian and M. Lelarge. Expressive power of invariant and equivariant graph neural net- works. arXiv preprint arXiv:2006. 15646, 2020.
[5] L. Babai, Erdos, and S. M. Selkow. Random graph isomorphism. SIAM Journal on Com- puling, 9(3):228-633 1980.
[6] M. Balcilar, P. Héroux, Gauzere, Vasseur, S. Adam, and Honeine. Breaking the limits of message passing graph neural networks. In International Conference on Machine Learning, pages 599-608. PMLR, 2021.
[7] W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanzhez-Gonzalez, Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprini arXiv: 18000012666,,,, 2018.
[8] B. Bevilacqua, F. Frasca, D. Lim, B. Srinivasan, C. Cai, G. Balamurugan, M. M. Bronstein, and H. Maron. Equivariant subgraph aggregation networks. arXiv preprint 1rXiv:2110.02911 2021.
[9] C. Bodnar, Frasca, N. Otter, Y. Wang, Lio, G. F. Montufar, and M. Bronstein. Weisfeiler and lehman go cellular: Cw networks. Advances in Neural Information Processing Systems, 34:2625-2666 2021.
[10] G. Bouritsas, F. Frasca, S. P. Zafeiriou, and M. Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[11] J.-Y. Cai, M. Fürer, and N. Immerman. An optimal lower bound on the number of variables for graph identification. Combinatorica, 12(4):389-410, 1992.
[12] D. Chen, Y. Lin, W. Li, Li, Zhou, and X. Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3438 3445, 2020.
[13] L. Chen, Z. Chen, and J. Bruna. On graph neural networks versus graph-augmented mlps. arXiv preprint arXiv:2010.151/ 16, 2020.
[14) 7. Chen, I.. Li, and J. Bruna. Supervised community detection with line graph neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.nedl forum?id-HH g023A99m.
[15] Z. Chen, L. Chen, S. Villar, and J. Bruna. Can graph neural networks count substructures? Advances in neural information processing systems, 13:10383-10395 2020.
[16] K. Choudhary and B. DeCost. Atomistic line graph neural network for improved materials property predictions. upj Computational Materials, 7(1):188, 2021.
[17] G. Corso, L. Cavalleri, D. Beaini, P. Liò, and P. Veličković. Principal neighbourhood aggrega- tion for graph nets. Advances in Neural Information Processing Systems, volume 33, pages 13260- 13201, 2020.

---

[18] M. H. Dupty, Y. Dong, and W. S. Lee. PF-GNN:
Differentiable particle filterin based approx- imation of universal graph representations. International Conference on Learning Repre- sentations, 2022. URL http:///pppppeeeeeeeeee/oodddddd
[19] P. P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson. Benchmarking graph neural networks, arXiv preprint arXiv:2003.00990 2020.
[20] M. Fey and J. E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop Representation Learning on Graphs and Manifolds, 2019.
[21] F. Frasca, B. Bevilacqua, M. M. Bronstein, and H. Maron. Understanding and extending subgraph gnns by rethinking their symmetries. arXiv preprint arXiv:2206. /40, 2022.
[22] V. Garg, S. Jegelka, and T. Jaakkola. Generalization and representational limits of graph neural networks. In International Conference on Machine Learning, pages 3419-3430. PMLR, 2020.
[23] Geerts. The expressive power of kth-order invariant graph networks. arXiv preprint arXiv:2007. 12035, 2020.
[24] F. Geerts and J. L. Reutter. Expressiveness and approximation properties of graph neu- ral networks. In International Conference on Learning Representations, 2022. URL https:
[25] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR, 2017.
[26] M. Grohe. The logic of graph neural networks. In 2021 36th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS), pages 1-17. IEEE, 2021.
[27] M. Grohe. Multiset wl. Personal communication, 2022.
[28] M. Grohe and M. Otto. Pebble games and linear equations. The Journal of Symbolic Logic, 80 (3):997-844, 2015.
[29] D. M. Hawkins. The problem of overfitting. Journal of chemical information and computer sciences, 44(1):1-12, 2004.
[30] L. Hella. Logical hierarchies in ptime. Information and Computation, 129(1):1-19, 1996.
[31] K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989.
[32] W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec. Strategies for pre- training graph neural networks. In International Conference on Learning Representations, 2020. URL
[33] N. Keriven and G. Peyré. Universal invariant and equivariant graph neural networks. Advances Neural Information Processing Systems, 32, 2019.
[34] J. Kim, T. D. Nguyen, S. Min, S. Cho, M. Lee, H. Lee, and S. Hong. Pure transformers are powerful graph learners. arXiv, abs/2207.02505. 2022. URL https///arxixorr//b//22777 002505.
[35] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. arXiv preprini arXiv: 1609.0090. 2016.
[36] T. N. Kipf and M. Welling, Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
[37] Klicpera, A. Bojehevski. and S. Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv: /8000099 2.18.
[38] A. Loukas. What graph neural networks cannot learn: depth vs width. In Interna- tional Conference on Learning Representations, 2020. URL https://operreviee.nteefeeeeeeee neeformmm ld=B112bp4YwS.
12

---

[39] Y. Ma. X. Liu, T. Zhao, Y. Liu, J. Tang, and N. Shah. A unified view on graph neural net- works as graph signal denoising. Proceedings of the 30th ACM International Conference on Information& & Knowledge Management, pages 1202-1211, 2021.
[40] H. Maron, H. Ben-Hamu, H. Serviansky, and Y. Lipman. Provably powerful graph networks. Advances in neural information processing systems, 32, 2019.
[41] H. Maron, E. Fetaya, N. Segol, and Y. Lipman. On the universality of invariant networks. In International conference machine learning, pages 4363-4371. PMLR, 2019.
[42] C. Morris, K. Kersting, and P. Mutzel. Glocalized weisfeiler-lehman graph kernels: Global- local feature maps of graphs. In 2017 IEEE International Conference on Data Mining (ICDM), pages 327-336. IEEE, 2017.
[43] C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 4602-4609, 2019.
[44] C. Morris, G. Rattan, and P. Mutzel. Weisfeiler and leman go sparse: Towards scalable higher- order graph embeddings. Advances in Neural Information Processing Systems, 33:21824- 21 21840, 2020.
[45] C. Morris, G. Rattan, S. Kiefer, and S. Ravanbakhsh. Speqnets: Sparsity-aware permutation- equivariant graph networks. arXiv preprint arXiv:2203. 2022.
[46] R. Murphy, B. Srinivasan, V. Rao, and B. Ribeiro. Relational pooling for graph representations. In International Conference on Machine Learning, pages 4663-4673, PMLR, 2019.
[47] K. Oono and T. Suzuki. Graph neural networks exponentially lose expressive power for node classification. arXiv preprint arXiv: 10947, 2019.
[48] C. Qian, G. Rattan, F. Geerts, C. Morris, and M. Niepert. Ordered subgraph aggregation networks. arXiv preprint arXiv:2206. / 168, 2022.
[49] R. C. Read and D. G. Corneil. The graph isomorphism disease. Journal of graph theory, 1(4): 339-363, 1977.
[50] E. Rossi, F. Frasca, B. Chamberlain, D. Eynard, M. Bronstein, and F. Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.11199 2020.
[51] A. Sankar, Y. Liu, J. Yu, and N. Shah. Graph neural networks for friend ranking in large-scale social platforms. In Proceedings of the Web Conference 2021, pages 1535-2546 2021.
[52] R. Sato. A survey on the expressive power of graph neural networks. CoRR, abs/2003.04078, 2020. URL http://dblp.um uni- tnieerdeebb/joommass/ntoo/oooooo htt.://oo/cccr20000ommm/aaaa//aa htll#abs-2003-0440
[53] R. Sato, M. Yamada, and H. Kashima. Random features strengthen graph neural networks. In Proceedings of the 202/ SIAM International Conference Data Mining (SDM), pages 333-341, SIAM, 2021.
[54] N. Shervashidze, P. Schweitzer, E. J. Van Leeuwen, K. Mehlhorn, and K. M. Borgwardt. Weisfeiler-lchman graph kernels. Journal of Machine Learning Research, 12(9). 2011.
[55] Topping, Di Giovanni, B. P. Chamberlain, Dong, and M. M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. arXiv preprint arX1v:2111.14522, 2021.
[56] Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. arXiv preprint arXiv:1710.10090: 2017.
[57] B. Weisfeiler. On construction and identification of graphs. In LECTURE NOTES IN MATH- EMATICS. Citeseer, 1976.
[58] WU, F. Sun, W. Zhang, X. Xie, and B. Cui. Graph neural networks in recommender systems: survey, ACM Computing Surveys (CSUR), 2020.
13

---

[59] Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9(2): 513-530, 2018.
[60] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1): 4-24, 2020.
[61] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How Powerful are Graph Neural Networks? In Proceedings of the 7th International Conference on Learning Representations, ICLR '19, pages 1-17, 2019. URL https://ppnneeissseee//oootttt/e
[62] You, R. Ying, and J. Leskovec. Position-awarr graph neural networks. In International Conference Machine Learning, pages 3143.. PMLR, 2019.
[63] J. You, J. Gomes-Selman R. Ying, and J. Leskovec. Identity-aware graph neural networks. arXiv preprint 11tXiv:2101.13200 2021.
[64] M. Zaheer, Kottur, Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and J. Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[65] M. Zhang and P. Li. Nested graph neural networks. Advances in Neural Information Processing Systems, 34:15734-15747, 2021.
[66] M. Zhang, P. Li, Y. Xia, K. Wang, and L. Jin. Revisiting graph neural networks for link prediction. 2020.
[67] I.. Zhao and I.. Akoglu. Pairnorm: Tackling oversmoothing in [gnn]s. In International Confer- ence on Learning Representations, 2020. URL
lee rtw
[68] L. Zhao, W. Jin, L. Akoglu, and N. Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. ICLR, 2022.
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], , [No]. or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes]
Did you include the license to the code and datasets? [No] proprietary.
The code and the data
Did you include license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] In Appendix
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] Will add later into the URL
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "pkXoFgeluTq", "reviewer": "Reviewer_KTkq", "summary": "The paper introduces a family of models, namely (k, c)(≤)-SETGNN, whose expressiveness increases with k and c.\nThe expressiveness is measured in terms of the newly proposed hierarchy (k, c)(≤)-SETWL which is more \"granular\" than the standard k-WL hierarchy.\nThe presentation guides the reader to the derivation of the (k, c)(≤)-SETWL by iteratively refining the k-WL hierarchy.\nThe paper comes with proofs relating the various refinements and experiments that investigate the progressive expressiveness of the proposed architecture for varying k and c.\n", "questions": "- Is it possible to analyse the expressiveness of existing approaches in term on the new hierarchy?\n- Did you use the parameter budget (100K or 500K) as prescribed by ZINC-12K?\n- The linked repository is empty, please update it.", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The empirical evaluation shows increased performance wrt existing methods\n- The proposed architecture's expressiveness can be modulated easily through the k and c parameters.\n\n**Weaknesses**\n- It's unclear whether one can use the new proposed \"ruler\" (the (k, c)(≤)-SETWL hierarchy) to compare existing models and whether new models other than (k, c)(≤)-SETGNN can be built around it. Consequently, the significance appears rather limited.\n\n\nMinor comments:\n- $s$ is not defined in line 135 but introduced much later in line 175\n- Fig. 1 needs a few more iterations. Many edges are disconnected from their endpoints and the labels (1, 5 etc.) of the nodes are not centered.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "a66uBxBPMhq", "reviewer": "Reviewer_g44M", "summary": "This work proposes a more fine-grained graph isomorphism test hierarchy, namely (k,c)(<=)-SetWL, by further considering removing order, removing repetition, and graph sparsity over the original k-WL algorithm. Based on the obtained graph isomorphism test algorithm, it further develops a progressively expressive GNN accordingly. Experiments on synthetic and real datasets are performed to evaluate the effectiveness of the proposed GNN.", "questions": "(1) I am not fully understand the derivation of Theorem 4 ((k,c)(<=)-SetGNN is as expressive as (k,c)(<=)-SetWL). In the given proof, it is mainly based on the results of DeepSet. In other words, MLP with summation is universal so that it can learn the desired injective function. However, this may not prove that Eq. (9) is injective directly, since the input of the outer MLP in Eq. (9) has four parts. It is unclear and non-trivial that how to ensure that this MLP can learn an injective function.\n\n+++minor points\nLine 182, a should be an\n", "limitations": "Not applicable.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+++ Pros\n\n(1) It is technically sound to consider removing order, removing repetition, and graph sparsity over the original k-WL algorithm. The step-by-step derivation is sound and clearly present.\n\n(2) This work also characterizes the expressiveness of the obtained k-WL variants clearly. It is impressive to me to know that k-MultisetWL is as expressive as the original k-WL. This result might have impact to the community.\n\n(3) The presentation of this work is great. Although it has a lot of math, the flow is generally easy to follow.\n\n(4) The empirical results are strong to demonstrate the effectiveness of the proposed GNN.\n\n+++ Cons\n\n(1) Although the presentation is good enough, there are several places which are hard to understand, given that this is a math-heavy work. For example, the result given in Theorem 1 is amazing, it would be great to provide some intuitive explanation of the proof in the main text. This can also help readers to understand why we consider removing orders.\n\n(2) Similarly, the derivation of Theorem 4 is also unclear to me. Please see the following question for details.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YuqR5h-eBWS", "reviewer": "Reviewer_ZZnY", "summary": "The goal of this paper is to develop more expressive GNNs using the k-WL hierarchy. While the one-dimensional Weisfeiler Leman (1-WL) helps to distinguish isomorphic graphs, the use of k-WL hierarchy helps build more expressive GNNs.  The authors propose (k,c) <= SETWL which reduces the complexity of the k-WL hierarchy. Tweaking k and c in this model, helps develop practical and progressively expressive GNNs. \n", "questions": "- What is the basic definition of \"expressiveness\" of a GNN model? Does increased expressiveness of the super-graph imply better generalization performance?\n \n- Is expressiveness related to interpretability of the model? If not, what marks the fundamental difference between them?\n \n- Why is the work consistently using GNNs when the literature is replete with many other kinds of neural networks whose expressiveness can be studied and interpreted?\n \n- The motivation for (k,c) (<=)-SETGNN has not been well discussed.\n \n - Section 5: What is a SOTA expressive GNN? The acronym has not been defined.\n -For ease of readability, consider having a notation table instead of the section 3\n -Table 1,2,3,4 and 5 should simply use two places of decimals for ease of readability\n \n The authors claim \"Such a hierarchy could enable us to gradually build more expressive models which admit improved scaling, and avoid unnecessary leaps in model complexity for tasks which do not require them, guarding against overfitting.\"\n What is the relation between expressivity of a graph and graph isomorphism?\n \n In Table 2, with k=3 or 4, and more than one connected components, the performance on the test set is much worse than on the train set. So for these highly expressive graphs, how is generalization performance demonstrated to be better?\n \n", "limitations": "Complexity of the proposed technique in addition to scalability are clearly limitations which the authors try to solve and are successful to a good extent. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well written with both theoretical and empirical work. \n- An interesting problem on expressiveness of GNNs is explored using set theoretic tools along with message passing algorithms\n\nWeakness\n- The problem needs to be better motivated -- how is expressiveness related to generalization performance? Why use (k,c) (<=)-SETGNN at all?\n- The paper assumes an audience well versed with graph isomorphism, Weisfeiler Leman hierarchy, and other kinds of isomorphism test. A reader unfamiliar with these concepts would wonder why even this is relevant. Motivating with concrete real world examples would perhaps make it easier to see why this is an important problem to solve.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0OJXkw93fxg", "reviewer": "Reviewer_UZY3", "summary": "The paper deals with supervised learning on graph structured data for which the most common framework of GNN is limited in expressivity by 1-WL test in terms of its identifying capacity of graphs. \nHigher order $K$-WL hierarchy of algorithms are well-known to improve the expressivity of GNN models with increasing $K$. However, the complexity of the $K$-WL increases exponentially with increasing $K$, making it infeasible to apply in practice. The authors propose $(k,c)(\\leq)$-SetWL hierarchy in place of WL-hierarchy. a It is a “finer-grained ruler” than WL-hierarchy and mainly depends on sets (size $\\leq k$) instead of tuples of nodes and connected components ($\\leq c$). The authors develop the method moving from tuples in WL to multisets, then to sets and then to sets with connected components and design its neural counterpart $(k,c)(\\leq)$-SETGNN. Experimental results on both synthetic and real datasets show that the method is promising.\n", "questions": "Please address the weakness mentioned above.\n", "limitations": "The authors have discussed some limitations. There is no potential negative societal impact.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n1. The paper addresses an important problem of going beyond 1-WL, however, in such a manner such that expressivity is improved in a progressive way while runtime is still manageable \n1. The idea naturally follows from K-WL models of moving from tuples to multisets, then to sets. The paper rigorously explores this.\n1. Experimental results on graph classification and substructure identification as well as regression Zinc12k are promising.\n1. The paper is well-written with needed rigor. However, the presentation could be improved for better readability.\n\n### Weaknesses/Comments:\n+ The result in Theorem-1 of k-Multiset-WL being equal to K-WL is surprising. I think this should have been discussed with some intuitive figures to show where exactly is the redundancy in K-WL that is being exploited by K-Multiset-WL. I could not follow the proof properly, however it seems to me that with the perm[] function, you would spend same amount of time as K-WL in updating the node colors. Exact number of supernodes in K-Multiset-WL in comparison with K-WL is also missing.\n+ Runtime analysis is not complete. Though it is nice to see the exact ratio of speedup of K-SETWL with respect to WL, it would be much clearer if we see the runtime of WL, Multiset-WL, SET-WL,  in BigO notation. Further, what is the cost with respect to $c$ in (k,c)(\\leq)-SETWL is also not clear. Asymptotic runtime would also help in visualizing where exactly the speedup comes from.\n+ The empirical evaluation should include additional recent expressive GNN models to properly understand the benefits of SET-GNN in comparison with other more expressive models like GraphSNN (Wijesinghe, et al. (2021)), PF-GNN (Dupty et al. (2021)), GNNML3 (Balcilar et al. (2021)) etc. This would also be important since these methods employ different techniques to achieve expressivity and SETGNN can be analysed in perspective.\n+ The paper can be significantly improved in readability if some pictures are added to describe the key points where redundancy is removed from tuple to multisets, then multisets to sets.\n+ Line 238: derivation not in Appendix\n+ Line 183, it should n_u in place of n_n\n\n**Overall**, I find the idea of the paper novel as well as very significant for improving the expressivity of GNNs. Also, the idea is systematically and rigorously developed towards providing a finer scale for measuring expressivity. \n\nHowever, I could not fully grasp the proofs and hence I’m less confident, especially since the results are somewhat surprising. Furthermore, experimental results need to be properly placed with respect to the recent literature.\n\n### References:\n\n+ [1] Wijesinghe, et al. (2021).  A New Perspective on\" How Graph Neural Networks Go Beyond Weisfeiler-Lehman?\". In International Conference on Learning Representations.\n+ [2] Dupty et al. (2021) \"PF-GNN: Differentiable particle filtering based approximation of universal graph representations.\" International Conference on Learning Representations.\n+ [3] Balcilar et al. (2021)  \"Breaking the limits of message passing graph neural networks.\" International Conference on Machine Learning. PMLR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept