Weisfeiler and Leman Go Infinite: Spectral and Combinatorial Pre-Colorings
Authors: Or Feldman
Amit Boyarski
Shai Feldman
Dani Kogan
Avi Mendelson
Chaim Baskin
Venue: Accepted by TMLR ()

Abstract: The limit in the expressivity of Message Passing Graph Neural Networks (MPGNNs) has recently led to the development of end-to-end learning GNN architectures. These advanced GNNs usually generalize existing notions in the GNN architecture or suggest new ones that break the limit of the existing, relatively simple MPGNNs. In this paper, we focus on a different solution, the two-phase approach (or pre-coloring), which enables to use of the same simple MPGNNs while improving their expressivity. We prove that using pre-colorings could strictly increase the expressivity of MPGNNs ad infinitum. We also suggest new pre-coloring based on the spectral decomposition of the graph Laplacian and prove that it strictly improves the expressivity of standard MPGNNs. An extensive evaluation of the proposed method with different MPGNN models on various graph classification and node property prediction datasets consistently outperforms previous pre-coloring strategies. The code to reproduce our experiments is available at \url{https://github.com/TPFI22/Spectral-and-Combinatorial}.

Amit Boyarski Published in Transactions on Machine Learning Research (03/2023)
Weisfeiler and Leman Go Infinite: Spectral and Combinatorial Pre-Colorings
Or Feldman
orfeldmmon@ammpssteennttn
Technion Israel Institute of Technology, Hoifa, Israel
amithooy@ess.echnnion..cc
Technion Israel Instibute of Technology, Hoifa, Israel
Shai Feldman
Technion Israel Institute of Technology, Hoffa, Israel
Dani Kogan
kogan, Sogeen.ddnnEess..ecntioo.ccc
Technion Israel Institute of Technology, Hoifa, Israel
Avi Mendelson
mendloon&tttehhiion..cc
Technion Israel Institute of Technology, Hoifa, Israel
Chaim Baskin
Chaimboskkin&teccriiin..cc
Technion Israel Institute of Technology, Hoifa, Israel
Reviewed on OpenReview: https: // optpr/veeneeee.... net/ forum? YJDQQSAuB6
Abstract
The limit in the exprossivity of Mcssage Passing Graph Neural Networks (MPGNNs) has recently led to the development of end-to-end learning GNN architectures. These advanced GNNs usually generalize existing notions in the GNN architecture or suggest new ones that break the limit of the existing, relatively simple MPGNNs. In this pa- per, we focus on a different solution, the two-phase approach (or pre-coloring), which enables to use of the same simple MPGNNs while improving their expressivity. We prove that using pre-colorings could strictly increase the expressivity of MPGNNs ad in- finitum. We also suggest new pre-coloring based on the spectral decomposition of the graph Laplacian and prove that it strictly improves the expressivity of standard MPGNNs. An extensive evaluation of the proposed method with different. MPGNN models on var- ious graph classification and node property prediction datasets consistently outperforms previous pre-coloring strategics. The code to reproduce our experiments is available at http:/gwww...ao//////////eeaaaaa and- Combinatorial
1 Introduction
The term Graph Neural Networks (GNNs), coined by Bronstein al. 2017)) denotes neural networks designed to learn the hon-Euclddean structure of graph data. According to Zhou et al. (2020) the two main motivations that led to the modern GNN architectures used in CNNs (LCunn et al.) 1998), and graph reprosentation learning (Hamiton et al. 2017b). Message Passing Neural Networks (MPNNs or MPGNNs)(Gilmee et al. 2017) are collections of GNNs with common properties. MPGNNs use first order locality by recursively updating the features of each node from its neighborhoods' aggregated features. Then they create descriptor for the graph by pooling all the node features together. MPGNNs are popular due to their efficiency (Balcilar al. 2021) and their ability learn real world graph-structured data (Xu et al. 2019
the notion of locality and weight sharing

---

Published in Transactions on Machine Learning Research (03/2023)
Figure 1: The pair of molecule graphs colored by the degree coloring (upper) and the spectral coloring (lower).
Despite their success, MPGNNs are bounded in their expressive power (i.e., two different graphs may be encoded to the same descriptor by the same MPGNN). is known that any two graphs that pass the WL test (described in detail in section will be encoded by the same descriptor (Xu al.) 2019 For example, MPGNNs cannot distinguish between the Decalin and Bicyclopentyl molecules graphs (Figure 2) although their graphs non-isomorphic (Sato 2020). Attempts have been made to improve the expressive power of MPGNNs by suggesting new and arguably complicated GNN architectures that are not bounded by the WL test, e.g., by using high order networks (Morris et al. 2019 generalizing graphs to simplicial complexes Bodnar et al. 2021a b), etc.
In this paper we focus on the two phase approach (or pre-coloring), i.e., generating new feat for the nodes before the learning process. This approach is based on the traditional and relatively simple MPGNN architectures and does not require them be changed at all. We present rigorous proof that the two phase approach can be used to improve the expressiveness of the WL test infinite number of times in the WL hierarchy (defined in section 2). In addition, we propose expressive permutation equivariant pre-coloring based on the spectral decomposition of the graph Laplacian that is also efficient to compute, explainable, and generates constant size features with respect to the graph size. Figure 1 demoostt rates the improvement in expressivity that our suggested pre-coloring adds compared to the relatively simple degree coloring, on the Decalin and Bicyclopentyl molccules graphs. Using our proposed pre-coloring, one can casily diffcrentiato between the two graphs, even though standard MPGNNs and the WL test cannol distinguish between them.
Contributions.
We prove that the expressive power of WL can be improved ad infinitum by a sequence of equivariant pre-colorings and that each of the latter can be computed in polynomial time. Thus, the upper bound of the existing MPGNNs can be improved accordingly. This contribution serves as motivation for the two phase approach.
We suggest expressive and informative pre-coloring based the spectral decomposition of the graph Laplacian, and explicitly prove that it improves the expressivity of the WL test, and MPGNNs accordingly.
We perform extensive experiments showing that this simple extension consistently improves the performance of various MPGNNs different benchmarks, compared to previous works.
2

---

Published in Transactions on Machine Learning Research (03/2023)
2 Preliminaries
Graph isomorphism. An undirected graph of size N is a pair g = (V,E) where V = {oI...u} is a set of vertices and E is a set of edges. Each edge is a set of two vertices from V. We say that two graphs 91 = Ey) and G2 = (V2, F2z) are isomorphic if there exists bijection a : V1 - V22 s.t. {er,,) E E = ((()) E Eg. There is no known polynomial time algorithm for determining isomorphism between any arbitrary pair of graphs. Nevertheless, there are some classes of graphs (trees, planar) ween which isomorphism can be determined using the polynomial time algorithm k- WL test (Kiefer et al. 2019: Immerman & Lander 1990). Moreover, Grohe (2012) showed that for almost any class of graphs, the k- WL lest can determine isomorphism.
Graph coloring. Graph coloring is a mapping from a vertex and its graph to a label (color), from a known set of labels. We say that coloring C refines coloring D if for any two graphs O1 and G2, and for any two vertices 01 E V1, 02 1/2 s.l. C(v1) = C(v2), then D(v1) = D(e2). A column vector x can be used to represent. the node colors of coloring C where C(V)) = Ia, for a specific graph. We say that C' is permutation equivariant if for any two graphs with adjacency matrices A and B, and column vectors x, y representing C on A and B respectively s.t. A = =BPP" where P is a permutation matrix, then x = Py. Ideally, we would like to find the following permutation equivariant coloring: for each tn E V1,02 E V/2 = C(e)) +> there exists isomorphism a : G - G2 s.t,, a(v))= U2-
k- WL test. The Weisfeiler-Leman (WL) Weisfeiler & Leman 1968 test of isomorphism is an algorithm for testing necessary but insufficient condition for graph isomorphism. Two graphs that do not pass the test are necessarily non-isomorphic. First, the algorithm assigns to each node the same color using the constant coloring CW (v) = CONST. Then the algorithm continues with iterations. At each iteration i, each node receives its neighbors' colors, and together with its color, it generates a new color for the next iteration, i.e., (w)(v)- = (CWL(v) (v), ([CW)())) E (())))), where '(0)' denotes a multi-set, and (v()) denotes the set of neighbors of v. This process continues until convergence whereupon the colors are collected into a histogram. If the two graphs have different histograms, they failed the test and are called distinguishable. If after the convergence, the two histograms are the same, the graphs did not fail. Having thus passed the test, they are called indistinguishabl. It was proved by Bevilacqua et al. (2022) that C++1 always refines Cirr. The WL test can be exlended to k-tuple coloring instead of verlex (1-tuple) coloring (the full definition of the algorithm can be found in the appendix). This extension is called the k-WL test. It was proved by Cail et al. (1992) that any pair of graphs that are indistinguishable by h + 1-WL are also indistinguishable by k- WL. Morcover, for any h 2 2, there exists a pair of graphs s.t. they are distinguishable by k + 1-WL but indistinguishable by k-WT., i.e., ls + 1-WL. is strielly more expressive than k-WL, for k > 2. This hierarchy of expressiveness is called the WL hierarchy. The diagonal k-WL coloring on the graph vertices is defined to be (()) = Ck, v) where Ck-WL is the coloring after the k-LL. converges. It was proved by Rattan& & Seppelt 2021) that AKk + 1-WL) refines ALk-WL).
Message Passing Graph Neural Networks. MPGNNs are a specific type of GNNs, MPGNNs work in layers; cach layer I has its own Multi Layer Perceptron (MLP); and itcrates over the nodes in the graph. For each node, its neighbors' features are aggregated logether with its own features using some aggregation operation. The result of the aggregation is then used as input to the MLP of the current layer, and the output is the node's 's new features. To create descriptor of the graph, the node features of each layer are aggregated separately, and the results are combined together. In other words, the node features of vertex after I layers are (( MLP (n) (UPDATE(N!- (!-1) (CCREGATE(((D (()1)) (N()))))) where the graph (LAAAIIN((AAGGGGAAEEEEE (((h) x (V())) layers)). It was proved by Xu et al.
descriptor is hg (2099) that the expressive power of MPGNNs is bounded by the expressive power of 1-WL, i.c., for any two graphs Gi and Go s.l. Gi and G2 are indistinguishable by 1-WL, their descriptors created by any MPGNN will be equal. Moreover, it was proved that MPGNNs whose node features are aggregated using summation are strictly more expressive than MPGNNs that use other popular operations such as MAX and MEAN. The MPGNN based on summation called Graph Isomorphism Network (GIN), and it has also been shown to produce state-of-the-art results in addition the theoretical superiority in expressive power.
3

---

Published in Transactions on Machine Learning Research (03/2023)
Graph Laplacian Let G = EV, be undirected graph with adjacency matrix denoted by A. Given function x E [ on the vertices, the Dirichlet energy of the function x on the graph is defined to be
x
Σ (UR))R
(1)
The matrix L is the (combinatorial) graph Laplacian, and given by = D-A, A, where D the degree matriz, i.., diagonal matrix where Dv u) = [v)]] L is symmetric and positive semi-definite and, thercfore, admits spectral decomposition L - Since the sum of each row in L is 0, λ 0 is always eigenvalue of L. The eigenpairs (Ø;: A;) can be thought of as the graph analogue of 'harmonic' and 'frequency'. The graph Laplacian is the discrete generalization of the Laplace-Belle Beltrami operator and hence it has similar properties to it.
The spectrum of the graph Laplacian holds structural information about it. For example, the multiplicity of the zero eigenvalue represents the number of connected components in the graph. Another example is the second eigenvalue (counting multiple eigenvalues separately) that measures the connectivity of the graph (Spielman 2009). We say that pair of graphs are cospectral or cospectral with respect to the Laplacian if their spectra of the Laplacians are equal.
Heat, kernel. The heat equation associated with the Laplacian is given by:
OH, 0 LHH
(2)
where H₃ the heat kernel and is time. The heat kernel describes the process of heat diffusion on the graph through time. Each column of the heat kernel matrix represents different diffusion process the graph. When each column i is initialized with a value of 1 the i-th entry and the rest of the entries are zeros the solution of the heat equation is:
H e
(3)
Then the heat kernel at time can computed by exponentiating the Laplacian eigenspectrum:
H, - - A00000
(4)
where is the i-th eigenvalue of the graph Laplacian and de is its corresponding eigenvector. The element at the index (u, v) can also be computed by:
H.(u, - Ad
(5)
(, v) is the amount of heat transferred from node to node e until time i, from the beginning of a diffusion process where all the nodes had 0 heat and u had exactly 1. When the observed point in time tends to zero, the kernel is affected mostly by the local structures of the graphs. When the obscrved time point is relatively large, the global structure the graphs becomes the dominant structure.
3 Expressive power of 1-WL with pre-colorings
In Section 2 we noted that the expressive power of 1-WT, is limited. In particular, it is strictly limited by the expressive power of k-WL, for any & 2. In this section, we present theoretical support for the two phase approach by showing that the expressive power of the standard 1-WL algorithm can be improved up anywhere in the WL hierarchy when using pre-colorings.
If we pre-color 1-WL with coloring C, we mark the new algorithm 1-C'WL.
Theorem 1. Let Rí, Ra be two colorings s.t. R2 refines Rí and Ra is permulation equivariant. Accordingly, 1-Ra WL is at least as expressive 1-R1 WL.
Proof outline (the full proof be found in the appendix):
4

---

Published in Transactions on Machine Learning Research (03/2023)
1. We show that for any pair of isomorphic graphs, their histogram of 1-R2WL is the same when using the permutation equivariant property of the coloring.
2. We show that two graphs distinguishable by 1-R, WL are also distinguishable byL-R2WL. This is corollary of the color refinement property of the 1-WL iterations.
For Ri and Rg that satisfy Theorem 1 is enough to find a single pair of graphs that are indistinguishable by 1-R, WL but distinguishable by 1-R2WL to prove strictness in expressive power,
Theorem Let 91,92 be any two graphs. Their A(kWWI) hislograms are equal histograms are equal.
their Ck- W
Proof outline (the full proof can be found in the appendix):
We prove the first direction by rumning k- WL for extra k 1 iterations after converged. Following the structure of the k-WL, we show that the coloring of the graph effectively 'folded' onto the diagonal. Since these iterations happen after convergence, they do not change the final coloring, which means that the full graph histogram is encoded onto the diagonal tuples.
2. We prove the second direction by the fact that in the initialization of k-WL, if tuple is colored with the same color diagonal tuple, then it is necessarily diagonal tuple. It, therefore, retains this property throughout the iterations.
Theorem 3. For any k 2 2, 1-A(k+1- MIN++--W)WW WL is strictly more expressive than 1- (k- WL)WL.
A corollary of Theorem and Theorem is that 1-WL can become as powerful as k-WL for any when using A(kWW L) as pre-coloring. This means that the expressive power of MPGNNs, which is provably bounded by the expressive power of 1-WL, can be improved ad infinitum in the WL hierarchy using the right permutation equivariant pre-coloring as pre-process before the MPGNN learning phase.
Not every permutation equivariant coloring C makes 1-CWL strictly more expressive than 1-WL. Example 3.1. If D(u) = [W()),, i.e., the degree coloring, then 1-D WL equal to 1-WL in terms of expressive power.
4 Spectral pre-coloring
Spectral WL We propose expressive pre-coloring based on the spectrum of the graph Laplacian, which can be used to color the nodes instead of the constant coloring of the 1-WL algorithm. We will call this variant the spectral WL algorithm. To calculate the pre-coloring, we first compute m heat kernel matrices for evenly spaced points in lime on the logarithmic scale. Then for each node u, we give the following color: (He, (u, u), Ht.. (u, u)). Finally, we choose a constant amount of quantiles r from the row of u (ignoring the element on the diagonal) and append them ascending order, c.g., ((gft...g') existing color of the node. The spectral coloring with simple setting of m = ...9t ("""..... sufficient to compute the ideal equivariant coloring of the molecules graphs mentioned in |section
to the 1, and no quantiles is
Theorem 4. The spectval pre-coloring is permutaliin equivariant.
Many spectral based methods generate node features that depend on the spectral decomposition represen- tation of the graph Laplacian (Srinivasan Ribeiro 2020 Dwivedi Bresson 2020 Kreuzer et al. 2021) In /section we demonstrate the flaws of this property. By Thoorem 4 we get that the spectral pre-coloring that we suggest does not depend the representation of the spectral decomposition of the Laplacian and, therefore, does not suffer from these flaws.
Theorem 5. Spectral WL is strictly more expressive than 1-WL.
Proof outline (the full proof can be found in the appendix):
We prove that spectral WL is as expressive as 1-WL using Theorem and Theorem 1 and the fact that any coloring refines the constant coloring.
5

---

Published in Transactions on Machine Learning Research (03/2023)
2.
We show a concrete example where spectral WL can distinguish between a pair of non-isomorphic graphs and 1-WL, does not.
Although spectrol WL is strictly more expressive than 1-WL, the exact expressive power of spectral- based approaches is currently unknown and it is stated as an open question by Fürer 2010). However, Fürer (2010) also showed that specific spectral methods based on the adjacency matrix are bounded in their expressive
power by 2-WT.. Rattan & Seppelt (2021) reached the same conclusion for the graph Laplacian and other matrices that are computed by the adjacency matrix.
Spectral features for GNNs. This pre-coloring can be used to create initial node features for MPGNNs as a pre-process before the learning phasc. Instead of applying the coloring, we can appond it to the existing node features of any graph. As hinted by [Theorem 5] in the results of the experiments we will see that it is enough to add a relatively small feature vector, e.g., with 10 entries, achieve great expressivity even for real world graphs with hundreds and thousands of nodes. One can, however, refine the pre-processing by adding more quantiles and time samples. The features that we added to each node have the desirable property of being explainable, and they have the following meaning: For node u, the feature at entry S is the amount of heat left at u at time t, from the beginning of a diffusion process where all the nodes had heat and u had exactly 1. The features at entrics m represent the distribution of the heat diffusion through time on the other nodes.
Scaling for large graphs. For enormous graphs, calculating all the eigenvalues and eigenvectors is im-- practical. For large and small values of t, one can approximate H, via spectral or spatial techniques. For large values of and X, e-t become negligible, hence it is enough to rely on the i smallest eigenvalues. For small values of i, H, can be computed iteratively using explieit/implicit Euler iterations. Alternatively, we can use the model order reduction (MOR) technique to oblain approximale dynamics. The technique has been used successfully for computing isometry invariant descriptors for shape analysis (Bähr et al. 2018 To that end, given the discrctized heat equation
X + Lx
(6)
we use the eigendecomposition L = T
to obtain
s x A x T x=
(7)
Since the dynamics are governed by the smaller eigenvalues of L, we can truncate the eigendecompooitionn to obtain lower-dimensional approdmmtiin of the dynamics. Denoting Wa $ x, with Ds being the first (smallest) & eigenvectors of L, we compute the approximated heat kernel at time by integrating
Wk + Aw =0 0
(8)
up until the relevant time. This simple PDE with a diagonal matrix A that can easily be solved via the unconditionally stable implicit Fuler method.
Despite the fact that we thus obtain only an approximation of the heat kernel, it is important to remember that our task is not to solve the heat equation but rather to provide useful spectral features for graph learning tasks. Tn the related problem non-rigid shape retrieval, descriptors obtained via the approximated dynamics provide better retrieval results compared to descriptors based on the full dynamics (Bähr et al. 2018). An approximation analysis of the suggested scaling technique can be found in the appendix.
5 Experimental study on synthetic benchmarks
To demonstrate the improvement in expressivity that the spectral features add, we built two benchmarks, each of which is based on a single pair of graphs. The first pair of graphs is the Decalin and Bicyclopentyl molecule graphs that have the same 1-WL histogram but their spectrum is different (Sato 2020). The
second pair of graphs is presented in Figure 3 and the graphs are distinguishable by 1-WL. but cospectral
6

---

Published in Transactions on Machine Learning Research (03/2023)
with respect to the Laplacian. For each benchmark, created 1000 examples by adding removing single edge at random from the original graphs and reordering their node indices randomly. For each benchmark, we split all the instances into training and test sets with a ratio of 9:1. The goal of a classifier for the benchmark is: Given a graph from the test sct, identify the original graph from which it was perturbed. We trained GIN (Xu al. 2019), GCN (Kipl Welling 2017], GraphSAGE (Hamillon et al. 2017a) and GAT (Veličković et al. 2018 and their appropriate Spectral Pre-processod (SP) classifiers with the same settings of five message passing layers, a hidden dimension of 64, learning rate of 0.01 and spectral features from 10 points in time using only the maximum quantile, for 100 epochs. We repeated the experiment 100 times and report the average accuracy and standard deviation of each classifier. We used bold to denote significantly high accuracy for each type of MPGNN.
Figure 2: First pair the original graphs. The graphs are 1-WL indistinguishable but not cospectral.
Figure Second pair of the original graphs. The graphs are 1-WL distinguishable but cospectral with respect to the Laplacian.
Table 1: Experimental study results
GNN / Test set 1-WL indistinguishable Cospectral
GIN
64.1#4.0
93.3
SP-GIN
99.0:15.2
93.3+44.5
GCN
51.5#4.00
73.5+16.7
SP-GCN
98.116.5
92.1+5.5
GAT
50.210.9
49.0+0.8
SP-GAT
97.7+11.8
77.6+11.0
GraphSAGE
49.8+-0.9
49.4++0.9
SP SP-GraphSAGE
95.1112.1
91.611..8
From the results in Table I for the 1-WL indistinguishable pair of graphs, the MPGNNs struggle to identify the source of each graph. This is probably because 1-WL cannot differentiate between the sources. The spectral features help them to overcome this issue easily. GIN, which has the most expressive aggregation

---

Published in Transactions on Machine Learning Research (03/2023)
operation among all the MPGNNs, achieves great accuracy the cospectral graphs; the other uniformly colored MPGNNs, however, do not. These results make sense since cospectral graphs have common structural properties. In Figure 4 and Figure 5 we can see the spectral coloring of the cospectral graphs introduced by the spectral pre-prossessing nodes with the same color have the same spectral features. In Figure 4 the pre-processing does not use any quantiles and in Figure 5) the pre-processing uses only the maximum quantile. We can see that not only do both colorings strictly refine the constant coloring, but that the coloring that uses the maximum quantile strictly refines the one that does not.
Figure 4: Cospectral graph coloring based only on the diagonal of the heat kernel.
Figure 5: Cospectral graph coloring based on the diagonal of the heat kernel and the maximum quantile.
In addition to the theoretical justification in Theorem I we conducted synthetic experiment that demon- strates the importance of the permutation equivariant property of pro-colorings. Non permutation equiv- ariant pre-colorings such as random node initialization (Salo et al.) 2021 Abboud et al. 2021 are able to distinguish between two 1-WL indistinguishable graphs, but at the cost of generating different embeddings for isomorphic graphs, which makes them scemingly expressive. Hence, we trained GIN on a datasot contain- ing permuted examples of two 1-WL indistinguishable graphs pre-colored with random node initialization. Given a permuted graph, the purpose of the classifier is identify to which of the two 1-WT. indistinguish- able graphs the permuted graph is isomorphic. We used GIN with hidden dimension of 64, learning rate of 0.01, 50 epochs, and repeated each training-teste testing session 10 times. Since GIN generated different em- beddings for isomorphic graphs, it achieved poor performance of 50% average accuracy. When we switched the random node initialization pre-coloring with our suggested spectral pre-coloring the average accuracy increased to 100%.
6 Evaluation on real benchmarks
We evaluate our pre-processing method on two graph learning tasks: graph classification and node classifica- tion. For each task, we used four types MPGNNs (GIN, GCN, GAT, and GraphSAGE) from the Pytorch Goometric framework (ooy Lenssen 2019 compare the standard use of the network our method. Statistics regarding all of the benchmarks we used can be found in the appendix.
6.1 Graph classification
For this part of the evaluation, we used standard benchmarks and settings that were suggested by Xu ct al. (2019) and followed by many other works (Kolouri et al. 2021 Cai et al. 2021 Maron al.] 2019 Bouritsas al., 2022 Bodnar et al., 2021a b). We compared our method previously suggested techniques of equivariant pre-coloring that generate constant size features. The setting includes eight, graph classification
8

---

Published in Transactions on Machine Learning Research (03/2023)
benchmarks: five social network datasets (COLLAB, IMDB-BINARY, IMDB-MULTI, REDDITBINARY and REDDITTMMUTTTEK); and three molecule datasets (MUTAG, PTC, NCN) (Yanardag & Vishwanathan 2015). The task of the benchmarks here is to achieve the highest average validation accuracy with 10-fold cross-validation. We used GNNs with five layers where in cach layer's MLP a single hidden layer was used. We used concatenation to create the final graph descriptor and a linear layer to creale the final output. We fine-tuned the dropout of the linear layer to be one of (0.0.5). For the molecule datasets, we fine-tuned the hidden dimension of all the MLPs to be one of {16,32), while for the social network benchmarks we consistently used a hidden dimension of size 64. The number of epochs that achieved the best vrossvvaldation accuracy, averaged over the 10 folds, was selected. We examined 700 epochs for each configuration. For fair comparison, we did not tune any of the SP parameters and used constant configuration of 10 time samples on the logarithmicc scale from 4 = 10-2 to 4o0 10², and no quantiles. We report the average validation accuracy and standard deviation over 10 folds. The results for the molecules datasets are located in Table and the results for the social networks datasets are located in Table 3
Table 2: Graph classification results Molecules
Bold accuracy denotes the best coloring for specific type of MPGNN specific benchmark and underlined italic bold accuracy denotes the best coloring for any MPGNN and any coloring on spe- cific benchmark.
Method
MUTAG
PTC
NCI1
GIN
89.3
65.6=6.5
82.1+1.4
LDP-GIN-
87.7+9.2
69.0+4.11
79.6+1166
3SI-GIN2
89.3
69.0+8.11
80.6
SP-GIN (ours)
90.9+19.4
69.3+12.0
81.5+1.7
GCN
79.3
10.3
81.4
LDP-GCN-
88.2+7.77
70.247.8
79.2+1.5
3SI-GCN2
89.211.00
68.7+8.5
80.0
SP-GCN (ours)
93.015.8
69.4+9.33
81.7+1.4
GAT
81.917.4
69.0
81.2
LDP-GAT
88.7+10
68.725.9
78.8+1.8
3SI-GAT
90.4+66.2
68.4+9.0
79.9+2.2
SP-GAT (ours)
91.416.8
69.0+8..7
81.2+11.9
GraphSAGE
84.0+1..1
68.1
81.9+16
LDP-GraphSAGE
90.3+99.9
68.7+10.0
80.1+1.7
3SI-GraphSAGE'
80.8+5..6
68.1+7.4
80.6+1.2
SP-GraphSAGE (ours)
92.4+55.5
70.8+6.0
82.9 82.
1 Local Degree Profile
Cai & Wang
2018),
Subgraph Isomorphism counting (Bouritsas al.
2022
On most of the benchmarks, both molecules and social networks, the SP-MPGNNs performed better than other pre-colored MPGNNs.
6.2 Node classification
We used node classification as an additional method of evaluation and used four node classification bench- marks for this task: three citation network datasets (Cora, CiteSeer, and PubMed) (Yang et al., 2016) and

---

Published in Transactions on Machine Learning Research (03/2023)
Table 3: Graph classification results Social networks Bold accuracy denotes the best coloring for a specific type of MPGNN on specific benchmark and underlined italic bold accuracy denotes the best coloring for any MPGNN and any coloring on a spe- cific benchmark.
Method
COLLAB
IMDB-B
IMDB-M
REDDIT-B
REDDIT-M
GIN
70.4+244
72.744.9
51.343.6
79.64
53.742.1
LDP-GIN
73.8#1.4
72.644.3
49.7
86.142.3
55.7:2.0
3SI-GIN²
73.1
72.14:..6
49.2:14.1
SP-GIN (ours)
76.4
73.4
51.0#4.2
86.9122.0
56.9+1..6
GCN
76.6+22.5
65.444..5
41.5+12.8
90.9+1.6
57.1+119
LDP-GCN
74.8
68.6413.1
44.4+2.6
92.1+0.7
5..5
3SI-GCN2
75.5+1.8
64.6+3..7
42.5+3.4
SP-GCN (ours)
76.7#1.9
73.6+4.5
49.8
91.9++2.5
57.7+1.3
GAT
3.8
52.643.1
36.4+2.5
72.545.9
24.31:6.3
LDP-GAT
7..241.99
69.8
44.6+2.0
90.6+2.00
56.8
3SI-GAT2
73.9+1.9
65.8+6.0
43.5+22.8
SP-GAT (ours)
73.7+225
73.0
50.4#3.6
92.141.5
57.8
GraphSAGE
36.6+7.9
53.4+2.2
35.7+4.1
74.6+3.44
35.7+2.3
LDP-GraphSAGE
76.8+1.2
71+4.3
44.7+
91.4+11.7
57.141.1
3SI 3SI-GraphSAGE
77.6+1.5
68.5
43.8+3.1
SP SP-GraphSAGE (ours)
74.4+22.8
73.2
50.2
91.8+1..8
57.9+1.7
Local Degree Profile CCai & Wang
2018
Subgraph Isomorphism counting (Bouritsas al.
2022
10

---

Published in Transactions on Machine Learning Research (03/2023)
The task of the benchmarks here is to achieve the highest average test accuracy upon 100 random initializations of the GNNs. For the citation networks, only the number of message passing layers, the hidden dimension of the MLPs, and the number of training epochs were fine-tuned, using the validation set. The number of the layers was one of {2. 3, 4}, the hidden dimension was one of {128, 256, 384, 512) and each model was trained for at most 200 epochs. Specifically for PPI, there were two layers, the hidden dimension was 512 and the models were trained for 800 epochs. The spectral pre-process was calibrated exactly as in the synthetic experiment. We repeated each training- testing session 100 times and report in Table 4 the average accuracy and standard deviation of the test set. We used the standard train-test split supplied by Pytorch Geometric.
biochemistry dataset (PPI) ZZiniik & Leskovee 2017
Table 4: Node classification results
Bold denotes significantly high accuracy for each type of MPGNN.
Method
CiteScer
Cora
PubMed
PPI
GIN
71.9+0.6
81.8+0.5
79.6
91.1+0.2
SP-GIN (ours)
71.3++0.6
81.9++.88
78.8++0.7
91.4+0.2
GCN
63.5
78.1
80.4+0.5
88.8
SP-GCN (ours)
72.1
82.3+1.4
80.8+004
89.2
GAT
64.114.55
81.6+1.0
79.9
79.6+0.2
SP-GAT (ours)
7..3+1.5
79.2
80.420.7
80.7+0..3
GraphSAGE
72.8:0.66
82.9
80.2+0066
95.8
3P-GraphSAGE (ours)
72.9
81.942.3
80.8
96.0+0.11
Even though each node in the benchmark contains a feature vector with hundreds of entries, appending it a relatively small number of spectral features usually improved the accuracy of the MPGNNs. Tt can be explained by the fact that the spectral features also contain global information about the graph and the node's position. This information cannot be learned using a small amount of message passing iterations. It is known that deeper MPGNNs (many message passing iterations), however, suffer from two phenomena called Over-squashing (Alon & Yahav 2021 Topping et al. 2022) and Over-smoothing (Wu et al. 2020 Li
et al. 2018 Chen et al. 2020) that are also make it difficult for them to learn global graph information.
6.3 Ablation study
Alongside the benchmark testing conducted, we also wanted to examine the effect of the parameters of our spectral pre-processing method, including the number of points in time to sample, the range of the sample, and the quantiles to use. For this experiment, we chose one benchmark with initial features (NCI1) and one benchmark with none (COLLAB). We again used the four MPGNNs. Each was trained using a hidden dimension of 64 with five message passing layers, for 700 epochs. We report the average and standard 10 different folds of the datasets. The configuration setting is reported
deviation of the lest accuracy "(start of the sampling range in powers of 10, end of the sampling range in powers of 10, number of samples, used )MMM in the quantiles entry denotes the use of the max, min and median quantiles. The full results can be found in the appendix.
For the NCI1 benchmark, the range and the quantiles amount with which we chose to train the MPGNNs achieve the best results when the number of features is limited to 10. For the NCT1 and COLLAB benchmarks, we can see that the performance of SP-MPGNNs can be improved even further by choosing spectral features with more than 10 entries.
11

---

Published in Transactions on Machine Learning Research (03/2023)
7
Related works
This section surveys works related to our research. We split the section into paragraphs by the method used to improve the expressive power of MPGNNs spectral methods and methods for generalizing the message passing scheme.
Use of spectral decomposition in GNNs. Work has been done to improve the expressive power of GNNs, with some studies adopting the spectral based approaches. An example of such an approach is the SAN architecture (Kreuzer et al. (021). First, SAN finds the spectral decomposition of the graph Laplacian using the k-th smallest eigenvalues and their appropriate eigenvectors. Then it encodes them into node features using a transformer with self-attention. Another example is the DGN architecture (Beani et all 2021), which uses the eigendecomposition of the Laplacian to calculate the derivative direction wet the nodes. The directions are then encoded as node features for the GNN. Both SAN and DGN have been shown lo produce state-of-the-arl results on real world benchmarks. These methods, however, allow different descriptors for isomorphic graphs since they are dependent on the eigendeecomposition representation.
Breaking the limits of MPGNNs Several other works tried to break the limit of expressivity of MPGNNs. Some introduce new sophisticated GNN architectures that are based on interesting concepts from various fields of research. These concepts usually generalize the GNNs' message passing scheme, which makes the new architectures more expressive but less efficient. The first attempt. to extend the expressive power of GNNs the K-Dimensional Graph Neural Network (Morris et al.) 2019). These networks gen- eralize MPGNNs in the same way the k-WL lest generalizes the WL lest. Hence, their expressive power is naturally better, upper bounded by the &L instead of the WL. Unfortunately, these networks require tremendous memory and computation time as K inereases, similar to the k-WL test.
The Simplicial Isomorphism Network (SIN) (Bodnar et al. 2021b) another approach that exlends the expressive power of GNNs. This method treats graphs as a general algebraic object called simplicial complex and performs the message passing between every two neighbors in the simplicial complex instcad of the adjacent vertices. Bodnar et al. proved that SIN is strictly more powerful than the WL test and at least as powerful as the 3-WL. test. The Cell Isomorphism Network (CIN) (Bodnar et al., 2021a) is yet. another architecture based on an algebraic object. This object is called a regular cell complex and it generalizes the simplicial complex. In their paper, the rescarchers use the definition of 'cell complex adjacencics' to define the new scheme of message passing. Similar to SIN, it was proved that CIN is strictly more powerful than the WL test and at least as powerful as the 3- test. They also present great results on learning tasks for molecular problems.
8 Discussion
In this work we demonstrated how one can strictly improve the expressive power of the WL test an infinite number of times in the WL hierarchy using the diagonal coloring of the k-WL algorithm, and simultaneously improve the upper bound for MPGNNs, without any change in their architecture. We also proposed spectral pre-processing for MPGNNs that is based on the diagonal and quantiles of the heat kernel matrix. From the results of the graph classification and node classification benchmarks, we conclude that method of pre- processing improves the performance of MPGNNs works. For example, the classification accuracy of GIN, the most expressive MPGNN, on social networks graphs, improved by 3.5%, and when using GAT the improvement. is much more significant and stands at 17%, when comparing it to the uniform pre-coloring. Moreover, our suggested pre-coloring consistently outperformed previously suggested pro-colorings on wide range of real world graph classification benchmarks. We also saw that the spectral pre-coloring is possible research direction for dealing with the Over-squashing problem.
real world graph-structured data, compared to previous
12

---

Published in Transactions on Machine Learning Research (03/2023)
9 Acknowledgement
This rescarch was partially supported by the Technion Hiroshi Fujiwara Cyber Security Research Center, the Israel National Cyber Directorate and the Pazy foundation.
13

---

Published in Transactions on Machine Learning Research (03/2023)
References
Ralph Abboud, ismail Īlka Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. In Proceedings of the Thirtieth International Joint Conference on Arlificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021,
pp. 2112 2218, 2021.
Uri Alon and Fran Yahav. On the bottleneck of graph neural networks and its practical implications. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Martin Bähr, Robert Dachsel, and Michael BreuB. Fast solvers for solving shape matching by time integration. In Annual Workshop of the AAPR (42), pp. 65 72, 2018.
Muhammet Balcilar, Pierre Héroux, Benoit Gauzere, Pascal Vasseur, Sébastien Adam, and Paul Honeine. Breaking the limits of message passing graph neural networks. In International Conference Machine Learning, pp. 599 608. PMLR, 2021.
Dominique Beani, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Di- rectional graph networks. In International Conference on Machine Learning, pp. 748 758. PMLR, 2021.
Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Bal- amurugan, Michael M Bronstcin, and Haggai Maron. Equivariant subgraph aggregation networks. The Tenth International Conference on Learning Representations, ICLR 2022, Virlual Event, April 25-29,
2022, 2022.
Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Lio, Guido Montufar, and Michael Bronstein. Weisfeiler and lehman go cellular: Cw networks. Advances in Neural Information Processing Systems, 34:2625-266 2640, 2021a.
Cristian Bodnar, Fabrizio Frasca, Yuguang Wang, Nina Otter, Guido F Montufar, Pietro Lio, and Michael Bronstein. Weisfeiler and lehman go topological: Message passing simplicial networks. In International Conference on Machine Learning, pp. 1026-1037, PMLR, 2021b.
Giorgos Bouritsas, Fabrizio Frasca, Stefanos P Zafeiriou, and Michael Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2022.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep lcarning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
Chen Cai and Yusu Wang. A simple yet effective baseline for non-attributed graph classification. anXiv preprint (Xi:::811.03555 2018.
Jin-Yi Cai, Martin Fürer, and Neil Immerman. An optimal lower bound on the number of variables for graph identification. Combinatorica, 12(4):389 410,,410000 1992.
Tianle Cai, Shengjie Luo, Keyulu Xu, Di Hc, Tic-yan Liu, and Liwei Wang. Graphnorm: A principled approach to accelerating graph neural network training. In International Conference on Machine Learning, pp. 1204 1215. PMLR, 2021.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3438 3445, 2020.
Vijay Prakash Dwivedì and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprinl ar Xiv:2001 09699, 2020.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint ar Xrxx::1905 02428, 2019.
14

---

Published in Transactions on Machine Learning Research (03/2023)
Martin Fürer. On the power of combinatorial and spectral invariants. Linear algebra and its applications, 432(9))2373-225) -3800 2010.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference machine leorning, pp. 1263 1272. PMLR, 2017.
Martin Grohe. Fixed-point definability and polynomial time on graphs with excluded minors. Journal of the ACM (JACM), 59(5):1-64, 2012.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017a.
William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs Methods and applications. IEEE Dala Eng. Bull., 40(3):52 74, 2017b.
Neil Immerman and Eric Lander. Describing graphs: A first-order approach to graph canonization. In Complexity theory retrospective, pp. 59-81. Springer, 1990.
Sandra Kiefer, Ilia Ponomarcnko, and Pascal Schweitzer. The weisfeiler-leman dimension of planar graphs is al most 3. Journal of the ACM (JACM), 66(6):1 31, 2019.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. 5th International Conference Learning Represenlations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Soheil Kolouri, Navid Naderializadeh, Gustavo K Rohde, and Heiko Hoffmann. Wasserstein embedding for graph learning. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34:
2021.
Stig Larsson and Vidar Thomée. Partial differential equations with numerical methods, volume 45. Springer, 2003.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied docu- ment recognition. Proceedings of the IEFE, 86(11):2278-25)) 2324, 1998.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi- supervised learning. In Secrrt--eccond AAAI conference on artificial intelligence, 2018.
Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. Advances in neural information processing systems, 32, 2019.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4602 4609, 2019.
Gaurav Rattan and Tim Seppelt. Weisfcller-eeman, graph spectra, and random walks. arXio preprint arXiv:2103.0220 2021.
Ryoma Sato. A survey on the expressive power of graph neural networks. arx preprint arXiv:2003.04078, 2020.
Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In Proceedings of the 2021 SIAM International Conference on Dala Mining (SDM), pp. 333-341 SIAM, 2021.
Daniel Spielman. Spectral graph theory. Lecture Notes, Yale Universily, pp- 740 0776, 2009.
15

---

Published in Transactions on Machine Learning Research (03/2023)
Balasubramaniam Srinivasan and Bruno Ribciro. On the equivalence between positional node embeddings and structural graph representations. 8th International Conference on Learning Representations, TCL.R 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M Bron- stein. Understanding over-squashing and bottlenecks graphs via curvature. The Tenth International Conference Leurning Represenlattons, ICLR 2022, Virlual Evend, April 25-29, 2022, 2022.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
Boris Weisfeiler and Andrei Leman. The reduction of graph to canonical form and the algebra which appears therein. NTI, Series, 2(9):12-16, 1968.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4 24 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Pinar Yanardag and S. V. N. Vishwanathan. Deep graph kernels. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pp. 40-48. PMLR, 2016.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020.
Marinka Zitnik and Jure Leskovee. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14):1190 i198, 2017.
16

---

Published in Transactions on Machine Learning Research (03/2023)
A Proofs
In the following proofs we assume the definition of the k-WL as defined by Morris et al. 2019).
Ch-wz is
defined to be equal between any two tuples of vertices from 91 and G2, if and only if the two subgraphs of g1 and G2 comprising all the vertcies in each tuple are isomorphic. We first define the multiset for iteration at index j to be (--wz(P),...x))
( ... D)-1, W, Us+1s...z) w VJ}. Finally, we define the k- WL K coloring at iteration on tuple to be C% Fk-wL(8 (8) = (c.11 (6), k- WL (8))
A.1 Theorem 1 proof
Proof. Let 91 and G2 be two isomorphic graphs where V1 - V2 is the isomorphism. We will prove by induction that after n message passing ilerations of 1-WL initilized with permutation equivariant coloring, R2, the coloring of every pair v E V1 and u E 1/2 s.t. (v)) = u is the same.
Base (n=0): R2 is permutation equivariant and hence by its definition R2(r) = R2(u) for each VE E V1 and u E V22 s.t. d(u)= u. u.
Step: From the induction assumption we know that every two nodes w E Vi and u = (v) E V/2 have the same color after n message passing itcrations of 1-WL. For cach such and u we will look at the coloring after the n + ileration of 1-WL. These are equal to (CT-RawI(o) ([CT-RR R.) E (()))) and (CT-Raw (w), (CCT-R,wLe (x))ar E ((u)))), respec- tively. CP
CT- Rawr,()) (u) and ()-Rwwr(() are equal from the induction assumption. is an isomor- phism and hence x E NNv)) + (xx E N(u) and, therefore, {1CT-Rau (2)]] E [Vv)] and (HCT-n,w ( E [())] are equal. Since a is a bijection, we get that the coloring histogram of I1 and G2 is the same for each n.
Let 91 and G2 be any two graphs and let R1, R2 be two initial colorings for 1-WL s.t. R2 refines Ry. We will prove by induction that for each v E V1 and u E 122, s.t. CT-RawLoo) = Wi--,wL(C (w), u also satisfy CP- CT-R.we w.
CP- R. w. (u) for any number of 1-WL message passing iterations. Therefore, if CT- CT-nww (v) + CT- wi-wwwll (u) then CT-Rww((U) + CT- (w.)r
Base (n=0): For any v E V1 and u E V22 if CP (--RWL(U) = C³ CT-R then C Ci-R,wZ(v) = (-nnwwl()) (u) since R2 refines Ri.
Ch+1 Cz-Roz (u). Their coloring in the n + I iteration is equal to (CT-R (.)r(()) (v), (ICT- (CTT-Rwi(t)lz E ((v)) and (CT- UPT-RWLLC (L),,, (w), ([CT- ((CT-R_L(())) (x) E N())))), respectively. From the induction assumption we find that CT-R (--RWL(() (v) = (T-R,wz(u). In addition, we know that the two multisets in the second part of the tuples are equal, this means that there exists an injective mapping μ : :N(u) + Nv) s.L. (T-nww((1) = CT-nwi (u(x)) and hence by the induction assumption (CTT-awI(<)) e (()))} = (, (u))) and therefore CT=*,, w (v) = CT
Step: Let E Vi and E V/2 be any two vertices s.t. Cm+1 CT-RawZ(1)
R-R W--R.WL(C (w)
1 and G2 are 1-R, WL distinguishable they have different 1-R, WL histograms after some iteration n. Hence, an injective mapping μ : V1 + V/2 s.t. C1-R, = C1- (z-zx for any x e V1, does not exist. From the claim proved by induction, an injective mapping μ : Vi - V22 s.t.
C (1-) = Ci R.W (u(x)) for any x E Vı does not exist. Therefore 91 and G2 have different
1 WL histograms and are distinguishable by 1-R2 WL.
A.2 Theorem 2 proof
1. Proof. Given Gi and G2 s.t. HCK-wwz(n,...nnJll..
E V22), we will prove that ([A(l-W))e)Jl E ViH = ([A(k-W))oo)e E 122}.. From the initialization of k-WL we know that the color of each tuple of the form (v,.,, is equal only to other tuples of this form since they are the only ones that represents a graph with a single vertex. Hence, for any v E V12 and U1 42. E ug- Since any e E Vi is
V1}) = ((C.-w2(eL....a)l
1/2 if C,-wi(U,.., v) = Ck-WL(u, Ux]; then necessarily in = U2 injectively mapped to E V/2 with the same diagonal coloring, we get that [(A(k-WL)(o)jee Vi]}= = [[A(k- W WL)) V2)).
17

---

Published in Transactions on Machine Learning Research (03/2023)
In the second part of the proof we are given 91 and G2 s.t. ((A( E V1)) = ((A(k-WL)(t)) e } and we need to prove that [{Ck- (CC,, (x) ... Tk E Vi}) = (ICE- Th [1 2k E V2}}.
From the definition of k- W.W,, it immediately derived that if for any (I1, Ii, 25+1 Dkk E V V there ex- ists (y1, ye, V1+I..9kk) E V s.t. K-WL(a, 31, ...,,,1111 31, W #++1....4x) = K WLyn, U1, yb Vk)...)kk) then for any (11, (1. E V there exists (V1, VI....11, 9i-1, V/....js)) (I, ,
1/2 V2- s.t. K- = K-W Lyn, U1....1 11 1: Vk). Since we know that this condition holds for i = k + 1, ie. constant tuples, it also accrues for i k, 2..... Since the condition holds for i = 2, ( xx)....,))))) [2)]21 IA [i1} = ([Ck-w2(21,, (0t,,,.,,))))a 11)] ... Ih E }/2) also holds.
A.3 Theorem proof
Proof. From Theorem 1 it immediately is derived that 1-W(L+1-WLWLL is as expressive at least as 1-A(k- WLW To show that. this inequality is striet, we will find a pair of graphs for each K 2 2 s.t. they are indistinguushaabe by 1-A(k-WL)WL but distinguishable by --(k++-WL)WWZ there exists 91 and G2 s.t. they are distinguishable by k+1-WL and indistinguishable by k-WL. From Theorem 2 we know that this pair of graphs is also distinguishable by the 1-A(k+1-W LLL algorithm. We also know from Theorem 2 that the A(k-WT. histograms of the graphs are equal. We will prove that the 1-A(k-WL))M histograms of the graphs are also equal by showing that the message passing iterations of 1-WL does not change the nodes' colors except for the marking/reppeeentaan
For any K > 2 we know
of the colors, ie., the message passing iterations of the 1-WT. does not add any new information to the coloring. After a single iteration of 1-A(k-W) (B--W-WWDDD WL, the new coloring of any vertex is (A()-WD)()), ((A(A-WD(
((e)))), i.c., the new information added to the coloring is the coloring histogram of the neighbors. We will show that this information can be derived from A(k-W W (v) for any v. From the initialization of k-WL, we can find in any color of a tuple (v, v,...u such that u E (Nu)) since their representing graphs are isomorphic and different from the representing graphs for (v, U, v....) where x & N(v). In this way, we can find any color of a tuple (v, u....) s.t. u E (v)). Again from the initialization of k- we can find the color of any (u, u, u....) = L) s.t. u (())...
Since the coloring of 1-A(k-WL)WL does not change in any iteration and because the coloring histograms are equal from the beginning, 1-A(k-WT.jW cannot distinguish between the pair of graphs.
A.4 Example 1 proof
Proof. We will prove that CL-wL = D, i.c., the coloring generated after a single iteration of 1-WL, ini- tialized with constant coloring equals D. For any vertex v, it is colored with the following coloring: e N(()))) (CONST, {(CONST,
(CT-wr(u), ((C--wi(Ce set size is equal to the size of N()).. Hence CL-WL= = D.
where the multi-
A.5 Theorem 4 proof
Proof. Even though the eigenvalues and eigenvectors of the Laplacian are being used to compute the heat kernel, it does not depend their representation (e.g. & -0). This is because the eigenspectrum is being used here to compute the matrix exponentiation in Equation 3 which is well defined. Now we prove that the spectral pre-coloring is indeed permutation equivariant. Let G1 and G2 be two graphs and let A and B be their associated adjacency matrices s.t., A- = PBP" for some permutation matrix P. In order to prove that the spectral pre-coloring is permutation equivariant it is sufficient to prove that 1f,AA = PHBP" for every We know that LA PLgP". Hence, LA and LB have the same set of eigenvalues, and o, an eigenvector of LB then Pb is eigenvector of LA- Then:
H.a DE - Po pT_ =P(S// e Avt 0i )PT = -PHHPTT
(9)
18

---

Published in Transactions on Machine Learning Research (03/2023)
Coloring Histogram
Color
0.1914
0.1929
0.2891
0.291
0.3078
0.3098
Graph
91
2
0
4
0
4
0
G2
0
2
0
0
4
Table 5: Coloring histograms after initialization of Spectral WL
A.6
Theorem 5 proof
Proof. From Theorem I it is immediately derived that Spectral WL is as expressive at least as 1-WL since the spectral pre-coloring is permutation equivariant and any coloring refines the constant coloring. We will show that there exist two graphs that are indistinguishable by 1-WT. but distinguishable by Spectral WL and hence Spectral WL is strictly more expressive than 1-WL.
Let 91 and G2 be the graphs representing the Decalin and Bicyclopentyl molecules Figure 2). It previously shown that 91 and G2 are not isomorphic but cannot be distinguished by the 1-WL test (Sato, 2020) Their Spectral WL histograms using m = 1 with t = 1 and r after the initialization phase are shown in Table 5) Since these histograms are different, Spectral WL will determine that these graphs are not isomorphic.
19

---

Published in Transactions on Machine Learning Research (03/2023)
B
Ablation study - Full results
Table 6: Ablation study on the COLLAB results. Highest accuracies per feature size.
GNN
Feature size
Configuration
Accuracy
SP-GIN
5
(-1,1,5,none)
0.777=0.018
SP-GIN
10
(-1,1,10,none) 0.780
SP-GIN
20
(-1,1,20,maxx 0.788+00021
SP-GCN
5
(-2,2,5,none) 0.774
SP-GCN
10
(-2,2,10,none) 0.778
SP-GCN
20
(-2,2,5,MMM)
0.787+0.021
SP-GAT
5
0.758+0.002
SP-GAT
10
(-1,1,10,nonee)
0.758+0.011
SP-GAT
20
(-2,2,20,none) 0.764+0.016
SP-GraphSAGE
5
(-1,1,5,none) 0.787+0.017
SP-GraphSAGE
10
(-1,1,10,none) 0.795+0.011
SP-GraphSAGE
20
(-3,3,10,maxx)
0.779+10.015
20

---

Published in Transactions on Machine Learning Research (03/2023)
Table 7: Ablation study on the NC11 results. Highest accuracies per feature size.
GNN
Features size
Configuration
Accuracy
SP-GIN
5
(-1,1,5,none)
0.806+0.024
SP-GIN
10
(-2,2,5,max)
0.81240.014
SP-GIN
20
(-2,2,10,maxx
0.81240.014
SP-GCN
5
(-1,1,5,none)
0.801#0.021
SP-GCN
10
(-2,2,5,max) 0.807+0.015
SP-GCN
20
(-2,2,20,nonee)
0.811
SP-GAT
5
(-2,2,5,none)
0.806+0.015
SP-GAT
10
(-2,2,10,nooee)
0.807+0.014
SP-GAT
20
(-3,3,10,maxx) 0.805+0.014
SP-GraphSAGE
5
(-1,1,5,none)
0.82140.010
SP-GraphSAGE
10
(-2,2,5,max)
0.816+0.010
SP-GraphSAGE
20
(-1,1, 10,,max
0.81740.018
21

---

Published in Transactions on Machine Learning Research (03/2023)
Table 8: Ablation study on COLLAB results on GIN
GNN Feature size
Configuration
Accuracy
SP-GIN
5
(-2,2,5,none)
0768+0.015
SP-GIN
5
(-1,1,5,none)
0.7774:10011
SP-GIN
10
(-2,2,5,max)
0.7774+0.014
SP-GIN
10
(-2,2,10,none))
0.7761
SP-GIN
10
(-1,1,10,nnoee)
0.780+10.021
SP-GIN
10
(-3,3,10,nnone)
0.763++0.020
SP-GIN
10
(-1,1,5,max)
0.778++0.022
SP-GIN
20
0.755++0.019
SP-GIN
20
(-2,2,10,max)
0.786+10.020
SP-GIN
20
(-1,1,10,max)
0.788+
SP-GIN
20
(-3,3,10,max)
0.772440.012
SP-GIN
20
(-2,2,20,nonee)
0.780+40.018
SP-GIN
20
(-3,3,20,none)
0.770440.015
22

---

Published in Transactions on Machine Learning Research (03/2023)
Table 9: Ablation study on COLLAB results on GCN
GNN Feature size
Configuration
Accuracy
SP-GCN
5
(-2,2,5,none)
0.7744
SP-GCN
5
(-1,1,5,none)
0.7604+400016
SP-GCN
10
(-2,2,5,max)
0.777
GC-
10
(-2,2,10,none))
0.7781+100119
SP-GCN
10
(-1,1,10,nnoee)
0.762410.022
SP-GCN
10
(-3,3,10,nnone)
0.7774+0.019
SP-GCN
10
(-1,1,5,max)
0.771++0.021
SP-GCN
20
0.787+40.021
SP-GCN
20
(-2,2,10,max)
0.774+10.020
SP-GCN
20
(-1,1,10,max)
0.770+
SP-GCN
20
(-3,3,10,max)
0.784+40.023
SP-GCN
20
(-2,2,20,nonee)
0.770+
SP-GCN
20
(-3,3,20,none)
0.780+40.024
23

---

Published in Transactions on Machine Learning Research (03/2023)
Table 10: Ablation study on COLLAB results on GAT
GNN
Feature size
Configuration
Accuracy
SP-GAT
5
(-2,2,5,none)
0.753++0.022
SP-GAT
5
(-1,1,5,none) 0.7581+40.222
SP-GAT
10
(-2,2,5,max)
0.750
SP-GAT
10
(-2,2,10,none))
0.7534+4.0119
SP-GAT
10
(-1,1,10,nnoee)
0.7581+000115
SP-GAT
10
(-3,3,10,nnone)
0.750++0.020
SP-GAT
10
(-1,1,5,max)
0.758++0.022
SP-GAT
20
0.749++0.022
SP-GAT
20
(-2,2,10,max)
0.763410.022
SP-GAT
20
(-1,1,10,max)
0.759440.018
SP-GAT
20
(-3,3,10,max)
0.758++0.020
SP-GAT
20
(-2,2,20,nnoee)
0.764+10.016
SP-GAT
20
(-3,3,20,none)
0.758+40.022
24

---

Published in Transactions on Machine Learning Research (03/2023)
Table 11: Ablation study on COLLAB results on GraphSAGE
GNN
Feature size
Configuration
Accuracy
SP-GrapESAGE
(-2,2,5,none)
0.7791+40.018
SP-GraphSAGE
5
(-1,1,5,none)
0.787
SP-GraphSAGE
10
(-2,2,5,max)
0.778++10.020
SP-GraphSAGE
10
(=2n2,10,none)
0.7791+40.016
SP-GraphSAGE
10
(-1,1,10,none)
0.795+
SP-GraphSAGE
10
(-3,3,10,none)
0.702
SP-GraphSAGE
10
(-1,1,5,max)
0.792++10:019
SP-GraphSAGE
20
MMM')
0.7761
SP-GraphSAGE
20
(-2,2,10,maxx
0.7741
SP-GraphSAGE
20
(-1,1,10,max)
0.723
SP-GraphSAGE
20
(-3,3,10,max)
0.779++00005
SP-GraphSAGE
20
(-nn),20,nonee)
0.778++00.000
SP-GraphSAGE
20
(-n,))
0.774++0.022
25

---

Published in Transactions on Machine Learning Research (03/2023)
Table 12: Ablation study on NCT1 results on GIN
GNN
Feature size
Configuration
Accuracy
SP-GIN
5
(-2,2,5,none)
0.806+++0022
SP-GIN
5
(-1,1,5,none)
0.8094:100014
SP-GIN
10
(-2,2,5,max)
0.8124:100114
SP-GIN
10
(-2,2,10,none))
0.803
SP-GIN
10
(-1,1,10,nnoee)
0.003
SP-GIN
10
(-3,3,10,noone)
0.808++000116
SP-GIN
10
(-1,1,5,max)
0.804+40.015
SP-GIN
20
0.809+40.018
SP-GIN
20
(-2,2,10,max)
0.812410.014
SP-GIN
20
(-1,1,10,max)
0.8104400016
SP-GIN
20
(-3,3,10,max)
0.807440.015
SP-GIN
20
(-2,2,20,nnnee)
0.808+40.021
SP-GIN
20
(-3,3,20,none)
0.811440.016
26

---

Published in Transactions on Machine Learning Research (03/2023)
Table 13: Ablation study on NCT results on GCN
GNN
Feature size
Configuration
Accuracy
SP-GCN
5
(-2,2,5,none)
0.801++0.019
SP-GCN
5
(-1,1,5,none) 0.801+:00021
SP-GCN
10
(-2,2,5,max)
0.8074+0.015
GC-GCN
10
(-2,2,10,none))
0.804440.010
SP-GCN
10
(-1,1,10,nnoee)
0.80341100016
SP-GCN
10
(-3,3,10,nooee)
0.795
SP-GCN
10
(-1,1,5,max)
0.806+40.021
SP-GCN
20
0.010
SP-GCN
20
(-2,2,10,max)
0.805++0.011
SP-GCN
20
(-1,1,10,max)
0.803
SP-GCN
20
(-3,3,10,max)
0.805+40.017
SP-GCN
20
(-2,2,20,nnoee)
0.811440.014
SP-GCN
20
(-3,3,20,none)
0.810440.016
27

---

Published in Transactions on Machine Learning Research (03/2023)
Table 14: Ablation study on NCT1 results on GAT
GNN
Feature size
Configuration
Accuracy
SP-GAT
5
(-2,2,5,none)
0.806++0.015
SP-GAT
5
(-1,1,5,none) 0.8014400018
SP-GAT
10
(-2,2,5,max)
0.806
SP-GAT
10
(-2,2,10,none))
0.8074+10004
SP-GAT
10
(-1,1,10,nnoee)
0.79011100016
SP-GAT
10
(-3,3,10,noone)
0.806+
SP-GAT
10
(-1,1,5,max)
0.800+10.018
SP-GAT
20
0.791+40.018
SP-GAT
20
(-2,2,10,max)
0.792410.015
SP-GAT
20
(-1,1,10,max)
0.794+.40021
SP-GAT
20
(-3,3,10,max)
0.805440.014
SP-GAT
20
(-2,2,20,nnone)
0.798110.013
SP-GAT
20
(-3,3,20,none)
0.798+40.013
28

---

Published in Transactions on Machine Learning Research (03/2023)
Table 15: Ablation study on NCh results on GraphSAGE
GNN
Feature size
Configuration
Accuracy
SP-GrapESAGE
(-2,2,5,none)
0..11
SP-GraphSAGE
5
(-1,1,5,none)
0.8214:40.011
SP-GraphSAGE
10
(-2,2,5,max)
0.816+++0.010
SP-GraphSAGE
10
(=2n2)10,none)
0.8101+00021
SP-GraphSAGE
10
(-1,1,10,none)
0.8104:40.010
SP-GraphSAGE
10
(-3,3,10,none)
0.0114
SP-GraphSAGE
10
(-1,1,5,max)
0.8104:40.013
SP-GraphSAGE
20
MMM))
0.007
SP-GraphSAGE
20
(-2,2,10,maxx
0.809++400111
SP-GraphSAGE
20
(-1,1,10,max)
0.8174400.01
SP-GraphSAGE
20
(-3,3,10,max)
0..11
SP-GraphSAGE
20
(-n))))0,nonee) 0.815+400.018
SP-GraphSAGE
20
(-n,))
0.816140.013
29

---

Published in Transactions on Machine Learning Research (03/2023)
C Addition experiments settings
Table 16: Graph classification datasets statistics
Name
Graphs
Classes
Avg.
Nodes
Avg. Edges
MUTAG
188
2
17.93
19.79
PTC
336
2
13.97
14.32
PROTEINS
1113
2
39.06
72.82
NCI1
4110
2
29.87
32.30
COLLAB
5000
3
74.49
2457.78
IMDB-BINARY
1000
2
19.77
96.53
IMDB-MULTI 1500
3
13.00
65.94
REDDIT-BINARY
2000
2
429.63
497.75
REDDIT-MULTI
4999
5
508.52
594.87
Table 17: Graph classification datasets statistics
Name
Classes
Avg. Nodes
Avg. Edges
CORA
7
2,708
5,429
CITESEER
6
3,327
4,732
PUBMED
3
19,717
44,338
PPI
121 (multilabel)
56944
818716
We used single GeForce RMX 3090 to run each benchmark. The time took to train and test each GNN on a single benchmark was approximately one to four hours, depends on the benchmark size.
30

---

Published in Transactions on Machine Learning Research (03/2023)
D Approximation analysis of the scaling technique
The scaling technique can be theoretically analyzed using the following theorems:
Theorem 6. The upper bound for the approcimation lional to the inverse of the number of ileralions, i.e., error O inam_tteratonss
of the heat kernel using implicit Euler is propor-
Proof. The step size in each iteration is proportional to the upper bound of the approximation error (Larsson) & Thomée 2003). Since the number of iterations is inversely proportionate to the step size, we get that the approximation error is proportionate to the inverse of the number of iterations.
Theorem 7. The upper bound for the approximation error of the heat kernel using truncated spectrum decays erponentially with respect to the value of the smallest eigenvalue truncated, i.c., error = O (
Proof.
error = 1 dct
Art 6:0! I (n - -k)e-Ants Autie
(10)
We also conducted an empirical approximation analysis on the scaling technique. For that, we used graphs from the REDDIT-BINARY dataset. statistics regarding 0.1 and 1, and computed their approximated heat kernel matrices
are available in the appendix.
We chose two points in time, of the graphs For t used explicit Euler iterations since t is relatively small. For i2 truncated the largest eigenvalues since to is relatively large.
0.00210
Average approxímation error
0.00205
0.00200
0.00195
0.00190
0.00185
0.00180
2.5
7.5
10.0
12.5
15.0
17.5
20.0
Iterations
Figure 6: Average approximation error by the number iterations.
31

---

Published in Transactions on Machine Learning Research (03/2023)
0.035
Average approximation error
0.030
0.025
0.020
0.015
0.010
0.005
0.000
10
20
30
40
50
Eigenvalues
Figure 7: Average approximation error by the number of eigenvalues.
For both t and to the average approximation error decays exponentially with the number of Euler iterations and the number of eigenvalues respectively. For < 0.1 or t > 10 the approximation error will decay faster. From these results, we can understand that the suggested method of approximation able to reduce the computation time of the heat kernel matrix dramatically, and our suggested spectral per-coloring specifically, while still being able approximate accurately.
32


Reviews:
[{"review_id": "hPDpRxUhhJ", "reviewer": "Reviewer_6e9B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: the paper is well written, and provides good empirical evidence that including \"colorings\" as additional features in the input nodes can improve performance of simple GNN architectures across many different tasks.\n\nWeaknesses/general comments:\n\n* The expressiveness results are mostly about comparing to the relatively weak class of 1-WL colorings. It would be good to have a more crisp characterization of what class of graphs spectral colorings can further discriminate, e.g. can they beat k-WL for k > 1?\n\n* If I understand correctly, the \"infinite\"/\"ad infinitum\" mentioned in the title/abstract refers to iteratively refining colorings. While it is clear that this can improve expressivity, can this improvement be quantified somehow? e.g. in Thm 3 you show \"strict\" improvements in expressivity with k-WL colorings, but is there a way to quantify how much the corresponding class of graphs grows?\n\n* it seems that the spectral colorings considered in the paper could be obtained with a few spectral filters with a one-layer spectral GNN (e.g. the diffusion can be implemented with exponential filters). If so, what are benefits of using pre-colorings instead of using such richer spectral architectures? A more detailed comparison would be useful here.\n\n* Recent works have compared the use of richer graph-based input features to using GNNs, see, e.g. this paper: https://arxiv.org/abs/2010.15116 and references therein. How does the conclusions of the present manuscript relate to this line of work?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies expressivity of message passing GNNs with certain pre-colorings of the input nodes, in particular based on spectral features.\nThe authors study improvements in the expressivity of 1-WL approaches (related to basic GNNs) with different pre-colorings, including colorings given by k-WL procedures, as well as spectral features obtained from a few steps of graph diffusions.\nThese pre-colorings are then tested empirically on various architectures and datasets, where it is found that they consistently improve performance across a range of tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above for general questions that could improve the paper.\n\nOther minor comments:\n- many references should be fixed (citet -> citep)\n- the figures and tables also seem quite big, could you make them smaller?\n- p.3, MPGNN paragraph:\n   * not sure what \"descriptor theoretical\" means.\n   * please include references for the results mentioned at the end of the paragraph (\"it was proved\", \"it has been shown\")\n- theorem 3: K -> k", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "i1yFenj7lu", "reviewer": "Reviewer_M4aF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. Graph heat kernel provides a new spectral method for designing message passing.\n\n2. The proposed spectral message passing is implemented by the spectral WL which expressivity is easily measured by the WL algorithm.\n\n3. Proved the equivariance of \\Delta(k-WL) and Ck-WL.\n\n**Weakness:**\n\n1. For large graphs, the computational cost of the spectral method may be high. \n\n2. The error of truncation approximation in spectral WL is not discussed.\n\n3. The expressiveness of \\Delta(k-WL) is discussed in theory. However, it is not used in studying the proposed spectral WL which is the focus of the paper.\n\n4. In experiments, most SOTA results are achieved by the spectral WL-pretrained GNNs, which is not sufficient to tell the superiority of the proposed message passing.\n\n5. There are other spectral GNNs, like Graph Wavelet Neural Network [Wu et al. (ICLR, 2018)], and Framelet convolution [Zheng et al. (ICML, 2021)]. These need to be compared with the spectral WL of the paper.\n\n6. If one uses spectral WL as a message-passing module of GNNs, can the GNNs be very deep? That is, will spectral WL limit oversmoothing?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the WL test for message passing of GNNs. It proposes a new message passing using the spectral information in heat kernel on the graph, and the WL algorithm. It shows in theory the spectral WL is more expressive than 1-WL which is the expressiveness of many GNN methods. A scaling method for reducing the computational cost of spectral message passing is discussed. Experiments for the proposed spectral method for message passing on synthetic and real datasets for both node-level and graph-level classifications are carried on and compared with existing GNN models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The spectral WL which is the focus of the paper appears in Section 4. It can be moved forward. \n\n2. The discussion of \\Detal(k-WL) is not major topic in the paper. It can be moved to the appendix.\n\n3. There are some typos to be corrected. For example, on page 4, in Theorem 4 “are equal ” is a repetition of $\\equiv$.\n\n4. In Table 1, at least 1 decimal digit should be preserved when showing accuracy like Table 2. The variance of the accuracy is big.\n", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qt8PrfIsj1", "reviewer": "Reviewer_fyxi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The colorings defined seem to provide a genuine improvement in GNN performance beyond the constant feature representation.\n- The experiments are comprehensive and examine a large variety of labeling configurations and GNN architectures.\n- The claims are theoretically well-justified. There is a detailed discussion comparing the expressiveness of the WL hierarchy under different initial colorings.\n\nWeaknesses\n\n- It's not clear to me why we need to compute the heat kernel at various times. Is it not just better to encode the eigenvectors themselves? This could benefit from a more intuitive explanation.\n\n- Spectral labelings are compared with constant features, but not with random node features. If GNNs can reap the same benefit in expressivity from random node features, I do not see a reason to use the computationally more expensive spectral colorings.\n\n- The writing could use some improvement (see Requested Changes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper describes methods of improving the expressiveness of MPNNs beyond the 1-WL graph isomorphism test. It does so by preprocessing the graph using a coloring defined by the eigendecomposition of the graph's Laplacian. This is equivariant to vertex permutations, and provably allows the GNN to distinguish between graphs in a finer way. Experiments are performed on synthetic and real-world datasets showing that classification accuracy increases when these labelings are added and for which settings they are most effective.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical points:\n\n- The proof of Theorem 2 is hard to follow; I think it needs more details and clearer writing. Additionally, it only proves the statement for $k = 3$.\n\nMinor points:\n\n- In table 3: What does \"specific\" mean in the numbers?\n\n- The $k$-WL test is introduced but never precisely defined in the text (only the 1-WL test is fully defined). This section in the preliminaries could use a little bit more exposition.\n\n- There are some formatting errors in the document. In the proof of Theorem 1, it says \"wide, labelwidth=!, labelindent=0pt\".\n\n- On page 8, \"provingly\" should be replaced with \"provably\".\n\n- For the base case of the proof of Theorem 1, \"equivarinat\" is a typo.\n\n- Instead of k-WL, it should say $k$-WL; there are several other parts where the correct math font is not used. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In Section 3 it is mentioned that the expressive power of 1-WL can be improved to anywhere in the k-WL hierarchy when using precoloring. If this is to be interpreted as stating that 1-WL with appropriate precoloring can be made as powerful as any k-WL, it would be good to add an explicit statement to this amount, otherwise rephrase for clarify. 

Section 4 presents a theorem stating that spectral precoloring is strictly more expressive than 1-WL. It would be good to mention how the proposed spectral precoloring compares, or how it is known to compare, with diagonal k-WL precoloring and how 1-WL with the proposed precoloring compares with k-WL. 

The article still has a number of typos, including the following. 
* There are repeated references 
* There are typos in the citation formatting. For orientation see section References in https://www.jmlr.org/format/format.html. 
* pg 3 k≥2 vs $k\geq2$; pg 5 k-1 vs $k-1$, k-WL vs $k$-WL
* pg 3, pg 4 \Delta(k - WL) vs \Delta(\text{$k$-WL}) 
* pg 4 t vs $t$ 
* pg 4 i-th vs $i$-th 
* pg 5 space in $1$-$\Delta(...)$
* pg 5, pg 6 misplaced . before reference 
* Tables 1, 2, 3 reference from the text seems to be missing. Font size in the tables seems to be inconsistent. 
Decision: Accept (poster)