Title: Spatio-Spectral Graph Neural Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Neural Networks

long-range interactions
Abstract: Spatial Message Passing Graph Neural Networks (MPGNNs) are widely used for learning on graph-structured data. However, key limitations of *ℓ*-step MPGNNs are that their "receptive field" is typically limited to the *ℓ*-hop neighborhood of a node and that information exchange between distant nodes is limited by over-squashing. Motivated by these limitations, we propose *Spatio-Spectral Graph Neural Networks (S²GNNs)* – a new modeling paradigm for Graph Neural Networks (GNNs) that synergistically combines spatially and spectrally parametrized graph filters. Parameterizing filters partially in the frequency domain enables global yet efficient information propagation. We show that S²GNNs vanquish over-squashing and yield strictly tighter approximation-theoretic error bounds than MPGNNs. Further, rethinking graph convolutions at a fundamental level unlocks new design spaces. For example, S²GNNs allow for free positional encodings that make them strictly more expressive than the 1-Weisfeiler-Leman (WL) test. Moreover, to obtain general-purpose S²GNNs, we propose spectrally parametrized filters for directed graphs. S²GNNs outperform spatial MPGNNs, graph transformers, and graph rewirings, e.g., on the peptide long-range benchmark tasks, and are competitive with state-of-the-art sequence modeling. On a 40 GB GPU, S²GNNs scale to millions of nodes.

Spatio-Spectral Graph Neural Networks

Abstract

Spatial Message Passing Graph Neural Networks (MPGNNs) are widely used for learning on graph-structured data. However, key limitations of ℓ -step MPGNNs are that their 'receptive field' is typically limited to the ℓ -hop neighborhood of a node and that information exchange between distant nodes is limited by oversquashing. Motivated by these limitations, we propose Spatio-Spectral Graph Neural Networks (S 2 GNNs) - a new modeling paradigm for Graph Neural Networks (GNNs) that synergistically combines spatially and spectrally parametrized graph filters. Parameterizing filters partially in the frequency domain enables global yet efficient information propagation. We show that S 2 GNNs vanquish over-squashing and yield strictly tighter approximation-theoretic error bounds than MPGNNs. Further, rethinking graph convolutions at a fundamental level unlocks new design spaces. For example, S 2 GNNs allow for free positional encodings that make them strictly more expressive than the 1-Weisfeiler-Leman (WL) test. Moreover, to obtain general-purpose S 2 GNNs, we propose spectrally parametrized filters for directed graphs. S 2 GNNs outperform spatial MPGNNs, graph transformers, and graph rewirings, e.g., on the peptide long-range benchmark tasks, and are competitive with state-of-the-art sequence modeling. On a 40 GB GPU, S 2 GNNs scale to millions of nodes.

1 Introduction

Spatial Message-Passing Graph Neural Networks (MPGNNs) ushered in various recent breakthroughs. For example, MPGNNs are able to predict the weather with unprecedented precision (Lam et al., 2023), can be composed as a foundation model for a rich set of tasks on knowledge graphs (Galkin et al., 2023), and are a key component in the discovery of millions of AI-generated crystal structures (Merchant et al., 2023). Despite this success, MPGNNs produce nodelevel signals solely considering limited-size neighborhoods , effectively bounding their expressivity. Even with a large number of message-passing steps, MPGNNs are limited in their capability of propagating information to distant nodes due to over-squashing . As evident by the success of global models like transformers (Vaswani et al., 2017), modeling long-range interactions can be pivotal and an important step towards foundation models that understand graphs.

We propose Spatio-Spectral Graph Neural Networks (S 2 GNNs) , a new modeling paradigm for tackling the aforementioned limita-

tions, that synergistically combine message passing with spectral filters , explicitly parametrized in the spectral domain. Spectral filters are virtually ignored by prior work but go beyond stacks of message-passing layers or polynomial parametrizations. Due to message passing's finite number

of propagation steps, it comes with a distance cutoff p cut (# hops, see Fig. 1). Conversely, spectral filters act globally ( p max ), even on a truncated frequency spectrum λ cut . Truncating the frequency spectrum for spectral filters is required for efficiency, yet message passing has access to the entire spectrum (right plots in Fig. 1). The combination of message passing and spectral filters provably leverages the strengths of each parametrization. Utilizing this combination, S 2 GNNs generalize the concept of 'virtual nodes' and distill many important properties of hierarchical message-passing schemes, graph-rewirings, and pooling into a single GNN (see Fig. 3). Outside of GNNs, a similar composition is at the core of some State Space Models (SSM) models (Poli et al., 2023), that deliver transformer-like properties with superior scalability on sequences - as do S 2 GNNs on graphs.

Our analysis of S 2 GNNs (§ 3.1) validates their capability for modeling long-range interactions. We prove in § 3.1.1 that combining spectral and spatial filters alleviates the over-squashing phenomenon (Alon & Yahav, 2020; Di Giovanni et al., 2023a,b), a necessity for effective informationexchange among distant nodes. Our approximation-theoretic analysis goes one step further and proves strictly tighter error bounds in terms of approximation of the target idealized GNN (§ 3.1.2).

Design space of S 2 GNNs (§ 3.2). Except for initial works like (Bruna et al., 2014) and in contrast to spatial MPGNNs, the design decisions for spectral filters are virtually unexplored -and so is their composition. The novel aspects of S 2 GNN's design space include the spectral filter parametrization (§ 3.2.1). We propose the first permutation-

Synergistic composition of spectral filters & spatial message passing

over-squashing

(§ 3.1.1)

Superior approximation capabilities (§ 3.1.2)

§ 3.2

Spatial message passing (see literature)

Spectral filter parametrization (§ 3.2.1)

Spectral-domain MLP (§ 3.2.2)

Spectral filter for directed graphs (§ 3.2.3)

Dual use of partial Eigendecomposition

(

EVD

):

free-of-cost

Positional Encodings (

PE

) (§ 3.2.4)

equivariance-preserving neural network in the spectral domain (§ 3.2.2) and generalize spectral filters to directed graphs (§ 3.2.3). The dual use of the partial eigendecomposition, required for spectral filters, allows us to propose 'free-of-cost' positional encodings (§ 3.2.4), that are permutationequivariant, stable, and increase expressivity strictly beyond the 1-Weisfeiler-Leman (WL) test.

S 2 GNNsare effective and practical. Weempirically verify the shortcomings of MPGNNs and how S 2 GNNs overcome them (§ 4). E.g., we set a new state-of-the-art on peptides-func (Dwivedi et al., 2022) with ≈ 35% fewer parameters, outperforming MPGNNs and graph transformers. Although sequences are just a subdomain of (directed) graphs, we also study how S 2 GNNs compare to specialized sequence models like transformers (Vaswani et al., 2017) or Hyena (Poli et al., 2023). We find that S 2 GNNs are highly competitive even though they operate on a much more general domain (un-/directed graphs). Last, the runtime and space complexity of S 2 GNNs is equivalent to MPGNNs and, with vanilla full-graph training, S 2 GNNs can handle millions of nodes with a 40 GB GPU.

2 Background

We study graphs G ( A , X ) with adjacency matrix A ∈ { 0 , 1 } n × n (or A ∈ R n × n ≥ 0 if weighted), node features X ∈ R n × d and edge count m . A is symmetric for undirected graphs and, thus, has eigendecomposition λ , V = EVD( A ) with eigenvalues λ ∈ R n and eigenvectors V ∈ R n × n : A = V Λ V ⊤ using Λ = diag( λ ) . Instead of A , we decompose the Laplacian L := I -D -1 / 2 AD -1 / 2 , with diagonal degree matrix D = diag( A ⃗ 1) , since its ordered eigenvalues 0 = λ 1 ≤ λ 2 ≤ · · · ≤ λ n ≤ 2 are similar to frequencies (e.g., low eigenvalues relate to low frequencies, see Fig. 4). Likewise, one could use, e.g., L = I -D -1 A or more general variants (Yang et al., 2023); however, we focus our explanations on the most common choice L := I -D -1 / 2 AD -1 / 2 . We choose the matrix of eigenvectors V ∈ R n × n to be orthogonal V V ⊤ = I . We refer to V as the Fourier basis of the graph, with Graph Fourier Transformation (GFT) ˆ X = V ⊤ X and its inverse X = V ˆ X . To provide an overview, Table 5 lists the symbols used in this work.

Spectral graph filters. Many GNNs implement a graph convolution, where node signal X ∈ R n × d is convolved g ∗ G X for every d with a scalar filter g ∈ R n . The graph convolution (Hammond et al., 2011) is defined in the spectral domain as g ∗ G X := V ([ V ⊤ g ] ⊙ [ V ⊤ X ]) , with elementwise product ⊙ and broadcast of V ⊤ g to match shapes. Instead of spatial g , spectral graph filters parametrize ˆ g : [0 , 2] → R explicitly and yield V ⊤ g := ˆ g ( λ ) ∈ R n as a function of the eigenvalues.

Message Passing Graph Neural Networks (MPGNNs) circumvent the EVD via polynomial ˆ g ( λ ) u = ∑ p j =0 γ j λ j u since V [ ∑ p j =0 γ j diag( λ ) j ] V ⊤ X = ∑ p j =0 γ j L j X . In practice, many MPGNNs use p = 1 : H ( l ) = ( γ 0 I + γ 1 L ) H ( l -1) with H (0) = X , and stack 1 ≤ l ≤ ℓ layers interleaved with node-wise transformations and activations σ . Werefer to Balcilar et al. (2021b) for similar interpretations of MPGNNs like GAT (Veliˇ ckovi´ c et al., 2018) or GIN (Xu et al., 2019).

3 Method

S 2 GNNs symbiotically pair spatial Spatial( H ( l -1) ; A ) MPGNNs and Spectral( H ( l -1) ; V , λ ) filters, using a partial eigendecomposition. Even though the spectral filter operates on a truncated eigendecomposition ( spectrally bounded ), it is spatially unbounded . Conversely, spatial MPGNNs are spatially bounded yet spectrally unbounded (see Fig. 1).

A spectrally bounded filter is sensible for modeling global pair-wise interactions, considering its message-passing interpretation of Fig. 3. Conceptually, a spectral filter consists of three steps: ① Gather : The multiplication of the node signal with the eigenvectors v ⊤ u X (GFT) is a weighted and signed aggregation over all nodes; ② Apply : the 'Fourier co-

efficients' are weighted; and ③ Scatter broadcasts the signal v u ˆ X back to the nodes (inverse GFT). The first eigenvector (here for L = D -A ) acts like a 'virtual node' (Gilmer et al., 2017) (see also § E). That is, it calculates the average embedding and then distributes this information, potentially interlayered with neural networks. Importantly, the other eigenvectors effectively allow messages to be passed within or between clusters. As we show for exemplary graphs in Fig. 4, low frequencies/eigenvalues capture coarse structures, while high(er) frequencies/eigenvalues capture details. For example, the second eigenvector in Fig. 4b contrasts the inner with the outer rectangle, while the third eigenspace models both symmetries up/down and left/right. In conclusion, S 2 GNNs augment spatial message-passing with a graph- adaptive hierarchy (spectral filter). Thus, S 2 GNNsdistill many important properties of hierarchical message-passing schemes (Bodnar et al., 2021) , graph-rewirings (Di Giovanni et al., 2023a) , pooling (Lee et al., 2019) etc. See § J.1 for more examples.

S 2 GNN's composition. We study (1) an additive combination for its simpler approximationtheoretic interpretation (§ 3.1.2), or (2) an arbitrary sequence of filters due to its flexibility. In both cases, residual connections may be desirable (see § J.2).

Spectral Filter. The building block that turns a spatial MPGNN into an S 2 GNNis the spectral filter:

with a point-wise transformation f ( l ) θ : R n × d ( l -1) → R n × d ( l ) , a learnable spectral filter ˆ g ( l ) ϑ ( λ ) ∈ R k × d ( l ) parameterized element-wise as ˆ g ( l ) ϑ ( λ ) u,v := ˆ g ( l ) v ( λ u ; ϑ v ) (see § 3.2.1), and truncated V ∈ R n × k , λ ∈ R k . Due to the combination of message passing with spectral filters, S 2 GNNs' hypothesis class goes beyond (finite-order) polynomials of the Laplacian L (or stacks message passing layers), unlocking a larger class of filters. In Algo. 1, we provide pseudo code for S 2 GNNs (Eq. 1).

Truncated spectrum. We omit extra notation for the truncated eigendecompositon EVD( L , k ) since it is equivalent to define ˆ g ( λ j ) = 0 for j > k . However, truncating after the k -th eigenvector requires care with the last eigenspance to maintain permutation equivariance. Due to the

(a)

(b)

̸

ambiguity of eigenvectors in the presence of repeated eigenvalues, we must ensure that we only include eigenspaces in their entirety. That is, we only include { λ j | j ≤ k ∧ λ j = λ k +1 } . Thus, Spectral( H ( l -1) ; EVD( L , k )) is permutation equivariant nonetheless. We defer all proofs to § H.

Theorem 1. Spectral( H ( l -1) ; EVD( L , k )) of Eq. 3 is equivariant to all n × n permutation matrices P ∈ P : Spectral( PH ( l -1) ; EVD( PLP ⊤ , k )) = P Spectral( H ( l -1) ; EVD( L , k )) .

Complementary high-resolution filters. Our Spectral filters are highly discriminative between the frequencies and, e.g., can readily access a single eigenspace. Yet, for efficiency, we limit the spectral filter to a specific frequency band. Due to the combination with message passing, this choice of band does not decide on, say, low-pass behavior; it solely determines where to increase the spectral selectivity. While S 2 GNNs with subsequent guarantees adapt to domain-specific choices for the spectral filter's frequency band, a sensible default is to focus on the low frequencies. The two main reasons for this are (see § J.3 for an extensive list): (1) Low frequencies model the smoothest global signals w.r.t. the graph structure (see Fig. 3 & 4). (2) Under a relative perturbation model (perturbation budget proportional to degree), stability implies C -integral-Lipschitzness ( ∃ C > 0: | λ d ˆ g / dλ | ≤ C ), i.e., the filter can vary strongly around zero but must level out for large λ (see Gama et al. (2020)).

3.1 Theoretical Analysis

We show that how S 2 GNNs alleviate oversquashing in § 3.1.1. Next, § 3.1.2 makes the approximation-theoretic advantages precise.

3.1.1 S 2 GNNs Vanquish Over-Squashing

Alon & Yahav (2020) pointed out that MPGNNs must pass information through bottlenecks that connect different communities using fixedsize embedding vectors. Topping et al. (2022) and Di Giovanni et al. (2023a) formalize this via an L 1 -norm Jacobian sensitivity analysis: ∥ ∂ h ( ℓ ) v / ∂ h (0) u ∥ L 1 models the output's h ( ℓ ) v change if altering input h (0) u .

5

10

15

Source to target distance

0.0

0.5

1.0

GCN

Spec.

Rand.

MPGNNs' Jacobian sensitivity typically decays O (exp ( -r )) with node distance r if the number of walks between the two nodes is small. See § F for results of Di Giovanni et al. (2023a).

S 2 GNNs are not prone to such an exponential sensitivity decay due to their global message scheme. We formalize this in Theorem 2, refer to Fig. 4 for intuition and Fig. 5 for empirical verification. All theoretical guarantees hold if a θ exists such that f θ = I .

Theorem 2. An ℓ -layer S 2 GNN can be parametrized s.t. output h ( ℓ ) v has a uniformly lowerbounded Jacobian sensitivity on a connected graph: ∥ ∂ h ( ℓ ) v / ∂ h (0) u ∥ L 1 ≥ C ϑ d / m with rows h (0) u , h ( ℓ ) v of H (0) , H ( ℓ ) for nodes u , v ∈ G , a parameter-dependent C ϑ , network width d and edge count m .

In contrast to Di Giovanni et al. (2023a), we prove a lower bound for S 2 GNNs, guaranteeing a minimum 'influence' for any u on v . This is true since S 2 GNNs contain a virtual node as a special case with ˆ g ( l ) ϑ ( λ ) = 1 { 0 } , with 1 S denoting the indicator function of a set S (see also § E). However, we find that a virtual node is insufficient for some long-range tasks, including our long-range clustering ( LR-CLUSTER ) of Fig. 10b. Hence, the exponential sensitivity decay of spatial MPGNNs only shows their inadequacy in long-range settings. Proving its absence is not sufficient to quantify long-range modeling capabilities, noting that the lower bound is not tight for S 2 GNNs on many graphs. We close this gap with our subsequent analysis rooted in polynomial approximation theory.

3.1.2 Approximation Theory: Superior Error Bounds Despite Spectral Cutoff

To demonstrate how S 2 GNNs can express a more general hypothesis class than MPGNNs, we study how well an 'idealized' GNN (IGNN) can be approximated. Each IGNN layer l can express convolution operators g ( l ) of any spectral form ˆ g ( l ) : [0 , 2] → R . We approximate IGNNs with S 2 GNNs from Eq. 1, with a spectral filter as in Eq. 3 and a spatial part parametrized by a polynomial. While we assume here that the S 2 GNNspectral filter is bandlimited to and a universal approximator on the interval [0 , λ max ] , the findings generalize to, e.g., a high-pass interval. In the main body, we focus on the key insights for architectures without nonlinear activations. Wang & Zhang (2022) prove that even linear IGNNs can produce any one-dimensional output under certain regularity assumptions on the graph and input signal. Thus, we solely need to consider a single layer. In § H.4, we cover the generic setting including nonlinearities, where multiple layers are helpful.

Locality relates to spectral smoothness. The locality of the true/ideal filter g is related to the smoothness of its Fourier transform ˆ g . For instance, if g is a low-order polynomial of L , it is localized to a few-hop neighborhood, and ˆ g is regularized to vary slowly (Fig. 6a w/o discontinuity). The other extreme is a discontinuous spectral filter ˆ g , such as the entirely non-local virtual node filter, ˆ g = 1 { 0 } (discontinuity in Fig. 6a, details in § E). This viewpoint of

0

1

2

Eigenvalue

λ

0

Filter

ˆ

g

(

λ

)

0

10

20

Node

u

0

Signal

Raw

True

Spa. (

p

=6

)

Spec. (

k

=6

)

S

2

(

p

=3

, k

=3

)

(a) Spectral domain

(b) Spatial domain

spectral smoothness illuminates the limitations of finite-hop message passing from an angle that complements spatial analyses in the over-squashing picture. It informs a lower bound on the error, which shows that spatial message passing, i.e, orderp polynomial graph filters g γ p with p + 1 coefficients γ p ∈ R p +1 , can converge exceedingly slowly - slower than any inverse root (!) of p -to a discontinuous ground truth in the Frobenius-induced operator norm:

Theorem 3. Let ˆ g be a discontinuous spectral filter. For any approximating sequence ( g γ p ) p ∈ N of polynomial filters, an adversarial sequence ( G p ) p ∈ N of input graphs exists such that

̸

Superior S 2 GNNerror bound. Aspatio-spectral convolution wins over a purely spatial filter when the sharpest irregularities of the ground truth ˆ g are within reach of its expressive spectral part. The spatial part, which can 'focus' on learning the remaining, smoother part outside of this window, now needs much fewer hops to give a faithful approximation. We illustrate this principle in Fig. 6 where we approximate an additive combination of an order-three polynomial filter with discontinuous lowpass. Only the S 2 filter is faithfully approximating this filter. Formally, we find:

Theorem 4. Assume ˆ g ∣ ∣ [ λ cut , 2] is r -times continuously differentiable on [ λ cut , 2] , and a bound K r (ˆ g, λ cut ) ≥ 0 such that ∣ ∣ d r dλ r ˆ g ( λ ) ∣ ∣ ≤ K r (ˆ g, λ cut ) ∀ λ ∈ [ λ cut , 2] . An approximating S 2 GNN sequence with parameters ( ϑ ∗ p , γ ∗ p ) p ∈ N exists such that, for arbitrary graph sequences ( G p ) p ∈ N ,

̸

with a scaling constant that depends only on r , not on ˆ g or ( G p ) p ∈ N .

The above bound extends to purely spatial convolutions in terms of K r (ˆ g, 0) if ˆ g is r -times continuously differentiable on the full interval [0 , 2] . The S 2 GNN bound of Theorem 4 is then still strictly tighter if K r (ˆ g, λ cut ) < K r (ˆ g, 0) . In particular, taking the limit K 1 (ˆ g, 0) →∞ towards discontinuity makes the purely spatial upper bound arbitrarily loose, whereas a benign filter might still admit a small K 1 (ˆ g, λ cut ) for some λ cut > 0 . Theorem 3 suggests that this is not an artifact of a loose upper bound but that there is an inherent difficulty in approximating unsmooth filters with polynomials.

We conclude the analysis by instantiating the bounds: assuming ˆ g is C -integral-Lipschitz for stability reasons (see Gama et al. (2020) and the paragraph before § 3.1.1) yields K 1 (ˆ g, λ cut ) = C / λ cut , whereas for the electrostatics example ˆ g σ in § G, we find upper bounds K r (ˆ g σ , λ cut ) = r ! / λ ( r +1) cut . In both cases, the pure spatial bound diverges as smoothness around 0 remains unconstrained.

3.2 Design Space

As shown in Fig. 2, we identify three major, yet unexplored, directions in S 2 GNNs' design space. In § 3.2.1, we discuss how we parametrize the spectral filter. In § 3.2.2, we propose the first neural network for the spectral domain. That is, we allow transformations and non-linearities in the 'Fourier' domain. In § 3.2.3, we are the first to instantiate spectral filters for directed graphs. Additionally,

due to the availability of the partial eigendecomposition, positional encodings may dual use them to improve epxressivity at negligible cost. In § 3.2.4, we propose the first permutation equivariant, stable and efficient positional encodings that provably admit an expressivity beyond 1-WL. § J provides further details and considerations, like some remarks on batching (§ J.7). For the (sub-) design space of spatial message passing (You et al., 2020), we refer to its rich literature.

3.2.1 Parametrizing Spectral Filters

For spectral filter function ˆ g ϑ ( λ ) of Eq. 3, we learn a channelwise linear combination of translated Gaussian basis functions (see "Gaussian smearing" used by Schütt et al. (2017)), as depicted in Fig. 7. This choice (1) may represent any possible ˆ g ϑ ( λ ) with sufficient resolution (assumption in § 3.1.2); (2) avoids overfitting towards numerical inaccuracies of the eigenvalue calculation; (3) limits the discrimination of almost repeated eigenvalues and, in turn, should yield stability (similar to § 3.2.4). Strategies to cope with a variable λ cut and k (e.g., using attention similar to SpecFormer (Bo et al., 2023a)) did usually not yield superior experimental results.

Window. We multiply the learned combinations of Gaussians by an envelope function (we choose a Tukey window) that decays smoothly to zero around cutoff λ cut . This counteracts the so-called 'Gibbs phenomenon' (aka 'ringing'): as visualized for a path graph/sequence of 100 nodes in Fig. 8, trying to approximate a spatially-discontinuous target signal using an ideal low-pass range of frequency components results in an overshooting oscillatory behavior near the spatial discontinuity. Dampening the frequencies near λ cut by a smooth envelope/window function alleviates this behavior. Wenote that the learned filter may, in principle, overrule the windowing at the cost of a higher weight decay penalty. See Algo. 2 for ˆ g ϑ ( λ ) 's algorithmic description.

Depth-wise separable convolution (Sifre, 2014; Howard et al., 2017): Applying different filters for each dimension is computationally convenient for spectral filters. While 'full' convolutions are also possible, we find that such a construction is more prone to over-fitting. In practice, we even use parameter sharing and apply fewer filters than dimensions to counteract over-fitting. We argue that sharing filters among dimensions is similar to the heads in a transformer (Vaswani et al., 2017).

0

50

100

u

0.0

0.5

1.0

Signal

x

Window

w/

w/o

Node

Feature transformations f ( l ) θ . As sketched in Fig. 3 & 4, all nodes participate in the global data transfer. While this global message-passing scheme is graph-adaptive , it does not adjust to the inputs. For adaptivity, we typically consider non-linear feature transformations f ( l -1) θ ( H ( l -1) ) , like gating mechanism f ( l -1) θ ( H ( l -1) ) = H ( l -1) ⊙ σ ′ ( H ( l -1) W ( l ) G + ⃗ 1 b ⊤ )) with element-wise multiplication ⊙ , SiLU function σ ′ , learnable weight W , and bias b . A linear transformation f ( l ) θ ( H ( l -1) ) = H ( l -1) W ( l ) is another interesting case since we may first apply the GFT and then the transformation: ( V ⊤ H ( l -1) ) W ( l ) . Next, we extend this linear transformation to a neural network in the spectral domain by adding multiple transformations and nonlinearities.

3.2.2 Neural Network for the Spectral Domain

Applying a neural network s ζ in the spectral domain is highly desirable due to its negligible computational cost if k ≪ n . Moreover, s ζ allows the spectral filter to become data-dependent and may mix between channels. Data-dependent filtering is one of the properties that is hypothesized to make transformers powerful Fu et al. (2023). We propose the first neural network for the spectral domain of graph filters s ( l ) ζ : R k × d ( l ) → R k × d ( l ) that is designed to preserve permutation equivariance.

We achieve permutation equivariance via sign equivariance s ζ ( S ⊙ X ) = S ⊙ s ζ ( X ) , ∀ S ∈ {-1 , 1 } k × d ( l ) , combined with a permutation equivariance s ζ ( PX ) = P s ζ ( X ) , P ∈ P k , where P k is the set of all k × k permutation matrices. Specifically, we stack linear mappings W s ∈ R d ( l ) × d ( l ) ( without bias ) with a gated nonlinearity ϕ ( ˆ H ) = ˆ H ⊙ σ ( ⃗ 1 [ m ⊤ W a + b ⊤ a ]) with sigmoid σ , column-wise norm m j = ∥ ˆ H : ,j ∥ , and learnable W a ∈ R d ( l ) × d ( l ) as well as b a ∈ R d ( l ) .

3.2.3 Directed Graphs

Directed graphs are an important topic that did not discuss so far. For S 2 GNNs to generalize the capabilities of non-local sequence models like transformers (Vaswani et al., 2017) or SSMs (Poli et al., 2023; Gu & Dao, 2023) it is vital to support direction, e.g., for distinguishing source/beginning and sink/end. However, all discussion before assumed the existence of the eigenvalue decomposition of L . This was the case for symmetric L ; however, for directed graphs, L may be asymmetric.

To guarantee L is diagonalizable with real eigenvalues, we use the Magnetic Laplacian (Forman, 1993; Shubin, 1994; De Verdière, 2013) which is Hermitian and models direction in the complex domain: L q = I -( D -1 / 2 s A s D -1 / 2 s ) ⊙ exp[ i 2 πq ( A -A ⊤ )] with symmetrized adjacency/degrees A s / D s , potential q ∈ [0 , 2 π ] , element-wise exponential exp , and imaginary unit i 2 = -1 . While other parametrizations of a Hermitian matrix are also possible, with A ∈ { 0 , 1 } n × n and appropriate choice of q , L q : { 0 , 1 } n × n → C n × n is injective . In other words, every possible asymmetric A maps to exactly one L q and, thus, this representation is lossless. Moreover, for sufficiently small potential q , the order of eigenvalues is well-behaved (Furutani et al., 2020). In contrast to Koke & Cremers (2024), a Hermitian parametrization of spectral filters does not require a dedicated propagation for forward and backward information flow. For simplicity we choose q < 1 / n max with maximal number of nodes n max (with binary A ). This choice ensures that the first eigenvector suffices to obtain, e.g., the topological sorts of a Directed Acyclic Graph (DAG). Due to the real eigenvalues of a Hermitian matrix, the presented content generalizes with minor adjustments. Most notably, we use a feature transformation f ( l ) θ : R n × d ( l -1) → C n × d ( l ) and map back into the real domain after the spectral convolution. We give more implementation details in § J.6 and provide additional background on directed graphs in § C.

3.2.4 Efficient Yet Stable and Expressive Positional Encodings

The availability of the partial eigendecomposition allows for their dual use for positional encodings at negligible cost. Motivated by this, we propose the first efficient ( O ( km ) ) and (fully) permutation equivariant spectral Positional Encodings PE that provably increase the expressivity strictly beyond the 1-Weisfeiler-Leman (1-WL) test (Xu et al., 2019; Morris et al., 2019). In contrast to the Laplacian encodings of Dwivedi & Bresson (2021), our PE do not require augmenting eigenvectors w.r.t. their sign and maintain permutation equivariance also in the presence of repeated eigenvalues. In comparison to Huang et al. (2024), our PE come with drastically lower computational cost and have no learnable parameters. Due to the absence of learnable parameters, we need to calculate our PE only once.

We construct our k -dimensional positional encodings PE( V , λ ) ∈ R n × k as

with concatenation || and binary adjacency A ∈ { 0 , 1 } n × n . We use a Radial Basis Function (RBF) filter with normalization around each eigenvalue ˆ h j ( λ ) = softmax( ( λ j -λ ) ⊙ ( λ j -λ ) / σ 2 ) with small width σ ∈ R > 0 . This parametrization is not only permutation equivariant but also stable according to the subsequent definition via the Hölder continuity. Note that C depends on the eigengap between 1 / ( λ k +1 -λ k ) at the frequency cutoff (for exact constant C see proof in § H.5).

Definition 1 (Stable PE ) . (Huang et al., 2024) A PE method PE : R n × k × R k → R n × k is called stable, if there exist constants c, C > 0 , such that for any Laplacian L , L ′ , and P ∗ = arg min P ∥ L -PL ′ P ⊤ ∥ F

(a)

(b)

(c)

(d)

(e)

Next to their stability, our PE can discriminate certain degree-regular graphs (e.g., Fig. 9). Since degree-regular graphs cannot be distinguished by 1-WL, our PE makes the equipped GNN (as expressive as 1-WL) strictly more expressive than 1-WL. See § I for continued expressivity analyses.

Theorem 6. S 2 GNNs are strictly more expressive than 1-WL with the PE of Eq. 5.

4 Empirical Results

With state-of-the-art performance on the peptides-func task of the long-range benchmark (Dwivedi et al., 2022), plus strong results on further benchmarks, we demonstrate that S 2 GCN, a GCN paired with spectral filters , is highly capable of modeling long-range interactions (§ 4.1) . We assess S 2 GNNs' long sequence performance (§ 4.2) (mechanistic in-context learning) and show that S 2 GCN, a graph machine learning method, can achieve competitive results to state-of-the-art sequence models, including H3, Hyena, and transformers. We exemplify S 2 GNNs' practicality and competitiveness at scale on large-scale benchmarks (§ 4.3) like TPUGraphs (Phothilimthana et al., 2023), PCQM4Mv2 (Hu et al., 2021), and Open Graph Benchmark (OGB) Products (Hu et al., 2020). Further, in § M.8, we report state-of-the-art performance on the heterophilic arXiv-year (Lim et al., 2021) and,in § M.4, we study combinations of spatial and spectral filters beyond Eq. 1 & 2.

Setup. We pair different MPGNNs with spectral filters and name the composition S 2 <base> . For example, a S 2 GNN with GAT as base will be called S 2 GAT. We typically perform 3 to 10 random reruns and report the mean ± standard deviation. The experiments of § 4.1 require <11 GB (e.g. Nvidia GTX 1080Ti); for the experiments in § 4.2 & 4.3 we use a 40 GB A100. We usually optimize weights with AdamW (Loshchilov & Hutter, 2019) and cosine annealing scheduler (Loshchilov & Hutter, 2017). We use early stopping based on the validation loss/score. See § M for more details and https://www.cs.cit.tum.de/daml/s2gnn for code as well as supplementary material.

4.1 Long-Range Interactions

Finding (I): S 2 GCN outperforms state-of-the-art graph transformers, MPGNNs, and graph rewirings on the peptides-func long-range benchmarks (Dwivedi et al., 2022) by a substantial margin. Simultaneously, we remain approximately 35% below the 500k parameter threshold and. On peptides-struct we are only outperformed by NBA-GIN (Park et al., 2023). We extend the best configuration for a GCN of Tönshoff et al. (2023) (see GCN in Table 1), lower the number of message passing steps from six to three, and interleave spatial and spectral filters (Eq. 2) with λ cut = 0 . 7 .

Dataset contribution: Clustering , given a single seed node per cluster, measures the ability (1) to spread information within the cluster and (2) to discriminate between the clusters. We complement the semisupervised task CLUSTER from Dwivedi et al. (2023) with (our) LR-CLUSTER dataset, a scaled-up version with longrange interactions (1). We closely follow Dwivedi et al. (2023), but instead of using graphs sampled from Stochastic Block Models (SBMs), we sample coordinates from a Gaussian Mixture Model (GMM) and then connect nearby nodes. CLUSTER has 117 nodes on average, while ours has 896. LR-CLUSTER has an average diameter of ≈ 33 and often contain hub nodes

that cause over-squashing. For full details on the dataset construction, see § M.6.

Dataset contribution: Distance regression is a task with long-range interactions used in prior work (Geisler et al., 2023; Lim et al., 2023). Here, the regression targets are the shortest path distances to the only root node (in-degree 0). We generate random trees/DAGs with ≈ 750 # of nodes on average (details are in § M.7). The target distances often exceed 30 hops. We evaluate on similarly sized graphs as in the training data, i.e., in-distribution ( ID ) samples, and out-of-distribution ( OOD ) samples that consist of slightly larger graphs. Details on the dataset construction are in § M.7.

Finding (II): spatial MPGNNs are less effective as S 2 GNNs , for long-range interactions. This is evident for peptides Table 1, clustering Fig. 10, distance regression Fig. 11, and over-squashing Fig. 12. Specifically, if the task requires long-range interactions beyond the receptive field of MPGNNs, they return crude estimates. E.g., in Fig. 11, the MPGNN predicts (approx.) constantly 20 for all distances beyond its receptive field - roughly the mean in the training data. Moreover,

0

1

2

4

6

8

10

# eigenvectors

k

0.6

0.8

MPGNN

Virtual node

2

3

4

5

6

7

8

9 10

# spatial MP layers

0.25

0.50

0.75

0

20

40

Ground truth distance

0

20

40

Prediction

DirGCN

S 2

DirGCN

0

50

100

n

0.0

0.5

1.0

GatedGCN

S

2

GatedGCN

Number of nodes

A

V

diag(ˆ

gϑ

(

λ

)

:

,

1)

V

/latticetop

V

diag(ˆ

gϑ

(

λ

)

:

,

2)

V

/latticetop

V

diag(ˆ

gϑ

(

λ

)

:

,

3)

V

/latticetop

V

diag(ˆ

gϑ

(

λ

)

:

,

4)

V

/latticetop

S 2 GNNs may converge faster (see Fig. 25 in § M.6.2) and are more parameter-efficient, as we show on PCQM4Mv2 (Hu et al., 2021) in § M.9.

Finding (III): virtual nodes are insufficient. We frequently find that including more than a single eigenvector ( k > 1 ) yields substantial gains. We make this explicit in Fig. 10a, where we append a single spectral layer and sweep over the number of eigenvectors k . We complement these findings with an ablation for the frequency cutoff λ cut on peptides-func in § M.5.

Finding (IV): our Positional Encodings PE consistently help , when concatenated to the node features. While this finding is true throughout our evaluation, the differences are more pronounced in certain situations. For example, on LR-CLUSTER in Fig. 10, the PE help with spectral filter and a small k or without spectral filter and many message passing steps.

Finding (V): spectral filters align with clusters , as we illustrate in Fig. 13 for four arbitrary spectral filters learned on LR-CLUSTER . We observe that (a) the spectral filters reflect the true clustering structure, (b) some filters are smooth while others contain details, and (c) they model coarser or finer cluster structures (e.g., first vs. third filter).

4.2 Sequence Modelling: Mechanistic In-Context Learning

Following the evaluation of Hyena (Poli et al., 2023) and H3 (Fu et al., 2023), we benchmark S 2 GCN with sequence models on the associative recall in-context learning task, stemming from mechanistic interpretability (Elhage et al., 2021; Power et al., 2022; Zhang et al., 2023; Olsson et al., 2022). In associative recall, the model is asked to retrieve the value for a key given in a sequence. For example, in the sequence a,0,e,b,z,9,h,2,=>,z , the target is the value for key z , which is 9 since it follows z in its prior occurrences. We create a sequence/path graph with a node for each 'token' (separated by ',' in the example above) and label the target node with its value. We assess the performance of S 2 GCN on graphs that vary in size by almost two orders of magnitude and follow Poli et al. (2023) with a vocabulary of 30 tokens. Moreover, we finetune our S 2 GCN on up to 30k nodes.

0

500

1000

n

0.0

0.5

1.0

Dir.

Undir.

Number of nodes

Finding (VI): our spectral filter for directed are effective and may improve generalization, as we find in Fig. 14 (and Table 13 of § M.7).

Finding (VII): S 2 GCN a state-of-the-art sequence model , as it performs on par with Hyena and, here, outperforms transformers (Table 2).

4.3 Large-Scale Benchmarks

Finding (VIII): S 2 GNNs is practical and scalable. We demonstrate this on the OGB Products graph (2.5 mio. nodes, Table 3) and the (directed) 10 million graphs dataset TPUGraphs (average number of nodes ≈ 10,000, Table 4). In both cases, we find full-graph training (without segment training (Cao et al., 2023)) using 3 (Dir-) GCN layers interlayered with spectral filters, a reasonable configuration on a 40 GB A100. However, for OGB Products, we find that batching is superior, presumably because the training nodes are drawn from a 'small' region of the graph (see § K).

The cost of partial EVD for each dataset (excluding TPUGraphs and distance regression) is between 1 to 30 minutes on CPUs. We report the detailed costs of EVD and experiments in § M.3.

5 Related Work

Combining spatial and spectral filters has recently attracted attention outside of the graph domain in models like Hyena (Poli et al., 2023), Spectral State Space Models (Agarwal et al., 2024), etc. with different flavors of parametrizing the global/FFT convolution. Nevertheless, the properties of spatial and spectral filter parametrization (e.g., local vs. global) are well-established in classical signal processing. A combination of spectral and spatial filters was applied to (periodic) molecular point clouds (Kosmala et al., 2023). For GNNs, Stachenfeld et al. (2020) compose a spatial and spectral message passing but do not handle the ambiguity of the eigendecomposition and, thus, do not maintain permutation equivariance. Moreover, Beaini et al. (2021) use the EVD for localized anisotropic graph filters; Liao et al. (2019) propose an approach that combines spatial and spectral convolution via the Lanczos algorithm; and Huang et al. (2022) augment message passing with power iterations. Behrouz & Hashemi (2024) apply a Mamba-like state space model to graphs via arbitrarily ordering the nodes and, thus, sacrifice permutation equivariance.

Long-range interactions on graphs. Works that model long-range interactions can be categorized into: (a) MPGNNs on rewired graphs (Gasteiger et al., 2019a,b; Gutteridge et al., 2023); (b) higherorder GNNs (Fey et al., 2020; Wollschläger et al., 2024) that, e.g., may pass information to distant nodes through hierarchical message passing schemes; and (c) message passing adaptations to facilitate long-range interactions. For example, Park et al. (2023) propose 'non-backtracking' message passing, Errica et al. (2024) adaptively choose the numbers of message passing steps, and Ding et al. (2024) use linear RNNs to aggregate over each node's neighborhoods. While approaches (a-c) can increase the receptive field of GNNs, they are typically still spatially bounded. In contrast, (d) alternative architectures, like graph transformers (Ma et al., 2023; Dwivedi & Bresson, 2021; Kreuzer et al., 2021; Rampášek et al., 2022; Geisler et al., 2023; Deng et al., 2024) with global attention, may model all possible n × n interactions. We provide notes on the limitations of graph transformers with absolute positional encodings in § D, which highlights the importance of capturing the relative relationships between nodes, as S 2 GNNs do. Moreover, in a recent/contemporary non-attention model for all pair-wise interactions, Batatia et al. (2024) use a resolvent parametrization of matrix functions relying on the LDL factorization of a matrix, but do not characterize their approximation-theoretic properties, over-squashing, expressivity on graphs, nor how to deal with directed graphs.

In § B, we discuss additional related work w.r.t. expressivity and directed graphs.

6 Discussion

We propose S 2 GNNs, adept at efficiently modeling complex long-range interactions via the synergistic composition of spatially and spectrally parametrized filters (§ 3). We show that S 2 GNNs share many properties with graph rewirings, pooling, and hierarchical message passing schemes (Fig. 3 &4). S 2 GNNs outperform the aforementioned techniques with a substantial margin on the peptides long-range benchmark (§ 4.1), and we show that S 2 GNNs are also strong sequence models, performing on par or outperforming state-of-the-art like Hyena or H3 in our evaluation (§ 4.2). Even though we find global graph models, like S 2 GNNs, more prone to overfitting (see § K/L for further limitations/impact), moving to global models aligns with the trend for other deep learning domains.

Acknowledgments and Disclosure of Funding

We want to express our gratitude to Nicholas Gao for his feedback and the discussions about modeling choices. Moreover, we thank Leo Schwinn and Tim Beyer for their helpful and on-point feedback and suggestions.

This research was supported by the Helmholtz Association under the joint research school 'Munich School for Data Science - MUDS', as well as by the Munich Data Science Institute (MDSI) via the Linde/MDSI Doctoral Fellowship program and the MDSI Seed Fund.

References

Naman Agarwal, Daniel Suo, Xinyi Chen, and Elad Hazan. Spectral State Space Models, arXiv, 2024.

Uri Alon and Eran Yahav. On the Bottleneck of Graph Neural Networks and its Practical Implications. In International Conference on Learning Representations, ICLR , 2020.

Muhammet Balcilar, Pierre Héroux, Benoit Gaüzère, Pascal Vasseur, Sébastien Adam, and Paul Honeine. Breaking the Limits of Message Passing Graph Neural Networks. In International Conference on Machine Learning, ICML , 2021a.

Muhammet Balcilar, Guillaume Renton, and Pierre Heroux. Analyzing the Expressive Power of Graph Neural Networks in a Spectral Perspective. In International Conference on Learning Representations, ICLR , 2021b.

Ilyes Batatia, Lars L Schaaf, Gabor Csanyi, Christoph Ortner, and Felix A Faber. Equivariant Matrix Function Neural Networks. In International Conference on Learning Representations, ICLR , 2024.

Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks, arXiv, 2018.

Dominique Beaini, Saro Passaro, Vincent Létourneau, William L. Hamilton, Gabriele Corso, and Pietro Liò. Directional Graph Networks. In International Conference on Machine Learning, ICML , 2021.

Ali Behrouz and Farnoosh Hashemi. Graph Mamba: Towards Learning on Graphs with State Space Models, arXiv, 2024.

Deyu Bo, Chuan Shi, Lele Wang, and Renjie Liao. Specformer: Spectral Graph Neural Networks Meet Transformers. In International Conference on Learning Representations, ICLR , 2023a.

Deyu Bo, Xiao Wang, Yang Liu, Yuan Fang, Yawen Li, and Chuan Shi. A Survey on Spectral Graph Neural Networks, arXiv, 2023b.

Cristian Bodnar, C˘ at˘ alina Cangea, and Pietro Liò. Deep Graph Mapper: Seeing Graphs Through the Neural Lens. Frontiers in Big Data , 4:38, 2021.

Xavier Bresson and Thomas Laurent. Residual Gated Graph ConvNets, arXiv, 2018.

Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges . arXiv, 2021.

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral Networks and Locally Connected Networks on Graphs, arXiv, 2014.

Chen Cai, Truong Son Hy, Rose Yu, and Yusu Wang. On the Connection Between MPNN and Graph Transformer. In International Conference on Machine Learning, ICML . arXiv, 2023.

Shaofei Cai, Liang Li, Xinzhe Han, Jiebo Luo, Zheng-Jun Zha, and Qingming Huang. Automatic Relation-Aware Graph Network Proliferation. In Conference on Computer Vision and Pattern Recognition, CVPR , 2022.

Kaidi Cao, Phitchaya Mangpo Phothilimthana, Sami Abu-El-Haija, Dustin Zelle, Yanqi Zhou, Charith Mendis, Jure Leskovec, and Bryan Perozzi. Learning Large Graph Property Prediction via Graph Segment Training. In Neural Information Processing Systems, NeruIPS . arXiv, 2023.

Zhe Chen, Hao Tan, Tao Wang, Tianrun Shen, Tong Lu, Qiuying Peng, Cheng Cheng, and Yue Qi. Graph Propagation Transformer for Graph Representation Learning. In International Joint Conference on Artificial Intelligence, IJCAI , 2023.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive Universal Generalized PageRank Graph Neural Network. In International Conference on Learning Representations, {ICLR} , 2021.

Yun Young Choi, Sun Woo Park, Minho Lee, and Youngho Woo. Topology-Informed Graph Transformer, arXiv, 2024.

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Neural Information Processing Systems, NeruIPS . arXiv, 2022.

Yves Colin De Verdière. Magnetic interpretation of the nodal defect on graphs. Analysis & PDE , 6 (5):1235-1242, 2013.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In Neural Information Processing Systems, NeurIPS , 2017.

Chenhui Deng, Zichao Yue, and Zhiru Zhang. Polynormer: Polynomial-Expressive Graph Transformer in Linear Time. In International Conference on Learning Representations, ICLR , 2024.

Francesco Di Giovanni, Lorenzo Giusti, Federico Barbero, Giulia Luise, Pietro Liò, and Michael Bronstein. On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topology. In International Conference on Machine Learning, ICML . arXiv, 2023a.

Francesco Di Giovanni, T. Konstantin Rusch, Michael M. Bronstein, Andreea Deac, Marc Lackenby, Siddhartha Mishra, and Petar Veliˇ ckovi´ c. How does over-squashing affect the power of GNNs?, arXiv, 2023b.

Yuhui Ding, Antonio Orvieto, Bobby He, and Thomas Hofmann. Recurrent Distance Filtering for Graph Representation Learning. In International Conference on Machine Learning, ICML , 2024.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, ICLR , 2021.

Vijay Prakash Dwivedi and Xavier Bresson. A Generalization of Transformer Networks to Graphs. Deep Learning on Graphs at AAAI Conference on Artificial Intelligence , 2021.

Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long Range Graph Benchmark. In Neural Information Processing Systems, NeruIPS . arXiv, 2022.

Vijay Prakash Dwivedi, Chaitanya K. Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking Graph Neural Networks. Journal of Machine Learning Research, JMLR , 2023.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, and et al. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021.

Federico Errica, Henrik Christiansen, Viktor Zaverkin, Takashi Maruyama, Mathias Niepert, and Francesco Alesiani. Adaptive Message Passing: A General Framework to Mitigate Oversmoothing, Oversquashing, and Underreaching, arXiv, 2024.

Matthias Fey and Jan Eric Lenssen. Fast Graph Representation Learning with PyTorch Geometric, arXiv, 2019.

Matthias Fey, Jan-Gin Yuen, and Frank Weichert. Hierarchical Inter-Message Passing for Learning on Molecular Graphs. In Graph Representation Learning and Beyond (GRL+) Workhop at ICML 2020 . arXiv, 2020.

Robin Forman. Determinants of Laplacians on graphs. Topology , 32(1):35-46, 1993.

Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry Hungry Hippos: Towards Language Modeling with State Space Models. In International Conference on Learning Representations, ICLR , 2023.

Satoshi Furutani, Toshiki Shibahara, Mitsuaki Akiyama, Kunio Hato, and Masaki Aida. Graph Signal Processing for Directed Graphs Based on the Hermitian Laplacian. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD , 2020.

Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. Towards Foundation Models for Knowledge Graph Reasoning, arXiv, 2023.

Fernando Gama, Joan Bruna, and Alejandro Ribeiro. Stability Properties of Graph Neural Networks. IEEE Transactions on Signal Processing , 68:5680-5695, 2020.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized PageRank. International Conference on Learning Representations, ICLR , 2019a.

Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. Diffusion Improves Graph Learning. Neural Information Processing Systems, NeurIPS , 2019b.

Floris Geerts. On the Expressive Power of Linear Algebra on Graphs. Theory Comput. Syst. , 65(1): 179-239, 2021.

Simon Geisler, Yujia Li, Daniel Mankowitz, Ali Taylan Cemgil, Stephan Günnemann, and Cosmin Paduraru. Transformers Meet Directed Graphs. In International Conference on Machine Learning, ICML , 2023.

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, ICML , 2017.

Lorenzo Giusti, Teodora Reu, Francesco Ceccarelli, Cristian Bodnar, and Pietro Liò. CIN++: Enhancing Topological Message Passing, arXiv, 2023.

Martin Grohe, Kristian Kersting, Martin Mladenov, and Pascal Schweitzer. Color Refinement and Its Applications. In Guy Van Den Broeck, Kristian Kersting, Sriraam Natarajan, and David Poole (eds.), An Introduction to Lifted Probabilistic Inference , pp. 349-372. The MIT Press, 2021.

Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv, 2023.

Yuhe Guo and Zhewei Wei. Graph Neural Networks with Learnable and Optimal Polynomial Bases. In International Conference on Machine Learning, ICML . arXiv, 2023.

Benjamin Gutteridge, Xiaowen Dong, Michael Bronstein, and Francesco Di Giovanni. DRew: Dynamically Rewired Message Passing with Delay. In International Conference on Learning Representations, ICLR , 2023.

David K. Hammond, Pierre Vandergheynst, and Rémi Gribonval. Wavelets on graphs via spectral graph theory. Applied and Computational Harmonic Analysis , 30(2):129-150, 2011.

Mingguo He, Zhewei Wei, Zengfeng Huang, and Hongteng Xu. BernNet: Learning Arbitrary Graph Spectral Filters via Bernstein Approximation. In Neural Information Processing Systems, NeruIPS , 2021.

Mingguo He, Zhewei Wei, and Ji-Rong Wen. Convolutional Neural Networks on Graphs with Chebyshev Approximation, Revisited. In Neural Information Processing Systems, NeurIPS , 2022.

Xiaoxin He, Bryan Hooi, Thomas Laurent, Adam Perold, Yann LeCun, and Xavier Bresson. A Generalization of ViT/MLP-Mixer to Graphs. In International Conference on Machine Learning, ICML , 2023.

Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR , abs/1606.08415, 2016.

Sepp Hochreiter and J Urgen Schmidhuber. Long Short-term Memory. Neural Computation , 9(8): 17351780-17351780, 1997.

Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, arXiv, 2017.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In Neural Information Processing Systems, NeurIPS , 2020.

Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. OGBLSC: A Large-Scale Challenge for Machine Learning on Graphs, arXiv, 2021.

Ningyuan Huang, Soledad Villar, Carey E. Priebe, Da Zheng, Chengyue Huang, Lin Yang, and Vladimir Braverman. From Local to Global: Spectral-Inspired Graph Neural Networks. In New Frontiers in Graph Learning at NeurIPS . arXiv, 2022.

Yinan Huang, William Lu, Joshua Robinson, Yu Yang, Muhan Zhang, Stefanie Jegelka, and Pan Li. On the Stability of Expressive Positional Encodings for Graph Neural Networks. In International Conference on Learning Representations, ICLR , 2024.

MdShamim Hussain, Mohammed J. Zaki, and Dharmashankar Subramanian. Global Self-Attention as a Replacement for Graph Convolution. In International Conference on Knowledge Discovery and Data Mining, KDD , pp. 655-665, 2022.

Md Shamim Hussain, Mohammed J. Zaki, and Dharmashankar Subramanian. Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformers, arXiv, 2024.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, ICLR , 2017.

Christian Koke and Daniel Cremers. HoloNets: Spectral Convolutions do extend to Directed Graphs. In International Conference on Learning Representations, ICLR , 2024.

Arthur Kosmala, Johannes Gasteiger, Nicholas Gao, and Stephan Günnemann. Ewald-based LongRange Message Passing for Molecular Graphs. In International Conference on Machine Learning, ICML , 2023.

Devin Kreuzer, Dominique Beaini, William L. Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking Graph Transformers with Spectral Attention. In Neural Information Processing Systems, NeurIPS , 2021.

Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, and Peter Battaglia. Learning skillful medium-range global weather forecasting. Science , 382(6677):1416-1421, 2023.

LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation , 1(4):541551, 1989.

Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In International Conference on Machine Learning, ICML , 2019.

B. Lehoucq, D. C. Sorensen, and C. Yang. ARPACK Users' Guide: Solution of Large-Scale Eigenvalue Problems with Implicitly Restarted Arnoldi Methods . Society for Industrial and Applied Mathematics, 1998.

Pan Li and Jure Leskovec. The Expressive Power of Graph Neural Networks. In Lingfei Wu, Peng Cui, Jian Pei, and Liang Zhao (eds.), Graph Neural Networks: Foundations, Frontiers, and Applications , pp. 63-98. Springer Nature Singapore, Singapore, 2022.

Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning. In Neural Information Processing Systems, NeurIPS , 2020.

Renjie Liao, Zhizhen Zhao, Raquel Urtasun, and Richard S. Zemel. LanczosNet: Multi-Scale Deep Graph Convolutional Networks. In International Conference on Learning Representations, ICLR . arXiv, 2019.

Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods. In Advances in Neural Information Processing Systems , 2021.

Derek Lim, Joshua Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and Basis Invariant Networks for Spectral Graph Representation Learning. In International Conference on Learning Representations, ICLR , 2023.

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, ICLR , 2017.

Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. International Conference on Learning Representations, ICLR , 2019.

Yuankai Luo, Hongkang Li, Lei Shi, and Xiao-Ming Wu. Enhancing Graph Transformers with Hierarchical Distance Structural Encoding, arXiv, 2024.

Liheng Ma, Chen Lin, Derek Lim, Adriana Romero-Soriano, Puneet K. Dokania, Mark Coates, Philip Torr, and Ser-Nam Lim. Graph Inductive Biases in Transformers without Message Passing. In International Conference on Machine Learning, ICML , 2023.

Amil Merchant, Simon Batzner, Samuel S. Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature , 624(7990):80-85, 2023.

Gaspard Michel, Giannis Nikolentzos, Johannes Lutzeyer, and Michalis Vazirgiannis. Path Neural Networks: Expressive and Accurate Graph Neural Networks. In International Conference on Machine Learning, ICML , 2023.

Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and Leman Go Neural: Higher-Order Graph Neural Networks. AAAI Conference on Artificial Intelligence , 33:4602-4609, 2019.

P. Natanson. Constructive function theory. Vol. I: Uniform approximation. Translated by Alexis N. Obolensky. New York: Frederick Ungar Publishing Co. IX, 232 p. (1964)., 1964.

Nhat Khang Ngo, Truong Son Hy, and Risi Kondor. Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structures. The Journal of Chemical Physics , 159(3):034109, 2023.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context Learning and Induction Heads, arXiv, 2022.

Seonghyun Park, Narae Ryu, Gahee Kim, Dongyeop Woo, Se-Young Yun, and Sungsoo Ahn. Nonbacktracking Graph Neural Networks, arXiv, 2023.

Phitchaya Mangpo Phothilimthana, Sami Abu-El-Haija, Kaidi Cao, Bahare Fatemi, Charith Mendis, and Bryan Perozzi. TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs. In Neural Information Processing Systems, NeruIPS , 2023.

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena Hierarchy: Towards Larger Convolutional Language Models. In International Conference on Machine Learning, ICML . arXiv, 2023.

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, arXiv, 2022.

Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a General, Powerful, Scalable Graph Transformer. In Neural Information Processing Systems, NeurIPS , 2022.

Emanuele Rossi, Bertrand Charpentier, Francesco Di Giovanni, Fabrizio Frasca, Stephan Günnemann, and Michael Bronstein. Edge Directionality Improves Learning on Heterophilic Graphs. In Learning on Graphs Conference, LoG , 2023.

Kristof T. Schütt, Pieter-Jan Kindermans, Huziel E. Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. SchNet: A continuous-filter convolutional neural network for modeling quantum interactions. In Neural Information Processing Systems, NeruIPS . arXiv, 2017.

Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J. Sutherland, and Ali Kemal Sinop. Exphormer: Sparse Transformers for Graphs. In International Conference on Machine Learning, ICML , 2023.

A. Shubin. Discrete Magnetic Laplacian. Communications in Mathematical Physics , 164(2): 259-275, 1994.

Laurent Sifre. Rigid-Motion Scattering For Image Classification . PhD thesis, Ecole Polytechnique, CMAP, 2014.

Kimberly Stachenfeld, Jonathan Godwin, and Peter Battaglia. Graph Networks with Spectral Message Passing, arXiv, 2020.

Zekun Tong, Yuxuan Liang, Changsheng Sun, David S. Rosenblum, and Andrew Lim. Directed Graph Convolutional Network, arXiv, 2020.

Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. In International Conference on Learning Representations, ICLR , 2022.

Jan Tönshoff, Martin Ritzert, Eran Rosenbluth, and Martin Grohe. Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark. In Learning on Graphs Conference , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, NeurIPS , 2017.

Petar Veliˇ ckovi´ c, Arantxa Casanova, Pietro Liò, Guillem Cucurull, Adriana Romero, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, ICLR , 2018.

Ulrike von Luxburg. A tutorial on spectral clustering. Statistics and Computing , 17(4):395-416, 2007.

Haorui Wang, Haoteng Yin, Muhan Zhang, and Pan Li. Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks. In International Conference on Learning Representations, ICLR , 2022.

Xiyuan Wang and Muhan Zhang. How Powerful are Spectral Graph Neural Networks. In International Conference on Machine Learning, ICML . arXiv, 2022.

Tom Wollschläger, Niklas Kemper, Leon Hetzel, Johanna Sommer, and Stephan Günnemann. Expressivity and Generalization: Fragment Biases for Molecular GNNs, arXiv, 2024.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken Ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning, ICML , 2018.

Keyulu Xu, Stefanie Jegelka, Weihua Hu, and Jure Leskovec. How powerful are graph neural networks? In International Conference on Learning Representations, ICLR , 2019.

Mingqi Yang, Wenjie Feng, Yanming Shen, and Bryan Hooi. Towards Better Graph Representation Learning with Parameterized Decomposition & Filtering. In International Conference on Machine Learning, ICML , 2023.

Jiaxuan You, Rex Ying, and Jure Leskovec. Design Space for Graph Neural Networks. pp. 13, 2020.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. In Neural Information Processing Systems, NeruIPS , 2017.

Xitong Zhang, Yixuan He, Nathan Brugnone, Michael Perlmutter, and Matthew Hirn. MagNet: A Neural Network for Directed Graphs. In Neural Information Processing Systems, NeurIPS , 2021.

Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling Transformers with LEGO: a synthetic reasoning task, arXiv, 2023.

Appendix

A Notation

B Related Work for Expressivity and Directed Graphs

Expressivity. Laplacian eigenvectors have been used previously to construct positional encodings that improve the expressivity of GNNs or Transformers (Lim et al., 2023; Wang et al., 2022; Geisler et al., 2023; Huang et al., 2024). Our positional encodings are similar to the preprocessing of Balcilar et al. (2021a), where the authors design an edge-level encoding/mask to surpass 1-WL. The hierarchy of Weisfeiler-Leman (WL) tests is a common way to categorize the expressivity of GNNs (Grohe

et al., 2021). Xu et al. (2019) showed that most MPGNNs are bound by or as strong as the 1-WL test. Lim et al. (2023) point out that spectral GNNs suffer from similar limitations as MPGNNs w.r.t. their expressivity. Generally, the development of expressive GNNs is an active research direction, and we refer to Li & Leskovec (2022) for a broad overview.

Directed graphs. Rossi et al. (2023) also extend the WL test to directed graphs and propose an MPGNN for directed graphs. How to model direction in graphs is also still an open question and various approaches were proposed (Battaglia et al., 2018; Tong et al., 2020; Zhang et al., 2021; Rossi et al., 2023; Koke & Cremers, 2024). We utilize a Hermitian Laplacian for direction awareness, namely the Magnetic Laplacian, which was also used by Zhang et al. (2021) for an MPGNN and Geisler et al. (2023) for positional encodings.

C Background for Directed Graphs

Undirected vs. directed graphs. For spatial filtering, it is straightforward to plausibly extend the message passing (e.g. Battaglia et al. (2018); Rossi et al. (2023)). However, the spectral motivation and spectral filter on directed graphs require more care. The eigendecomposition is guaranteed to exist for real symmetric matrices. Real symmetric matrices are always diagonalizable, and the eigenvectors will then span a complete orthogonal basis to represent all possible signals X ∈ R n × d . Note that some non-symmetric square matrices are also diagonalizable and, thus, also have an eigendecomposition, albeit the eigenvectors may not be orthogonal. Thus, further consideration is required to generalize the graph Laplacian to general directed graphs.

Magnetic Laplacian. For the spectral filter on directed graphs, we build upon a direction-aware generalization, called Magnetic Laplacian (Forman, 1993; Shubin, 1994; De Verdière, 2013; Furutani et al., 2020; Geisler et al., 2023)

where A s = A ∨ A ⊤ is the symmetrized graph with diagonal degree matrix D s . ⊙ denotes the element-wise product, exp the element-wise exponential, i = √ -1 an imaginary number, and q the potential (hyperparameter). By construction L q is a Hermitian matrix L q = L H q where the conjugate transpose is equal to L q itself. Importantly, Hermitian matrices naturally generalize real symmetric matrices and have a well-defined eigendecomposition L q = V Λ V H with real eigenvalues Λ and unitary eigenvectors V V H = I . For appropriate choices of the potential q , the order of eigenvalues is well-behaved (Furutani et al., 2020). Recently Geisler et al. (2023) demonstrated the efficacy of these eigenvectors for positional encodings for transformers. Moreover, the Magnetic Laplacian was used for a spectrally designed spatial MPGNN (Zhang et al., 2021), extending Defferrard et al. (2017). Due to the real eigenvalues, one could, in principle, also apply a monomial basis (Chien et al., 2021), or different polynomial bases stemming from approximation theory (He et al., 2021; Wang & Zhang, 2022; He et al., 2022; Guo & Wei, 2023).

To see why Eq. 7 describes an injection for appropriate choices of q , consider that the sparsity pattern of L q matches A up to the main diagonal. If A contains a self-loop the main diagonal will have a 0 instead of 1 entry at the self-loop location. A -A ⊤ can be directly inferred from the phase exp[ i 2 πq ( A -A ⊤ )] , assuming that q < 1 / (2 max u,v A u,v ) . Thus, it is solely left to obtain A s from I -D -1 / 2 s A s D -1 / 2 s , which is trivial for a binary adjacency but more involved for real-valued weights. Determining if and when L q is injective for real-valued A is left for future work.

̸

Properties of the eigendecomposition. The eigendecomposition is not unique, and thus, one should consider the result of the eigensolver arbitrary in that regard. One ambiguity becomes apparent from the definition of an eigenvalue itself Lv = λ v since one can multiply both sides of the equation with a scalar c ∈ C \ { 0 } : L ( c v ) = λ ( c v ) . We already implicitly normalized the magnitude of the eigenvectors V by choosing them to be orthogonal ( V V ⊤ = I ) or unitary ( V V H = I ). Thus, after this normalization, c only represents an arbitrary sign for real-valued eigenvectors or a rotation on the unit circle in the complex case. Another reason for ambiguity occurs in the case of repeated / multiple eigenvalues (e.g., λ u = λ v for u = v ). In this case, the eigensolver may return an arbitrary set of orthogonal eigenvectors chosen from the corresponding eigenspace.

D Limitations of Graph Transformers Using Absolute Positional Encodings

Here, we consider a vanilla graph transformer f ( X ) that solely becomes structure-aware due to the addition (or concatenation) of positional encodings: f ( X + PE( A )) . The main point we are going to demonstrate is that a vanilla transformer with such absolute positional encodings PE( A ) ∈ R n × d will be limited in its expressivity if the positional encodings are permutation equivariant P PE( A ) = PE( PAP ⊤ ) w.r.t. any n × n permutation matrix P ∈ P .

The limitation particularly arises in the presence of automorphisms P a AP ⊤ a = A with specifically chosen permutation P a . To be more specific, assume that nodes u and v are automorphic to each other. That is, there exists a P a that will swap the order of u and v (among other nodes) s.t. P a AP ⊤ a = A . By permutation equivariance, we know P a PE( A ) = PE( P a AP ⊤ a ) = PE( A ) and, hence, PE( A ) u = PE( A ) v .

We have just shown that automorphic nodes will have the same positional encodings PE if the positional encodings are permutation equivariant. This implies that permutation equivariant positional encodings PE are not even able to capture simple neighboring relationships. For example, consider an undirected sequence/path graph o-o-o-o-o with five nodes. Here, the two end nodes, which we also all first and last node, are automorphic. So are the second and second-last nodes. Assuming the second and second last nodes have different node features (e.g., A-B-C-D-A ), that breaks the symmetry, it is still not possible for a transformer with absolute positional encodings to tell the first and last node apart. In other words, in the example, the transformer cannot tell apart the end node with neighboring feature B from the end node with neighboring feature D . This shows a severe limitation of architectures without additional components capturing the relative distances (e.g., as S 2 GNNs can). This concern is not as critical for architectures where the positional encodings are not entirely permutation equivariant (Dwivedi & Bresson, 2021; Kreuzer et al., 2021), with relative positional encodings (Ma et al., 2023), and might also be of lesser concern for directed graphs (Geisler et al., 2023).

E S 2 GNN Generalizes a Virtual Node

Adding a fully connected virtual node (Gilmer et al., 2017) is among the simplest ways to add the ability for long-range information exchange. An equivalent method was proposed as a simple oversquashing remedy in the seminal work by Alon & Yahav (2020). A single Spectral layer amounts to a type of virtual nodes in the special case of f θ = I and

Assuming a simply-connected graph G , the unique normalized zero-eigenvector v of the symmetrically-normalized graph Laplacian L = I -D -1 / 2 AD -1 / 2 has components v u = √ d u 2 | E | , where d u denotes the degree of node u ∈ G , and | E | the number of edges in the graph. At node u ∈ G , we therefore find

with h ( l -1) v denoting the row of H ( l -1) corresponding to node v ∈ G . In other words, filtering out the zero-frequency component of the signal means scattering a global, degree-weighted embedding average to all nodes of the graph. For the unnormalized graph Laplacian, Eq. 9 instead becomes an unweighted average, which is consistent with the usual definition of a virtual node. We refer to Fig. 3 & 4 for additional intuition.

F Existing Results on Over-Squashing

We restate two key results from Di Giovanni et al. (2023a) using our notation. They imply the existance of a regime in which 1-hop MPNN architectures suffer from exponentially decaying Jacobian sensitivity. Meanwhile, S 2 GNNs can easily learn a signal of constantly lower-bounded sensitivity, as shown by invoking its trivial subcase of a virtual node in Theorem 2.

Theorem 7 (Adapted from Di Giovanni et al. (2023a)) . In an ℓ -layer spatial MPGNN with messagepassing matrix S = c r I + c a A ( c r , c a ∈ R + ) and a Lipschitz nonlinearity σ ,

the Jacobian sensitivity satisfies the following upper bound:

with h (0) u , h ( ℓ ) v denoting the rows of H (0) , H ( ℓ ) corresponding to the nodes v , u ∈ G , c σ the Lipschitz constant of the nonlinearity, w the maximum entry value over all weight matrices W ( l ) , and d the network width.

The dependence of the upper bound on the matrix power ( S ℓ ) vu - not generally present for S 2 GNN by Theorem 2 - leads to a topology-dependence which becomes explicit in the following theorem. It concerns the typical shallow-diameter regime, in which the number ℓ of MPGNN layers is comparable to the graph diameter.

Theorem 8 (Adapted from Di Giovanni et al. (2023a)) . Given an MPNN as in Eq. 10, with c a ≤ 1 , let v, u ∈ G be at distance r . Let c σ be the Lipschitz constant of σ, w the maximal entry-value overall weight matrices, d min the minimal degree of G , and γ ℓ ( v, u ) the number of walks from v to u of maximal length ℓ . For any 0 ≤ k < r , there exists C k > 0 independent of r and of the graph, such that

For 1-hop MPGNNs with 2 c σ wd < d min , we therefore identify an exponential decay of sensitivity with node distance r in the weak-connectivity limit for which γ r + k ( v, u ) increases subexponentially with r . As Di Giovanni et al. (2023a) point out, sharper bounds can be derived under graph-specific information about ( S r ) vu .

G Construction of an explicit ground truth filter

We express the electric potential along a periodic sequence of screened 1D charges as a convolution of a corresponding graph signal with a consistently defined graph filter. This closed-form example underscores our default use of a low-pass window for the spectral part of S 2 GNNs by showing how a continuous problem with a convolutional structure and quickly flattening spectral response (typical for pair interactions in physics and chemistry) discretizes into a graph problem with similar features.

The approach exploits the surjective mapping of Fourier modes on [0 , n ] onto the Laplacian eigenvectors of a cycle graph C n . We consider two corresponding representations of the same problem:

A continuous representation (Eq. 12) in terms of a 1D distribution ρ of n point charges q 1 , . . . , q n and their periodically repeating image charges, written as a sum of Dirac combs at equidistant offsets l with 0 ≤ l ≤ n -1 , interacting via the potential profile ϕ σ obtained from solving in Gauss' law of electrostatics for a 1D point charge screened by a Gaussian cloud of opposite background charge with standard deviation σ . The screening ensures

convergence to a finite potential and its exact form is insignificant (we choose the Gaussiantype screening due to its analytical tractability). Note that ϕ σ ( x ) ≃ const. -| x | for x → 0 (the unscreened 1D potential in the direction normal to an infinitely wide capacitor plate), while the screening guarantees an exponential dropoff to zero as x →∞ ,

A graph representation (Eq. 14) by placing the n charges q 1 , . . . , q n onto a cycle graph C n .

We derive the graph filter g σ from a consistency condition (Eq. 13) between both representations: the graph convolution ( g σ ∗ G q ) has to yield the electric potential V sampled at the charge loci if we want g σ to act like the continuous convolution kernel ϕ σ in the discrete graph picture.

The Fourier transform of ϕ σ (in the convention without integral prefactor and with a 2 π frequency prefactor) reads ˆ ϕ σ ( κ ) = 1 πκ 2 ( 1 -exp ( -1 2 σ 2 κ 2 )) . For the density, the Poisson sum formula gives ˆ ρ ( κ ) = ∑ n -1 k =0 1 √ n ˆ q k ∆ 1 ( κ -k n ) with ˆ q k = 1 √ n ∑ n -1 j =0 q i exp ( -i 2 π k n j ) . The coefficients ˆ q k are precisely the components of the graph Fourier transform of q (physically, they amount for the structure factor). By the convolution theorem, ˆ V ( κ ) = ˆ ϕ σ ( κ )ˆ ρ ( κ ) . By noting that all integer-shifted frequencies in the Dirac combs ∆ 1 ( · -k n ) (or all Brillouin zones, in physics terminology) yield the same phase exp ( i 2 π k n l ) if we only sample V ( x ) at the charge loci x = l , 0 ≤ l ≤ n -1 , we can write V ( l ) = 1 2 π ∑ n -1 k =0 ˆ q k ( ∑ m ∈ Z ˆ ϕ σ ( k n + m ) ) 1 √ n exp ( i 2 π k n l ) . Through pattern-matching with the consistency condition of Eq. 13, we can therefore identify that the graph filter is a sum over Brillouin zones, (ˆ g σ )( λ k ) = 1 2 π ∑ m ∈ Z ˆ ϕ σ ( k n + m ) , where λ k denotes the eigenvalues of the normalized C n graph Laplacian, λ k = 1 -cos ( 2 πk n ) . To fulfill this relation for all n , k we set

We claim now (and prove in a later paragraph) that for λ > λ 0 > 0 and a sufficiently large choice σ > σ ( r, λ 0 ) , the absolute r -th derivative satisfies the upper bound | d r dλ r ˆ g σ ( λ ) | ≤ | d r dλ r ˆ g ∞ ( λ ) | , where we can think of ˆ g ∞ as the limit of taking σ →∞ (i.e., a constant background charge):

The merit of this is that unlike the screened ˆ g σ ( λ ) , ˆ g ∞ ( λ ) can be solved analytically to find closedform bounds on the absolute derivatives | d r dλ r ˆ g σ ( λ 0 ) | . By invoking the sum expansion form of the trigamma function Ψ 1 ( z ) = ∑ ∞ m =0 1 ( z + m ) 2 , the reflection identity ψ 1 (1 -z ) + ψ 1 ( z ) = π 2 sin 2 πz , and the half-angle formula sin 2 ( x 2 ) = 1 -cos( x ) 2 , we find

a remarkably simple result. We can now readily evaluate | d r dλ r ˆ g ∞ ( λ ) | = r ! λ r +1 , but it remains to prove that this upper-bounds | d r dλ r ˆ g σ ( λ ) | for any λ > λ 0 > 0 and sufficiently large σ > σ ( r, λ 0 ) . For compactness, define the expressions z ( λ ) := 1 2 π arccos(1 -λ ) ∈ [ 0 , 1 2 ] (strictly increasing in

λ ), y σ ( z ) := exp ( -1 2 σ 2 z 2 ) , and ˜ z = 1 -z ≥ z . Consider the series of 'term-by-term' derivatives

They converge uniformly on [ 0 , 1 2 ] as they clearly are Cauchy sequences under uniform bound (moreover, well-definedness in z = 0 follows by applying l'Hospital's rule - physically, this is the merit provided by including Gaussian screening in our model). Therefore, they indeed converge to the respective derivatives d r dz r ˆ g σ ( λ ( z )) (justifying the above notation). The same holds for the corresponding series for d r dz r ˆ g ∞ ( λ ( z )) : they are not defined in z = 0 , but otherwise still converge as they match the known series expansion of the polygamma function. Given λ 0 > 0 and thus z ( λ 0 ) > 0 , taking σ larger than some σ ( r, λ 0 ) guarantees that d r dz r ˆ g σ ( λ ( z )) and d r dz r ˆ g ∞ ( λ ( z )) are of the same sign for λ > λ 0 ( z ( λ ) > z ( λ 0 ) ). This holds for all orders r ∈ N since we see by induction that the product rule always yields one term analogous to the first respective terms above, and otherwise only terms of O ( y σ ( m ))) . Then, observing that 0 ≤ 1 -y σ ( x ) < 1 ∀ x ≥ 0 and ˜ z ≥ z implies | d r dz r ˆ g σ ( λ 0 ( z 0 )) | ≤ | d r dz r ˆ g ∞ ( λ 0 ( z 0 )) | . The same must hold for the λ -derivatives by the chain rule.

One interesting question is whether ˆ g σ is also C -integral-Lipschitz for some constant C > 0 . We discuss this stability-informed criterion (Gama et al., 2020) in the main body as a domain-agnostic prior assumption about the 'ideal' graph filter if no other ground truth knowledge informing additional smoothness bounds (such as here) is available. While the above bound is too loose to certify this directly ( | d dλ ˆ g σ ( λ ) | ≤ Cλ -1 would be needed), integral-Lipschitzness under some constant follows from the fact that | d dλ ˆ g σ ( λ ) | is bounded on [0 , 2] : by the uniform convergence of the term-by-term derivative series, it is continuous. Well-definedness of the product d dz ˆ g σ dz dλ has to be checked in λ = 0 , where it follows by continuous extension using l'Hospital's rule. As a continuous function defined on a compact interval, | d dλ ˆ g σ | assumes a maximum.

H Proofs

H.1 Proof of Theorem 1

We next prove the permutation equivariance of the spectral filter in Eq. 3:

Theorem 1. Spectral( H ( l -1) ; EVD( L , k )) of Eq. 3 is equivariant to all n × n permutation matrices P ∈ P : Spectral( PH ( l -1) ; EVD( PLP ⊤ , k )) = P Spectral( H ( l -1) ; EVD( L , k )) .

for the general case of parametrizing a Hermitian 'Laplacian' L ∈ C n × n , L H = L . Note that this proof does not rely in any means on the specifics of L , solely that the eigendecomposition exists L = V Λ V H with unitary eigenvectors V V H = I . For practical reasons, it is suitable to define L ( A ) as a function of A . Asimilar proof for real-valued eigenvectors is given by (Lim et al., 2023). The specific spectral filter we consider is

with arbitrary f : C d 1 → C d 2 , applied row-wise to H ( l -1) ∈ C n × d 1 . Analogously, h : C d 2 → C d 3 is applied row-wise. We choose complex functions to emphasize generality, although we restrict

Spectral to real in- and outputs in all experiments. The graph filter is defined as element-wise function ˆ g u,v ( λ ) := ˆ g v ( λ u , { λ 1 , λ 2 , . . . , λ k } ) that depends on the specific eigenvalue λ and potentially the set of eigenvalues { λ 1 , λ 2 , . . . , λ k } (or its vector representation λ ) of the partial eigendecomposition.

̸

Weneed to make sure that the partial decomposition includes all eigenvalues of the same magnitude, i.e., λ u = λ u ′ , ∀ u ∈ { 1 , 2 , . . . , k } , u ′ ∈ { k + 1 , k + 2 , . . . , n } . In practice, this is achieved by choosing large enough k to accommodate all eigenvalues λ cut < λ k +1 , or by dropping trailing eigenvalues where λ j = λ k +1 for j ∈ { 1 , 2 , . . . , k } . Generally, it is also not important that we consider the k smallest eigenvalues in the spectral filter. We only need to ensure that the spectral filter is either calculated on all or no eigenvalues/-vectors of an eigenspace.

Proof. Assuming functions ϕ ( X ) and ψ ( X ) are permutation equivariant, then ϕ ( ψ ( X )) is permutation equivariant ϕ ( ψ ( PX )) = ϕ ( P ψ ( X )) = P ϕ ( ψ ( X )) for any n × n permutation P ∈ P . Thus, it sufficies to prove permutation equivariance for h, f, V (ˆ g ( λ ) ⊙ [ V H X ]) independently, where X ∈ C n × d 2 .

Regardless of the complex image and domain of h and f , they are permuation equivariant since they are applied row-wise

and reordering the rows in X ∈ C n × d 1 also reorders the outputs: f ( PX ) = P f ( X ) .

For finalizing the proof of permutation equivariance, we first rearrange Y = V (ˆ g ( λ ) ⊙ [ V H X ]) = ∑ k u =1 v u (ˆ g u, : ( λ u ) ⊙ [ v H u X ]) and Y : ,v = ∑ k u =1 ˆ g u,v ( λ u ) v u v H u X : ,v .

This construction (a) is invariant to the ambiguity that every eigenvector v u can be arbitrarily rotated c u v u by { c u ∈ C | | c u | = 1 } . That is, ( c u v u )( c u v u ) H = c u ¯ c u v u v H u = v u v H u .

Moreover, (b) in the case of j repeated eigenvalues { s +1 , s +2 , . . . , s + j } where λ s +1 = λ s +2 = · · · = λ s + j , we can choose a set of orthogonal eigenvectors arbitrarily rotated/reflected from the j -dimensional eigenspace (basis symmetry). The given set of eigenvectors can be arbitrarily transformed V : ,s +1: s + j Γ j by a matrix chosen from the unitary group Γ j ∈ U ( j ) . Since

we simply need to show that the expression is invariant to a transformation by Γ j :

To see why Γ j ∈ U ( j ) is a sufficient choice in the light of repeated/multiple eigenvalues, consider the defintion of eigenvalues/vectors

we can now multiply both sides from the right with an arbitrary matrix B ∈ C j × j . To preserve the unitary property V : ,s +1: s + j V H : ,s +1: s + j = I , we require ( V : ,s +1: s + j B )( V : ,s +1: s + j B ) H = I . Thus, the eigenvectors can be arbitrarily transformed by Γ j ∈ U ( j ) instead of B ∈ C j × j .

This concludes the proof.

H.2 Proof of Theorem 2

We restate Theorem 2 in more detail and also considering graphs that contain multiple connected components. The unchanged bottom line is that S 2 GNNs can express signals lower-bounded by a constant that is unaffected by local properties of the graph topology, instead of suffering from exponential sensitivity decay like spatial MPGNNs.

Theorem (Theorem 2, formal) . Consider an ℓ -layer S 2 GNN of the form Eq. 1. Let ( ˜ ϑ, ϑ, θ ) be parameters of the spatial GNN, spectral filters ˆ g ( l ) ϑ , and feature transformation f θ . Assume the existence of parameters ˜ ϑ such that Spatial ( l ) ( H ( l -1) ; A , ˜ ϑ ) = 0 ∀ 1 ≤ l ≤ ℓ and θ such that f θ = I . Then, a filter choice ϑ exists such that the ℓ -layer S 2 GNN of the additive form Eq. 1 can express a signal h ( ℓ ) v ( H (0) ; ˜ ϑ, ϑ, θ ) with uniformly lower-bounded Jacobian sensitivity,

̸

with h (0) u , h ( ℓ ) v denoting the rows of H (0) , H ( ℓ ) corresponding to the nodes u = v ∈ G , connected component C ⊂ G containing | E C | edges, network width d and parameter-dependent constant K ϑ .

Proof. Choose ˜ ϑ such that Spatial ( l ) ( H ( l -1) ; A ) = 0 ∀ 1 ≤ l ≤ ℓ (typically by setting all weights and biases to zero), θ such that f θ = I , and set ϑ such that

for some K ϑ > 0 . This choice of filter parameters ϑ lets Spectral act like a type of virtual node across all hidden dimensions k : In the standard orthonormal basis of the 0 -eigenspace given by

where C enumerates all connected components, and d u denotes the degree of node u , we find

with C ( v ) denoting the connected component containing v . Particularly, note that applying the spectral layer more than once does not affect the result since the projector onto an eigenvector is idempotent (up to K ϑ ). The result must also hold in any other orthonormal basis of the 0 -eigenspace due to the invariance of Spectral under orthogonal eigenbasis transformations. Differentiating with respect to h (0) u , taking the L 1 norm and using √ d u d v ≥ 1 shows the statement.

H.3 Proof of Theorem 3

Theorem 3. Let ˆ g be a discontinuous spectral filter. For any approximating sequence ( g γ p ) p ∈ N of polynomial filters, an adversarial sequence ( G p ) p ∈ N of input graphs exists such that

̸

Theorem 9 (Bernstein) . Let f : [0 , 2 π ] → C be a 2 π -periodic function. Then f is α -Hölder continuous for some α ∈ (0 , 1) if, for every p ∈ N , there exists a degreep trigonometric polynomial T p ( x ) = a 0 + ∑ p j =1 a j cos( jx ) + ∑ p j =1 b j sin( jx ) with coefficients a j , b j ∈ C , such that

where C(f) is a positive number depending on f .

Proof. Given a discontinuous filter ˆ g : [0 , 2] → R , construct the function f : [0 , 2 π ] → C fulfilling the prerequisites of Theorem 9 by pre-composing f := ˆ g ◦ (cos( · ) + 1) . We proceed via contradiction. Suppose that there is an α ∈ (0 , 1) and a sequence of degreep polynomial filters, ˆ g γ p ( λ ) = ∑ p j =0 γ j λ j , γ = ( γ 0 , . . . , γ p ) ⊤ ∈ R p +1 , such that ∥ ˆ g γ p -ˆ g ∥ ∞ = O ( p -α ) . Then, the sequence of trigonometric polynomials T p := ˆ g γ p ◦ (cos( · ) + 1) fulfills the condition of Theorem 9. This would imply that f = ˆ g ◦ (cos( · ) + 1) is α -Hölder continuous, meaning that a constant K > 0 exists such that

Considering λ 0 ∈ [0 , 2] , λ → λ 0 and x = arccos( λ 0 -1) , y = arccos( λ -1) (using the arccos branch in which both λ 0 , λ eventually end up) shows a contradiction to the assumed discontinuity of ˆ g . Therefore, no polynomial filter sequence ( ˆ g γ p ) p ∈ N together with an α ∈ (0 , 1) exist such that ∥ ˆ g γ p -ˆ g ∥ ∞ = O ( p -α ) . In particular, for any sequence ( ˆ g γ p ) p ∈ N , a sequence of adversarial values ( λ p ) p ∈ N , λ p ∈ [0 , 2] exists such that

The proof is finished if we can find a sequence of graphs ( G p ) such that the symmetricallynormalized graph Laplacian L p of G p contains λ p as an eigenvalue. In this case, we can construct adversarial input signals X p on the graphs G p by setting the first embedding channel to an eigenvector corresponding to λ p , and the remaining channels to zero, such that ( g γ p -g ) ∗ G p X p = | ˆ g γ p ( λ p ) -ˆ g ( λ p ) | X p . In particular, it then holds that

̸

If we assume only simple graphs, such a construction is unfortunately not possible since the set of all simple graphs and therefore the set of all realizable eigenvalues is countable, whereas the adversarial values λ p could lie anywhere in the uncountable set [0 , 2] . We can, however realize arbitrary eigenvalues by using weighted graphs with three nodes. Consider a cyclic graph structure and tune the weight of edge (1 , 2) to sin 2 ( θ p ) and the weight of edges (2 , 3) and (3 , 1) to cos 2 ( θ p ) with θ p ∈ [ 0 , π 2 ] . The symmetrically-normalized graph Laplacian,

has eigenvalues λ (1) p = 1 , λ (2) p = sin 2 ( θ p ) , λ (3) p = 2 -sin 2 ( θ p ) . λ (2) p can assume all values λ p ∈ [0 , 1] , whereas λ (3) p can assume all values λ p ∈ [1 , 2] . This finishes the proof.

̸

Remark. If one wishes to restrict the set of possible adversarial graph sequences ( G p ) p ∈ N to include only simple graphs, a version of Theorem 3 still holds where we restrict the assumption to filters ˆ g which are piecewise-continuous with discontinuities on a finite set of points D ⊂ S , where S ⊂ [0 , 2] denotes the countable set of eigenvalues realizable by simple graphs. This still covers a large class of filters to which orderp polynomial filters can provably converge slower than any inverse root of p in the operator norm, and includes the virtual node filter (discontinuous only in λ = 0 ) presented as an example in the main body. The proof is fully analogous up to the point of constructing λ p . If λ p ∈ D , we can find a graph that realizes it exactly. Now assume λ p / ∈ D . We note that the set S is dense in [0 , 2] (clear from considering, e.g., the cyclic graphs C n with symmetricallynormalized Laplacian eigenvalues λ k = 1 -cos ( 2 πk n ) ). Since we assume that ˆ g and therefore also | ˆ g γ p -ˆ g | is piecewise-continuous anywhere but on D ⊂ S and D is finite, we can find an open neighborhood N ( λ p ) for any λ p / ∈ D on which ˆ g is continuous. Using that S is dense in [0 , 2] , we find a graph sequence ( ˜ G ( l ) p ) l ∈ N with eigenvalues ˜ λ ( l ) p ∈ N ( λ p ) ∀ l ∈ N , ( ˜ λ ( l ) p ) l ∈ N → λ p for which ∥ ˆ g γ p ( ˜ λ ( l ) p ) -ˆ g ( ˜ λ ( l ) p ) ∥ → ∥ ˆ g γ p ( λ p ) -ˆ g ( λ p ) ∥ . Therefore, by the same reasoning as in the proof of Theorem 3, we find that there can be no α ∈ (0 , 1) for which sup 0 = X ∈ R |G p |× d ∥ ( g γ p -g ) ∗ G p X ∥ F ∥ X ∥ F is of O ( p -α ) .

H.4 Proof of Theorem 4

We first introduce the setting and notation to state Theorem 4 in its general version. We study how well S 2 GNNs can approximate 'idealized' GNNs ( IGNNs ) containing L graph convolution layers 1 ≤ l ≤ L , each of which can express a convolution operator g with any spectral representation ˆ g ( l ) : [0 , 2] → R d ( l ) . An IGNN layer therefore has the structure

We compare this to S 2 GNNs with ℓ = ( m +1) L layers for m ≥ 1 , in the additive form of Eq. 1,

Each layer 1 ≤ l ≤ ℓ parametrizes a spatio-spectral convolution. The spectral part satisfies Eq. 3,

with embeddings H ( l ) ∈ R n × d ( l ) , linear feature transforms f ( l ) θ := W ( l ) spec ∈ R d ( l ) × d ( l -1) and a spectral filter ˆ g ( l ) ϑ : [0 , 2] → R that is fully supported and a universal approximator on [0 , λ cut ] . Note we assume here that in every layer, there is only one spectral filter which gets reshaped as to act on every hidden component, whereas in practice, we relax this assumption to different filters per component, which can only be more expressive. The spatial part is a polynomial filter of the form

with W ( l ) spat ∈ R d ( l ) × d ( l -1) , polynomial order p (fixed across layers), and a spectral representation ˆ g ( l ) γ ( λ ) = ∑ p j =0 γ ( l ) j λ j with coefficients γ ( l ) = ( γ ( l ) 0 , . . . , γ ( l ) p ) ⊤ ∈ R p +1 . We note that theorem 4 extends immediately to the case of directed graphs if the spatial part is instead a polynomial of the magnetic Laplacian (see section 3.2.3) over complex-valued embeddings like in Zhang et al. (2021).

Note that the layer-wise hidden dimensions D ( l ) vs. d ( l ) of the IGNN vs. S 2 GNN do not have to agree except at the input layer, d (0) = D (0) (of course, both networks receive the same input H (0) = H (0) = X ), and at the output layer, d ( ℓ ) = D ( L ) . We now state the general version of Theorem 4.

Theorem (Theorem 4, general) . Assume an L -layer IGNN with filters ˆ g ( l ) such that ˆ g ( l ) ∣ ∣ [ λ cut , 2] ∈ C r [ λ cut , 2] and ∥ ∥ ∥ d r dλ r ˆ g ( l ) ∣ ∣ [ λ cut , 2] ∥ ∥ ∥ ∞ ≤ K max r ( λ cut ) for all 1 ≤ l ≤ L . Let ∥ ˆ g ( l ) ∥ ∞ ≤ ∥ ˆ g ∥ max ∞ and ∥ W ( l ) ∥ 2 ≤ ∥ W ∥ max 2 for all 1 ≤ l ≤ L . Assume that σ = [ · ] ≥ is the ReLu function. Then,

(1) For a fixed polynomial order p ≥ 2 , an approximating sequence ( S 2 GNN m ) m ∈ N of [( m +1) L ] -layer S 2 GNNs exists such that, for arbitrary graph sequences ( G m ) m ∈ N ,

̸

with a leading-order scaling constant that depends only on r . Here, ( · ) G m denotes the instantiation of all model filters on the eigenvalues of an input graph G m , which maps both models onto a G m -dependent function R D (0) → R D ( L ) .

(2) For fixed m ≥ 1 , an approximating sequence ( S 2 GNN p ) p ∈ N of [( m +1) L ] -layer S 2 GNNs with increasing layer-wise polynomial order p exists such that, for all ( G p ) p ∈ N , the same bound holds.

Proof. We first prove the following lemma, narrowing down the previous theorem to a single layer.

Lemma 1. Let IGNN ( l ) denote a single IGNN layer as in Eq. 20, with a filter ˆ g ( l ) such that ˆ g ( l ) ∣ ∣ [ λ cut , 2] is r -times continuously differentiable on [ λ cut , 2] and satisfies a bound K r ( ˆ g ( l ) , λ cut ) ≥ 0 , ∣ ∣ d r dλ r ˆ g ( l ) ( λ ) ∣ ∣ ≤ K r ( ˆ g ( l ) , λ cut ) ∀ λ ∈ [ λ cut , 2] . Let σ = [ · ] ≥ be the ReLu function, and let ∥ W ( l ) ∥ 2 denote the spectral norm of W ( l ) . Then,

(1) For fixed polynomial order p ≥ 2 , an approximating sequence ( S 2 GNN ( l ) m ) m ∈ N of ( m +1) -layer S 2 GNNs exists such that, for arbitrary graph sequences ( G m ) m ∈ N ,

̸

with a scaling constant that depends only on r . Here, ( · ) G m denotes the instantiation of all model filters on the eigenvalues of an input graph G m , which maps both models onto a G m -dependent function R D ( l -1) → R D ( l ) .

(2) For fixed m ≥ 1 , an approximating sequence ( S 2 GNN ( l ) p ) p ∈ N of ( m + 1) -layer S 2 GNNs with increasing layer-wise polynomial order p exists such that, for all ( G p ) p ∈ N , the same bound holds.

Remark. The proof of the simplified Theorem 4 used in the main body is analogous to the proof of Lemma 1 just without the nonlinearity, which has the following consequences:

The final layer m +1 which we only need to apply one last nonlinearity to the output (since the spectral part of all layers, including the previous layer m , has none) becomes obsolete, so the final layer instead becomes m ,

The two limits (1) and (2) are equivalent by the reduction to an mp -order polynomial filter,

Wedo not need the dimension-doubling 'trick' outlined below to get rid of the nonlinearity in the proof and instead set all feature transform matrices in layers 1 through m -1 to the identity and the final ones to W ∗ ( m ) spec = W ( l ) , W ∗ ( m ) spat = W ( l ) .

Proof of Lemma 1. We first note that m S 2 GNN spatial parts, each of order p , would act like an ( mp ) -order polynomial filter (factorized into m orderp polynomials), were it not for the nonlinearities in between. However, using the fact that σ is the ReLu function, we can choose intermediate hidden dimensions twice the size of the input dimension and then use the linear transforms to store a positive and a negative copy of the embeddings, add them back together after applying each ReLu, just to split the result back into a positive and negative copy for the next layer. This essentially gets us rid of σ . Throughout the proof, we use a star superscript to denote the specific parameters that will ultimately satisfy our bound, whereas we put no star above parameters that are yet to be fixed in a later part of the proof.

For m ≥ 2 , the trick discussed above works if we set

In the case m = 1 , pick the matrices W ∗ (1) spec , W ∗ (1) spat from above for the first, and the matrices W ∗ ( m +1) spec , W ∗ ( m +1) spat from above for the second layer.

Set ˆ g ∗ ( m +1) γ ( λ ) = 1 and ˆ g ∗ ( m +1) ϑ ( λ ) = 0 . Given these choices and a graph G with eigenvalues λ ,

We see that ˆ g spsp ∣ ∣ [ λ max , 2] = ∏ m j =1 ˆ g ( j ) γ since ˆ g ( j ) ϑ ∣ ∣ [ λ max , 2] = 0 for 1 ≤ j ≤ m . This can express any polynomial up to order mp on [ λ max , 2] , since we assumed a layer-wise p ≥ 2 and any polynomial with real coefficients factorizes into real-coefficient polynomials of degree less or equal to 2 by the fundamental theorem of algebra. On the interval [0 , λ max ] , on the other hand, the filter ˆ g spsp ∣ ∣ [0 ,λ max ] can express any IGNN filter ˆ g ( l ) ∣ ∣ [0 ,λ max ] . For m = 1 , this is immediately clear. Else, set ˆ g ( j ) ϑ to constants C j ∈ R ≥ , 1 ≤ j ≤ m -1 large enough that none of the polynomials ( C j +ˆ g ( j ) γ ) ,

1 ≤ j ≤ m -1 , has a zero in [0 , λ max ] . Defining ˆ g ( m ) ϑ = ˆ g ( l ) ∣ ∣ [0 ,λ max ] ∏ j m =1 ( C j +ˆ g ( j ) γ ) -ˆ g ( m ) γ ∣ ∣ [0 ,λ max ] gives the desired function.

We proceed by making use of a result by D. Jackson (Natanson, 1964), which is essentially a converse to Theorem 9 which we used to prove Theorem 3:

Theorem (Jackson's theorem on an interval) . Let a < b ∈ R , k, r ∈ N with k ≥ r -1 ≥ 0 , f ∈ C r [ a, b ] . Then, a polynomial p k of degree less or equal to k exists such that

Since ˆ g spsp can express any polynomial up to order mp on [ λ max , 2] and, for any such polynomial, find parameters for the spectral parts that match the ideal filter ˆ g ( l ) ∣ ∣ [0 ,λ max ] exactly (not contributing to the supremum error), we can directly transfer this theorem to our case. Define S 2 GNN ( l ) m from the lemma by setting the linear feature transforms and final-layer filters as above. For the filters in layers 1 through m , define γ ∗ (1) , . . . , γ ∗ ( m ) such that ∏ m j =1 ˆ g ∗ ( j ) γ factorizes into into the polynomial from Jackson's theorem on [ λ max , 2] , and ϑ ∗ (1) , . . . , ϑ ∗ ( m ) to match ˆ g ( l ) on [0 , λ max ] . This defines a filter ˆ g ( l ) spsp. We then find, for mp ≥ r -1 ≥ 0 ,

̸

Therefore, ∥ ˆ g ( l ) spsp -ˆ g ( l ) ∥ ∞ is of O ( K r (ˆ g, λ cut )( mp ) -r ) and we can find a scaling constant that depends only on r . Since the Lipschitz constant of σ is 1 , we find for any graph G with eigenvalues λ and any graph signal 0 = X ∈ R |G| ,

with a scaling constant that depends only on r . Exactly the same procedure and bounds hold if we instead keep m fixed and increase p . This finishes the proof of Lemma 1.

We can now prove the main theorem by induction. Lemma 1 gives the initial step. Now, assume the theorem holds for L IGNN layers. We can then choose ( S 2 GNN m ) m ∈ N =

ing Lemma 1, while S 2 GNN ( L ◦···◦ 1) m fulfill the induction assumption. We assume fixed p and increasing m , but the proof is fully analogous in the other case. Applying the same decomposition to ( IGNN m ) m ∈ N lets us express the error on a graph sequence ( G m ) m ∈ N as

We first used the triangle inequality in line 3 to split the difference into two terms. Next, we bound the first term using the induction assumption, as well as the Lipschitz constant of S 2 GNN ( L +1) m , which in turn, by Lemma 1, is the Lipschitz constant of IGNN ( L +1) m up to a term of O ( K max r ( λ cut ) ( pm ) -r ) . We moreover bound the second term using the Lipschitz constant of IGNN ( L ◦···◦ 1) m , as well as Lemma 1 to arrive at the final result.

H.5 Proof of Theorem 5

We next prove the stability of our positional encodings:

Theorem 5. The Positional Encodings PE in Eq. 5 are stable according to Definition 1.

Recall the definition of stability via Hölder continuity:

Definition 1 (Stable PE ) . (Huang et al., 2024) A PE method PE : R n × k × R k → R n × k is called stable, if there exist constants c, C > 0 , such that for any Laplacian L , L ′ , and P ∗ = arg min P ∥ L -PL ′ P ⊤ ∥ F

For this proof, we build on the work of Huang et al. (2024) where the authors show that under the assumptions of Definition 2, and some minor adjustments, a positional encoding of the following form Eq. 23 is stable (Theorem 10).

Definition 2. The key assumptions for SPE are as follows:

ϕ ℓ and ρ are permutation equivariant.

ϕ ℓ is K ℓ -Lipschitz continuous: for any λ , λ ′ ∈ R k , ∥ ϕ ℓ ( λ ) -ϕ ℓ ( λ ′ ) ∥ F ≤ K ℓ ∥ λ -λ ′ ∥ .

ρ is J-Lipschitz continuous: for any [ B 1 , B 2 , . . . , B k ] ∈ R n × n × k and [ B ′ 1 , B ′ 2 , . . . , B ′ k ] ∈ R n × n × k , ∥ ρ ( B 1 , B 2 , . . . , B k ) -ρ ( B ′ 1 , B ′ 2 , . . . , B ′ k ) ∥ F ≤ J ∑ k l =1 ∥ B ℓ -B ′ ℓ ∥ F .

Theorem 10 (Stability of Eq. 23 by Huang et al. (2024)) . Under Definition 2, SPE (Eq. 23) is stable with respect to the input Laplacian: for Laplacians L , L ′ ,

where the constants are α 1 = 2 J ∑ k l =1 K ℓ , α 2 = 4 √ 2 J ∑ k l =1 M ℓ , and α 3 = J ∑ k l =1 K ℓ . Here M ℓ = sup λ ∈ [0 , 2] k ∥ ϕ ℓ ( λ ) ∥ and again P ∗ = arg min P ∈ Π( n ) ∥ ∥ L -P ∗ LP ⊤ ∗ ∥ ∥ F . The eigengap γ = λ k +1 -λ k is the difference between the ( k +1) -th and k -th smallest eigenvalues, and γ = + ∞ if k = n .

We prove a similar bound for general weighted adjacency matrices A ∈ R n × n ≥ 0 (note that such a stability result would be trivial if we restrict A ∈ { 0 , 1 } n × n , since any function on a finite set is Lipschitz continuous). To achieve this, we need a technical assumption in order to ensure that the function values do not blow up and degree normalization is indeed a Lipschitz continuous function: We assume that the domain of A is restricted to (symmetric) matrices whose degrees are uniformly bounded by some constants 0 < ˜ D min < ˜ D max :

To decompose the proof into smaller pieces we commonly use the well-known fact that the composition of Lipschitz continuous functions f 1 ◦ f 2 , with constants C 1 and C 2 , is also Lipschitz continuous ∥ f 1 ( f 2 ( y )) -f 1 ( f 2 ( x )) ∥ ≤ C 1 C 2 ∥ y -x ∥ with constant C 1 C 2 .

Proof. Our proposed encoding (Eq. 5) matches roughly Eq. 23. Specifically, ϕ ℓ ( λ ) = softmax( ( λ j -λ ) ⊙ ( λ j -λ ) / σ 2 ) with σ ∈ R > 0 . However, ρ ℓ ( B 1 , B 2 , . . . , B k ) does not directly match || k j =1 [ B j ⊙ A ] · ⃗ 1 , since it is also a function of the adjacency A . Nevertheless, we show that ϕ ℓ is K ℓ -Lipschitz continuous and ρ is J -Lipschitz continuous, where we also bound the change of A .

We will start with ϕ ℓ ( λ ) = softmax( ( λ j -λ ) ⊙ ( λ j -λ ) / σ 2 ) . The softmax is well-known to be of Lipschitz constant 1 w.r.t. the L 2 vector norm/Frobenius norm. -x / σ has a Lipschitz constant of 1 /σ . This leaves us with the quadratic term ψ u ( λ ) = ( λ u -λ ) ⊙ ( λ u -λ ) where we bound the norm of the Jacobian

that is zero everywhere except for the diagonal entries, excluding its u -th entry. Thus, ∥ J ψ u ∥ F ≤ 2 k max v ∈{ 1 , 2 ,...,k } ( λ v -λ u ) ≤ 2 k ( λ k -λ 1 ) ≤ 4 k , as 0 = λ 1 ≤ λ k ≤ 2 . We can therefore use K ℓ := 4 k/σ .

Nowwecontinue with ˜ ρ ℓ ( A , B 1 , B 2 , . . . , B k ) . For f ( A , B ) = ( B ⊙ A ) · ⃗ 1 with a general weighted adjacency A ∈ R n × n , we consider

(A) holds by Cauchy-Schwarz, (B) by triangle inequality, (C) by Cauchy-Schwarz, (D) follows from the domain of A , and (E) is true since the largest eigenvalue of B = V ϕ ℓ ( λ ) V ⊤ is 1 because ϕ ℓ ( λ ) j ≤ 1 , ∀ 1 ≤ j ≤ k .

To further bound (F), i.e. ∥ A -A ′ ∥ F , note that ∥ L -L ′ ∥ F = ∥ D -1 / 2 AD -1 / 2 -D ′-1 / 2 A ′ D ′-1 / 2 ∥ F . For g ( A ) := D 1 / 2 AD 1 / 2 , our initial assumption from Eq. 25 yields the existence of a Lipschitz constant C ˜ D min , ˜ D max for g , which can be verified by computing the partial derivatives of g . Thus, we can bound

As concatenation of k vectors || k j =1 x has a Lipschitz constant of 1, we have J = √ n ˜ D max . Moreover, we have an additional term for the RHS of Eq. 24 with constant α 4 √ nk , coming from (F) and Eq. 28.

To finalize the proof, we restate the beginning of the proof of Huang et al. (2024) and incorporate the additional A -dependency of ˜ ρ ℓ ( A , B 1 , B 2 , . . . , B k ) with B j = V diag ( ϕ j ( λ )) V ⊤ for 1 ≤ j ≤ k .

Including the extra term stemming from our A -dependent ˜ ρ ℓ ( A , B 1 , B 2 , . . . , B k ) , the stability guarantee reads

with the newly introduced α 4 arising as Lipschitz constant of (inverse) degree normalization. The proof is complete.

Windowing for 'eigengap' independent bounds. Note that C depends on the eigengap between 1 / λ k +1 -λ k at the frequency cutoff. One should be able to improve upon this bound with windowing (see Fig. 7)), effectively lowering the Lipschitz constant of ˆ h j ( λ ) around λ k . We leave a formal treatment of this insight to future work.

H.6 Proof of Theorem 6

We next prove the expressivity of a GNN/S 2 GNN in combination with our positional encodings: Theorem 6. S 2 GNNs are strictly more expressive than 1-WL with the PE of Eq. 5.

For this, we assume that the positional encodings are the only node attributes, subsuming a constant feature or that there is a linear transformation on the raw features. We require that the choice of spatial MPGNN / spectral filter is at least as expressive as the 1-WL test, which is the case, e.g., for GIN. Moreover, we assume that the node-level embeddings are aggregated to the graph level using summation.

Proof. To show that GNN(PE( V , λ )) is strictly more expressive as 1-WL. For all graphs that 1WL can distinguish, the GNN may learn to ignore the PE . Thus, we only need to prove that the positional encodings/node features of PE( V , λ ) suffice to distinguish some graphs that 1-WL could

not distinguish. For all graphs that 1-WL can distinguish we know, by assumption, that the GNN can distinguish the graphs.

As Li et al. (2020) point out, 1-WL (and MPGNN that are as capable as 1-WL) cannot distinguish degree-regular graphs with the same number of nodes and degrees. A degree regular graph is a graph where each node has the same degree. This is closely related to Theorem 11.

We next show that our PE alone distinguishes certain degree-regular graphs. In this construction, we consider all 3-regular graphs with n = 8 nodes for this (see Fig. 15). The encodings ⃗ 1 PE result in the following values with σ = 0 . 001 and rounded to max 2 decimal places:

By constructing examples, this shows that our PE can distinguish 4 out of the 5 3-regular graphs with 8 nodes. Thus, our PE may distinguish at least some graphs that 1-WL cannot. This concludes the proof.

I Expressivity of Spectral Filters and Spectrally Designed Spatial Filters

While it is well-known that common spatial MPGNNs are at most as expressive as 1-WL and that spectrally designed GNNs can be more expressive than 1-WL (Theorem 2 of Balcilar et al. (2021a)), we show that spectral GNNs are not able to distinguish degree-regular graphs. This upper bound was

not known/formalized prior to our work (Bo et al., 2023b). Fortunately, our PE largely mitigates the limitation. The improved expressivity of our positional encodings, along with their efficiency, stems from the element-wise product with A (see also Geerts (2021)).

Theorem 11. Spectral filters V diag(ˆ g ( λ )) V ⊤ ⃗ 1 are strictly less expressive than 3-WL with Laplacian L = D -A , L = I -D -1 A , or L = I -D -1 / 2 AD -1 / 2 .

Proof. The proof relies on properties of the eigenvectors for the different choices L u = D -A , L rw = I -D -1 A , or L s = I -D -1 / 2 AD -1 / 2 . For L u ⃗ 1 = λ 0 ⃗ 1 = 0 and L rw ⃗ 1 = λ 0 ⃗ 1 = 0 the first eigenvector is constant. The first eigenvector of L s is D 1 / 2 ⃗ 1 (ignoring normalization). Thus, for degree-regular graphs, the first eigenvector of L s is also constant.

̸

By the orthogonality of eigenvectors, v u ⊥ v v if u = v , we know that all other eigenvectors are orthogonal to constant node features. Consequently, the 'Fourier transformed' node features are V ⊤ ⃗ 1 = [ √ n 0 . . . 0 ] for all three choices L u, L rw, and L s . Since this is true for all degreeregular graphs, spectral GNNs cannot distinguish degree-regular graphs with the same number of nodes.

Since the 3-WL test can distinguish some degree-regular graphs, 3-WL is strictly more expressive than a spectral GNN.

Corollary 1. 'Spectrally designed' MPGNNs that use a polynomial parametrization of filter diag(ˆ g ( λ )) are strictly less expressive than 3-WL with the same choices for L .

Proof. With a polynomial parametrization of the spectral filter ˆ g ( λ ) , we know V (ˆ g ( λ ) ⊙ [ V ⊤ x ]) = V diag(ˆ g ( λ )) V ⊤ x = ˆ g ( L ) x = ∑ p j =0 γ j L j x (see § 2). Due to this equivalence between a spectral and spatial filter and the constant node features x = ⃗ 1 , any polynomial filter ∑ p j =0 γ j L j ⃗ 1 cannot distinguish degree-regular graphs. This argument also holds if the polynomial filter is normalized by the maximum eigenvalue as done by ChebNet (Defferrard et al., 2017).

J Further Remarks on S 2 GNNs

Wenext provide insights, details, and remarks on the details and variants of S 2 GNNs, accompanying the main section § 3. The structure roughly follows the main body.

Next to the overview in Fig. 2 and the method description of the main part, we provide pseudocode in Algo. 1 for a Spatio-Spectral Graph Neural Network on a graph with node attributes, and in Algo. 2 for a spectral filter.

Algorithm 1 Spatio-Spectral Graph Neural Network (S 2 GNN), implementing Eq. 1

1: Input: Adjacency A ∈ R , , node attributes X ∈ R , number of eigenvectors k

n × n ≥ 0 n × d (0) 2: V , λ ← EVD( L ( A ) , k ) 3: H (0) ← X +PE( V , λ ) 4: for l ∈ { 1 , 2 , . . . , ℓ } do 5: H ( l ) ← Spectral ( l ) ( H ( l -1) ; V , λ ) + Spatial ( l ) ( H ( l -1) ; A ) 6: Return H ( ℓ )

Algorithm 2 Real-valued spectral filter of Eq. 4

1: Input: Node embeddings H ( l -1) ∈ R n × d ( l -1) , eigenvalues λ ∈ [0 , 2] k , eigenvectors V ∈ R n × k

2: ˆ g ( l ) ϑ ( λ ) ← Smearing( λ ) W ⊙ Window( λ )

4: ˆ H ( l ) ← s ( l ) ζ (ˆ g ( l ) ϑ ( λ ) ⊙ ˆ H ( l -1) )

3: ˆ H ( l -1) ← V ⊤ H ( l -1)

5: H ( l ) ← V ˆ H ( l )

6: Return H ( l )

(a)

(b)

(c)

(d)

(e)

A1)

十

A2)

+

A3)

+

(4)

十

A1)

十

入2)

十

A3）

十

入4)

+

A1)

十

A2)

+

A3)

+

（4)

十

A1)

+

入2

十

A3）

+

入4)

十

A1)

+

+

二

A3

+

A4

十

J.1 Visualization of Spectral Filters

In Fig. 16, we provide further examples of hierarchies/eigenspaces spectral filters have access to, complementing Fig. 3 & 4. Here and in the main part, we use the main diagonal of ∑ j s.t. λ j = λ u v j v ⊤ j for deciding on the edge weights of the graph structures, potentially summing over multiple eigenvectors with identical values λ j = λ u . Wetake the product ∏ j s.t. λ j = λ u sign( v j ) for visualizing the sign of the n edges for the global aggregation.

For all graphs, the first eigenvector denotes the constant signal (for L = D -A ). For (a-d), we observe that the second eigenspace roughly describes a half oscillation, i.e., the left vs. right part of the graph. The third eigenspace separates the middle parts. For (a), the fourth eigenspace models the interactions between the extremal nodes. For (b-d), the frequency increments again, effectively clustering the graph in four roughly equal pieces. For (e), the eigenspaces model the interplay between (automorphic) inner and outer structures, as well as the vertical and horizontal symmetry.

J.2 Composition of Filters

Composing a residual connection with a graph filter G = diag(ˆ g ( λ )) ∈ R n × n yields Y = V GV ⊤ H + H = V ( G + I ) V ⊤ H , chaining multiple filters (without nonlinearities) results in V G 2 V ⊤ V G 1 V ⊤ H = V G 2 G 1 V ⊤ H . Chaining and residual connections resolve to V ( G 2 G 1 + G 2 + G 1 + I ) V ⊤ H . Hence, an arbitrary sequence of graph filters (Eq. 2) can be more flexible due to the interactions between filters. Note that this composition is only true in the absence of nonlinearities. Nevertheless, the main intuition about how filters interact remains approximately the same also in the light of nonlinearities.

J.3 Exhaustive Reasons Why Low Frequencies Are Sensible

A sensible default is to focus on the low frequencies. We specifically identify the following six reasons: (1) Low frequencies model the smoothest global signals w.r.t. the high-level graph structure (see Fig. 3 & 4). (2) Gama et al. (2020) find that, under a relative perturbation model (perturbation budget proportional to connectivity), stability implies C -integral-Lipschitzness ( ∃ C > 0: | λ d ˆ g / dλ | ≤ C ), i.e., the filter can vary arbitrarily around zero but must level out towards larger λ . Stability to graph perturbations is a strong domain-agnostic prior. (3) Many physical long-range interactions are power laws with a flattening frequency response. For example, we construct an explicit graph filter modeling the electric potential of charges in a 1D 'ion crystal' (§ G) and find that a lowpass window is optimal. (4) Sequence models like Hyena (Poli et al., 2023) apply global low-pass filters through their exponential windows. (5) Cai et al. (2023) prove that an MPGNN plus virtual node (see § E) can emulate DeepSets (Zaheer et al., 2017) and, thus, approximate self-attention to any precision. Nonetheless, we find that a virtual node alone does not necessarily yield good generalization (§ 3.1.1 & 4.1). (6) Nonlinearities 'spill' features between frequency bands (Gama et al., 2020). This includes spillage from higher frequencies to the band of the spectral filter. Gama et al. (2020) argue that this spillage makes it possible to learn stable yet expressive graph filters and is also a feature of stable message passing models.

J.4 Scaling to Graphs of Different Magnitude

For scaling a single to graphs of different orders of magnitude, it can be beneficial to rescale the eigenvalues before learning the filter ˆ g ϑ ( λ ) . That is, we use ˆ g ( l ) ϑ ( ˜ λ ) with rescaled ˜ λ .

For example, the eigenvalues for a path/sequence are λ j ≈ (1 -cos( πj / n )) . Thus, the resolution is poor, especially for the eigenvalues close to zero since cos approaches slope 0. For this reason, we consider rescaling the eigenvalues with

or

The latter is, e.g., convenient for identifying the second lowest eigenvalue regardless of n . Due to the poor numerical properties of these relations, we evaluate cos -1 (1 -λ j ) = tan -1 ( √ 2 λ j -λ 2 j / 1 -λ j ) instead.

J.5 Spectral Normalization

While the GFT and its inverse preserve the norm of the input (e.g., ∥ ˆ x ∥ 2 = ∥ V ⊤ x ∥ 2 = ∥ x ∥ 2 ), this is not true if operating on a truncated frequency spectrum or if the filter ˆ g ϑ ( λ ) suppresses certain frequencies. For example, in the example of a virtual node (for simplicity here with L = D -A ), a signal x that is zero at every node but one at a single node, then the signal will be equally scattered to every frequency. Then, suppressing all frequencies but λ = 0 , yields ∥ V 1 { 0 } V ⊤ x ∥ 2 = 1 / √ n .

Motivated by this unfortunate scaling, we also consider normalization in the spectral domain. Specifically, we normalize ˆ H = ˆ g ϑ ( λ ) ⊙ [ V ⊤ f θ ( H ) ] ∈ R k × d s.t. ˆ H j ← (1 -a j ) ˆ H j + a j ˆ H j / ∥ ˆ H j ∥ 2 with learnable a ∈ [0 , 1] d . This allows, e.g., broadcasting a signal from one node without impacting its scale. However, we empirically find that this normalization only helps only marginally in the over-smoothing experiment (Di Giovanni et al., 2023a) and otherwise can destabilize training. We also consider variants where the norm in the spectral domain is scaled with the norm of the signal

in the spatial domain with more or less identical results. We hypothesize that such normalization is counter-productive for, e.g., a bandpass filter if the signal does not contain the corresponding frequencies.

J.6 Adjusting S 2 GNNs to Directed Graphs

For the spectral filter of Eq. 3, we use f ( l ) θ ( ˆ H ( l ) ) = H ( l ) ⊙ [ σ ( H ( l ) W ( l ) G, ℜ ) + i · σ ( H ( l ) W ( l ) G, ℑ )] and subsequently map the result of Spectral back the real domain, e.g., using w ( l ) ℜ ℜ (Spectral ( l ) ( H ( l -1) )) + w ( l ) ℑ ℑ (Spectral ( l ) ( H ( l -1) )) , with learnable weights w ( l ) ℜ , w ( l ) ℑ ∈ R d and real ℜ ( · ) as well as imaginary component ℑ ( · ) . For the positional encodings PE( V , λ ) of § 3.2.4, we use A s in Eq. 5 and concatenate real as well as imaginary components. The neural network for the spectral domain s ζ of § 3.2.2 generalizes without adjustment. Similar to Koke & Cremers (2024), one could also employ complex weights; however, we do not.

J.7 Computational Remarks

We use readily available eigensolvers ( scipy ) and, thus, use a fixed number of eigenvectors (typically k ≪ n ) instead of determining k based on λ cut. The partial eigendecomposition is of complexity O ( km ) for m edges, while the spectral filter has complexity O ( kdn ) . On a different remark, we batch multiple graphs using block diagonal matrices (Fig. 17).

Spectral graph-level readouts. The key insight is that frequencies are a global concept, and hence, the GFT can be used for global readouts in graph-level tasks. With k ≪ n , such a readout is practically free in the presence of intermediate spectral layers and of O ( kn ) otherwise. Thus, there is the opportunity for a computationally convenient aggregation of global information, including a sort of graph-level 'jumping knowledge' (Xu et al., 2018). The only caveat is that the Fourier coefficients are not unique due to the ambiguity in the eigendecomposition. To maintain permutation equivariance, we take









































































































































A

1

A

2

. . .

A

b

0

0

∑

i

ni

∑

i

ni









































































































































V

1

V

2

. . .

V

b

0

0

∑

i

k i

∑

i

ni

the absolute value and aggregate over dimension k in Eq. 4 instead of the multiplication with V . We observe that such intermediate readout can improve performance slightly, e.g., on TPUGraphs. However, we leave a systematic evaluation of its benefits for future work.

Linear bottle necks. To circumvent overfitting, we commonly replace the linear transformations WX in f ( l ) θ ( ˆ H ( l ) ) and ˆ g ϑ ( λ ) with low-rank bottlenecks W 2 W 1 X , s.t. W ∈ R d × d , W 2 ∈ R d × d ′ , W 1 ∈ R d ′ × d , and d ′ < d .

K Limitations

We expect that many common graph benchmarks do not have or only insignificant long-range interactions. We observe that MPGNNs are less likely to overfit, perhaps since locality is a good inductive bias in many circumstances (Bronstein et al., 2021). Moreover, we observe that the spectral filter (§ 3.2.1) may converge slowly and get stuck in local optima. We find that a sufficient amount of randomly initialized filters mitigates this issue to a large extent. Further, one can introduce inductive biases via windowing functions (Fig. 7), like the exponential window used by Hyena (Poli et al., 2023).

Even if the true causal model generating the target consists of long-range interactions, it might be sufficient to model the training data solely using (potentially spurious) local interactions. This might be especially true if the training nodes are samples from a 'small' vicinity of the graph (e.g., OGB Products (Hu et al., 2020)).

Closely related to the previous point is the amount of available training data. We hypothesize that S 2 GNNs are more data-hungry than their purely spatial counterpart. That is, to reliably detect (nonspurious) long-range interactions in the training data, a sufficient amount of data is required. Similar findings have been made, e.g., in the image domain (Dosovitskiy et al., 2021).

Except for heterophilic graphs, direction plays a small role in graph machine learning even though many benchmark tasks actually consist of directed graphs (Rossi et al., 2023). Moreover, there is a lack of benchmarks involving directed graphs, which require long-range interactions. Note that most of the theoretical findings generalize to directed graphs under appropriate modeling decisions/assumptions. However, we do not make this discussion explicit since MPGNNs for directed graphs are still actively researched.

Since a lot of the previous points hover around the insufficiency of the available benchmarks, we propose two new tasks § 4.1 and derive further datasets, e.g., for associative recall.

While we demonstrate the practicality of S 2 GNNs in § 4.3 on large-scale benchmarks, the partial eigendecomposition EVD starts to become costly on the largest graphs we use for evaluation. Even though we did not experiment with lowering the requested precision, etc., we expect that for scaling further, naïve approaches might not be sufficient. One direction could be to utilize GPUs instead of CPUs or to adapt concepts, e.g., from spectral clustering (von Luxburg, 2007).

Even though there are many important reasons why we should utilize a spectral filter on the low end of the spectrum, there might be tasks for which this choice is suboptimal. One way to estimate the frequency band to which one should apply a spectral filter is via a polynomial regression and then determine where the derivative is maximal. Note that it is efficient to calculate the eigenvectors around an arbitrary location of the spectrum, e.g., with the 'shift-invert mode' of scipy / ARPACK (Lehoucq et al., 1998).

Due to the many possible design decisions of spectrally parametrized filters, the neglect of spectral filters in prior work, and the lack of appropriate benchmarks, it was not possible to ablate all the details. We expect that future work will discuss the specific building blocks in greater detail.

L Broader Impact

We expect that S 2 GNNs will have similar societal implications as other model developments like Convolutional Neural Networks (CNNs) (LeCun et al., 1989), LSTMs (Hochreiter & Urgen Schmidhuber, 1997), transformers (Vaswani et al., 2017), or modern Graph Neural Networks (Gilmer et al., 2017). Since such models may be used as building blocks in architectures for predictive tasks, generative modeling, etc., they have a wide range of positive and negative implications. Nevertheless, we expect that S 2 GNNs will not have more negative implications than other machine learning model innovations.

M Experimental Results

This section provides further details on the experimental setup (§ M.1), the computational cost (§ M.3), and graph constructions with additional experimental results for the clustering tasks (§ M.6); likewise we provide details for the distance regression (§ M.7), arXiv-year (§ M.8), and provide nodes on the graph construction in TPUGraphs (§ M.10). Note that the sections on clustering (§ M.6) and distance regression (§ M.7) also contain ablations and further insights.

M.1 Experimental Details

Implementation. The code base is derived from Cao et al. (2023), which on the other hand derive the code of Rampášek et al. (2022). The implementation heavily relies on PyTorch geometric (Fey &Lenssen, 2019).

Datasets. We collect the main statistics, including licenses, for the datasets in Table 6. The provided code will download all datasets along with the experiment execution, except for TPUGraphs, where one should follow the official instructions. Due to the high variation in results, we merge all 'layout' datasets and present the results on this joint dataset. We use the fixed public splits for all experiments and proceed accordingly for our datasets (see § M.6 and § M.7).

Hyperparameters. While we provide full parameters for all experiments and models in our code, we gather an overview of the used S 2 GNNs variants here. The parameters were determined through cascades of random search throughout the development of the method. We list the most important parameters in Table 7.

Usage of external results. The performance of baselines is commonly taken from leaderboards and the respective accompanying papers. This specifically includes the results in Table 1, Table 2, and Table 11.

Setup. For clustering (§ M.6), distance regression (§ M.7), and arXiv-year (§ M.8) we report the detailed setup in the respective sections. For the other tasks, the relevant details are:

Peptides: We follow the setup and implementation of Rampášek et al. (2022). That is, we train for 250 epochs with a batch size of 200. We rerun experiments on 10 random seeds.

Over-squashing: We derive the setup from Di Giovanni et al. (2023a). In the main part (Fig. 5), for the GCN, we report the numbers of their Figure 3 for a GCN on 'Clique Path' graphs. For the spectral filter, we actually consider the more challenging setting where we do not train one model per graph size. Instead, we train one model for all sequence lengths. The task is to retrieve the correct of five possible classes on the other end of the graph. In the extended experiment of Fig. 12, we compose the dataset of 'Clique Path' and 'Ring' graphs (see Di Giovanni et al. (2023a)). To avoid m = O ( n 2 ) , we limit the fully connected clique to 15 nodes. For training and validation, we enumerate all graphs with even n ∈ { 4 , 6 , . . . , 50 } and train for 500 epochs. For test, we enumerate the graphs with even n ∈ { 52 , 54 , . . . , 100 } . We rerun experiments on 10 random seeds.

Associative recall: We construct one dataset consisting of key-value sequences of length 20 to 999. As Poli et al. (2023), we use a vocabulary of 30. We sample 25,000/500 random graphs for train/validation. For the test set, we randomly generate 500 graphs for the sequence lengths of 1,000 to 1,199. We train for 200 epochs. In the experiment with validation/test sequence length 30k (Table 2), we generate 10,000 training graphs of length 29,500 to 30,499 and finetune S 2 GNNsGCN from the smaller setup. We rerun experiments on 10 random seeds.

OGB Products: Even though full-graph training with 3 layers GCN plus one spectral layer fits into a 40 GB A100 GPU, we find that batched training works better. We randomly divide the graph during training into 16 parts and train a 6-layer S 2 GAT with spectral layers after the second and last message passing step. Inference is performed on the entire graph at once. We rerun experiments on 5 random seeds.

TPUGraphs: This is the only dataset where we use a transformer to model ˆ g instead of the procedure detailed in § 3.2.1. We fix the number of eigenvectors to k = 100 and do not apply any windowing. Due to the large variation of results, we merge all 'layout' tasks into a single dataset. Since the default graph construction is not able to express all relevant information, we adapt it as detailed in § M.10, however, the empirical impact was small. TPUGraphs 'layout' consists of a few hundred distinct graph structures with a large variation on the node-level configuration/features. We sample 10,000 configurations for each graph structure of each 'layout' sub-split. Here, we introduce two batch dimensions: (1) batching over multiple graphs and (2) batching over the configurations. In each training step of the 1,000 epochs, we sample a small subset of configurations per graph structure and apply a pairwise hinge loss to rank the configurations. We do not perform random reruns due to the computational cost.

M.2 Qualitative Experiments

In Fig. 6 and Fig. 8, we provide qualitative insights about the approximation of filters and ringing.

In Fig. 6, we construct a true filter by adding a discontinuous filter at λ = 0 and a polynomial filter of order 3. For the spectral part, we use the true filter values and fit a Chebyshev polynomial on the remaining part. We then plot the response of the true filter and its approximations on a path graph with 21 nodes and L = I -D -1 A .

Similarly, in Fig. 8, we use a path graph with 100 nodes and L = I -D -1 A . We then construct a perfect low pass ( k = 25 ) and approximate a rectangular wave.

M.3 Computational Cost

We report the computational cost for the experiments in Table 7 for a single random seed. On top of the pure cost of reproducing our numbers, we conducted hyperparameter searches using random search. Partially, we required 100s of runs to determine good parameter ranges. A generally wellworking approach was first to reproduce the results of the best available MPGNN in prior work. Thereafter, we needed to assess how likely additional capacity would lead to overfitting. Usually, we reduced the number of message-passing steps, added the spectral filter, and determined appropriate values for the number of eigenvectors k . In the light of overfitting, it is a good idea to lower the number of Gaussians in the smearing of the filter parametrization (§ 3.2.1), introduce bottle-neck layers (§ J), and use fewer spectral filters than hidden dimensions.

Runtime with precalculated eigenvectors. In Fig. 18, we contrast the runtime cost of a spectral convolution with spatial messages passing on ogb-arXiv (170k nodes) of Hu et al. (2020), using an Nvidia GTX 1080Ti. This essentially compares a sparse matrix multiplication (adjacency matrix)

10

1000

2000

3000

4000

5000

Number of eigenvectors

k

100

200

Runtime / ms

2

4

6

Number message passing steps

GCN

Spec.

with matrix multiplications on dense "tall and skinny" matrices (GFT). we find that one GCN-layer here is as costly as a spectral filter with approx. k = 2 , 500 eigenvectors.

Large-scale benchmarks. On the large-scale datasets OGB Products and TPUGraphs, we perform full-graph training (without, e.g., segment training (Cao et al., 2023)) using 3 DirGCN layers interlayered with spectral filters targeting a pair-wise hinge loss. The spectral GNN uses the Magnetic Laplacian to incorporate direction. The spatial MPGNN closely resembles the model of Rossi et al. (2023), except that we half the dimension for the forward and backward message passing and concatenate the result. We shrink the dimensions to model the direction at a very low cost. We conclude that S 2 GNNs can be very practical even if applied at scale and can effectively model long-range interactions also on large graphs.

Dense

Sparse

0

2000

Number of nodes

n

10

2

10

1

10 0

10 1

Time cost / s

(a)

scipy

(CPU)

0

10000

Number of nodes

n

10

2

10

1

10 0

10 1

Time cost / s

(b)

PyTorch

(GPU)

Eigendecompositon. We show the computational cost for the eigendecomposition of a random Erd˝ os Rényi graph (every edge has equal likelihood to be drawn). We use scipy (CPU) and PyTorch (GPU) with default arguments. For the sparse decomposition with PyTorch , we use the svd_lowrank method. Note that the default parameters for PyTorch are usually leading to large numerical errors. Fig. 19 demonstrates that the cost of the eigendecomposition is manageable. For large graphs like ogbn-products (2.5 mio. nodes), the EVD takes around 30 minutes with k = 100 on 6 CPU cores of an AMD EPYC 7542. Note that the default parameters of the eigensolver allow for 1000s of iterations or until the error in the 32-bit float representation achieves machine precision.

M.4 S 2 GNN Aggregation Ablation

In the main body we present two ways to combine a spatial and spectral filter: An additive combination (Eq. 1) and an arbitrary sequence of filters (Eq. 2). In this section, we perform an ablation analysis on the peptides-func benchmark and report the results in Table 8.

Instead of summation of the spatial and spectral parts, concatenation is another possible option. Getting input features H ( l -1) ∈ R n × d ( l -1) , we design Spectral ( l ) and Spatial ( l ) to map to R n × d ( l ) / 2 and update the embeddings as

Additionally, we consider normalization of the addends at the end of each embedding update, dividing by 1 / √ 2 (concatenation w/ residual) and 1 / √ 3 (summation w/ residual) as an attempt to keep the variance constant.

Inspired by recent advancements in state space models like Mamba (Gu & Dao, 2023), we also consider modeling an update step in a similar way, identifying the convolutional part with Spatial ( l ) and the SSM part with Spectral ( l ) .

The following table shows results for the different design choices to combine the spatial and spectral parts, with all hyperparameters being precisely the ones reported in Table 7 for peptides-func.

M.5 Number of Eigenvectors Ablation on Peptides-Func

In Fig. 20, we ablate the number of eigenvectors on the real-world dataset peptides-func. Since we here limit the number of eigenvalues by cut-off frequency λ cut, we report the average number of eigenvectors.

0.00

0.25

0.50

0.75

λ

cut

0.66

0.68

0.70

0.72

0.74

Average Precision (AP)

test

val

40

60

80

Average number of eigenvectors

k

0.66

0.68

0.70

0.72

0.74

Average Precision (AP)

test

val

M.6 Clustering Tasks

We use a clustering task LR-CLUSTER based on Gaussian Mixture Models (GMMs), which requires long-range interactions to measure the ability of S 2 GNN to spread information within clusters and consider the original CLUSTER task from Dwivedi et al. (2023) based on Stochastic Block Models (SBMs) in order to measure the ability to discriminate between the clusters. The differences are apparent from an illustration of some exemplary graphs in Fig. 21 & 22. While LR-CLUSTER has long-range interactions, the challenge of CLUSTER is to discriminate between the clusters. Without the arrangement of nodes, colors, and different edge weights, for CLUSTER , it is virtually impossible to discriminate the clusters by visual inspection.

(a)

(b)

(c)

(a)

(b)

(c)

Nevertheless, we find that the spectral filter is well aligned with the cluster structure in these tasks. We plot this some exemplary filter in Fig. 23. The findings match the explanations of § 4.1 also for CLUSTER .

(a)

A

V

diag(ˆ

gϑ

(

λ

)

:

,

1)

V

/latticetop

V

diag(ˆ

gϑ

(

λ

)

:

,

2)

V

/latticetop

V

diag(ˆ

gϑ

(

λ

)

:

,

3)

V

/latticetop

V

diag(ˆ

gϑ

(

λ

)

:

,

4)

V

/latticetop

(b)

A

V

diag(ˆ

gϑ

(

λ

)

:

,

1)

V

/latticetop

V

diag(ˆ

gϑ

(

λ

)

:

,

2)

V

/latticetop

V

diag(ˆ

gϑ

(

λ

)

:

,

3)

V

/latticetop

V

diag(ˆ

gϑ

(

λ

)

:

,

4)

V

/latticetop

In the remainder of the section, we provide full details of the experiment setups. Moreover, we provide additional results not presented in the main text, including ablations.

M.6.1 GMMClustering LR-CLUSTER

Setup. To sample an input graph, we start by generating C = 6 p -dimensional cluster centers µ c ∼ U [0 , 10] p for c ∈ { 0 , . . . , C -1 } (we use p = 2 ). Next, we draw n c ∈ { 100 , . . . , 199 } points x ic ∼ N ( µ c , 4 I p ) which will represent the nodes of the graph. Subsequently, we update the class memberships such that every point is in its most likely class according to the underlying probabilistic model. Finally, we connect each node v to its e v ∼ U ( { 1 , . . . , 10 } ) closest neighbors by Euclidean distance ∥ · ∥ 2 . This whole procedure is repeated until the generated graph is connected. We then discard the location information and only keep the graph structure. In this way, we generate graphs of an average diameter of ≈ 33 . See Fig. 21 for depictions of example graphs.

Apart from the graph generation procedure, we adhere closely to Dwivedi et al. (2023): We introduce input features in { 0 , 1 , 2 , . . . , C } , where a feature value of c = 1 , . . . , C corresponds to the node

being in class c -1 and a feature value of 0 means that the class is unknown and has to be inferred by the model. Only one node v c per class is randomly chosen to be labeled and all remaining node features are set to 0 . The output labels are defined as the class labels. We use weighted cross entropy loss for training and class-size-weighted accuracy as a target metric. We generate 10,000 training and 1,000 val/test graphs each and report the average ± standard deviation over 3 random reruns.

Models. As an underlying spatial model baseline, we use a vanilla GCN (Kipf & Welling, 2017). We compare this to S 2 GCN, only applying one spectral convolution immediately before the last spatial layer. We investigate the influence of the number k ∈ { 0 , 1 , . . . , 10 } of eigenvectors to be taken into account with 4 spatial layers, with k = 0 indicating the absence of a spectral layer (see Fig. 10a, and Table 9 for the underlying data). We also vary the number of spatial MP layers from 2 to 10 and compare the performance of a purely spatial GCN to the corresponding S 2 GCN with one spectral convolution (see Fig. 10b, and Table 10 for the underlying data).

Throughout all evaluations, we maintain a consistent hyperparameter configuration: Specifically, we use an inner dimension of 128, GELU (Hendrycks & Gimpel, 2016) as an activation function, no dropout, and residual connections for all spatial and spectral layers. For the spectral layer, we implement the gating mechanism f ( l ) θ , but abstain from a neural network in the spectral domain (§ 3.2.2), bottlenecks, or parameter sharing. We train for 50 epochs with a batch size of 50 , using the AdamW optimizer (Loshchilov & Hutter, 2019) with a base learning rate of 0 . 003 , a weight decay of 0 . 0001 , a cosine scheduler and 5 warmup epochs.

Further discussion. The clustering task comes naturally to S 2 GCN, as a spectral layer can simulate certain variations of spectral clustering (von Luxburg, 2007): Suppose H ( l -1) ∈ R n × C is a one-hot encoding of the cluster labels, i.e. H ( l -1) v,c = δ v,v c , with c ∈ { 1 , . . . , C } and v c being the unique labeled node per class. In its simplest form, taking ˆ g ( l ) ϑ ( λ ) ≡ 1 and f ( l ) θ ≡ id , the spectral layer Spectral ( l ) from Eq. 3 turns into H ( l ) = V V ⊤ H ( l -1) . Hence, H ( l ) v,c = V ⊤ v, : V v c , : encodes a notion of similarity between a node v and each labeled node v c . This relates to the Euclidean distance ∥ V v, : -V v c , : ∥ 2 = √ ∥ V v, : ∥ 2 2 + ∥ V v c , : ∥ 2 2 -2 V ⊤ v, : V v c , : which is more typically used for spectral clustering.

M.6.2 SBM Clustering CLUSTER (Dwivedi et al., 2023)

Setup. We conduct an ablation study on the original CLUSTER task (Dwivedi et al., 2023), which uses a similar setup to our GMM clustering task, however drawing from a SBM instead: For each cluster, n c ∈ { 5 , . . . , 35 } nodes are sampled. Nodes in the same community are connected with a probability of p = 0 . 55 , while nodes in different communities are connected with a probability of q = 0 . 25 . While there is no need for long-range interactions in this task, considering that the average diameter of the graphs is just ≈ 2 . 17 , separating the clusters is much harder than in the GMM clustering task (see Fig. 23 for example adjacency matrices from the SBM and GMM models). We use weighted cross entropy loss for training and class-size-weighted accuracy as a target metric. We report the average ± standard deviation over 3 random reruns.

Models. In our ablation study, we consider GCN (Kipf & Welling, 2017), GAT (Veliˇ ckovi´ c et al., 2018), and GatedGCN (Bresson & Laurent, 2018) as MPGNN baselines, following a setup similar to Dwivedi et al. (2023). We consider models with 4 layers (roughly 100k parameters) and 16 layers (roughly 500k parameters), while keeping most hyperparameters the same as in the benchmark, including inner dimension, dropout, and the number of heads for GAT. However, our reported baseline results and parameter counts differ slightly as we are using a different post-MP head, where we maintain a constant dimension until the last layer, in contrast to Dwivedi et al. (2023) who progressively shrink the inner dimension. We construct the corresponding S 2 GNNs by modifying each baseline model, replacing the 3 rd and the 5 th / 15 th layers with spectral layers, ensuring a roughly equivalent parameter count. Additionally, each model is optionally supplemented by our positional encodings PE (§ 3.2.4).

We further conduct a hyperparameter search on the most promising base MPGNN candidate, GatedGCN, which leads to an optimized version of S 2 GNN. This optimized model has 18 spatial MPGNN layers, spectral layers between all spatial layers, and additional RWSE encodings. The inner dimension is adjusted to keep the model well below a parameter budget of 500k. Finally, we also evaluate S 2 GCN and S 2 GAT using these hyperparameter settings.

Throughout all evaluations, we use GELU (Hendrycks & Gimpel, 2016) as an activation function, residual connections for all spatial and spectral layers, and implement the gating mechanism f ( l ) θ without employing a neural network in the spectral domain. We use a batch size of 128 for training the 4-layer models and 64 for all other models. For

4 layers

16 layers

0.5

0.6

0.7

0.8

Test accuracy

GCN

4 layers

16 layers

0.5

0.6

0.7

0.8

Test accuracy

GAT

4 layers

16 layers

0.5

0.6

0.7

0.8

Test accuracy

GatedGCN

Base MPGNN

S 2

GNN

Base MPGNN (

+PE

)

S 2

GNN (

+PE

)

the spectral layer, we use the partial eigendecomposition corresponding to the lowest k = 50 eigenvalues ( k = 100 for the optimized S 2 GNNversions), spectral normalization, and λ cut = 1 . 3 . For the optimized models, we employ parameter sharing with 128 heads, and a bottleneck of 0.25 in feature gating. We use 8 attention heads for all GAT versions in accordance with Dwivedi et al. (2023) (in-

ner dimension is not expanded but split up), except for the optimized version, which uses 4 heads. For the purely spatial models, we use p = 0 . 0 as dropout (similar to Dwivedi et al. (2023)). We observe this to lead to overfitting for models with spectral layers, for which we set p ∈ { 0 . 1 , 0 . 2 } . Hyperparameters differing between the compared models are listed in Table 12. We train for 100 epochs using the AdamW optimizer (Loshchilov & Hutter, 2019) with a base learning rate of 0.001, no weight decay, and a cosine scheduler with 5 warmup epochs.

Results. Results for the CLUSTER task are presented in Table 12, Table 11 and Fig. 24. Introducing a spectral layer significantly enhances performance on the 4-layer architectures, both with and without positional encodings. The effect is most pronounced on GCN, where replacing just a single GCN layer by a spectral layer boosts accuracy from 0 . 504 to 0 . 655 . Notably, introducing two spectral layers still has a consistent positive effect on all 16-layer architectures.

M.7 Distance Regression

Setup. We generate directed random trees with one source by sampling trees with n ∈ { 500 , . . . , 999 } nodes, picking one node at random to declare as a source and introducing edge directions accordingly. To construct random DAGs with long distances, we start from such directed random trees and proceed by adding ⌊ n/ 10 ⌋ edges at random, choosing each edge direction such that the resulting graph is still a DAG. Additionally, we mark the source node with a node feature. Besides evaluating all models in an in-distribution regime, we also assess the generalization power of the methods by drawing out-of-distribution val/test splits from slightly larger graphs of n ∈ { 1000 , . . . , 1099 } and n ∈ { 1100 , . . . , 1199 } nodes each. We use L 2 loss for training and R 2 as a target metric. We sample 50,000 training and 2,500 val/test graphs each and report the average ± standard deviation over 3 random reruns.

0

50

100

Epoch

0.2

0.4

0.6

0.8

4 layers

0

50

100

Epoch

0.2

0.4

0.6

0.8

0

50

100

Epoch

0.2

0.4

0.6

0.8

0

50

100

Epoch

0.2

0.4

0.6

0.8

GCN

16 layers

GCN

16 layers

0

50

100

Epoch

0.2

0.4

0.6

0.8

GAT

0

50

100

Epoch

0.2

0.4

0.6

0.8

GatedGCN

Models. As a MPGNN baseline, we use a five-layer directed version of GCN, DirGCN (Rossi et al., 2023), with three post-message-passing layers, and concatenating instead of averaging over the source-to-target and target-to-source parts. We compare these baselines to S 2 DirGCN of the form Eq. 2 with four spectral layers, alternating spatial and spectral convolutions and employing residual connections. We benchmark versions of S 2 DirGCN that ignore edge direction in the spectral convolution against directed versions in which we set q = 0 . 001 . In all cases, we use the partial eigendecomposition corresponding to the k = 50 lowest eigenvalues. All models are optionally enriched by the positional encodings from § 3.2.4. Throughout all evaluations, we use an inner dimension of 236, GELU (Hendrycks & Gimpel, 2016) as an activation function, and dropout p = 0 . 05 .

( l )

For the spectral layers, we utilize the gating mechanism f θ , not employing a neural network in the spectral domain, we use spectral normalization, λ cut = 0 . 1 , and a bottleneck of 0 . 03 in the spectral layer. We train for 50 epochs, using a batch size of 36 and the AdamW optimizer (Loshchilov & Hutter, 2019) with a base learning rate of 0.001, a weight decay of 0.008, and a cosine scheduler with 5 warmup epochs.

Results. In Table 13, we show the performance of the different models on DAGs and trees. We observe that the simple MPGNNs are notably surpassed by all versions of S 2 DirGCN. While S 2 DirGCN achieves nearly perfect predictions on the tree tasks in both the directed and undirected case, the undirected version is outperformed by the directed version on the DAG tasks. Here, performance also reduces slightly in the out-of-distribution regime. The great performance on the tree

0

20

40

Ground truth

0

5

10

15

RMSE

0

20

40

Ground truth

0

5

10

15

RMSE

0

50

Ground truth

0

10

20

30

RMSE

0

50

Ground truth

0

10

20

30

RMSE

0

20

40

Ground truth

0

20

40

Prediction

DAGs (in-sample)

0

20

40

Ground truth

0

20

40

Prediction

DAGs (out of sample)

0

50

Ground truth

0

20

40

60

Prediction

Trees (in-sample)

0

50

Ground truth

0

20

40

60

Prediction

Trees (out of sample)

DirGCN

DirGCN (

+PE

)

S 2

DirGCN (

+PE

, undir.)

S 2

DirGCN (

+PE

, direc.)

task is due to the fact that trees are collision-free graphs (Geisler et al., 2023), where the phase of each eigenvector is exp( i 2 πq ( d v + c )) for each node v , with d v representing the distance to the source node and c ∈ R being an arbitrary constant (due to phase invariance of the eigenvector). It is noteworthy that a simple MPGNN with positional encodings, despite having the distances (shifted by c ) readily available, fails the task, as the information about the phase of the source node cannot be effectively shared among all nodes. In Fig. 26, we compare the distance predictions by the different models. While the prediction of all models is close to perfect below a distance of 5, the spatial MPGNNs are almost unable to distinguish higher distances. By contrast, S 2 DirGCN predicts reasonable distances regardless of the ground truth, with the absolute error only increasing slowly.

M.8 Heterophilic arXiv-year (Lim et al., 2021)

Setup. We evaluate S 2 GNN on a large-scale heterophilic dataset, namely arXiv-year . arXiv-year is based on OGB arXiv (Hu et al., 2020), but instead of paper subject areas, the year of publication (divided into 5 classes) is the prediction target. While there are no long-range interactions in this dataset, preliminary experiments indicated that the phase of the Magnetic Laplacian eigenvectors on its own can also be predictive of the class label. We report average ± standard deviation over 5 reruns with the splits from Lim et al. (2021), using a different random seed for each run.

Models. We use DirGCN (Rossi et al., 2023) as a baseline and largely follow the original setup. However, we observe that using 4 layers (instead of 6) and introducing a dropout of p = 0 . 5 improves baseline performance. Furthermore, we drop the jumping knowledge used by Rossi et al. (2023). We compare this baseline to S 2 DirGCN with two spectral layers (after the second and third spatial layers) and apply residual connections only for the spectral layers. For the spectral layers, we set q = 0 . 0001 and use the partial eigendecomposition with k = 100 , a NN in the spectral domain § 3.2.2, no feature gating, and a bottleneck of 0 . 05 . All other parameters are kept similar to the DirGCN base of Rossi et al. (2023). We train for 2000 epochs using the AdamW optimizer (Loshchilov & Hutter, 2019) with a base learing rate of 0.005, no weight decay, and a cosine scheduler with 50 warmup epochs.

Results. We report the results in Table 14. Notably, our S 2 DirGCN outperforms both our baseline DirGCN as well as the recent FaberNet (Koke & Cremers, 2024), albeit by a very tight margin. However, we found hyperparameter optimizations to be quite noisy, and as such, the resulting performance metrics should be interpreted cautiously. A more comprehensive evaluation of S 2 GNN's power on heterophilic datasets, potentially with long-range interactions, is left for future work.

M.9 Large-Scale PCQM4Mv2 (Hu et al., 2021)

We show that S 2 GNNs are very parameter efficient. Even though we only conduct a very rudimentary hyperparameter search, S 2 GNNskeep up with state of the art approaches. Specifically, we adapt the hyperparameters from peptides-func and achieve comparable performance to the state of the art (excluding external data) with about 3-20% of the number of parameters.

M.10 TPUGraphs Graph Construction

The 'XLA' collections of TPUGraphs contain many constructs that are most certainly suboptimal for a machine-learning-based runtime prediction. However, in our preliminary experiments, we could not show that our graph construction yielded better results in a statistically significant manner. Nevertheless, we include this discussion since it might be insightful.

To understand the challenges with the default graph construction, note that in the TPUGraphs dataset each node represents an operation in the compuational graph of Accelerated Linear Algebra (XLA). Its incoming edges are the respective operands, and the outgoing edges signal where the operation's result is used. Thus, the graph describes how the tensors are being transformed. An (perhaps unnecessary) challenge for machine learning models arises from using tuple , which represents a sequence of tensors of arbitrary shapes. In this case, the model needs to reason how the tuple is constructed, converted, and unpacked again. Moreover, directly adjacent tensors/operations can be very far away in the graphs of TPUGraphs.

We identified and manually 'fixed' three cases to eliminate this problem largely in the TPUGraphs dataset: Tuple-GetTupleElement, While, and Conditional. Since we could not access the configurations in the HLO protobuf files and C++ XLA extraction code, we decided to perform these optimizations ourselves. However, it might be a better strategy to utilize the existing XLA compiler etc.

Additionally, to the subsequently described graph structure changes, we extract the order of operands from the HLO protobud files. Outgoing edges are assumed to be unordered except for the GetTupleElement operation, where the tuple index is used as order. Moreover, we extracted all features masked in the C++ code and then excluded constant features.

M.10.1 Tuple-GetTupleElement

The dataset contains aggregations via the XLA Tuple operation that are often directly followed by a GetTupleElement operation. To a large extent, these constructs are required for the subsequently discussed While and Conditional operations. Importantly, the model could not extract the relationships through a tuple aggregation since the tuple_index was not included in the default features. Moreover, the resulting tuple hub nodes severely impact the Fourier basis of the graphs (see § 2). We illustrate the graph simplification in Fig. 27 and denote the edge order of incoming edges from top to bottom. The edge order represents the order of operands.

We propose dropping immediate Tuple-GetTupleElement constructs and directly connecting predecessors and successors. For this, we generate a graph solely consisting of direct connections and then resolve multiple consecutive Tuple-GetTupleElement constructs via a graph traversal (depthfirst search).

We perform the Tuple-GetTupleElement simplification after dealing with While and Conditionals. However, for the sake of simplicity, we will avoid using tuples in the subsequent explanations for While and Conditional. In other words, the subsequent explanations extend to functions with multiple arguments via the use of tuples.

M.10.2 While Operation

The While operation has the signature While(condition, body, init) where condition is a function given the init or output of the body function. Note that in the TPUGraph construction, body as well as condition only represent the outputs of the respective function and their operands need to be extracted from HLO.

To avoid hub nodes and to retain the dataflow between operations (important for decisions about the layout), operands and outputs are connected directly. Technically, we am modeling a do-while construct because the condition is not connected to the inputs. Since the successors of the while are of type GetTupleElement, they relabeled to a new node type, signaling the end of a while loop. To support nested while loops, each node in the body is assigned a new node feature signaling the number of while body statements it is part of.

M.10.3 Conditional Operation

Conditional(branch_index, branch_computations, branch_operands) is the most common signature of the Conditional operation, where the integer-valued branch_index selects which branch_computations is executed with the respective input in branch_operands . Similarly to the While operation, we introduce new node types for the inputs of computations and the successors (they are GetTupleElement operations).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We discuss the shortcomings of prior MPGNNs and that our new framework S 2 GNN takes a new approach to overcome the limitations. We detail our theoretical analysis/justification, outline the yet largely unexplored design space of S 2 GNN, and summarize the experimental evaluation.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide approximation-theoretic error bound in § 3.1.2 that also gives light on the limitations. Moreover, we discuss important considerations of S 2 GNNs throughout the method section § 3. Additionally, we elaborately list and summarize the limitations in § K.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The theoretical analysis of § 3, § 3.1.1, § 3.1.2, § 3.2.4, and § I state important assumptions in the main text. All theorems, definitions, formulas, and proofs are numerated. Due to the page limit, we do not provide proof sketches in the main body. Nevertheless, we provide accompanying sketches and illustrations to convey the main intuition. The proofs, full assumptions, and details are given in § H (& § I).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe our S 2 GNNs in length in § 3 and detail the experimental setup in § 4. Additional relevant details are provided in § M.1. We provide code, including configuration, at https://www.cs.cit.tum.de/daml/s2gnn .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will open source fully automized scripts for all datasets that download and preprocess the data prior to the execution of the configured experiment (except TPUGraphs (Phothilimthana et al., 2023) due to its dedicated access). This includes our own datasets on clustering, distance prediction, and associative recall. We provide code at https://www.cs.cit.tum.de/daml/s2gnn . Moreover, we provide reasonable details for reproduction in § M.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the setup and details to a reasonable detail in § 4 and § M.1. For additional details, we refer to the provided code, including configuration, at https://www.cs.cit.tum.de/daml/s2gnn .

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report mean, standard deviation, and details on randomization for all relevant results on benchmark datasets. For qualitative insights and the large-scale TPUGraphs dataset, we solely provide mean estimates. The collective experimental results underline the claims and are strong evidence against the statistical insignificance of our results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide a reasonable summary of used resources in § M.1 and specifically Table 7. Here, we list the hardware accelerator used and the runtime of the experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: [Yes]

Guidelines: The proposed framework S 2 GNN does not pose an unusual risk on top of the common risks for research on machine learning architectures. Our evaluation does not include human subjects, participants, or data. We discuss the broader impact in § L. Thus, in summary, we conform to the ethics guidelines in every aspect.

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impact in § L. It should be noted that the proposed framework S 2 GNN does not pose an unusual risk on top of the common risks for research on machine learning architectures. We discuss the broader impact in § L.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The proposed framework S 2 GNN does not pose an unusual risk on top of the common risks for research on machine learning architectures. Our evaluation does not include human subjects, participants, or data.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all origins of the datasets, even if we derive the benchmark from someone else. We provide an overview of the datasets, including the licenses in Table 6. We provide code, including configuration, at https://www.cs.cit.tum.de/daml/s2gnn . It also contains an (anonymized) license for its usage and discloses the provenance of the project setup.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code and configuration for the dataset on GMM clustering, distance regression, and associative recall will be provided. The generation of all resources takes around one hour and uses solely CPUs. We provide code, including configuration, at https://www.cs.cit.tum.de/daml/s2gnn . The datasets are described in § M, § M.6, and § M.7.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work neither involves crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work neither involves crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper neither involves crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors are proposing Spatio-Spectral Graph Neural Networks ($S^2GNNs$), a hybrid model that combines (Spatial) Message Passing Neural Networks (MPNNs) and Spectral Graph Neural Networks. The authors argue that, by combining message-passing with spectral filters, the model can better model both local and global information, alleviating phenomena such as over-squashing and under-reaching. The authors discuss the design space of their proposed method and showcase a technique for obtaining position encodings that can make their model more expressive than the 1-WL test. The authors perform multiple experiments on both synthetic and real-world datasets.\n\nSTRENGTHS:\n- The paper is well-written, and the figures and tables are aesthetically pleasing.\n- The tackled problem is well-motivated.\n- The method is relatively easy to comprehend. The idea is simple and elegant, yet it has very strong empirical results.\n- The proposed $S^2 GNN$ is analyzed very well. Including mechanistic in-context learning experiments is very interesting in the context of long-range modeling on graphs. Moreover, the connections between $S^2 GNNs$ and SSMs that the authors make are insightful and could be very interesting to some readers.\n- The empirical results on the Peptides datasets are very good. $S^2 GNNs$ obtain state-of-the-art results on peptides-func *without* using Position Encodings, which is very hard to accomplish even with models that consider the graph as being complete (such as Graph Transformers). Moreover, the proposed method uses significantly fewer parameters than other competing models.\n- The theoretical analysis is well thought out. The authors also make some remarks about the advantages of having spectral filters, and they provide some very useful intuitions that connect them to using virtual nodes (Fig 3, Fig 4).\n- The authors introduce two new datasets - a long-range clustering dataset (\\textsc{LR-CLUSTER}) and a long-range regression dataset that is then used to make several observations about their model - such as the ability to solve long-range tasks and the alignment between the clusters and the learned spectral filters.\n- The authors also propose a way of obtaining some Laplacian Position Encodings that improve the overall performance and make their model more expressive than the 1-WL test.\n- The paper contains an extensive amount of supplementary materials detailing other experiments and proofs for the theoretical results.\n\nWEAKNESSES:\n- The paper's main weaknesses are that the main text is somewhat unfocused, and some related work is not mentioned. I will detail in the following:\n\t- While the authors generally do a very good job of providing valuable theoretical insights regarding their design choices, I would argue that some of them are unnecessary and distract the reader from the main points of the paper. For instance, I believe the discussion about \"high-resolution filters for low frequencies\" from lines 149-167 to be somewhat hand-wavy and distracting, especially the remark that \"as graph problems discretize continuous phenomena, stability to the graph perturbations is a strong domain-agnostic prior\". While I don't think that the discussion is bad, I feel that the supplementary materials would be a much better place in which to include it. \n\t- I believe that Subsection 3.3 (Parametrizing Spectral Filters) is somewhat confusing, especially for a reader who is not very familiar with spectral methods. I did not initially understand the discussion about the window (Lines 247-253), and I believe it should contain more context. The authors should either expand the section or move it to the supplementary materials.\n\t- The proposed positional encodings are a nice addition, but similar laplacian positional encodings, such as [[1]], have not been discussed as related work. Other works, such as [[2]], are cited in the paper but not in the context of the laplacian positional encodings that they propose.\n\t- Some other recent methods deal with long-range relationships on graphs and have not been discussed in the related work. Notably, GRED [[3]] uses Linear Recurrent Units to model distant relationships with a mechanism similar to k-hop MPNNs and also draws inspiration from SSMs; AMP [[4]] has a message filtering mechanism and is adaptively increasing the number of layers depending on the dataset; NBA [[5]] is passing messages in a non-backtracking manner, obtaining a slightly better score than $S^2 GCN$ on *peptides-struct* ($0.2424$ v.s. $0.2447$), and PR-MPNN [[6]] rewires graphs adaptively by learning a probability distribution for the adjacency matrices.\n- The authors mention on line 69 that the runtime of their method is equivalent to that of an MPNN, but I could not find a table containing a direct comparison. It would be great if this were included somewhere. There are training times on page 35, Table 6, but no comparison with a simple MPNN (such as a GCN or a GIN model).\n\n\nOverall, I believe that the paper is very strong. The model is conceptually simple but contains novel ideas. The empirical results are very good, and the authors test their method on many different scenarios. I found the mechanistic in-context learning experiments to be particularly interesting. I believe the work could have a considerable impact on how we design GNN architectures going forward.\n\nAs stated previously, the main weakness is that the paper feels somewhat unfocused, and some important related work is omitted. This is a somewhat strange complaint to have, but I feel that the scope of the paper is very broad. In my opinion, the overall style of the paper would be better suited for a journal submission if some details from the appendix were moved into the main text. For a conference with more limited space, I would advise the authors to move some of the more in-depth details to the appendix. This is in no way an argument for rejection, I just believe that the paper would be a lot nicer to read if it didn't contain these details in the main text.\n\nI recommend a 7 (Accept), but I will update my score to an 8 (Strong Accept) if the authors expand their related work discussion and provide more details for Subsection 3.3. I would also recommend to the authors that they move some of the details from Section 3 to the Appendix, but not doing so will not affect my final score. \n\n\n\n[1]: https://arxiv.org/abs/2110.07875\n[2]: https://arxiv.org/pdf/2012.09699\n[3]: https://arxiv.org/pdf/2312.01538\n[4]: https://arxiv.org/pdf/2312.16560\n[5]: https://arxiv.org/pdf/2310.07430\n[6]: https://arxiv.org/abs/2310.02156\n\nQUESTIONS:\n- In the graph rewiring literature, the models are often benchmarked on other graph-level datasets (such as QM9, Zinc, or the TUDataset collection). Did the authors try their method on these datasets? I am not requesting these experiments, but they would be a nice addition.\n- The Long-Range Graph Benchmark also contains COCO-SP, PascalVOC-SP, and PCQM-Contact. Did the authors try to benchmark their models on these datasets? I'm personally not a fan of COCO-SP and PascalVOC-SP because they're obtained from Computer Vision datasets, so the GNN inductive bias might not be appropriate for them. Still, PCQM-Contact is a molecular dataset, and it would be interesting to see some results on it.\n\nLIMITATIONS:\n- The limitations are addressed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Spatio-Spectral Graph Neural Networks (S²GNNs) to address the limitations of ℓ-step Spatial Message Passing Graph Neural Networks (MPGNNs), such as limited receptive fields and over-squashing. S²GNNs combine spatial and spectral graph filters for efficient global information propagation, offering tighter error bounds and greater expressiveness than MPGNNs. They outperform traditional MPGNNs, graph transformers, and graph rewirings on benchmarks and scale efficiently to millions of nodes.\n\nSTRENGTHS:\n1. Combining spectral and spatial methods is interesting. The proposed method can overcome many problems associated with MPNNs.\n2. The theoretical proofs in this paper are sufficient and, in my opinion, sound.\n3. This paper offers a lot of intuitive analysis, with examples that help readers understand the concepts.\n4. S²GNN performs excellently in multiple tasks and can be extended to various applications. It also demonstrates scalability, running efficiently on graphs with millions of nodes.\n\nWEAKNESSES:\n1. This paper's writing and structure can be further improved. Adding descriptions to the notation and including pseudocode would enhance readability.\n2. The experiment includes too few baselines. For instance, only GAT and GCN are used as baselines in Tables 3 and 4, while some of the latest methods, such as SpecFormer, which also uses Eigenvector, are not considered.\n\nQUESTIONS:\n1. Please refer to the weaknesses mentioned above.\n2. What is the upper bound of the graph size that the EVD used in S²GNN can support? For example, can it handle graphs with billions of nodes, such as papers100m?\n3. In real tasks, are the eigenvectors $k$ sensitive to performance? For instance, on products and TPU graphs, would using a larger $k$ improve performance?\n\nLIMITATIONS:\nThe authors discussed limitations in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Spatio-Spectral Graph Neural Networks (S$^2$GNNs), a novel paradigm that combines spatial and spectral parameters to overcome the limitations of $\\ell$-step MPGNNs, notably their restricted receptive fields and over-squashing issues. S$^2$GNNs achieve global information propagation efficiently, surpassing MPGNNs in approximation accuracy, expressivity, and scalability, demonstrated through superior performance on various graph tasks and ability to handle large-scale datasets.\n\nSTRENGTHS:\n1. The paper presents a rigorous theoretical basis and comprehensive analysis, reinforcing the validity of the proposed approach.\n\n2. The proposed method is straightforward and practical, with the added advantage of being easily integrable into a wide range of existing GNN models.\n\n3. The method has been extensively tested across various backbones and datasets of differing types and sizes, demonstrating its robustness and versatility.\n\nWEAKNESSES:\n1. **Deficiency in Ablation Studies for Embedding Updates**: Eq.~(1) within the paper that updates to the latent embeddings are achieved through the summation of spectral and spatial layers. This method warrants a broader range of ablation studies, such as exploring concatenation, normalization, or adopting a randomized approach to select between spectral or spatial optimizations like [1], to validate the robustness and efficiency of these embeddings.\n\n2. **Inadequate Comparative Analysis for Over-Squashing and Long-Range Interactions**: The paper's treatment of over-squashing and long-range interactions could be enhanced by incorporating comparative analyses with established methods. Notably, the GCNII[2], which utilizes skip connections and PageRank to facilitate up to 64 layers in a GNN, and methods based on random walks that intensify depth-first search capabilities for extended influence, should be examined. This inclusion would enhance the analysis and provide a more rigorous evaluation of the proposed model’s capabilities in handling long-range interactions.\n\n[1] Chang, Heng, et al. \"Not all low-pass filters are robust in graph convolutional networks.\" Advances in Neural Information Processing Systems 34 (2021): 25058-25071.\n[2] Chen, Ming, et al. \"Simple and deep graph convolutional networks.\" International conference on machine learning. PMLR, 2020.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The proposed method Spatio-Spectral Graph Neural Networks (S2GNNs) combine the spectral GCN and spatial GCNs embeddings linearly. The paper considers a deep dive into spectral filter properties and attempt to motivate the combination with spatial filtering. There are well known results repeated in the method section, which could have been part of background. Also, the contribution towards directed graphs is not clear.

STRENGTHS:
1.	The paper considers a deep dive into spectral filter properties and attempt to motivate the combination of spectral filters with spatial filtering. 
2.	The linear combination of spectral and spatial embeddings is new.

WEAKNESSES:
1.	There are well known results repeated in the method section. For example, it is established that the spectral filters are permutation equivariant, Locality relates to spectral smoothness.
2.	The paper sometimes fails to connect the dots. For example, MPNNs (spatial) suffer from oversquashing, but at the same time spectral GNNs can handle oversquashing issue. So why do we need to combine them both? Its not clear in Introduction.
3.	Contribution towards the directed graphs is unclear, any spectral filters can be adapted to directed graphs via magnetic Laplacian. The main point of the paper is to demonstrate the superiority against existing methods for undirected settings first.

QUESTIONS:
GNNs aim to learn data driven filters. “Ideal” filters may NOT be a “good” choice always. Then what is the need of implementing an ideal discontinuous spectral filter discussed in Section 3.2?

In the experiments, did you take the directionality into account in S2GNN? If, yes what is the performance of S2GNN without directionality?

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-14