SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nicolas Keriven
CNRS, IRISA, Rennes, France  
nicolas. keriven@cnrs.fr
CNRS, LJAD, Nice, France  
samuel.vaiter@cnrs.fr
We aim to deepen the theoretical understanding of Graph Neural Networks (GNNs) on large graphs, with a focus on their expressive power. Existing analyses relate this notion to the graph isomorphism problem, which is mostly relevant for graphs of small sizes, or studied graph classification or regression tasks, while prediction tasks on nodes are far more relevant on large graphs. Recently, several works showed that, on very general random graphs models, GNNs converge to certain functions as the number of nodes grows. In this paper, we provide a more complete and intuitive description of the function space generated by equivariant GNNs for node-tasks, through general notions of convergence that encompass several previous examples. We emphasize the role of input node features, and study the impact of node Positional Encodings (PEs), a recent line of work that has been shown to yield state-of-the-art results in practice. Through the study of several examples of PEs on large random graphs, we extend previously known universality results to significantly more general models. Our theoretical results hint at some normalization tricks, which is shown numerically to have a positive impact on GNN generalization on synthetic and real data. Our proofs contain new concentration inequalities of independent interest.
Machine learning on graphs with Graph Neural Networks (GNNs) [53, 5] is now a well-established domain, with application fields ranging from combinatorial optimization [6] to recommender systems [50, 11], physics [45, 1], chemistry [16], epidemiology [37], physical networks such as power grids [41], and many more. Despite this, there is still much that is not properly understood about GNNs, both empirically and theoretically, and their performances are not always consistent [52, 22], compared to simple baselines in some cases. It is generally admitted that a better theoretical understanding of GNNs, especially of their fundamental limitations, is necessary to design better models in the future.
Theoretical studies of GNNs have largely focused on their expressive power, kickstarted by a seminal study [54] that relates their ability to distinguish non-isomorphic graphs to the historical Weisfeiler-Lehman (WL) test [51]. Following this, many works have defined improved versions of GNNs to be "more powerful than WL" [34, 35, 26, 49, 38], often by augmenting GNNs with various features, or by implementing "higher-order" versions of the basic message-passing paradigm. Among the simplest and most effective idea to "augment" GNNs is the use of Positional Encodings (PE) as input to the GNN, inspired by the vocabulary of Transformers [48]. The idea is to equip nodes with carefully crafted input features that would help break some of the indeterminancy in the subsequent message-passing framework. In early works, unique and/or random node identifiers have been used [32, 47], but they technically break the permutation-invariance/equivalence - consistency with a reordering of the nodes in the graph - of the GNN. Most PEs in the current literature are based on eigenvectors of the adjacency matrix or Laplacian of the graph [12, 13] (with recent variants to handle the sign/basis indeterminancy
[31]), random-walks [13], node metrics [56, 30], or subgraphs [4]. Some of these have been shown to have an expressive power beyond WL [30, 4, 31].
In some contexts however, WL-based analyses have limitations: they pertain to tasks on graphs (e.g. graph classification or regression) and have limited to no connections to tasks on nodes; and they are mostly relevant for small-scale graphs, as medium or large graphs have a negligible chance of being exactly isomorphic to one another, but exhibit different characteristics (e.g. community structures) that might be useful for learning. At the other end of the spectrum, the properties of GNNs on large graphs have been analysed in the context of latent positions Random Graphs (RGs) [24, 25, 43, 44, 29, 2, 36], a family of models slightly more general than graphons [33]. Such statistical models of large graphs are classically used in graph theory [18, 9] to model various data such as epidemiological [27, 39], biological [17], social [20], or protein-protein interaction [19] networks, and are still an active area of research [9]. For GNNs, the use of such models has shed light on their stability to deformations of the model [44, 29, 24], expressive power [25], generalisation [15, 36], or some phenomena such as oversmoothing [23, 3]. One basic idea is that, as the number of nodes in a random graph grows, GNNs converge to "continuous" equivalents [24, 8], whose properties are somewhat easier to characterize than their discrete counterpart. As prediction tasks on nodes are far more common and relevant on large graphs modelled by random graphs, this paper will focus on permutation-equivariant GNNs, rather than permutation-invariant. In the limit, it has been shown that their output converges to functions over some latent space to label the nodes, but the descriptions of this space of functions and its properties are still very much incomplete. A partial answer was given in [25], in which some universality properties are given for specific models of GNNs, but for limited models of random graphs with no random edges, and specific models of GNNs that no not include node features or PEs. We will in particular extend some of their results to random edges, with the proper choice of PEs.
Contributions. In this paper, we significantly extend existing results by providing a complete description of the function space generated by permutation-equivariant GNNs (Theorem 1), in terms of simple stability rules, and show that it is equivalent to previous implicit definitions that were based on convergence bounds. We outline the role of the input node features, and particularly of Positional Encodings (PEs). We then study the several representative examples of PEs on large random graphs. In particular, we analyze SignNet [31] (eigenvector-based) PEs (Theorem 2), and distance-based PEs [30] (Theorem 3). We derive simple normalization rules that are necessary for convergence, and illustrate them on real data. Finally, our proofs contain new universality results for square-integrable functions and new concentration inequalities that are of independent interest. All technical proofs are available in the Appendix. The code to reproduce the figures can be found at https://github.com/nkeriven/random-graph-gnn.
2 Background on Random Graphs and Graph Neural Networks
Let us start with generic notations and definitions. The norm  $\| \cdot \|$  is the Euclidean norm for vectors and the operator norm for matrices and compact operators between Hilbert spaces. The latent space  $\mathcal{X}$  is a compact metric set with a probability distribution  $P$  over it. Square-integrable functions from  $\mathcal{X}$  to  $\mathbb{R}^q$  w.r.t.  $P$  are denoted by  $L_{q}^{2}$ , and are equipped with the Hilbertian norm  $\| f\|_{L^2}^2 \stackrel{\mathrm{def.}}{=} \int_{\mathcal{X}} \| f(x)\|^2 dP(x)$ . The (disjoint) union of multidimensional functions  $L_{\sqcup}^{2} \stackrel{\mathrm{def.}}{=} \bigcup_{q \in \mathbb{N}^*} L_{q}^{2}$  is a metric space for a metric defined as  $\| f - g\|_{L^2}$  if  $f, g \in L_{q}^{2}$  for some  $q$ , and 1 otherwise. Continuous Lipschitz functions between metric spaces  $\mathcal{X} \rightarrow \mathcal{Y}$  are denoted by  $\mathcal{C}_{\mathrm{Lip}}(\mathcal{X},\mathcal{Y})$ . For  $X = \{x_1,\ldots ,x_n\}$  where  $x_i \in \mathcal{X}$ , we define the sampling of  $f: \mathcal{X} \to \mathbb{R}^d$  as  $\iota_Xf = [f(x_i)]_{i=1}^n \in \mathbb{R}^{n\times d}$ . Given  $Z \in \mathbb{R}^{n\times d}$ , the Frobenius norm is  $\| Z\|_{\mathrm{F}}$  and we define the normalized Frobenius norm as  $\| Z\|_{\mathrm{MSE}} = n^{-\frac{1}{2}} \| Z\|_{\mathrm{F}}$ . The notation comes from the fact that  $\| \iota_X(f - f^\star)\|_{\mathrm{MSE}}^2 = n^{-1}\sum_i \| f(x_i) - f^\star(x_i)\|^2$  which is akin to a Mean Square Error.
Latent position Random Graphs. In this paper, we consider latent position random graphs [20, 28, 33], a family of models that includes Stochastic Block Models (SBM), graphons, random geometric graphs, and many other examples. They are the primary models used for the study of GNNs in the literature [24, 25, 29, 43]. We generate a graph  $G = (X,A,Z)$ , where  $X \in \mathbb{R}^{n \times d}$  are unobserved latent variables,  $A \in \{0,1\}^{n \times n}$  its symmetric adjacency matrix, and  $Z \in \mathbb{R}^{n \times p}$  are (optional) observed node features. The latent variables and adjacency matrix are generated as such:
$$
\forall i, x _ {i} \stackrel {i i d} {\sim} P, \quad \forall i <   j, a _ {i j} \sim \operatorname {B e r n o u l l i} \left(\alpha_ {n} w \left(x _ {i}, x _ {j}\right)\right) \quad \text {i n d e p e n d e n t l y} \tag {1}
$$
where  $w: \mathcal{X} \times \mathcal{X} \to [0,1]$  is a continuous connectivity kernel and  $\alpha_{n}$  is the sparsity-level of the graph, such that the expected degrees are in  $\mathcal{O}\left(n^{2}\alpha_{n}\right)$ . Non-dense graph can be obtained with  $\alpha_{n} = o(1)$ , here we will go down to the relatively sparse case  $\alpha_{n} \gtrsim (\log n) / n$ , a classical choice in the literature [28, 24]. Note that the continuity hypothesis of the kernel  $w$  is not really restrictive: neither  $\mathcal{X}$  nor the support of the distribution  $P$  need be connected. For instance, SBMs can be obtained by taking  $\mathcal{X}$  to be a finite set. We do not specify a model for the node features yet, see Sec. 4.
Graph shift matrix and operator. When the number of nodes  $n$  grows on random graphs, it is known that certain discrete operators associated to the graph converge to their continuous version, as well as the GNNs that employ them [24, 8]. Here, some of our results will be valid under quite general assumptions, hence we use generic notations for our graph representations. When the results are only valid for particular examples, this will be specifically expressed.
We consider a graph shift matrix [46]  $S = S(G) \in \mathbb{R}^{n \times n}$ , which can be either directly the adjacency matrix of the graph or various notions of graph Laplacians. We define an associated graph shift operator  $\mathbf{S}: L_{\square}^{2} \to L_{\square}^{2}$  such that the restriction  $\mathbf{S}|_{L_q^2}$  is a compact linear operator of  $L_{q}^{2}$  onto itself. Note that we reserve "matrix" and "operator" respectively for the discrete and continuous versions. The results in Sec. 3 will be valid under generic convergence assumptions from  $S$  to  $\mathbf{S}$ , while the results of Sec. 4 will focus on the following two representative examples.
Example 1 (Normalized adjacency matrix and kernel operator). Here  $S = \tilde{A} = (n\alpha_{n})^{-1}A$  and  $\mathbf{S}f = \mathbf{A}f = \int w(\cdot ,x)dP(x)$ . This choice requires to know, or estimate, the sparsity level  $\alpha_{n}$ . In this case, our results will hold whenever  $\alpha_{n}\gtrsim (\log n) / n$  with an arbitrary multiplicative constant.
Note that this choice requires to know (or estimate) the parameter  $\alpha_{n}$ , otherwise we will not have convergence between  $S$  and  $\mathbf{S}$ , which can be limiting. This is not the case of the next example.
Example 2 (Normalized Laplacian matrix<sup>1</sup> and operator). Here  $S = L = D_A^{-1/2} AD_A^{-1/2}$  where  $D_A = \mathrm{diag}(A1_n)$  is the degree matrix of  $G$ , and  $\mathbf{S}f = \mathbf{L}f = \int \frac{w(\cdot, x)}{\sqrt{d(\cdot)d(x)}} dP(x)$  where  $d(\cdot) = \int w(\cdot, x)dP(x)$ .
is the degree function. Whenever we opt for this choice, we assume that  $d_{\mathrm{min}} \stackrel{\mathrm{def.}}{=} \inf_{\mathcal{X}} d(x) > 0$ , and our results will hold whenever  $\alpha_n \geqslant C(\log n)/n$  with a multiplicative constant  $C$  that depends (in a non-trivial way) on  $w$ , see Thm. 9 in App. D.
To sometimes unify notations, when we adopt these examples, we define  $w_{\mathbf{S}}$  such that  $w_{\mathbf{S}}(x,y) = w(x,y)$  in the adjacency case and  $w_{\mathbf{S}}(x,y) = \frac{w(x,y)}{\sqrt{d(x)d(y)}}$  in the normalized Laplacian case. Therefore for these two examples the continuous operator has a single expression  $\mathbf{S}f = \int w_{\mathbf{S}}(\cdot ,x)dP(x)$ .
Graph Neural Network. As mentioned in the introduction, we focus on equivariant GNNs that can compute functions over nodes, as this makes the most sense on large graphs that RGs seek to model. Recall that we observe a graph shift operator  $S$  and node features  $Z \in \mathbb{R}^{n \times p}$ , and we return a vector per node  $\Phi(S,Z) \in \mathbb{R}^{n \times d_L}$ . We adopt a traditional GNN that uses the graph shift matrix  $S$ : given input features  $Z^{(0)} \in \mathbb{R}^{n \times d_0}$ ,
$$
Z ^ {(\ell)} = \rho \left(Z ^ {(\ell - 1)} \theta_ {0} ^ {(\ell - 1)} + S Z ^ {(\ell - 1)} \theta_ {1} ^ {(\ell - 1)} + 1 _ {n} (b ^ {(\ell)}) ^ {\top}\right) \in \mathbb {R} ^ {n \times d _ {\ell}},
$$
$$
\Phi_ {\theta} (S, Z ^ {(0)}) = Z ^ {(L - 1)} \theta^ {(L - 1)} + 1 _ {n} (b ^ {(L)}) ^ {\top} \tag {2}
$$
where  $\rho$  is the ReLU function applied element-wise, and  $\theta_i^{(\ell)}\in \mathbb{R}^{d_\ell \times d_{\ell +1}}$ ,  $b^{(\ell)}\in \mathbb{R}^{d_{\ell}}$  are learnable parameters gathered in  $\theta \in \Theta$ . We denote by  $\Theta$  the set of all possible parameters. For all classic choices of  $S$ , our definition of GNNs are a special case of message-passing NN (MPNN), which can be defined with a more general "aggregation" function. For the two examples above (adjacency and Laplacian), the aggregation function used is a sum, or a normalized sum.
We note that here we employ the ReLU function as a non-linearity, as some of our results will use its specific properties. Multi-Layer Perceptrons (MLP, densely connected networks) using the ReLU activation, and with potentially more than one hidden layer, will be denoted by  $f_{\gamma}^{\mathrm{MLP}}$ , where  $\gamma$  gathers their parameters.
<sup>1</sup>Note that the normalized Laplacian is traditionally defined as  $\operatorname{Id} - L$ , here it does not change our definition of GNNs since they include residual connections
Following recent literature [12, 13], we consider inputting Positional Encoding (PE) at each node. Such PE are generally computed using only the graph structure and concatenated to existing node features  $Z$ , here we simply introduce a generic notation:
$$
Z ^ {(0)} = \mathrm {P E} _ {\gamma} (S, Z) \in \mathbb {R} ^ {n \times d _ {0}} \tag {3}
$$
with some parameter  $\gamma \in \Gamma$ . In our notations, the PE module uses the node features  $Z$ , generally by concatenating them to its output. For short, we may denote the whole architecture with PE and GNN as  $\Phi_{\theta,\gamma}(S,Z) \stackrel{\mathrm{def.}}{=} \Phi_{\theta}(S,\mathrm{PE}_{\gamma}(S,Z))$ . It is not difficult to see that if the PE computation is equivariant, then the whole GNN is equivariant: denoting by  $\sigma$  a permutation matrix of  $\{1,\ldots,n\}$ ,
$$
\forall \sigma ,   \Phi_ {\theta , \gamma} (\sigma S \sigma^ {\top}, \sigma Z) = \sigma \Phi_ {\theta , \gamma} (S, Z) \quad \Leftrightarrow \quad \forall \sigma ,   \mathrm {P E} _ {\gamma} (\sigma S \sigma^ {\top}, \sigma Z) = \sigma \mathrm {P E} _ {\gamma} (S, Z).
$$
All the examples of PEs examined in Sec. 4 are equivariant.
3 Function spaces of Graph Neural Networks
In this section, we provide a complete and intuitive description of the function space approximated by equivariant GNNs applied on RGs. All technical proofs are provided in App. A. It has been shown [24, 25, 8, 29] that GNNs converge to functions over the latent space: when the node features are a sampling of a certain function  $\iota_X f^{(0)}$ , then the output of the GNN is close to being a sampling of another function  $\iota_X f^{(L)}$ . Assuming the node features or PEs approximate some function set  $\mathcal{B} \subset L_{\square}^2$ , we define the space of functions that a GNN can approximate as follows.
Definition 1. Given a base set  $\mathcal{B} \subset L_{\sqcup}^{2}$ , the set of functions approximated by GNNs  $\mathcal{F}_{\mathrm{GNN}}(\mathcal{B})$  is formed by all the functions  $f \in L_{\sqcup}^{2}$  such that: for all  $\varepsilon > 0$ , there are  $\theta \in \Theta$ ,  $f^{(0)} \in \mathcal{B}$  such that
$$
\mathbb {P} \left(\left\| \Phi_ {\theta} (S, \iota_ {X} f ^ {(0)}) - \iota_ {X} f \right\| _ {\mathrm {M S E}} \geqslant \varepsilon\right) \xrightarrow [ n \rightarrow \infty ]{} 0. \tag {4}
$$
In other words,  $\mathcal{F}_{\mathrm{GNN}}(\mathcal{B})$  are the functions whose sampling can be  $\varepsilon$ -approximated by the output of a GNN, with probability going to 1 as  $n$  grows. Note that if the quantifiers of  $\theta$ ,  $f^{(0)}$  and  $\varepsilon$  were reversed, the MSE would converge to 0 in probability. Here this is not the case:  $\theta$ ,  $f^{(0)}$  may depend on  $\varepsilon$ , which is akin to an approximation level. Similar to the permutation equivariance of GNNs, there is a notion of continuous equivariance for functions well-approximated by GNNs [24, 25, 8], where the permutations are replaced by bijections over the latent space  $\mathcal{X}$ . We adopt the notations  $\mathcal{F}_{\mathrm{GNN}}(\mathcal{B}) = \mathcal{F}_{\mathrm{GNN}}(\mathcal{B}, w, P)$ . For all continuous bijections  $\phi$  over  $\mathcal{X}$ , we define  $w_{\phi}(x, y) = w(\phi(x), \phi(y))$ ,  $P_{\phi} = \phi^{-1} \sharp P$  where  $\sharp$  is the push-forward operation, and  $\mathcal{B}_{\phi} = \{f \circ \phi \mid f \in \mathcal{B}\}$ . Then, we have the following result.
Proposition 1. Let  $S = S(A)$  be a graph shift operator that only depends on the adjacency matrix of the graph in a permutation-equivariant manner. Then, for all continuous bijections  $\phi : \mathcal{X} \to \mathcal{X}$ ,
$$
\mathcal {F} _ {\mathrm {G N N}} \left(\mathcal {B} _ {\phi}, w _ {\phi}, P _ {\phi}\right) = \left\{f \circ \phi \mid f \in \mathcal {F} _ {\mathrm {G N N}} (\mathcal {B}, w, P) \right\}.
$$
That is, if one "permutes" the kernel  $w$ , the distribution  $P$  and the base set  $\mathcal{B}$ , then the function space  $\mathcal{F}_{\mathrm{GNN}}$  contains exactly the permuted version of the original space.
The goal of this section is to provide a more intuitive description of the space  $\mathcal{F}_{\mathrm{GNN}}$ , which we will do under some basic convergence assumption from  $S$  to  $\mathbf{S}$ . GNNs (2) basically include two components: dense connections and MLPs that can approximate any continuous function by the universality theorem [40], and applications of  $S$ . Hence, we define the following function space.
Definition 2. We define  $\mathcal{F}_{\mathbf{S}}(\mathcal{B}) \subset L_{\sqcup}^{2}$  the (minimal) S-extension of a base set  $\mathcal{B} \subset L_{\sqcup}^{2}$  by the following rules:
(i)Base space:  $\mathcal{B}\subset \mathcal{F}_{\mathbf{S}}(\mathcal{B})$
(ii) Stability by composition with continuous functions: for all  $f \in \mathcal{F}_{\mathbf{S}}(\mathcal{B})$  with a  $p$ -dimensional output and  $g \in \mathcal{C}_{\mathrm{Lip}}(\mathbb{R}^p, \mathbb{R}^q)$ , it holds² that  $g \circ f \in \mathcal{F}_{\mathbf{S}}(\mathcal{B})$ ;
(iii) Stability by graph operator: for all  $f \in \mathcal{F}_{\mathbf{S}}(\mathcal{B})$ , it holds that  $\mathbf{S}f \in \mathcal{F}_{\mathbf{S}}(\mathcal{B})$ ;
2Note that, since  $g$  is Lipschitz, when  $f\in L_{\sqcup}^{2}$  we indeed have  $g\circ f\in L_{\sqcup}^{2}$
(iv) Linear span: for all  $q$ ,  $\mathcal{F}_{\mathbf{S}}(\mathcal{B}) \cap L_q^2$  is a vector space;
(v) Closure:  $\mathcal{F}_{\mathbf{S}}(\mathcal{B})$  is closed in  $L_{\square}^{2}$ ;
(vi) Minimality: for all  $\mathcal{G} \subset L_{\sqcup}^{2}$  satisfying all the properties above,  $\mathcal{F}_{\mathbf{S}} \subset \mathcal{G}$ .
In words,  $\mathcal{F}_{\mathrm{S}}$  take a base set  $\mathcal{B}$ , and extend it to be stable by composition with Lipschitz functions, application of the graph operator, and linear combination (of its elements with the same dimensionality). Our result will use the following assumption, which is naturally true for our running examples.
Assumption 1. With probability going to 1,  $\| S \|$  is bounded. Moreover, for all  $f \in L_{\sqcup}^{2}$ ,
$$
\| S \iota_ {X} f - \iota_ {X} \mathbf {S} f \| _ {\mathrm {M S E}} \xrightarrow [ n \to \infty ]{\mathcal {P}} 0
$$
where  $\stackrel{\mathcal{P}}{\longrightarrow}$  indicates convergence in probability.
Proposition 2. Assumption 1 is true for the adjacency matrix (ex. 1) and normalized Laplacian (ex. 2).
Under this assumption, the main result of this section states that the functions well-approximated by GNNs are exactly the S-extension of the base input features  $\mathcal{B}$ .
Theorem 1. Under Assumption 1, for all  $\mathcal{B} \subset L_{\sqcup}^{2}$ , we have:
$$
\mathcal {F} _ {\mathrm {G N N}} (\mathcal {B}) = \mathcal {F} _ {\mathbf {S}} (\mathcal {B})
$$
Given the definition of GNNs (2) and construction of  $\mathcal{F}_{\mathbf{S}}$ , Theorem 1 appears quite natural. Its proof, provided in App. A.3, is however far from trivial. The inclusion  $\mathcal{F}_{\mathbf{S}}(\mathcal{B}) \subset \mathcal{F}_{\mathrm{GNN}}(\mathcal{B})$  is similar in spirit to previous convergence results [24], since one has to construct a GNN that approximates a particular function. It involves however a new extended universality theorem for MLPs for square-integrable functions (Lemma 3 in App. A.3), which uses the special properties of ReLU. The reverse inclusion  $\mathcal{F}_{\mathrm{GNN}}(\mathcal{B}) \subset \mathcal{F}_{\mathbf{S}}(\mathcal{B})$  is quite different from previous work on GNN convergence: given  $f \in \mathcal{F}_{\mathrm{GNN}}(\mathcal{B})$  whose only property is to be well-approximated by GNNs, one must construct a sequence of functions in  $\mathcal{F}_{\mathbf{S}}(\mathcal{B})$  that converge to  $f$ , and uses the closure of  $\mathcal{F}_{\mathbf{S}}(\mathcal{B})$ . The need to work within square-integrable function is here obvious, as we only have convergence of the MSE, an approximation of the  $L^2$ -norm. For instance, this inclusion would not be true in the space of continuous functions.
Using composition with continuous functions, if  $\mathcal{F}_{\mathbf{S}}(\mathcal{B})$  contains a continuous bijection  $\phi : \mathcal{X} \to \operatorname{Im}(\phi)$ , then  $\mathcal{F}_{\mathbf{S}}(\mathcal{B})$  contains all continuous functions, and by density all square integrable functions. That is, the equivariant GNNs are then universal over  $\mathcal{X}$ : they can generate any function to label the nodes. Another criterion using the Stone-Weierstrass theorem (e.g. [21]), similar to the proofs in [25], is the following.
Proposition 3. Assume that for all  $x \neq x'$  in  $\mathcal{X}$ , there is a continuous function  $f \in \mathcal{F}_{\mathbf{S}}(\mathcal{B}) \cap \mathcal{C}_{\mathrm{Lip}}(\mathcal{X}, \mathbb{R})$  such that  $f(x) \neq f(x')$ . Then,  $\mathcal{F}_{\mathbf{S}}(\mathcal{B}) = L_{\square}^{2}$ .
In the rest of the paper, we study several examples of PEs and corresponding set  $\mathcal{B}$ , that will generalize the results of [25]. We expect many other interesting characteristics of  $\mathcal{F}_{\mathbf{S}}$  to be derived in the future.
4 Node features and Positional encodings
In the previous section, we have provided a complete description of the function space generated by equivariant GNNs when fed samplings of functions as node features, and the set of  $\mathcal{B}$  is thus crucial for the properties of  $\mathcal{F}_{\mathbf{S}}(\mathcal{B})$ . For instance, in the absence of node features and PEs, it is classical to input constant features to GNNs [25], such that the space of interest is  $\mathcal{F}_{\mathbf{S}}(1)$ . However, similar to the failure of the WL test on regular graphs, if  $\mathbf{S}1 \propto 1$  (e.g. constant degree function), then  $\mathcal{F}_{\mathbf{S}}(1)$  contains only constant functions! The role of PEs is often to mitigate such situations.