Year: 2021

Abstract:
Characterizing the separation power of graph neural networks (GNNs) provides an understanding of their limitations for graph learning tasks. Results regarding separation power are, however, usually geared at specific GNN architectures, and tools for understanding arbitrary GNN architectures are generally lacking. We provide an elegant way to easily obtain bounds on the separation power of GNNs in terms of the Weisfeiler-Leman (WL) tests, which have become the yardstick to measure the separation power of GNNs. The crux is to view GNNs as expressions in a procedural tensor language describing the computations in the layers of the GNNs. Then, by a simple analysis of the obtained expressions, in terms of the number of indexes and the nesting depth of summations, bounds on the separation power in terms of the WL-tests readily follow. Furthermore, the tensor language point of view allows for the derivation of universality results for classes of GNNs in a natural way. Our approach provides a toolbox with which GNN architecture designers can analyze the separation power of their GNNs, without needing to know the intricacies of the WL-tests. We also provide insights in what is needed to boost the separation power of GNNs.

heading: 1 INTRODUCTION
text: Graph Neural Networks (GNNs) (Merkwirth & Lengauer, 2005; Scarselli et al., 2009) cover many popular deep learning methods for graph learning tasks (see Hamilton (2020) for a recent overview). These methods typically compute vector embeddings of vertices or graphs by relying on the underlying adjacency information. Invariance (for graph embeddings) and equivariance (for vertex embeddings) of GNNs ensure that these methods are oblivious to the precise representation of the graphs.
Separation power. Our primary focus is on the separation power of GNN architectures, i.e., on their ability to separate vertices or graphs by means of the computed embeddings. It has become standard to characterize GNN architectures in terms of the separation power of graph algorithms such as color refinement (CR) and k-dimensional Weisfeiler-Leman tests (k-WL), as initiated in Xu et al. (2019) and Morris et al. (2019). Unfortunately, understanding the separation power of any given GNN architecture requires complex proofs, geared at the specifics of the architecture. We provide a tensor language-based technique to analyze the separation power of general GNNs.
Tensor languages. Matrix query languages (Brijder et al., 2019; Geerts et al., 2021b) are defined to assess the expressive power of linear algebra. Balcilar et al. (2021a) observe that, by casting various GNNs into the MATLANG (Brijder et al., 2019) matrix query language, one can use existing separation results (Geerts, 2021) to obtain upper bounds on the separation power of GNNs in terms of 1-WL and 2-WL. In this paper, we considerably extend this approach, by defining, and studying, a new general-purprpose tensor language specifically designed for modeling GNNs. As in Balcilar et al. (2021a), our focus on tensor languages allows us to obtain new insights about GNN architectures. First, since tensor languages can only define invariant and equivariant graph functions, any GNN that can be cast in our tensor language inherits these desired properties. More importantly, the separation power of our tensor language is as closely related to CR and k-WL as GNNs are. Loosely speaking, if tensor language expressions use k + 1 indices, then their separation power is bounded by k-WL. Furthermore, if the maximum nesting of summations in the expression is t, then t rounds of k-WL are needed for to obtain an upper bound on the separation power. A similar connection is obtained for CR and a fragment of tensor language that we call “guarded” tensor language.
We thus reduce problem of assessing the separation power of any specific GNN architecture to the problem of specifying it in our tensor language, analyzing the number of indices used and counting their summation depth. As we show in the paper, this is usually much easier than dealing with intricacies of CR and k-WL. In fact, casting GNNs in our tensor language is often as simple as writing down their layer-based definition found in research articles. We believe that this provides a nice toolbox for GNN designers to assess the separation power of their architecture. We use this toolbox to recover known results about the separation power of specific GNN architectures such as GINs (Xu et al., 2019), GCNs (Kipf & Welling, 2017), Folklore GNNs (Maron et al., 2019b), k-GNNs (Morris et al., 2019), and several others. We also derive new results: we answer an open problem posed by Maron et al. (2019a) by showing that the separation power of Invariant Graph Networks (k-IGNs), introduced by Maron et al. (2019b), is bounded by (k − 1)-WL. In addition, we revisit the analysis by Balcilar et al. (2021b) of ChebNet (Defferrard et al., 2016), and show that CayleyNet (Levie et al., 2019) is bounded by 2-WL.
When writing down GNNs in our tensor language, the less indices needed, the stronger the bounds in terms of k-WL we obtain. After all, (k − 1)-WL is known to be strictly less separating than k-WL (Otto, 2017). Thus, it is important to minimize the number of indices used in tensor language expressions. We connect this number to the notion of treewidth: expressions of treewidth k can be translated into expressions using k+1 indices. This corresponds to optimizing expressions, as done in many areas in machine learning, by reordering the summations (a.k.a. variable elimination).
Approximation and universality. We next consider the ability of GNNs to approximate general invariant or equivariant graph functions. Once more, instead of focusing on specific architectures, we use our tensor languages to obtain general approximation results, which naturally translate to universality results for GNNs. We show: k+1 index tensor language expressions suffice to approximate any (invariant/equivariant) graph function whose separating power is bounded by k-WL, and we can further refine this by comparing the number of rounds in k-WL with the summation depth of the expressions. These results provide a finer picture than the one obtained by Azizian & Lelarge (2021). Furthermore, focusing on “guarded” tensor expressions yields a similar universality result for CR, a result that, to our knowledge, was not known before. We also provide the link between approximation results for tensor expressions and GNNs, enabling us to transfer our insights into universality properties of GNNs. As an example, we show that k-IGNs can approximate any graph function that is less separating than (k−1)-WL. This case was left open in Azizian & Lelarge (2021). In summary, we draw new and interesting connections between tensor languages, GNN architectures and classic graph algorithms. We provide a general recipe to bound the separation power of GNNs, optimize them, and understand their approximation power. We show the usefulness of our method by recovering several recent results, as well as new results, some of them left open in previous work.
Related work. Separation power has been studied for specific classes of GNNs. (Morris et al., 2019; Xu et al., 2019; Maron et al., 2019b; Chen et al., 2019; Morris et al., 2020; Azizian & Lelarge, 2021). Balcilar et al. (2021a) follow a more general approach in which the MATLANG matrix query language is used to obtain upper bounds on the separation power of various GNNs. MATLANG can only be used to obtain bounds up to 2-WL and is inherently limited to matrices. Our tensor language is more general and flexible and allows for reasoning over the number of indices, treewidth, and summation depth of expressions. These are all crucial for our main results. The tensor language introduced resembles sum-MATLANG (Geerts et al., 2021b), but with the added ability to represent tensors. Neither separation power nor guarded fragments were considered in Geerts et al. (2021b). See the supplementary material for a detailed comparison. For universality, Azizian & Lelarge (2021) is closest in spirit. Our tensor language approach provides an elegant way to recover and extend their results. Azizian & Lelarge (2021) describe how their work (and hence also ours) encompasses previous works (Keriven & Peyré, 2019; Maron et al., 2019c; Chen et al., 2019). Our results use connections between k-WL and logics (Immerman & Lander, 1990; Cai et al., 1992), and CR and guarded logics (Barceló et al., 2020). The optimization of algebraic computations and the use of treewidth relates to the approaches by Aji & McEliece (2000) and Abo Khamis et al. (2016).

heading: 2 BACKGROUND
text: We denote sets by {} and multisets by {{}}. For n ∈ N, n > 0, [n] := {1, . . . , n}. Vectors are denoted by v,w, . . ., matrices by A,B, . . ., and tensors by S,T, . . .. Furthermore, vi is the i-th
entry of vector v,Aij is the (i, j)-th entry of matrix A and Si denotes the i = (i1, . . . , ik)-th entry of a tensor S. If certain dimensions are unspecified, then this is denoted by a “:”. For example, Ai: and A:j denote the i-th row and j-th column of matrix A, respectively. Similarly for slices of tensors.
We consider undirected simple graphs G = (VG, EG, colG) equipped with a vertex-labelling colG : VG → Rℓ. We assume that graphs have size n, so VG consists of n vertices and we often identify VG with [n]. For a vertex v ∈ VG, NG(v) := {u ∈ VG | vu ∈ EG}. We let G be the set of all graphs of size n and let Gs be the set of pairs (G,v) with G ∈ G and v ∈ V sG. Note that G = G0. The color refinement algorithm (CR) (Morgan, 1965) iteratively computes vertex labellings based on neighboring vertices, as follows. For a graph G and vertex v ∈ VG, cr(0)(G, v) := colG(v). Then, for t ≥ 0, cr(t+1)(G, v) := (cr(t)(G, v), {{cr(t)(G, u) | u ∈ NG(v)}}). We collect all vertex labels to obtain a label for the entire graph by defining gcr(t)(G) := {{cr(t)(G, v) | v ∈ VG}}. The k-dimensional Weisfeiler-Leman algorithm (k-WL) (Cai et al., 1992) iteratively computes labellings of k-tuples of vertices. For a k-tuple v, its atomic type in G, denoted by atpk(G,v), is a vector in R2( k 2)+kℓ. The first ( k 2 ) entries are 0/1-values encoding the equality type of v, i.e., whether vi = vj
for 1 ≤ i < j ≤ k. The second ( k 2 ) entries are 0/1-values encoding adjacency information, i.e., whether vivj ∈ EG for 1 ≤ i < i ≤ k. The last kℓ real-valued entries correspond to colG(vi) ∈ Rℓ for 1 ≤ i ≤ k. Initially, for a graph G and v ∈ V kG , k-WL assigns the label wl (0) k (G,v) := atpk(G,v). For t ≥ 0, k-WL revises the label according to wl (t+1) k (G,v) := (wl (t) k (G,v),M) with
M := {{(
atpk+1(G,vu),wl (t) k (G,v[u/1]), . . . ,wl (t) k (G,v[u/k]) ) | u ∈ VG }} , where v[u/i] :=
(v1, . . . , vi−1, u, vi+1, . . . , vk). We use k-WL to assign labels to vertices and graphs by defining: vwl(t)k (G, v) := wl (t) k (G, (v, . . . , v)), for vertex-labellings, and gwl (t) k := {{wl (t) k (G,v) | v ∈ V kG}}, for graph-labellings. We note that 1-WL differs from CR in that 1-WL also uses information from non-adjacent vertices. This distinction only matters for vertex embeddings (Grohe, 2021). Moreover, we use cr(∞), gcr(∞), vwl(∞)k , and gwl (∞) k to denote the stable labellings produced by the corresponding algorithm over an arbitrary number of rounds.
We emphasize that we are using the “folklore” k-WL of Cai et al. (1992), except that Cai et al. use 1-WL to refer to CR. It is equivalent to “oblivious” (k + 1)-WL (Grohe, 2021), used in some other works on GNNs. Care is thus needed when comparing our results to related works.
Let G be a graph with VG = [n] and let σ be a permutation of [n]. We denote by σ ⋆ G the isomorphic copy of G obtained by applying the permutation σ. Similarly, for v ∈ V kG , σ ⋆ v is the permuted version of v. Let F be some feature space. A function f : G0 → F is called invariant if f(G) = f(σ ⋆ G) for any permutation π. More generally, f : Gs → F is equivariant if f(σ ⋆G, σ ⋆ v) = f(G,v) for any permutation σ. The functions cr(t) : G1 → F and vwl(t)k : G1 → F are equivariant, whereas gcr(t) : G0 → F and gwl(t)k : G0 → F are invariant, for any t ≥ 0 and k ≥ 1.

heading: 3 SPECIFYING GNNS
text: Many GNN architectures are described in terms of linear algebra computations on vectors, matrices or tensors, interleaved with the application of activation functions or multilayer perceptrons (MLPs). We aim at understanding the separation power of GNNs, without being tied to a specific GNN architecture. In order to do so, we introduce the specification language TL, for tensor language, and show how to use it to specify several GNN architectures. The idea of this language is to express algebraic computations in a more procedural way by explicitly stating how each entry is to be computed. For simplicity of exposition, we focus on GNNs that use summation aggregation and present TL using summation aggregation only. In the supplementary material, we extend TL with all kinds of aggregation functions on multisets of real values; our results carry over to this general setting.
To introduce TL, consider a typical layer in a GNN of the form F ′ = σ(A·F ·W ), where A ∈ Rn×n is an adjacency matrix, F ∈ Rn×ℓ are vertex features such that Fi: ∈ Rℓ is the feature vector of vertex i, σ is a non-linear activation function, and W ∈ Rℓ×ℓ is a weight matrix. By exposing the indices in the matrices and vectors we can equivalently write: for i ∈ [n] and s ∈ [ℓ]:
F ′is := σ (∑ j∈[n]Aij · (∑ t∈[ℓ]Wts · Fjt )) .
In TL, we do not work with specific matrices or indices ranging over [n], but focus instead on expressions applicable to any matrix. We use index variables x1 and x2 instead of i and j, replace Aij with a placeholder E(x1, x2) and Fjt with placeholders Pt(x2), for t ∈ [ℓ]. We then represent the above computation in TL by ℓ expressions ψs(x1), one for each feature column, as follows:
ψs(x1) = σ (∑
x2 E(x1, x2) · (∑ t∈[ℓ]Wts · Pt(x2) )) .
These are pure syntactical expressions. To give them a semantics, we assign to E a matrix A ∈ Rn×n, to Pt column vectors F:t ∈ Rn×1, for t ∈ [ℓ], and to x1 an index i ∈ [n]. By letting the variable x2 under the summation range over 1, 2, . . . , n, the TL expression ψs(i) evaluates to F ′is. As such, F ′ = σ(A ·F ·W ) can be represented as a specific instance of the above TL expressions. Throughout the paper we reason about expressions in TL rather than specific instances thereof. Importantly, by showing that certain properties hold for expressions in TL, these properties are inherited by all of its instances. We use TL to enable a theoretical analysis of the separating power of GNNs; It is not intended as a practical programming language for GNNs.
Syntax. We first give the syntax of TL expressions. We have a binary predicate E, to represent adjacency matrices, and unary vertex predicates Ps, s ∈ [ℓ], to represent column vectors encoding the ℓ-dimensional vertex labels. In addition, we have a (possibly infinite) set Ω of functions, such as activation functions or MLPs. Then, TL(Ω) expressions are defined by the following grammar:
φ := 1x op y | E(x, y) | Ps(x) | φ · φ | φ+ φ | a · φ | f(φ, . . . , φ) | ∑ x φ
where op ∈ {=, ̸=}, x, y are index variables that specify entries in tensors, s ∈ [ℓ], a ∈ R, and f ∈ Ω. Summation aggregation is captured by ∑ x φ.
1 We sometimes make explicit which functions are used in expressions in TL(Ω) by writing TL(f1, f2, . . .) for f1, f2, . . . in Ω. For example, the expressions ψs(x1) are in TL(σ).
The set of free index variables of an expression φ, denoted by free(φ), determines the order of the tensor represented by φ. It is defined inductively: free(1x op y) = free(E(x, y)) := {x, y}, free(Ps(x)) = {x}, free(φ1 ·φ2) = free(φ1+φ2) := free(φ1)∪ free(φ2), free(a ·φ1) := free(φ1), free(f(φ1, . . . , φp)) := ∪i∈[p]free(φi), and free( ∑ x φ1) := free(φ1) \ {x}. We sometimes explicitly write the free indices. In our example expressions ψs(x1), x1 is the free index variable.
An important class of expressions are those that only use index variables {x1, . . . , xk}. We denote by TLk(Ω) the k-index variable fragment of TL(Ω). The expressions ψs(x1) are in TL2(σ).
Semantics. We next define the semantics of expressions in TL(Ω). Let G = (VG, EG, colG) be a vertex-labelled graph. We start by defining the interpretation πG[[·, ν]] of the predicates E, Ps and the (dis)equality predicates, relative toG and a valuation ν assigning a vertex to each index variable:
πG[[E(x, y), ν]] := if ν(x)ν(y) ∈ EG then 1 else 0 πG[[Ps(x), ν]] := colG(ν(x))s∈ R πG[[1x op y, ν]] := if ν(x) op ν(y) then 1 else 0.
In other words, E is interpreted as the adjacency matrix of G and the Ps’s interpret the vertexlabelling colG. Furthermore, we lift interpretations to arbitrary expressions in TL(Ω), as follows:
πG[[φ1 · φ2, ν]] := πG[[φ1, ν]] · πG[[φ2, ν]] πG[[φ1+φ2, ν]] := πG[[φ1, ν]] + πG[[φ2, ν]] πG[[ ∑ x φ1, ν]] := ∑ v∈VGπG[[φ1, ν[x 7→ v]]] πG[[a · φ1, ν]] := a · πG[[φ1, ν]]
πG[[f(φ1, . . . , φp), ν]] := f(πG[[φ1, ν]], . . . , πG[[φp, ν]])
where, ν[x 7→ v] is the valuation ν but which now maps the index x to the vertex v ∈ VG. For simplicity, we identify valuations with their images. For example, πG[[φ(x), v]] denotes πG[[φ1(x), x 7→ v]]. To illustrate the semantics, for each v ∈ VG, πG[[ψs, v]] = F ′vs for F ′ = σ(A · F ·W ) when A is the adjacency matrix of G and F represents the vertex labels.
Representing GNNs. Consider a function f : Gs → Rℓ : (G,v) 7→ f(G,v) ∈ Rℓ for some ℓ ∈ N. We say that the function f can be represented in TL(Ω), if there exists ℓ expressions φ1(x1, . . . , xs), . . . , φℓ(x1, . . . , xs) in TL(Ω) such that for each graph G and each s-tuple v ∈ V sG:
f(G,v) = ( πG[[φ1,v]], . . . , πG[[φℓ,v]] ) .
1We can replace ∑
x φ by a more general aggregation construct aggr(F, x, φ) for arbitrary functions F that assign a real value to multisets of real values. We refer to the supplementary material for details.
When f is described in terms of a GNN, we say that the GNN can be represented in TL(Ω). For example, we can interpret F ′ = σ(A·F ·W ) as a function f : G1 → Rℓ, such that f(G, v) := F ′v:. We have seen that for each s ∈ [ℓ], πG[[ψs, v]] = F ′vs. Hence, f(G, v) = (πG[[ψ1, v]], . . . , πG[[ψℓ, v]]) and thus f (that is, F ′ = σ(A ·F ·W )) can be represented in TL(Ω). Most functions corresponding to GNNs can be represented in TL(Ω), as will be illustrated later in the paper.
TL represents equivariant or invariant functions. We make a simple observation which follows from the type of operators allowed in expressions in TL(Ω).
Proposition 3.1. Any function f : Gs → Rℓ represented in TL(Ω) is equivariant (invariant if s = 0). An immediate consequence is that when a GNN can be represented in TL(Ω) it is automatically invariant or equivariant, depending on whether graph or vertex tuple embeddings are considered.

heading: 4 SEPARATION POWER OF TENSOR LANGUAGES
text: Our first main results concern the characterization of the separation power of tensor languages in terms of the color refinement and k-dimensional Weisfeiler-Leman algorithms. We provide a finegrained characterization by taking the number of rounds of these algorithms into account. This will allow for measuring the separation power of classes of GNNs in terms of their number of layers.

heading: 4.1 SEPARATION POWER
text: We define the separation power of graph functions in terms of an equivalence relation, based on the definition from Azizian & Lelarge (2021), hereby first focusing on their ability to separate vertices.2
Definition 1. Let F be a set of functions f : G1 → Rℓf . The equivalence relation ρ1(F) is defined by F on G1 as follows: ( (G, v), (H,w) ) ∈ ρ1(F) ⇐⇒ ∀f ∈ F , f(G, v) = f(H,w).
In other words, when ((G, v), (H,w)) ∈ ρ1(F), no function in F can separate v in G from w in H . For example, we can view cr(t) and vwl(t)k as functions from G1 to some Rℓ. As such ρ1(cr(t)) and ρ1(vwl(t)k ) measure the separation power of these algorithms. The following strict inclusions are known: for all k ≥ 1, ρ1(vwl(t)k+1) ⊂ ρ1(vwl (t) k ) and ρ1(vwl (t) 1 ) ⊂ ρ1(cr(t)) (Otto, 2017; Grohe, 2021). It is also known that more rounds (t) increase the separation power of these algorithms (Fürer, 2001).
For a fragment L of TL(Ω) expressions, we define ρ1(L) as the equivalence relation associated with all functions f : G1 → Rℓf that can be represented in L. By definition, we here thus consider expressions in TL(Ω) with one free index variable resulting in vertex embeddings.

heading: 4.2 MAIN RESULTS
text: We first provide a link between k-WL and tensor language expressions using k+1 index variables: Theorem 4.1. For each k ≥ 1 and any collection Ω of functions, ρ1 ( vwl(∞)k ) = ρ1 ( TLk+1(Ω) ) .
This theorem already gives us new insights: if we wish to understand how a new GNN architecture compares against the k-WL algorithms, all we need to do is to show that such an architecture can be represented in TLk+1(Ω), an arguably much easier endeavor. As an example of how to use this result, it is well known that triangles can be detected by 2-WL but not by 1-WL. Thus, in order to design GNNs that can detect triangles, layer definitions in TL3 rather than TL2 should be used.
We can do much more, relating the rounds of k-WL to the notion of summation depth of TL(Ω) expressions. We also present present similar results for functions computing graph embeddings. The summation depth sd(φ) of a TL(Ω) expression φ measures the nesting depth of the summations∑ x in the expression. It is defined inductively: sd(1x op y) = sd(E(x, y)) = sd(Ps(x)) := 0, sd(φ1 · φ2) = sd(φ1 + φ2) := max{sd(φ1), sd(φ2)}, sd(a · φ1) := sd(φ1), sd(f(φ1, . . . , φp)) := max{sd(φi)|i ∈ [p]}, and sd( ∑ x φ1) := sd(φ1) + 1. For example, expressions ψs(x1) above have summation depth one. We write TL(t)k (Ω) for the class of expressions in TLk(Ω) of summation depth
2We differ slightly from Azizian & Lelarge (2021) in that they only define equivalence relations on graphs.
at most t. We can now refine Theorem 4.1, taking into account the number of rounds used in k-WL. Theorem 4.2. For all t ≥ 0, k ≥ 1 and any collection Ω of functions, ρ1 ( vwl(t)k ) = ρ1 ( TL(t)k+1(Ω) ) .
Guarded TL and color refinement. As noted by Barceló et al. (2020), the separation power of vertex embeddings of simple GNNs, which propagate information only through neighboring vertices, is usually weaker than that of 1-WL. For these types of architectures, Barceló et al. (2020) provide a relation with the weaker color refinement algorithm, but only in the special case of firstorder classifiers. We can recover and extend this result in our general setting, with a guarded version of TL which, as we will show, has the same separation power as color refinement.
The guarded fragment GTL(Ω) of TL2(Ω) is inspired by the use of adjacency matrices in simple GNNs. In GTL(Ω) only equality predicates 1xi=xi (constant 1) and 1xi ̸=xi (constant 0) are allowed, addition and multiplication require the component expressions to have the same (single) free index, and summation must occur in a guarded form ∑ xj ( E(xi, xj) · φ(xj) ) , for i, j ∈ [2]. Guardedness means that summation only happens over neighbors. In GTL(Ω), all expressions have a single free variable and thus only functions from G1 can be represented. Our example expressions ψs(x1) are guarded. The fragment GTL(t)(Ω) consists of expressions in GTL(Ω) of summation depth at most t. Theorem 4.3. For all t ≥ 0 and any collection Ω of functions: ρ1 ( cr(t) ) = ρ1 ( GTL(t)(Ω) ) .
As an application of this theorem, to detect the existence of paths of length t, the number of guarded layers in GNNs should account for a representation in GTL(Ω) of summation depth of at least t. We recall that ρ1(vwl(t)1 ) ⊂ ρ1(cr(t)) which, combined with our previous results, implies that TL (t) 2 (Ω) is strictly more separating than GTL(t)(Ω).
Graph embeddings. We next establish connections between the graph versions of k-WL and CR, and TL expressions without free index variables. To this aim, we use ρ0(F), for a set F of functions f : G → Rℓf , as the equivalence relation over G defined in analogy to ρ1: (G,H) ∈ ρ0(F) ⇐⇒ ∀f ∈ F , f(G) = f(H). We thus consider separation power on the graph level. For example, we can consider ρ0(gcr(t)) and ρ0(gwl(t)k ) for any t ≥ 0 and k ≥ 1. Also here, ρ0(gwl (t) k+1) ⊂ ρ0(gwl (t) k ) but different from vertex embeddings, ρ0(gcr(t)) = ρ0(gwl(t)1 ) (Grohe, 2021). We define ρ0(L) for a fragment L of TL(Ω) by considering expressions without free index variables. Also here, the connection between the number of index variables in expressions and k-WL holds. Apart from k = 1, no clean relationship exists between summation depth and rounds, however.3
Theorem 4.4. For all t ≥ 0, k ≥ 1 and any collection Ω of functions, we have that: (1) ρ0 ( gcr(t) ) = ρ0 ( TL(t+1)2 (Ω) ) = ρ0 ( gwl(t)1 ) (2) ρ0 ( gwl(∞)k ) = ρ0 ( TLk+1(Ω) ) .
Intuitively, in (1) the increase in summation depth by one is incurred by the additional aggregation needed to collect all vertex labels computed by gwl(t)1 .
Optimality of number of indices. Our results so far tell that graph functions represented in TLk+1(Ω) are at most as separating as k-WL. What is left unaddressed is whether all k + 1 index variables are needed for the graph functions under consideration. It may well be, for example, that there exists an equivalent expression using less index variables. This would imply a stronger upper bound on the separation power by ℓ-WL for ℓ < k. We next identify a large class of TL(Ω) expressions, those of treewidth k, for which the number of index variables can be reduced to k+1.
Proposition 4.5. Expressions in TL(Ω) of treewidth k are equivalent to expressions in TLk+1(Ω).
Treewidth is defined in the supplementary material. Intuitively, treewidth k implies that the computation of tensor language expressions can be decomposed, by reordering summations, such that each local computation requires at most k + 1 indices. This is similar to the generalized distributive law approach for optimizing computations by Aji & McEliece (2000) (see also Abo Khamis et al. (2016)
3Indeed, the best one can obtain for general tensor logic expressions is ρ0 ( TL(t+k)k+1 (Ω) ) ⊆ ρ0 ( gwl(t)k ) ⊆
ρ0 ( TL(t+1)k+1 (Ω) ) . This follows from Cai et al. (1992) and connections to finite variable logics.
for a more recent account). As a simple example, consider θ(x1) = ∑
x2 ∑ x3 E(x1, x2) ·E(x2, x3)
in TL(2)3 such that πG[[θ, v]] counts the number of paths of length two starting from v. This expression has a treewidth of one. And indeed, it is equivalent to the expression θ̃(x1) = ∑ x2 E(x1, x2) ·(∑
x1 E(x2, x1) ) in TL(2)2 (and in fact in GTL
(2)). As a consequence, no more vertices can be separated by θ(x1) than by cr(2), rather than vwl22 as the original expression in TL (2) 3 suggests.
On the impact of functions. All separation results for TL(Ω) and fragments thereof hold irregardless of the chosen functions in Ω, including when no functions are present at all. Function applications hence do not add expressive power. While this may seem counter-intuitive, it is due to the presence of summation and multiplication in TL that are enough to separate graphs or vertices.

heading: 5 CONSEQUENCES FOR GNNS
text: We next interpret the general results on the separation power from Section 4 in the context of GNNs.
1. The separation power of any GNN vertex embedding architecture that can be represented in GTL(t)(Ω) is bounded by the power of t rounds of color refinement.
As an example, we consider the Graph Isomorphism Networks (GINs) (Xu et al., 2019). Let gin be such a network; it updates vertex embeddings as follows. Initially, gin(0) : G1 → Rℓ0 : (G, v) 7→ F (0)v: := colG(v) ∈ Rℓ0 . For layer t > 0, gin(t) : G1 → Rℓt is given by: (G, v) 7→ F (t)v: := mlp(t) ( F (t−1)v: , ∑ u∈NG(v) F (t−1) u: ) , with F (t) ∈ Rn×ℓt and mlp(t) = (mlp(t)1 , . . . ,mlp (t) ℓt ) : R2ℓt−1 → Rℓt is an MLP. We denote by GIN(t) the class of GINs consisting t layers. Clearly, gin(0) can be represented in GTL(0) by considering the expressions φ(0)i (x1) := Pi(x1) for each i ∈ [ℓ0]. To represent gin(t), assume that we have ℓt−1 expressions φ(t−1)i (x1) in GTL
(t−1)(Ω) representing gin(t−1). That is, we have πG[[φ(t−1)i , v]] = F (t−1) vi for each vertex v and i ∈ [ℓt−1]. Then gin
(t) is represented by ℓt expressions φ(t)i (x1) defined as:
mlp(t)i ( φ(t−1)1 (x1), . . . , φ (t−1) ℓt−1 (x1), ∑ x2 E(x1, x2) · φ(t−1)1 (x2), . . . , ∑ x2 E(x1, x2) · φ(t−1)ℓt−1 (x2) ) ,
which are now expressions in GTL(t)(Ω) where Ω consists of MLPs. We have πG[[φ(t)i , v]] = F (t) v,i for each v ∈ VG and i ∈ [ℓt] as desired. Hence, Theorem 4.3 tells that t-layered GINs cannot be more separating than t rounds of color refinement, in accordance with known results (Xu et al., 2019; Morris et al., 2019). We thus simply cast GINs in GTL(Ω) to obtain an upper bound on their separation power. In the supplementary material we give similar analyses for GraphSage GNNs with various aggregation functions (Hamilton et al., 2017), GCNs (Kipf & Welling, 2017), simplified GCNs (SGCs) (Wu et al., 2019), Principled Neighborbood Aggregation (PNAs) (Corso et al., 2020), and revisit the analysis of ChebNet (Defferrard et al., 2016) given in Balcilar et al. (2021a).
2. The separation power of any GNN vertex embedding architecture that can be represented in TL(t)k+1(Ω) is bounded by the power of t rounds of k-WL.
For k = 1, we consider extended Graph Isomorphism Networks (eGINs) (Barceló et al., 2020). For an egin ∈ eGIN, egin(0) : G1 → Rℓ0 is defined as for GINs, but for layer t > 0, egin(t) : G1 → Rℓt is defined by (G, v) 7→ F (t)v: := mlp (t) ( F (t−1)v: , ∑ u∈NG(v) F (t−1) u: , ∑ u∈VG F (t−1) u: ) , where mlp(t) is
now an MLP from R3ℓt−1 → Rℓt . The difference with GINs is the use of ∑
u∈VG F (t−1) u: which corresponds to the unguarded summation ∑
x1 φ(t−1)(x1). This implies that TL rather than GTL
needs to be used. In a similar way as for GINs, we can represent eGIN layers in TL(t)2 (Ω). Theorem 4.2 tells that t rounds of 1-WL bound the separation power of t-layered extended GINs, conform to Barceló et al. (2020). More generally, any GNN looking to go beyond CR must use non-guarded aggregations. For k ≥ 2, it is straightforward to show that t-layered “folklore” GNNs (k-FGNNs) (Maron et al., 2019b) can be represented in TL(t)k+1(Ω) and thus, by Theorem 4.2, t rounds of k-WL bound their separation power. One merely needs to cast the layer definitions in TL(Ω) and observe that k + 1 indices and summation depth t are needed. E.g., for k = 2, a 2-FGNN updates features according to F (t)v1v2: = mlp0 ( F (t−1)v1v2: , ∑ u∈VG mlp1(F (t−1) v1u: ) ·mlp2(F (t−1) uv2: ) ) , resulting in expressions
φ(t)(x1, x2) := mlp0 ( φ(t−1)(x1, x2), ∑ x3 mlp1(φ (t−1)(x1, x3)) ·mlp2(φ(t−1)(x3, x2)) )
in TL(t)3 (Ω). We thus refine and recover the k-WL bound for k-FGNNs by Azizian & Lelarge (2021). We also show that the separation power of (k+1)-Invariant Graph Networks ((k+1)-IGNs) (Maron et al., 2019b) are bounded by k-WL, albeit with an increase in the required rounds.
Theorem 5.1. For any k ≥ 1, the separation power of a t-layered (k + 1)-IGNs is bounded by the separation power of tk rounds of k-WL.
We hereby answer open problem 1 in Maron et al. (2019a). The case k = 1 was solved in Chen et al. (2020) by analyzing properties of 1-WL. By contrast, Theorem 4.2 shows that one can focus on expressing (k+1)-IGNs in TLk+1(Ω) and analyzing the summation depth of expressions. The proof of Theorem 5.1 requires non-trivial manipulations of tensor language expressions; it is a simplified proof of (anonymous). The additional rounds (tk) are needed because (k + 1)-IGNs aggregate information in one layer that becomes accessible to k-WL in k rounds. We provide all details in the supplementary material, where we also identify a simple class of t-layered (k + 1)-IGNs that are as powerful as (k + 1)-IGNs but whose separation power is bounded by t rounds of k-WL.
As another example, we study “augmented” GNNs, which are combined with a preprocessing step in which higher-order graph information is computed. In the supplementary material we show how TL encodes the preprocessing step, and how this leads to separation bounds in terms of k-WL, where k depends on the treewidth of the graph information used. Finally, our approach can also be used to show that the spectral CayleyNets (Levie et al., 2019) are bounded in separation power by 2-WL. This result complements the spectral analysis of CayleyNets given in Balcilar et al. (2021b).
3. The separation power of any GNN graph embedding architecture that can be represented in TLk+1(Ω) is bounded by the power of k-WL.
Graph embedding methods are commonly obtained from vertex (tuple) embeddings methods by including a readout layer in which all vertex (tuple) embeddings are aggregated. For example, mlp( ∑ v∈V egin
(t)(G, v)) is a typical readout layer for eGINs . Since egin(t) can be represented in TL(t)2 (Ω), the readout layer can be represented in TL (t+1) 2 (Ω), using an extra summation. Hence, their separation power is bounded by gwl(t)1 , in accordance with Theorem 4.4. This holds more generally. If a vertex (tuple) embedding method can be represented in TLk+1(Ω), then the same holds for their graph versions, which are then bounded by gwl(∞)k by our Theorem 4.4.
4. To go beyond the separation power of k-WL, it is necessary to use GNNs whose layers are represented by expressions of treewidth > k.
Hence, to design expressive GNNs one needs to define the layers such that treewidth of the resulting TL expressions is large enough. For example, to go beyond 1-WL, TL3 representable linear algebra operations should be used. Treewidth also sheds light on the open problem from Maron et al. (2019a) where it was asked whether polynomial layers (in A) increase the separation power. Indeed, consider a layer of the form σ(A3·F ·W ), which raises the adjacency matrix A to the power three. Translated in TL(Ω), layer expressions resemble ∑ x2 ∑ x3 ∑ x4 E(x1, x2)·E(x2, x3)·E(x3, x4), of treewidth one. Proposition 4.5 tells that the layer is bounded by wl(3)1 (and in fact by cr (3)) in separation power. If instead, the layer is of the form σ(C ·F ·W ) whereCij holds the number of cliques containing the edge ij. Then, in TL(Ω) we get expressions containing ∑ x2 ∑ x3 E(x1, x2) ·E(x1, x3) ·E(x2, x3). The variables form a 3-clique resulting in expressions of treewidth two. As a consequence, the separation power will be bounded by wl(2)2 . These examples show that it is not the number of multiplications (in both cases two) that gives power, it is how variables are connected to each other.

heading: 6 FUNCTION APPROXIMATION
text: We next provide characterizations of functions that can be approximated by TL expressions, when interpreted as functions. We recover and extend results from by taking the number of layers of GNNs into account. We also provide new results in the context of color refinement.

heading: 6.1 GENERAL TL APPROXIMATION RESULTS
text: We assume that Gs is a compact space by requiring that vertex labels come from a compact set K ⊆ Rℓ0 . Let F be a set of functions f : Gs → Rℓf and define its closure F as all functions h from Gs for
which there exists a sequence f1, f2, . . . ∈ F such that limi→∞ supG,v∥fi(G,v) − h(G,v)∥ = 0 for some norm ∥.∥. We assume F to satisfy two properties. First, F is concatenation-closed: if f1 : Gs → Rp and f2 : Gs → Rq are in F , then g := (f1, f2) : Gs → Rp+q : (G,v) 7→ (f1(G,v), f2(G,v)) is also in F . Second, F is function-closed, for a fixed ℓ ∈ N: for any f ∈ F such that f : Gs → Rp, also g ◦ f : Gs → Rℓ is in F for any continuous function g : Rp → Rℓ. For such F , we let Fℓ be the subset of functions in F from Gs to Rℓ. Our next result is based on a generalized Stone-Weierstrass Theorem (Timofte, 2005), also used in Azizian & Lelarge (2021). Theorem 6.1. For any ℓ, and any set F of functions, concatenation and function closed for ℓ, we have: Fℓ = {f : Gs → Rℓ | ρs(F) ⊆ ρs(f)}.
This result gives us insight on which functions can be approximated by, for example, a set F of functions originating from a class of GNNs. In this case, Fℓ represent all functions approximated by instances of such a class and Theorem 6.1 tells us that this set corresponds precisely to the set of all functions that are equally or less separating than the GNNs in this class. If, in addition, Fℓ is more separating that CR or k-WL, then we can say more. Let alg ∈ {cr(t), gcr(t), vwl(t)k , gwl (∞) k }.
Corollary 6.2. Under the assumptions of Theorem 6.1 and if ρ(Fℓ) = ρ(alg), then Fℓ = {f : Gs → Rℓ | ρ(alg) ⊆ ρ(f)}.
The properties of being concatenation and function-closed are satisfied for sets of functions representable in our tensor languages, as long as Ω contains all continuous functions g : Rp → Rℓ, for any p, or alternatively, all MLPs (by Lemma 32 in Azizian & Lelarge (2021)). Together with our results in Section 4, the corollary implies that GTL(t)(Ω), TL(t+1)2 (Ω), TL (t) k+1(Ω) or TL(Ω), when viewed as functions, can approximate all functions with equal or less separation power than cr(t), gcr(t), vwl(t)k or gwl (∞) k , respectively. Interestingly, Proposition 3.1 tells that the closure necessarily consists of invariant (when s = 0) and equivariant (when s > 0) functions.

heading: 6.2 CONSEQUENCES FOR GNNS
text: All our results combined provide a recipe to guarantee that a given function can be approximated by GNN architectures. Indeed, let f be a function that is not more separating than cr(t) (respectively, gcr(t), vwl(t)k or gwl (∞) k , for some k ≥ 1). Most classes of GNNs in the literature are concatenationclosed and allow the application of arbitrary MLPs. Then, f can be approximated by a class of GNNs, as long as these are at least as separating as GTL(t) (respectively, TL(t+1)2 , TL (t) k+1 or TL (∞) k+1). This, in turn, amounts to showing that the GNNs can be represented in the corresponding tensor language fragment, and that they can match the corresponding labeling algorithm in separation power.
For example, GIN(t)ℓ contains any function f : G1 → Rℓ satisfying ρ1(cr(t)) ⊆ ρ1(f); eGIN (t) ℓ any function satisfying ρ1(wl(t)1 ) ⊆ ρ1(f); and when extended with a readout layer, their closures consist of functions f : G0 → Rℓ satisfying ρ0(gcr(t)) = ρ0(vwl(t)1 ) ⊆ ρ0(f). Also, k-FGNN (t) ℓ consist of functions f such that ρ1(vwl(t)k ) ⊆ ρ1(f). We thus recover and extend results by Azizian & Lelarge (2021) by including layer information (t) and by treating color refinement separately from 1-WL for vertex embeddings. Furthermore, Theorem 5.1 implies that (k + 1)-IGNℓ consists of functions f satisfying ρ1(vwl(∞)k ) ⊆ ρ1(f) and ρ0(gwl (∞) k ) ⊆ ρ0(f), a case left open in Azizian & Lelarge (2021).
These results follow from Corollary 6.2, that the GNNs can simulate CR or k-WL on either graphs with discrete (Xu et al., 2019; Barceló et al., 2020) or continuous labels (Maron et al., 2019b), and that they can be cast in the corresponding tensor language. We can use these GNNs to approximate functions in our tensor languages, i.e., GIN(t)ℓ = GTL (t)(Ω), and similarly for the other cases.

heading: 7 CONCLUSION
text: Connecting GNNs and tensor languages allows us to translate our general analysis of tensor languages to an understanding of the separation and approximation power of GNNs. The number of indices and summation depth needed to represent the layers in GNNs determine their separation power in terms of color refinement and Weisfeiler-Leman tests. We thus provide a handy toolbox to understand existing and new GNN architectures, and we demonstrate this by recovering several results about the power of GNNs presented recently in the literature, as well as proving new results.

heading: ETHICS STATEMENT
text: The results in this paper do not include misleading claims; their correctness is theoretically verified. Related work is accurately represented.

heading: SUPPLEMENTARY MATERIAL

heading: A RELATED WORK CNT’D
text: We provide additional details on how the tensor language TL(Ω) considered in this paper relates to recent work on other matrix query languages. Closest to TL(Ω) is the matrix query language sum-MATLANG (Geerts et al., 2021b) whose syntax is close to that of TL(Ω). There are, however, key differences. First, although sum-MATLANG uses index variables (called vector variables), they all must occur under a summation. In other words, the concept of free index variables is missing, which implies that no general tensors can be represented. In TL(Ω), we can represent arbitrary tensors and the presence of free index variables is crucial to define vertex, or more generally, k-tuple embeddings in the context of GNNs. Furthermore, no notion of summation depth was introduced for sum-MATLANG. In TL(Ω), the summation depth is crucial to assess the separation power in terms of the number of rounds of color refinement and k-WL. And in fact, the separation power of sumMATLANG was not considered before, and neither are finite variable fragments of sum-MATLANG and connections to color refinement and k-WL studied before. Finally, no other aggregation functions were considered for sum-MATLANG. We detail in Section C.5 that TL(Ω) can be gracefully extended to TL(Ω,Θ) for some arbitrary set Θ of aggregation functions.
Connections to 1-WL and 2-WL and the separation power of another matrix query language, MATLANG (Brijder et al., 2019) were established in Geerts (2021). Yet, the design of MATLANG is completely different in spirit than that of TL(Ω). Indeed, MATLANG does not have index variables or explicit summation aggregation. Instead, it only supports matrix multiplication, matrix transposition, function applications, and turning a vector into a diagonal matrix. As such, MATLANG can be shown to be included in TL3(Ω). Similarly as for sum-MATLANG, MATLANG cannot represent general tensors, has no (free) index variables and summation depth is not considered (in view of the absence of an explicit summation).
We also emphasize that neither for MATLANG nor for sum-MATLANG a guarded fragment was considered. The guarded fragment is crucial to make connections to color refinement (Theorem 4.3). Furthermore, the analysis in terms of the number of index variables, summation depth and treewidth (Theorems 4.1,4.2 and Proposition 4.5), were not considered before in the matrix query language literature. For none of these matrix query languages, approximation results were considered (Section 6.1).
Matrix query languages are used to assess the expressive power of linear algebra. Balcilar et al. (2021a) use MATLANG and the above mentioned connections to 1-WL and 2-WL, to assess the separation power of GNNs. More specifically, similar to our work, they show that several GNN architectures can be represented in MATLANG, or fragments thereof. As a consequence, bounds on their separation power easily follow. Furthermore, Balcilar et al. (2021a) propose new architectures inspired by special operators in MATLANG. The use of TL(Ω) can thus been seen as a continuation of their approach. We note, however, that TL(Ω) is more general than MATLANG (which is included in TL3(Ω)), allows to represent more complex linear algebra computations by means summation (or other) aggregation, and finally, provides insights in the number of iterations needed for color refinement and k-WL. The connection between the number of variables (or treewidth) and k-WL is not present in the work by Balcilar et al. (2021a), neither is the notion of guarded fragment, needed to connect to color refinement. We believe that it is precisely these latter two insights that make the tensor language approach valuable for any GNN designer who wishes to upper bound their GNN architecture.

heading: B DETAILS OF SECTION 3

heading: B.1 PROOF OF PROPOSITION 3.1
text: Let G = (V,E, col) be a graph and let σ be a permutation of V . As usual, we define σ ⋆ G = (V σ, Eσ, colσ) as the graph with vertex set V σ := V , edge set vw ∈ Eσ if and only if σ−1(v)σ−1(w) ∈ E, and colσ(v) := col(σ−1(v)). We need to show that for any expression φ(x) in TL(Ω) either πσ⋆G[[φ, σ⋆v]] = πG[[φ,v]], or when φ has no free index variables, πσ⋆G[[φ]] = πG[[φ]]. We verify this by a simple induction on the structure of expressions in TL(Ω).
• If φ(xi, xj) = 1xi op xj , then for a valuation ν mapping xi to vi and xj to vj in V :
πG[[1xi op xj , ν]] = 1vi op vj = 1σ(vi) opσ(vj) = πσ⋆G[[1xi op xj , σ ⋆ ν]],
where we used that σ is a permutation.
• If φ(xi) = Pℓ(xi), then for a valuation µ mapping xi to vi in V :
πG[[Pℓ, µ]] = (col(vi))ℓ = (col σ(σ(vi))ℓ = πσ⋆G[[Pℓ, σ ⋆ ν]],
where we used the definition of colσ .
• Similarly, if φ(xi, xj) = E(xi, xj), then for a valuation ν assigning xi to vi and xj to vj :
πG[[φ, ν]] = 1vivj∈E = 1σ(vi)σ(vj)∈Eσ = πσ⋆G[[φ, σ ⋆ ν]],
where we used the definition of Eσ .
• If φ(x) = φ1(x1) · φ2(x2), then for a valuation ν from x to V :
πG[[φ, ν]] = πG[[φ1, ν]] · πG[[φ2, ν]] = πσ⋆G[[φ1, σ ⋆ ν]] · πσ⋆G[[φ2, σ ⋆ ν]] = πσ⋆G[[φ, σ ⋆ ν]],
where we used the induction hypothesis for φ1 and φ2. The cases φ(x) = φ1(x1) + φ2(x2) and φ(x) = a · φ1(x) are dealt with in a similar way. • If φ(x) = f(φ1(x1), . . . , φp(xp)), then
πG[[φ, ν]] = f(πG[[φ1, ν]], . . . , πG[[φp, ν]])
= f(πσ⋆G[[φ1, σ ⋆ ν]], . . . , πσ⋆G[[φp, σ ⋆ ν]])
= πσ⋆G[[φ, σ ⋆ ν]],
where we used again the induction hypothesis for φ1, . . . , φp. • Finally, if φ(x) = ∑
y φ1(x, y) then for a valuation ν of x to V : πG[[φ, ν]] = ∑ v∈V πG[[φ1, ν[y 7→ v]]] = ∑ v∈V πσ⋆G[[φ1, σ ⋆ ν[y 7→ v]]]
= ∑ v∈V σ πσ⋆G[[φ1, σ ⋆ ν[y 7→ v]]] = πσ⋆G[[φ, σ ⋆ ν]],
where we used the induction hypothesis for φ1 and that V σ = V because σ is a permutation.
We remark that when φ does not contain free index variables, then πG[[φ, ν]] = πG[[φ]] for any valuation ν, from which invariance follows from the previous arguments. This concludes the proof of Proposition 3.1.

heading: C DETAILS OF SECTION 4
text: In the following sections we prove Theorem 4.1, 4.2, 4.3 and 4.4. More specifically, we start by showing these results in the setting that TL(Ω) only supports summation aggregation ( ∑ x e) and in which the vertex-labellings in graphs take values in {0, 1}ℓ. In this context, we introduce classical logics in Section C.1 and recall and extend connections between the separation power of these logics and the separation power of color refinement and k-WL in Section C.2. We connect TL(Ω) and logics in Section C.3, to finally obtain the desired proofs in Section C.4. We then show how these results can be generalized in the presence of general aggregation operators in Section C.5, and to the setting where vertex-labellings take values in Rℓ in Section C.6.

heading: C.1 CLASSICAL LOGICS
text: In what follows, we consider graphs G = (VG, EG, colG) with colG : VG → {0, 1}ℓ. We start by defining the k-variable fragment Ck of first-order logic with counting quantifiers, followed by the definition of the guarded fragment GC of C2. Formulae φ in Ck are defined over the set {x1, . . . , xk} of variables and are formed by the following grammar:
φ := (xi = xj) | E(xi, xj) | Ps(xi) | ¬φ | φ ∧ φ | ∃≥mxi φ,
where i, j ∈ [k], E is a binary predicate, Ps for s ∈ [ℓ] are unary predicates for some ℓ ∈ N, and m ∈ N. The semantics of formulae in Ck is defined in terms of interpretations, denoted by πBG, relative to a given graph G and a (partial) valuation µ : {x1, . . . , xk} → VG. Such an interpretation maps formulae, graphs and valuations to Boolean values B := {⊥,⊤}, in a similar way as we did for tensor language expressions.
More precisely, given a graph G = (VG, EG, colG) and partial valuation µ : {x1, . . . , xk} → VG, we define πBG[[φ, µ]] ∈ B for valuations defined on the free variables in φ. That is, we define:
πBG[[xi = xj , µ]] := if µ(xi) = µ(xj) then ⊤ else ⊥; πBG[[E(xi, xj), µ]] := if µ(xi)µ(xj) ∈ EG then ⊤ else ⊥; πBG[[Ps(xi), µ]] := if colG(µ(xi))s = 1 then ⊤ else ⊥;
πBG[[¬φ, µ]] := ¬πBG[[φ, µ]]; πBG[[φ1 ∧ φ2, µ]] := πBG[[φ1, µ]] ∧ πBG[[φ2, µ]];
πBG[[∃≥mxi φ1, µ]] := if |{v ∈ VG | πBG[[φ, µ[xi 7→ v]]] = ⊤}| ≥ m then ⊤ else ⊥.
In the last expression, µ[xi 7→ v] denotes the valuation µ modified such that it maps xi to vertex v. We will also need the guarded fragment GC of C2 in which we only allow equality conditions of the form xi = xi, component expressions of conjunction and disjunction should have the same single free variable, and counting quantifiers can only occur in guarded form: ∃≥mx2(E(x1, x2)∧φ(x2)) or ∃≥mx1(E(x2, x1) ∧ φ(x1)). The semantics of formulae in GC is inherited from formulae in C2. Finally, we will also consider Ck∞ω , that is, the logic C
k extended with infinitary disjunctions and conjunctions. More precisely, we add to the grammar of formulae the following constructs:∨
α∈A φα and ∧ α∈A φα
where the index set A can be arbitrary, even containing uncountably many indices. We define GC∞ω in the same way by relaxing the finite variable conditions. The semantics is, as expected: πBG[[ ∨ α∈A φα, µ]] = ⊤ if for at least one α ∈ A, πBG[[φα, µ]] = ⊤, and πBG[[ ∧ α∈A φα, µ]] = ⊤ if for all α ∈ A, πBG[[φα, µ]] = ⊤. We define the free variables of formulae just as for TL, and similarly, quantifier rank is defined as summation depth (only existential quantifications increase the quantifier rank). For any of the above logics L we define L(t) as the set of formulae in L of quantifier rank at most t. To capture the separation power of logics, we define ρ1 ( L(t) )
as the equivalence relation on G1 defined by (
(G, v), (H,w) ) ∈ ρ1 ( L(t) ) ⇐⇒ ∀φ(x) ∈ L(t) : πBG[[φ, µv]] = πBH [[φ, µw]],
where µv is any valuation such that µ(x) = v, and likewise for w. The relation ρ0 is defined in a similar way, except that now the relation is only over pairs of graphs, and the characterization is over all formulae with no free variables (also called sentences). Finally, we also use, and define, the relation ρs, which relates pairs from Gs: consisting of a graph and an s-tuple of vertices. The relation is defined as(
(G,v), (H,w) ) ∈ ρs ( L(t) ) ⇐⇒ ∀φ(x) ∈ L(t) : πBG[[φ, µv]] = πBH [[φ, µw]],
where x consist of s free variables and µv is a valuation assigning the i-th variable of x to the i-th value of v, for any i ∈ [s].

heading: C.2 CHARACTERIZATION OF SEPARATION POWER OF LOGICS
text: We first connect the separation power of the color refinement and k-dimensional Weisfeiler-Leman algorithms to the separation power of the logics we just introduced. Although most of these connections are known, we present them in a bit of a more fine-grained way. That is, we connect the number of rounds used in the algorithms to the quantifier rank of formulae in the above logics. Proposition C.1. For any t ≥ 0, we have the following identities:
(1) ρ1 ( cr(t) ) = ρ1 ( GC(t) ) and ρ0 ( gcr(t) ) = ρ0 ( gwl(t)1 ) = ρ0 ( C2,(t+1) ) ;
(2) For k ≥ 1, ρ1 ( vwl(t)k ) = ρ1 ( Ck+1,(t) ) and
ρ0 ( Ck+1,(t+k) ) ⊆ ρ0 ( gwl(t)k ) ⊆ ρ0 ( Ck+1,(t+1) ) .
As a consequence, ρ0 ( gwl (∞) k ) = ρ0 ( Ck+1 ) .
Proof. For (1), the identity ρ1 ( cr(t) ) = ρ1 ( GC(t) ) is known and can be found, for example, in
Theorem V.10 in Grohe (2021). The identity ρ0 ( gcr(t) ) = ρ0 ( gwl(t)1 ) can be found in Proposition
V.4 in Grohe (2021). The identity ρ0 ( gwl(t)1 ) = ρ0 ( C2,(t+1) ) is a consequence of the inclusion shown in (2) for k = 1.
For (2), we use that ρk ( wl(t)k ) = ρk ( Ck+1,(t) ) , see e.g., Theorem V.8 in Grohe (2021). We argue
that this identity holds for ρ1 ( vwl(t)k ) = ρ1 ( Ck+1,(t) ) . Indeed, suppose that (G, v) and (H,w) are
not in ρ1 ( Ck+1,(t) ) . Let φ(x1) be a formula in Ck+1,(t) such that πBG[[φ, v]] ̸= πBH [[φ,w]]. Con-
sider the formula φ+(x1, . . . , xk) = φ(x1) ∧ ∧k i=1(x1 = xi). Then, π B G[[φ
+, (v, . . . , v)]] ̸= πBH [[φ +, (w, . . . , w)]], and hence (G, (v, . . . , v)) and (H, (w, . . . , w)) are not in ρk ( Ck+1,(t) ) either. This implies that wl(t)k (G, (v, . . . , v)) ̸= wl (t) k (H, (w, . . . , w)), and thus, by definition, vwl(t)k (G, v) ̸= vwl (t) k (H,w). In other words, (G, v) and (H,w) are not in ρ1 ( vwl(t)k ) , from which
the inclusion ρ1 ( vwl(t)k ) ⊆ ρ1 ( Ck+1,(t) ) follows. Conversely, if (G, v) and (H,w) are not in
ρ1 ( vwl(t)k ) , then wl(t)k (G, (v, . . . , v)) ̸= wl (t) k (H, (w, . . . , w)). As a consequence, (G, (v, . . . , v))
and (H, (w, . . . , w)) are not in ρk ( Ck+1,(t) ) either. Let φ(x1, . . . , xk) be a formula in Ck+1,(t) such that πBG[[φ, (v, . . . , v)]] ̸= πBH [[φ, (w, . . . , w)]]. Then it is readily shown that we can convert φ(x1, . . . , xk) into a formulaφ−(x1) in Ck+1,(t) such that πBG[[φ
−, v]] ̸= πBH [[φ−, w]], and thus (G, v) and (H,w) are not in ρ1 ( Ck+1,(t) ) . Hence, we also have the inclusion ρ1 ( vwl(t)k ) ⊇ ρ1 ( Ck+1,(t) ) , form which the first identity in (2) follows.
It remains to show ρ0 ( Ck+1,(t+k) ) ⊆ ρ0 ( gwl(t)k ) ⊆ ρ0 ( Ck+1,(t+1) ) . Clearly, if (G,H) is not
in ρ0 ( gwl(t)k ) then the multisets of labels wl(t)k (G,v) and wl (t) k (H,w) differ. It is known that with each label c one can associate a formula φc in Ck+1,(t) such that πBG[[φ c,v]] = ⊤ if and only if wl(t)k (G,v) = c. So, if the multisets are different, there must be a c that occurs more often in one multiset than in the other one. This can be detected by a fomulae of the form ∃=m(x1, . . . , xk)φc(x1, . . . , xk) which is satisfied if there are m tuples v with label c. It is now easily verified that the latter formula can be converted into a formula in Ck+1,(t+k). Hence, the inclusion ρ0 ( Ck+1,(t+k) ) ⊆ ρ0 ( gwl(t)k ) follows.
For ρ0 ( gwl(t)k ) ⊆ ρ0 ( Ck+1,(t+1) ) , we show that if (G,H) is in ρ0 ( gwl(t)k ) , then this implies that πBG[[φ, µ]] = π B H [[φ, µ]] for all formulae in C
k+1,(t+1) and any valuation µ (notice that µ is superfluous in this definition when formulas have no free variables). Assume that (G,H) is in ρ0(gwl(t)k ). Since any formula of quantifier rank t+1 is a Boolean combination of formulas of less rank or a formula of the form φ = ∃≥mxi ψ where ψ is of quantifier rank t, without loss of generality consider a formula of the latter form, and assume for the sake of contradiction that πBG[[φ, µ]] = ⊤ but πBH [[φ, µ]] = ⊥. Since πBG[[φ, µ]] = ⊤, there must be at least m elements satisfying ψ. More precisely, let v1, . . . , vp in G be all vertices in G such that for each valuation µ[x 7→ vi] it holds that πBG[[ψ, µ[x 7→ vi]] = ⊤. As mentioned, it must be that p is at least m. Using again the fact that ρk ( wl(t)k ) = ρk ( Ck+1,(t) ) , we infer that the color wl(t−1)k (G, (vi, . . . , vi)) is the same, for each such vi.
Now since gwl(t−1)k (G) = gwl (t−1) k (H), it is not difficult to see that there must be exactly p vertices w1, . . . , wp in H such that wl (t−1) k (G, (vi, . . . , vi)) = wl (t−1) k (H, (wi, . . . , wi)). Otherwise, it would simply not be the case that the aggregation step of the colors, assigned by k-WL is the same in G and H . By the connection to logic, we again know that for valuation µ[x 7→ wi] it holds that πBH [[ψ, µ[x 7→ wi]] = ⊤. It then follows that πBH [[φ, µ]] = ⊤ for any valuation µ, which was to be shown.
Finally, we remark that ρ0 ( gwl (∞) k ) = ρ0 ( Ck+1 ) follows from the preceding inclusions in (2).
Before moving to tensor languages, where we will use infinitary logics to simulate expressions in TLk(Ω) and GTL(Ω), we recall that, when considering the separation power of logics, we can freely move between the logics and their infinitary counterparts:
Theorem C.2. The following identities hold for any t ≥ 0, k ≥ 2 and s ≥ 0:
(1) ρ1 ( GC(t)∞ω ) = ρ1 ( GC(t) ) ;
(2) ρs ( Ck,(t)∞ω ) = ρs ( Ck,(t) ) .
Proof. For identity (1), notice that we only need to prove that ρ1 ( GC(t) ) ⊆ ρ1 ( GC(t)∞ω ) , the other direction follows directly from the definition. We point out the well-known fact that two tuples (G, v) and (H,w) belong to ρ1 ( GC(t) ) if and only if the unravelling of G rooted at v up to depth t is isomorphic to the unravelling of H rooted at w up to root t. Here the unravelling is the infinite tree whose root is the root node, and whose children are the neighbors of the root node (see e.g. Barceló et al. (2020); Otto (2019). Now for the connection with infinitary logic. Assume that the unravellings of G rooted at v and of H rooted at w up to level t are isomorphic, but assume for the sake of contradiction that there is a formula φ(x) in GC(t)∞ω such that π B G[[φ, µv]] ̸= πBH [[φ, µw]], where µv and µw are any valuation mapping variable x to v and w, respectively. Now sinceG andH are finite graphs, one can construct, from formula ϕ, a formula ϕ′ in GC(t) such that πBG[[ψ, µv]] ̸= πBH [[ψ, µw]]. Notice that this is in contradiction with our assumption that unravellings where isomorphic and therefore indistinguishable by formulae in GC(t). To construct ψ, consider an infinitary disjunction∨
a∈A αa. Since G and H have a finite number of vertices, and the formulae have a finite number of variables, the number of different valuations from the variables to the vertices in G or H is also finite. Thus, one can replace any extra copy of αa, αa′ such that their value is the same in G and H . The final result is a finite disjunction, and the truth value over G and H is equivalent to the original infinitary disjunction.
For identity (2) we refer to Corollary 2.4 in Otto (2017).
C.3 FROM TL(Ω) TO Ck∞ω AND GC∞ω
We are now finally ready to make the connection between expressions in TL(Ω) and the infinitary logics introduced earlier.
Proposition C.3. For any expression φ(x) in TLk(Ω) and c ∈ R, there exists an expression φ̃c(x) in Ck∞ω such that πG[[φ,v]] = c if and only if π B G[[φ̃
c,v]] = ⊤ for any graph G = (VG, EG, colG) in G and v ∈ V kG . Furthermore, if φ(x) ∈ GTL(Ω) then φ̃c ∈ GC∞ω . Finally, if φ has summation depth t then φ̃c has quantifier rank t.
Proof. We define φ̃c inductively on the structure of expressions in TLk(Ω).
• φ(xi, xj) := 1xi op xj . Assume first that op is “=”. We distinguish between (a) i ̸= j and (b) i = j. For case (a), if c = 1, then we define φ̃1(xi, xj) := (xi = xj), if c = 0, then we define φ̃0(xi, xj) := ¬(xi = xj), and if c ̸= 0, 1, then we define φ̃c(xi, xj) := xi ̸= xi. For case (b), if c = 1, then we define φ̃1(xi, xj) := (xi = xi), and for any c ̸= 1, we define φ̃c(xi, xj) := ¬(xi = xi). The case when op is “̸=” is treated analogously.
• φ(xi) := Pℓ(xi). If c = 1, then we define φ̃1(xi) := Pℓ(xi), if c = 0, then we define φ̃0(xi) := ¬Pj(xi). For all other c, we define φ̃c(xi, xj) := ¬(xi = xi).
• φ(xi, xj) := E(xi, xj). If c = 1, then we define φ̃1(xi, xj) := E(xi, xj), if c = 0, then we define φ̃0(xi, xj) := ¬E(xi, xj). For all other c, we define φ̃c(xi, xj) := ¬(xi = xi).
• φ := φ1 + φ2. We observe that πG[[φ,v]] = c if and only if there are c1, c2 ∈ R such that πG[[φ1,v]] = c1 and πG[[φ2,v]] = c2 and c = c1 + c2. Hence, it suffices to define
φ̃c := ∨
c1,c2∈R c=c1+c2
φ̃c11 ∧ φ̃ c2 2 ,
where φ̃c11 and φ̃ c2 2 are the expressions such that πG[[φ1,v]] = c1 if and only if πBG[[φ̃ c1 1 ,v]] = ⊤ and πG[[φ2,v]] = c2 if and only if πBG[[φ̃ c2 2 ,v]] = ⊤, which exist by induction.
• φ := φ1 · φ2. This is case is analogous to the previous one. Indeed, πG[[φ,v]] = c if and only if there are c1, c2 ∈ R such that πG[[φ1,v]] = c1 and πG[[φ2,v]] = c2 and c = c1 · c2. Hence, it suffices to define
φ̃c := ∨
c1,c2∈R c=c1·c2
φ̃c11 ∧ φ̃ c2 2 .
• φ := a · φ1. This is case is again dealt with in a similar way. Indeed, πG[[φ,v]] = c if and only if there is a c1 ∈ R such that πG[[φ1,v]] = c1 and c = a · c1. Hence, it suffices to define
φ̃c := ∨ c1∈R c=a·c1 φ̃c11 .
• φ := f(φ1, . . . , φp) with f : Rp → R. We observe that πG[[φ,v]] = c if and only if there are c1, . . . , cp ∈ R such that c = f(c1, . . . , cp) and πG[[φi,v]] = ci for i ∈ [p]. Hence, it suffices to define
φ̃c := ∨
c1,...,cp∈R c=f(c1,...,cp)
φ̃c11 ∧ · · · ∧ φ̃cpp .
• φ := ∑
xi φ1. We observe that πG[[φ, µ]] = c implies that we can partition VG into ℓ parts
V1, . . . , Vℓ, of sizes m1, . . . ,mℓ, respectively, such that πG[[φ1, µ[xi → v]]] = ci for each v ∈ Vi, and such that all ci’s are pairwise distinct and c = ∑ℓ i=1 ci ·mi. It now suffices to consider the following formula
φ̃c := ∨
ℓ,m1,...,mℓ∈N c1,...,cℓ∈R
c= ∑ℓ
i=1 mici
ℓ∧ i=1 ∃=mixi φ̃ci1 ∧ ∀xi ℓ∨ i=1 φ̃ci1 ,
where ∃=mixi ψ is shorthand notation for ∃≥mixi ψ ∧ ¬∃≥mi+1xi ψ, and ∀xi ψ denotes ¬∃≥1xi ¬ψ.
This concludes the construction of φ̃c. We observe that we only introduce a quantifiers when φ =∑ xi φ1 and hence if we assume by induction that summation depth and quantifier rank are in sync, then if φ1 has summation depth t−1 and thus φ̃c1 has quantifier rank t−1 for any c ∈ R, then φ has summation depth t, and as can be seen from the definition of φ̃c, this formula has quantifier rank t, as desired.
It remains to verify the claim about guarded expressions. This is again verified by induction. The only case requiring some attention is φ(x1) := ∑ x2 E(x1, x2) ∧ φ1(x2) for which we can define
φ̃c := ∨
ℓ,m1,...,mℓ∈N c1,...,cℓ∈R
c= ∑ℓ
i=1 mici m= ∑ℓ
i=1 mi
∃=mx2E(x1, x2) ∧ ℓ∧
i=1
∃=mix2E(x1, x2) ∧ φ̃ci1 (x2),
which is a formula in GC again only adding one to the quantifier rank of the formulae φ̃c1 for c ∈ R. So also here, we have the one-to-one correspondence between summation depth and quantifier rank.
C.4 PROOF OF THEOREM 4.1, 4.2, 4.3 AND 4.4
Proposition C.4. We have the following inclusions: For any t ≥ 0 and any collection Ω of functions:
• ρ1 ( cr(t) ) ⊆ ρ1 ( GTL(t)(Ω) ) ;
• ρ1 ( vwl(t)k ) ⊆ ρ1 ( TL(t)k+1(Ω) ) ; and
• ρ0 ( gwl(t)k ) ⊆ ρ0 ( TL (t+1) k+1 (Ω) ) .
Proof. We first show the second bullet by contraposition. That is, we show that if (G, v) and (H,w) are not in ρ1 ( TL(t)k+1(Ω) ) , then neither are they in ρ1 ( vwl(t)k ) . Indeed, suppose that there exists an expression φ(x1) in TL(t)k+1(Ω) such that πG[[φ, v]] = c ̸= c′ = πH [[φ,w]]. From Proposition C.3 we know that there exists a formula φ̃c in Ck+1,(t)∞ω such that π B G[[φ̃
c, v]] = ⊤ and πBH [[φ̃c, w]] = ⊥. Hence, (G, v) and (H,w) do no belong to ρ1 ( Ck+1,(t)∞ω ) . Theorem C.2 implies that (G, v) and (H,w) also
do not belong to ρ1 ( Ck+1,(t) ) . Finally, Proposition C.1 implies that (G, v) and (H,w) do not belong
to ρ1(vwl(t)k ) , as desired. The third bullet is shown in precisely the same, but using the identities for ρ0 rather than ρ1, and gwl(t)k rather than vwl (t) k .
Also the first bullet is shown in the same way, using the connection between GTL(t)(Ω), GC2,(t)∞ω , GC(t) and cr(t), as given by Proposition C.1, Theorem C.2, and Proposition C.3.
We next show that our tensor languages are also more separating than the color refinement and k-dimensional Weisfeiler-Leman algorithms.
Proposition C.5. We have the following inclusions: For any t ≥ 0 and any collection Ω of functions:
• ρ1 ( GTL(t)(Ω) ) ⊆ ρ1 ( cr(t) ) ;
• ρ1 ( TL(t)k+1(Ω) ) ⊆ ρ1 ( vwl(t)k ) ; and
• ρ0 ( TL (t+k) k+1 (Ω) ) ⊆ ρ0 ( gwl(t)k ) .
Proof. For any of these inclusions to hold, for any Ω, we need to show the inclusion without the use of any functions. We again use the connections between the color refinement and k-dimensional Weisfeiler-Leman algorithms and finite variable logics as stated in Proposition C.1. More precisely, we show for any formula φ(x) ∈ Ck,(t) there exists an expression φ̂(x) ∈ TL(t)k such that for any graph G in G, πBG[[φ,v]] = ⊤ implies πG[[φ̂,v]] = 1 and πBG[[φ,v]] = ⊥ implies πG[[φ̂,v]] = 0. By appropriately selecting k and t and by observing that when φ(x) ∈ GC then φ̂(x) ∈ GTL, the inclusions follow.
The construction of φ̂(x) is by induction on the structure of formulae in Ck.
• φ := (xi = xj). Then, we define φ̂ := 1xi=xj .
• φ := Pℓ(xi). Then, we define φ̂ := Pℓ(xi).
• φ := E(xi, xj). Then, we define φ̂ := E(xi, xj).
• φ := ¬φ1. Then, we define φ̂ := 1xi=xi − φ̂1.
• φ := φ1 ∧ φ2. Then, we define φ̂ := φ̂1 · φ̂2. • φ := ∃≥mxi φ1. Consider a polynomial p(x) := ∑ j ajx j such that p(x) = 0 for x ∈
{0, 1, . . . ,m− 1} and p(x) = 1 for x ∈ {m,m+ 1, . . . , n}. Such a polynomial exists by interpolation. Then, we define φ̂ := ∑ j aj (∑ xi φ̂1 )j .
We remark that we here crucially rely on the assumption that G contains graphs of fixed size n and that TLk is closed under linear combinations and product. Clearly, if φ ∈ GC, then the above translations results in an expression φ̂ ∈ GTL(Ω). Furthermore, the quantifier rank of φ is in oneto-one correspondence to the summation depth of φ̂.
We can now apply Proposition C.1. That is, if (G, v) and (H,w) are not in ρ1 ( cr(t) )
then by Proposition C.1, there exists a formula φ(x) in GC(t) such that πBG[[φ, v]] = ⊤ ≠ πBH [[φ,w]] = ⊥. We have just shown when we consider φ̃, in GTL(t), also πG[[φ̃, v]] ̸= πH [[φ̃, w]] holds. Hence, (G, v) and (H,w) are not in ρ1 ( GTL(t)(Ω) ) either, for any Ω. Hence, ρ1 ( GTL(t)(Ω) ) ⊆ ρ1 ( cr(t) )
holds. The other bullets are shown in the same way, again by relying on Proposition C.1 and using that we can move from vwl(t)k and gwl (t) k to logical formulae, and to expressions in TL (t) k+1 and TL (t+k) k+1 , respectively, to separate (G, v) from (H,w) or G from H , respectively.
Theorems 4.1, 4.2, 4.3 and 4.4 now follow directly from Propositions C.4 and C.5.

heading: C.5 OTHER AGGREGATION FUNCTIONS
text: As is mentioned in the main paper, our upper bound results on the separation power of tensor languages (and hence also of GNNs represented in those languages) generalize easily when other aggregation functions than summation are used in TL expressions.
To clarify what we understand by an aggregation function, let us first recall the semantics of summation aggregation. Let φ := ∑ xi φ1, where ∑ xi
represents summation aggregation, let G = (VG, EG, colG) be a graph, and let ν be a valuation assigning index variables to vertices in VG. The semantics is then given by:
πG[[ ∑
xi φ1, ν]] := ∑ v∈VG πG[[φ1, ν[xi 7→ v]]],
as explained in Section 3. Semantically, we can alternatively view ∑
xi φ1 as a function which takes
the sum of the elements in the following multiset of real values:
{{πG[[φ1, ν[xi 7→ v]]] | v ∈ VG}}.
One can now consider, more generally, an aggregation function F as a function which assigns to any multiset of values in R a single real value. For example, F could be max, min, mean, . . .. Let Θ be such a collection of aggregation functions.
We extend the syntax of TL(Ω) to expressions in TL(Ω,Θ) by generalizing the construct ∑
xi φ in
the grammar of TL(Ω) expression to the construct
aggr(F, xi, φ)
with the obvious semantics:4
πG[[aggr(F, xi, φ), µ]] := F ({{πG[[φ, µ[xi 7→ v]]] | v ∈ VG}}) .
The notion of free index variables for expressions in TL(Ω,Θ) is defined as before, where now free(aggr(F, xi, φ)) := free(φ) \ {xi}. Moreover, summation depth is replaced by the notion of aggregation depth, agd(φ), defined in the same way as summation depth except that agd(aggr(F, xi, φ)) := agd(φ) + 1. Similarly, the fragments TLk(Ω,Θ) and its aggregation depth restricted fragment TL(t)k (Ω,Θ) are defined as before, using aggregation depth rather than summation depth. For the guarded fragment, GTL(Ω,Θ), expressions are now restricted such that aggregations must occur in a guarded form aggr(F, xj , E(xi, xj) ·φ(xj)) just as before. That is, aggregation only happens on multisets of values obtained from neighboring vertices.
We now argue that our upper bound results on the separation power remain valid for the extension TL(Ω,Θ) of TL(Ω) with arbitrary aggregation functions Θ. Proposition C.6. We have the following inclusions: For any t ≥ 0, any collection Ω of functions and any collection Θ of aggregation functions:
• ρ1 ( cr(t) ) ⊆ ρ1 ( GTL(t)(Ω,Θ) ) ;
• ρ1 ( vwl(t)k ) ⊆ ρ1 ( TL(t)k+1(Ω,Θ) ) ; and
4We remark that we can also consider aggregation functions F over multisets of values in Rℓ for some ℓ ∈ N. This requires extending the syntax with aggr(F, x1, . . . , xℓ, φ) and with semantics F ({{πG[[φ, ν[(x1, . . . , xℓ) 7→ (v1, . . . , vℓ)]]] | v1, . . . , vℓ ∈ VG}}).
• ρ0 ( gwl(t)k ) ⊆ ρ0 ( TL (t+1) k+1 (Ω,Θ) ) .
Proof. It suffices to show that Proposition C.3 also holds for expressions in the fragments of TL(Ω,Θ) considered. In particular, we only need to revise the case of summation aggregation (that is, φ := ∑ xi φ1) in the proof of Proposition C.3. Indeed, let us consider the more general case:
• φ := aggr(F, xi, φ1). We then define
φ̃c := ∨ ℓ∈N ∨ (m1,...,mℓ)∈Nℓ ∨ (c,c1,...,cℓ)∈C(m1,...,mℓ,F ) ℓ∧ i=1 ∃=mi xiφ̃ci1 ∧ ∀xi ℓ∨ i=1 φ̃ci1 ,
where C(m1, . . . ,mℓ, F ) now consists of all (c, c1, . . . , cℓ) ∈ Rℓ+1 such that c = F ( {{c1, . . . , c1︸ ︷︷ ︸
m1 times , . . . , cℓ, . . . , cℓ︸ ︷︷ ︸ mℓ times
}} ) .
It is readily verified that πG[[aggr(F, xi, φ1),v]] = c iff πBG[[φ̃ c,v]] = ⊤, as desired. The case of guarded expressions is dealt with similarly.
We will illustrate later on (Section D) that this generalization allows for assessing the separation power of GNNs that use a variety of aggregation functions.
The choice of supported aggregation functions has, of course, an impact on the ability of TL(Ω,Θ) to match color refinement or the k-WL procedures in separation power. The same holds for GNNs, as shown by Xu et al. (2019). And indeed, the proof of Proposition C.5 relies on the presence of summation aggregation. We note that most lower bounds on the separation power of GNNs in terms of color refinement or the k-WL procedures assume summation aggregation since summation suffices to construct injective sum-decomposable functions on multisets (Xu et al., 2019; Zaheer et al., 2017), which are used to simulate color refinement and k-WL. A more in-depth analysis of lower bounding GNNs with less expressive aggregation functions, possibly using weaker versions of color refinement and k-WL is left as future work.

heading: C.6 GENERALIZATION TO GRAPHS WITH REAL-VALUED VERTEX LABELS
text: We next consider the more general setting in which colG : VG → Rℓ for some ℓ ∈ N. That is, vertices in a graph can carry real-valued vectors. We remark that no changes to neither the syntax nor the semantics of TL expressions are needed, yet note that πG[[Ps(x), ν]] := colG(ν)s is now an element in R rather than 0 or 1, for each s ∈ [ℓ]. A first observation is that the color refinement and k-WL procedures treat each real value as a separate label. That is, two values that differ only by any small ϵ > 0, are considered different. The proofs of Theorem 4.1, 4.2, 4.3 and 4.4 rely on connections between color refinement and k-WL and the finite variable logics GC and Ck+1, respectively. In the discrete context, the unary predicates Ps(x) used in the logical formulas indicate which label vertices have. That is, πBG[[Ps, v]] = ⊤ iff colG(v)s = 1. To accommodate for real values in the context of separation power, these logics now need to be able to differentiate between different labels, that is, different real numbers. We therefore extend the unary predicates allowed in formulas. More precisely, for each dimension s ∈ [ℓ], we now have uncountably many predicates of the form Ps,r, one for each r ∈ R. In any formula in GC or Ck+1 only a finite number of such predicates may occur. The Boolean semantics of these new predicates is as expected:
πBG[[Ps,r(x), ν]] := if colG(µ(xi))s = r then ⊤ else ⊥.
In other words, in our logics, we can now detect which real-valued labels vertices have. Although, in general, the introduction of infinite predicates may cause problems, we here consider a specific setting in which the vertices in a graph have a unique label. This is commonly assumed in graph learning. Given this, it is easily verified that all results in Section C.2 carry over, where all logics involved now use the unary predicates Ps,r with s ∈ [ℓ] and r ∈ R.
The connection between TL and logics also carries over. First, for Proposition C.3 we now need to connect TL expressions, that use a finite number of predicates Ps, for s ∈ [ℓ], with the extended logics having uncountably many predicates Ps,r, for s ∈ [ℓ] and r ∈ R, at their disposal. It suffices to reconsider the case φ(xi) = Ps(xi) in the proof of Proposition C.3. More precisely, πG[[Ps(xi), ν]] can now be an arbitrary value c ∈ R. We now simply define φ̃c(xi) := Ps,c(xi). By definition πG[[Ps(xi), ν]] = c if and only if πBG[[Ps,c(xi), ν]] = ⊤, as desired. The proof for the extended version of proposition C.5 now needs a slightly different strategy, where we build the relevant TL formula after we construct the contrapositive of the Proposition. Let us first show how to construct a TL formula that is equivalent to a logical formula on any graph using only labels in a specific (finite) set R of real numbers.
In other words, given a setR of real values, we show that for any formula φ(x) ∈ Ck,(t) using unary predicates Ps,r such that r ∈ R, we can construct the desired φ̂. As mentioned, we only need to reconsider the case φ(xi) := Ps,r(xi). We define
φ̂ := 1∏ r′∈R,r ̸=r′ r − r′ ∏ r′∈R,r ̸=r′ (Ps(xi)− r′1xi=xi).
Then, πG[[φ̂, ν]] evaluates to∏ r′∈R,r ̸=r′(r − r′)∏ r′∈R,r ̸=r′(r − r′) = { 1 [[Ps,r, ν]] = ⊤ 0 [[Ps,r, ν]] = ⊥ .
Indeed, if [[Ps,r, ν]] = ⊤, then colG(v)s = r and hence πG[[Ps, v]] = r, resulting in the same nominator and denominator in the above fraction. If [[Ps,r, ν]] = ⊥, then colG(v)s = r′ for some value r′ ∈ R with r ̸= r′. In this case, the nominator in the above fraction becomes zero. We remark that this revised construction still results in a guarded TL expression, when the input logical formula is guarded as well.
Coming back to the proof of the extended version of Proposition C.5, let us show the proof for the the fact that ρ1 ( GTL(t)(Ω) ) ⊆ ρ1 ( cr(t) ) , the other two items being analogous. Assume that there is
a pair (G, v) and (H,w) which is not in ρ1 ( cr(t) ) . Then, by Proposition C.1, applied on graphs with real-valued labels, there exists a formula φ(x) in GC(t) such that πBG[[φ, v]] = ⊤ ≠ πBH [[φ,w]] = ⊥. We remark that φ(x) uses finitely many Ps,r predicates. Let R be the set of real values used in both G and H (and φ(x)). We note that R is finite. We invoke the construction sketched above, and obtain a formula φ̂ in GTL(t) such that πG[[φ̃, v]] ̸= πH [[φ̃, w]]. Hence, (G, v) and (H,w) is not in ρ1 ( GTL(t)(Ω) ) either, for any Ω, which was to be shown.

heading: D DETAILS OF SECTION 5
text: We here provide some additional details on the encoding of layers of GNNs in our tensor languages, and how, as a consequence of our results from Section 4, one obtains a bound on their separation power. This section showcases that it is relatively straightforward to represent GNNs in our tensor languages. Indeed, often, a direct translation of the layers, as defined in the literature, suffices.

heading: D.1 COLOR REFINEMENT
text: We start with GNN architectures related to color refinement, or in other words, architectures which can be represented in our guarded tensor language.
GraphSage. We first consider a “basic” GNN, that is, an instance of GraphSage (Hamilton et al., 2017) in which sum aggregation is used. The initial features are given by F (0) = (f (0)1 , . . . ,f (0) d0 ) where f (0)i ∈ Rn×1 is a hot-one encoding of the ith vertex label in G. We can represent the initial embedding easily in GTL(0), without the use of any summation. Indeed, it suffices to define φ(0)i (x1) := Pi(x1) for i ∈ [d0]. We have F (0) vj = πG[[φ (0) j , v]] for j ∈ [d0], and thus the initial features can be represented by simple expressions in GTL(0).
Assume now, by induction, that we can also represent the features computed by a basic GNN in layer t − 1. That is, let F (t−1) ∈ Rn×dt−1 be those features and for each i ∈ [dt−1] let φ(t−1)i (x1) be expressions in GTL(t−1)(σ) representing them. We assume that, for each i ∈ [dt−1], F (t−1)vi = πG[[φ (t−1) i , v]]. We remark that we assume that a summation depth of t− 1 is needed for layer t− 1.
Then, in layer t, a basic GNN computes the next features as F (t) := σ ( F (t−1) · V (t) +A · F (t−1) ·W (t) +B(t) ) ,
where A ∈ Rn×n is the adjacency matrix of G, V (t) and W (t) are weight matrices in Rdt−1×dt , B(t) ∈ Rn×dt is a (constant) bias matrix consist of n copies of b(t) ∈ Rdt , and σ is some activation function. We can simply use the following expressions φ(t)j (x1), for j ∈ [dt]:
σ (dt−1∑ i=1 V (t)ij · φ (t−1) i (x1) ) + ∑ x2 ( E(x1, x2) · (dt−1∑ i=1 W (t)ij · φ (t−1) i (x2) )) + b(t)j · 1x1=x1  . Here, W (t)ij , V (t) ij and b (t) j are real values corresponding the weight matrices and bias vector in layer t. These are expressions in GTL(t)(σ) since the additional summation is guarded, and combined with the summation depth of t − 1 of φ(t−1)i , this results in a summation depth of t for layer t. Furthermore, F (t)vi = πG[[φ (t) i , v]], as desired. If we denote by bGNN
(t) the class of t-layered basic GNNs, then our results imply
ρ1 ( cr(t) ) ⊆ ρ1( ( GTL(t)(Ω) ) ⊆ ρ1 ( bGNN(t) ) ,
and thus the separation power of basic GNNs is bounded by the separation power of color refinement. We thus recover known results by Xu et al. (2019) and Morris et al. (2019).
Furthermore, if one uses a readout layer in basic GNNs to obtain a graph embedding, one typically applies a function ro : Rdt → Rdt in the form of ro (∑ v∈VG F (t) v ) , in which aggregation takes places over all vertices of the graph. This corresponds to an expression in TL(t+1)2 (σ, ro): φj := roj (∑ x1 φ(t−1)j (x1) ) , where roj is the projection of the readout function on the jthe coordinate. We note that this is indeed not a guarded expression anymore, and thus our results tell that
ρ0 ( gcr(t) ) ⊆ ρ0(TL(t+1)2 (Ω) ) ⊆ ρ0 ( bGNN(t) + readout ) .
More generally, GraphSage allows for the use of general aggregation functions F on the multiset of features of neighboring vertices. To cast the corresponding layers in TL(Ω), we need to consider the extension TL(Ω,Θ) with an appropriate set Θ of aggregation functions, as described in Section C.5. In this way, we can represent layer t by means of the following expressions φ(t)j (x1), for j ∈ [dt]:
σ (dt−1∑ i=1 V (t)ij · φ (t−1) i (x1) ) + dt−1∑ i=1 W (t)ij · aggr ( F, x2, E(x1, x2) · φ(t−1)i (x2) ) + b(t)j · 1x1=x1  . which is now an expression in GTL(t)({σ},Θ) and hence the bound in terms of t iterations of color refinement carries over by Proposition C.6. Here, Θ simply consists of the aggregation functions used in the layers in GraphSage.
GCNs. Graph Convolution Networks (GCNs) (Kipf & Welling, 2017) operate alike basic GNNs except that a normalized Laplacian D−1/2(I + A)D−1/2 is used to aggregate features, instead of the adjacency matrix A. Here, D−1/2 is the diagonal matrix consisting of reciprocal of the square root of the vertex degrees in G plus 1. The initial embedding F (0) is just as before. We use again dt to denote the number of features in layer t. In layer t > 0, a GCN computes F (t) := σ(D−1/2(I + A)D−1/2 · F (t−1)W (t) + B(t)). If, in addition to the activation function σ we add the function 1√
x+1 : R → R : x 7→ 1√ x+1 to Ω, we can represent the GCN layer, as follows. For
j ∈ [dt], we define the GTL(t+1)(σ, 1√x+1 ) expressions
φ(t)j (x1) := σ ( f1/ √ x+1 (∑ x2 E(x1, x2) ) · (dt−1∑ i=1 W (t)ij · φ (t−1) i (x1) ) · f1/√x+1 (∑ x2 E(x1, x2) )
+f1/ √ x+1 (∑ x2 E(x1, x2) ) · (∑ x2 E(x1, x2)·f1/√x+1 (∑ x1 E(x2, x1) ) · (dt−1∑ i=1 W (t)ij ·φ (t−1) i (x2) )) ,
where we omitted the bias vector for simplicity. We again observe that only guarded summations are needed. However, we remark that in every layer we now add two the overall summation depth, since we need an extra summation to compute the degrees. In other words, a t-layered GCN correspond to expressions in GTL(2t)(σ, 1√
x+1 ). If we denote by GCN(t) the class of t-layered GCNs, then our
results imply ρ1 ( cr(2t) ) ⊆ ρ1 ( GTL(2t)(Ω) ) ⊆ ρ1 ( GCN(t) ) .
We remark that another representation can be provided, in which the degree computation is factored out (Geerts et al., 2021a), resulting in a better upper bound ρ1 ( cr(t+1) ) ⊆ ρ1 ( GCN(t) ) . In a similar
way as for basic GNNs, we also have ρ0 ( gcr(t+1) ) ⊆ ρ0 ( GCN(t) + readout ) .
SGCs. As an other example, we consider a variation of Simple Graph Convolutions (SGCs) (Wu et al., 2019), which use powers the adjacency matrix and only apply a non-linear activation function at the end. That is, F := σ(Ap ·F (0) ·W ) for some p ∈ N and W ∈ Rd0×d1 . We remark that SGCs actually use powers of the normalized Laplacian, that is, F := σ ( (D−1/2(I + AG)D
−1/2))p · F (0) · W ) but this only incurs an additional summation depth as for GCNs. We focus here on our simpler version. It should be clear that we can represent the architecture in TL(p)p+1(Ω) by means of the expressions:
φ(t)j (x1) := σ ∑ x2 · · · ∑ xp+1 p∏ k=1 E(xk, xk+1) · ( d0∑ i=1 Wij · φ(0)i (xp+1) ) ,
for j ∈ [d1]. A naive application of our results would imply an upper bound on their separation power by p-WL. We can, however, use Proposition 4.5. Indeed, it is readily verified that these expressions have a treewidth of one, because the variables form a path. And indeed, when for example, p = 3, we can equivalently write φ(t)j (x1) as
σ (∑ x2 E(x1, x2) · (∑ x1 E(x2, x1) · (∑ x2 E(x1, x2) · ( d0∑ i=1 Wij · φ(0)i (x2) )))) ,
by reordering the summations and reusing index variables. This holds for arbitrary p. We thus obtain guarded expressions in GTL(p)(σ) and our results tell that t-layered SGCs are bounded by cr(p) for vertex embeddings, and by gcr(p) for SGCs + readout.
Principal Neighbourhood Aggregation. Our next example is a GNN in which different aggregation functions are used: Principal Neighborhood Aggregation (PNA) is an architecture proposed by Corso et al. (2020) in which aggregation over neighboring vertices is done by means of mean, stdv, max and min, and this in parallel. In addition, after aggregation, three different scalers are applied. Scalers are diagonal matrices whose diagonal entries are a function of the vertex degrees. Given the features for each vertex v computed in layer t − 1, that is, F (t−1)v: ∈ R1×ℓ, a PNA computes v’s new features in layer t in the following way (see layer definition (8) in (Corso et al., 2020)). First, vectors G(t)v: ∈ R1×4ℓ are computed such that
G(t)vj =  mean ( {{mlpj(F (t−1)w: ) | w ∈ NG(v)}} ) for 1 ≤ j ≤ ℓ stdv ( {{mlpj(F (t−1)w: ) | w ∈ NG(v)}} ) for ℓ+ 1 ≤ j ≤ 2ℓ max ( {{mlpj(F (t−1)w: ) | w ∈ NG(v)}} ) for 2ℓ+ 1 ≤ j ≤ 3ℓ
min ( {{mlpj(F (t−1)w: ) | w ∈ NG(v)}} ) for 3ℓ+ 1 ≤ j ≤ 4ℓ,
where mlpj : Rℓ → R is the projection of an MLP mlp : Rℓ → Rℓ on the jth coordinate. Then, three different scalers are applied. The first scaler is simply the identity, the second two scalers s1 and s2 depend on the vertex degrees. As such, vectors H (t)v: ∈ R12ℓ are constructed as follows:
H (t)vj =  H (t)vj for 1 ≤ j ≤ 4ℓ s1(degG(v)) ·H (t) vj for 4ℓ+ 1 ≤ j ≤ 8ℓ
s2(degG(v)) ·H (t) vj for 8ℓ+ 1 ≤ j ≤ 12ℓ,
where s1 and s2 are functions from R → R (see (Corso et al., 2020) for details). Finally, the new vertex embedding is obtained as
F (t)v: = mlp ′(H (t)v: )
for some MLP mlp′ : R12ℓ → Rℓ. The above layer definition translates naturally into expressions in TL(Ω,Θ), the extension of TL(Ω) with aggregate functions (Section C.5). Indeed, suppose that for each j ∈ [ℓ] we have TL(Ω,Θ) expressions φ(t−1)j (x1) such that πG[[φ (t−1) j , v]] = F (t−1) vj for any vertex v. Then, G(t)vj simply corresponds to the guarded expressions
ψ(t)j (x1) := aggr(mean, x2, E(x1, x2) ·mlpj(φ (t−1) 1 (x2), . . . , φ (t−1) ℓ (x2)),
for 1 ≤ j ≤ ℓ, and similarly for the other components of G(t)v: using the respective aggregation functions, stdv, max and min. Then, H (t)vj corresponds to
ξ(t)j (x1) =  ψ(t)j (x1) for 1 ≤ j ≤ 4ℓ s1(aggr(sum, x2, E(x1, x2))) · ψ(t)j (x1) for 4ℓ+ 1 ≤ j ≤ 8ℓ s2(aggr(sum, x2, E(x1, x2))) · ψ(t)j (x1) for 8ℓ+ 1 ≤ j ≤ 12ℓ,
where we use summation aggregation to compute the degree information used in the functions in the scalers s1 and s2. And finally,
φ(t)j := mlp ′ j(ξ (t) 1 (x1), . . . , ξ (t) 12ℓ(x1))
represents F (t)vj . We see that all expressions only use two index variables and aggregation is applied in a guarded way. Furthermore, in each layer, the aggregation depth increases with one. As such, a t-layered PNA can be represented in GTL(t)(Ω,Θ), where Ω consists of the MLPs and functions used in scalers, and Θ consists of sum (for computing vertex degrees), and mean, stdv, max and min. Proposition C.6 then implies a bound on the separation power by cr(t).
Other example. In the same way, one can also easily analyze GATs (Velickovic et al., 2018) and show that these can be represented in GTL(Ω) as well, and thus bounds by color refinement can be obtained.
D.2 k-DIMENSIONAL WEISFEILER-LEMAN TESTS
We next discuss architectures related to the k-dimensional Weisfeiler-Leman algorithms. For k = 1, we discussed the extended GINs in the main paper. We here focus on arbitrary k ≥ 2.
Folklore GNNs. We first consider the “Folklore” GNNs or k-FGNNs for short (Maron et al., 2019b). For k ≥ 2, k-FGNNs computes a tensors. In particular, the initial tensor F(0) encodes atpk(G,v) for each v ∈ V kG . We can represent this tensor by the following k2(ℓ + 2) expressions in TL(0)k :
φ(0)r,s,j(x1, . . . , xk) :=  1xr=xs · Pj(xr) for j ∈ [ℓ] E(xr, xs) for j = ℓ+ 1 1xr=xs for j = ℓ+ 2 ,
for r, s ∈ [k] and j ∈ [ℓ + 2]. We note: πG[[φ(0)r,s,j , (v1, . . . , vk)]] = F (0) v1,...,vk,r,s,j for all (r, s, j) ∈ [k]2 × [ℓ+ 2], as desired. We let τ0 := [k]2 × [ℓ+ 2] and set d0 = k2 × (ℓ+ 2). Then, in layer t, a k-FGNN computes a tensor
F(t)v1,...,vk,• := mlp (t) 0 ( F(t−1)v1,...,vk,•, ∑ w∈VG k∏ s=1 mlp(t)s (F (t−1) v1,...,vs−1,w,vs+1,...,vk,•) ) ,
where mlp(t)s : Rdt−1 → Rd ′ t , for s ∈ [k], and and mlp(t)0 : Rdt−1×d ′ t → Rdt are MLPs. We here use • to denote combinations of indices in τd for F(t) and in τd−1 for F(t−1).
Let F(t−1) ∈ Rnk×dt−1 be the tensor computed by an k-FGNN in layer t − 1. Assume that for each tuple of elements j in τdt−1 we have an expression φ (t−1) j (x1, . . . , xk) satisfying
πG[[φ (t−1) j , (v1, . . . , vk)]] = F (t−1) v1,...,vk,j and such that it is an expression in TL(t−1)k+1 (Ω). That is, we need k + 1 index variables and a summation depth of t− 1 to represent layer t− 1. Then, for layer t, for each j ∈ τdt , it suffices to consider the expression
φ(t)j (x1, . . . , xk) := mlp (t) 0,j (( φ (t−1) i (x1, . . . , xk) ) i∈τdt−1 ,
∑ xk+1 k∏ s=1 mlp(t)s,j ( (φ(t−1)i (x1, . . . , xs−1, xk+1, xs+1, . . . , xk) ) i∈τdt−1 ) ,
where mlp(t)o,j and mlp (t) s,j are the projections of the MLPs on the j-coordinates. We remark that we need k + 1 index variables, and one extra summation is needed. We thus obtain expressions in TL(t)k+1(Ω) for the tth layer, as desired. We remark that the expressions are simple translations of the defining layer definitions. Also, in this case, Ω consists of all MLPs. When a k-FGNN is used for vertex embeddings, we now simply add to each expression a factor ∏k s=1 1x1=xs . As an immediate consequence of our results, if we denote by k-FGNN(t) the class of t-layered k-FGNNs, then for vertex embeddings:
ρ1 ( vwl(t)k ) ⊆ ρ1 ( TL(t)k+1(Ω) ) ⊆ ρ1 ( k-FGNN(t) ) in accordance with the known results from Azizian & Lelarge (2021). When used for graph embeddings, an aggregation layer over all k-tuples of vertices is added, followed by the application of an MLP. This results in expressions with no free index variables, and of summation depth t+ k, where the increase with k stems from the aggregation process over all k-tuples. In view of our results, for graph embeddings:
ρ0 ( gwl(∞)k ) ⊆ ρ0 ( TLk+1(Ω) ) ⊆ ρ0 ( k-FGNN ) in accordance again with Azizian & Lelarge (2021). We here emphasize that the upper bounds in terms of k-WL are obtained without the need to know how k-WL works. Indeed, one can really just focus on casting layers in the right tensor language!
We remark that Azizian & Lelarge (2021) define vertex embedding k-FGNNs in a different way. Indeed, for a vertex v, its embedding is obtained by aggregating of all (k−1) tuples in the remaining coordinates of the tensors. They define vwlk accordingly. From the tensor language point of view, this corresponds to the addition of k − 1 to the summation depth. Our results indicate that we loose the connection between rounds and layers, as in Azizian & Lelarge (2021). This is the reason why we defined vertex embedding k-FGNNs in a different way and can ensure a correspondence between rounds and layers for vertex embeddings.
Other higher-order examples. It is readily verified that t-layered k-GNNs (Morris et al., 2019) can be represented in TL(t)k+1(Ω), recovering the known upper bound by vwl (t) k (Morris et al., 2019). It is an equally easy exercise to show that 2-WL-convolutions (Damke et al., 2020) and Ring-GNNs (Chen et al., 2019) are bounded by 2-WL, by simply writing their layers in TL3(Ω). The invariant graph networks (k-IGNs) (Maron et al., 2019b) will be treated in Section E, as their representation in TLk+1(Ω) requires some work.
Augmented GNNs. Higher-order GNN architectures such as k-GNNs, k-FGNNs and k-IGNs, incur a substantial cost in terms of memory and computation (Morris et al., 2020). Some recent proposals infuse more efficient GNNs with higher-order information by means of some pre-processing step. We next show that the tensor language approach also enables to obtain upper bounds on the separation power of such “augmented” GNNs.
We first consider F-MPNNs (Barceló et al., 2021) in which the initial vertex features are augmented with homomorphism counts of rooted graph patterns. More precisely, let P r be a connected rooted graph (with root vertex r), and consider a graph G = (VG, EG, colG) and vertex v ∈ VG. Then, hom(P r, Gv) denotes the number of homomorphism from P to G, mapping r to v. We recall that a homomorphism is an edge-preserving mapping between vertex sets. Given a collection F = {P r1 , . . . , P rℓ } of rooted patterns, an F-MPNN runs an MPNN on the augmented initial vertex features:
F̃ (0)v: := (F (0) v: , hom(P r 1 , G v), . . . , hom(P rℓ , G v)).
Now, take any GNN architecture that can be cast in GTL(Ω) or TL2(Ω) and assume, for simplicity of exposition, that a t-layer GNN corresponds to expressions in GTL(t)(Ω) or TL(t)2 (Ω). In order to analyze the impact of the augmented features, one only needs to revise the expressions φ(0)j (x1) that represent the initial features. In the absence of graph patterns, φ(0)j (x1) := Pj(x1), as we have seen before. By contrast, to represent F̃ (0)vj we need to cast the computation of hom(P r i , G
v) in TL. Assume that the graph pattern Pi consists of p vertices and let us identify the vertex set with [p]. Furthermore, without of loss generality, we assume that vertex “1” is the root vertex in Pi. To obtain hom(P ri , G
v) we need to create an indicator function for the graph pattern Pi and then count how many times this indicator value is equal to one in G. The indicator function for Pi is simply given by the expression ∏ uv∈EPi
E(xu, xv). Then, counting just pours down to summation over all index variables except the one for the root vertex. More precisely, if we define
φPi(x1) := ∑ x2 · · · ∑ xp ∏ uv∈EPi E(xu, xv),
then πG[[φPi , v]] = hom(P r i , G v). This encoding results in an expression in TLp. However, it is well-known that we can equivalently write φPi(x1) as an expression φ̃Pi(x1) in TLk+1 where k is the treewidth of the graph Pi. As such, our results imply that F-MPNNs are bounded in separation power by k-WL where k is the maximal treewidth of graphs in F . We thus recover the known upper bound as given in Barceló et al. (2021) using our tensor language approach.
Another example of augmented GNN architectures are the Graph Substructure Networks (GSNs) (Bouritsas et al., 2020). By contrast to F-MPNNs, subgraph isomorphism counts rather than homomorphism counts are used to augment the initial features. At the core of a GSN thus lies the computation of sub(P r, Gv), the number of subgraphs H in G isomorphic to P (and such that the isomorphisms map r to v). In a similar way as for homomorphisms counts, we can either directly cast the computation of sub(P r, Gv) in TL resulting again in the use of p index variables. A possible reduction in terms of index variables, however, can be obtained by relying on the result (Theorem 1.1.) by Curticapean et al. (2017) in which it shown that sub(P r, Gv) can be computed in terms of homomorphism counts of graph patterns derived from P r. More precisely, Curticapean et al. (2017) define spasm(P r) as the set of graphs consisting of all possible homomorphic images of P r. It is then readily verified that if the maximal treewidth of the graphs in spasm(P r) is k, then sub(P r, Gv) can be cast as an expression in TLk+1. Hence, GSNs using a pattern collection F can be represented in TLk+1, where k is the maximal treewidth of graphs in any of the spams of patterns in F , and thus are bounded in separation power k-WL in accordance to the results by Barceló et al. (2021).
As a final example, we consider the recently introduced Message Passing Simplicial Networks (MPSNs) (Bodnar et al., 2021). In a nutshell, MPSNs are run on simplicial complexes of graphs instead of on the original graphs. We sketch how our tensor language approach can be used to assess the separation power of MPSNs on clique complexes. We use the simplified version of MPSNs which have the same expressive power as the full version of MPSNs (Theorem 6 in Bodnar et al. (2021)).
We recall some definitions. Let Cliques(G) denote the set of all cliques in G. Given two cliques c and c′ in Cliques(G), define c ≺ c′ if c ⊂ c′ and there exists no c′′ in Cliques(G), such that c ⊂ c′′ ⊂ c′. We define Boundary(c,G) := {c′ ∈ Cliques(G) | c′ ≺ c} and Upper(c,G) := {c′ ∈ Cliques(G) | ∃c′′ ∈ Cliques(G), c′ ≺ c′′ and c ≺ c′′}. For each c in Cliques(G) we have an initial feature vector F (0)c: ∈ R1×ℓ. Bodnar et al. (2021) initialize all initial features with the same value. Then, in layer t, for each c ∈ Cliques(G), features are updated as follows:
G(t)c: = FB({{mlpB(F (t−1)c: ,F (t−1) c′: ) | c ′ ∈ Boundary(G, c)}}) H (t)c: = FU ({{mlpU (F (t−1)c: ,F (t−1) c′: ,F (t−1) c∪c′: ) | c
′ ∈ Upper(G, c)}}) F (t)c: = mlp(F (t−1) c: ,G (t) c: ,H (t) c: ),
where FB and FU are aggregation functions and mlpB , mlpU and mlp are MLPs. With some effort, one can represent these computations by expressions in TLp(Ω,Θ) where p is largest clique inG. As such, the separation power of clique-complex MPSNs on graphs of clique size at most p is bounded
by p − 1-WL. And indeed, Bodnar et al. (2021) consider Rook’s 4 × 4 graph, which contains a 4-clique, and the Shirkhande graph, which does not contain a 4-clique. As such, the analysis above implies that clique-complex MPSNs are bounded by 2-WL on the Shrikhande graph, and by 3-WL on Rook’s graph, consistent with the observation in Bodnar et al. (2021). A more detailed analysis of MPSNs in terms of summation depth and for other simplicial complexes is left as future work.
This illustrates again that our approach can be used to assess the separation power of a variety of GNN architectures in terms of k-WL, by simply writing them as tensor language expressions. Furthermore, bounds in terms of k-WL can be used for augmented GNNs which form a more efficient way of incorporating higher-order graph structural information than higher-order GNNs.

heading: D.3 SPECTRAL GNNS
text: In general, spectral GNNs are defined in terms of eigenvectors and eigenvalues of the (normalized) graph Laplacian (Bruna et al., 2014; Defferrard et al., 2016; Levie et al., 2019; Balcilar et al., 2021b)). The diagonalization of the graph Laplacian is, however, avoided in practice, due to its excessive cost. Instead, by relying on approximation results in spectral graph analysis (Hammond et al., 2011), the layers of practical spectral GNNs are defined in term propagation matrices consisting of functions, which operate directly on the graph Laplacian. This viewpoint allows for a spectral analysis of spectral and “spatial” GNNs in a uniform way, as shown by Balcilar et al. (2021b). In this section, we consider two specific instances of spectral GNNs: ChebNet (Defferrard et al., 2016) and CayleyNet (Levie et al., 2019), and assess their separation power in terms of tensor logic. Our general results then provide bounds on their separation power in terms color refinement and 2-WL, respectively.
Chebnet. The separation power of ChebNet (Defferrard et al., 2016) was already analyzed in Balcilar et al. (2021a) by representing them in the MATLANG matrix query language (Brijder et al., 2019). It was shown (Theorem 2 (Balcilar et al., 2021a)) that it is only the maximal eigenvalue λmax of the graph Laplacian used in the layers of ChebNet that may result in the separation power of ChebNet to go beyond 1-WL. We here revisit and refine this result by showing that, when ignoring the use of λmax, the separation power of Chebnet is bounded already by color refinement (which, as mentioned in Section 2, is weaker than 1-WL for vertex embeddings). In a nutshell, the layers of a ChebNet are defined in terms of Chebyshev polynomials of the normalized Laplacian Lnorm = I − D−1/2 · A · D−1/2 and these polynomials can be easily represented in GTL(Ω). One can alternatively use the graph Laplacian L = D−A in a ChebNet, which allows for a similar analysis. The distinction between the choice of Lnorm and L only shows in the needed summation depth (in as in similar way as for the GCNs described earlier). We only consider the normalized Laplacian here.
More precisely, following Balcilar et al. (2021a;b), in layer t, vertex embeddings are updated in a ChebNet according to:
F (t) := σ
( p∑
s=1
C(s) · F (t−1) ·W (t−1,s) ) ,
with
C(1) := I,C(2) = 2
λmax Lnorm − I,C(s) = 2C(2) ·C(s−1) −C(s−2), for s ≥ 3,
and where λmax denotes the maximum eigenvalue of Lnorm. We next use a similar analysis as in Balcilar et al. (2021a). That is, we ignore for the moment the maximal eigenvalue λmax and redefine C(2) as cLnorm − I for some constant c. We thus see that each C(s) is a polynomial of the form ps(c,Lnorm) := ∑qs i=0 a (s) i (c) · (Lnorm)i with scalar functions a (s) i : R → R and where we interpret (Lnorm)0 = I . To upper bound the separation power using our tensor language approach, we can thus shift our attention entirely to representing (Lnorm)i · F (t−1) · W (t−1,s) for powers i ∈ N. Furthermore, since (Lnorm)i is again a polynomial of the form qi(D−1/2 · A · D−1/2) :=∑ri
j=0 bij · (D−1/2 ·A ·D−1/2)j , we can further narrow down the problem to represent
(D−1/2 ·A ·D−1/2)j · F (t−1) ·W (t−1,s)
in GTL(Ω), for powers j ∈ N. And indeed, combining our analysis for GCNs and SGCs results in expressions in GTL(Ω). As an example let us consider (D−1/2 ·A ·D−1/2)2 ·F (t−1) ·W (t−1), that is we use a power of two. It then suffices to define, for each output dimension j, the expressions:
ψ2j (x1) = f1/ √ x (∑ x2 E(x1, x2) ) · ∑ x2 ( E(x1, x2) · f1/x (∑ x1 (E(x2, x1) ) ·
∑ x1 ( E(x2, x1) · f1/√x( ∑ x2 E(x1, x2)) · (∑dt−1 i=1 W (t−1) ij φ (t−1) i (x1) ))) ,
where the φ(t−1)i (x1) are expressions representing layer t− 1. It is then readily verified that we can use ψ2j (x1) to cast layer t of a ChebNet in GTL(Ω) with Ω consisting of f1/√x : R → R : x 7→ 1√x , f1/x : R → R : x 7→ 1x , and the used activation function σ. We thus recover (and slightly refine) Theorem 2 in Balcilar et al. (2021a): Corollary D.1. On graphs sharing the same λmax values, the separation power of ChebNet is bounded by color refinement, both for graph and vertex embeddings.
A more fine-grained analysis of the expressions is needed when interested in bounding the summation depth and thus of the number of rounds needed for color refinement. Moreover, as shown by Balcilar et al. (2021a), when graphs have non-regular components with different λmax values, ChebNet can distinguish them, whilst 1-WL cannot. To our knowledge, λmax cannot be computed in TLk(Ω) for any k. This implies that it not clear whether an upper bound on the separation power can be obtained for ChebNet taking λmax into account. It is an interesting open question whether there are two graphs G and H which cannot be distinguished by k-WL but can be distinguished based on λmax. A positive answer would imply that the computation of λmax is beyond reach for TL(Ω) and other techniques are needed.
CayleyNet. We next show how the separation power of CayleyNet (Levie et al., 2019) can be analyzed. To our knowledge, this analysis is new. We show that the separation power of CayleyNet is bounded by 2-WL. Following Levie et al. (2019) and Balcilar et al. (2021b), in each layer t, a CayleyNet updates features as follows:
F (t) := σ
( p∑
s=1
C(s) · F (t−1)W (t−1,s) ) ,
with
C(1) := I,C(2s) := Re ((hL− ıI hL+ ıI )s) ,C(2s+1) := Re ( ı (hL− ıI hL+ ıI )s) ,
where h is a constant, ı is the imaginary unit, and Re : C → C maps a complex number to its real part. We immediately observe that a CayleyNet requires the use of complex numbers and matrix inversion. So far, we considered real numbers only, but when our separation results are concerned, the choice between real or complex numbers is insignificant. In fact, only the proof of Proposition C.3 requires a minor modification when working on complex numbers: the infinite disjunctions used in the proof now need to range over complex numbers. For matrix inversion, when dealing with separation power, one can use different expressions in TL(Ω) for computing the matrix inverse, depending on the input size. And indeed, it is well-known (see e.g., Csanky (1976)) that based on the characteristic polynomial of A, A−1 for any matrix A ∈ Rn×n can be computed as a polynomial −1cn ∑n−1 i=1 ciA
n−1−i if cn ̸= 0 and where each coefficient ci is a polynomial in tr(Aj), for various j. Here, tr(·) is the trace of a matrix. As a consequence, layers in CayleyNet can be viewed as polynomials in hL− ıI with coefficients polynomials in tr((hL− ıI)j). One now needs three index variables to represent the trace computations tr((hL− ıI)j). Indeed, let φ0(x1, x2) be the TL2 expression representing hL− ıI . Then, for example, (hL− ıI)j can be computed in TL3 using
φj(x1, x2) := ∑ x3 φ0(x1, x3) · φj−1(x3, x2)
and hence tr((hL−ıI)j) is represented by ∑
x1 ∑ x2 φj(x1, x2)·1x1=x2 .. In other words, we obtain
expressions in TL3. The polynomials in hL − ıI can be represented in TL2 just as for ChebNet.
This implies that each layer in CayleyNet can be represented, on graphs of fixed size, by TL3(Ω) expressions, where Ω includes the activation function σ and the function Re. This suffices to use our general results and conclude that CayleyNets are bounded in separation power by 2-WL. An interesting question is to find graphs that can be separated by a CayleyNet but not by 1-WL. We leave this as an open problem.

heading: E PROOF OF THEOREM 5.1
text: We here consider another higher-order GNN proposal: the invariant graph networks or k-IGNs of Maron et al. (2019b). By contrast to k-FGNNs, k-IGNs are linear architectures. If we denote by k-IGN(t) the class of t layered k-IGNs, then following inclusions are known (Maron et al., 2019b)
ρ1 ( k-IGN(t) ) ⊆ ρ1 ( vwl(t)k−1 ) and ρ0 ( k-IGN ) ⊆ ρ0 ( gwl(∞)k−1 ) .
The reverse inclusions were posed as open problems in Maron et al. (2019a) and were shown to hold by Chen et al. (2020) for k = 2, by means of an extensive case analysis and by relying on properties of 1-WL. In this section, we show that the separation power of k-IGNs is bounded by that of (k − 1)-WL, for arbitrary k ≥ 2. Theorem 4.2 tells that we can entirely shift our attention to showing that the layers of k-IGNs can be represented in TLk(Ω). In other words, we only need to show that k index variables are needed for the layers. As we will see below, this requires a bit of work since a naive representation of the layers of k-IGNs use 2k index variables. Nevertheless, we show that this can be reduced to k index variables only.
By inspecting the expressions needed to represent the layers of k-IGNs in TLk(Ω), we obtain that a t layer k-IGN(t) require expressions of summation depth of tk. In other words, the correspondence between layers and summation depth is precisely in sync. This implies, by Theorem 4.2:
ρ1 ( k-IGN ) = ρ1 ( vwl(∞)k−1 ) ,
where we ignore the number of layers. We similarly obtain that ρ0 ( k-IGN ) = ρ0 ( gwl(∞)k−1 ) , hereby answering the open problem posed in Maron et al. (2019a). Finally, we observe that the k-IGNs used in Maron et al. (2019b) to show the inclusion ρ1 ( k-IGN(t) ) ⊆ ρ1 ( vwl(t)k−1 ) are of very simple form.
By defining a simple class of k-IGNs, denoted by k-GINs, we obtain ρ1 ( k-GIN(t) ) = ρ1 ( vwl(t)k−1 ) , hereby recovering the layer/round connections.
We start with the following lemma: Lemma E.1. For any k ≥ 2, a t layer k-IGNs can be represented in TL(tk)k (Ω).
Before proving this lemma, we recall k-IGNs. These are architectures that consist of linear equivariant layers. Such linear layers allow for an explicit description. Indeed, following Maron et al. (2019c), let ∼ℓ be the equality pattern equivalence relation on [n]ℓ such that for a, b ∈ [n]ℓ, a ∼ℓ b if and only if ai = aj ⇔ bi = bj for all j ∈ [ℓ]. We denote by [n]ℓ/∼ℓ the equivalence classes induced by ∼ℓ. Let us denote by F(t−1) ∈ Rn
k×dt−1 the tensor computed by an k-IGN in layer t−1. Then, in layer t, a new tensor in Rnk×dt is computed, as follows. For j ∈ [dt] and v1, . . . , vk ∈ [n]k:
F (t)v1,...,vk,j := σ ( ∑ γ∈[n]2k/∼2k ∑ w∈[n]k 1(v,w)∈γ ∑ i∈[dt−1] cγ,i,jF (t−1)w1,...,wk,i + ∑ µ∈[n]k/∼k 1v∈µbµ,j ) (1) for activation function σ, constants cγ,i,j and bµ,j in R and where 1(v,w)∈γ and 1v∈µ are indicator functions for the 2k-tuple (v,w) to be in the equivalence class γ ∈ [n]2k/∼2k and the k-tuple v to be in class µ ∈ [n]k/∼k . As initial tensor F (0) one defines F (0)v1,...,vk,j := atpk(G,v) ∈ R d0 , with d0 = 2(k2) + kℓ where ℓ is the number of initial vertex labels, just as for k-FGNNs.
We remark that the need for having a summation depth of tk in the expressions in TLk(Ω), or equivalently for requiring tk rounds of (k− 1)-WL, can intuitively be explained that each layer of a k-IGN aggregates more information from “neighbouring” k-tuples than (k−1)-WL does. Indeed, in each layer, an k-IGN can use previous tuple embeddings of all possible k-tuples. In a single round of (k − 1)-WL only previous tuple embeddings from specific sets of k-tuples are used. It is only after an additional k−1 rounds, that k-WL gets to the information about arbitrary k-tuples, whereas this information is available in a k-IGN in one layer directly.
Proof of Lemma E.1. We have seen how F(0) can be represented in TLk(Ω) when dealing with k-FGNNs. We assume now that also the t − 1th layer F(t−1) can be represented by dt−1 expressions in TL((t−1)k)k (Ω) and show that the same holds for the tth layer.
We first represent F(t) in TL2k(Ω), based on the explicit description given earlier. The expressions use index variables x1, . . . , xk and y1, . . . , yk. More specifically, for j ∈ [dt] we consider the expressions:
φ(t)j (x1, . . . , xk) = σ  ∑ γ∈[n]2k/∼2k dt−1∑ i=1 cγ,i,j
∑ y1 · · · ∑ yk ψγ(x1, . . . , xk, y1, . . . , yk) · φ(t−1)i (y1, . . . , yk)
+ ∑
µ∈[n]k/∼k
bµ,j · ψµ(x1, . . . , xk)  , (2) where ψµ(x1, . . . , xk) is a product of expressions of the form 1xi op xj encoding the equality pattern µ, and similarly, ψγ(x1, . . . , xk, y1, . . . , yk) is a product of expressions of the form 1xi op xj , 1yi op yj and 1xi op yj encoding the equality pattern γ. These expressions are indicator functions for the their corresponding equality patterns. That is,
πG[[ψγ , (v,w)]] = { 1 if (v,w) ∈ γ 0 otherwise πG[[ψµ,v]] = { 1 if v ∈ µ 0 otherwise
We remark that in the expressions φ(t)j we have two kinds of summations: those ranging over a fixed number of elements (over equality patterns, feature dimension), and those ranging over the index variables y1, . . . , yk. The latter are the only ones contributing the summation depth. The former are just concise representations of a long summation over a fixed number of expressions.
We now only need to show that we can equivalently write φ(t)j (x1, . . . , xk) as expressions in TLk(Ω), that is, using only indices x1, . . . , xk. As such, we can already ignore the term ∑ µ∈[n]k/∼k
bµ,j · ψµ(x1, . . . , xk) since this is already in TLk(Ω). Furthermore, this expressions does not affect the summation depth.
Furthermore, as just mentioned, we can expand expression φ(t)j into linear combinations of other simpler expressions. As such, it suffices to show that k index variables suffice for each expression of the form: ∑
y1 · · · ∑ yk ψγ(x1, . . . , xk, y1, . . . , yk) · φ(t)i (y1, . . . , yk), (3)
obtained by fixing µ and i in expression (2). To reduce the number of variables, as a first step we eliminate any disequality using the inclusion-exclusion principle. More precisely, we observe that ψγ(x,y) can be written as:∏
(i,j)∈I
1xi=xj · ∏
(i,j)∈Ī
1xi ̸=xj · ∏
(i,j)∈J
1yi=yj · ∏
(i,j)∈J̄
1yi ̸=yj ∏
(i,j)∈K
1xi=yj · ∏
(i,j)∈K̄
1xi ̸=yj
= ∑ A⊆Ī ∑ B⊆J̄ ∑ C⊆K̄ (−1)|A|+|B|+|C| ∏ (i,j)∈I∪A 1xi=xj ∏ (i,j)∈J∪B 1yi=yj · ∏ (i,j)∈J∪C 1xi=yj , (4)
for some sets I , J and K of pairs of indices in [k]2, and where Ī = [k]2 \ I , J̄ = [k]2 \ J and K̄ = [k]2\K. Here we use that 1xi ̸=xj = 1−1xi=xj , 1yi ̸=yj = 1−1yi=yj and 1xi ̸=yj = 1−1yi=yj and use the inclusion-exclusion principle to obtain a polynomial in equality conditions only.
In view of expression (4), we can push the summations over y1, . . . , yk in expression (3) to the subexpressions that actually use y1, . . . , yk. That is, we can rewrite expression (3) into the equivalent expression:∑
A⊆Ī ∑ B⊆J̄ ∑ C⊆K̄ (−1)|A|+|B|+|C| · ∏ (i,j)∈I∪A 1xi=xj
· ∑ y1 · · · ∑ yk ∏ (i,j)∈J∪B 1yi=yj · ∏ (i,j)∈K∪C 1xi=yj · φ (t−1) i (y1, . . . , yk)  . (5) By fixing A, B and C, it now suffices to argue that
∏ (i,j)∈I∪A 1xi=xj · ∑ y1 · · · ∑ yk ∏ (i,j)∈J∪B 1yi=yj · ∏ (i,j)∈K∪C 1xi=yj · φ (t−1) i (y1, . . . , yk)  , (6) can be equivalently expressed in TLk(Ω).
Since our aim is to reduced the number of index variables from 2k to k, it is important to known which variables are the same. In expression (6), some equalities that hold between the variables may not be explicitly mentioned. For this reason, we expand I ∪A, J ∪B and K ∪C with their implied equalities. That is, 1xi=xj is added to I ∪A, if for any (v,w) such that
πG[[ ∏
(i,j)∈I∪A
1xi=xj · ∏
(i,j)∈J∪B
1yi=yj · ∏
(i,j)∈K∪C
1xi=yj , (v,w)]] = 1 ⇒ πG[[1xi=xj ,v]] = 1
holds. Similar implied equalities 1yi=yj and 1xi=yj are added to J ∪ B and K ∪ C, respectively. let us denoted by I ′, J ′ and K ′. It should be clear that we can add these implied equalities to expression (6) without changing its semantics. In other words, expression (6) can be equivalently represented by
∏ (i,j)∈I′ 1xi=xj · ∑ y1 · · · ∑ yk ∏ (i,j)∈J′ 1yi=yj · ∏ (i,j)∈K′ 1xi=yj · φ ((t−1) i (y1, . . . , yk)  , (7) There now two types of index variables among the y1, . . . , yk: those that are equal to some xi, and those that are not. Now suppose that (j, j′) ∈ J ′, and thus yj = yj′ , and that also (i, j) ∈ K ′, and thus xi = yj . Since we included the implied equalities, we also have (i, j′) ∈ K ′, and thus xi = yj′ . There is no reason to keep (j, j′) ∈ J ′ as it is implied by (i, j) and (i, j′) ∈ K ′. We can thus safely remove all pairs (j, j′) from J ′ such that (i, j) ∈ K ′ (and thus also (i, j′) ∈ K ′). We denote by J ′′ be the reduced set of pairs of indices obtained from J ′ in this way. We have that expression (7) can be equivalently written as
∏ (i,j)∈I′ 1xi=xj · ∑ y1 · · · ∑ yk ∏ (i,j)∈K′ 1xi=yj · ∏ (i,j)∈J′′ 1yi=yj · φ (t−1) i (y1, . . . , yk)  , (8) where we also switched the order of equalities in J ′′ andK ′. Our construction of J ′′ andK ′ ensures that none of the variables yj with j belonging to a pair in J ′′ is equal to some xi.
By contrast, the variable yj occurring in (i, j) ∈ K ′ are equal to xi. We observe, however, that also certain equalities among the variables {x1, . . . , xk} hold, as represented by the pairs in I ′. let I ′(i) := {i′ | (i, i′) ∈ I ′} and define ı̂ as a unique representative element in I ′(i). For example, one can take î to be smallest index in I ′(i). We use this representative index (and corresponding x-variable) to simplify K ′. More precisely, we replace each pair (i, j) ∈ K ′ with the pair (̂ı, j). In terms of variables, we replace xi = yj with xî = yj . Let K
′′ be the set K ′′ modified in that way. Expression (8) can thus be equivalently written as
∏ (i,j)∈I′ 1xi=xj · ∑ y1 · · · ∑ yk ∏ (ı̂,j)∈K′′ 1xı̂=yj · ∏ (i,j)∈J′′ 1yi=yj · φ (t−1) i (y1, . . . , yk)  , (9) where the free index variables of the subexpression∑
y1 · · · ∑ yk ∏ (ı̂,j)∈K′′ 1xı̂=yj · ∏ (i,j)∈J′′ 1yi=yj · φ (t−1) i (y1, . . . , yk) (10)
are precisely the index variables xı̂ for (̂ı, j) ∈ K ′′. Recall that our aim is to reduce the variables from 2k to k. We are now finally ready to do this. More specifically, we consider a bijection
β : {y1, . . . , yk} → {x1, . . . , xk} in which ensure that for each ı̂ there is a j such that (̂i, j) ∈ K ′′ and β(yj) = xı̂. Furthermore, among the summations ∑ y1 · · · ∑ yk we can ignore those for which β(yj) = xı̂ holds. After all, they only contribute for a given xı̂ value. Let Y be those indices in [k] such that β(yj) ̸= xı̂ for some ı̂. Then, we can equivalently write expression (9) as∏
(i,j)∈I′ 1xi=xj · ( ∑ β(yi),i∈Y ∏ (ı̂,j)∈K′ 1xı̂=β(yj) · ∏ (i,j)∈J′′ 1β(yi)=β(yj)
· β(φ(t−1)i (y1, . . . , yk))
) , (11)
where β(φ(t−1)i (y1, . . . , yk)) denotes the expression obtained by renaming of variables y1, . . . , yj in φ (t−1) i (y1, . . . , yk) into x-variables according to β. This is our desired expression in TLk(Ω). If we analyze the summation depth of this expression, we have by induction that the summation depth of φ(t−1)i is at most (t − 1)k. In the above expression, we are increasing the summation depth with at most |Y |. The largest size of Y is k, which occurs when none of the y-variables are equal to any of the x-variables. As a consequence, we obtained an expression of summation depth at most tk, as desired.
As a consequence, when using k-IGNs(t) for vertex embeddings, using (G, v) → F(t)v,...,v,: one simply pads the layer expression with ∏ i∈[k] 1x1=xi which does not affect the number of variables or summation depth. When using k-IGNs(t) of graph embeddings, an additional invariant layer is added to obtain an embedding fromG→ Rdt . Such invariant layers have a similar (simpler) representation as given in equation 1 (Maron et al., 2019c), and allow for a similar analysis. One can verify that expressions in TL((t+1)k)k (Ω) are needed when such an invariant layer is added to previous t layers. Based on this, Theorem 4.2, Lemma E.1 and Theorem 1 in Maron et al. (2019b), imply that ρ1 ( k-IGN ) = ρ1 ( vwl(∞)k−1 ) and ρ0 ( k-IGN ) = ρ0 ( gwl(∞)k−1 ) hold.
k-dimensional GINs. We can recover a layer-based characterization for k-IGNs that compute vertex embeddings by considering a special subset of k-IGNs. Indeed, the k-IGNs used in Maron et al. (2019b) to show ρ1(wl(t)k−1) ⊆ ρ1(k-IGN
(t)) are of a very special form. We extract the essence of these special k-IGNs in the form of k-dimensional GINs. That is, we define the class k-GINs to consist of layers defined as follows. The initial layers are just as for k-IGNs. Then, for t ≥ 1:
F(t)v1,...,vk,: := mlp (t) 0 ( F(t−1)v1,...,vk,:, ∑ u∈VG mlp(t)1 (F (t−1) u,v2,...,vk,: ), ∑ u∈VG mlp(t)1 (F (t−1) v1,u,...,vk,: )
, . . . , ∑ u∈VG mlp (t) 1 (F (t−1) v1,v2,...,vk−1,w,: )) ) ,
where F (t−1)v1,v2,...,vk,: ∈ R dt−1 , mlp(t)1 : Rdt−1 → Rbt and mlp (t) 1 : Rdt−1+kbt → Rdt are MLPs. It is now an easy exercise to show that k-GIN(t) can be represented in TL(t)k (Ω) (remark that the summations used increase the summation depth with one only in each layer). Combined with Theorem 4.2 and by inspecting the proof of Theorem 1 in Maron et al. (2019b), we obtain: Proposition E.2. For any k ≥ 2 and any t ≥ 0: ρ1(k-GIN(t)) = ρ1(vwl(t)k−1).
We can define the invariant version of k-IGNs by adding a simple readout layer of the form∑ v1,...,vk∈VG mlp(F(t)v1,...,vk,:),
as is used in Maron et al. (2019b). We obtain, ρ0(k-GIN) = ρ0(gwl(∞)k−1), by simply rephrasing the readout layer in TLk(Ω).

heading: F DETAILS OF SECTION 6
text: Let C(Gs,Rℓ) be the class of all continuous functions from Gs to Rℓ. We always assume that Gs forms a compact space. For example, when vertices are labeled with values in {0, 1}ℓ0 , Gs is a
finite set which we equip with the discrete topology. When vertices carry labels in Rℓ0 we assume that these labels come from a compact set K ⊂ Rℓ0 . In this case, one can represent graphs in Gs by elements in (Rℓ0)2 and the topology used is the one induced by some metric ∥.∥ on the reals. Similarly, we equip Rℓ with the topology induced by some metric ∥.∥.
Consider F ⊆ C(Gs,Rℓ) and define F as the closure of F in C(Gs,Rℓ) under the usual topology induced by f 7→ supG,v∥f(G,v)∥. In other words, a continuous function h : Gs → Rℓ is in F if there exists a sequence of functions f1, f2, . . . ∈ F such that limi→∞ supG,v∥fi(G,v)−h(G,v)∥ = 0. The following theorem provides a characterization of the closure of a set of functions. We state it here modified to our setting. Theorem F.1 ((Timofte, 2005)). Let F ⊆ C(Gs,Rℓ) such that there exists a set S ⊆ C(Gs,R) satisfying S · F ⊆ F and ρ(S) ⊆ ρ(F). Then,
F := { f ∈ C(Gs,Rℓ) ∣∣ ρ(F ) ⊆ ρ(f),∀(G,v) ∈ Gs, f(G,v) ∈ F(G,v)}, where F(G,v) := {h(G,v) | h ∈ F} ⊆ Rℓ. We can equivalently replace ρ(F) by ρ(S) in the expression for F .
We will use this theorem to show Theorem 6.1 in the setting that F consists of functions that can be represented in TL(Ω), and more generally, sets of functions that satisfy two conditions, stated below. We more generally allow F to consist of functions f : Gs → Rℓf , where the ℓf ∈ N may depend on f . We will require F to satisfy the following two conditions:
concatenation-closed: If f1 : Gs → Rp and f2 : Gs → Rq are in F , then g := (f1, f2) : Gs → Rp+q : (G,v) 7→ (f1(G,v), f2(G,v)) is also in F .
function-closed: For a fixed ℓ ∈ N, for any f ∈ F such that f : Gs → Rp, also h ◦ f : Gs → Rℓ is in F for any continuous function h ∈ C(Rp,Rℓ).
We denote by Fℓ be the subset of F of functions from Gs to Rℓ. Theorem 6.1. For any ℓ, and any set F of functions, concatenation and function closed for ℓ, we have: Fℓ = {f : Gs → Rℓ | ρs(F) ⊆ ρs(f)}.
Proof. The proof consist of (i) verifying the existence of a set S as mentioned Theorem F.1; and of (ii) eliminating the pointwise convergence condition “∀(G,v) ∈ Gs, f(G,v) ∈ Fℓ(G,v) in the closure characterization in Theorem F.1.
For showing (ii) we argue that Fℓ(G,v) = Rℓ such that the conditions f(G,v) ∈ Fℓ(G,v) is automatically satisfied for any f ∈ C(Gs,Rℓ). Indeed, take an arbitrary f ∈ Gk → Rℓ and consider the constant functions bi : Rℓ → Rℓ : x 7→ bi with bi ∈ Rℓ the ith basis vector. Since F is functionclosed for ℓ, so is Fℓ. Hence, bi := gi ◦ f ∈ Fℓ as well. Furthermore, if sa : Rℓ → Rℓ : x 7→ a×x, for a ∈ R, then sa ◦ f ∈ Fℓ and thus Fℓ is closed under scalar multiplication. Finally, consider + : R2ℓ → Rℓ : (x,y) 7→ x + y. For f and g in Fℓ, h = (f, g) ∈ F since F is concatenationclosed. As a consequence, the function + ◦ h : Gs → Rℓ is in Fℓ, showing that Fℓ is also closed under addition. All combined, this shows that Fℓ is closed under taking linear combinations and since the basis vectors of Rℓ can be attained, Fℓ(G,v) := Rℓ, as desired. For (i), we show the existence of a set S ⊆ C(Gs,R) such that S · Fℓ ⊆ Fℓ and ρs(S) ⊆ ρs(Fℓ) hold. Similarly as in Azizian & Lelarge (2021), we define
S := { f ∈ C(Gs,R) ∣∣ (f, f, . . . , f)︸ ︷︷ ︸ ℓ times ∈ Fℓ } .
We remark that for s ∈ S and f ∈ Fℓ, s · f : Gs → Rℓ : (G,v) 7→ s(G,v)⊙ f(G,v), with ⊙ being pointwise multiplication, is also in Fℓ. Indeed, s · f = ⊙ ◦ (s, f) with (s, f) the concatenation of s and f and ⊙ : R2ℓ → Rℓ : (x,y) → x⊙ y being pointwise multiplication. It remains to verify ρs(S) ⊆ ρs(Fℓ). Assume that (G,v) and (H,w) are not in ρs(Fℓ). By definition, this implies the existence of a function f̂ ∈ Fℓ such that f̂(G,v) = a ̸= b = f̂(H,w) with a, b ∈ Rℓ. We argue that (G,v) and (H,w) are also not in ρs(S) either. Indeed, Proposition 1 in Maron et al. (2019b) implies that there exists natural numbers α = (α1, . . . , αℓ) ∈ Nℓ such
that the mapping hα : Rℓ → R : x → ∏ℓ i=1 x αi i satisfies hα(a) = a ̸= b = hα(b), with a, b ∈ R. Since F (and thus also Fℓ) is function-closed, hα ◦ f ∈ Fℓ for any f ∈ Fℓ. In particular, g := hα ◦ f̂ ∈ Fℓ and concatenation-closure implies that (g, . . . , g) : Gs → Rℓ is in Fℓ too. Hence, g ∈ S, by definition. It now suffices to observe that g(G,v) = hα(f̂(G,v)) = a ̸= b = hα(f̂(H,w)) = g(H,w), and thus (G,v) and (H,w) are not in ρs(S), as desired.
When we know more about ρs(Fℓ) we can say a bit more. In the following, we let alg ∈ {cr(t), gcr(t), vwl(t)k , gwl (∞) k } and only consider the setting where s is either 0 (invariant graph functions) or s = 1 (equivariant graph/vertex functions). Corollary 6.2. Under the assumptions of Theorem 6.1 and if ρ(Fℓ) = ρ(alg), then Fℓ = {f : Gs → Rℓ | ρ(alg) ⊆ ρ(f)}.
Proof. This is just a mere restatement of Theorem 6.1 in which ρs(Fℓ) in the condition ρs(Fℓ) ⊆ ρs(f) is replaced by ρs(alg), where s = 1 for alg ∈ {cr(t), vwl(t)k } and s = 0 for alg ∈ {gcr(t), gwl(∞)k }.
To relate all this to functions representable by tensor languages, we make the following observations. First, if we consider F to be the set of all functions that can be represented in GTL(t)(Ω), TL(t+1)2 (Ω), TL(t)k+1(Ω) or TL(Ω), then F will be automatically concatenation and function-closed, provided that Ω consists of all functions in ⋃ p C(Rp,Rℓ). Hence, Theorem 6.1 applies. Furthermore, our results
from Section 4 tell us that for all t ≥ 0, and k ≥ 1, ρ1 ( cr(t) ) = ρ1 ( GTL(t)(Ω) ) , ρ0 ( gcr(t) ) =
ρ0 ( TL(t+1)2 (Ω) ) = ρ0 ( gwl(t)1 ) , ρ1 ( ewl(t)k ) = ρ1 ( TL(t)k+1(Ω) ) , and ρ0 ( TLk+1(Ω) ) = ρ0 ( gwl(∞)k ) . As a consequence, Corollary 6.2 applies as well. We thus easily obtain the following characterizations:
Proposition F.2. For any t ≥ 0 and k ≥ 1:
• If F consists of all functions representable in GTL(t)(Ω), then Fℓ = {f : G1 → Rℓ | ρ1 ( cr(t) ) ⊆ ρ1(f)};
• If F consists of all functions representable in TL(t)k+1(Ω), then Fℓ = {f : G1 → Rℓ | ρ1 ( vwl(t)k ) ⊆ ρ1(f)}; • If F consists of all functions representable in TL(t+1)2 (Ω), then Fℓ = {f : G0 → Rℓ | ρ0 ( gwl(t)1 ) ⊆ ρ0(f)}; and finally, • If F consists of all functions representable in TLk+1(Ω), then Fℓ = {f : G0 → Rℓ | ρ0 ( gwl(∞)k ) ⊆ ρ0(f)},
provided that Ω consists of all functions in ⋃
p C(Rp,Rℓ).
In fact, Lemma 32 in Azizian & Lelarge (2021) implies that we can equivalently populate Ω with all MLPs instead of all continuous functions. We can thus use MLPs and continuous functions interchangeably when considering the closure of functions.
At this point, we want to make a comparison with the results and techniques in Azizian & Lelarge (2021). Our proof strategy is very similar and is also based on Theorem F.1. The key distinguishing feature is that we consider functions f : Gs → Rℓf instead of functions from graphs alone. This has as great advantage that no separate proofs are needed to deal with invariant or equivariant functions. Equivariance incurs quite some complexity in the setting considered in Azizian & Lelarge (2021). A second major difference is that, by considering functions representable in tensor languages, and based on our results from Section 4, we obtain a more fine-grained characterization. Indeed, we obtain characterizations in terms of the number of rounds used in CR and k-WL. In Azizian & Lelarge (2021), t is always set to ∞, that is, an unbounded number of rounds is considered. Furthermore, when it concerns functions f : G1 → Rℓf , we recall that CR is different from 1-WL. Only 1-WL is considered in Azizian & Lelarge (2021). Finally, another difference is that we define the equivariant version vwlk in a different way than is done in Azizian & Lelarge (2021), because in this
way, a tighter connection to logics and tensor languages can be made. In fact, if we were to use the equivariant version of k-WL from Azizian & Lelarge (2021), then we necessarily have to consider an unbounded number of rounds (similarly as in our gwlk case).
We conclude this section by providing a little more details about the consequences of the above results for GNNs. As we already mentioned in Section 6.2, many common GNN architectures are concatenation and function-closed (using MLPs instead of continuous functions). This holds, for example, for the classes GIN(t)ℓ , eGIN (t) ℓ , k-FGNN (t) ℓ and k-GIN (t) ℓ and k-IGN
(t), as described in Section 5 and further detailed in Section E and D. Here, the subscript ℓ refers to the dimension of the embedding space.
We now consider a function f that is not more separating than cr(t) (respectively, gcr(t), vwl(t)k or gwl(∞)k , for some k ≥ 1), and want to know whether f can be approximated by a class of GNNs. Proposition F.2 tells that such f can be approximated by a class of GNNs as long as these are at least as separating as GTL(t) (respectively, TL(t+1)2 , TL (t) k+1 or TL (∞) k+1). This, in turn, amounts showing that the GNNs can be represented in the corresponding tensor language fragment, and that they can match the corresponding labeling algorithm in separation power. We illustrate this for the GNN architectures mentioned above.
• In Section 5 we showed that GIN(t)ℓ can be represented in GTL (t)(Ω). Theorem 4.3 then
implies that ρ1(cr(t)) ⊆ ρ1(GIN(t)ℓ ). Furthermore, Xu et al. (2019) showed that ρ1(GIN (t) ℓ ) ⊆ ρ1(cr (t)). As a consequence, ρ1(GIN(t)ℓ ) = ρ1(cr
(t)). We note that the lower bound for GINs only holds when graphs carry discrete labels. The same restriction is imposed in Azizian & Lelarge (2021).
• In Section 5 we showed that eGIN(t)ℓ can be represented in TL (t) 2 (Ω). Theorem 4.2 then
implies that ρ1(vwl(t)1 ) ⊆ ρ1(eGIN (t) ℓ ). Furthermore, Barceló et al. (2020) showed that ρ1(eGIN (t) ℓ ) ⊆ ρ1(vwl (t) 1 ). As a consequence, ρ1(eGIN (t) ℓ ) = ρ1(vwl (t) 1 ). Again, the lower
bound is only valid when graphs carry discrete labels. • In Section 5 we mentioned (see details in Section D) that k-FGNN(t)ℓ can be represented
in TL(t)k+1(Ω). Theorem 4.2 then implies that ρ1(vwl (t) k ) ⊆ ρ1(k-FGNN (t) ℓ ). Furthermore, Maron et al. (2019b) showed that ρ1(k-FGNN(t)ℓ ) ⊆ ρ1(vwl (t) k ). As a consequence, ρ1(k-FGNN(t)ℓ ) ⊆ ρ1(vwl (t) k ). Similarly, ρ1((k + 1)-GIN (t) ℓ ) = ρ1(vwl (t) k ) for the special class of (k + 1)-IGNs described in Section E. No restrictions are in place for the lower bounds and hence real-valued vertex-labelled graphs can be considered.
• When GIN(t)ℓ or eGIN (t) ℓ are extended with a readout layer, we showed in Section 5 that
these can be represented in TL(t+1)2 (Ω). Theorem 4.4 and the results by Xu et al. (2019) and Barceló et al. (2020) then imply that ρ0(vwl(t)1 ) and ρ0(gcr
(t)) coincide with the separation power of these architectures with a readout layer. Here again, discrete labels need to be considered.
• Similarly, when k-FGNN or (k+1)-IGNs are used for graph embeddings, we can represent these in TLk+1(Ω) resulting again that their separation power coincides with that of gwl(∞)k . No restrictions are again in place on the vertex labels.
So for all these architectures, Corollary 6.2 applies and we can characterize the closures of these architectures in terms of functions that not more separating than their corresponding versions of cr or k-WL, as described in the main paper. In summary, Proposition F.3. For any t ≥ 0:
GIN(t)ℓ = {f : G1 → R ℓ | ρ1(cr(t)) ⊆ ρ1(f)} = GTL(t)(Ω)ℓ
eGIN(t)ℓ = {f : G1 → R ℓ | ρ1(vwl(t)1 ) ⊆ ρ1(f)} = TL (t) 2 (Ω)ℓ
and when extended with a readout layer:
GIN(t)ℓ = eGIN (t) ℓ = {f : G0 → R ℓ | ρ0(gwl(t)1 ) ⊆ ρ0(f)} = TL (t+1) 2 (Ω)ℓ.
Furthermore, for any k ≥ 1
k-FGNN(t)ℓ = k-GIN (t) ℓ = {f : G1 → R ℓ | ρ1(vwl(t)k ) ⊆ ρ1(f)} = TL (t) k+1(Ω)ℓ
(k + 1)-IGNℓ = {f : G1 → Rℓ | ρ1(vwl(∞)k ) ⊆ ρ1(f)} = TLk+1(Ω)ℓ and when converted into graph embeddings:
k-FGNNℓ = k-GINℓ = (k + 1)-IGNℓ = {f : G0 → Rℓ | ρ0(gwl(∞)k ) ⊆ ρ0(f)} = TLk+1(Ω)ℓ,
where the closures of the tensor languages are interpreted as the closure of the graph or graph/vertex functions that they can represent. For results involving GINs or eGINs, the graphs considered should have discretely labeled vertices.
As a side note, we remark that in order to simulate CR on graphs with real-valued labels, one can use a GNN architecture of the form F (t)v: = ( F (t−1)v: , ∑ u∈NG(v) mlp(F (t−1) u: ) ) , which translates in GTL(t)(Ω) as expressions of the form
φ(t)j (x1) :=
{ φ(t−1)j (x1) 1 ≤ j ≤ dt−1∑
x2 E(x1, x2) ·mlpj ( φ(t−1)1 (x1), . . . , φ (t−1) dt (x1) ) dt−1 < j ≤ dt.
The upper bound in terms of CR follows from our main results. To show that CR can be simulated, it suffices to observe that one can approximate the function used in Proposition 1 in Maron et al. (2019b) to injectively encode multisets of real vectors by means of MLPs. As such, a continuous version of the first bullet in the previous proposition can be obtained.

heading: G DETAILS ON TREEWIDTH AND PROPOSITION 4.5
text: As an extension of our main results in Section 4, we enrich the class of tensor language expressions for which connections to k-WL exist. More precisely, instead of requiring expressions to belong to TLk+1(Ω), that is to only use k + 1 index variables, we investigate when expressions in TL(Ω) are semantically equivalent to an expression using k + 1 variables. Proposition 4.5 identifies a large class of such expressions, those of treewidth k. As a consequence, even when representing GNN architectures may require more than k + 1 index variables, sometimes this number can be reduced. As a consequence of our results, this implies that their separation power is in fact upper bounded by ℓ-WL for a smaller ℓ < k. Stated otherwise, to boost the separation power of GNNs, the treewidth of the expressions representing the layers of the GNNs must have large treewidth.
We next introduce some concepts related to treewidth. We here closely follow the exposition given in Abo Khamis et al. (2016) for introducing treewidth by means variable elimination sequences of hypergraphs.

heading: G.1 ELIMINATION SEQUENCES
text: We first define elimination sequences for hypergraphs. Later on, we show how to associate such hypergraphs to expressions in tensor languages, allowing us to define elimination sequences for tensor language expressions.
With a multi-hypergraph H = (V, E) we simply mean a multiset E of subsets of vertices V . An elimination hypergraph sequences is a vertex ordering σ = v1, . . . , vn of the vertices of H. With such a sequence σ, we can associate for j = n, n−1, n−2, . . . , 1 a sequence of nmulti-hypergraphs Hσn,Hσn−1, . . . ,Hσ1 as follows. We define
Hn := (Vn, En) := H ∂(vn) := {F ∈ En | vn ∈ F}
Un := ⋃
F∈∂(vn)
F.
and for j = n− 1, n− 2, . . . , 1 :
Vj := {v1, . . . , vj} Ej := (Ej+1 \ ∂(vj+1)) ∪ {Uj+1 \ {vj+1}}
∂(vj) := {F ∈ Ej | vj ∈ F}
Uj := ⋃
F∈∂(vj)
F.
The induced width on H by σ is defined as maxi∈[n] |Ui| − 1. We further consider the setting in which H has some distinguished vertices. As we will see shortly, these distinguished vertices correspond to the free index variables of tensor language expressions. Without loss of generality, we assume that the distinguished vertices are v1, v2, . . . , vf . When such distinguished vertices are present, an elimination sequence is just as before, except that the distinguished vertices come first in the sequence. If v1, . . . , vf are the distinguished vertices, then we define the induced width of the sequence as f +maxf+1≤i≤n |Ui \ {v1, . . . , vf}| − 1. In other words, we count the number of distinguished vertices, and then augment it with the induced width of the sequence, starting from vf+1 to to vn, hereby ignoring the distinguished variables in the Ui’s. One could, more generally, also try to reduce the number of free index variables but we assume that this number is fixed, similarly as how GNNs operate.

heading: G.2 CONJUNCTIVE TL EXPRESSIONS AND TREEWIDTH
text: We start by considering a special form of TL expressions, which we refer to as conjunctive TL expressions, in analogy to conjunctive queries in database research and logic. A conjunctive TL expression is of the form
φ(x) = ∑ y ψ(x,y).
where x denote the free index variables, y contains all index variables under the scope of a summation, and finally, ψ(x,y) is a product of base predicates in TL. That is, ψ(x,y) is a product of E(zi, zj) and Pℓ(zi) with zi, zj variables in x or y. With such a conjunctive TL expression, one can associate a multi-hypergraph in a canonical way (Abo Khamis et al., 2016). More precisely, given a conjunctive TL expression φ(x) we define Hφ as:
• Vφ consist of all index variables in x and y; • Eφ: for each atomic base predicate τ in ψ we have an edge Fτ containing the indices
occurring in the predicate; and • the vertices corresponding to the free index variables x form the distinguishing set of ver-
tices.
We now define an elimination sequence for φ as an elimination sequence for Hφ taking the distinguished vertices into account. The following observation ties elimination sequences of φ to the number of variables needed to express φ. Proposition G.1. Let φ(x) be a conjunctive TL expression for which an elimination sequence of induced with k − 1 exists. Then φ(x) is equivalent to an expression φ̃(x) in TLk.
Proof. We show this by induction on the number of vertices in Hφ which are not distinguished. For the base case, all vertices are distinguished and hence φ(x) does not contain any summation and is an expression in TLk itself.
Suppose that in Hφ there are p undistinguished vertices. That is, φ(x) = ∑ y1 · · · ∑ yp ψ(x,y).
By assumption, we have an elimination sequence of the undistinguished vertices. Assume that yp is first in this ordering. Let us write
φ(x) = ∑ y1 · · · ∑ yp ψ(x,y)
= ∑ y1 · · · ∑ yp−1 ψ1(x,y \ yp) · ∑ yp ψ2(x,y)
where ψ1 is the product of predicates corresponding to the edges F ∈ Eφ \ ∂(yp), that is, those not containing yp, and ψ2 is the product of all predicates corresponding to the edges F ∈ ∂(yp), that is,
those containing the predicate yp. Note that, because of the induced width of k − 1, ∑
yp ψ2(x,y)
contains all indices in Up which is of size ≤ k. We now replace the previous expression with another expression
φ′(x) = ∑ y1 · · · ∑ yp−1 ψ1(x,y \ yp) ·Rp(x,y)
Where Rp is regarded as an |Up| − 1-ary predicate over the indices in Up \ yp. It is now easily verified that Hφ′ is the hypergraph Hp−1 corresponding to the variable ordering σ. We note that this is a hypergraph over p− 1 undistinguished vertices. We can apply the induction hypothesis and replace φ′(x) with its equivalent expression φ̃′(x) in TLk. To obtain the expression φ̃(x) of φ(x), it now remains to replace the new predicate Rp with its defining expression. We note again that Rp contains at most k − 1 indices, so it will occur in φ̃′(x) in the form Rp(x, z) where |z| ≤ k − 1. In other words, one of the variables in z is not used, say zs, and we can simply replace Rp(x, z) by∑
zs ψx(x, z, zs).
As a consequence, one way of showing that a conjunctive expression φ(x) in TL is equivalently expressible in TLk, is to find an elimination sequence of induced width k − 1. This in turn is equivalent to Hφ having a treewidth of k − 1, as is shown, e.g., in Abo Khamis et al. (2016). As usual, we define the treewidth of a conjunctive expression φ(x) in TL as the treewidth of its associated hypergraph Hφ. We recall the definition of treewidth (modified to our setting): A tree decomposition T = (VT , ET , ξT ) of Hφ with ξT : VT → 2V is such that
• For any F ∈ E , there is a t ∈ VT such that F ⊆ ξT (t); and • For any v ∈ V corresponding to a non-distinguished index variable, the set {t | t ∈ VT , v ∈ ξ(t)} is not empty and forms a connected sub-tree of T .
The width of a tree decomposition T is given by maxt∈VT |ξT (t)| − 1. Now the treewidth of Hφ, tw(H) is the minimum width of any of its tree decompositions. We denote by tw(φ) the treewidth of Hφ. Again, similar modifications are used when distinguished vertices are in place. Referring again to Abo Khamis et al. (2016), tw(φ) = k− 1 is equivalent to having a variable elimination sequence for φ of an induced width of k − 1. Hence, combining this observation with Proposition G.1 results in: Corollary G.2. Let φ(x) be a conjunctive TL expression of treewidth k−1. Then φ(x) is equivalent to an expression φ̃(x) in TLk.
That is, we have established Proposition 4.5 for conjunctive TL expressions. We next lift this to arbitrary TL(Ω) expressions.

heading: G.3 ARBITRARY TL(Ω) EXPRESSIONS
text: First, we observe that any expression in TL can be written as a linear combination of conjunctive expressions. This readily follows from the linearity of the operations in TL and that equality and inequality predicates can be eliminated. More specifically, we may assume that φ(x) in TL is of the form ∑
α∈A aαψα(x,y),
with A finite set of indices and aα ∈ R, and ψα(x,y) conjunctive TL expressions. We now define tw(φ) := max{tw(ψα) | α ∈ A}
for expressions in TL. To deal with expressions in TL(Ω) that may contain function application, we define tw(φ) as the maximum treewidth of the expressions: (i) φnofun(x) ∈ TL obtained by replacing each top-level function application f(φ1, . . . , φp) by a new predicateRf with free indices free(φ1) ∪ · · · ∪ free(φp); and (ii) all expressions φ1, . . . , φp occurring in a top-level function application f(φ1, . . . , φp) in φ. We note that these expression either have no function applications (as in (i)) or have function applications of lower nesting depth (in φ, as in (ii)). In other words, applying this definition recursively, we end up with expressions with no function applications, for which treewidth was already defined. With this notion of treewidth at hand, Proposition 4.5 now readily follows.

Review summaries:
This paper introduces a new approach to study the separation power and approximation properties of graph neural networks (GNN). The authors introduce the Tensor Languages (TL) and show the representation of GNN as TL expressions. In Section 4, they show the separation power of TL and relate it to color refinement algorithms like k-WL. In particular, they are able to characterize the separation power of such algorithms after a given number of iterations. These results allows them to compute in Section 5 the separation power of GNNs. They are able to recover all results in the literature in an unified way and prove some new results. In Section 6, they give consequences of their results in term of approximation of GNNs, recovering known results and proving new ones. This is a theoretical paper without any experimental results.

This paper defined formal languages for describing tensor expressions. It showed that their expressive power is identical to (suitably parameterized) color refinement algorithms or (vertex/graph) WL algorithms. Then, by translating existing GNNs into the tensor languages, this paper provided upper bounds for the expressive power of GNNs, which recovered several existing results. In addition, this paper characterized the closure of a function class by functions whose expressive power is equal to or less than the functions (Theorem 6.1). As an application, this paper characterized the expressive power of several GNNs via the tensor language (Corollary 6.2, Proposition E.3).

The paper proposes Tensor Language (TL), a language with which popular GNN models can be uniformly studied, so as to yield insights about their expressiveness and separation power. More specifically, the paper shows how TL corresponds to the WL hierarchy, and shows that summation depth in TL, as well as the number of index variables, correspond directly to iterations and tuple size for k-WL, respectively. Moreover, the paper establishes a direct correspondence between GNN update equations and expressions in TL, thus offering a simpler means to understand the expressive and separation power of GNNs, even considering their number of layers, by mapping their architectures to expressions in TL. TL , as well as its guarded fragment GTL, are then used to re-establish a set of known GNN results from the literature.   Beyond expressive power, TL is also used to quantify the function approximation power of GNNs, and also shows that GNNs, characterized by a TL fragment, can learn functions with separation power upper-bounded by a refinement algorithm corresponding to this fragment. Finally, the TL framework is used to establish new results. In particular, it shows that k-IGN cannot achieve expressiveness beyond (k-1)-WL, as this model corresponds to tk iterations of (k-1)-WL, and also offers insights as to the expressiveness of k-IGN with a polynomial number of layers: Indeed, model power does not increase with more standard layers, e.g., those relying on the adjacency matrix, but rather, any increase in layers can only improve expressiveness by deriving GNN functions with e.g., increased treewidth.

The paper proposed a novel way of evaluating the expressive power of any arbitrary GNNs in terms of WL test order using tensor language. The main motivation of this paper is that there is no straight forward expressive power evaluation tool so far. All existing methods need architecture specific proofs. Main challenge is the translating given GNN into tensor language. Later, they easily parametrize given GNN by two numbers which are the number of indices of tensor language ($k$ ) and number of summation depth ($t$). Then, they claimed that given GNN's expressive power is equal to $k-1$-WL test when it applies $t$ coloring rounds.