Title: Approximately Equivariant Graph Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural networks

equivariant machine learning

symmetry

generalization

statistical learning
Abstract: Graph neural networks (GNNs) are commonly described as being permutation equivariant with respect to node relabeling in the graph. This symmetry of GNNs is often compared to the translation equivariance of Euclidean convolution neural networks (CNNs). However, these two symmetries are fundamentally different: The translation equivariance of CNNs corresponds to symmetries of the fixed domain acting on the image signals (sometimes known as active symmetries), whereas in GNNs any permutation acts on both the graph signals and the graph domain (sometimes described as passive symmetries). In this work, we focus on the active symmetries of GNNs, by considering a learning setting where signals are supported on a fixed graph. In this case, the natural symmetries of GNNs are the automorphisms of the graph. Since real-world graphs tend to be asymmetric, we relax the notion of symmetries by formalizing approximate symmetries via graph coarsening. We present a bias-variance formula that quantifies the tradeoff between the loss in expressivity and the gain in the regularity of the learned estimator, depending on the chosen symmetry group. To illustrate our approach, we conduct extensive experiments on image inpainting, traffic flow prediction, and human pose estimation with different choices of symmetries. We show theoretically and empirically that the best generalization performance can be achieved by choosing a suitably larger group than the graph automorphism, but smaller than the permutation group.

Approximately Equivariant Graph Networks

Abstract

Graph neural networks (GNNs) are commonly described as being permutation equivariant with respect to node relabeling in the graph. This symmetry of GNNs is often compared to the translation equivariance of Euclidean convolution neural networks (CNNs). However, these two symmetries are fundamentally different: The translation equivariance of CNNs corresponds to symmetries of the fixed domain acting on the image signals (sometimes known as active symmetries ), whereas in GNNs any permutation acts on both the graph signals and the graph domain (sometimes described as passive symmetries ). In this work, we focus on the active symmetries of GNNs, by considering a learning setting where signals are supported on a fixed graph. In this case, the natural symmetries of GNNs are the automorphisms of the graph. Since real-world graphs tend to be asymmetric, we relax the notion of symmetries by formalizing approximate symmetries via graph coarsening. We present a bias-variance formula that quantifies the tradeoff between the loss in expressivity and the gain in the regularity of the learned estimator, depending on the chosen symmetry group. To illustrate our approach, we conduct extensive experiments on image inpainting, traffic flow prediction, and human pose estimation with different choices of symmetries. We show theoretically and empirically that the best generalization performance can be achieved by choosing a suitably larger group than the graph automorphism, but smaller than the permutation group.

1 Introduction

Graph Neural Networks (GNNs) are popular tools to learn functions on graphs. They are commonly designed to be permutation equivariant since the node ordering can be arbitrary (in the matrix representation of a graph). Permutation equivariance serves as a strong geometric prior and allows GNNs to generalize well [1-3]. Yet in many applications, the node ordering across different graphs is matched or fixed a priori, such as a time series of social networks where the nodes identify the same users, or a set of skeleton graphs where the nodes represent the same joints. In such settings, the natural symmetries arise from graph automorphisms, which effectively only act on the graph signals; This is inherently different from the standard equivariance in GNNs that concerns all possible permutations acting on both the signals and the graph domain. A permutation of both the graph and the graph signal can be seen as a change of coordinates since it does not change the object it represents, just the way to express it. This parallels the passive symmetries in physics, where physical observables are independent of the coordinate system one uses to express them [4, 5]. In contrast, a permutation of the graph signal on a fixed graph potentially transforms the object itself, not only its representation. This parallels the active symmetries in physics, where the coordinate system (in this case, the graph or domain) is fixed but a group transformation on the signal results in a predictable

transformation of the outcome (e.g., permuting left and right joints in the human skeleton changes a left-handed person to right-handed). The active symmetries on a fixed graph are similar to the translation equivariance symmetries in Euclidean convolutional neural networks (CNNs), where the domain is a fixed-size grid and the signals are images.

In this work, we focus on active symmetries in GNNs. Specifically, we consider a fixed graph domain G with N nodes, an adjacency matrix A ∈ R N × N , and input graph signals X ∈ R N × d . We are interested in learning equivariant functions f that satisfy (approximate) active symmetries

where G is a subgroup of the permutation group S N that depends on G . For example, G can be the graph automorphism group A G = { Π : Π A = A Π } . Each choice of G induces a hypothesis class H G for f : the smaller the group G , the larger the class H G . We aim to select G so that the learned function f ∈ H G generalizes well, also known as the model selection problem [6, Chp.4]. In contrast, standard graph learning methods (GNNs, spectral methods) are defined to satisfy passive symmetries, by treating A as input and requiring f (Π A Π ⊤ , Π X ) = Π f ( A,X ) for all permutations Π ∈ S N and all N . But for the fixed graph setting, we argue that active symmetries are more relevant. Thus we use A to define the hypothesis class rather than treating it an input. By switching from passive symmetries to active symmetries, we will show how to design GNNs for signals supported on a fixed graph with different levels of expressivity and generalization properties.

While enforcing symmetries has been shown to improve generalization when the symmetry group is known a priori [7-13], the problem of symmetry model selection is not completely solved, particularly when the data lacks exact symmetries (see for instance [14-16] and references therein). Motivated by this, we study the symmetry model selection problem for learning on a fixed graph domain. This setting is particularly interesting since (1) the graph automorphism group serves as the natural oracle symmetry; (2) real-world graphs tend to be asymmetric, but admit cluster structure or local symmetries. Therefore, we define approximate symmetries of graphs using the cut distance between graphs from graphon analysis. An approximate symmetry of a graph G is a symmetry of any other graph G ′ that approximates G in the cut distance. In practice, we take G ′ as coarse-grainings (or clusterings) of G , as these are typically guaranteed to be close in cut distance to G . We show how to induce approximate symmetries for G via the automorphisms of G ′ . Our main contributions include:

We formalize the notion of active symmetries and approximate symmetries of GNNs for signals supported on a fixed graph domain, which allows us to study the symmetry group model selection problem. (See Sections 2, 4)

We theoretically characterize the statistical risk depending on the hypothesis class induced from the symmetry group, and show a bias-variance tradeoff between the reduction in expressivity and the gain in regularity of the model. (See Sections 3, 4)

We illustrate our approach empirically for image inpainting, traffic flow prediction, and human pose estimation. (See Section 5 for an overview, and Appendix D for the details on how to implement equivariant graph networks with respect to different symmetry groups).

1.1 Related Work

Graph Neural Networks and Equivariant Networks. Graph Neural Networks (GNNs) [17-19] are typically permutation-equivariant (with respect to node relabeling). These include message-passing neural networks (MPNNs) [19-21], spectral GNNs [18, 22-24], and subgraph-based GNNs [2530]. Permutation equivariance in GNNs can extend to edge types [31] and higher-order tensors representing the graph structure [32, 33]. Equivariant networks generalize symmetries on graphs to other objects, such as sets [34], images [35], shapes [36, 37], point clouds [38-42], manifolds [1, 43, 44], and physical systems [45, 46] among many others. Notably, many equivariant machine learning problems concern the setting where the domain is fixed, e.g., learning functions on a fixed sphere [47], and thus focus on active symmetries rather than passive symmetries such as the node ordering in standard GNNs. Yet learning on a fixed graph domain arises naturally in many applications such as molecular dynamics modeling [48]. This motivates us to consider active symmetries of GNNs for learning functions on a fixed graph. Our work is closely related to Natural Graph Networks (NGNs) in [49], which use global and local graph isomorphisms to design maximally expressive GNNs for distinguishing different graphs . In contrast, we focus on generalization and thus consider symmetry model selection on a fixed graph.

Generalization of Graph Neural Networks and Equivariant Networks. Most existing works focus on the graph-level tasks, where in-distribution generalization bounds of GNNs have been derived using statistical learning-theoretic measures such as Rademacher complexity [50] and Vapnik-Chervonenkis (VC) dimension [51, 52], or uniform generalization analysis based on random graph models [53]; Out-of-distribution generalization properties have been investigated using different notions including transferability [54, 55] (also known as size generalization [56]), and extrapolation [57]. In general, imposing symmetry constraints improves the sample complexity and the generalization error [7-13]. Recently, Petrache and Trivedi [58] investigated approximation-generalization tradeoffs using approximate symmetries for general groups. Their results are based on uniform convergence generalization bounds, which measure the worst-case performance of all functions in a hypothesis class and differ from our non-uniform analysis.

Approximate Symmetries. For physical dynamical problems, Wang et al. [45] formalized approximate symmetries by relaxing exact equivariance to allow for a small equivariance error. For reinforcement learning applications, Finzi et al. [59] proposed Residual Pathway Priors to expand network layers into a sum of equivariant layers and non-equivariant layers, and thus relax strict equivariance into approximate equivariance priors. In the context of self-supervised learning, Suau et al. [60], Gupta et al. [61] proposed to learn structural latent representations that satisfy approximate equivariance. Inspired by randomized algorithms, Cotta et al. [62] formalizes probabilistic notions of invariances and universal approximation. In [63], scattering transforms (a specific realization of a CNN) are shown to be approximately invariant to small deformations in the image domain, which can be seen as a form of approximate symmetry. Scattering transforms on graphs are discussed in [64-66]. Additional discussions on approximate symmetries can be found in [67, Section 6].

2 Problem Setup: Learning Equivariant Maps on a Fixed Graph

Notations. We let R , R + denote the reals and the nonegative reals, I denote the identity matrix and ✶ denote the all-ones matrix. For a matrix Y ∈ R N × k , we write the Frobenious norm as ∥ Y ∥ F . We denote by X ⊙ Y the element-wise multiplication of matrices X and Y . We write [ N ] = { 1 , . . . , N } and S N as the permutation group on the set [ N ] . Groups are typically noted by calligraphic letters. Given H , K groups, we denote a semidirect product by H ⋊ K . For a group G with representation ϕ , we denote by χ ϕ | G the corresponding character (see Definition 6 in Appendix A.1). Denoting H ≤ G means that H is a subgroup of G .

Equivariance. We consider a compact group G with Haar measure λ (the unique G -invariant probability measure on G ). Let G act on spaces X and Y by representations ϕ and ψ , respectively. We say that a map f : X → Y is G -equivariant if for all g ∈ G , x ∈ X , ψ ( g -1 ) f ( ϕ ( g ) x ) = f ( x ) . Given any map f : X → Y , a projection of f onto the space of G -equivariant maps can be computed by averaging over orbits with respect to λ

For u, v ∈ Y , let ⟨ u, v ⟩ be a G -invariant inner product, i.e. ⟨ ψ ( g ) u, ψ ( g ) v ⟩ = ⟨ u, v ⟩ , for all g ∈ G , for all u, v ∈ Y . Given two maps f 1 , f 2 : X → Y , we define their inner product as ⟨ f 1 , f 2 ⟩ µ = ∫ X ⟨ f 1 ( x ) , f 2 ( x ) ⟩ d µ ( x ) , where µ is a G -invariant measure on X . Let V be the space of all (measurable) map f : X → Y such that ∥ f ∥ µ = √ ⟨ f, f ⟩ µ < ∞ .

Graphs. We consider edge-node weighted graphs G = ([ N ] , A, b ) , where [ N ] is a finite set of nodes, A ∈ [0 , 1] N × N is the adjacency matrix describing the edge weights, and b = { b 1 , . . . , b N } ⊂ R are the node weights. An edge weighted graph is a special case of G where all node weights are 1 . A simple graph is a special case of an edge weighted graph, where all edge weights are binary. Let A G be the automorphism group of a graph defined as A G := { Π ∈ S N : Π A Π ⊤ = A, Π b = b } , which characterizes the symmetries of G . Hereinafter, G is assumed to be a subgroup of S N .

Graph Signals and Learning Task. We consider graph signals supported in the nodes of a fixed graph G and maps between graphs signals. Let X = R N × d , Y = R N × k be the input and output graph signal spaces. We denote by f a map between graph signals, f : X → Y . Even though the functions f depend on G , we don't explicitly write G as part of the notation of f because G is fixed. Our goal is to learn a target map between graph signals f ∗ : X → Y . To this end, we assume access to a training set { ( X i , Y i ) } where the X i are i.i.d. sampled from an S N -invariant distribution µ on

X where S n acts on X by permuting the rows, and Y i = f ∗ ( X i ) + ξ i for some noise ξ i . A natural assumption is that f ∗ is approximately equivariant with respect to A G in some sense. Our symmetry model selection problem concerns a sequence of hypothesis class {H G } indexed by G , where we choose the best class H ∗ G such that the estimator ˆ f ∈ H ∗ G gives the best generalization performance.

Equivariant Graph Networks. We propose to learn the target function f ∗ on the fixed graph using G -equivariant graph networks ( G -Net), which are equivariant to a chosen symmetry group G depending on the graph domain. Using standard techniques from representation theory, such as Schur's lemma and projections to isotypic components [68], we can parameterize G -Net by interleaving G -equivariant linear layers f G : R N × d → R N × k with pointwise nonlinearity, where the weights in the linear map f G are constrained in patterns depending on the group structure (also known as parameter sharing or weight tying [69], see Appendix A.1 for technical details). In practice, we can make G -Net more flexible by systematically breaking the symmetry, such as incorporating graph convolutions (i.e., Af G ) and locality constraints (i.e., A ⊙ f G ). Compared to standard GNNs (described in Appendix D.1), G -Net uses a more expressive linear map to gather global information (i.e., weights are not shared among all nodes). Importantly, G -Net yields a suite of models that allows us to flexibly choose the hypothesis class (and estimator) reflecting the active symmetries in the data, and subsumes standard GNNs that are permutation-equivariant with respect to passive symmetries (but not necessarily equivariant with respect to active symmetries).

Graphons. A graphon is a symmetric measurable function W : [0 , 1] 2 → [0 , 1] . Graphons represent (dense) graph limits where the number of nodes goes to infinity; they can also be viewed as random graph models where the value W ( x, y ) represents the probability of having an edge between the nodes x and y . Let W denote the space of all graphons and η denote the Lebesgue measure. Lovász and Szegedy [70] introduced the cut norm on W as

Based on the cut norm, Borgs et al. [71] defined the cut distance between two graphons W,U ∈ W ,

where S [0 , 1] is the set of all measure-preserving bijective measurable maps between [0 , 1] and itself, and U f ( x, y ) = U ( f ( x ) , f ( y )) . Note that the cut distance is 'permutation-invariant,' where measure preserving bijections are seen as the continuous counterparts of permutations.

3 Generalization with Exact Symmetry

In this section, we assume the target map f ∗ is A G -equivariant and study the symmetry model selection problem by comparing the (statistical) risk of different models. Concretely, the risk quantifies how a given model performs on average on any potential input. The smaller the risk, the better the model performs. Thus, we use the risk gap of two functions f, f ′ , defined as

as our model selection metric. Following ideas from Elesedy and Zaidi [8], we analyze the risk gap between any function f ∈ V and its G -equivariant version, for G ⊆ S N .

Lemma 1 (Risk Gap) . Let X = R N × d , Y = R N × k be the input and output graph signal spaces on a fixed graph G . Let X ∼ µ where µ is a S N -invariant distribution on X . Let Y = f ∗ ( X ) + ξ , where ξ ∈ R N × k is random, independent of X with zero mean and finite variance and f ∗ : X → Y is A G -equivariant. Then, for any f ∈ V and for any compact group G ⊆ S N , we can decompose f as

where ¯ f G = Q G f, f ⊥ G = f -¯ f G . Moreover, the risk gap satisfies

Lemma 1, proven in Appendix B.1, adapts ideas from [8] to equate the risk gap to the symmetry 'mismatch' between the chosen group G and the target group A G , plus the symmetry 'constraint' captured by the norm of the anti-symmetric part of f with respect to G . Our symmetry model selection problem aims to find the group G ⊆ S N such that the risk gap is maximized. Note that for G ≤ A G , the mismatch term vanishes since f ∗ is A G -equivariant, but the constraint term decreases with G ; When G = A G , we recover Lemma 6 in [8]. On the other hand, for G > A G , the mismatch term can be positive, negative or zero (depending on f ∗ ) whereas the constraint term increases with G .

3.1 Linear Regression

In this section, we focus on the linear regression setting and analyze the risk gap of using the equivariant estimator versus the vanilla estimator. We consider linear estimator ˆ Θ : R N × d → R N × k that predicts ˆ Y = f ˆ Θ ( X ) = ˆ Θ ⊤ X . Given a compact group G and any linear map Θ , we obtain its G -equivariant version via projection to the G -equivariant space (also known as intertwiner average),

We denote Ψ ⊥ G (Θ) = Θ -Ψ G (Θ) as the projection of ˆ Θ to the orthogonal complement of the G -equivariant space. Here Ψ G instantiates the orbit-average operator Q G (eqn. 2) for linear functions.

Theorem 2 (Bias-Variance-Tradeoff) . Let X = R N × d , Y = R N × k be the graph signals spaces on a fixed graph G . Let S N act on X and Y by permuting the rows with representations ϕ and ψ . Let G be a subgroup of S N acting with restricted representations ϕ | G on X and ψ | G on Y . Let X [ i,j ] i.i.d. ∼ N ( 0 , σ 2 X ) and Y = f ∗ ( X ) + ξ where f ∗ ( x ) = Θ ⊤ x is A G -equivariant and Θ ∈ R Nd × Nk . Assume ξ [ i,j ] is random, independent of X , with mean 0 and E [ ξξ ⊤ ] = σ 2 ξ I < ∞ . Let ˆ Θ be the least-squares estimate of Θ from n i.i.d. examples { ( X i , Y i ) : i = 1 , . . . , n } , Ψ G ( ˆ Θ) be its equivariant version with respect to G . Let ( χ ψ | G | χ ϕ | G ) = ∫ G χ ψ | G ( g ) χ ϕ | G ( g )d λ ( g ) denote the inner product of the characters. If n > Nd +1 the risk gap is

The bias term depends on the anti-symmetric part of Θ with respect to G , whereas the variance term captures the difference of the dimension between the space of linear maps R Nd → R Nk (measured by N 2 dk ), and the space of G -equivariant linear maps (measured by ( χ ψ | G | χ ϕ | G ) ; a proof can be found in [72, Section 2.2]). The bias term reduces to zero for G ≤ A G and we recover Theorem 1 in [8] when G = A G . Notably, by using a larger group G , the dimension of the equivariant space measured by ( χ ψ | G | χ ϕ | G ) is smaller, and thus the variance term increases; meanwhile, the bias term decreases due to extra (symmetry) constraints on the estimator. We remark that ( χ ψ | G | χ ϕ | G ) depends on d, k as well: For d = k = 1 , ϕ ( g ) = ψ ( g ) ∈ R N × N are standard permutation matrices; For general d > 1 , k > 1 , ϕ ( g ) ∈ R Nd × Nd , ψ ( g ) ∈ R Nk × Nk are block-diagonal matrices, where each block is a permutation matrix.

To understand the significance of the risk gap, we note that (see Appendix B.1) when n > Nd +1 , the risk of the least square estimator is

Thus, when n is small enough so that the risk gap in Theorem 2 is dominated by the variance term, enforcing equivariance gives substantial generalization gain of order N 2 dk -( χ ψ | G | χ ϕ | G ) n -Nd -1 .

Example 3.1. In Appendix C.1 we construct an example with X = R 3 , Y = R 3 , and x ∼ N (0 , σ 2 X I d ) . We consider a target linear function f ∗ that is S 2 -equivariant and approximately S 3 -equivariant, and compare the estimators f ψ S 2 ( ˆ Θ) versus f ψ S 3 ( ˆ Θ) . When the number of training samples n is small,

blow-up

[Defn 1]

[Defn 2]

[Defn 3]

[Defn 4]

(1,1)

(1,6)

(1,1)

(1,6)

(a)

(b)

(c)

(1,6)

(1,1)

(1,6)

(1,1)

using a S 3 -equivariant least square estimator yields better test error than the S 2 -equivariant one, as shown in figure inset. The dashed vertical line denotes the theoretical threshold n ∗ ≈ 35 , before which using S 3 yields better generalization than S 2 . This is an example where enforcing more symmetries than the target (symmetry) results on better generalization properties.

4 Generalization with Approximate Symmetries

Large graphs are typically asymmetric. Therefore, the assumption of f ∗ being A G -equivariant in Section 3 becomes trivial. Yet, graphon theory asserts that graphs can be seen as living in a 'continuous' metric space, with the cut distance (Definition 1). Since graphs are continuous entities, the combinatorial notion of exact symmetry of graphs is not appropriate. We hence relax the notion of exact symmetry to a property that is 'continuous' in the cut distance. The regularity lemma [73, Theorem 5.1] asserts that any large graph can be approximated by a coarser graph in the cut distance. Since smaller graphs tend to exhibit more symmetries than larger ones, we are motivated to consider the symmetries of coarse-grained versions of graphs as their approximate symmetries (Definition 2, 3). We then present the notion of approximately equivariant mappings (Definition 4), which allows us to precisely characterize the bias-variance tradeoff (Corollary 3) in the approximate symmetry setting based on Lemma 1. Effectively, we reduce selecting the symmetry group to choosing the coarsened graph G ′ . Moreover, our symmetry model selection perspective allows for exploring different coarsening procedures and choosing the one that works best for the problem.

Definition 1 (Induced graphon) . Let G be a (possibly edge weighted) graph with node set [ N ] and adjacency matrix A = { a i,j ∈ [0 , 1] } N i,j =1 . Let P N = { I n N = ( n -1 N , n N ) } N n =1 be the equipartition of [0 , 1] to N intervals (where formally the last interval is the closed interval [ N -1 N , 1] ). We define the graphon W G induced by G as

where ✶ I i N is the indicator function of the set I i N ⊂ [0 , 1] , i = 1 , . . . , N . We induce a graphon from an edge-node weighted graph G ′ = ([ M ] , A, b ) , with rational node weights b i , as follows. Let N ∈ N be a value such that the node weights can be written in the form b = { b m = q m N } M m =1 , for some q m ∈ N ≥ 0 , m = 1 , . . . , M . We blow-up G ′ to an edge weighted graph G N of ∑ q m nodes by splitting each node m of G ′ into q m nodes { n j m } q m j =1 of weight 1 , and defining the adjacency matrix A N with entries a n j m ,n j ′ m ′ = a m,m ′ . Note that for any two choices N 1 , N 2 ∈ N in the above construction, δ □ ( W G N 1 , W G N 2 ) = 0 , where the infimum in the definition of δ □ is realized on some permutation of intervals in [0 , 1] , as explained below. We hence define the induced graphon of the edge-node weighted graph G ′ by W G ′ := W G N for some N , where, in fact, each choice of N gives a representative of an equivalence class of piecewise constant graphons with zero δ □ distance.

Definition 2. Let G ′ be an edge-node weighted graph with M nodes and node weights b = { b m = q m N ′ } M m =1 , satisfying { q m } m ⊂ N ≥ 0 and ∑ q m = N , and let G be an edge weighted graph with N

nodes. We say that G ′ coarsens G up to error ϵ if

Suppose that G ′ coarsens G . This implies the existence of an assignment of nodes of G to nodes of G ′ . Namely, the supremum underlying the definition of δ □ over the measure preserving bijections ϕ : [0 , 1] → [0 , 1] is realized by a permutation Π : [ N ] → [ N ] applied on the intervals { I i N } N i =1 . With this permutation, the assignment C G → G ′ defined by [ N ] ∋ Π( n j m ) ↦→ m ∈ [ M ] , for every m = 1 , . . . , M and j = 1 , . . . , q m , is called a cluster assignment of G to G ′ .

Let G ′ , with M nodes, be a edge-node weighted graph that coarsens the simple graph G of N nodes. Let C G → G ′ be a cluster assignment of G to G ′ . Let A G ′ be the automorphism group of G ′ . Note that two nodes can belong to the same orbit of A G ′ only if they have the same node weight. Hence, for every two nodes m,m ′ in the same orbit of A G ′ (i.e., equivalent clusters), there exists a bijection ψ m,m ′ : C -1 G → G ′ { m } → C -1 G → G ′ { m ′ } . We choose a set of such bijections, such that for every m,m ′ on the same orbit of A G ′

Wemoreover choose ψ m,m as the identity mapping for each m = 1 , . . . , M . We identify each element g of A G ′ with an element g in a permutation group A G ′ of [ N ] , called the blown-up symmetry group , as follows. Given g ∈ A G ′ , for every m ∈ [ M ] and every n ∈ C -1 G → G ′ { m } , define

Definition 3. Let G ′ , with M nodes, be a edge-node weighted graph that coarsen the (simple or weighted) graph G of N nodes. Let C G → G ′ be a cluster assignment of G to G ′ with cluster sizes c 1 , . . . , c M . Let A G ′ be the automorphism group of G ′ , and A G ′ be the blown-up symmetry group of [ N ] . For every m = 1 , . . . M , let S c m be the symmetry group of C -1 G → G ′ { m } . We call the group of permutations

the symmetry group of G induced by the coarsening G ′ . We call any element of G G → G ′ an approximate symmetry of G .

Specifically, every element s ∈ G G → G ′ can be written in coordinates by

for a unique choice of s j ∈ S c j , j = 1 , . . . , M , and a ∈ A G ′ (see Appendix B.2 for details).

In words, we consider the symmetry of the graph G of N nodes not by its own automorphism group, but via the symmetry of its coarsened graph G ′ . The nodes in the same orbit in G are either in the same cluster of G ′ (i.e., from the same coarsened node), or in the equivalent clusters of G ′ (i.e., they belong to the same orbit of the coarsened graph and share the same cluster size). See Figure 1 (and Figure 9 in Appendix A.2) for examples.

Definition 4. Let G be a graph with N nodes, and X = R N × d be the space of graph signals. Let X ∼ µ where µ is a S N -invariant measure on R N × d . We call f ∗ : R N × d → R N × k an approximately equivariant mapping if there exists a function κ : R + → R + satisfying lim ϵ → 0 κ ( ϵ ) = 0 (called the equivariance rate ), such that for every ϵ > 0 and every edge-node weigted graph G ′ that coarsen G up to error ϵ ,

where f ∗ G G → G ′ = Q G G → G ′ ( f ) is the intertwining projection of f ∗ with respect to G G → G ′ .

Example 4.1. Here we give an example of an approximately equivariant mapping in a natural setting. Suppose that G is a random geometric graph, namely, a graph that was sampled from a metric-probability space M by randomly and independently sampling nodes { x n } N n =1 ⊂ M . Hence, G can be viewed as a discretization of M , and the graphs that coarsen G can be seen as coarser discretizations of M . Therefore, the approximate symmetries are local deformations of G , namely, permutations that swap nodes if they are close in the metric of M . This now gives an interpretation for approximately equivariance mappings of geometric graphs: these are mappings that are stable to local deformations. In Appendix C.2 we develop this example in detail.

We are ready to present the bias-variance tradeoff in the setting with approximate symmetries. The proofs are deferred to Appendix B.2.

Corollary 3 (Risk Gap via Graph Coarsening) . Let X = R N × d , Y = R N × k be the input and output graph signal spaces on a fixed graph G . Let X ∼ µ where µ is a S N -invariant distribution on X . Let Y = f ∗ ( X ) + ξ , where ξ ∈ R N × k is random, independent of X with zero mean and finite variance, and f ∗ : R N × d → R N × k be an approximately equivariant mapping with equivariance rate κ . Then, for any G ′ that coarsens G up to error ϵ , for any f ∈ V , we have

Notably, Corollary 3 illustrates the tradeoff explicitly in the form of choosing the coarsened graph G ′ : If we choose G ′ close to G such that the coarsening error ϵ and ∥ f ⊥ G G → G ′ ∥ µ are small, then the mismatch term is close to zero; meanwhile the constraint term is also small since A G is typically trivial. On the other hand, if we choose G ′ far from G that yields large coarsening error ϵ , then the constraint term ∥ f ⊥ G G → G ′ ∥ 2 µ also increases.

Corollary 4 (Bias-Variance-Tradeoff via Graph Coarsening) . Consider the same linear regression setting in Theorem 2, except now f ∗ is an approximately equivariant mapping with equivariance rate κ , and G = G G → G ′ is controlled by G ′ that coarsens G up to error ϵ . Denote the canonical permutation representations of G G → G ′ on X , Y as ϕ ′ , ψ ′ , respectively. Let ( χ ψ ′ | χ ϕ ′ ) = ∫ G G → G ′ χ ψ ′ ( g ) χ ϕ ′ ( g )d λ ( g ) denote the inner product of the characters. If n > Nd + 1 the risk gap is bounded by

5 Experiments

We illustrate our theory in three real-world tasks for learning on a fixed graph: image inpainting, traffic flow prediction, and human pose estimation 1 . For image inpainting, we demonstrate the biasvariance tradeoff via graph coarsening using G -Net; For the other two applications, we show that the tradeoff not only emerges from G -Net that enforces strict equivariance, but also G -Net augmented with symmetry-breaking modules, allowing us to recover standard GNN architectures as a special case. Concretely, we consider the following variants (more details in Appendix D.4.2):

G -Net with strict equivariance using equivariant linear map f G .

G -Net augmented with graph convolution Af G ( x ) , denoted as G -Net(gc).

G -Net augmented with graph convolution and learnable edge weights: G -Net(gc+ew).

G -Net augmented with graph locality constraints ( A ⊙ f G )( x ) and learnable edge weights, denoted as G -Net(pt+ew).

G -Net serves as a baseline to validate our generalization analysis for equivariant estimators (see Section 3). Yet in practice, we observe that augmenting G -Net with symmetry breaking can further improve performance, thereby justifying the analysis for approximate symmetries (see Section 4). In particular, G -Net(gc) and G -Net(gc+ew) are motivated by graph convolutions used in standard GNNs; G -Net(pt+ew) is inspired by the concept of locality in CNNs, where A ⊙ f G effectively restricts the receptive field to the 1 -hop neighrborhood induced by the graph A .

5.1 Application: Image Inpainting

We consider a 28 × 28 grid graph as the fixed domain, with grey-scale images as the graph signals. The learning task is to reconstruct the original images given masked images as inputs (i.e., image inpainting). We use subsets of MNIST [74] and FashionMNIST [75], each comprising 100 training samples and 1000 test samples. The input and output graph signals are ( m i ⊙ x i , x i ) , where x i ∈ R 28 × 28 ≡ R 784 denotes the image signals and m i denotes a random mask (size 14 × 14 for MNIST and 20 × 20 for FashionMNIST). We investigate the symmetry model selection problem by clustering

the grid into M patches with size d × d , where d ∈ { 28 , 14 , 7 , 4 , 2 , 1 } . Here d = 28 means one cluster (with S N symmetry); d = 1 is 784 singleton clusters with no symmetry (trivial).

We consider G G → G ′ -equivariant networks G -Net with ReLU nonlinearity. We parameterize the equivariant linear layer f : R N → R N with respect to G G → G ′ = ( S c 1 × . . . ×S c M ) ⋊ A G ′ using the following block-matrix form (assuming the nodes are ordered by their cluster assignment), with f kl denoting block matrices, and a k , b k , e kl representing scalars:

̸

The coarsened graph symmetry A G ′ induces constraints on a k , b k , e kl . If A G ′ is trivial, then these scalars are unconstrained. In the experiment, we consider a reflection symmetry on the coarsened grid graph, i.e., A G ′ = S 2 which acts by reflecting the left (coarsened) patches to the right (coarsened) patches. Suppose the reflected patch pairs are ordered consecutively, then a k = a k +1 , b k = b k +1 for k ∈ { 1 , 3 , . . . , M -1 } , and e kl = e k +1 ,l -1

for k ∈ { 1 , 3 , . . . , M -1 } , l ∈ { 2 , 4 , . . . , M } (see Figure inset for an illustration). In practice, we extend the parameterization to f : R N × d → R N × k . More details can be found in Appendix D.2. (1,1) (1,2) (1,3) (1,4)

Coarsened 2×2 Grid

f:R→R4,unique=8

1

2

Figure 2 shows the empirical risk first decreases and then increases as the group decreases, illustrating the bias-variance tradeoff from our theory. Figure 2 (left) compares a 2 -layer G -Net with a 1 -layer linear G -Net, demonstrating that the tradeoff occurs in both linear and nonlinear models. Figure 2 (right) shows that using reflection symmetry of the coarsened graph outperforms the trivial symmetry baseline, highlighting the utility of modeling coarsened graph symmetries.

MNIST

FashionMNIST

FashionMNiST(+hflip) with Reflection Symmetry Ag = S2

0.26

0.44

2-layer C-Net

2-layer C-Net

Trivial AG

0.42

1-layer

0.24

1-layer

0.22

(Sn/4)4S2

AG=S2

0.22

MSE

0.40

3SW

0.20

0.20

Test

0.38

0.18

0.18

0.36

0.16

0.16

(Sn/4g)49>S2

(Sn/ige)196>4S2

0.34

0.14

0.14

Sn

(Sn/4)4

(Sn/6)16

(Sn/49)49

(Sn/196)196 (Sn784)784

Sn

(Sn4)4

(Sn/16)6

(Sn/49)49

(Sn/196)196 (Sn784)784

Sn

(Sn4)4

(Sn6)26

(Sn/49)49

(Sn/196)196 (Sn784)784

5.2 Application: Traffic Flow Prediction

The traffic flow prediction problem can be formulated as follows: Let X ( t ) ∈ R n × d represent the traffic graph signal (e.g., speed or volume) observed at time t . The goal is to learn a function h ( · ) that maps T ′ historical traffic signals to T future graph signals, assuming the fixed graph domain G :

The traffic graph is typically large and asymmetric. Therefore we leverage our approximate symmetries to study the symmetry model selection problem (Section 4). We use the METR-LA dataset which represents traffic volume of highways in Los Angeles (see figure inset). We use both the original graph G from [76], and its sparsified version G s which is more faithful to the road geometry. In G s , the ( i, j ) -edge exists if and only if nodes i, j lie on the

same highway. We first validate our Definition 4 in such dataset (see Appendix D.3.1). In terms of the models, we consider a simplified version of DCRNN model proposed in [76], where the spatial module is modelled by a standard GNN, and the temporal module is modelled via an encoder-decoder architecture for sequence-to-sequence modelling. We follow the same temporal module and extending its GNN module using G -Net(gc), which recovers DCRNN when choosing S N . We then consider different choices of the coarsened graph G ′ (to induce approximate symmetries). As shown in Table 3, using 2 clusters as approximate symmetries yields better generalization error than using 9 clusters, or the full permutation group S N .

5.3 Application: Human Pose Estimation

We consider the simple (loopy) graph G with adjacency matrix A ∈ { 0 , 1 } 16 × 16 representing 16 human joints, illustrated on the right. The input graph signals X ∈ R 16 × 2 describe the joint 2D spatial location. The goal is to learn a map to reconstruct the 3D pose Y ∈ R 16 × 3 . We use the standard benchmark dataset Human3.6M [77] and follow the evaluation protocol in [78]. The generalization performance is measured by mean per joint position error (MPJPE) and MPJPE after aligntment (P-MPJPE). We implement a 4 -Layer G -Net(gc+ew), which recovers the same model architecture in SemGCN [78] when choosing G = S 16 (further details are provided in Appendix D.4.1).

̸

We consider the following symmetry groups: the permutation group S 16 , the graph automorphism A G , and the trivial symmetry. Note that the human skeleton graph has A G = ( S 2 ) 2 (corresponding to the arm flip and leg flip). Additionally, we investigate an approximate symmetry called RelaxS 16 ; This relaxes the global weight sharing in S 16 (that only learns 2 scalars per equivariant linear layer f S 16 : R N → R N , where f S 16 [ i, i ] = a, f S 16 [ i, j ] = b for i = j ) to local weight sharing (that learns 2 × 16 = 32 scalars, where f RelaxS 16 [ i, i ] = a i , f RelaxS 16 [ i, j ] = b i for i = j ). Table 4 shows that the best performance is achieved at the hypothesis class induced from RelaxS 16 , which is smaller than S 16 but differs from A G . Furthermore, using G -Net(gc+ew) with RelaxS 16 gives a substantial improvement over S 16 (representing SemGCN in [78]). This demonstrates the utility of enforcing active symmetries in GNNs that results in more expressive models. More details and additional ablation studies can be found in Appendix D.4.

6 Discussion

In this paper, we focus on learning tasks where the graph is fixed, and the dataset consists of different signals on the graph. We developed an approach for designing GNNs based on active symmetries and approximate symmetries induced by the symmetries of the graph and its coarse-grained versions. A layer of an approximately equivariant graph network uses a linear map that is equivariant to the chosen symmetry group; the graph is not used as an input but rather induces a hypothesis class. In practice, we further break the symmetry by incorporating the graph in the model computation, thus combining symmetric and asymmetric components.

We theoretically show a bias-variance tradeoff between the loss of expressivity due to imposing symmetries, and the gain in regularity, for settings where the target map is assumed to be (approximately) equivariant. For simplicity, the theoretical analysis focuses on the equivariant models without symmetry breaking; Theoretically analyzing the combination of symmetric and asymmetric components in machine learning models is an interesting open problem. The bias-variance formula is computed only for a simple linear regression model with white noise and in the underparamterized setting; Extending it to more realistic models and overparameterized settings is a promising direction.

As a proof of concept, our approximately equivariant graph networks only consider symmetries of the fixed graph. An interesting future direction is to extend our approach to also account for (approximate) symmetries in node features and labels, using suitably generalized cut distance (e.g., graphon-signal cut distance in [79]; see Appendix C.2 for an overview). Our network architecture consists only of linear layers and pointwise nonlinearity. Another promising direction is to incorporate pooling layers (e.g., [22, 80]) and investigate them through the lens of approximate symmetries. Finally, extending our analysis to general groups is a natural next step. Our techniques are based on compact groups with orthogonal representation. While we believe that the orthogonality requirement can be lifted straightforwardly, relaxing the compactness requirement appears to be more challenging (see related discussion in [46]).

̸

Acknowledgments

This research project was started at the BIRS 2022 Workshop 'Deep Exploration of non-Euclidean Data with Geometric and Topological Representation Learning' held at the UBC Okanagan campus in Kelowna, B.C. The authors thank Ben Blum-Smith (JHU), David Hogg (NYU), Erik Thiede (Cornell), Wenda Zhou (Open AI), Carey Priebe (JHU), and Rene Vidal (UPenn) for valuable discussions, and Hong Ye Tan (Cambridge) for pointing out typos in an earlier version of the manuscript. The authors also thank the anonymous NeurIPS reviewers and area chair for helpful feedback. Ningyuan Huang is partially supported by the MINDS Data Science Fellowship from Johns Hopkins University. Ron Levie is partially funded by ISF (Israel Science Foundation) grant #1937/23. Soledad Villar is partially funded by the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (MoDL) (NSF DMS 2031985), NSF CISE 2212457, ONR N00014-22-1-2126 and an Amazon AI2AI Faculty Research Award.

References

M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine , 34(4):18-42, 2017.

Fernando Gama, Joan Bruna, and Alejandro Ribeiro. Stability properties of graph neural networks. IEEE Transactions on Signal Processing , 68:5680-5695, 2020.

Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021.

Carlo Rovelli and Marcus Gaul. Loop quantum gravity and the meaning of diffeomorphism invariance. In Towards Quantum Gravity: Proceeding of the XXXV International Winter School on Theoretical Physics Held in Polanica, Poland, 2-11 February 1999 , pages 277-324. Springer, 2000.

Soledad Villar, David W Hogg, Weichi Yao, George A Kevrekidis, and Bernhard Schölkopf. The passive symmetries of machine learning. arXiv preprint arXiv:2301.13724 , 2023.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Alberto Bietti, Luca Venturi, and Joan Bruna. On the sample complexity of learning under invariance and geometric stability, 2021.

Bryn Elesedy and Sheheryar Zaidi. Provably strict generalisation benefit for equivariant models. In International Conference on Machine Learning , pages 2959-2969. PMLR, 2021.

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Learning with invariances in random features and kernel models. arXiv preprint arXiv:2102.13219 , 2021.

Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy. On the benefits of invariance in neural networks. arXiv preprint arXiv:2005.00178 , 2020.

Bryn Elesedy. Group symmetry in pac learning. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning , 2022.

Bryn Elesedy. Provably strict generalisation benefit for invariance in kernel methods. Advances in Neural Information Processing Systems , 34:17273-17283, 2021.

Arash Behboodi, Gabriele Cesa, and Taco Cohen. A pac-bayesian generalization bound for equivariant networks. arXiv preprint arXiv:2210.13150 , 2022.

Gregory Benton, Marc Finzi, Pavel Izmailov, and Andrew G Wilson. Learning invariances in neural networks from training data. Advances in neural information processing systems , 33: 17605-17616, 2020.

Jameson Cahill, Dustin G Mixon, and Hans Parshall. Lie pca: Density estimation for symmetric manifolds. Applied and Computational Harmonic Analysis , 65:279-295, 2023.

Vasco Portilheiro. A tradeoff between universality of equivariant models and learnability of symmetries. arXiv preprint arXiv:2210.09444 , 2022.

F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE Transactions on Neural Networks , 20(1):61-80, 2009.

J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun. Spectral networks and deep locally connected networks on graphs. In International Conference on Learning Representation , 2014.

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning , pages 1263-1272, 2017.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems , pages 3837-3845, 2016.

F. Gama, A. G. Marques, G. Leus, and A. Ribeiro. Convolutional neural network architectures for signals supported on graphs. IEEE Transactions on Signal Processing , 67(4):1034-1049, 2019.

R. Levie, F. Monti, X. Bresson, and M. M. Bronstein. Cayleynets: Graph convolutional neural networks with complex rational spectral filters. IEEE Transactions on Signal Processing , 67(1): 97-109, 2019.

Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count substructures? Advances in neural information processing systems , 33:10383-10395, 2020.

Erik Henning Thiede, Wenda Zhou, and Risi Kondor. Autobahn: Automorphism-based graph neural nets, 2021. URL https://arxiv.org/abs/2103.01710 .

Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. arXiv preprint arXiv:2110.02910 , 2021.

Giorgos Bouritsas, Fabrizio Frasca, Stefanos P Zafeiriou, and Michael Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Fabrizio Frasca, Beatrice Bevilacqua, Michael M Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. arXiv preprint arXiv:2206.11140 , 2022.

Bohang Zhang, Guhao Feng, Yiheng Du, Di He, and Liwei Wang. A complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests. CoRR , abs/2302.07090, 2023.

Jianfei Gao, Yangze Zhou, and Bruno Ribeiro. Double permutation equivariance for knowledge graph completion. arXiv preprint arXiv:2302.01313 , 2023.

Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. arXiv preprint arXiv:1812.09902 , 2018.

Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI Conference on Artificial Intelligence , pages 4602-4609, 2019.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems , 30, 2017.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning , pages 2990-2999. PMLR, 2016.

Yashil Sukurdeep et al. Elastic shape analysis of geometric objects with complex structures and partial correspondences . PhD thesis, Johns Hopkins University, 2023.

Jameson Cahill, Joseph W Iverson, Dustin G Mixon, and Daniel Packer. Group-invariant max filtering. arXiv preprint arXiv:2205.14039 , 2022.

Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv:1802.08219 , 2018.

Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Neural Information Processing Systems , 33, 2020.

Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. arXiv preprint arXiv:2207.09453 , 2022.

Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. Advances in Neural Information Processing Systems , 34:28848-28863, 2021.

Ben Finkelshtein, Chaim Baskin, Haggai Maron, and Nadav Dym. A simple and universal rotation equivariant point-cloud network. In Topological, Algebraic and Geometric Learning Workshops 2022 , pages 107-115. PMLR, 2022.

Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-gordan nets: a fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems , 31:10117-10126, 2018.

Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolutional networks-isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2106.06020 , 2021.

Rui Wang, Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly symmetric dynamics. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 2307823091. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/wang22aa. html .

Soledad Villar, Weichi Yao, David W Hogg, Ben Blum-Smith, and Bianca Dumitrascu. Dimensionless machine learning: Imposing exact units equivariance. Journal of Machine Learning Research , 24(109):1-32, 2023.

Taco S. Cohen, Mario Geiger, Jonas Koehler, and Max Welling. Spherical cnns, 2018.

Jiaqi Han, Wenbing Huang, Tingyang Xu, and Yu Rong. Equivariant graph hierarchy-based neural networks. Advances in Neural Information Processing Systems , 35:9176-9187, 2022.

Pim de Haan, Taco S Cohen, and Max Welling. Natural graph networks. Advances in Neural Information Processing Systems , 33:3636-3646, 2020.

Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and representational limits of graph neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 3419-3430. PMLR, 13-18 Jul 2020. URL https://proceedings. mlr.press/v119/garg20c.html .

Pascal Esser, Leena Chennuru Vankadara, and Debarghya Ghoshdastidar. Learning theory can (sometimes) explain generalisation in graph neural networks. Advances in Neural Information Processing Systems , 34:27043-27056, 2021.

Christopher Morris, Floris Geerts, Jan Tönshoff, and Martin Grohe. Wl meet vc, 2023.

Sohir Maskey, Ron Levie, Yunseok Lee, and Gitta Kutyniok. Generalization analysis of message passing neural networks on large random graphs. In Advances in Neural Information Processing Systems , 2022.

Ron Levie, Wei Huang, Lorenzo Bucci, Michael Bronstein, and Gitta Kutyniok. Transferability of spectral graph convolutional neural networks. Journal of Machine Learning Research , 22 (272):1-59, 2021.

Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. Graphon neural networks and the transferability of graph neural networks. Advances in Neural Information Processing Systems , 33: 1702-1712, 2020.

Beatrice Bevilacqua, Yangze Zhou, and Bruno Ribeiro. Size-invariant graph representations for graph classification extrapolations. In International Conference on Machine Learning , pages 837-851. PMLR, 2021.

Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009.11848 , 2020.

Mircea Petrache and Shubhendu Trivedi. Approximation-generalization trade-offs under (approximate) group equivariance, 2023.

Marc Finzi, Gregory Benton, and Andrew G Wilson. Residual pathway priors for soft equivariance constraints. Advances in Neural Information Processing Systems , 34:30037-30049, 2021.

Xavier Suau, Federico Danieli, T. Anderson Keller, Arno Blaas, Chen Huang, Jason Ramapuram, Dan Busbridge, and Luca Zappella. Duet: 2d structured and approximately equivariant representations. In International Conference on Machine Learning . PMLR, 2023.

Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Structuring representation geometry with rotationally equivariant contrastive learning. arXiv preprint arXiv:2306.13924 , 2023.

Leonardo Cotta, Gal Yehuda, Assaf Schuster, and Chris J Maddison. Probabilistic invariant learning with randomized linear classifiers. arXiv preprint arXiv:2308.04412 , 2023.

Stéphane Mallat. Group invariant scattering. Communications on Pure and Applied Mathematics , 65(10):1331-1398, 2012. doi: https://doi.org/10.1002/cpa.21413.

Fernando Gama, Alejandro Ribeiro, and Joan Bruna. Diffusion scattering transforms on graphs. arXiv preprint arXiv:1806.08829 , 2018.

Michael Perlmutter, Feng Gao, Guy Wolf, and Matthew Hirn. Understanding graph neural networks with asymmetric geometric scattering transforms. arXiv preprint arXiv:1911.06253 , 2019.

Dongmian Zou and Gilad Lerman. Graph convolutional neural networks via scattering. Applied and Computational Harmonic Analysis , 49(3):1046-1074, 2020.

Alexander Bogatskiy, Sanmay Ganguly, Thomas Kipf, Risi Kondor, David W Miller, Daniel Murnane, Jan T Offermann, Mariel Pettee, Phiala Shanahan, Chase Shimmin, et al. Symmetry group equivariant architectures for physics. arXiv preprint arXiv:2203.06153 , 2022.

William Fulton and Joe Harris. Representation theory: a first course , volume 129. Springer Science & Business Media, 2013.

Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parametersharing. In International conference on machine learning , pages 2892-2901. PMLR, 2017.

László Lovász and Balázs Szegedy. Limits of dense graph sequences. Journal of Combinatorial Theory, Series B , 96(6):933-957, 2006.

Christian Borgs, Jennifer T Chayes, László Lovász, Vera T Sós, and Katalin Vesztergombi. Convergent sequences of dense graphs i: Subgraph frequencies, metric properties and testing. Advances in Mathematics , 219(6):1801-1851, 2008.

Mark Reeder. Notes on representations of finite groups, 2014.

László Lovász and Balázs Szegedy. Szemerédi's lemma for the analyst. GAFA Geometric And Functional Analysis , 17(1):252-270, 2007.

Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 1998.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv:1707.01926 , 2017.

Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence , 36(7):1325-1339, 2013.

Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas. Semantic graph convolutional networks for 3d human pose regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3425-3435, 2019.

Ron Levie. A graphon-signal analysis of graph neural networks, 2023.

Alejandro Parada-Mayorga, Zhiyang Wang, and Alejandro Ribeiro. Graphon pooling for reducing dimensionality of signals and convolutional operators on graphs, 2022.

Daniel A. Spielman. Testing isomorphism of graphs with distinct eigenvalues, 2018.

Bruce Sagan. The symmetric group: representations, combinatorial algorithms, and symmetric functions , volume 203. Springer Science & Business Media, 2001.

Jean-Pierre Serre et al. Linear representations of finite groups , volume 42. Springer, 1977.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949-986, 2022.

Ningyuan Huang, David W. Hogg, and Soledad Villar. Dimensionality reduction, regularization, and generalization in overparameterized regressions. SIAM Journal on Mathematics of Data Science , 4(1):126-152, feb 2022. doi: 10.1137/20m1387821. URL https://doi.org/10. 1137%2F20m1387821 .

Somesh Das Gupta. Some aspects of discrimination function coefficients. Sankhy¯ a: The Indian Journal of Statistics, Series A , pages 387-400, 1968.

Hosagrahar V Jagadish, Johannes Gehrke, Alexandros Labrinidis, Yannis Papakonstantinou, Jignesh M Patel, Raghu Ramakrishnan, and Cyrus Shahabi. Big data and its technical challenges. Communications of the ACM , 57(7):86-94, 2014.

Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7753-7762, 2019.

A Parameterization of equivariant linear Maps

We consider a group G acting on spaces X and Y via representations ϕ and ψ , respectively. Our goal is to find the equivariant linear maps f : X → Y such that f ( ϕ ( g ) x ) = ψ ( g ) f ( x ) for all g ∈ G and x ∈ X . The standard way to do this, used extensively in the equivariant machine learning literature (e.g. [40, 43]), is to decompose ϕ and ψ in irreducibles and use Schur's lemma.

In a nutshell, a group representation φ is an homomorphism G → GL( V ) , where GL( V ) denotes the General Linear group of the vector space V (sometimes mathematicians say that V is a representation of G , but we need to know the homomorphism φ too). One way to interpret the group homomorphism (i.e. φ ( gh ) = φ ( g ) ◦ φ ( h ) ) is that the group multiplication corresponds to the composition of linear invertible maps (i.e. matrix multiplication). A linear subspace W of V is said to be a subrepresentation of φ if φ ( G )( W ) ⊂ W . An irreducible representation is one that only has itself and the trivial subspace as subrepresentations.

Schur's lemma states that if V , W are vector spaces over C and φ V , φ W are irreducible representations, then either (1) φ V and φ W are not isomorphic as representations (and the only equivariant linear map between V , W is the zero map), or (2) φ V and φ W are isomorphic and the only non-trivial equivariant maps are of the form λI where λ ∈ C and I is the identity (See Chapter 1 of [68]).

Given G acting on spaces X and Y via representations ϕ and ψ , respectively, one can decompose ϕ and ψ in irreducibles over C

(this notation assumes the same irreducibles appear in both decompositions, which can be done if we allow some of the a k and b k to be zero). Then one can parameterize the equivariant linear maps by having one complex parameter per irreducible that appears in both decompositions.

These ideas can be applied to real spaces too. By Maschke's theorem we can decompose the representation in irreducibles over R . Then we can check further how to decompose these irreducibles over C , and apply Schur's lemma. We have 3 cases for the decomposition:

The irreducible over R is also irreducible over C . In this case we directly apply Schur's lemma.

The irreducible over R decomposes in two different irreducibles over C . In this case we can send each C -irreducible to their isomorphic counterpart.

The irreducibles over R decompose in two copies of the same irreducible over C . In this case we can send each irreducible to any isomorphic copy independently.

Therefore, finding the equivariant linear maps reduces to decomposing the corresponding representations in irreducibles. In the next sections we explain in detail how to do this for the specific problems described in this paper. The appendix is organized as follows: We first show how to parameterize equivariant linear layers for abelian group using isotypical decomposition (Section A.1), and then discuss the case for the symmetry group induced by graph coarsening, which further considers parameterizing permutation-equivariant linear maps to model the within-cluster symmetries (Section A.2).

A.1 Equivariant Linear Maps via Isotypical Decomposition

In this section, we assume that the graph adjacency matrix A has distinct eigenvalues λ 1 > λ 2 > . . . > λ n . Then A G is an abelian group [81, Lemma 3.8.1]. Under this assumption, we present the construction of A G -equivariant linear maps using isotypical decomposition (i.e. decomposition into isomorphism classes of irreducible representations) and group characters. We remark that such construction extends to non-abelian groups and refer the interested reader to [82], but we omit it here for the ease of exposition.

We consider the simplest setting where f : R N → R N is a linear function that maps graph signals. Let x ∈ R N be the node features, then A G -equivariance requires

To construct equivariant linear functions f , our roadmap is outlined as follows:

Decompose the vector space X = R N into a sum of components such that different components cannot be mapped to each other equivariantly (also known as the isotypic decomposition);

Given X = ⊕ i X i an isotypic representation, we then parameterize f by linear maps at each X i such that for all i, f ( X i ) ⊆ X i .

To this end, we need the following definitions.

Definition 5. ( G -module, [82, Defn 1.3.1]) Let X be a vector space and G be a group. We say the vector space X is a G -module or X carries a representation of G if there is a group homomorphism ρ : G → GL ( X ) , where GL denotes the General Linear group. Equivalently, if the following holds:

gv ∈ X ,

g ( cv + dw ) = c ( gv ) + d ( gw ) ,

( gh ) v = g ( hv ) ,

ev = v

for all g, h ∈ G ; v, w ∈ X and scalars c, d ∈ C ( e ∈ G denotes the identity element).

In what follows, we consider X = R N carries a representation of G .

Definition 6. (Group characters) Given a group G and a representation ϕ : G → GL ( V ) , the group character is a function χ : G → R (or C ) defined as χ ( g ) := tr ϕ ( g ) .

Definition 7. (Group orbits) Let X be a vector space and G be a group. The group orbit of an element x ∈ X is O ( x ) := { gx : g ∈ G} .

Let g 1 , . . . , g s be the generators of A G ⊂ ( S 2 ) n , or simply A G ≡ ( S 2 ) k for some k ≤ n . Since A G is abelian, any irreducible representation is 1 -dimensional [68, p.8]. In other words, the irreducible representations of an abelian group are homomorphisms

Since all the elements of the group A G = ( S 2 ) k is of order 1 or 2 , the homomorphisms are ρ : A G →{± 1 } ⊂ R . By Definition 6, the irreducible characters (i.e., characters of irreducible matrix representation) are also homomorphisms ρ : A G →{± 1 } . In other words, χ ( g ) ∈ {± 1 } for all g ∈ A G . Then we can write down the 2 k × 2 k character table, where the rows are the characters χ , and the columns are the group elements g ∈ A G (see Table 1 as an example). Now, define the projection onto the isotypic component of the representation X as

where the second equality uses the fact that A G is abelian.

Intuitively, applying P χ on X = span( { e 1 , . . . , e N } ) picks out all v ∈ X that stays in the same subspace defined by the group character χ . (Note that for the ( S 2 ) k case χ -1 ( g ) = χ ( g ) since χ ( g ) ∈ {± 1 } ).

Algorithm 1 presents the construction of equivariant linear map f with respect to an abelian group. Such construction can parameterize all abelian A G -equivariant linear maps, as shown in Lemma 5.

Lemma 5. f is linear, equivariant with respect to the abelian group A G if and only if f can be written as (13) in Algorithm 1.

Proof. By construction in Algorithm 1, f is linear and equivariant. To show the converse, we first recall some useful facts and notations. Since A G is abelian with all irreducible representations being one-dimensional, for isotypic components X χ 1 ≇ X χ 2 , we have

̸

where there exists some g ∈ G such that λ 1 ( g ) = λ 2 ( g ) . To show f being linear and equivariant implies for all v ∈ X χ , f ( v ) ∈ X χ , we prove by contradiction. Without loss of generality, suppose

Algorithm 1 Parameterizing equivariant linear functions f : R N → R N for abelian group

Require: Abelian group A G = ( S 2 ) k

Construct the character table of χ irreps for A G , i.e. χ i : A G →{± 1 } i = 1 , . . . ℓ ;

For each character χ i in the character table, compute the projection matrix

followed by computing the basis from P χ i ( X ) and call it X χ i = [ b (1) χ i , . . . , b ( K i ) χ i ] .

X = ⊕ ℓ i =1 X χ i where X χ i are the isotypic component. Then f is any linear function satisfying that f ( X χ i ) ⊆ X χ i for all i = 1 , . . . , ℓ . In particular, in the basis [ b ( s ) χ i ] 1 ≤ i ≤ ℓ, 1 ≤ s ≤ K i f can be written as a block diagonal matrix R n × n with each block M χ i being the linear map from X χ i →X χ i ,

for some scalars α 1 , α 2 and v χ 1 ∈ X χ 1 , v χ 2 ∈ X χ 2 . Then by (14), for all g ∈ G ,

Now, since f is equivariant, for all g ∈ G ,

̸

But there exists some g ′ ∈ G such that λ 1 ( g ′ ) = λ 2 ( g ′ ) , which leads to f ( g ′ v χ 1 ) = f ( g ′ v χ 1 ) , a contradiction. One can easily extend the proof strategy to the general case for f ( v χ 1 ) = ∑ l i =1 v χ i .

̸

Example A.1. Consider the path graph on 4 nodes (i.e., P 4 ). We have aut( P 4 ) = { e, (14)(23) } ∼ = Z 2 .

Steps 1 : Note that Z 2 is abelian and thus all irreducible characters χ ( g ) ∈ {± 1 } , for all g ∈ Z 2 . The character table is shown in Table 1.

Step 2 : using (11) we have

Step 3 : Parameterize f : R 4 → R 4 by f : B ( P χ e ) → B ( P χ e ) and f : B ( P χ 2 ) → B ( P χ 2 ) . For all v ∈ R 4 , write v = c 1 ( e 1 + e 4 ) + c 2 ( e 2 + e 3 ) + c 3 ( e 1 -e 4 ) + c 4 ( e 2 -e 3 ) , then

where α 1 , . . . , α 8 are (learnable) real scalars. Now f is linear, equivariant by construction.

return f

A.2 Equivariant Linear Map for Symmetries Induced by Graph Coarsening

In this section, we provide additional details of constructing equivariant linear maps for using the symmetry group induced by graph coarsening (Definition 3). Recall the symmetry group with M clusters of G (with the associated coarsened graph G ′ ) is given by

We first assume that A G ′ is trivial and show how to parameterize equivariant functions with respect to products of permutations. Then we discuss more general cases, for instance if A G ′ is abelian. For the ease of exposition, we consider X ∈ R N , Y ∈ R N .

Suppose A G ′ is trivial, we claim that a linear function f : R N → R N is equivariant with respect to G G → G ′ if and only if it admits the following block-matrix form:

̸

where f kl are block matrices, I k is a sizek identity matrix, 1 k is a lengthk vector of all ones, and a k , b k , e kl are scalars. The subscript c k denotes the size of the k -th cluster. Figure 5 illustrates the block structure of f .

We provide a proof sketch of our claim. Since A G ′ is trivial by assumption, it remains to show any function that is equivariant to the product of permutations ( S 1 ×S 2 . . . ×S M ) admits the form in eqn. 20. We justify as follows:

(within-cluster) f kk is a linear permutation-equivariant function if and only if its diagonal elements are the same and its off-diagonal elements are the same ([34, Lemma 3.]);

̸

(across-cluster) f kl for k = l is a constant matrix since nodes within a cluster are indistinguishable; moreover, e kl are unconstrained since the coarsened symmetry A G ′ is trivial.

f11

f12

f1M

f21

f22

f2M

fM1

fM2

fMM

As an example, we illustrate the equivariant linear layer for two-cluster graph coarsening. Without loss of generality, assume that the node signals X are ordered according to the cluster assignment (e.g., X [1: | V 1 | ] are node features for the first cluster, etc). Let X (1) , X (2) denote the node features for the first and second cluster, W s (1) , W s (2) denote the weights on the block diagonal for self-feature transformation, W n (1) , W n (2) denote the weights on the block diagonal for within-cluster neighbors, and W n (12) , W n (21) denote the weights off the block diagonal for across-cluster neighbors. Let I denote the identity matrix, 1 (1) , 1 (2) denote the all-ones matrices with the same size as the corresponding

cluster, and 1 (12) , 1 (21) denote the all-ones matrices mapping across clusters. Recall ⊙ denotes the element-wise multiplication of two matrices. Then the equivariant linear layer is parameterized as

For cases where A G ′ is nontrivial, if A G ′ is abelian, we can use a construction by Serre ([83] Section 8.2). Observe that the symmetry of A G ′ effectively constrains the patterns of e kl in equation 20. We provide an example where A G ′ = S 2 in our image inpainting experiment (see Section 5.1), and defer the investigation of general cases for future work.

B Proofs of Our Theoretical Results

B.1 Proofs of Generalization with Exact Symmetries

Lemma 1 (Risk Gap) . Let X = R N × d , Y = R N × k be the input and output graph signal spaces on a fixed graph G . Let X ∼ µ where µ is a S N -invariant distribution on X . Let Y = f ∗ ( X ) + ξ , where ξ ∈ R N × k is random, independent of X with zero mean and finite variance and f ∗ : X → Y is A G -equivariant. Then, for any f ∈ V and for any compact group G ⊆ S N , we can decompose f as

where ¯ f G = Q G f, f ⊥ G = f -¯ f G . Moreover, the risk gap satisfies

Lemma 1 is a straightforward extension of Lemma 6 in [8], which makes use of Lemma 1 in [8].

Lemma 1 in [8]. Let U be any subspace of V that is closed under Q . Define the subspaces S and A of, respectively, the G -symmetric and G -anti-symmetric functions in U : S = { f ∈ U : f is G -equivariant } and A = { f ∈ U : Q f = 0 } . Then U admits admits an orthogonal decomposition into symmetric and anti-symmetric parts

Proof of Lemma 1. The first part of Lemma 1 f = ¯ f G + f ⊥ G follows from Lemma 1 in [8]. For the second part, by the assumption that the noise ξ is independent of X with zero mean and finite variance, we can simplify the risk gap as

Substituting f = ¯ f G + f ⊥ G yields

We remark that Lemma 6 in [8] assumes that f ∗ is G -equivariant, so the first term in (23) vanishes. We are motivated from the symmetry model selection problem, and thereby relax the assumption of the chosen symmetry group G can differ from the target symmetry group A G .

Theorem 2 (Bias-Variance-Tradeoff) . Let X = R N × d , Y = R N × k be the graph signals spaces on a fixed graph G . Let S N act on X and Y by permuting the rows with representations ϕ and ψ . Let G be a subgroup of S N acting with restricted representations ϕ | G on X and ψ | G on Y .

Let X [ i,j ] i.i.d. ∼ N ( 0 , σ 2 X ) and Y = f ∗ ( X ) + ξ where f ∗ ( x ) = Θ ⊤ x is A G -equivariant and Θ ∈ R Nd × Nk . Assume ξ [ i,j ] is random, independent of X , with mean 0 and E [ ξξ ⊤ ] = σ 2 ξ I < ∞ . Let ˆ Θ be the least-squares estimate of Θ from n i.i.d. examples { ( X i , Y i ) : i = 1 , . . . , n } , Ψ G ( ˆ Θ) be its equivariant version with respect to G . Let ( χ ψ | G | χ ϕ | G ) = ∫ G χ ψ | G ( g ) χ ϕ | G ( g )d λ ( g ) denote the inner product of the characters. If n > Nd +1 the risk gap is

Theorem 2 presents the risk gap in expectation, which follows from Lemma 1, by taking f as the least-squares estimator and using assumptions in the linear regression setting. To this end, we denote X ∈ R n × Nd , Y ∈ R n × Nk , ξ ∈ R n × Nk as the training data arranged in matrix form, where Y = f ∗ ( X ) + ξ . Recall that the least-squares estimator of Θ in the classic regime ( n > Nd ) is given by

while its equivariant map is

Our proof makes use of the following results in [8], which we restate adapted versions here for our setting.

Proposition 11 in [8]. Let V = { f W : f W ( x ) = W ⊤ x, W ∈ R d × k , x ∈ R d } denote the space of linear functions. Let X ∼ µ with E [ XX ⊤ ] = Σ . For any linear functions f W 1 , f W 2 ∈ V , the inner product on V satisfies

Theorem 13 in [8] (Simplified, Adapted) . Consider the same setting as Theorem 2. For n > Nd +1 ,

Proof of Theorem 2. We first plug in the least-squares expressions ˆ Θ , Ψ G ( ˆ Θ) to Lemma 1 and treat the mismatch term and constraint term separately; We complete the proof by collecting common terms together.

For the mismatch term, our goal is to compute

where the expectation is taken over the test point X and the training data X , ξ .

To that end, we write

Taking expectation yields

Note that ξ is independent with X and mean 0, so the second term in (29) vanishes. Similarly, the part E X, X , ξ ∫ G ψ ( g -1 ) ( ξ ⊤ X ( X ⊤ X ) -1 ) ϕ ( g ) x d λ ( g ) also vanishes (by first taking conditional

expectation of ξ conditioned on X ). Thus, we arrive at

where the last equality follows from Proposition 11 in [8] with the assumption that Σ = σ 2 X . This finishes the computation for the mismatch term.

Now for the constraint term, we have

where the last equality follows from linearity of expectation, E [ ξ ] = 0 and ξ independent of x

.

Combining the mismatch term in (30) with the constraint term in (33), the risk gap becomes

Applying Theorem 13 in [8], the second term in (34) reduces to

from which the theorem follows immediately.

Finally, we state a well-known result for the risk of (Ordinary) Least-Squares Estimator (see [84, 85] and references therein).

Lemma 6 (Risk of Least-Squares Estimator) . Consider the same set-up as Theorem 2. For n > Nd +1 ,

Proof of Lemma 6. Recall X,Y denote the test sample. We denote the risk of the least-squares estimator conditional on the training data X ∈ R n × Nd as R ( ˆ Θ | X ) , which has the following bias-variance decomposition:

where the last equality follows from ξ being zero mean and independent with X . The second term σ 2 ξ is also known as irreducible error . We decompose the first term into

Recall that ˆ Θ a.e. = ( X ⊤ X ) -1 X ⊤ Y = ( X ⊤ X ) -1 X ⊤ ( X Θ+ ξ ) = Θ + ( X ⊤ X ) -1 X ⊤ ξ . Thus E [ ˆ Θ] = Θ and (39) simplifies to E [ ∥ ( E [ ˆ Θ] -ˆ Θ) ⊤ X ∥ 2 F | X ] .

We finish computing the risk by taking expectation over X , and using E [ ˆ Θ] -ˆ Θ = ( X ⊤ X ) -1 X ⊤ ξ ,

By [86, Lemma 2.3], for n > Nd +1 , E [( X ⊤ X ) -1 ] = Nd n -Nd -1 I . Putting this in (43) completes the proof.

B.2 Proofs of Generalization with Approximate Symmetries

In Definition 3, we construct the symmetry group of G induced by the coarsening G ′ via a semidirect product,

We explain the construction in more details here. We first recall the definition of semidirect product. Given two groups G 1 , G 2 and a group homomorphism φ : G 2 → aut( G 1 ) , we can construct a new group G 1 ⋊ φ G 2 , called the semidirect product of G 1 , G 2 with respect to φ as follows:

The underlying set is the Cartesian product G 1 ×G 2 ;

The group operation ◦ is determined by the homomorphism φ , such that

Take G 1 = ( S c 1 ×S c 2 . . . ×S c M ) , G 2 = A G ′ . Note that G 1 is a normal subgroup in G G → G ′ ; Namely, for all s ∈ G G → G ′ , g 1 ∈ G 1 , we have sg 1 s -1 ∈ G 1 . Thus, the map g 1 ↦→ sg 1 s -1 is an automorphism of G 1 . In particular, the homomorphism φ g 2 ( g 1 ) = g 2 g 1 g -1 2 for g 2 ∈ G 2 describes the action of G 2 on G 1 by conjugation 2 . In the context of a graph G with N nodes and its coarsening G ′ with M clusters, the homomorphism φ describes how across-cluster permutations in G ′ act on the original graph G . Note that in the special case where A G ′ acts transitively on the coarsened nodes, we recover the wreath product - a special kind of semidirect product. Our construction of G G → G ′ can be seen as a natural generalization of the wreath product.

Corollary 3 (Risk Gap via Graph Coarsening) . Let X = R N × d , Y = R N × k be the input and output graph signal spaces on a fixed graph G . Let X ∼ µ where µ is a S N -invariant distribution on X . Let Y = f ∗ ( X ) + ξ , where ξ ∈ R N × k is random, independent of X with zero mean and finite variance, and f ∗ : R N × d → R N × k be an approximately equivariant mapping with equivariance rate κ . Then, for any G ′ that coarsens G up to error ϵ , for any f ∈ V , we have

Proof of Corollary 3. We start by simplifying the mismatch term in Lemma 1,

Putting this together with the constraint term completes the proof.

Corollary 4 (Bias-Variance-Tradeoff via Graph Coarsening) . Consider the same linear regression setting in Theorem 2, except now f ∗ is an approximately equivariant mapping with equivariance rate κ , and G = G G → G ′ is controlled by G ′ that coarsens G up to error ϵ . Denote the canonical permutation representations of G G → G ′ on X , Y as ϕ ′ , ψ ′ , respectively. Let ( χ ψ ′ | χ ϕ ′ ) = ∫ G G → G ′ χ ψ ′ ( g ) χ ϕ ′ ( g )d λ ( g ) denote the inner product of the characters. If n > Nd + 1 the risk gap is bounded by

Proof of Corollary 4. It follows immediately from applying Theorem 13 in [8] to Corollary 3 with G = G G → G ′ .

C Examples

C.1 Example: Gaussian data model with approximate symmetries

Example 3.1 considers G = S 3 , G = S 2 , X = R 3 , Y = R 3 , and x ∼ N (0 , σ 2 X I d ) . The target function is linear, i.e., f ∗ ( x ) = Θ ⊤ x for some Θ ∈ R 3 × 3 . In other words, we are learning linear functions on a fixed graph domain with 3 nodes. Suppose the target function is S 2 -equivariant such that it has the form

Now, we project Θ in (44) to S 3 -equivariant space using the intertwined average 6 with the canonical permutation representation of S 3 . A direct calculation yields

Therefore, the bias term evaluates to

For the variance term, recall χ ψ S 3 , χ ϕ S 3 are both the canonical permutation representations of S 3 , we have

(a) 1

(b) S2

(c) S2/S3

(d) S3

[[ 0.46 0.10 0.26]

[[0.38 0.09 0.21]

[[0.57 0.13 0.12]

[[0.55 0.12 0.12]

[0.59 0.49 -0.26]

[0.09 0.38 0.21]

[0.13 0.57 0.12]

[0.12 0.55 0.12]

[0.25 -0.04-0.03]]

[0.48 0.48 0.40]]

[0.12 0.12 0.52]]

[0.12 0.12 0.55]]

Error

S2

S2

S2

S2

Generalization

S3

S3

S3

S3

20

30

40

20

30

40

20

30

40

20

30

40

n

n

n

n

Therefore, the variance term evaluates to

Putting (47) and (49) together yields the generalization gap of for the least square estimator f ˆ Θ compared to its S 3 -equivariant version f Ψ S 3 ( ˆ Θ) .

As a comparison, when choosing the symmetry group of the target function G = S 2 , the bias vanishes and note that ( χ ψ S 2 | χ ϕ S 2 ) = 1 2 (3 2 +1 2 ) = 5 , so generalization gap is

We see that choosing G = S 3 is better if a ≈ e, b ≈ c ≈ d (i.e., f ∗ is approximately S 3 -invariant) and the training sample size n small, whereas S 2 is better vice versa. This analysis illustrates the advantage of choosing a (suitably) larger symmetry group to induce a smaller hypothesis class when learning with limited data, and introduce useful inductive bias when the target function is approximately symmetric with respect to a larger group. We further illustrate our theoretical analysis via simulations, with details and results shown in Figure 6.

C.2 Example: Approximately Equivariant Mapping on a Geometric Graph

In this section, we illustrate a construction of an approximately equivariant mapping. We focus on a version of Definition 3 that does not take to account the symmetries of G ′ . Namely, we consider a definition of the approximate symmetries as

Equivalently, we restrict the analysis to coarsening graphs G ′ that are asymmetric.

Background from graphon-signal analysis. To support our construction, we cite some definitions and results from [79].

Definition 8. Let r > 0 . The graphon-signal space with signals bounded by r is WL r := W × L ∞ r [0 , 1] , where L ∞ r [0 , 1] is the ball of radius r in L ∞ [0 , 1] . The distance in WL r is defined for ( W,s ) , ( V, g ) ∈ WL r by

Moreover,

where g ϕ ( x ) = g ( ϕ ( x )) and ϕ is a measure preserving bijection.

Any graph-signal induces a graphon signal in the natural way, as in Definition 1. The cut norm and distance between two graph-signals is defined to be the cut norm and distance between the two induced graphon-siganl respectively. Similarly, the L 1 distance between a signal q on a graph and a signal s on [0 , 1] is defined to be the L 1 distance between the induced signal from q and s . The supremum in the definition of cut distance between two induced graphon-signals is realized by some measure preserving bijection.

Sampling graphon-signals. The following construction is from [79, Section 3.4]. Let Λ = ( λ 1 , . . . λ N ) ∈ [0 , 1] N be N independent uniform random samples from [0 , 1] , and ( W,s ) ∈ WL r . We define the random weighted graph W (Λ) as the weighted graph with N nodes and edge weight w i,j = W ( λ i , λ j ) between node i and node j . We similarly define the random sampled signal s (Λ) with value s i = s ( λ i ) at each node i . Note that W (Λ) and s (Λ) share the sample points Λ . We then define a random simple graph as follows. We treat each w i,j = W ( λ i , λ j ) as the parameter of a Bernoulli variable e i,j , where P ( e i,j = 1) = w i,j and P ( e i,j = 0) = 1 -w i,j . We define the random simple graph G ( W, Λ) as the simple graph with an edge between each node i and node j if and only if e i,j = 1 . The following theorem is [79, Theorem 3.6]

Theorem 3.6 from [79] (Sampling lemma for graphon-signals) . Let r > 1 . There exists a constant N 0 > 0 that depends on r , such that for every N ≥ N 0 , every ( W,s ) ∈ WL r , and for Λ = ( λ 1 , . . . λ N ) ∈ [0 , 1] N independent uniform random samples from [0 , 1] , we have

By Markov's inequality and (51), for any 0 < p < 1 , there is an event of probability 1 -p (regarding the choice of Λ ) in which

Stability to deformations of mappings on geometric graphs. Let M be a metric space with an atomless standard probability measure defined over the Borel sets (up to completion of the measure). Such a probability space is equivalent to the standard probabiltiy space [0 , 1] with Lebesgue measure. Namely, there are co-null sets A ⊂ M and B ⊂ [0 , 1] , and a measure preserving bijection ϕ : A → B . Hence, graphon analysis applied as-is when replacing the domain [0 , 1] with M . Suppose that we are interested in a target function f M : L 1 ( M ) → L 1 ( M ) that is stable to deformations in the following sense.

Definition 9. Let ϵ > 0 . A measurable bijection ν : M→M is called a deformation up to ϵ , if there exists an event B ϵ ⊂ M with probability greater than 1 -ϵ such that for every x ∈ B ϵ

The mapping f M : L 1 ( M ) → L 1 ( M ) is called stable to deformations with stability constant C , if for any deformation ν up to ϵ , and every s ∈ L 1 ( M ) , we have

Suppose that we observe a discretized version of the domain M , defined as follows. There is a graphon W : M 2 → [0 , 1] defined as

where r : R + → [0 , 1] is a decreasing function with support [0 , ρ ] . Instead of observing W , we observe a graph G = G ( W, Λ) with node set [ N ] , sampled from W on the random independent points Λ = { λ n } N n =1 ⊂ M as above. Suppose moreover that any graph signal is sampled from a signal in L 1 ( M ) , on the same random points Λ , as above. Suppose that the target f M on the continuous

domain is well approximated by some mapping f ∗ : L 1 [ N ] → L 1 [ N ] on the discrete domain in the following sense. For every s ∈ L 1 ( M ) , let s G be the graph signal sampled on the random samples { λ n } n . Then there is an event of high probability such that

for some small e . We hence consider f ∗ as the target mapping of the learning problem. One example of such a scenario is when there exists some Lipschitz continuous mapping Θ : WL r →WL r with Lipschitz constant L , such that f M = Θ( W, · ) and f ∗ = Θ( G, · ) . Indeed, by (52), for some p as small as we like, there is an event of probability 1 -p in which, up to a measure preserving bijection,

A concrete example is when Θ is a message passing neural network (MPNN) with Lipschitz continuous message and update functions, and normalized sum aggregation [79, Theorem 4.1].

Let G ′ be a graph that coarsens G up to error ϵ . In the same event as above, by (52), up to a measure preserving bijection,

We next show an approximation property that we state here informally: Since W ( x, y ) ≈ 0 for x away from y , we must have W G ′ ( x, y ) ≈ 0 as well for a set of high measure. Otherwise, δ □ ( W G ′ , W ) cannot be small. By this, any approximate symmetry of G is a small deformation, and, hence, f ∗ is an approximately equivariant mapping.

Equivariant mappings on geometric graphs. In the following, we construct a scenario in which f ∗ can be shown to be approximately equivariant in a restricted sense. For simplicity, we assume f ∗ ( s G ) ∈ L 2 [0 , 1] , and restrict to the case r = ✶ [0 ,ρ ] in the geometric graphon W of (53). Denote the induced graphon W G ′ = T . Given h > 0 , define the h -diagonal d h = { ( x, y ) ∈ M 2 | d M ( x, y ) ≤ h } .

In the following, all distances are assumed to be up to the best measure preserving bijection.

If there is a domain S ′ × T ′ ∈ M 2 outside the ρ -diagonal in which T ( x, y ) > c for some c > 0 , by reverse triangle inequality, we must have

Hence, since by (55), ∥ W -T ∥ □ < u , for every S ′ × T ′ that does not intersect d ρ , we must have

In other words, for any two sets S, T with distance more than ρ ( inf s ∈ S,t ∈ T d µ ( s, t ) > ρ ), we have

This formalizes the statement ' W G ′ ( x, y ) ≈ 0 for x away from y ' from above.

Next, we develop the analysis for the special case M = [0 , 1] with the standard metric and Lebesgue probability measure. We note that the analysis can be extended to M = [0 , 1] D for a general dimension D ∈ N . For every z ∈ [0 , 1] , we have

and

Let ν > 0 . We take a grid { x j } ∈ [0 , 1] of spacing √ 2 ν . The sets

cover d c ν (where d c ν is the complement of d ν ). Hence,

We take 2 √ 2 ν u = t , for u ≪ t ≪ 1 , namely, ν = √ 2 u t . For example, we may take t = √ 2 u 1 / 3 , and ν = u 2 / 3 , assuming that ρ < u 1 / 3 . Hence, we have

To conclude, the probability of having an edge between nodes λ i and λ j in G ′ N which are further away than u 2 / 3 , namely, d M ( λ i , λ j ) > u 2 / 3 , is less than √ 2 u 1 / 3 .

Suppose that G ′ is asymmetric. This means that symmetries of G G → G ′ can only permute between nodes that have an edge between them in the blown-up graph G ′ N . The probability of having an edge between nodes further away than u 2 / 3 is less than √ 2 u 1 / 3 . Hence, a symmetry in G G → G ′ can be seen as a small deformation, where for each node λ i and a random uniform g ∈ G G → G ′ , the probability that λ i it is mapped by g to a node of distance less than u 2 / 3 is more than 1 -√ 2 u 1 / 3 .

Any symmetry g in G G → G ′ induces a measure preserving bijection ν in M = [0 , 1] , by permuting the intervals of the partition P N of Definition 1. As a result, the set of points that are mapped further away than u 2 / 3 under ν has probability upper bounded by √ 2 u 1 / 3 , and symmetries in G G → G ′ can be seen as a small deformation ν according to Definition 9 (in high probability). This means that, for any g ∈ G G → G ′ , √

so by the triangle inequality, combining with equation 54, we have

Equation (56) leads to

Since for any q ∈ L 2 [0 , 1] ∩ L ∞ [0 , 1] we have ∥ q ∥ 2 2 ≤ ∥ q ∥ ∞ ∥ q ∥ 1 , we can bound

where the last inequality follows from ∥Q G G → G ′ ( f ∗ )( s G ) ∥ ∞ ≤ ∥ f ∗ ( s G ) ∥ ∞ .

Denote ∥ f ∗ ∥ ∞ := ∫ ∥ f ∗ ( s G ) ∥ ∞ dµ ( s G ) , and suppose that ∥ f ∗ ∥ ∞ is finite. Hence, if µ is a probability measure, we have

This shows an example of approximately equivariant mapping based on random geometric graph, where the approximation rate is also a function of the size of the graph N , and goes to zero as N →∞ and ϵ → 0 .

In future work, we will extend this example to more general metric space M and to non-trivial symmetry groups A G ′ . Intuitively, most random geometric graphs are 'close to asymmetric'. This means that for 'most' G ′ , the symmetries of A G ′ can only permute between nodes connected by an edge, and so are the symmetries of G G → G ′ . For this, we need to extend Definition 9 by treating G ′ probabilistically.

D Experiment Details

In this section, we provide additional details of our experiments. We first give a brief introduction of standard graph neural networks (Section D.1), followed by in-depth explanations of our applications in image inpainting (Section D.2), traffic flow prediction (Section D.3), and human pose estimation (Section D.4). All experiments were conducted on a server with 256 GB RAM and 4 NVIDIA RTX A5000 GPU cards.

D.1 Graph Neural Networks (GNNs)

We consider standard message-passing graph neural networks (MPNNs) [19-21] defined as follows. A L -layer MPNN maps input X ∈ R N × d to output Y ∈ R N × k following an iterative scheme: At initialization, h (0) = X ; At each iteration l , the embedding for node i is updated to

where ϕ, ψ are the update and message functions, N ( i ) denotes the neighbors of node i , and A [ i,j ] represents the ( i, j ) -edge weight. MPNNs typically have two key design features: (1) ϕ, ψ are shared across all nodes in the graph, typically chosen to be a linear transformation or a multi-layer perceptions (MLPs), known as global weight sharing ; (2) the graph A is used for (spatial) convolution.

D.2 Application: Image Inpainting

We provide additional details of the data, model, and optimization procedure used in the experiment.

Data. We consider the datasets MNIST [74] and FashionMNIST [75]. For each dataset, we take 100 training samples and 1000 test samples via stratified random sampling. The input and output graph signals are ( m i ⊙ x i , x i ) ( ⊙ is entrywise multiplication). Here x i ∈ R 28 × 28 ≡ R 784 denotes the image signals and m i denotes a random mask (size 14 × 14 for MNIST and 20 × 20 for FashionMNIST). For experiments with reflection symmetry on the coarsened graph, we further transform each image in the FashionMNIST subset using horizontal flip with probability 0 . 5 (FashionMNIST+hflip). We remark that it is possible to model the dihedral group D 4 on the square as the coarsened graph symmetry, yet we only consider the reflection symmetry S 2 to simplify the parameterization (since it is Abelien while D 4 is not) while capturing the most relevant symmetries.

Model. We consider a 2 -layer G G → G ′ -equivariant networks G -Net, a composition of f out ◦ ReLU ◦ f in, where f in , f out denote the input/output equivariant linear layers. The input and output feature dimension is 1 (since the signals are grayscale images), and the hidden dimension is set as 28 . For comparison, we also consider a simple 1 -layer G G → G ′ -equivariant networks G -Net.

We train the models with ADAM (learning rate 0 . 01 , no weight decay, at most 1000 epochs). We report the best test accuracy at the model checkpoint selected by the best validation accuracy (with a 80 / 20 training-validation split).

We supplement Figure 2 with Table 7 for further numerical details.

D.3 Application: Traffic Flow Prediction

Data. The METR-LA traffic dataset, [76], contains traffic information collected from 207 sensors in the highway of Los Angeles County from Mar 1st 2012 to Jun 30th 2012 [87]. We use the same traffic data normalization and 70 / 10 / 20 train/validation/test data split as [76]. We consider two different traffic graphs constructed from the pairwise road network distance matrix: (1) the sensor graph G introduced in [76] based on applying a thresholded Gaussian kernel (degree distribution in Figure 8e); (2) the sparser graph G s based on applying the binary mask where the ( i, j ) entry is nonzero if and only if nodes i, j lie on the same highway (degree distribution in Figure 8d). We construct the second variant to more faithfully model the geometry of the highway, illustrated in Figure 8a.

Graph coarsening. We choose 2 clusters based on highway intersection and flow direction, indicated by colors (Figure 8b (b)), and 9 clusters based on highway labels (Figure 8c (c)).

(d) Our faithful graph degree dis-

tribution

(e) Original sensor graph degree

distribution

34.225

34.225

34.225

34.200

34.200

34.200

34.175

34.175

34.175

34.150

34.150

34.150

34.125

34.125

34.125

34.100

34.100

34.100

34.075

34.075

34.075

34.050

34.050

34.050

118.55

118.50

118.45

118.40

118.35

118.30

118.25

118.20

118.55

118.50

118.45

118.40

118.35

118.30

118.25

118.20

118.55

118.50

118.45

118.40

118.35

118.30

118.25

118.20

20.0

175

17.5

150

15.0

125

12.5

Count

100

uno

10.0

75

7.5

50

5.0

25

2.5

0

Degree

2

3

0.0

2 3 4 5 6 7 8 910111213141516171819202122

Degree

25

Model. We use a standard baseline, DCRNN proposed in [76]. DCRNN is built on a core recurrent module, DCGRU cell, which iterates as follows: Let x i,t , h i,t denote the i -th node feature and hidden state vector at time t ; Let X t , R t , H t -1 be the matrices of stacking feature vectors x i,t , r i,t , h i,t -1 as rows.

where W z , U z , b z , U r , W r , b r , W h , U h , b h are learnable weights and biases, σ g is the sigmoid function, ϕ g is the hyperbolic tangent, and h i, 0 = 0 for all i at initialization. The crucial different from a vanilla GRU lies in eqn (63) where graph convolution replaces matrix multiplication.

We then modify the graph convolution in (63) from global weight sharing to tying weights among clusters of nodes, similar to the implementation in Appendix D.4 for RelaxS 16 . For example, in the

case of two clusters (orbits), we change XW h to

where X c i denotes the submatrix of X including the rows of nodes from cluster i only, and W h,c 1 , W h,c 2 are two learnable matrices. In words, we perform cluster-specific linear transformation, combine the transformed features, and reorder the rows (i.e., swap ) to ensure compatibility with the graph convolution.

Experiment Set-up. For our experiments, we use DCRNN model with 1 RNN layer and 1 diffusion step. We choose T ′ = 3 (i.e., 3 historical graph signals) and T = 3 (i.e., predict the next 3 period graph signals). We train all variants for 30 epochs using ADAM optimizer with learning rate 0 . 01 . We report the test set performance selected by the best validation set performance.

D.3.1 Assumption Validation: Approximate Equivariant Map

Before applying our construction of approximate symmetries, we validate the assumption of the target function f ∗ being an approximately equivariant mapping using a trained DCRNN model as a proxy. We proceed as follows:

Data. We use the validation set of METR-LA (traffic graph signals in LA), which has 207 nodes and consists of 14 , 040 input and output signals. Each input X ∈ R 207 × 2 represents the traffic volume and speed in the past at the 207 stations, and output Y ∈ R 207 representing future traffic volume.

Model. We use a trained DCRNN model on our faithful graph, with input being 3 historical signals X = ( X T -3 , X T -2 , X T -1 ) ∈ R 3 × 207 × 2 to predict the future signals Y = ( X T , X T +1 , X T +2 ) ∈ R 3 × 207 . We denote this model as f . It gives reasonable performance with Mean Absolute Error ≈ 3 , and serves as a good proxy for the target (unknown) function f ∗ .

Neighbors. Wetake our faithful traffic graph that originally has 397 non-loop edges, and only consider a subset of 260 edges by thresholding the distance values to eliminate geometrically far-away nodes. This defines our 260 neighboring node pairs.

Equivariance error. For each node pair ( i, j ) , we swap their input signals by interchanging the ( i, j ) -th slices in the node dimension of the tensor X , denoted as X ( i,j ) , and check if the swapped output ˆ Y ( i,j ) = f ( X ( i,j ) ) is close to the original output ˆ Y = f ( X ) with ( i, j ) -th slices swapped. We measure 'closeness' via the relative equivariant error at the node pair. Concretely, let X [ i, j ] denote the tensor slices at the ( i, j ) node pair, and X [ j, i ] being the swapped version by interchanging ( i, j ) -th slices. The relative different is computed as

where / denotes element-wise division. We then compute the mean relative equivariance error over all instances in the validation set, which equals to 5.17%. This gives concrete justification to enforce approximate equivariance in the traffic flow prediction problems.

D.4 Application: Human Pose Estimation

D.4.1 Equivariant Layer for Human Skeleton Graph

We apply the constructions in Section A.1 to our human skeleton graph. We first show how to parameterize all linear A G -equivariant functions. Observe that A G ∼ = ( S 2 ) 2 = { e, a, l, al } , where the nontrivial actions correspond to the a rm flip with respect to the spine, the l eg flip with respect to the spine, and their composition. To fix ideas, we first treat both input and output graph signals as vectors, and construct A G -equivariant linear maps f : R 16 → R 16 .

Step 1 : Obtain the character table for ( S 2 ) 2

Step 2 : Construct the basis for isotypic decomposition. Here we choose to index the leg joint pairs as (1 , 4) , (2 , 5) , (3 , 6) , arm joint pairs as (10 , 13) , (11 , 14) , (12 , 15) , and spline joints 0 , 7 , 8 , 9 .

Step 3 : Parameterize f : R 16 → R 16 by f : B ( P χ e ) →B ( P χ e ) and f : B ( P χ 2 ) →B ( P χ 2 ) , i.e. for all v ∈ R 16 , let v = B ( P χ e ) c e + B ( P χ 2 ) c 2 + B ( P χ 3 ) c 3 , then

where W e ∈ R 10 × 10 , W 2 ∈ R 3 × 3 , W 3 ∈ R 3 × 3 are (learnable) weight matrices. Now f expresses all linear, equivariant maps w.r.t ( S 2 ) 2 .

The following calculation based on f : R 16 → R 16 shows how much degree of freedom (measured by learnable parameters) is gained by relaxing the symmetry from global (group S 16 ), exact A G ∼ = ( S 2 ) 2 , to trivial group (i.e., no symmetry).

To parameterize equivariant linear function f : R 16 × d → R 16 × d ′ , we proceed by decoupling the input space into R 10 × d , R 3 × d , R 3 × d and the output space into R 10 × d ′ , R 3 × d ′ , R 3 × d ′ . Now the learnable weight matrices for multidimensional input/output become W e ∈ R 10 d × 10 d ′ , W 2 ∈ R 3 d × 3 d ′ , W 3 ∈ R 3 d × 3 d ′ . The construction is summarized in Algorithm 2.

Algorithm 2 Equivariant layer f A G : R 16 × d → R 16 × d ′ for A G ∼ = ( S 2 ) 2

Require: The basis B ∈ R 16 × 16 in (66) for isotypic decomposition of A G = ( S 2 ) 2 , input h ( l ) ∈ R 16 × d .

Initialize: The learnable weights W ( l ) e ∈ R 10 d ′ × 10 d ; W ( l ) 2 , W ( l ) 3 ∈ R 3 d ′ × 3 d ; M ( l ) ∈ R 16 × 16 .

Project h ( l ) to the isotypic component: z ( l ) = B ⊤ h ( l ) ;

Perform block-wise linear transformation:

z e = W e flatten( z ( l ) [: , :10] )

z 2 = W 2 flatten( z ( l ) [: , 10:13] )

z 3 = W 3 flatten( z ( l ) [: , 13:] )

z ( l +1) = concat[ z e ; z 2 ; z 3 ] ∈ R 16 × d ′

Project back to the standard basis: ¯ h ( l +1) = Bz ( l +1) .

Perform pointwise nonlinearity: h ( l +1) = σ ( ¯ h ( l +1) ) . return h ( l +1)

3

13

2

15

12

15

12

15

12

15

coarsen

blow-up

more sym.

AG = S2 × S2

9g'→G = (S2 × S2) × AG

g = S2 × S2 × AG

D.4.2 Experiment Details

Data. We use the standard benchmark dataset, Human3.6M [77], with the same protocol as in [78]: We train the models on 1 . 56 Mposes (from human subjects S 1 , S 5 , S 6 , S 7 , S 8 ) and evaluate them on 0 . 54 Mposes (from human subjects S 9 , S 11 ). We use the method described in [88] to normalize the inputs (2D joint poses) to [ -1 , 1] and align the targets (3d joint poses) with the root joint.

Model. We give a detailed description of G -Net and its variants used in the experiments. Figure inset illustrates the architecture of G -Net.

For the human skeleton graph with N = 16 , we have f G : R 16 × d → R 16 × k , where d, k represent the input dimension and output dimension (for each layer). Let f G [ i, j ] : R 16 → R 16 denote its ( i, j ) -th slice.

1. G -Net with strict equivariance using equivariant linear map f G (see Table 3):

S 16 : f S 16 [ i, j ] ∈ R 16 × 16 is a diagonal matrix, with one learnable scalar a on diagonal and another learnable scalar b off diagonal.

̸

RelaxS 16 : We relax f S 16 [ i, j ] by having 16 different pairs of scalars ( a i , b i ) , i ∈ [16] , such that each node i can map to itself and communicate to its neighbors in a different way (controlled by ( a i , b i ) ), while still treat all neighbors equally (by using the same b i for nodes j = i ).

A G = S 2 2 : We use Algorithm 2.

Trivial: We allow f [ i, j ] ∈ R 16 × 16 to be arbitrary, i.e., it has 16 × 16 learnable scalars.

Linear Map

Nonlinearity

Linear Map

Nonlinearity

Linear Map

We remark that for S 16 and RelaxS 16 , we implement them by tying weights; for A G , we implement them by projecting to isotypic component as shown in Algorithm 2.

G -Net augmented with graph convolution Af G ( x ) , denoted as G -Net(gc) (see Table 3): We apply the equivariant linear map f G in 1. and obtain the output f G ( x ) ∈ R 16 × k ; We then apply graph convolution by multiplication from the left, i.e., Af G ( x ) ∈ R 16 × k .

̸

G -Net augmented with graph convolution and learnable edge weights, denoted as G -Net(gc+ew) (see Table 4): We further learn the edge weights for the adjacency matrix A , by softmax ( M ⊙ A ) where M ∈ R 16 represents the learnable edge weights, and M i,j is nonzero when A i,j = 0 and 0 elsewhere. This is inspired from SemGCN [78]. Besides the groups discussed above, we also implemented Relax-( S 6 ) 2 which corresponds to tying weights among the coarsened graph orbits, consists of 4 spline nodes (singleton orbits) and 2 orbits for the left/right arm and leg nodes.

4. G -Net augmented with graph locality constraints ( A ⊙ f G )( x ) and learnable edge weights, denoted as G -Net(pt+ew) (see Table 3): We perform pointwise multiplication A ⊙ f G [ i, j ] at each ( i, j ) -th slice of f G . In practice, we also allow learnable edge weights as done in 3.

Experimental Set-up. We design G -Net to have 4 layers (with batch normalization and residual connections in between the hidden layers), 128 hidden units, and use ReLU nonlinearity. This allows G -Net(gc+ew) to recover SemGCN [78] when choosing G = S 16 . We train our models for at most 30

epochs with early stopping. For comparison purpose, we use the same optimization routines as in SemGCN [78] and perform the hyper-parameter search of learning rates { 0 . 001 , 0 . 002 } .

Evaluation. Table 3 shows results of G -Net and its variants when varying the choice of G . We observe that using the automorphism group A G does not give the best performance, while imposing no symmetries (Trivial) or a relaxed version of S 16 yields better results. Here, enforcing no symmetry achieves better performance since the human skeleton graph is very small with 16 nodes only. As shown in other experiments with larger graphs (e.g. image inpainting), enforcing symmetries indeed yields better performance.

Additional Evaluation. Table 4 shows the experiments when we keep the number of parameters roughly the same across different choices of G .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work discusses a non-trivial case of equivariant graph networks. In this scenario, the input graph $G$ remains fixed, and therefore, the relevant permutations that act on the graph signals are the automorphisms of $G$. Under this assumption, the authors describe a bias-variance tradeoff with respect to the symmetry group to which the hypothesis class is restricted. Since the automorphism group tends to be trivial in large graphs, the authors introduce the concept of approximated symmetry by considering a symmetry group associated with a coarser version of the graph. The main result of the paper formalizes the bias-variance tradeoff in relation to the approximated symmetries.\n\nSTRENGTHS:\n* The paper introduces a novel perspective and analysis of equivariant graph networks, both in terms of the fixed graph scenario and the approximated equivariance via graph coarsening.  \n\n* The use of Graphons to define a metric over graph spaces is interesting and could have many applications.\n\n* The theoretical claims are very clear and explained in a detailed manner.\n\nWEAKNESSES:\n* An explanation regarding the clustering methods is missing in the main text. The method section in the paper does not suggest how to coarsen a graph G, therefore it would be helpful to explain how it is done, specifically in the traffic flow experiment.\n\n* Although the appendix provides some explanation, additional information about the architectures employed in the experimental section would be beneficial for a more comprehensive understanding of the empirical results.\n\n* The empirical results presented in Table 2 do not provide strong evidence supporting the necessity of approximated equivariance.\n\nQUESTIONS:\n* Definition 1 - What does the J in summation stand for? should it be N?\n\n*  The implementation of this method on an arbitrary graph, which lacks any inductive bias concerning the graph's structure, remains unclear. Are the authors able to offer any insights or intuition regarding a general approach or recipe for graph coarsening in such cases?\n\nLIMITATIONS:\nThere is no potential negative impact to this work", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

comments: SUMMARY:
This paper formalizes the notion of active symmetries and approximate symmetries of GNNs on a fixed graph domain. Furthermore, it theoretically characterizes the statistical risk of linear regression with symmetries and show a bias-variance tradeoff. For graph tasks, it utilize 
coarsed graph for approximate symmetries. Experimental results on human pose estimation and traffic flow prediction valid the proposed method.

STRENGTHS:
1. Novel theoretical results and methods. To my best knowledge, its the first risk gap with a combination of symmetries and graphon. The approximating symmetry by coarsened graph is also fancy.

WEAKNESSES:
1. The result in Section 3.1 seems trivial and useless.
2. The experiment section is hard to read as all implementations are put in Appendix. 
3. Graph coarsening is vital for the proposed method. But coarsening method for general graph task in unknown.
4. The experiments is conducted on small and specific tasks only rather than general graph task, like link prediction [1].

[1] https://ogb.stanford.edu/docs/linkprop/

QUESTIONS:
1. As shown in the Weaknesses section, please add a more concrete description of your method in the maintext. 
2. Please provide experimental results on more general graph tasks, like link prediction.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors discuss the generalization of learning a map that is equivariant to one group, while the ground truth is (approximately) equivariant to another group. Their theoretical analysis first considers the case where the ground truth is exactly equivariant and finds a bias-variance trade-off: making the hypothesis equivariant to a larger group increases bias, while reducing variance. When the hypothesis equivariance group is smaller than the ground-truth group, the bias is 0. Next, the authors define a graph coarsening criterion via graphon theory, that results in a coarse graph $G'$ and whose nodes are a clustering of the nodes of the fine graph $G$. This gives a new symmetry group on the fine graph $G$, called $\\mathcal G_{G \\to G'}$, which equals the automorphisms of the coarse graph times the permutations of nodes in the clusters. The ground truth graph function is then assumed to be $\\epsilon$ close to a function equivariant to $\\mathcal G_{G \\to G'}$. Theoretically, the authors find a trade-off in a lower bound the risk gap (the difference in test loss between a hypothesis and its symmetrization to the approximate symmetry group). In their experiments, on fixed graphs, the authors use neural networks that combine graph methods on $G$ with methods equivariant to permutations in clusters in the graph. They evaluate different cluster sizes and find an optimum between clusters between size 1 and all nodes clustered together.\n\nSTRENGTHS:\n- Very interesting formalization of approximate graph symmetry via graph coarsening with graphons and the group $\\mathcal G_{G \\to G'}$ .\n- The authors develop novel theoretical results on generalization with approximate symmetries.\n- The ideas give an interesting theoretical insight in using clustering and permutation equivariant networks on graphs.\n- The paper mostly clearly written.\n\nWEAKNESSES:\n- The paper should explain better (in the main paper) why the risk gap is an important criterion for model selection. Lines 183-186 are hard to follow.\n- There appears to be a substantial mismatch between the theory and the experiments: the experiments rely substantially on combining equivariant nets to $\\mathcal G_{G \\to G'}$ with the ( $G$ -automorphism equivariant) graph nets on $G$ . The theory does not discuss this combination of symmetries. Also, the graph cut distance doesn't appear to be computed in practice.\n- The theoretical results talk about the symmetry group  $\\mathcal G_{G \\to G'}$ being a combination of cluster permutations and coarse graph $G'$ automorphisms. However, in their experiments the authors assume those automorphisms to be trivial, so that they only work with cluster permutations. This is a substantially less interesting class of groups, which should be more fairly stated in the paper.\n- The coarsening group $\\mathcal G_{G \\to G'}$ appears quite similar to the groupoid of nodes used in [1], which contains the automorphism group of node neighbourhood (akin to the cluster permutations in this paper) and maps between similar neighbourhoods (akin to the automorphism group of $G'$ ). The authors should consider comparing to this paper.\n- The neural network architecture $\\mathcal G$ -Nets used in the experiments should be more fully defined in the main paper.\n\n[1] P de Haan and TS Cohen. 2020. “Natural Graph Networks.”\n\nQUESTIONS:\n- Schur's lemma doesn't apply always naively on the real numbers (as they're not algebraically closed.) How do the authors apply the complex results to the real field, as mentioned in line 575 of the appendix? Do they ignore the possible additional intertwiners?\n- In section 3, it is not so clear which results rely on the group acting via a permutation of rows (so related to graphs), and which parts could generalize to other groups and representations. Could the authors comment on that?\n- In line 168, the authors say that the mismatch term goes below zero, presumably because an inner product goes above zero. However, it is unclear to me why the sign of the inner product is known in general.\n- In line 178, the authors make a comment on an inner product of characters increasing when the group increases. Can the authors clarify why this holds in general, or make the comment more specific?\n\nMinor points:\n- The authors appear to be confusingly using both the symbols $\\Psi$ and $\\mathcal Q$ for the symmetrization operation.\n- What are the two measures in (3)? Shouldn't these both be the Lebesgue measure? $\\mu$ is already used for a measure on $\\mathcal X$ .\n\nLIMITATIONS:\nThe authors are not quite transparent about the mismatch between their theory and experiments.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors observe that – while graphs considered as a class have global permutation symmetry – for specific problems, e.g. when learning on a fixed graph, the graph has a much smaller symmetry. Consequently, they attempt to answer the question of how symmetric the model should be in comparison with global permutation symmetry or the graph’s natural symmetries. They quantify this model selection problem as a bias-variance tradeoff, which they validate with numerical experiments.\n\nSTRENGTHS:\nTo my knowledge, this paper contains several new ideas. The concept of choosing a supergroup of an object’s natural symmetry group as a statistical strategy motivated by bias-variance trade-off is a novel one in equivariant ML as well as in graph-ML. Similarly, the concept of explicitly considering the natural symmetry of a fixed graph in devising the learning algorithm rather than assuming global permutation symmetry is novel as well. Finally, the concept of using graphon distances to define a notion af approximate equivariance on graphs is yet another novelty that addresses the issue of defining approximate equivariance on a space of objects that is fundamentally discrete.\n\nMethodologically, I found the mathematical approaches taken to be appropriate, and the experiments to be clean tests of the ideas presented. In particular, the authors demonstrate the existence of the proposed bias-variance trade-off in multiple settings.\n\nFinally, I found the author’s argument that the use of overly-equivariant models should be viewed as a statistical strategy that purposefully introduces some systematic error to gain regularity to be a good perspective on the current state of graph neural networks, where people have seen good practical success with permutation-equivariant GNNs on problems that have much lower natural symmetry.\n\nWEAKNESSES:\nThe paper could do with being run through a spell-check: there are a few obvious typos, e.g. “neighrborhood” in line 289. Moreover, I did not feel like the choice of using graphons to construct a distance between graphs was properly motivated: additional discussion on why the authors chose to use graphons to compare graphs would help motivate the direction of the paper. Is it only to embed the graphs in a continuous space?\n\nQUESTIONS:\nThis question might reflect my unfamiliarity with a subset of the literature, but to what extent is the bias-variance tradeoff that the authors propose a perspective that has been used for more general equivariant neural networks where the symmetry of the model is greater than the symmetry of the data? Am I correct in thinking that this perspective on the use of overly-symmetric models is novel for general equivariant-ML, not just for graph ML? If so, maybe this should be noted in the text, e.g. by noting in the discussion that these results should translate to other groups and analyzing this as a future direction.\n\nLIMITATIONS:\nLimitations and potential negative societal impacts are appropriately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}