Rethinking the Benefits of Steerable Features in 3D Equivariant Graph Neural Networks
Authors: Shih-Hsin Wang
Yung-Chang Hsu
Justin Baker
Andrea L. Bertozzi
Jack Xin
Bao Wang
Venue: ICLR 2024 poster (2024)
Keywords: Steerable features
Equivariant graph neural networks
Message passing
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We discuss the benefits of steerable features of different types for 3D equivariant graph neural networks

Abstract: Theoretical and empirical comparisons have been made to assess the expressive power and performance of invariant and equivariant GNNs. However, there is currently no theoretical result comparing the expressive power of $k$-hop invariant GNNs and equivariant GNNs. Additionally, little is understood about whether the performance of equivariant GNNs, employing steerable features up to type-$L$, increases as $L$ grows -- especially when the feature dimension is held constant. In this study, we introduce a key lemma that allows us to analyze steerable features by examining their corresponding invariant features. The lemma facilitates us in understanding the limitations of $k$-hop invariant GNNs, which fail to capture the global geometric structure due to the loss of geometric information between local structures. Furthermore, we investigate the invariant features associated with different types of steerable features and demonstrate that the expressiveness of steerable features is primarily determined by their dimension -- independent of their irreducible decomposition. This suggests that when the feature dimension is constant, increasing $L$ does not lead to essentially improved performance in equivariant GNNs employing steerable features up to type-$L$. We substantiate our theoretical insights with numerical evidence.

Published as a conference paper at ICLR 2024
RETHINKING THE BENEFITS OF STEERABLE FEATURES IN 3D EQUIVARIANT GRAPH NEURAL NETWORKS
Shih-Hsin Wang', Yung-Chang Justin Baker',
Andrea Bertozzi", Jack Xin & Bao Wang**
1 *Dearrtment of Mathematics and Scientific Computing and Imaging (SCI) Institute University of Utah, Salt Lake City, UT 84102, USA
DDepartment of Mathematics, Purdue University, West Lafayette, IN 47907, USA *Department of Mathematics, UCLA, Los Angeles, CA 90095, USA *Department of Mathematics, UC Irvine, Irvine, CA 92697, USA
ABSTRACT
Theoretical and empirical comparisons have been made to assess the expressive power and performance of invariant and equivariant GNNs. However, there is currently no theoretical result comparing the expressive power of k-hop invariant GNNs and equivariant GNNs. Additionally, little is understood about whether the performance of equivariant GNNs, employing steerable features up to type-L, increases as L grows especially when the feature dimension is held constant. this study, we introduce key lemma that allows us to analyze stecrable features by examining their corresponding invariant features. The lemma facilitates us in understanding the limitations of k-hop invariant GNNs, which fail to capture the global geometric structure due to the loss of geometric information between local structures. Furthermore, we analyze the ability of steerable features to carry information by studying their corresponding invariant features. In particular, establish that when the input spatial embedding has full rank, the information- carrying ability of steerable features characterized by their dimension and remains independent of the feature types. This suggests that when the feature dimension is constant, increasing T. does not lead to essentially improved performance equivariant GNNs employing steerable features up to type-L. We substantiate our theoretical insights with numerical evidence.
INTRODUCTION
Machine learning (ML) tasks have different kinds of inherent Euclidean symmetries. For instance, translating or rotating an image does not change its label (invariance) 21 while applying Euclidean isometries to molecule results in corresponding changes in its dynamics (equivariance) [3] ().
Enforcing Euclidean symmetry neural network design [35 significantly improves sample efficiency [16] (8. generalizability (31), and robustness [13), and preserves the principles of physics 12]. Utilizing steerable features provides a powerful framework for crafting symmetry-award neural networks. In particular, it enables the design of neural networks with equivariance to 3D Euclidean transformations by leveraging the group representations of O(3) and SO(3). The reducibility of decompose any steerable vector into a direct sum of steerable vectors of different types. These types of stecrable vectors are defined 442 in particular, type-0 and type-1 steerable vectors corresponding to scalars and 3D vectors, resp. Research in designing equivariant models, especially graph neural networks (GNNs) that update steerable features up to type L, has surged. For instance, SchNet 34,, DimeNet [15], SphereNeet [25], and ComENet 440] are invariant GNNs that propagate invariant features (L = 0). Equivariant GNNs, e.g., as proposed in (32 33 update steerable features up to
the group representations of ((3) or SO(3) 30 42) allows
by Wigner-D matrices
L = 1. Meanwhile, numerous architectures are designed to handle steerable features for L > 1; see
e.g. 8 45 37
42
[3
I8
43 22 5.
1.1 REMARKS ON SOME REMARKABLE STUDIES
Recent research, especially (20). highlights performance disparities between 1-hop message-passing invariant GNNs and equivariant GNNs. [20 introduces the geometric Weisfeiler-Lehman (GWL) test and its invariant version (IGWL) to characterize the expressive power of 1-hop equivariant and
Cordespond to wangbaon sangbaonj@gmai com

---

Published as a conference paper at ICLR 2024
invariant GNNs, resp., and then shows that 1-hop invariant GNNs may underperform equivariant GNNs- - primarily due to their limited ability to capture the global geometry of the graph. Intriguingly, 20 also reveals that 1-hop invariant GNNs can be as expressive as equivariant GNNs for fully connected input graphs. However, the theory in (20] is limited to 1-hop message passing, leaving questions about the implications of introducing multi-hop aggregation into the message-passing of invariant GNNs. E.g.. ComENet [40] and SphereNet [25] introduce the concept of completeness of edge attributes - a property held by invariant edge features that enable them to uniquely determine the spatial embedding of a point cloud up to the group action. Specifically, ComENet achieves this completeness by computing dihedral angles between local structures, gathering information from up to 2 hops neighbors. Both ComENet and SphereNet can outperform equivariant GNNs in molecular modeling tasks, even when dealing with non-fully connected input graphs.
The effects of steerable features of different types have also been explored. E.g. B 23 2
28 show
that the performance equivariant GNNs that use steerable features up to type- - can be enhanced by increasing L. However, these experiments typically lack the control of feature dimensions [12]. This requires rethinking the benefits of increasing L while maintaining the feature dimension. Indeed, [4] first compares the performance of SEGNN a particular equivariant GNN using different L, reporting limited improvement beyond L = 1 when feature dimensions are held constant.
1.2 OUR CONTRIBUTIONS
and build new understandings of the ex- pressive power and peessag--passing mechanisms of invariant and equivariant GNNs. We summarize our major theoretical contribution as follows:
We aim to address the theoretical questions in Section 1.1
We establish Lemma] [ a key lemma showing that any steerable feature corresponds to some invariant features. This lemma serves as a comnestone for theoretical analysis, enabling to examine stecrable features by studying their corresponding invariant counterparts.
We demonstrate that message--assiingg using steerable features can be interpreted as message- passing with their corresponding invariant features. This perspective reveals that both equivariant and invariant GNNs propagate invariant features across different local neighborhoods. However, invariant GNNs, when compared to equivariant GNNs, lack the intrinsic capability to capture geometric information between local neighborhoods, even when adopting the k-hop message- passing. In particular, k-hop invariant GNNs may struggle to capture the changing geometry between k-hop local structures and fail to obtain accurate global invariant features.
We analyze the ability of steerable features to carry information by studying their corresponding invariant features. In particular, we establish that when the input spatial embedding has full rank, the infoommttionncarryyie
ability of steerable features is characterized by their dimension and remains independent of the feature types utilized in steerable features. This result indicates that when preserving the feature dimension, the performance of equivariant GNNs employing steerable features up to type-L may not increase as L grows.
We provide numerical evidence that consistently echos the above theoretical insights.
2 BACKGROUND
We recap group theory, steerable vector spaces introduced in [4, the concepts of geometric graphs and geometric Graph Neural Networks (GNNs). We also review the Geometric Weisfeiler-Lehman test (GWL) and its relevant results from 20 in Appendix D
Group theory. Let 6 be a group, and consider a space X on which @ acts When an element I X undergoes the group action of G, denoted as g x, we define the G-orbit of x as the set C x = {o- x E X g E }} C X. The quotient space X/6 contains all @-orbits. Additionally. we define the stabilizer of x as the subgroup 6_- == {g G g. J = x) S G.
Assuming that @ acts on both spaces X and Y. we refer to function f X + Y as G-equivariant it satisfies f(g x) = g (xr). and as ---invariant if it satisfies f(g xz) = f()).
Steerable vector space. We revisit the concept of steerable vector space following [4) where the group action is induced by group representation. A vector space V over said be G-steerable for G if group representation p G + CL. of & is assigned to V. In other words, for any vector e E V, the transformation g v given by The matrix multiplication g a = p(g)v.
When G is 50(3) or (3), we can decompose the representation into irreducible representations. Consequently, is sufficient to investigate steerable vector spaces transformed by these irreducible
(V) denotes the general linear group of the vector space V.
2

---

Published as a conference paper at ICLR 2024
representations. It is known that the irreducible representations of (3 have dimensions 21 + 1 for = {0] U U, and they are defined by the Wigner-D matrices (D'). We refer to the vector space transformed by D' of order as a rype-l (3(3) (003))-tterrblee vector space, denoted as Vi.
Moreover, notice that ((3) is the direct product of (3)3) and the inversion group I = (1, -}. implying that any representation of (3) can be written the product of representation of SO(3) and a representation of I. The inversion group has only two irreducible representations: the trivial representation Pr (I) (r(-
and the sign representation Ps (T( (-11) -1. As 0 Pr or p.}
result, all the irreducible representation of O(3) can be expressed as ((())
by writing the element of ((3) as a product i g where i E I. E SO(3). ) An alternative way to represent these irreducible representations involves the determinant, as follows:
(g) = de)( D' (d))(g))) and (g)
(g) det(g)(+1 (D((d)))t.D((de(()g)))))
where E ((3) and I 2 0. To avoid ambiguity, we use the terminology of ind and aug for clarification and ease of subsequent study. Lastly, let Vi,ind denote the (33)-steerable vector space acted upon by D Dind and Vi,ang denote the -(3)-steerable vector space acted upon by D We may aug
denote Vnd simply as Vo since both correspond to trivial representations.
Geometric graphs. Let (V,E) be an attributed graph comprising m nodes, where each node E V has feature embedding fi E R" and a spatial embedding æ; E R The input embeddings can be organized as two matrices F = .i,....,Jee fm] E R and = [ , E R We can represent this as G = (V. E, F.X). referring to this attributed graph two geometric graphs G1 = (V1, & ,
geometric graph. Consider = (V2, Eg, F , x²)) where the underlying graph structures and feature embeddings are isomorphic. In other words, there is adge-preserving bijection b V1= V/2 s.t.
f bbi) where do not assume the uniqueness of b. We say I1 and
G2 are identical up to group action if there is graph isomorphism b such that x g- for some b(s)
g E Let (x represent the k-hop neighborhood of node i, the set of nodes in V that are reachable from i through path with & edges or fewer. Then we say Gi and G2 are k-hop identical if there is graph isomorphism b such that for any node i E V1, there exists g; E G satisfying o gi a G2 b(j)
for any j E (,(4) {i). Otherwise, we say 91 and G2 are k-hop distinct if, for all isomorphisms b, there is node E V1 such that for any E 6, we have x5'' 4g
Go for some j E A²( u {i}. bb))
k-hop geometric GNNs. We extend the framework of geometric GNNs in 20 to k-hop setting. This framework can be regarded as an abstract of several existing invariant and equivariant GNNs, e.g. 33
32
4 2 ; see Appendix B for details. Consider geometric graph G EV,EFF F, XX) with &
being (3)) or (33). Geometric GNNs propagate features from iteration to 1. + 1 as follows:
(I) where F.y = σ xy. {- denotes a multiset, and UPD and AGG are learnable update and aggregate (G-equivariant) functions. Notably, the equivariance of UPD and AGG ensures that this message- message-passing passing mechanism remains equivariant, that is, we have
f((+1) = UPD
f("),, j (EEN
HJ), with f = ft
(1422) (o))(t))))) UPD (p (o),!") AGG( ( (o)s!" , (0) .g gg. 1j E E B)).
(2)
where ((++)((g)) (g) and p² (g) are group representations acting on f(((+1) and -!(') resp., and g Bij is the regular representation. The features f in geometric GNNs consist of steerable features up to type-L, where L = 0(> 0) corresponds G Saauriinnt/--quvvriinnte GNNs.
3 MAIN THEORY
Steerable features and invariant features. From now on, let 6 be group acting on 3xm We define two key concepts: steerable features refer to the steerable vectors that are generated equivariantly from the input spatial embedding X, meaning that they can be expressed as f(X) for some 6 (--qquvvariant function f mapping from 3 to steerable vector space. Similarly, invariant features are characterized as the steerable vectors f(X) produced by some G-invariant function f. Notice that these two concepts cover all the notions of existing terms, such as steerable feature fields and steerable feature vectors [37]: see Section for more discussion.
[2 4T
Steerable features and their corresponding invariant features. By the axiom of choice 177. can choose a "represennative" for each orbit in 33xm /6. Specifically, there is a (set-theoretical function e 3x /6 3x3m that maps each orbit element within that orbit.
3

---

Published as a conference paper at ICLR 2024
Note that for any X, X' E Raxm they belong to the same 6-orbit if and only if X g X' for = is not empty since c(6 X) and X lie in the same orbit @ X. Once again, applying the axiom of choice,
some g E 8. Consequently, for any given X, the set {g E @ g g. x) = x
can select one element from this set, denoting it as gx- Then we can represent any X E R using the pair
X (c(B... X): gx), where dx c(6 X) X. Next, introduce a key lemma that serves as the cornerstone of our theoretical framework; a similar finding can be found in [44].
Lemma 1. Let V be d-dimensional (-steerable vector space with the assigned group representation
P:6 + GL( If RR3mm V is --equivariant, then there exists unique G-invariant function
A: 3RR Vidid s.1. [(X) - p(gx) )x(x) ), where Vn denotes the ID trivial representation of e In particular, the following map is well-defined
{S f:R3xm= + V f (-equivariant) > [A: R3 - -ood X G-invariant).
(3)
Remark 1. While we consider the group action on R3xm, it is important to note that the same result
applies to group actions on any space. One can interpret that 9x absorbs the group action on X and constrains X(X)
) to be invariant. More precisely, we see that ((b) (XX) p(b)
f(h X) = p(gh-x (X) for any E B. However, is not necessary that h gx = - X-
We observe from Lemma that any steerable feature with dimension d, denoted f(X), can be substituted with a group element fx and d-dimensional invariant features X(X). However, since the selection of gx remains unaffected by the function f. what we truly observe is that the steerable feature f(XX) corresponds to a unique d-dimensional invariant feature X(X). The
correspondence of ((X to X(X) shares similarities with the notion of scalarization in and A detailed discussion of their distinctions is provided in Appendix A
3.1 MESSAGE- PASSIIG MECHANISMS OF GEOMETRIC GNNs
It has been demonstrated that 1-hop invariant GNNs underperform equivariant GNNs 20 However, the underlying mechanisms of message passing that distinguish these two approaches remain unclear. In this section, we leverage our framework to shed some light on this issue. particular, we will show that k-hop invariant GNNs do not possess an inherent capability to capture geometric information between local structures, whereas equivariant GNNs do.
Steerable features propagate like invariant features. We first treat the steerable features f(() in the propagation scheme defined in equation (L). as functions of X and denote them as f(() = f: (x).
By applying Lemma we can express (X) = p (a) (X), where A is G-invariant, and for simplicity, we represent gx as g. Additionally, since we have X = 9 c( X) where c(B X) is G-invariant, and -r' are the invariant features corresponding to Tzy æ; 2y due to the uniqueness described in Lemma| [ Consequently, we arrive at the following relationship:
UPD (X),
(/) (X). ay Iie E (!
= =UPD (g)a (X), AGG (g) (X), (0) (o)afo (X),9 (0 aa,)) 1 E ATT 1/0 (X), 'z,,, j E EN H)).
-0"(1) (g) UPD (A (X) AGG
Since f (1+1) (x) = ()+))) (g)A (X), the uniqueness of corresponding invariant features implies
1 ((()(() UPD (2 (X), AGG (x), (X). m. l E ())
(4)
This reveals that the propagation of steerable features can be effectively understood as the propagation of their corresponding invariant features. Therefore, we can analyze the message- paassagg-pssingggggg mecha- nism by examining how the corresponding invariant features are aggregated and updated.
Message aggregated from multi-hop neighborhoods. To investigate the aggregation of local features from multi-hop neighborhoods, we explicitly specify the input spatial embeddings for each steerable feature f((); namely, for any iteration t and node i, we write f(()) = f((() (x ()) where x! [=]]) u()) represents the spatial embedding, consisting of all coordinates of node i
and its tk-hop neighbors M((())) In particular, the tk- neighbor Ag(th) includes all the nodes that can propagate information to node i through h-hhpp aggregation i times. Without loss of generality, we may assume the group representations are all the same for any iteration, i.c. p ((4) for any t. Next,
For G = SO(3), corresponds to the type-O steerable vector space we defined in Section For simplicity, we employ the same notation here.
We do make any assumptions about the order of the coordinates or indices here.
4

---

Published as a conference paper at ICLR 2024
utilize Lemma 1 to examine how corresponding invariant features propagate through different multi-hop neighborhoods. Specifically, we express these steerable features as p(g: ( )x where
x (1) (x
(t) (0)) corresponds to the local invariant features, and gi (!) denotes g xx(") for simplicity.
Our goal is to investigate how invariant features 2! at iteration l are aggregated and updated into invariant features 2 (++1)) at iteration t ++1 1. In particular, we have
(((1)) UPD (f AGG(H)! . , 1 J E H)) UPD (plo 2 AGG (A ((
(5)
((o..))) 1 E "
Next, by leveraging the fact that (!(tt) p(g. ) ( and the uniqueness of X ( established in Lemmal
| we deduce the propagation of the corresponding invariant features.
2 x(+))) = UPD (p(lo!" (o((o(+))) "((l""")))=""""" )a AGG (fl((a (1(() ) "! )a9),
(6)
p((g. (( g) D")" ((g) ((of+)))= 9: (o.") (1) æz 1 E N, B)):
We assert that the collection of elements { ((((o 0 lj A, U { plays a crucial role in enabling geometric GNNs to capture geometric information between the local structures A7(th) To clarify this, we consider k-hop invariant GNNs where is trivial, resulting in the set of elements {o((o, (+++ - ) j E N, U {i}} being inevitably overlooked. Then we have the following:
Theorem 1. If 91 and G2 two k-hop identical graphs, then any iteration of k-hop invariant GNNs will get the same output from these two graphs. That is, there is a graph isomorphism b such that
X (1+1)_
(1+1) for any i, even though 91 and G2 may not be identical up to group action.
b())
Remark 2. We also extend the IGWL, test in (20) to k--op setting and show that (1-hop) GWL. is
still more powerful than h-hop IGWL; see Appendix fioor details.
Theorem iiplies that k-hop invariant GNNs may strug- gle to capture the changing geometry between k-hop local structures. To illustrate this point, let's consider the examples of k-chains discussed in (20). For any k let's examine a pair of graphs, Gi and Go, each consisting of 2k: + 2 nodes. In these graphs, there are 2k nodes arranged in a line, with differentiation based on the ori- entation of the two endpoints, as demonstrated in Fig. [I] By assigning the same attributes and customizing the spatial embeddings, we can make these graphs k-hop identical but (k + 1)-hop distinct. Specifically,
80....
2k nodes
2 nodes
Figure 1: A pair of graphs cach consisting of 2k + nodes. These graphs are nearly identi- cal, differing only the orientation of a single edge, marked in blue. Despite this minor dis- tinction, these graphs remain k-hop identical.
the geometry between (/ and Ar(h)) and the geometry between NT,10 and Ar differ, as the unions j1
andA/
A U 9i and A7th) U. G2 are not identical. However, since Gi and G2 are k-hop identical, we expect that k-hop invariant GNNs will likely struggle to distinguish between the distinct geometries present in these scenarios. We further analyze this example empirically in Section
Let V and W be steerable vector spaces with the assigned faithful group representations PV and PW. i.e., the group homomorphisms PV : G + GL(V) and pw 6 + GL(W) are injective. The injectivity implies pv ((o. - )) and Pw ((+(()))))))))1))) ) 0,
(t) come from the group element
(o("+)))''g) They capture the same geometric information defined by (g. ( o With assumptions on the injectivity of UPD and AGG, we can show that equivariant GNNs that learn steerable features on faithful representations can distinguish any two k-hop distinct geometric graphs.
Theorem Consider 1-hop equivariant GNNs learning features on steerable vector space V where the aggregate function AGG learns features on steerable vector space W. Suppose V and W are faithful representations, and UPD and AGG satisfy certain assumptions on the injectivity outlined in Proposition 8 in the appendix. Then with k iterations, these equivariant GNNs learn different multisets of node f features { } on two k-hop distinct geometric graphs.
Remark 3. It can be verified that D and D which correspond to type-1 steerable vector spaces Vi for (3) and V ind for O(3), are faithful representations. As a result, we conclude that equivariant GNNs defined in equation 77. that propagating steerable features up type L. >> O, all exhibit the
5

---

Published as a conference paper at ICLR 2024
same capability to capture geometric information between two local neighborhoods For additional discussion on the importance of faithfulness. refer to Appendix
To mitigate the limited ability of invariant GNNs mentioned above, several approaches have emerged to encode geometric information between local structures into edge attributes. For instance, ComENet peeelllyyyy features. Completeness enables these edge features to uniquely determine the spatial embedding X up to isometries. ComENet achieves this completeness by encoding spherical coordinates (in triplets) and dihedral angles (in quadruplets) as edge features, which involves computation within 2-hop neighborhoods. The former specifies the locations of nodes within 1-hop neighborhoods, and the latter captures the angle between two 1-hop neighborhoods that share two nodes, thus addressing the issue of missing geometric information between these neighborhoods. Consequently, while ComENet employs 1-hop aggregation schemes, these edge attributes effectively contain all the necessary geometric information within and between 1 1-hop neighborhoods for molecular tasks [40]
introduces the concept of completeness of edge attributes, property associated with invariant edge
4 However, it's important to note that dihedral angles may not always be well-defined in quadruplet particularly when facing a chain structure within the graph, such as cis-trans stereoisomers or Fig. [ Additionally, the complete edge attributes that ComENet utilizes are extracted within 2-hop neighborhoods, while Theorem [ [suggests that k-hop invariant GNNs may still struggle to capture the true global geometry effectively. This result for ComENet is empirically validated in Section 5
Some remarks. We have pointed out that k-hop invariant GNNs inevitably ignore the information that captures the geometry between k-hop neighborhoods. Namely, they are limited to the information within k-hop neighborhoods. While encoding complete edge attributes emerges as potential remedy, it remains open question whether constructing complete edge attributes can be achieved for other tasks beyond molecular graphs, especially when dihedral angles cannot be well-defined in quadruplets. Theorem further suggests that invariant features confined to specifie k-hopp neighborhoods may not be sufficient to capture the accurate global geometry and global invariant features of geometric graphs, thereby emphasizing the need for encoding global features as a potential solution to address this limitation. We leave this intriguing avenue for future research.
3.2 COMPARING EQUIVARIANT GNNS USING DIFFERENT TYPES OF STEERABLE FEATURES
Remark 3 suggests that when equivariant GNNs learn steerable features up to type L, they exhibit the same capability to capture geometric information between two local neighborhoods. To understand if the performance of equivariant GNNs can be enhanced by increasing T., we analyze the information- carrying ability of steerable features by investigating their corresponding invariant features.
Lemma D states that any type-/ steerable feature f(X) E V1 corresponds to a (21 + 1)-dimensional invariant feature A( A due to the decomposition f(X) ) (ox)) Now, we raise the question: Does any (2l + -immeensola invariant feature X (X) correspond to a type-l steerable feature f(X)? Note that this question is essentially asking whether the space of all type-/ steerable features f(X) has a dimension of 21 + 1 since D' D'(gx) is invertible. While Lemma is not affected by rank( (X), the rank of the spatial embedding X, the answer to this question is contingent upon the rank. This is because equivariant functions must obey "Curie's principle" [36). For a brief intuition, refer Fig. in Appendix G Based on this principle, we present the following results, answering the aforementioned question. Specifically, we first examine G-equivariant functions without focusing on a spatial embedding and then shift our attention to a spatial embedding obtain a precise answer.
Theorem 3. Let X, denote the set {X E m vank(X) = r}. Then we have a one-to-one correspondence between -equivaara functions and -i)-invariaant functions:
[ X3- + Vi,ina ʃ: ((3) eqquivarra)))) = (X X3- + v À (33)-inaai, (7)
{f: X² + Vi,in f - } = {} X₂ + V@L+1 | | -i))iinvr)i
{f:X, - Vi,nd ʃ 0(3) (((()))quuvariamn = (A X + V A: (33)-invariam))
(o + Vi,ind | f -equivrrui } = {ʃ %% = {0} - [0)]-
[f: + Vi.ugg ( } = {} Xx + 1/
| (3))innariiant)
(8)
[f: - Vi,aug f (3) = (A X2 V
|x: (3)
{f X1 - Vi.ang f: ( = ( X + (0)),
[ Xn- Vi.nng
f (:)) = ( Xx =(0) (0))
*When the nodes are collinear, there are infinitely many planes containing them.
6

---

Published as a conference paper at ICLR 2024
Remark 4. Note that any G-equivariant function paxm + VV can be expressed summation: f 2- (X), where 1x, represents the indicator function, and f-1x. (X) can be considered as a G-equivariant function that maps from X, to V. Consequently, the results presented above suffice describe any (33--vvvvaaiiatt function that maps steerable vector space.
Corollary 1. Let X R3 be a spatial embedding. We have the following relation between (tee-alee features and invariant features:
1. If rank(X) = 3. there bijection between steerable features Viaind and (21 + 1)- dimensional invariant features, as well as a bijection between steerable features in VLang and
(21 + 1)-dimensional invariant features.
2. If rrank(X)
2, there is a bijection between steerable features in Viand and (1 + 1)- dimensional invariant features and a bijection between steerable features in Vi,aug and 1-dimensional invariant features.
3. If rank X) 1, there is a bijection between steerable features in Vnn and 1-dimensional invariant features, while there is no non-trivial steerable feature lying in Vi.g
4. There exist only trivial steerable feature O and trivial invariant feature 0 if rank(()) = 0.
Similar results for SO(3) are presented in Theorem 5 and Corollary Appendix C Due to the reducibility of G-steerable vector spaces, we can decompose any G-steerable vector space into direct sum of steerable vector spaces of different types. Then, we have the following two corollaries:
Let X3 denote the set { {X E Raxm rank( = 3]. Then for any G--tterabbe vector space of dimension d, denoted as V, we have a one-to-one correspondence:
Corollary 2.
U %3 + V f {-equivariant) = {x: : X3 - Vidd X O-invariant) where the map between these two spaces is induced by the map defined in equation 3
Corollary 3. Let V and W be two G-steerable vector spaces of dimension d. Then for any B-
equivariant function Jv Xx - V, there is a (--qqivaaiant function fw X3 - W such that for any X E X3, we have (x = C- 3--ivariant function À where pv, PW are the group representation on V and W, resp.
(xX) and fw (X) = (gxgx)m (XX)) for the same
Consider learning steerable features in V and W, resp. Corollary 3 suggests that if dim V = dim W, regardless of their irreducible decomposition, any (learnable) ---quivariant function fv X3 - V can be replaced by 6 ---quiiariaant function fw : Xx - W where they learn the same corresponding invariant features. That is, the invariant features carried by steerable features is primarily characterized by the feature dimension independent of the highest type utilized.
Specifically, we have the following equivalence among geometric GNNs, detailed in Appendix
Theorem 4. Consider two geometric GNNs learning features on steerable vector spaces V and W of the same dimension, resp. Denote their update and aggregation functions at iteration t as (PD UPD() (t) and AGG (t) AGG Then for any collection {(UPD)) (t) AGG())) }], there exists a collection ((UPDW)
}h such that for any fully connected graph, they learn the same corresponding invariant features x!4) for any iteration 2 on each node i.
Theorem 4 holds for any representation, especially non-faithful representations, but relies on the assumption of fully connected graphs. This result establishes the equivalence of geometric GNNs on fully connected graphs; this is similar to the equivalence of IGWL and GWL tests on fully connected graphs (20 but without strong assumptions on the injectivity of update functions and aggregate functions. Additionally, when dealing with non-fully connected graphs, our earlier exploration in Section 3.1 highlighted that learning features on faithful representations versus non-faithful representations results in different expressive powers. This discrepancy arises primarily because each node in non-fully connected graph can only capture the global geometry through message passing.
Some remarks. Our proof of Theorem 4 relies on a precise understanding of how these models capture global geometry through message passing, making it challenging to ascertain its validity for non-fully connected graphs under the assumption of faithfulness of representations. While we believe it may be more feasible to demonstrate this by considering specific architectures and gaining better understanding of how they obtain global geometry from local information, we consider exploring this aspect further future work. Nevertheless, Theorem 2 and Remark 3 suggest that equivariant
7

---

Published as a conference paper at ICLR 2024
GNNs learning steerable features up to type-L exhibit the same capacity to capture global geometry. We propose that when the feature dimension remains constant, the performance of equivariant GNNs employing steerable features up type-L may not increase as L grows. However, we cannot assert that using I. = 1 is sufficient. The concept of expressiveness includes two key aspects: the capacity of features to carry information and the ability of model extract it the latter is commonly referred to as universality [II]. We intended to focus on the former, as the latter is subject to the architecture of the given model, while the former not. Moreover, due the lack of regularity of functions appearing in Lemma 1 we decided to defer discussions on universality to future works.
More detailed discussions and potential methods are available in Appendix B
4 ADDITIONAL RELATED WORKS
Param.
Fear. Dim.
256
39M
2304
Numerical comparisons in steerable feature types. In a recent experiment in (20,, a comparison was made regard- ing using different types of steerable features. However, this experiment was not specifically designed as an in- variant classification task. Therefore, the conclusion that higher-type steerable features are superior may not directly apply this context. Additionally, in experiments conducted in 23 2 28). it was indeed observed that higher-type steerable features improved performance. Nevertheless, it should be noted that these experiments do not maintain fixed dimensions for hidden features, making direct comparisons challenging. Table T illustraaes the difference in the number of parameters and the steerable feature dimension for the eSCN model (28) with varying feature type L and channels c. The ablation study in [28] compares rows 1&3, while rows 2&3 provide a more suitable comparison.
2
824
113M
7416
256
107M
7424
Table 1: Total parameter count (# Param.) and stecrable feature dimension (Steer. Dim.) for eSCN models with varying order of steer- able features L and stecrable channels c.
EXPERIMENTS
In this section, we empirically verify our theory on several benchmark tasks. First, we verify the limitations of invariant models using the synthetic k-chain dataset. Second, we perform an ablation
study over the steerable feature dimension when training steerable models on the large-scale OC20
IS2RE and S2EF datasets [6]. The invariant models considered are SchNet [34], DimeNet++ [14),
SphereNet [25). and ComENet [40]; the equivariant models include EGNN [3 ClofNet (9] and
GVP [9:: the steerable models are eSCN (28). EquiformerV2 24 and MACE
We provide
details on each model, training procedure and hyperparameters in Appendix E
and F.4 resp.
k-Chain. In the k-chain task, motivated by (0 and illustrated in Figure I we aim to differentiate chains using the orientation of the terminal node. Each chain is made of l: + 2 nodes, all possessing constant features. The connecting edges are undirected with uniform distance ri = 5. Chains are assigned binary labels based on the orientation of their final node. The dataset consists of pairs of chains, one of each label, which undergoes the same random rotation/reflection and translation. 50 transformed graph pairs are split into 50%/33% train/validation/tess splits with balanced labels.
Layers
2
1
2
3
4
h-hup chain
=
k=
==3
L=
SchNet
50.0 + 0.0
50.0 = 0.0
50.0 0.0
50.0 = 0.0
60.1
= 0.2
5..0
50.0
+ 0.0
DimeNet++
50.0 1: 0.0
1
50.0 1 0.0
50.0
0.0
5..0
50.0
0.0
SphereNet
50.0 + 0.0
50.0 = 0.0
50.0 0.0
50.0 = 0.0
50.0
+ 0.0
5..0
50.0 : 0.0
ComENet
55.0 = 4.5
59.0 11.6
53.0
51.0 6.2
50.0
= 0.0
5.0
51.0+20 2.0
EquiormerV2
71.0 = 3.0
76.0 8.0
83.0 6.4
43.0 9.0
67.0 2 4.6
67.9 9.0
61.0 + 5.4
L L=1
FGNN
50.0
+ 0.0
100.0
95.0 15.0
50.0 0.0
50.0 1 0.0
90.0
20.0
100.0 -
GVP
50.0
1 0.0
100.0 4: 0.0
100.0 + 0.0
50.0 1 U.0
92.5 1 16.0
91.5 = 17.3
95.0 1: 15.0
ClofNet
50.0 1 0.0
50.0 1.00:00 0.0
100.0 0.0
50.0 1 0.0
50.0 1 0.0
100.0 1 0.0
100.0 1 0.0
MACE
50.0 - 0.0
100.0 0.0
100.0 1
50.0 0.0
0.0
100.0
10.00000 0.0
100.0 - 0.0
cSCN
61.0 + 8.0
60.5 10.0
613 + 18.2
53.0
= 4.6
63.0 = 9.0
60.0 18.0
56.0 + 10.2
EquiformerV2
90.0
1:
76.0 6.6
6.0
98.01140
L =
MACE
50.0 = 0.0
100.0 + 0.0
100.0 + 0.0
50.0 = 0.0
100.0 + 0.0
100.0 = U.O
100.0 + 0.0 6.6
eSCN
62.0 +: 7.5
52.0
10.8
56.0
10.2
EquiformerV2
73.0 1: 4.6
88.0
1 4.0
86.0
4.9
86.0 1 4.9
89.0
1 3.0
88.0
4.0
83.0 + 9.0
Table 2: Test accuracy for the k-chain dataset with different lss. Models are further distinguished by their use of type-I. features. Cell shading is based on two standard deviations above or below the expected value. Unit.%
Table 2 reports the 10-fold cross-vas validation mean test accuracy and standard deviation for varying chain length and model depth. The values are shaded based on the expected value from Section 3 In general, observe that models perform expected. The outliers can be categorized as equivariant models EGNN and ClofNet and quasi-equivariant models eSCN and EquiformerV2. The consistent underperformance of the equivariant models can be attributed to over-squashing as discussed in 20)
8

---

Published as a conference paper at ICLR 2024
The abnormal performance of the quasi-equivariant models cSCN and EquiformerV2 can be attributed to the error in their equivariance. More discussions on models with outlying performance are provided in Appendix F.I Table 2 shows that invariant features L 0 are insufficient to distinguish the geometry regardless of the model depth. Also, architectures with I. = 1 or 2 and finite k-hops can learn with sufficient depth. We show further comparisons up to A 4 in Appendix [.
IS2RE. The OC20 IS2RE dataset [6] is large-scale molecular property prediction task that uses molecule's ' initial structure to predict its adsorption energy. In this task, we perform an ablation study on the order of the steerable features L and the size of the steerable feature dimension In particular, we select a value e to roughly fix the steerable feature dimension when varying L.
Model eSCN eSCN
L
Dim.
Loss
Energy MAE [mcV]
EWT [%] f
2
206
1854
ICM
0.389 1 0.008
842
1: 13
1.94 1: 0.12
4
98
1862
9M
0.408 + 0.006
929 + 15
1.74 0.12
eSCN
6
64
1856
8M
0.3836 0.003
872 1 6
1.91 19911000 0.19
EquiformerV2
2
306
9M
0.369 + 0.009
2.02 =
EquiformerV2
4 16
30-1
15M
0.364 O 005
832 + 11
2.03 + 0.14
Table 3: Validation results of the stecrable model ablation study on L and over 4-folds of the IS2RE dataset
with 10k training molecules. We observe that higher type-L stecrable models may not perform best.
We list the ablation study in T showing that when the steerable feature dimension is fixed, lower order type- steerable features may provide improved results. These results are further validated in Figure [ which illustrates the validation curves over 4-fold cross-validation with plotted mean and shaded standard deviation. We include additional results for EquiformerV2 in Appendix F.2
eSCNI- -2 206
esCNIL- 206)
Velidation EWT
9,000
eSCNIL a 4.c eSCNI 66
Vnlccattn MAKE
escMiL -611
Valldaaton
0.8
eSCNIL 6-644
0.055
0.5
eSCNIL -2
1.0
3.030
esCNIL - -.6-98)
eSCNIC = 6,
Validation Epech Figure 2: Validation results for the ablation study on eSCN for the IR2SE datasct with 10k training molecules. Depieted is the loss (left), energy MAE (middle), and energy within the threshold(EwT)) (right) over 12 validation epochs. We plot the mean over four runs and shade the standard deviation.
Validation Epoch
Validation Epach
S2EF. The OC20 S2EF dataset 66] uses the molecular strueture to predict molecular ad- sorption energies and per-atom forces. Table [ lists the size and the steerable feature dimen- sion for each model. Table 4 reports the re- sults of our training procedure against the re- ported results from (2). We observe that with a fixed feature dimension and fewer training
Energy MAR
Force MAE
Force Coe
Coully dSCN
[meV]
ImeV/A1
256
204
0.063
1.40
L 2.0 924
240
23.1
0.596
1.77
Table Validation results after 8 training epochs for the eSCN model on the S2EF dataset with 2M training molecules. Results marked with * are reported from (28 and average over four after 12 training epochs.
2, c 824] consistently outperforms its lower dimensional counterpart eSCNIL- = 2, e = 256] and outperforms eSCN[L = 6, c = 256] on energy MAE and EFWT. These tasks do not rule out confounding factors like over-squashing II, but strongly support our results.
epochs, the eSCN[L-
6 CONCLUDING REMARKS
We analyze the advantages granted by steerable features in GNNs. Specifically, we highlight two key findings: (1) Propagating steerable features of type-L equips geometric GNNs with inherent ability to automatically capture the geometry between local structures and obtain global invariant features from local ones. However, relying solely on propagating invariant features confined to specific k-hop neighborhoods is insufficient for enabling geometric GNNs to capture global information of all geometric graphs precisely. (2) When keeping the feature dimension constant, increasing using steerable features up to type-I. cannot essentially improve the performance of equivariant GNNs. In sum, our findings highlight the necessity of incorporating global features, extending beyond k-hop neighborhoods for fixed k. to achieve the same expressiveness in invariant GNNs as in equivariant GNNs. Additionally, the traditional trade-off between performance and computational cost of using steerable features in equivariant GNNs should be reevaluated. Specifically, when maintaining constant feature dimension, achieving better performance may not be guaranteed by using higher-type steerable features in equívariant GNNs, and this choice may come with computational overhead.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This material is based on research sponsored by NSF grants DMS-1952339, DMS-1952644, DMS- 2151235, DMS-2152762, DMS-2152717, DMS-2219956, DMS-2219904, and DMS-2208361 and DOE grant DE-SC0023490.
ETHICS STATEMENT
Our paper focuses on developing new theoretical understandings of the benefits of steerable features of different types in 3D equivariant graph neural networks. The paper is mainly theoretical and we do not any potential ethical issues in research.
REPRODUCIBILITY STATEMENT
We are dedicated to upholding the principles of reproducible research. pursuit of this commitment, we have taken several measures, For the theoretical proofs, we have included comprehensive derivations to enhance accessibility for wide range of readers. Additionally, we have submitted the code, along with detailed documentation, to facilitate the reproduction of the numerical results presented in our paper.
REFERENCES
[1] Uni Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. In International Conference Learning Representations, 2021. URL https
/openreviie ew. net. htte://ooommmm1110000000
[2] llyes Batatia, Dávid Péter Kovács, Gregor NC Simm, Christoph Ortner, and Gábor Csányi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. arXiv preprint arXiv:2206.0769) 2022.
[3] Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data-cfficient and accurate interatomic potentials. Nature communications, 13(1):
1-11, 2022.
[4] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling. Geometric and physical quantities improve E(3) equivariant message passing. In International Conference on Learning Representations, 2022.
[5] Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build E(n)-equivariane steerable CNNs. In International Conference on Learning Representations, 2022.
[6] Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Mor- gane Riviere, Kevin Tran, Javier Heras-Dominge Caleb Ho, Weihua Hu, al. Open catalyst 2020 (oc20) dataset and community challenges. Acs Catalysis, 11(10):005-6607) 2021.
[7] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, PMLR, 2016.
[8] Taco Cohen and Max Welling. Stecrable CNNs. In International Conference on Learning Representations, 2017.
[9] Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie-Yan Liu. Se (3) equivariant graph neural networks with complete local frames. In International Conference on Machine Learning, pp. 5583-5608. PMLR, 2022.
[10] Weitao Du, Yuanqi Du, Limei Wang, Dieqiao Feng, Guifeng Wang, Shuiwang Ji, Carla Gomes, and Zhi-Ming Ma. A new perspective on building efficient and expressive 3d equivariant graph neural networks. arXiv preprint arXiv:2304.047.. 2023.
10

---

Published as a conference paper at ICLR 2024
[11] Nadav Dym and Haggai Maron. On the universality of rotation equivariant point cloud networks. In International Conference on Learning Representations, 2021.
[12] Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to Lie groups on arbitrary continuous data. In International Conference Machine Learning, pp. 3165-3176. PMLR, 2020.
[13] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. -tans-raasoormmrr: 3D roto-translation equivariant attention networks. Advances in Neural Information Processing Systems, 1911970-991, 2020.
[14] Johannes Gasteiger, Shankari Giri, Johannes TT' Margraf, and Stephan Günnemann. Fast and uncertainty- directional message passing for non-equinee con-equilibrium molecules. arXiv preprint arXiv:2011.14115, 2020.
[15] Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03/23, 2020.
[16] Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In International conference on artificial neural networks, pp. 44-51. Springer, 2011.
[17] Thomas J Jech. The axiom of choice. Courier Corporation, 2008
[18] Erik Jenner and Maurice Weiler. Steerable partial differential operators for equivariant neural networks. arXiv preprint arXiv:2 10/63, 2021.
[19] Bowen Jing, Stephan Fismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Con- ference on Learning Representations, 2021. URL (ttt:ttts: /aattps://oenneviiee.. wwwpeewwwwww.wweeeeeeeeeee
d=1YL_JDVSx5J4
[20] Chaitanya K. Joshi, Cristian Bodnar, Simon Mathis, Taco Cohen, and Pictro Lio. On the expressive power of geometric graph neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 15330-15355. PMLR, 23-29 Jul 2023. URL https:// proceedings mlr. wwwpss/wwe...//p 2wwp22222ooosh22222 html
[21] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basse learning applied to document 1998. doi: 10.1199/5.72679.
recognition. Proceedings of the IEEE,
[22] Bo Li, Qili Wang, and Gim Hee Lee. Filtra: Rethinking steerable CNN by filter transform. In International Conference on Machine Learning. pp. 6515-6522. PMLR, 2021.
[23] Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3D atomistic graphs. arXiv preprint arXiv:2206.1 1990, 2022.
[24] Yi-Lun Liao, Brandon Wood, Abhishek Das, and Tess Smidt. Equiformerv2: Improved equivari- ant transformer for scaling higher-degree representations. arXiv preprint urXiv:2306. /2059, 2023.
[25] Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3D molecular graphs. In International Conference Learning Representations, 2021.
[26] Google LLC. Google colaboratory, 2023. URL
com
[27] Peter Olver. Lectures moving frames. 2009.
[28] Saro Passaro and C Lawrence Zitnick. Reducing SO(3) convolutions to SO(2) for efficient equivariant gnns. arXiv preprint arXiv:2302.03655, 2023.

---

Published as a conference paper at ICLR 2024
[29] Omri Puny, Matan Atzmon, Edward J. Smith, Ishan Misra, Aditya Grover, Heli and Yaron Lipman. Frame averaging for invariant and equivariant network design. International Conference Learning Representations, 2022.
[30] Ryszard Raczka and Asim Orhan Barut. Theory of group representations and applications. World Scientific Publishing Company, 1986.
[31] David W Romero, Erik J Bekkers, Jakub M Tomczak, and Mark Hoogendoorn. Wavelet networks: Scale equivariant learning from raw waveforms. arXiv preprint arXiv:2006.05259 2020.
[32] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. International conference on machine learning, pp.
PMLR, 2021.
[33] Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction tensorial properties and molecular spectra. International Conference on Machine Learning, pp. 9377-9388, PMLR, 2021.
[34] Kristof Schütt,, Huziel Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. SchNet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24), 2018.
[35] John Shawe-Taylor. Building symmetries into feedforward networks. 1989 First IEE International Conference on Artificial Neural Networks, (Conf, Publ. No. 313), pp. 158-162. IET, 1989.
[36] Tess E Smidt, Mario Geiger, and Benjamin Kurt Miller. Finding symmetry breaking order parameters with euclidean neural networks. Physical Review Research, (10: 2021.
[37] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-eque -equivaiion-uuiaiiiin neural networks for 3D point clouds. arXiv preprint arXiv:/802.00 02222.082.... 2018.
[38] Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via cur- vature. In International Conference on Learning Representations, 2022. URL https //openreview net forum?id==mm
[39] Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. Advances in Neural Information Processing Systems, 34:28848 -28663,, 2021.
[40] Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. ComENet: Towards complete and efficient message passing for 3D molecular graphs. Advances in Neural Information Processing Systems, 35:650-664, 2022.
[41] Maurice Weiler and Gabriele Cesa. General E())-equivariant steerable CNNs. Advances in Neural Information Processing Systems, 32, 2019.
[42] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D steerable CNNs: Learning rotationally equivariant features in volumetric data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
and R. Garnett (eds.), Advances in
[43] Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolutional networks-isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint urXiv:2 2021.
[44] Robin Winter, Marco Bertolini, Tuan Le, Frank Noé, and Djork-Amé Clevert. Unsupervised learning of group invariant and equivariant representations. Advances in Neural Information Processing Systems, 35:31942-31956, 2022.
12

---

Published as a conference paper at ICLR 2024
[45] Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel Brostow. Har- monic networks: Deep translation and rotation equivariance. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5028-5031 2017.
[46] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2024
A ADDITIONAL RELATED WORKS
Characterization of equivariant functions. The characterization of some equivariant functions has been explored in [39]. Specifically, Lemma is analogous Proposition 4 in [39], as can be interpreted as expressing f f(X) as a linear combination of the column vectors in p(gx) with A(X) (X) as the coefficients. However, a significant distinction lies in the fact that the number of vectors involved in our linear combination is the same as the dimension of f(X), not the number of column vectors in X. Additionally, the vectors in our linear combination truly form a basis, ensuring the uniqueness of the coefficients ). Meanwhile, the vectors in [9] 's linear combination do not necessarily form a basis and therefore cannot guarantee the uniqueness of the coefficients. Furthermore, our results cover cases of higher-type steerable features (l > 1), whereas [39] only studied = 0 and 1.
Equivariant moving frames and scalarization with local frames. The existing concepts, such as equivariant moving frames (27 [9] and scalarization with local frames [0] exhibit similarities with Lemma] [ However, we emphasize the distinctions as follows: While equivariant moving frames in (27] and gxx both assign X to a single element in G, the distinction lies in the fact that gx is well-defined even in cases where the action of 6 is not free, making it not necessarily equivariant. The scalarization proposed in [ [0] shares similarities with how we define A x
p(gx)=1, ((X). Nevertheless, it differs because the local frames used for scalarization are generated from two coordinates rather than all the coordinates X. In contrast, our gx requires consideration of the entire spatial embedding X.
Alignment. Lemma D indicates that steerable features correspond certain invariant features. However, implementing this in practice may pose challenges. The selection of the group element gxx and the representative c(β XX) is closely related to the notion of alignment in 46 44] In (46., the
Kabsch alignment algorithm is adopted to optimally find c X), while [44] utilizes an encoder to
learn it. Nevertheless, it is worth mentioning that X(X)= == f((((( X)) might not be well-behaved. The topological properties of the Lie group 6 = S3(3) 0(3) prevent us from constructing
continuous function + gx. resulting poor regularity of + gx and X(X).
Various notions of steerable features. Several related concepts surround steerable features in the literature, and it's beneficial to briefly recap them. The notion of a "steerable vector space" " 4 refers to vector space transformed by a group O through group representation of 8. "Stcerable feature pertain to assigning each point a steerable vector. On the other hand, "stcerable feature vectors" (37 refer the actual outputs of these steerable feature fields. In the context of this paper, we use the term "steerable feature" to specifically refer to the steerable vectors generated by an equivariant map of the spatial embedding. Our notion aligns closely with the steerable feature fields concept but emphasizes including all relevant spatial information as input. This distinction helps clarify the focus and scope of our work in geometric graph neural networks.
fields" " 42 41
B ADDITIONAL REMARKS AND DISCUSSION
Significance of faithful representations in learning steerable features. It's worth noting that 20) suggests the extensibility of their setup to higher-order tensors. However, our finding indicates that similar results for this extension may not hold if the geometric objects lack faithfulness. One can see that faithfulness is necessary to guarantee the injectivity in the proof of Proposition & and Theorem 2
To illustrate this issue, we can straightforwardly extend the argument from Theorem T Let's consider a hhhop equivariant GNN learning features on fon-ffithful representations, such as D or D ang Our aim is to demonstrate the existence of two 2-hop distinct geometric graphs for which the considered 1-hop equivariant GNN produces the same feature representation for any iteration.
While our discussion here somewhat informal, the details we provided can be used to establish solid proof casily. Notice that the non-faithfulnee of representations implies that features may remain unchanged even when the input undergoes a transformation by some group element g. This aligns with the challenge of using invariant features, which expected to remain unchanged under any transformation.
In fact, it is sufficient to construct two 2-hop distinct geometric graphs, where their 1-hop subgraphs are identical up to certain group actions (cither or the identity). This pair of graphs can be obtained by adjusting the pair illustrated in Fig. The analogous argument in the proof of Theorem will
14

---

Published as a conference paper at ICLR 2024
then demonstrate the desired results. The underlying rationale for this approach lies in the fact that aggregating features on non- non-faithful representations, due to their nature, fail to capture the locally changed geometry induced by the group element g.
We did not explicitly delve into this in the main paper, as existing models typically focus on learning features involving faithful representations (1 1). However, crucial determining factor for achieving the maximal expressive power in equivariant GNNs is whether the steerable features lie on faithful representations.
Limitations: universality and the lack of regularity. In the final part of Section 3.2 we refrain from asserting that using 7, = 1 is sufficient due the unaddressed consideration of universality. Here, we delve into the challenges of studying the intricate relationship between universality and internal representation, specifically the representation involved steerable features. Additionally, we present potential method to address them.
Let us formally articulate the questions pivotal to the investigation of the relationship between universality and internal representation.
Consider 6 G-steerable vector space V. and let X3 denote the set {X E F3xmm rank(X)) = =3). For specific architecture of geometric GNNs capable of learning features on any G-steerable vector space, we denote the family of all possible continuous (--quivariant functions from X3 to V that this architecture can parameterize (with sufficient number of layers) as Av. The family consisting of all continuous G-equivariant functions from X3 to V is denoted as Fv.
The first question to address is as follows: Question 1.
Can any continuous G-equivariani function in Fy be uniformly approximated compact sets by fanctions Ay?
Now, let W be another G-steerable vector space with the same dimension as V. Let fv Xx3- - V be an arbitrary continuous (--qqivariaat function in Fv. According to Corollary 3 there exists 8-cquivariant function fw : X3 + W such that for any X E X3, we have fv pv (gx) and fw) pw pw(gx' A(X) ) for the same G-invariant function À where pv, PW are the group representation on V and W, resp. However, fw is not necessarily continuous, i.e. fw & Fw. implying that it may not be approximated by functions in Aw- In particular, the topological properties of the Lie group T = (3() or O(3) prevent us from constructing a continuous function X + gx. resulting in poor regularity of X + gx and X(X). Consequently, the continuity of sw cannot be guaranteed.
As result, present the following question:
Question 2. Let p denote the Lebesgue measure on 3xmm For any 6 > O and any compact set K C. Xs. does there exist continuous G-equivariant function fwy %%3 + W and measurable subset E C K with p(K) E) such that fi (X) = -ww (X) for all X E ES?
This implies that for any 0 and any compact set K C X3, there exists function f'v E Fw that can be uniformly approximated on compact sets by functions in Aw. Additionally, its corresponding invariant function X' X' = Pw (gx fly (X) is identical to X(X), the corresponding invariant function of fv (X). on some
Suppose obtain affirmative answers to both Question [ and 2
measurable set E. with [u('' \ F?) <
By restricting fv to a function in Av. we observe that the given architecture is able to learn a function in Aw such that its corresponding invariant function is identical to the corresponding invariant function of fv any compact set except an e-small measurable subset. In essence, the architecture can successfully learn approximated corresponding invariant functions on steerable vector spaces of the same dimension.
While the response to Question dependd on the architectures, the answer to Question 2 is inde- pendent of them. In the following context, propose potential approach to address Question 2 by constructing X + gx with better regularityf] More precisely, we aim to remedy the reg-
ularity issue of X + gx by extending the domain X3 to its covering space Xz. We proceed similarly
29
construct the covering space for the set of all generie spatial embeddings in
Thiis will impact the validity of Lemma 1 and Corollary
15

---

Published as a conference paper at ICLR 2024
Xz. Namely, given the spatial embedding X Xz. Principle Component Analysis (PCA) sug- gests that there are three vectors V1. U2, Vy E R uniquely determined up to sign corresponding to three distinct singular values 01 > 02 > Jg. As result, one obtains the equivariant moving frame X + F (X) == {arv 0222 a3Wyy E E (3)lo; E [1] as a set-valued function. In our situation, we take alternative viewpoint. Notice that any generic X X3, there's small enough neighborhood Ux of X such that any choice of sign (a1, a2 03) gives a homeo- morphism X + (x, [aie] 02222 03v3]) from Ux X3 to Ux 0x.00 may check that the neighborhood compatibility allows us to patch Ux Ux,an,a2.00
X3 O (3). One
into a topologi-
cal space X3. Intuitively speaking, we obtain X3 via locally embedding multiple copies of small neighborhoods into X3 x O (3). Each neighborhood Ux corresponds to 2 copies of themselves in X3 O (3). It becomes clear that X3 + X3 O (3) O (3) defines a continuous equivariant function X == (X, + 8 == g and so does X + Pw (ox). Now, for a given fw with the corresponding invariant function X, we lift À to an invariant function X + X(X) = À T (x) on
the covering space. Indeed, X ++ PW (0x) Ā produces a G-equivariant continuous function, but the function is defined on the covering space Xg. To produce function on X3. one need to modify on an e-small set and also À so that the modified invariant function X remains continuous and fw (X) == Pw (9x) X (XX actually defines function on X3. In particular, fwv and fw coincide on a given compact set except an c-small measurable subset.
Although addressing Question I and 2 may suggest that learning on steerable vector spaces of the same dimension demonstrates consistent expressive power, we cannot deny the possibility that certain choices of representation could benefit the model, allowing it to approximate the target function with fewer layers. Additionally, our exploration in Section 3.T indicates that learning on non-faithful representations might restrict the model's 's expressive power and, consequently, its universality. Nevertheless, we have chosen to reserve the exploration of Question and 2 for future work.
While our theory reveals this limitation, it is noteworthy that empirical results surprisingly align with the claim that learning lower types of steerable features can perform effectively as learning higher types of steerable features.
C MISSING PROOF
Lemma 1. Let V be d-ddimensional G-steerable vector space with the assigned group representation
p: GL(V). If f 123mm + V is C-equivariant, then there exists unique G-invariant functiont
A: 1max Vindd s.t. f(X)= p(gx) )xX). where Vu denotes the ID trivial representation of c
In particular, the following map is well-definee
R. + V (--quivariann) + (A: 33mm + 1ood
X (6-invariant).
(3)
Proof of Lemmal
7 The equivariance follows that,
f(x) [((x c(B X)) p(ax) (()
suffices to show that f(((B. x)) induces G-invariant map A R3xm -> Vidd First, sending X to [c((5. X)) defines a map from R3xm to a d-dimensional vector space. Since 6 (g- X) = 6- X for any E 6, we see that cBB X) is invariant, and hence f [c((S X)) is invariant to the group action. Therefore, A(X)= == fc((O 6 x)) is G-iivariant. One can check that the uniqueness of this function directly stems from the linear independence of the columns of p(gx). Alternatively, the invertibility of p(gx) can be utilized to conclude that X(X) = p(gx) -1 f(X).
For the ease of reading, restate Theorem Tbelow
Theorem 1. If and G2 are two k-hop identical graphs, then any iteration of k-hop invariant GNNs will get the same output from these two graphs. That is, there is a graph isomorphism b such that
X (1+1) = X b()
(t+1) for any i, even though G1 and G2 may not be identical up to group action.
Foor G x3) it corresponds to the type-O steerable vector space we defined in Section For simplicity, we employ the same notation here.
16

---

Published as a conference paper at ICLR 2024
Proof. For k-hop invariant GNNs, the propagation of the (corresponding) invariant features in equation 6 can be expressed as follows:
A ((+1) = UPD
(1)) AGG (H 25e ((0)
1 I N)(1
(10)
UPP 1 AGG (( 2 my |j M
where we use p(g) - I for any g E 6 and the invariance property of AGG.
Let 9 = EV,, , x) 92 = (V2, E2, F9;, xx2 ) be two k-hop identical geometric graphs By definition, there exists graph isomorphism b such that for any node i V1, can find group element gi E 6 satisfies s [(() x; Bi Db(()
G2 and f f66 Bi
Go
whenever j A, b(j)
Then, utilizing the invariance of AGG once more, we observe that:
2 UPD f AGG (1ff ffth zi lj N B))
UPD
fo b(j)
AGG
(H (H fiy Di z 16() No 11
fb0) AGG (M f (0 b()) E (1 })
UPD
(1)
Hence, it follows that X = b())
for any Following this step inductively, we can demonstrate that
X (++1)_ = 6()
(++1) for any node i.
For the ease of reading, restate Theorem below:
Theorem 4. Consider two geometric GNNs learning features steerable vector spuces V and W of the same dimension, resp. Denote their update and aggregation functions at iteration t as UPD UPD
a collection W and AGG AGG Then for any collection {(UPD) AGG Tr. there exists (t) (GG((W } such that for any fully connected graph, they learn the same corresponding invariant features X(r) for any iteration t 2 0 each node i.
Let us first provide more details about what we are going to show. We consider the following two geometric GNNs:
f((++) = UPD. (F AGG, (UJ
1/0 my 1 (NND)
(11)
where represents the steerable vector space V or W and f! denotes the i-th node feature learn at iteration f. Let PV and PW denote the group representations on and W, resp. Remark that the corresponding invariant features are given by:
1 = p. 1
(12)
where = gx is the group element associated with input spatial embedding X.
We aim to show that for any collection ( AGG(()) }}+ of 6-equivariant functions, there exists
a collection ((UPD() AGG(()) }. of &-equivariant functions such that x - for any / 2 0 and i.
Proof. We proceed with this proof by induction. First, (1) x(() holds for 0 since
5.00) f.:
"One also directly use equation
17

---

Published as a conference paper at ICLR 2024
where Si denotes the input node feature of node i. Suppose X Liw holds for any node
iteration t. At iteration t 1, we have
(++1) A L
=vv(g) VV
-Pv(l)) UPDV ( AGGV (1) 1 E.)))
= -UDDv (ov(0)) (,, AGG Se:: Pg()) 5 0
aiy 1j 0'
((D))
=UPDv (A AGGv (E 1. 8/
2/ 1 Ni
Consider the following construction of (UPD W AGG
UPDw ( (GGGw(1.))) J.W: Dij 1 E (N) = (w( UPDv (pw(a) S AGGv (p) Pw S T. 'as 1j ENBB)
(13)
More precisely, to guarantee G-equivariance, we define UPD and AGG to be: (UPD@) W AGG
AGGW (IS f/w mn 11 IJENiD)
= (pw(() (s) - (... pw (0) - p(t)
J,W. 8. 8 lj E ENB)).
(14)
UPDw (f (f(W, (m; 9)) = pw (g) UPDy (ow(g) (p) f -ww.PM PW ( .m:)
where m, denotes the first component of AGG )'s output. The 6-equivariance follows from the sense Remark
It is well-defined since connected graphs. Indeed, since we have Σx 23 = 0 (from eliminating the effects of translations), then Ljd; Bij ma, Ljzi Jtj (m + 1)æ; where is the number of nodes. Therefore,
construct g 9x from Fij by using the assumption of fully
m+1 Exes Dik = x; and m+1 Exes Iik)
Tij = 2j for any j + i. These coordinates then
determine X and hence determine gx.
Then using the inductive assumption (1)) i,W and !!! p.(0) (1 we obtain
(1)
((+1) A W.
PW (g) f,,
(()
PW (g) -1 UPD (1 AGGW (1 (tt)
T.y 1oyljjei (jEND))
UPDy (g) (g) l(() J,,,, AGG, (g) -1 fiw k.W.Pw(6 (0) f Jj g' 1j e NBD)
-UPDy (2 AGGv 111)
A... g1 1j N..i))
-UPDy (2 (A,, AGGv (1 v.V. Fxx 15 E (NH)
=X((t))
This shows the construction satisfies the desired result.
Proofs and Additional Details for Section 3.2 All the theoretical results in Section 3.2 stem from the following theorem:
Theorem 3. Let X, denote the set {X E 3 rank(XX) = r). Then we have a one-to-one
correspondence between -eniiiaaaa functions and ((3)-invarian functions:
{f : X3 + Viana f -equvvvrrv)) = {} : X3 + V@21+1
-(v)--iiariian)) (7)
{f Xg + Viand f -eueevrauuv = {A X, - V
X (033)innariant))
{ X, + Viand ʃ (03) -quiivarra)) = (A X, + 1/
|A: ((3)-invariaml) )
{f X Viand f (03))-qquvvrrimn { Xx {0) (0)).
18

---

Published as a conference paper at ICLR 2024
[f:X: + Vi.ug
f - equivariant } = {A X3 0 V022-1
| (3)) (8)
[f:X,
Vi
f (0((3))qquvarrian 2 (X X2 V
|A (33)-invrriant)
(f:X1 - Viang (3) -equivarian = (ʃ : X1 + +()),
:0 Viang
ʃ 0(3) -equivarrr) = ( X. [ + (0))
Theorem 5. Similarly, we have a one-to-one correspondence for -quivvranuv and -invaranna::
[f: f SO(3) equivarian) {A : X3 V0 1/22+1
À: SO(3) iivariant)
(15)
(f:X2 + Vi SO(3) -qquivaraaa {A: X2 + 1/241
À (0(3) -invariant),
[f:X; - V f -qquivaraa)) = (A: X1 4 1/ À (30(3)-invarian))
X, Vi V -qquivarraa {f X {0] {0]}
Consider @ as either ((3) or SO(3). Given the reducibility of G-stecrable vector spaces, we can decompose any (6-stecrable vector space V into a direct sum of steerable vector spaces of different types. Then equation 77) equation (8) in Theorem and equation (15) in Theorem S imply the following result:
Corollary 2. Let X3 denote the set {X E 13x rran(()) = 3}. Then for any G-steerable vector space of dimension d, denoted as V, we have a one-to-one correspondence:
: X33 > V f (-equivariant) 2 {x: X3 Viddd X (6-invriint).
where the map between these two spaces is induced by the map defined in equation
Corollary 3. Let V and W be two G-steerable vector spaces of dimension d. Then for any 8- equivariant function fy : Xx - V, there is a (--eqivvariant function fw Xx + W such that for any X Xz. have fx pv (gx )X()) and fw (X) PW (gx))(XX)) for the same 8- D-invariant function X where Pv: Pw are the group representation on and W, resp.
Proof. According to Lemma 1 there is a G-invariant function A X3 + veed such that (v pv pv (0x))(X) for any X X3. Then applying Theorem to W, there exist G-equivariant function fw X3 + W such that fw (X) Pw (w(gx)A(X) for any X X3. which shows the desired result.
The following two theorems follow from Theorem 3addd by focusing given spatial embedding. Corollary 1. Let X E RR be a spatial embedding. We have the following relation between 0(3) -tte-sttee features and invariant features:
1. If rank(X) = =3, there a bijection between steerable features in Viand and (21 + 1)- dimensional invariant features, well as a bijection between steerable features in Vi,ng and (2l + 1)-dimensional features
If rank(X) = 2, there is a bijection between steerable features Viand and (I 1)- dimensional invariant features and bijection between steerable features in Vi,aug and I-dimensional invariant features.
If frank( X) = 1, there a bijection between steerable features in Vi,and and 1-dimensional invariant features, while there is no non-trivial steerable feature lying in Vi
4. There exist only trivial steerable feature O and trivial invariant feature 0 if rark(X) = O.
Corollary 4. Let X E 1R3xm be a spatial embedding. We have the following relation between -steerrbee features and invariant features:
1. If (frnn((X) = or 3, there is a bijection between type-l steerable features and (2l + 1)- dimensional invariant featurs..
2. rankk)( there is a bijection between type-l steerable features and 1-dimensional invariant features.
3. ffrak((X) = 0, there exist only trivial steerable feature 0 and trivial invariant feature O.
Remark 5. It is worth mentioning that Corollary I and Asuggests that type-l steerable features
(1 0) is more sensitive the rank of the spatial embedding X than invariant features (1 = 0).
19

---

Published as a conference paper at ICLR 2024
Before proving Theorem Jand 5 we introduce the following lemma:
Lemma 2. For any fixed 1 2 O, any -equiivaaaa function f R2xm + Vi has unique
decomposition: f(l) p1 + faig into a sum of -33))- functions find 3axm + Vi,and
and fang IR3xm + Vi, where
(X) (X) (-1)² ( X) and Sun (x)
( - (-1) 100 (-x)
Proof of Lemma
Upon routine verification, it becomes apparent that:
Jind x)= = (-1)- (1 (X) and
(())
(-X) = (-1) faug (X). aug
Since both (X) and f( are SO(3) eqqiivariu, so are both Fnd and f Combined with the sign-change property, we conclude that f and Sang
Teuiavaauvaai establish
uniqueness, let's consider any decomposition of f()) Iind + Hang into the sum of a -euueqvvvarr
function (t) R3xm Vínind and 0(3) -(3-vvrauaaa function h R3xm 7 Vl,aug. Now, observe that the functions on both sides of the following equation,
Ind Ind
(()) ang
(1)
are -qquiivarr, which, due to sign-change property:
(-1)- (ond (X) Aidd (x)) - -ind
(1)
(-X) Find (-X)
(()) = Jang
(-X) - 111 (-X) = =(-1)+1 1 (falg(())
(X) - -/ (X)
(-1) Find (X) Sind (x)
can only be O. In other words, Innd = (1) and hang faug.
(1)
Proof of Theorem land 5
We can observe that Theorem imply Theorem 5y applying Lemma
Therefore, it remains to prove all the correspondences for (().. The proof strategy here similar to that of Lemma with additional care on the design of the choice function e R pm / + 2 We recall the following decomposition:
T 3xm_
Kr, where x= fx
3 rank (X) r).
Since 6 = -3) action does not affect the rank of spatial embedding X, the decomposition is preserved under quotient:
R 3xm /
(x./3).
This allows to define the choice function in ways that respect the geometry arising from different rank X conditions:
(r 3)
We shall that the argument here is special case of Remark applied the subset
X3 C paxm, Invoking axiom of choice, we obtain a choice function:
Xa/ / X3 with C3 (6 X) E 6 X.
X.
Notice that in the full-rank setting, all stabilizers are the same; in particular, they trivial:
G = 5 ( x) G3 = (T). VX E X3.
(r = 2) Here is where the novelty of our argument comes in. We observe that
x- plane
(6 x)
[ x0))
#0, VX E %2
20

---

Published as a conference paper at ICLR 2024
In other words, can take the following special choice function:
2 X X2 n (IR²
{0})* m with 02 (8
6 X.
Notably, all the choices here share the same stabilizer:
1 0
(85 x) = 6: = R ==
VX E X2.
0 -1
This will be useful later on.
(r 1) We modify the argument in the previous (r = 2) case to reflect the rank X geometry and apply to the (r 1) case. Again, start with the following observation:
z-axis
(6 X) (10)² x #0, v
We may, thus, take the following special choice function:
/ x n [0) x xR)
with (8 X) EB. X.
Similarly, all the choices here share the same stabilizer:
R 0 K (. 1 { 1 E 6
( v
(r (r=0) 0) This is the trivial case Xo {0). and thus the choice function is defined uniquely:
co % (6 0) - {0)
And obviously, (c = 6 = 6.
For convenience, we define the total choice function:
1
r=0
Via axiom of choice again, we can find a 6 valued function:
9() R 3xm + C such that X = gx c c(6. X).
As direct consequence, given 033)-equivariant f : X - Vi,., we have the following formula:
f = DÝ (gx) (c, (0. X)). VX E X,.
By design, the formula suggests a way to relate Vi,+ steerable feature (21 + 1) dimensional invariant features:
(X) = f (6 X)) D (ax))1 f(X,, VX E X,.
f(x))
Yet, upon further inspection, A (X) has some hidden structure relating to the stabilizer 6,. To be precise, the ((3) -quivaarrve of f implies that
D. (g) x (g- o, (6. X)) = f (cr (6 x)) = -XX), Ag 6, In other words, x (X) could lie in proper subspace.
=
(X) E n ker (D', (g) -1) = Vi...
gee,
We first go through the two trivial cases:
V3,a = ker (0) =
F F.em: em:
-ISmáI
Vo,- = n ker (D., (g) (0).
21

---

Published as a conference paper at ICLR 2024
For 2, 22,* is exactly an eigenspace of D (R.):
22, kcr (D', (Rz) I)
A direct calculation shows the following:
D'. (R.)
(T,0,00 [... (...(-1)mttttouue em e -ssmsi
Therefore, we have:
22
F Em
1- lang
mo82
In particular, we have dim V2,ind / + 1, dim V2 = l, and V3,+ = 12, + 22 To deal with the (r 1) case, we first notice that 11 can be generated by the following:
1
0
a a 0
Rg =
-
and R. = sin
cos a
a E 2m)
0 1
0 0
This simplifies our problem,
V1,- = ker (R,y) I)
ker (D', (R,) - 1).
Direct calculation gives:
D., (Ra) D' (a, 0, [.... iam em .. ".JJ-1mm1" -15m51
Therefore, we must have:
n ker (D', (Ra) - -)) = =F Cy- ee.
It remains to check whether eu lies in the eigenspace ker (Di (Ry) I), not. We perform the following calculation:
D'. (R2) en (-1)(tlmma))) D' (0, -) en (-1) lamx()))) en- We may now conclude that Vindnd = F eo and V Vi,ang (0). Combining what we have, we obtain the following picture:
ee
(X) (ex) f x)) XeX, A (X) E Vr.::
with each space being characterized as follows:
V 3,ddd Va,ang
F. Cm
2V
-15m51
V 2,ndd
F Cm
~VB1-1
-l
m mod2
2
F Cm
~V01
-ISmSI m of mod2
V 1inind
F.
~Vo
V V,ag Vo,ind, Vo,ang
(0)-
We now argue that the following space the invariant features
Ar, (A:
Vr -i)--nna))
22

---

Published as a conference paper at ICLR 2024
has 1-to-1 correspondence the following space of steerable features
Fr, = ( : X, + Vf.-If=
and the correspondence is exactly given by the following formula:
(x) D. (gx) (axx)1 (X).
Indeed, for f E Pr.e. the invertibility of D'd (gx) defines a unique X. Moreover, we've established that such À in3)-invariant and has its images contained in V,,, and thus, À E Ar, On the other hand, given À E Ay, and any X Xp, we have the following,
D'. (ox) (X) DỘ (gx) py.. (X),
where pv.,, is the matrix that represent the orthogonal projection onto V, Since
8..X.Ge @ g.g.X
g X
=0.X
8g g ox E AX E X,,
9 9 (8. X)
obtain
(D's (g 9 9x -1) pV.. = 0,
as direct consequence. After some algebraic manipulation, we derive the following equation:
D (0g x) pv..- = Dt. (g) D¹, (gx) .pv.... VX E X
In other words, the formula
((X) + (X) = DC (0x) (X)
XeX. D (gx) pv,. (X),
defines an -(3)-qquivariant function E Fr... With that, we establish the 1-to-1 correspondence.
D GEOMETRIC WEISFEILER-LEMMAN TEST (GWL)
In this section, we will provide an overview of the geometric Weisfeiler- test (GWL) and its invariant version (IGWL), as originally introduced in (20): however, for simplicity, exclude input vector features. Additionally, we will extend IGWL to the k-hop setting and demonstrate that k-hop IGWL cannot distinguish any two k-hop identical graphs. As corollary, remains strictly more powerful than k-hop IGWL. We will also prove the conditions under which k-hop invariant GNNs achieve the same expressive power as k-hop IGWL.
will establish that GWL
Before delving into the details, we introduce a graph-level readout, a G-invarriat multiset function, at the final layer of geometric GNNs. This function maps multisets { V} to invariant fcatures vidd' for some d' 0. This inclusion aids in understanding when geometric GNNs attain the maximum expressive power characterized by the GWL test.
GWL Consider geometric graph G (V, F, X). Let denote a countable space of colors.
Initially, we assign scalar color c; HASH based on their input features f;:
(0) E C to each node E V through an injective mapping function
(0) HASH(f:).
(16)
Additionally, assign extra geometric object 9; (0) to each node E V V/ gi (0) (0)
Then we define the inductive step. Assuming we have all the colors c; (t-1) and geometric objects
g (t-1) at iteration t for each node we aggregate the geometric information from its neighborhood A into new geometric object:
g = ((c) (t-1) -95 (1-1)),
H(ch .9 (1-1) ; 1j E ENN.)
(17)
23

---

Published as a conference paper at ICLR 2024
Notice that the geometric objects acted on by the group B: (t) 94 = (c) (1- 0 Di ") A(cl)- , 9/ 10-1) g (.:)))) 1 Evi)
(18)
One can check that the process of creating 9: is injective and G-equivariant. We then assign the color of iteration through G-invariant and (--orbit injective map, denoted as I-HASH(()
I-HASH (0: (1)
(19)
In other words, I-HASH (g) H-HAHHHH if and only g = g. g for some g E 6.
The iteration terminates when the colors induce the same partitions of nodes. Then, given two attributed graphs 91 &1) = E2), if there is some iteration s.t. ((e l E Vi *
(c,0) | E V2 then GWL determines that these two graphs are not geometrically isomorphic.
Invariant GWL. For the invariant version of GWL, we do not consider the equivariant geometric object. Thus, the iteration becomes:
I-HAHH (Q--1) A(c) ((c))-)) mx)) | E E
(20)
where the initialization remains the same c; (f)H(f.).
k-hop IGWL, To extend the 1-hop aggregation in IGWL to k-hop aggregation, we replace the 1-hop neighborhood Mr with the k-hop neighborhood Ar(!)
(t) = I-HASH ((tT)) Alce a.y)) 1j N 9).
(21)
Now we extend the results in (20 for IGWL to k-hop IGWL:
Proposition 1. k-hop IGWL can distinguish k-hop distinct geometric graphs with just one iteration, but it cannot differentiate h-h identical geometric graphs matter how many iterations used.
Proof of Proposition L Let G1 = (V,,E,, FC, X1),, = (V2, E2, F X&2 ) be two k-hop identical geometric graphs. By definition, there exists a graph isomorphism b such that for any node: V1, we can find a group element gi C satisfies f f x = gi =b(() and Jb())
G2 2 = di D0) whenever j E ((()) This implies for any i, we have c7 b(1) and 1 (b))
G2
are identical up to group action. Based the definition of k-hop IGWL iterations defined in 20 we then conclude that c/ = Cb())
hence the multisets [(c)0) x ,, and ((c/)) ((b))))))))))))))"""" j _o B
(1) for any i. By induction, it follows that any number of k-hop IGWL iterations cannot
distinguish G1 and G2-
Now, let 91 (V1, E1, Foi xx)) (V2, FS, X S2) be k-hop distinct geometric graphs By definition, for any graph isomorphism b, there is node E V1 such that the corresponding k-hop subgraphs forming by AA(10) U (i) and Ax (b((i) are distinct under the group action. This implies
(1) o * ((i)) Since is arbitrary, we conclude that (ef1) 1 E VI Hlo) (2) | E V2 and hence 1
iteration of k-hop IGWL is sufficient to distinguish Gi and G2-
We recap two key results regarding the expressive power GWL. from 20]:
Proposition GWL can distinguish any two k-hop distinct geometric graphs, and l iteration is sufficient.
Proposition 3. Up to iteration, GWL cannot distinguish any two k-hop identical geometric graphs.
Applying Proposition 2 and 3 along with Proposition we derive the following theorem. Notably, the case for k = corresponds to Theorem 8 in (20]:
Theorem 6. GWL is strictly more powerful than k-hop IGWL any k. while they have the same expressive power when applied fully connected graphs.
24

---

Published as a conference paper at ICLR 2024
Proof. We have demonstrated that k-hop IGWL can distinguish k-hop distinct geometric graphs, which are distinguishable by GWL using & iterations.
However, k-hop IGWL cannot distinguish k-hop identical geometric graphs. As illustrated in Figure !1 we provide an example where k-hop IGWL fails to distinguish such graphs, whereas GWL succeeds.
L
Proposition 4. Any pair of geometric graphs is distinguished by k-hop invariant GNNs is also distinguished by k-hop IGWL.
Proof. The proof is the same as the proof of Theorem 24 in (20 by replacing the 1-hop neighborhoods with h-hop neighborhoods.
Proposition 5. k-hop invariant GNNs have the same expressive power as k-hop IGWL if the following conditions hold: (1) The aggregate function AGG and update function UPD are G-orbit injective and O--invriiat multisel functions (2) The graph-level readout function is an injective multiset function.
Proof. The proof is the same as the proof of Proposition 25 in 20 by replacing the 1-hop neighbor- hoods with k-hop neighborhoods.
Significance of faithful representations in learning steerable features. It has been demonstrated in [20] that the expressive power of equivariant GNNs, when learning steerable features up to type bounded by the GWL test, with equality under certain assumptions on the injectivity of UPD, AGG, and the graph-level readout function. In particular, they prove the following results:
Proposition 6. Any pair of geometric graphs is distinguished by 1-hop equivariant GNNs is also distinguished by GWL.
Proposition 7. 1-hop equivariant GNNs learning steerable features up to type have the same expressive power as GWL if the following conditions hold: (1) The aggregate function AGG and update function UPD are 6-orbit injective and 6 B-equivariant multiset functions (2) The graph-level readout function is G-orbit injective and 6- invariant multiset function..
Remark 6. As we merge scalar and vector features into steerable featurrss fú, certain conditions in Proposition 7 have been udjusted accommodate our setup.
In this work, we extend Proposition to cover cases where equivariant GNNs learn steerable features on any representations. Notably, we highlight that achieving equality in this extended scenario necessitates an additional condition: the steerable features must lie faithful representations.
Proposition 8. Consider 1-hop equivariant GNNs learning features steerable vector space V where the aggregate function AGG learns features on steerable vector space W. Then these equivariant GNNs have the same expressive power GWL if the following conditions hold: (I) The aggregate function AGG and update function UPD are G-orbit injective and B- --equivariant multiset functions (2) The graph--evel readout function is a G-orbit injective and G-invariant multiset function. (3) V. W are faithful representations.
Proof. We employ similar strategy to the proof presented in Proposition 7 in (20 to establish our result. this proof, we use [...] to denote the equivalence class generated by the actions of G. Then, any G-orbit injective function can be expressed as an injective function over the equivalence classes [...].
The GWL test updates the node color c;" and geometric object 9; as:
gi (1)
((c
(1-1) "i-") xxs) ((("-"".)""")
e 40= h, ((2
where he is G-invariant and G-orbit injective map and hy is G-equivariant and injective operation, such as expanding the geometric multiset by copying, as shown in equation
Consider an equivariant GNN that satisfies the conditions outlined in the theorem statement. We will show by induction that at any iteration there always exist -e-quivariant and injective functions (!))
25

---

Published as a conference paper at ICLR 2024
such that f. = (0()) (s!") for any Let denote the graph-level readout function. Since h maps
different multisets of node features to unique invariant features, f! = (p(1)
[i
(()
implies that there
exists an injective function spe such that / (1 { l v()))) = ( (fl (9)
i E v)})) where V
denotes the set of nodes.
First, we observe that f! = ( (0())
holds for because 9 = c (0) HASH(f:) for all
i V. Now, suppose this holds for iteration t. At iteration + 1, substituting f! with (0())
g (t)
implies that.
f (++1) = UPD (1 AGG((A J,"... .5," .aylis (sENNH)
(22)
(1))) (1 .AGG (H(000) (1!") p(tt) (0,0) Fij 1j exi)))
= UPD
Consider the function ((c,"
(1)
(4) 9i (1)) = (()" Suppose ((c (1)
9 ( o(c ,g ) The injectivity
of (t)) then implies that gi = 0301 As h2 G-invariant and 6-oobit injective, we deduce that
c "g and therefore, Φ is injective. The eeeuiiaaren of & is inherited directly from uo()) By
substituting (
(9. () with p(c)). "!!"),, we obtain
f (1+1) = UPD (o (c. (0)) AGG
(c(".)))) ay 1 liE): (23)
It remains to show that there exists injective function ø' such that:
f((+1) ø' ((c)..9)) (e)"9:))) .. {(c".s))) as E &&Nill)
Indeed, we can define ((+1)) = Ø o hnj and then we have
1 ((+11) = o o (4 (1) 1
mi 1j EN
= (t(() hh ((49 19: . (0,0
(0)
(9
Dg 15 Ni
(24)
(9 (111).
Now consider the following construction of d',
d' (((c)) (e") (0)))((()
UUP (c( (0) (0) AGG (1 ( (G 00)) (0) mi 1 E NNi
To demonstrate o' is injective, we will first show that UPD and AGG are injective. This step requires the faithfulness of V and W. More precisely, the G-orbit injectivity and (-equivariance of UPD and AGG with the faithfulness of V and W imply that they are injective. Suppose UPD(f,, m.) = UPD(f), m;) where m, denotes the output of AGG. Due to the 6-orbit injectivity, we have mf = [fy. my]. Therefore, (fi, m.) = g my)) = (g fj.g m,,) for some E G. Applying the (eequivariance, obtain
UPD(f,, my) UPD(f:, m.)
(25)
UPD(g f..g. my
(26)
g UPD(f,,
(27)
Faithfulness implies g is the identity, proving the injectivity of UPD. Similarly, we can prove the injectivity of AGG. Now applying the injectivity of UPD and AGG, that:
9 gi
(1)
(1)
Φ (0
.9, (.)))
(28)
19 (0)) . ( { ( c
(0)
Tihh k NiIl
11
0 c ,9 ,,,
.
10 (() 2i k Nr
(29)
26

---

Published as a conference paper at ICLR 2024
The injectivity of then implies that
(d".9.") -c(":))))) (<2 (0) 9 mik A EENi {{
(30)
&k 1
) mik l E
(31)
This shows the injectivity of o' and thus completes the proof.
Theorem Consider 1-hop equivariant GNNs learning features on steerable vector space where the aggregate functiin AGG learns features steerable vector space W. Suppose V and W are faithful representations, and UPD and AGG satisfy certain assumptions on the injectivity outlined in Proposition 8 in the appendix. Then with k iterations, these equivariant GNNs learn different } on two k-hop distinct geometric graphs.
multisets of node features {
Proof. This can be derived from the proof of Proposition and
E AN OVERVIEW OF EXISTING INVARIANT AND EQUIVARIANT GNNS
In this section, discuss the invariant and equivariant architectures considered in the work. However our framework covers models beyond those mentioned here, for instance it includes the models discussed in Joshi et al. [20] It's worth noting that this framework is not confined to using 2-body aggregation; it can also employ multi-body aggregation methods.
E.1 --INAARIANT GNNS
The SchNet model 34 is 2-body G-invariant architecture which propagates type-0 features in 1-hop using relative distances:
SchNet
f ((+1) UPD (1 AGG(ES 5 | e,(
DimeNet++ The DimeNet++ model 14 is 3-body 6-invariant architecture which propagates type-0 features in 1-hop using relative distances and angles.
((+1)) = UPD (1 AGG ./ I| asy as 1i,k E E k f())
ComENet The ComENet model 40 is a 4-body G-invariant architecture which propagates type-0 features in 1-hop with complete edge attributes ([|=; |I.. Ozj, dij, Tzy):
((+)))) UPD ( AGG((S)) | ll. Ozg, dijj Tiy | Ag B)),
where fiys Oif, and Try are computed quadruplet (within a 2-hop neighborhood).
E.2 G-EQUIVARIANT GNNS
EGNN and GVP The EGNN 32 and GVP 19 models 3-body (G-qquivariant architecture which propagate type-0 and type-1 features in 1-hop using relative distances.
f (((1)) = UPD ( AGG(ES 10,0 || l E EN(
MACE and eSCN The MACE 2 and eSCN (8 models are 2-body (-equivariant architecture which propagate up to type-L features 1-hop using spherical harmonics.
(1-1) UPD (f AGG( s/ (()))) ||a lj (AP)))
Equiformer V2 The Equiformer V2 model [24] is 6 (e-qqiivaiaant architecture, similar to the one mentioned above. It propagates up to type-L features in 1-hop by fully utilizing all neighbors within the neighborhood to create the attention weights.
27

---

Published as a conference paper at ICLR 2024
F ADDITIONAL EXPERIMENTAL DETAILS
F.1
ADDITIONAL RESULTS FOR k-CHAINS
Layers
2
3
5
L =
SchNet
50.0 0.0
50.0
50.0 0.0
50.0
50.0 + 0.0
DimeNet++
50.0
5.0
50.0 0.0
5.0
50.0
SphereNet
50.0 0.0
50.0
50.0 0.0
50.0
50.0 0.0
ComENet
50.0 0.0
50.0
53.0 + 5.1
49.5 1.5
49.5 = 2.7
EquiformerV2
55.0 8.1
61.0 = 12.2
68.0 8.7 8.7
80.0 7.7
75.0 = 11.2
EGNN
50.0 0.0
0.0
95.0 15.0
100.0++.000
90.0 + 20.0
GVP
0.0
93.0 15.5
90.5 19.0
9.0
100.0 0.0
ClofNet
0.0
1.0
95.0 15.0
95.0
95.0 15.0
LEFTNet
0.0
0.0
50.0
5.0
50.0
MACE
50.0
1.0
100.0 1 0.0
1.0
100.0
eSCN
58.0
62.0 1 9.8
5.0
53.0 6.4
EquiformerV2
53.0 + 10.0
52.0 7.5
100.0 0.0
100.0 0.0
100.0 + 0.0
L
MACE
50.0 0.0
100.0 + 0.0
100.0 = 0.0
100.0 + 0.0
100.0 = 0.0
cSCN
57.0 10.0
54.0 8.0
60.0 10.0
55.0 6.7
60.0 = 7.7
EquiformerV2
53.0 = 10.0
16.0 + 9.2
45.0 1 6.7
75.0 6.7
77.0 + 7.8
Table 5:
Test accuracy for the k-hop chain dataset with 4. Cell shading is based standard deviations above or below the expected value. Unit:%.
5 the 10-fold cross validation mean test accuracy and standard deviation for chain length and varying model depth. The values are shaded based on the expected value from Section We remark that EGNN, ClofNet, LEFTNet [10], eSCN and EquiformerV2 V2 do not perform expected on this particular task. As discussed 220 the under performance of EGNN and ClofNet is likely due to oversquashing [.. For LEFTNet, the underlying cause is attributed to its invariant design. Consequently, this architecture can be categorized as 2-hop invariant GNN, leading to expected challenges in capturing changes in global geometry. The consistent underperformance of eSCN may be due to the task being 0(3) equivariant but not SO(3) equivariant. Additionally, eSCN and EquiformerV2 are quasi-equivariant methods which introduce error into the equivariance due their spherical activation function, see Appendix D of (28) for details. This additional error explains the significant difference in results for eSCN and EquiformerV? well as why they may over perform as well as under perform.
F.2 ADDITIONAL RESULTS FOR IS2RE
Model
Feat. Dim.
Param.
Energy MAE [meV]]
EwT[%)
eSCN
464
1856
IIM
0.380
0.006
865 1 14
1.91 0.09
eSCN
2
206
1854
10M
0.369
0.006
842 1 13
eSCN
3
133
1862
9M
0.397 0.001
904 3
1.85
ESCN
4
98
1862
9M
0.408 1 0.006
929 1 15
1.74 1.12
77
1848
0.409 0.003
933
1.61 +
eSCN
6
64
1856
8M
0.3836 + 0.003
872 + 6
1.91 0.19
Equiformerss V2
77
304
7M
COM
00M
00M
EquiformerV
34
306
9M
0.369 0.009
841 + 21
2.02
Equiformer
3
22
306
12M
0.363 + 0.009
828 = 21
1.94 = 0.08
EquiformerV2
16
0.364 0.005
832 + 11
2.03
Table 6: Validation results of the steerable model ablation study L and over 4-folds of the IS2RE dataset with 10k training molecules. We observe that higher type-L steerable models may not perform best. 0OM denotes models that run out memory during training.
Note that for EquiformerV2 L=1, the model will fit onto the GPU. However, during training the data and the model will exceed the GPU memory capabilities. We denote the out of memory phenomenon OOM in the Table and Table report the model parameters and size.
28

---

Published as a conference paper at ICLR 2024
Model
Run Time (min)
Memory
eSCN
464
151 =
12.3GB
eSCN
206
297
9.1GB
eSCN
3
133
207 i
9.0GB
eSCN
98
9.2GB
eSCN
77
246 3
10.9GB
eSCN
64
429 = 8
11.5GB
Equiformer V2
77
0OM
18.6GB
Equiformer V2
2
34
284 1
12.6GB
EquiformerV2
22
256
11.1GB
EquiformerV2
16
298
11.5GB
Table Run time and memory footprint for the steerable model ablation study on and over 4-folds of the IS2RE dataset with 10k training molecules. OOM denotes models which run out of memory during training.
Validatinn Loss
0.55
Equlformerv211
EquiformarVV -2.c-30
0.0200
8.50
Equitormervzic c.c-
EquiformeVV1 -,, 16)
validation
0.0175
8.45
1.0
0.0150
0.40
80.9
0.0125
Buloomr/2i-44.c-
Validation Epoch
Validation Epach
Validation Epoch
Figure 3: Validation results for the ahlation study on EquiformerV2 for the IR2SE: dataset with 10k training molecules. Depicted the loss (left) energy MAE (middle) and energy within the threshold(Fw/] (right) over 12 validation epochs. We plot the mean over four rums and shade the standard deviation. We observe no difference between the values for different type-L models.
F.3 EXPERIMENTAL DETAILS
k-Chain The reported optimal hyperparameters are used except in the case of steerable mod- els eSCN and EquiformerV2 where the steerable feature dimensions are reduced due to memory constraints. Steerable models using type L 0, 1, and dimension to preserve the dimension of the steerable convolution. The experiment is implemented in Google Colab (26] with invariant and equivariant models using the 16GB NVIDIA T4 GPUs and the steerable models using the 40GB NVIDIA A100 GPUs. To ensure the connectivity of the graph is respected, we adjust the cutoff radius of all models to 5.1 units.
used with adjusted steerable feature
The training procedure of (20 is modified to apply softmax function to model outputs, stabilizing 1000 training epochs using the Adam optimizer. The learning rate is scheduled based on the validation accuracy with an initial learning rate Le-4, 0.9 learning rate decay, 25 epochs of patience, and a minimum learning rate of 1e-5. We use fixed data splits and report the mean and standard deviation of the test accuracy from 10-fold voss-validation of random model weight initializations.
the classification training. The cross entropy minimized
The task is implemented using the OC20 framework using the baseline IS2RE training procedure with 12 training epochs for all models and the reported optimal model hyperparameters. Due to computational constraints, consider the 10k molecule training data split and reduce the steerable feature dimension to train each model on a single 24GB NVIDIA RTX3090 GPU.
IS2RE
The training procedure of [6 is used for training. In particular learning rate of 8e-4, a maximum epoch of 12 and gradient clipping if the gradient norm greater than 20.
the AdamW optimizer with
S2EF The task is implemented in the OC20 framework using the baseline S2EF training procedure and the reported optimal hyperparameters for each model. Training performed in parallel on two 24GB NVIDIA RTX4090 GPUs using the 2M molecule training data split. Due to the limited computational resources, we are only able to train a single model for 8 epochs over the span weeks. We note that we compare with 32GB of RAM and does not report the standard deviation.
results to (28 which trains 4-fold cross validation on 16GPUs
The training procedure of (6] is used. In particular
the AdamW optimizer with weighted decay of 1e-3, a cosine learning rate scheduler and an initial learning rate of 4e-4. The maximum number of epochs are 12 and gradient clipping is applied if the gradient norm greater than 100.
29

---

Published as a conference paper at ICLR 2024
F.4 HYPERPARAMETER DETAILS
In this section we provide details on the adjusted hyperparameters for each task. In general we implement the reported optimal hyperparameters for each model.
F.4.1 k-CHAIN
For models the cutoff hyperparameter (cutoff, max max_uu, r_max) 5.1. Table lists the hyperparameters for ComENet, eSCN and EquiformerV2. For all other models, we use the hyperparameters outlined in Joshi et al. [20]
ComeENet
Value
cSCN cutoff 5.1
Value
EquiformerV2
Value
cutoff
5.0
max_radius
5.1
hidden channels
256
hidden channels [256,113] att hidden channels
[64,16,7]
middle channels
64
Imax list
[1,2]
Imax list
[0,1,2]
num num_radin radial
mmax [1,2]
mmax list
[0.1,2]
sphericll
regress_forcee forces False
regress_ forcees
False
output output layers
Table 8: Hyperparameters for the ComeENet, eSCN and EquiformerV2 architectures trained on
k- k-Chain dataset. In brackets provide the values for each type-L model.
F,4.2 IS2RE
eSCN
Value
EquiformerV2
Value
num_layers
12
num_layers
8
max_neighbors
20
sphere_channels
128
cutoff
12.0
attn hidden_ channels
[34/16,16]
ccienee_
128
num heads 8
hidden channels
[206/64,98/64, 64]
attn alpha channels
64
Imax_ list
[2,4.6]
Imax_ _ist
[2,4]
mmax list
[2,2,2]
mmax list
[2,2]
num waap__shee___aapee
128
norm_typp 'layer norm sh'
distance_functio
function
"gaussian"
otf otf_graph True
regress forces forress
False
regress regress_forces forces
False
use_pbe
True
attn waa__alue_chhnnee fîn on__hiden__chmme channels 32
16
basis_width_ scallrr
2.0
otf graph True
Table 9: Hyperparameters for the eSCN and EquiformerV2 architectures trained IS2RE with 10k training molecules. In brackets, we provide the values for the ablation study on the stecrable feature dimension.
30

---

Published as a conference paper at ICLR 2024
F.4.3 S2EF
eSCN
Value
num Sum_layers 12
max_neighbors
20
cutoff
12.0
sphere_channed
128
hidden channell
[256,824]
Imax list [6.2]
mmax__ist 2
num sas___sheee_____p____
128
distance ficancceffnction
"gaussian"
regress_ forces True
use use_pbe True
basis_widee width_scalar otf _uf_graph True
2.0
Table 10: Hyperparameters for the eSCN architectures trained on S2EF with 2M training molecules.
In brackets provide the values for eSCN[L 6, 256] and eSCNIL 2, c 824] resp.
31

---

Published as a conference paper at ICLR 2024
G AN ILLUSTRATIVE EXAMPLE
While Lemma is not affected by rank(X) (,), the rank of the spatial em- bedding X, the answer to this question is contingent upon the rank. In Fig. we present an example to gain intuition into this phenomenon. The graph comprises two distinct points æ1, Z2. We assert that all type-1 fea- tures must lie on the line passing through æ1 and Z2, which corresponds to rank-1 space. Suppose there exists type-1 steerable feature + that does not lie on this line. In such a case, we can select rotation R around the line, which preserves 1 and #2 but changes the direction of Ro v. Consequently, v cannot be generated equivariantly from 21 and 02, leading to a contradiction.
22
x1
Rv
Figure 4: If a doesn't lie on the line passing through æi and 22, it can be trans- formed into another vector by some rotation R around line. Therefore, e can- not be generated equivari- andy from œ1 and T2.
The rationale behind this is that different ranks of X correspond to distinct underlying symmetries inherent to X. Specifically, when rank (X) = 3, there exists non-tre aniiiiiiill group action capable of preserving the spatial embedding X, meaning that the stabilizer 6x x is empty. In contrast, in the coordinates
cases where rank X)
confined to line a plane- certain group actions can be applied without affecting X. That is, 3g E 8 s.t. g X = X. Since g f(X) fg X) = f(X). we observe that any steerable feature f(X) must be preserved under the action of This limitation inherently constrains the complexity of steerable features.
32


Reviews:
[{"review_id": "Z27eSIlXns", "reviewer": "Reviewer_nbPE", "summary": "This paper studies the expressivity of geometric graph networks which are “multi-hop”, or aggregate information at each message passing step based on all nodes at most some number of steps from each node. They use the notion of equivariant moving frames to connect invariant features with equivariant features, illustrating that any equivariant activation space can be converted to an invariant activation space of equal dimension, so long as the frame is tracked (via an input-dependent group element). With this perspective, they show that k-hop invariant GNNs lose some geometric information relative to equivariant GNNs, but that the particular representations internal to equivariant GNNs matter less (in some sense) than the dimensions of the representations.", "questions": "1. Intuitively, where is the multi-hop aspect used in this work? The intuition about invariant features and moving frames seems true for 1-hop networks too. What is the takeaway regarding “multi-hop” architectures from this work, in contrast to 1-hop networks?\n2. eSCN is an architecture evaluated in the experiments section. But is this a multi-hop architecture? \n3. Could the authors comment further on Corollary 3, regarding point (1) from the Weaknesses section? For example, doesn’t the choice of internal representation affect the universality a given architecture (beyond just the dimension)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The problem studied in this paper is topical and practically meaningful, as geometric GNNs are quite widespread in application. Moreover, the insights presented in the paper are prescriptive, and give a rule of thumb for selecting hyper parameters such as the highest “L” (indexing the irrupts of O(3) or SO(3)) to include, and how many channels to use. The use of equivariant moving frames to convert to invariants is uncommon elsewhere in the literature, and its application to expressivity considerations is creative. The paper is generally clear and well-written. The authors’ theoretical claims, and even some less rigorous intuitions, are backed up with large-scale experiments on cutting edge architectures for the OpenCatalyst dataset.", "weaknesses": "1. My main qualm is with Corollary 3 and similar statements. In this Corollary, the authors note that given an equivariant function on a k-dimensional input representation space, it can be converted to an equivariant function on any other k-dimensional input representation space. As a result, they claim that the choice of representation to which the internal activations of a neural network transform is irrelevant (modulo its dimension). However, the conversion mapping from one equivariant vector space to a different (equal-dimensional) equivariant vector space may be arbitrarily complex or difficult to compute, and indeed, it may or may not be computable with a given architecture. In the case of rotations, one needs to recover g from a faithful representation (e.g. l=1), and then evaluate another representation at g — but computing a Wigner D matrix of high L may take many layers/parameters in the GNN. This is in the same spirit as universality results, which e.g. require increasing numbers of layers to approximate polynomials of increasing degree (see e.g. the appendix of Bogatskiy et al 2022, Lorentz Group Equivariant Neural Network for Particle Physics). In other words, there is a critical distinction between whether the **information** to compute a particular mapping is available, and whether a given architecture can actually **efficiently compute** that mapping. The authors of this work seem to focus more on the former perspective; namely, whether or not there is information loss from a certain architecture, which precludes one from ever computing a particular function (by any means). This has been a fruitful perspective for existing impossibility results on invariant GNNs — since these results roughly establish that, by only working with invariants, some neighborhoods are indistinguishable, and so **no** architecture can distinguish between them. This is a strong notion, but the converse does not hold: when the information is **not** lost, this does not imply that any architecture can actually compute the mapping. All of this is to say that, in my opinion, Corollary 3 does not sufficiently imply that the choice of equivariant representation is irrelevant up to dimension. I suspect even that the choice of representation may affect the universality of a fixed architecture family.\n2. On a related note, many of the paper’s claims are informal — e.g. “there is no fundamental distinction in message-passing mechanisms arising from different values of L” in Remark 3, or “analyzing the message passing using the corresponding invariant features is a reasonable approach” on page 4. It would be very helpful and important to make these precise. \n3. There are a few very related lines of work which aren’t cited, but probably should be. For example, the paper’s reasoning relies heavily on the concept of “equivariant moving frames,” which are a classical notion (dating back to Elie Cartan); yet this term does not appear in the paper, nor does a citation to e.g. Puny et al’s frame averaging paper, which is a more recent machine learning paper that harnesses the concept of frames. A small section (the start of Section 3.2 on page 6) in this paper also notes that the output symmetry of an equivariant function must be at least the input symmetry; this is a well-established fact about equivariant functions, e.g. see Smidt et al 2021. Finally, and perhaps most significantly, related ideas were discussed in ClofNet (Du et al 2022) and its follow-up LeftNet (Du et al 2023), both of which use the precise idea of moving frames and invariantization to obtain equivariants from invariants. The latter work in particular includes an expressivity result for two-hop geometric GNNs. \n\n\nReferences:\n* Frame Averaging for Invariant and Equivariant Network Design by Omri Puny, Matan Atzmon, Heli Ben-Hamu, Ishan Misra, Aditya Grover, Edward J. Smith, and Yaron Lipman\nFinding symmetry breaking order parameters with Euclidean neural networks by Tess E. Smidt, Mario Geiger, and Benjamin Kurt Miller \n* SE(3) Equivariant Graph Neural Networks with Complete Local Frames by Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Bin Shao, and Tie-Yan Liu \n* A new perspective on building efficient and expressive 3D equivariant graph neural networks by Weitao Du, Yuanqi Du, Limei Wang, Dieqiao Feng, Guifeng Wang, Shuiwang Ji, Carla Gomes, and Zhi-Ming Ma\n\n\nHere are a few minor typos and writing notes:\n* On page 3, in the definition of geometric graphs, in the last two sentences: g inconsistently has a subscript $g_i$. Also, the last equality should presumably be an inequality (k-hop distinct if for all isomorphism, there is a node I, such that for any g, that equation does NOT hold).\n* Page 3, k-hop geometry GNNs: “Given a geometric graph G=(V,E,F,X).” is not a full sentence; perhaps “Given” should have been “Consider”.\n* In Lemma 1, was $V_0$ defined somewhere?\n* As a very minor nitpick, the calligraphic g the authors have chosen for the group element is used almost universally to refer to an element of a Lie algebra, not of a Lie group. I would recommend sticking to regular $g$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ni9M9EmR4Y", "reviewer": "Reviewer_WxNP", "summary": "In recent years there has been growing interest in various forms of `geometric graph neural networks', where the input are graphs whose node have attributes in R^3, and the tasks discussed are equivariant to both relabeling of the nodes and applying a global rigid motion to all node coordinates. \n\nThere is a zoo of different methods for these problems, which differ among others in the use of invariant vs equivariant features and in the type of GNN used as a backbone. This paper attempts to understand the importance  of these various different choices. It  discusses k-hop GNNs with invariant/equivariant features. Its main claims are:\n(a) Invariant features are less expressive than equivariant features.\n(b) equivariant features of order 1 are as expressive as higher order features- the main issue is the dimension of the features and not the equivariant order", "questions": "Dear authors: please relate to the first two weaknesses above- what your opinion is about them, and how you intend to address them if you agree they should be addressed. \n\nDetailed remarks about writing\n* Page 4: Steerable features: It would make more sense to me to define f to be the steerable features. What does it mean to say that a vector is a steerable feature? Say you have a vector x=[1,2,5] in R^3. Is it a steerable feature or not?\n* Page 5: \"without loss of generality we may assume the group reps are all the same\" why?\n* Remark 3 is vague and out of context. Corollary 3 seems related, is much more formal and accurate, but does not require the action to be faithful. Can you formally explain why faithfulness is important? This could be the criitical missing link explaining why in terms of the irreducible order $\\ell$ we claim to have 0<1=2=3=4=... which is the message you are trying to convey but I don't think the theory currently supports.\n* Page 6: \"This question is essentially asking whether the space of all type-ell features has a dimension of 2\\ell +1\" This is well defined once f is fixed I believe and X is free? Perhaps clarifying this will also help me understand how you define steerable features in general.\n* Page 6: Could you specify v=f(x_1,x_2) or whatever the relation between v and x_1,x_2 is?\n* Also I think a similar observation appears in [Villar, Lemma 3] I suggest you cite them\n* Table 1: what are the significance of the specific three values of (L,c) chosen in Table 1? Presumably you can run eSCN with any choice of the c-s?\n* In Table 2 the invariant equiformerV2 actually does reasonably well, seemingly contradicting your theorem?\n\nsmall comments that should not be addressed in rebuttal:\n* Page 2: the stabilizer is not only a set but also a group, perhaps you would prefer to say 'group' \n* Page 3: I didn't understand the definition of k-hop distinct. Possibly a typo somewhere there?  \n* in Lemma 1 and elsewhere: in the formula f(X)=... the output of lambda is in $V_0^{d}$ and then you apply $\\rho(g)$ and get something in $V$. I understand what you mean but technically the group action should go from V to V.\n* Page 4: \"for simplicity we represent g as g_X\" I believe here and a line or two above you accidently swapped between g and g_X\n* In theorem 2 has V_l,aug been previouisly defined? Couldn't easily find the definition.\n  [Villar et al. Scalars are universal: Equivariant machine learning, structured like classical physics. Lemma numbering from arxiv version]\n *Page 9: `our work has evoked' I would suggest different wording focusing on what the works does rather than what it caused the reader to do.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper has some observations I believe are novel and interesting:\n(a) an interesting correspondence between equivariant functions and invariant functions, and between equivariant functions spaces of different order. \n(b) the conjecture that high dimensional representations do not matter, once the number of features is evened out, is interesting and recieves some empirical and theoretical evidence in the paper.", "weaknesses": "* Regarding (a): This correspondence has an important limitation which I believe the authors should mention:  Update and Aggregation functions which are actually useful for learning are reasonably `nice': for example, differentiable almost everywhere, continuous, etc. The correspondence suggested by lemma 1 uses a mapping of every orbit to a canonical element which indeed exists, but is not in general continuous or very well behaved. As a result an equivariant `nice' aggregation function will correspond to a `wild' invariant aggregation function.\n\n* The paper's stand on invariant vs equivariant features seems inconsistent. On the one hand the paper maintains that there is a one-to-one correspondence between invariant and equivariant features, and that \"propagation of steerable features can be effectively understood as propagation of invariant features, therefore analyzing the message passing using the corresponding invariant features is a reasonable approach (page 4)\" on the other hand once this analysis is carried out the paper maintains that it points to an advantage of equivariant over invariant.  \n\n*I have some issues with the writing, which could be addressed in a revision towards a camera ready version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xpXjrzPI0G", "reviewer": "Reviewer_BvXD", "summary": "This paper introduces a theoretical analysis of the expressive power of steerable equivariant GNNs. In particular, the authors show that every type-$L$ steerable feature can be analyzed through its corresponding invariant features. The authors use this lemma to study $k$-hop invariant GNNs and show limited expressive power. Then, the authors argue that any type-$L$ steerable feature is as expressive as its dimension. Specifically, there is a one-to-one correspondence between steerable features and $d $ invariants. Hence, increasing $L$, when accounting for the increase of feature dimension, does not improve the GNN's performance. The authors test their findings on several numerical experiments.", "questions": "- Can you contrast your results with Villar et al. (2022)? How do your results differ/improve upon theirs?\nLet's take a basis vector $e_1 \\in \\mathbb{R}^{3 \\times 1}$ and $f: \\mathbb{R}^{3 \\times 1} \\to \\mathbb{R}^3$ with $e_1 \\mapsto \\alpha_1 e_1, \\alpha \\in \\mathbb{R}$. This is clearly an equivariant function. What would the corresponding invariant function be such that Lemma 1 holds?\n- Let's take three basis vectors $e_1, e_2, e_3 \\in \\mathbb{R}^{3\\times3}$ and $f: \\mathbb{R}^{3 \\times 3} \\to \\mathbb{R}^3$ with $e_1, e_2, e_3 \\mapsto \\alpha_1 e_1 + \\alpha_2 e_2 + \\alpha_3 e_3, \\alpha \\in \\mathbb{R}$. \n  - Isn't the set stabilizer now only the trivial rotation?\n  - What would, in this case, be the corresponding unique $\\lambda$?\n- Why did you use $L=2$ and higher in your experiments in Tables 1, 2, and 3?\n- Could you elaborate on my last comments in the previous (weaknesses) section?\n- Do you know if your results were somehow already known in (geometric) invariant theory?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The writing is of high quality and clarity. Recent works have started analyzing the expressive power of steerable graph NNs, but little consensus has been reached. Hence I'd say this is a timely and significant work.", "weaknesses": "See Questions for clarifications.\n\n- I think a highly related work is Villar et al. (2022) (\"Scalars are universal...\") with several related and perhaps even identical results: please include this in your related work section.\n- The notations sometimes could be clarified a bit further.\n  - For example, the logic above Lemma 1 could be laid out a bit clearer.\n  - It's not clear to me how $\\rho(g_X)$ acts on $\\lambda(X) \\in V_0^{\\oplus d}$. First off, what is the action of $\\rho(g_X)$? Further, $\\lambda(X)$ is a tuple of scalars, so are they all left-invariant? Or are you considering them as a vector in $V$? This is also how you define $f$ in Lemma 1.\n  - In general, how can an *equivariant* function be described by *invariants*? I guess I'm not following some notations here.\n- I find it slightly confusing what the exact message is that the paper is trying to convey, especially towards the end of the paper. \n  - It is claimed that one needs more than invariants to capture the global symmetries, yet Lemma 1 states that one can use invariants to represent equivariant functions (see also my previous comments).  \n  - I find it slightly confusing that according to the paper, one doesn't need more than $ L=1 $, but many experiments (Table 1, 2, 3) use $ L=2$ and higher. \n  - In the conclusion, it is claimed that $L \\geq 1$ captures local structures, but one doesn't need more than type $L$. Why don't the authors claim that $L=1$ is enough?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WK7z6o6OMH", "reviewer": "Reviewer_3HqL", "summary": "This paper compares the expressive power of invariant and equivariant GNNs, from a theoretical perspective, backed by a couple of experiments (1 toy experiment and 2 real datasets).\nIt is an extension of what was recently done in [16] to the case of k-hop GNNs.\nThe main theoretical contribution is Lemma 1, which states that there is a unique invariant feature corresponding to any equiavariant one. It is made more concrete by theorem 2 and corrolary 1.\nThe experiments investigate the interplay between type order L (order of the Spherical Harmonics expansion, e.g. L=1 is vectors, L=0 scalars, L>1 are high order tensors), depth, and channel number, in the context of fixed feature dimension. This allows the authors to formulate the cautious claim \"preserving the feature dimension, the expressiveness of equivariant GNNs employing steerable features up to type-L may not increase as L grows.\"\n\nI update my rating to 6, marginally above acceptance threshold.", "questions": "1. In the definition of k-hop geometric GNNs, k is not defined (its role is not defined).\nActually, the definition of k-hop GNN is a bit overlooked (although it's rather intuitive what the meaning of k-hop is), and more importantly, it is not clear to me how these are not essentially equivalent to regular GNNs with more layers (since a single layer of a 2-hop GNN aggregates information with a similar receptive field as a 1-hop GNN with 2 layers).\nProbably authors should elaborate on that or cite appropriate references.\n\n2a. About Lemma 1, page 4. When you define the function c, why can't it simply be always the identity? One could always choose the neutral element from the group G.\nI do understand why the set {g ∈ G | g · c(G · X) = X} is not empty.\nI do not understand what is non trivial about c(G · X)\n\n2b. Also, V_0^{⊕d} is not explicitly defined early enough. What is d ? Does it relate to the maximal rotation order L ?\n\n2c. You proove that the decomposition of Lemma 1 is unique. But, concretely, what is lambda ? Isn't it almost always simply the norm of the irrep ? Like, for a vector, its norm, or for a higher-order tensor, also its norm ? And then \\rho(g_X) is just the usual (matrix) representation of g_X.\nCan it be otherwise ? Or is it more complicated ? Am I missing the point ?\nIf in practice things are simple, in most of the usual settings, saying so is helpful to the reader.\nOR, maybe g_X simply encodes the orientation of X (say if it's a vector)?\n\n\n2d. In ref [16], page 3, it says:\n|At initial-\n|isation, we assign to each node i ∈ V a scalar node colour\n|c i ∈ C ′ and an auxiliary object g i containing the geometric\n|information associated to it\n|i.e. they already separate (factorize) the scalar (feature only) contributions from the geometric ones (vectorial one v + previous color).\nIn this respect it is not obvious how contribution (1) (your lemma 1) is new.\nFurthermore, I lack an intuitive view of what may concretely go into your scalar function lambda(X).\n\n\n3. gothic{g} appears in the r.h.s. of Eq (4) or in the last line of the previous est of equations (let me call this last line Eq. 3bis) (actually the gothic-style has been forgotten in Eq. (4), this is a typo I believe).\nHow can we identify the right term in the r.h.s. of Eq. 3bis as a lambda (invariant), when g^-1 appears in it ? I don't see why g^-1\\cdot x_ij should be invariant, on the contrary, it seems to me like an equivariant feature.\n\n\nIn Eq. 7, first line, why isn't it 2l+2 instead of 2l+1 (in the r.h.s.) ? I understood that representing O(3) required one additional component ? Could you provide an intuitive explanation?\n\n4. You write, in page 6:\n|Does any (2l + 1)-dimensional invariant feature λ(X) correspond to a type-l steerable feature f (X)?\nWhich sounds like an interesting question (although ideally I'd like to know the mapping, not just know about its existence)\n|Note that this question is essentially asking whether the space of all type-l steerable features f (X) has a dimension of 2l + 1 since D l (g X ) is invertible.\nBut that sounds like a known fact: using Spherical Harmonics, it is known that  type-l steerable features  need components $h_m$ with  $m\\in[-l,+l]$ to be represented. That is, they need 2l+1 numbers (dimensions)\n\n5. The remark below corrolary 3 is a very strong, original conclusion.\nI think I understand corrolary 3 (altohough I feel like in practice lambda is the norm of the representation, and in that case it's kind of trivial..).\nIn any case I do not see how the remark  \"the expressiveness of learning steerable features is primarily characterized by the feature dimension – independent of the highest type utilized\"  follows from corrolary 3.\n\n\n6. In table I, I can do the maths for line 1 and 2, but not line 3.\n(5+3+1)*256 = 2304\n(5+3+1)*824 = 7416\nbut then,\n(1+3+5+7+9+11+13)*256 =12544 , not 7424\n\n\n7. experiments are a bit disappointing.\n7a. First, Table 2 shows a number of red (intersting) cases, but they turn out to be related to eSCN being SO(3)-eqivariant, when the task is O(3). This is not making the point of the paper, and instead is rather confusing, at first sight.\nMost importantly, it's not clear to me in which sense table 2 is different from the table 1 of ref [16] (which by the way seemed more readable and dense).\nPlease clarify this.\n\n7b. IS2RE. Table 3.\nHere I enjoyed a lot the idea of comparing models with comparable feature dimension.\nHowever, several points:\n- why not report also the total number of parameters ? They do not grow as fast as feature dimension since weights are shared for a given L (type order)\n- although I understand it's somewhat hardware dependent, please also report the training time for these models. Maybe even the memory footprint (on GPU).\n- Figure 3 gives an idea of how things perform over a reasonable number of epochs (and I salute the attempt to not go to an exceedingly large number of epochs), but it seems that a more thorough study of variations with L and c, reporting only the epoch=12 validation performances, would be useful to the reader (I did not have time to look at the content of ref [5])\n\n7c. S2EF. Table 4 is commented, but no conclusion stands out. What is the point of that table ?\n\n\n8. Conclusion. Two key findings are claimed:\n| (1) Propagating steerable features of type ≥ 1\n| equips geometric GNNs with the inherent ability to automatically capture the geometry between local\n| structures. (2) When the feature dimension remains constant, increasing the use of steerable features\n| up to type-L cannot essentially improve the performance of equivariant GNNs\nBut I think that  (1) is vague and already known. For instance from (16), which clearly distinguishes invariant (L=0) from equivariant L>=1\nnetworks.\nAnd that (2) is known as oversquashing and was known before.\nI note that the term oversquashing is absent from the manuscript, although it is mentionned several times in [16].\nI think the term should appear. If it sounds too buzzwordy to the authors, they should mention it at least once, and explain why they don't like it.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea of studying equivariant GNNs in the context of k-hop aggregation is new.\nThe formalism the authors use to formulate proofs is general and does not rely on Spherical Harmonics explictly (this is also a weakness though).\nThe idea of working at fixed budget to compare models, albeit probably not new, is to be saluted.", "weaknesses": "See my questions below for details. The main Weaknesses are:\n\nLack of novelty compared to what was done in [16]. See my points 2d., 4., 7a., 8.\n\nOverall clarity: although wording is clear, some definitions are not given (as k-hop, see questions below) and the fact that theorems prove existence but do not provide an example of such function (like lambda, in some concrete example) make it hard to follow. See questions.\n\nAlso, it is not clear whether all claims are correct (or, some of them may be correct but trivial if properly re-phrased -- but I may be wrong !).\n\nI may be misunderstood in my assessment, hence my numerous questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)