Improving Equivariant Networks with Probabilistic Symmetry Breaking
Authors: Hannah Lawrence
Vasco Portilheiro
Yan Zhang
Sékou-Oumar Kaba
Venue: ICLR 2025 Poster (2025)
Keywords: equivariance
symmetry
symmetry-breaking
canonicalization
graphs
GNNs
Primary area: learning on graphs and other geometries & topologies
TL;DR: We propose a probabilistic framework for breaking symmetries, e.g. in generative models' latent spaces, by combining equivariant networks with canonicalization.

Abstract: Equivariance encodes known symmetries into neural networks, often enhancing generalization. However, equivariant networks cannot *break* symmetries: the output of an equivariant network must, by definition, have at least the same self-symmetries as its input. This poses an important problem, both (1) for prediction tasks on domains where self-symmetries are common, and (2) for generative models, which must break symmetries in order to reconstruct from highly symmetric latent spaces. This fundamental limitation can in fact be addressed by considering *equivariant conditional distributions*, instead of equivariant functions. We therefore present novel theoretical results that establish necessary and sufficient conditions for representing such distributions. Concretely, this representation provides a practical framework for breaking symmetries in any equivariant network via randomized canonicalization. Our method, SymPE (Symmetry-breaking Positional Encodings), admits a simple interpretation in terms of positional encodings. This approach expands the representational power of equivariant networks while retaining the inductive bias of symmetry, which we justify through generalization bounds. Experimental results demonstrate that SymPE significantly improves performance of group-equivariant and graph neural networks across diffusion models for graphs, graph autoencoders, and lattice spin system modeling.

Published as a conference paper at ICLR 2025
IMPROVING EQUIVARIANT NETWORKS WITH PROBABILISTIC SYMMETRY BREAKING
Hannah Lawrence'; Vasco Portilheiro"; Yan Zhang*1, Sékou-Oumar Kaba 4,5
1 Department of Flectrical Engineering and Computer Science, MIT, 2 Gaatsby Computational Neuroscience Unit, UCL, *Samsung SAIT Al Lab, Montreal, 'Mila- Quebec Artficial Intelligence Institute, 'cGGill University
ABSTRACT
Equivariance encodes known symmetries into neural networks, often enhancing generalization. However, equivariant networks cannot break symmetries: the output of an equivariant network must, by definition, have at least the same self- symmetries as the input. This poses an important problem, both (1) for prediction tasks on domains where selm-mymmetises are common, and (2) for generative mod- els, which must break symmetries in order to reconstruct from highly symmetric latent spaces. This fundamental limitation can be addressed by considering equiv- ariant conditional distributions, instead of equivariant functions. We present novel thcoretical results that establish necessary and sufficient conditions for representing such distributions. Concretely, this representation provides practical framework for breaking symmetries in any equivariant network via randomized canonicaliza- tion. Our method, SymPE (Symmetry- -brekking Positional Encodings), admits simple interpretation in terms of positional encodings. This approach expands the representational power of equivariant networks while retaining the inductive bias of symmetry, which we justify through generalization bounds. Experimental results demonstrate that SymPE significantly improves performance of group- rrouu-equivariant and graph neural networks across diffusion models for graphs, graph autoencoders, and lattice spin system modeling.
INTRODUCTION
Learning tasks with known symmetries, such as rotations and permutations, abound in applications (Veeling et al.) 2018 Celledoni et al. [021] Bogatskiy et al. [022]] Velickovie] 2003]. Equivariant learning, which builds these symmetries directly into neural networks, has been shown provide powerful inductive bias for deep learning (Bronstein et al., 2021). However, even in domains that seemingly have clear symmetries, there are functions that equivariant networks simply cannot represent. For example, consider the problem of predicting one molecular three- dimennional graph from another, such as predicting a dichlorobenzene molecule from a benzene molecule (pictured at the top of Fig. [T. Such tasks are relevant in generative modeling of atomic systems (Satorras) (ct al.) [2021] Xie et al., 202I) and molecular editing (Liu et al.) 20024). Since we are working in 3D space, rotation equivariance is natural
intuitively, rotating the benzene molecule should only affect the rotation of the predicted dichlorobenzene, not the structure of the molecule itself.
While this approach seems reasonable, strange problem arises. When the input to equivariant model is self-symmetric, must remain self-symmetric in the output (as pointed out by e.g. |mmddt al. ).. Since benzene has sixfold rotational symmetry, equivariant model is unable to output dichlorobenzene, which is not rotationally symmetric.
In fact, self-symmetry arises in a variety of applications, often with more complex groups- e.g. non-trivial graph automorphisms, Hamiltonians of physical systems with symmetries, or rotationally symmetric point-clouds (Fig. [ Generative models and autoencoders, which reconstruct from latent space, are particularly noteworthy. By virtue of being embedded in a simple, low-dimensional space, the latent representation often has greater self self-symmetry than the input itself, i.e. certain
Autthor order decided by coin flip. Correspondence to: banlawemit edu, wascopo@gmai.l.cccccoooo Code available at: https:tpp... /tt::////ttiii cowphu/a.coo///aaaaaaaaaaaaaaaaa awrence/ www.a///eweae.eeeceeeeeeeebreaae

---

Published as a conference paper at ICLR 2025
transformations of the input will not affect its latent representation. An equivariant decoder must then map the more symmetric latent space to the less symmetric data space, which is just as impossible as predicting dichlorobenzene from benzene. To avoid this problem, we could simply discard symmetry structure entirely, but this loses the generalization benefits of equivariance on asymmetric inputs. The question we ask is therefore: How can retain the inductive bias of symmetry, while resolving the difficulty posed by self-symmetric inputs?
We will focus on equivariant distributions, i.e. func- tions from the input space to distributions over the output space, rather than the more standard setting of equivariant functions from input to output space. This provides greater flexibility in the types of systems we can model, because as we will see later, distribu- tion can equivariant without individual samples from that distribution transforming equivariantly. In- stead of mapping an input to a single output through an equivariant function directly, we map the input to an equivariant distribution, and sample from it. This distinction will allow resolve the symmetry breaking problem elegantly.
SO(3) Network
Sg Encoder
X
Decoder
Embedding
SOL3) Network
Figure 1: Example applications requiring symmetry breaking- Top: A rotation- equivariant network for molecules cannot transform benzene into dichlorobenzene due to benzene's s sixfold symmetry. Middle: A commutation-equivriann graph decoder can- not break latent space symmetries (see Ap- pendix H). Bottom: A rotation-equivarian network for point clouds cannot transform table into a chair, as the table's 's legs are rota- tionally indistinguishable, unlike the chair's.
Contributions Extending the results of [Bloem- Reddy & Teh (2020) probabilistic symmetries, we derive a method which can provably represent any equivariant conditional distribution (Section B). For this, we rely on an external source of randomness, coming from a canonicalization function which has been appropriately randomized. Our theory suggests a simple modification to existing equivariant models, akin to concatenating positional encoding, which allows them to break symmetries (Section[4) 4 In Section we show more generally that equivariant break symmetries, and provide theoretical justification for its generalization henefits in Section 6 Using our framework, we also show in Section 7 and Corollary AA.3 that several
noise injection
recently proposed approaches to symmetry breaking (Kaba & Ravanbakhsh [023] Xie & Smidt 2024) can be modified to represent any equivariant conditional distribution. Finally, we validate our
approach experimentally in tasks on graphs, atomic systems, and spin Hamiltonians (Section
2 BACKGROUND
Preliminaries Let X' and y be measurable input and output spaces, respectively. We assume G group which acts on both X' and Y. with the action of G E X' denoted by gæ X' Equivariance describes functions f satisfying f fgx) g (2). capturing the requirement that the output should transform predictably under transformations of the input. Given subgroup H C G,
we denote a left coset of H in G as gHl = [gh: f H! for E G. We denote right cosets similarly G}. The stabilizer of is denoted gx x} and is the subset of G that leaves unchanged. Any x with non-trivial G₂ termed self-symumetric. We consider probability distributions over X. y, and G, with P(D) denoting the space of all probability distributions over D. Also, denote by X and Y the input and output random variables, respectively. For random variables X, and X2, we write X1 =a.s. X₂ the equality holds with probability 1. Denote the distribution of X by IP (X), and the conditional distribution of Y given X by IP (X) The action of G on y naturally gives rise to an action on distributions, defined by g (Y) IP (gY) as shown in Figure 2 Essentially, the action of (YY is the distribution the random variable after transformation by g.
Hg. The orbit of x E X' with respect G [x]] {ga 9 E
by G2 [g E G
Following Bloem- Bloem-Reddy Reddy & Teh 2020), the definition of equivariance can be extended to conditional
distributions P (Y[X) simply by viewing P (Y|X) as a function from X to P(Y), and applying the standard definition of equivariance to the action of G' on P(Y). In essence, transforming the value of
IG' should be locally compact, second countable Hausdorff, with proper actions on X' and y (Chiu & Bloem-Reddy 2023).
2

---

Published as a conference paper at ICLR 2025
the input random variable E A' by just shifts the distribution of Y by g:
PP P = gz) = 9 P ( = = (gVlX = r).
(1)
Curie's Principle As discussed previously, the out- put of equivariant functions must be at least as self- symmetric as the input. This is known as Curic's Principle physies (Curie 1894). can stated formally in terms of stabilizers as G5 S f()) for any x E X if f is an equivariant function. The proof is quite simple: if g E Ga. then ga implies f(x) - f(gx) = g g
60
8 = 60
If we consider a probabilistic system, Curie's Prin- ciple holds at the level of distributions instead. That is, if IP (Y is equivariant, then for given the distribution P (Y = =x) inherits the same self- symmetries as G C GP(y)x=a)- A key insight is that this does not need to hold for individual samples from PP XY x), as shown in Figure 3 This known by physicists as spontaneous symmetry break- ing (Beekman et al., 2019). In this work, we posit that symmetry breaking is best understood as coming from the probabilistic nature of the world.
g= 60
Figure 2: Example of how group acts on distributions. Top: An element g E SO(2), the group of -iv-dimensiona rotations, can act unit vector in R² by standard rotation. Middle: This induces an action of on a distribution (blue) over unit vectors (orange). Under this action, rotates the entire distribu- tion. Bottom: When g = 60% acts on a distri- bution which is already 60% self-symmetric, the distribution remains unchanged even though a vector sampled from the distribu- tion has no 60% self-symmetry.
While equivariant distributions provide an elegant way of addressing symmetry breaking, equivariant modeling is usually done with deterministic networks. will now see how to bridge the two.
From equivariant functions to distributions Bloem-Reddy & Teh (2020) link together the two concepts by expressing equivariant distributions in terms of equivariant functions. They show that under certain conditions P (Y x) is equivariant if and only if there exists function f: X x (0, + equivariant in (i.e. f(gz, c) = of (I, e)) such that:
- ((X,))
with e (0, 1) a random variable independent of X. However, one necessary condition is that G acts on X freely: each x E X' must not have any self- eelf-symmetries (i.e. the stabilizer G, is trivial). Therefore, this result does not apply to the kinds of inputs for which we would like to break symmetries. In the following, we show how to address this limitation.
REPRESENTATION OF EQUIVARIANT DISTRIBUTIONS
Our aim is now to write equivariant distributions in terms of equivariant functions. while handling possibly self-symmetri inputs. The main idea is to introduce a mapping called the inversion kernel (Kallenberg] 2011). Intuitively, the inversion kernel maps an input to uniform distribution over subset of group elements describing the input's self-symmetry. Sampling from the inversion kernel will then let break the self-symmetry of the input, and ultimately represent equivariant distributions terms of equivariant functions.
To formally define the inversion kernel, we first use the concept of canonicalization (Kaba al. 2023). Canonicalization allows for transforming any input to a canonical "pose," or orbit representative.
Definition 3.1 (Canonicalization function). function т : X - is a canonicalization function (xx) = - is invariant for all æ E X. The function 7y X + X' is the orbit representative map associated with the canonicalization function, and 7(x) is the orbit representative.
Invariance of the orbit representative map ensures that each transformed version of gets assigned to single representative of the orbit of z. The canonicalization function can alternatively be seen as
2G' compact, P (X) = g (X), and there is measurable canonicalization map (as defined in Section 3
3

---

Published as a conference paper at ICLR 2025
P(Y (X-x)
sample
91 30
91 30
92 + 30
P(Y | - a
sample
91 x
92 y
Figure 3: When the imput is rotated by 30", as shown from the top the bottom row, the equivariant conditional distribution IP X = æ) (middle column) also rotates by 30". The distribution thus has the same self- self- symmetry æ., which is sixfold rotational symmetry. However, individual samples from (V = z) are free to break this self-symmetry, as shown in the rightmost column.
providing group element which maps the orbit representative to given input, i.e. r (t)). the absence of self-symmetries, the canonicalization function is uniquely defined (and equivariant), given 7. But consider the case where z has a self- self-symmetry.... For g Gz. have gz and therefore æ = g x. and so for any canonicalization T(z) taking x to ((x), it holds that
(gr(x))
r(x) ((z))-'-''_ -(z)
(xx)
The entire coset Gz (ar) comprises the possible ways to get to x from 7(x). The inversion kernel is map from to uniform distribution over these transformations that canonicalize x in the same way.
Definition 3.2 (Inversion kemne).. Let 7 - G be a canonicalization function. The corresponding P(G), is the conditional distribution IP X x)) (G.7(G..(T))
inversion kernel, mapping
Example 3.3. For example, consider G = SO()), and let be the benzene ring at upper left of Fig. 3 If we set (1r = 30 (such that the orbit representative (*)) the lower left), then G2 {0', 60% ..... 300%). and the inversion kernel at is uniform over TZ {30%, 90" ..... 330%)
While each choice of canonicalization map r gives rise to an inversion kernel, we often refer to "the" inversion kernel for simplicity, assuming tacitly some 7 has been chosen (and is universally measurable (Kallenberg 2017)). Using this, we can generalize [Bloem-Reddy & Teh (2020) to represent any equivariant distribution.
Theorem 3.4. IP (Y ) is equivariant and only if
Y a f(X,ğ,c)
(3)
for a function f X x G (0,1) + y jointly equivariant in its first two inputs (i.e. f(ar, hg, c) h1(2, (., g.e. E)), noise ~ Uni 1), and X| distributed according to some inversion kernel.
The proof follows in Appendix A where also prove related representation in terms of relaxed equivariant functions (Kaba & Ravanbakhsh 2023). Intuitively, our result decomposes the ran- domness in Y|X into that derived from symmetry breaking, Ø. and independent noise e. Here, selects among various transformations E G which canonicalize X, and since any group element has trivial stabilizer, this allows Y break self-symmetry: in essence. breaking of existing canonicalization techniques to that of P (Y [X). Note that in this work, we are primarily concerned with the randomness deriving from symmetry breaking and not learning generic equivariant distributions, and so we will focus primarily on omitting from experiments.
bootstrap the symmetry
METHOD: SYMMETRY- BREAKKY POSITIONAL ENCODING (SYMPE)
Following Theorem 3.4 and as shown in Fig. we propose to represent equivariant conditionals using g) as input (and if so desired). Two implementation questions remain: (1) how to sample ÿ. and (2) how to pass as input to an equivariant network f. We first introduce general approach to sample from inversion kernels using canonicalization.
equivariant neural network f. and pass
4

---

Published as a conference paper at ICLR 2025
X
X
Apply
Canonicalization Method
~
(0), 6T,
300))
gv
Equivariant Network
Figure 4: Illustration of our symmetry-breaking method. Here, a die indicates randomness, which is used in the canonicalization method (shown in the dotted box) to sample g. (Optionally, random variable e can also be input to the equivariant network f. to capture randomness unrelated to symmetry symmetry-breaking we do not include this variable in our experiments.) Ultimately, the input æ and the sampled group element ( are input to an equivariant network f as f(r
Then, show how to encode the group element 9 as input the neural network. We finally provide an interpretation of our method as a positional encoding (Vaswani et al., 2017)] Bello et al. 2019] You et al., 2019 Srinivasan & Ribeiro 2020; Lim et al. 2023a). As will see, advantage of our method is that all the components can be learned end-to-end, and do not require much hand-engineering beyond specifying which equivariant neural network use.
Inversion kernels with canonicalization Our method requires sampling ğ (nTT(G, for some choice of canonicalization function т. This can be implemented in a simple way by taking T as canonicalization network. The form of the distribution (ni seems to suggest that need to detect the stabilizer of x along with using a canonicalization function, but this is in fact not inversion kernel can be implemented by optimization of
necessary. Following Kaba et al.] (2023), an energy function function E : R such that that Gz)(z) = arg min geG E(g where the arg min is set. Sampling is done simply by selecting random element of the arg arg min set. As show in C
Appendix it is often possible to parameterize E such that the optimization is fast and poses no significant overhead. For example, sorting breaks permutation symmetries, and can be viewed as an optimization of an energy function (Blondel et al., 2020). Implementations of these canonicalization functions exist for images, sets, graphs and point clouds (Mondal et al. [023] Kim et al. 2023).
As explained in Appendix [[and justified in Section our method also works even when is sampled from an equivariant distribution with support larger than that of the inversion kernel. This arises when working with graph inputs, for example, since canonicalizing combinatorial graphs is likely not possible in polynomial time (Babai & Luks, 1983). On the other hand, when using an invariant loss, samples of ğ can be replaced with deterministic canonicalization r(x) with no impact on the loss.
Encoding of the group element Once a group element is sampled, we need to specify way to feed into an equivariant neural network. Our method is based on the observation that if the group acts freely on vector u, then the mapping g + gu is injective. The group element is therefore uniquely specified by the way it transforms v. We can furthermore make e learnable. We show in Appendix D that vector spaces with that property can be defined conveniently for group representations of finite groups and Lie groups, which comprise most groups interest in applications. The encoded group element is then given input to the equivariant neural network by concatenation with Linking back to Theorem 3.4, our model is defined as fzz = fo(æ @ gu), where fo is any equivariant neural network. The equivariance of (x, g) + (gv) is clear, so f is indeed jointly equivariant function. As shown in Appendix [ any jointly equivariant function f can be expressed in this way, which makes our method fully expressive.
Interpretation as positional encodings We term our method Symmetry-breakinn Positional En- coding (SymPE), since it can be naturally interpreted as type of positional encoding, similar to the ones used in transformers /Vaswani et al. (017) and GNNs (You et 2019). In general, absolute positional encodings can disambiguate between identical tokens or nodes by assigning each a unique identifier. In graphs, for example, nodes which are part of the same orbit of the automorphism group (and that are thus indistinguishable for an equivariant model) are assigned different "positions" by the positional encoding. The positional encoding therefore breaks symmetries in the input. We generalize this view to other data types and groups. Specifically, the sampled group element ğ represents the position (for translation groups), pose (for rotation groups) ordering (for permutation groups)
5

---

Published as a conference paper at ICLR 2025
Algorithm SymPE: Symmetry-Breakinn Positional Encodings
1: Inputs: input 2 E X'
2: Learnable parameters: learned vector v E V, equívariant neural network parameters canonicalization parameters ø
3: Sample g ~ he (z)
Sample group element for canonicalization
4: 0 4- gv
Apply group element to learned vector > Forward pass with positional encoding
5: Return fo(r @ 0)
of the input relative to canonical one. We apply this group element to a learned vector that concatenated to each component of the input (token, pixel, node, etc.). By virtue of the group acting freely on v, it fully specifies the "position" of the input. The learned vector therefore plays the same role the sinusoidal encoding in sequence models, with the group element shifting these encodings to specify which token is the first one.
Equivariance and symmetry-breaking of positional encodings The standard, absolute positional encodings used in Transformers (Vaswani et al. (017)) are not equivariant. In other words, the position nominally assigned to tokens in the sequence by the positional encoding does not shift the sequence shifts, which precludes translation equivariance. Similarly, the pixelwise positional encodings used in Vision Transformers do not allow translation equivariance (in contrast to CNNs). Relative positional encodings equivariant to shifts (Shaw et al. 2018). but cannot break symmetries because they rely only on invariant relative distances between tokens. The Laplacian positional encodings used in GNNs (Belkin & Niyogi 2003 Dwivedi et al.. 2023), also not equivariant permutations, but can be made so at the cost of losing the ability to break automorphism symmetries (Lim et al.) [202cc] Morris et al. 2024). The same true for positional encodings based on random walks (Dwivedi et al. 2022: Ma et al. 2023), which preserve automorphisms. By contrast, in our method, the absolute position 9 is sampled from the inversion kernel, which is an equivariant distribution. As a result, the model preserves the inductive bias of equivariance, while retaining the ability to differentiate between different symmetric configurations and therefore break symmetries.
5 SYMMETRY BREAKING WITH NOISE INJECTION
Adding noise to inputs is commonly used heuristic for symmetry breaking (Satorras et al., 2021 1
Sato et al. 202T Abboud et al., 202T Eliasof et al. 2023 Zhao al.] 2024). This approach involves using functional model similar to that of Theorem 3.4 but with an arbitrary equivariant noise variable Z replacing g. It is natural to ask whether this simple heuristic can also represent any equivariant distribution. We show that under some conditions, noise injection can indeed be used to break symmetries, while still representing any equivariant conditional distribution.
Proposition 5.1 (Noise injection). Let X,Y, Z be random variables in X, X,y, Z respectively, each space acted on by G. The following are equivalent: (I) G acts on Z freely (up to a set of probability zero) and P XZ is equivariant; (2) P (Y[X) is equivariant if there exists f XX' x Z x (0,1) - jointly equivariant in X and Z such that Y a.s. f(X, Z, t) for noise Unif(O, 1).
The proof follows in Appendix A.4 This result implies that one may sample ğ from a general equivariant distribution on G' instead of the inversion kernel specifically, and still obtain an equivariant conditional distribution This applies, for example, to the recently proposed symmetry breaking sets (SBS) of Xie & Smidi (2024) (see Section 7). Moreover, for many groups of interest, such as S,, and On)). simply sampling Z from an isotropic Gaussian satisfies the requirements of Proposition 5.1 However, this method potentially introduces more noise than is necessary into the learning process. More precisely, as we show in Appendix F. the inversion kernel is an equivariant distribution of minimal entropy, which conjecture facilitates learning (since the model need not learn map more random inputs to the same output than necessary). Our ablation studies, comparing the inversion kernel generic distribution, align with this intuition (Section 8)
"In fact, the optimization of energy mentioned above is akin to sampling from density p(g x)) oc exp(-F(g ( (=)//T). When 7 0 we sample from general equivariant distribution on G. As we lower T,, the entropy diminishes and we recover an exact inversion kernel arg min F(g 'z) when T 0.

---

Published as a conference paper at ICLR 2025
6 GENERALIZATION BENEFITS OF SYMMETRY BREAKING
Equivariance is understood to impart generalization benefits by meaningfully restricting the hypothesis class. In this section, we explore similar intuitions for symmetry-breaking
Elesedy & Zaidi (2021) formalized this intuition for ordinary equivariance. They showed that if P (X) is G-invariant, then the space (x(2 Y.P (, ((X) of -itterrieeeeaalee functions f. with inner product (2) = { (x), f2). f2)) (dr), decomposes into the orthogonal sum of equivariant and "anti-equivariant" parts, f and f = f-ff- f. Assuming Y (X) +c for an equivariant function f and mean-zerv zero noise c, it follows that for L risk R R(f) = [X) - YlIa). there is a non- non-negative generalization gap Aff. f) = R(f) R(f) = If [ when using a non- conequiivriaant model f. This provides some theoretical justification for the use of equivariant models: it implies that for any non-equivariant model f. there exists an equivariant model of strictly lower risk. In Theorem 66. (proved in Appendix A. we obtain a similar result for our probabilistic setting, simply assuming P (Y is equivariant and that one is injecting equivariant noise.
Theorem 6.1. Suppose P (X) is G-invariant, and P (Y|X) is equivariant with E{y?] < co. Consider
stochastic model y = fX, Z) with E[Y?] < cc, where Z Z is such that P (Z is equivariant, and where G acts on Z freely (up to a set of zero probability). Then f decomposes into jointly equivariant J and its orthogonal complement f = f - f, and there is a generalization gap
A(f R(f) - R(f) 11 (xx2)))
(4)
where R(f) = -I]
Remark 6.2. The condition that PP (Zlx) is equivariant is satisfied when P (Z) is simply G- invariant distribution independent of X, which is often the case. The result above says that even in this pure noise injection setting, the expected loss of model f(X, 2) can be decreased by projecting ff to an equivariant function, thereby producing conditionally equivariant distribution. The result also applies SymPE, by letting Z (g. c) as in Theorem 3.4, where g acts as gZ (gg. c).
We also emphasize that the risk used in the theorem measures error in each individual prediction of Y from X. This may not be the metric of interest, if, for example, one only cares about predictions up to group transformation, we prove a similar result when instead computing the loss between distributions, i.e. between a ground truth equivariant conditional density and an arbitrary model density.
wants to learn the entire distribution for generative task. In Appendix B
7 RELATED WORK
The problem of self-symmetric inputs yielding self-symmetric outputs in equivariant learning has been observed in several domains. In the context of graph representation learning, Srinnvasaa & Ribeiro (2teiro((002)) (2020) showed that isomorphic structures in graphs must be assigned the same representations by equivariant functions. It has been noted that this is problematic in several applications, including generative modeling on graphs (Liu et al.
2019: [Saooras et al.] 2021] Yan et al., 2023] ZZhao 2023b). Similar issues arise for discriminative and generation tasks on sets. Zhang et al. (2022) noted that equivariant functions are limited in both processing multisets (sets with self self-symmetries) and decoding from an invariant latent space to a set, introducing notion of multiset equivariance, of which relaxed equivariance (Kaba et al.) 2023 Kaba Ravanbakhsh] 2023) a generalization, as a solution. Viigna & Frossard (2022) also introduced a generalization of equivariance to address the problem of set generation, which our results subsume.
al., 2024) and link prediction (Lim et al.
Other studies have focused on symmetry breaking the context of machine learning for modeling physical systems. (Smidt et al. (2021) first formulated the preservation of symmetry in equivariant neural networks as an analogue to Curie's Principle, and proposed using gradients of equivariant neural networks to identify cases for which symmetry breaking is necessary. Kaba & Ravanbakhsh (2022) identified that for prediction tasks on crystal structures, symmetry breaking can be necessary, and proposed method based on non-equivariant positional encodings. For inputs defined on discrete grids, Wang et al.] (2024) proposed a flexible method for implementing and interpreting symmetry breaking based on relaxed group convolutions, the cost of only approximating equivariance.
On the theoretical side, our work builds on that Chiu & Gloem-Reddyy (2023), who appear to have been first in applying the inversion kernel in machine learning. Sampling from general equivariant
7

---

Published as a conference paper at ICLR 2025
conditional distribution on G was also explored by Dym et al. (2024). in the context of constructing continuous and efficient frames (Puny et al. 2022). Symmetry breaking more specifically was studied by Kaba & Ravanbakhsh (223)) who defined the notion of relaxed equivariance, and furthermore showed that relaxed equivariant layers can be constructed as solutions to systems of equations.
Perhaps most closely related to work is Xie & Smidt (2024). Inspired by the idea that symmetry
breaking arises from missing information, Xie & Smmdtt (2024) defined an equivariant symmetry breaking set (SBS) (xx) (for each input x)) as Ga-dependent equivariant set Bxx)) on which G2 acts freely. Elements b E (x) can then be input to an equivariant function f(x b), breaking the symmetry of x much like our g. Our work allows one to analyze SBS through a probabílistic lens: Proposition (...ssttionn3.mmmplii implies that uniformly sampling from an SBS can represent any equivariant distribution, one also adds e ~ (n, 1) to the input as f(x c). However, using an SBS requires detecting G2 and its normalizer, while our framework naturally suggests implementation via existing canonicalization methods. Additionally, Xie Smidt (2024) are largely concerned with identifying when "ideal" Bxx), i.e. of minimal size [B(x) Gal. exists (which they argue should facilitate learning). Indeed, their chosen restriction that (xx) depend only on G2 means such an ideal SBS may not exist. In contrast, by allowing supp(ğ) Gz)(z) to depend on itself and not only Gz, we effectively always obtain an "ideal" set because Tx)) = Gzll
8 EXPERIMENTS
Table 1: Cross-entropy loss and reconstruction
error in graph autoencoding.
We evaluate SymPE empirically on three tasks:
autoencoding graphs with EGNN (Section 8.1). graph generation with the DiGress diffuusin pro- cess (Section 8.2) and predicting ground states of Ising models Is with G-CNNs (Section 8.3). In all cases, find that SymPE outperforms base- lines, both without bymmetry-breaking and with other methods for bymamety--reakkigg.
Method
BCE
% Error
Param.
No SB
10.0
3.9
101,074
Noise
10.1
2.3
88,017
Uniform
5.7
1.3
88,890
Laplacian
5.6
0.98
88,885
SymPE (ours)
3.7
0.77
101,750
8.1 GRAPH AUTOENCODING / LINK PREDICTION
Autoencoders with symmetric latent spaces pose a problem for equivariant models. One example explored in Satorras et al. (2021) ) is autoencoding graphs using a node-wise latent space Z = R_ where n is the number of nodes and f is the feature dimension. This can alternatively be understood as a link prediction problem via node representations, for which equivariance is known to be problematic Srinivasan & Ribeiro (2020); /Zhang et al. (202T). From an S,-equivariant embedding in Z, the graph is decoded equivariantly; the presence of an edge between nodes with latents zz and zj is function of ||z - I1 If A is the adjacency matrix of the graph, its self-symmetries are GA {g E Sa : Ag = A). However, there may not even exist an embedding E Z such that GA G2 (see Appendix H for details), which by Curie's Principle results in an "overly symmetric" embedding with G & G. (atorras et al. 2021 Figure 3) when any equivariant encoder is used.
We consider reconstructing Erdös-Rényi random graphs with edge probability 0.25, using the data from Satooras et al. (202T) (which we compute, via networkx, contains at least 44% automorphic graphs) and their standard message-passing passing architecture as the encoder. To break symmetries using our method, we first compute learned scalar nodewise embeddings x using one message-passinn layer, and then obtain by sorting y, letting the sorting algorithm break ties. The symmetry breaking input ē = ğJ is obtained by correspondingly sorting learned vector v E IR", and is input to the encoder. As baselines, we consider no symmetry breaking "aacking""("" ("No SB"), randomly initialized node features ("Noise"), randomly sampling 9 from S,, ("Uniform"), and passing in spectral features directly ("Laplacian"). In Appendix [H] we consider other baselines (such as breaking symmetries at the embedding level.) Breaking symmetries via our method achieves the lowest error (Table T).
8.2 GRAPH GENERATION WITH DIFFUSION MODELS
We evaluate our framework on graph generation. Small graphs, which are of interest for molecular generation, are especially likely to have non-trivial automorphism groups (Godsil & Royle, 2001).
"To ensure fair comparison our method, we still compute y and input it the encoder.
8

---

Published as a conference paper at ICLR 2025
We apply SymPE, generation. We follow the setup and experimental protocol of a state-of- of-the-art method, DiGress (Vignac al. 2023), in which a discrete diffusion process is applied on a graph's 's node features and adjacency matrix. A graph transformer then predicts back the denoised graph. Symmetry breaking is potentially crucial in this setting: since the denoising network is equivariant, if the diffusion process ever introduces a symmetry in the graph, it is impossible to denoise back to the original graph due to Curie's s Principle. |Vignac al. (2023) note that heuristic that adds spectral features node features improves performance. These can indeed break symmetries, but in a less principled way than our method (see Appendix H.. and at a computational cost scaling cubically in the number of vertices.
caaammetr--beeaknnn positional encoding, to discrete diffusion-base based graph
We evaluate combining our method with DiGress on the QM9 (Wu et al. 2017) and MOSES (Polykovskiy et al., 2020) datasets. For QM9, we consider the more challenging version in which hydrogen atoms appear explicitly in the graphs. We use sorting- based S,, canonicalization with GIN architecture (Xu et al. 2019) to sample ÿ. following|Kin et al. (2023). The symmetry-brent -reekking positional
go, is concatenated to node features, and gv.g to the adjacency matrix, with learned 2n E RRuxd and V E RRaxxxd (with n set to the maximum graph size and d = =). Further details on the experimental procedure given in Appendix
Table 2: Evaluation metrics for molecular generation on QM9 with explicit hydrogens.
Method
Validt
Uniquet
Atomic stability?
Mol. stability?
NLL
Dataset
97.8
100
98.5
87.0
ConGress (a variant of DiGress)
86.7+1.
98.4+aa
97.2+b.2
69.5=1.6
DiGress (with Laplacian)
95.4+1.1
97.6+0.4
98.1+b.3
79.8=s.6
129.7
DiGress + SymPE (ours)
96.1
97.5
98.6
82.5
30.3
DiGress noise
90.7
97.6
97.8
73.1
126.5
DiGress + SymPE (nodes only)
96.2
97.4
98.4
83.9
128.8
Results for QM9 are shown in Table 2 and results for MOSES in Table f (Appendix G Our method leads to large improvement in negative log-likelihood (NLL) compared to the original DiGress. Note that the NLL captures the ability of the model to learn the right distribution, but not necessarily chemical validity of the generated samples. Another factor to consider is that for the other metrics, the baseline values are already close to the dataset values, making them more difficult to improve. We perform ablation studies with alternative methods to break symmetry. First, we consider breaking symmetry by concatenating noise sampled from a standard normal distribution ("DiGress + noise"). This does not lead to a similar increase in performance, which we hypothesize is because the model learns to ignore the uninformative noise. We also consider only using SymPE on the node features, and not on the adjacency matrix ("DiGress + SymPE (nodes only)"). This also does not have significant effect on the likelihood, showing that breaking symmetry directly on the adjacency matrix fed to the graph transformer crucial.
8.3 PREDICTING GROUND-STATES OF ISING MODELS
Spin systems are prototypical examples of physical systems that exhibit spontaneous symmetry breaking. Here, we consider unsupervised training of neural networks to obtain ground-states (states of minimal energy) of the Ising model given Hamiltonian parameters. The Ising model is an idealization of a magnetic system of spins, with the Hamiltonian describing its energy. Identifying low- energy configurations of spin systems is an important problem in statistical physics (Hu et al. 2017]] Carrasquilla & Melko 2017) and has applications to the graph max-cut problem (Fu & Anderson 1986). yet brute force optimization scales exponentially in system size. Monte-Carle simulation is possible, but does not benefit from the generalization of neural networks across Hamiltonians.
Formally, given set of lattice sites A, spin configuration E {1 specifies binary value for each site. The Hamiltonian H 1} + R assigns energy to each configuration o. In our experiments, we consider the anisotropic Ising model under an external field h on a square periodic lattice A, given by (o) = - -EE E Jij only if and j are horizontal neighbors in A, and similarly for J The task has symmetry under the automorphism group of the square grid, G = p4m (see Appendix ( for details on the group action). That is, for any g E G, have HgJ.gh (g σ) H,,( the energy is unchanged if both the
Σnj JM Ji
Σ, h4o, Here, Jú + O
9

---

Published as a conference paper at ICLR 2025
Ground truth
MLP
G-CNN
25
FM
23-
Thansvees 7164
2.-
2.3
13
APM
Steipes
13-
2.5
-05
Vertical
Vertical Interaction
Vertical
Figure 5: Phase diagrams predicted by the different methods. For each configuration predicted by the neural network on a test set Hamiltonian, we compute the values of the three order parameters: the ferromagnetic phase (red), the antiferromagnetic phase (green), and the stripes phase (blue). Brighter colors are associated with larger values of the order parameter, and black to the disordered phase.
Hamiltonian parameters and the spin configuration transformed in the same way. The Hamiltonian parameters themselves have a self-symmetry to reflection and translations, which corresponds to configurations of minimum energy, or non-zero probability, of the distribution P (a) Jo]J, J. f)) o xxp(-HJ, /T) in the T + 0 limit. While (o|J, f)) is equivariant G. ground states may break the pmmm self-symmetries of (J,
the stabilizer group GJ,, pmm G. Ground-states
We train a p4m--quivariant G-CNN combined with SymPE to take Hamiltonian parameters as input, and return a spin configuration as output. Training done by directly using the Hamiltonian as the loss function and minimizing energy. We build training and test set by sampling diverse Hamiltonian parameters J and h from given distribution. To evaluates the ability to generalize transformed data, we additionally build an are randomly rotated by 90% See Appendix F for details on the experimental setup.
(OOD) test set for which Hamiltonian parameters
We compare our method to vanilla G-CNN, non-equivariant MLP trained with data augmentation sampled from p4m, adding noise ~ ~ (0, 1) to the input of the G-CNN to break symmetry, and the relaxed group convolutions proposed by WWng et al. (2024). We use the average energy as an evaluation metric. The proposed method achieves significantly lower energy than baselines on the OOD test set, and is only slightly worse than the relaxed convolutions method on the in nn-distribution test set. (Table 3 Appendix FF. A more in-depth understanding of the baselines's 's shortcomings can be obtained from the predicted phase diagrams (Fig. 5 Spin systems show phase transitions depending on Hamiltonian parameters, similar to molecular systems (see Appendix [F). For the considered Ising models, there are three possible phases: ferromagnetism, antiferromagnetism,, and stripes order. The order parameters give a quantitative description of the predicted phases given Hamiltonian parameters. We see that the G-CNN is not able to predict antiferromagnetic and stripes phases, which break symmetries. The MLP also does not recover the correct phase diagram.
CONCLUSION
By considering symmetry- breakking from a probabilistic perspective, we derive a representation of any equivariant distribution in terms of a deterministic equivariant function of the input, and symmetry- breaking sample from the inversion kernel. Motivated by this theoretical result, we introduce a flexible method for breaking symmetries in existing equivariant architectures, by concatenating random symmetry-breakinn case (of lowest entropy) within the general class of equivariant noise-injection methods, which we prove are able to represent equivariant distributions and enjoy guaranteed generalization benefits. We observe in experiments that SymPE outperforms baselines, both with and without symmetry- breakkng.. on graph autoencoding, graph generation, and Ising model ground-state prediction.
positional encoding (SymPE). We show that our method is special
One limitation of SymPE is that it requires access to a canonicalization method, which may be less readily available for uncommon groups, as well as way to randomize its outputs. Future work remains to sample ğ in practice for general (including infinite) groups, such as: (I) testing our proposal of using energy- based modeling in a generic setting, (2) sampling from low-entropy equivariant distributions when the inversion kernel may be hard to sample from exactly, and (3) further exploring the tradeoff between structured symmetry breaking and generic noise injection. also remains to address partial symmetry breaking (when the output breaks some, but not all, symmetries of the input), as treated for example by Xie & Smidi (2024).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was enabled in part by computational resources provided by Mila, Compute Canada, and MIT Supercloud. S.-O. K. is supported by IVADO and the DeepMind Scholarship. H.L. is supported by the Fannie and John Hertz Foundation. V.P. is supported by the Gatsby Charitable Foundation. The authors are grateful to Siamak Ravanbakhsh, Daniel Levy, Tara Akhound-Sadege Vitória Barin Pacela, Bonaventure F. P. Dossou, Clément Vignac, Matthew Morris, Tess Smidt, YuQing Xie, Elyssa Hofgard, Nadav Dym, Jonathan Siegel and Peter Orbanz for insightful discussions.
REFERENCES
Ralph Abboud, ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization, In Proceedings of the Thirtieth International Joint Conference Artifical Intelligence (IJCAT), 2021.
László Babai and Eugene M Luks. Canonical labeling of graphs. In Proceedings of the fifteenth annual ACM symposium on Theory of computing, pp. 171-183, 1983.
Aron J. Beekman, Louk Rademaker, and Jasper van Wezel. An introduction to spontaneous symmetry breaking. SciPost Phys. Lect. Notes, pp. 11, 2019. doi: 10.21466/SciPostthysslccc ectNots.. 11. URL
https: sci post. /00 21 469/ Scc
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373-1333) 1396, 2003. doi: 10.1 10.1162208976666331111000
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V. Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Benjamin Bloem-Redee and Yee Whye Teh. Probabilistic symmetries and invariant neural net- works. Journal of Machine Learning Research, 21(90):1-61, 2020. URL htthttp:://mmr.oo org/ papers/v21/ 19 322. html
Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking. In International Conference on Machine Learning, pp. 950-959. PMLR, 2020.
Alexander Bogatskiy, Sanmay Ganguly, Thomas Kipf, Risi Kondor, David W Miller, Daniel Murnane, Jan T Offermann, Mariel Pettee, Phiala Shanahan, Chase Shimmin, et al. Symmetry group equivariant architectures for physics. arXiv preprint arXi::2003.06/53 2022.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovié.
Geometric deep learning:
Grids, groups, graphs, geodesics, and gauges. arXiv preprint 11t:::201413477 2021.
Juan Carrasquilla and Roger G. Melko. Machine learning phases of matter. Nature Physics, 13(5): 431-434, 2017. doi: 10. 1338mphy44433. /RL.httpss//000..rr//00000000
Elena Celledoni, Matthias J Ehrhardt, Christian Etmann, Brynjulf Owren, and Ferdia Sherry. Equivariant neural networks for inverse problems. Inverse Problems, 37(8): 085006, 2021.
Schönlieb,
Kenny Chiu and Benjamin Bloem- Redd-- Non-parametric hypothesis tests for distributional group symmetry, 2023.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, 2016.
Pierre Curie. Sur la symétrie dans les phénomènes physiques, symétrie d' un champ électrique et d' u'u champ magnétique. Journal de physique théorique el appliquée, 3(1 3(1):393-415, 1894.
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations, 2022. URL htttps://openzeviex www.eewwwwwwwwwweeeeeeeeeeeeeeee
id=wTT jnvGphy
11

---

Published as a conference paper at ICLR 2025
Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 24 (43):1-48, 2023.
Nadav Dym, Hannah Lawrence, and Jonathan W Siegel. Equivariant frames and the impossibility of continuous canonicalization. arXiv preprint arXiv:2402.16077, 2024.
Bryn Elesedy. Symmetry and Generalisation in Machine Learning. Ph.D. Thesis, University of Oxford, 2023. URL http:::ttt:///tttt http://aaaaaasssp////a.aaaaaaa
s.pff
Bryn Elesedy and Sheheryar Zaidi. Provably striet generalisation benefit for equivariant models. In
Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2959-2969, PMLR, 18- 24 Jul 2021.
URL http:: htttpp::/prooeeee: ngs ml www.m.................m..mmmmmm press v1339999 v1339//eeeeeeyy// hhmml
Moshe Eliasof, Fabrizio Frasca, Beatrice Bevilacqua, Eran Treister, Gal Chechik, and Haggai Maron. Graph positional encoding via random feature propagation. In International Conference Machine Learning, pp. 9202-9223, PMLR, 2023.
Yaotian Fu and Philip Anderson. Application of statistical mechanies np-complete problems in combinatorial optimisation. Journal of Physics A: Mathematical and General, 19(9):1605, 1986.
Chris Godsil and Gordon Royle. Algebraic graph theory, volume 207. Springer Science & Business Media, 2001.
Wenjian Hu, Rajiv RP Singh, and Richard Scalettar. Discovering phases, phase transitions. and crossovers through unsupervised machine learning: A critical examination. Physical Review E, 95 (6):062122, 2017.
Sékou-Oumar Kaba and Siamak Ravanbakhsh. Equivariant networks for crystal structures. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022.
Sékou-Oumar Kaba, Arnb Kumar Mondal, Yan Zhang, Yoshua Bengio, and Siamak Ravanbakhsh. Equivariance with learned canonicalization functions. In International Conference Machine Learning, pp. 15546 15566. PMLR, 2023.
Oukoar Kaba and Siamak Ravanbakhsh. Symmetry breaking and equivariant neural networks. In Symmetry and Geometry in Neural Representations Workshop, NeurlPS, 2023.
Olav Kallenberg. Invariant palm and related disintegrations via skew factorization. Probability Theory and Related Fields, 149:279-301, 2011. URL httpt:ttt//aaaa http:ppp...emmtiicccccc......a
Ttrg/CrrpusD::
121316040
Olav Kallenberg. Group Stationarity and Invariance, pp. 266-309, Springer International Publishing, Cham, 2017. ISBN 978 978-3-319-41598-7,
doi: 10. 10.10077978-33319-11998877 URL https
// . o/ooi.oo//0..1000000 httpg///1..111/99518 3-111-878-------11---111-
Olav Kallenberg, Foundations of Modern Probability, Springer International Publishing, Cham, 2021. ISBN 978-3-0000--11811111 doi: 10. 10000079788--000-688771111 URL /tttps://oo..rr//111
Jinwoo Kim, Dat Nguyen, Ayhan Suleymanzade, Hyeokjun An, and Seunghoon Hong. Learning probabilistic symmetrization for architecture agnostic equivariance. In A. Oh, T. Naumann, A. Globerson, Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 18582- 18612. Curran Associates, Inc., 2023.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook brain theory and neural networks, (10): 1995.
Derek Lim, Hannah Lawrence, Ningyuan Teresa Huang, and Erik Henning Thiede. Positional encodings group representations: A unified framework. 2023a.
12

---

Published as a conference paper at ICLR 2025
Derek Lim, Joshua Robinson, Stefanie Jegelka, Yaron Lipman, and Haggai Maron. Expressive sign equivariant networks for spectral geometric learning. In ICLR 2023 Workshop on Physics for Machine Learning, 2023b.
Derek Lim, Joshua David Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and basis invariant networks for spectral graph representation learning. In The Eleventh International Conference on Learning Reprexentations, 2023c. URL https
net.:/iewweeeeeeeeeeoeeeeeeee net/ /
Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky. Graph normalizing flows. Advances in Neural Information Processing Systems, 32, 2019.
Shengchao Liu, Chengpeng Wang, Jiarui Lu, Weili Nie, Hanchen Wang, Zhuoxinran Li, Bolei Zhou, and Jian Tang. Unsupervised discovery of steerable factors when graph deep generative models entangled. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
ht.tps: www.eeeeeeweweeeeeeeeeeeeeeeeeee net wttp://wwwweeeooooiiiieeeeeee
Liheng Ma, Chen Lin, Derek Lim, Adriana Romero-Soriano, Puneet K Dokania, Mark Coates, Philip Torr, and Ser-Nam Lim. Graph inductive biases in transformers without message passing. In International Conference Machine Learning, pp. 23321-23337, PMLR, 2023.
Brendan D McKay and Adolfo Piperno. Practical graph isomorphism. ii. Journal of symbolic
computation, 60:94-112, 112, 2014.
Arnab Kumar Mondal, Siba Smarak Panigrahi, Oumar Kaba, Sai Rajeswar Mudumba, and Siamak Ravanbakhsh. Equivariant adaptation of large pretrained models. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 50293-50309. Curran Associates, Inc., 2023. URL htttps: www.:eddddnnnneerrsrsse...ee http://c835111102223303000222223 Pappree--eppee--C-eeeeeeeeeeee Conferenceeeecee pdf
www.nnnuupp..o cc/ppppeepcpppppppppppeee httppppprrfillepppppppeee files//appe:2223/
Matthew Morris, Bernardo Cuenca Grau, and lan Horrocks. Orbit-equivariant graph neural networks.
In The Twelfth International Conference on Learning Representations, 2024. URL https?
/oopenreview.oo net.://wweeeeeeeeeeeeeeee..eeeee forum? Satprummmnd-GGoocaaaa322
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Simon Johansson, Hongming Chen, Sergey Nikolenko, Alán Aspuru- Guzik, and Alex Zhavoronkov. Molecular sets (moses): A benchmarking platform for molecular generation models. Frontiers in Pharmacology. 11, 2020.
Omri Puny, Matan Atzmon, Edward J. Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations, 2022. ht..stts: //tp://////pneeeeee/e net/ forum?
id z IUyj155XXR
Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In Proceedings of the 2021 SIAM international conference on data mining (SDM), pp. 333-341. SIAM, 2021.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural net- works. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Proceedings Machine Learning Research, pp. 9323 9332. PMLR, 18-24 Jul 2021. URL htttps::/ppoceeediiiiii satorras21 h
mll. wwt.p/eresssiii.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.). Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 2 (Short Papers), pp. 464 468, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 2.118653/y1/NEE 2074. URL https: / antho ogy. httpol///1oo///0020/00000
13

---

Published as a conference paper at ICLR 2025
Tess E. Smidt, Mario Geiger, and Benjamin Kurt Miller. Finding symmetry breaking order pa- rameters with euclidean neural networks. Phys. Rev. Research, 3:L012002. Jan 2021. doi:
1LL113/PhyyeeReeeeccc1LL..202000 PhysRevresearch.. 3. LO 2002
URL httph:://tiinttea aps wttp////lg.oooooeoooeeooooe 1103/ https//LLLi.aaaaargg///////000/0
Balasubramaniam Srinivasan and Bruno Ribeiro. On the equivalence between positional node embeddings and structural graph representations. In International Conference Learning Repre- sentations, 2020. URL
http://wwwwweee//roreoeeeeee net/ hott://oorrmii----aaaaaas.s
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: https::/prroceeinggs.. wtw.://ewee...iieeeeeii.e....ooo cwurips.cc/pppprr ott.re////pppooeoooo.e...rsss.os htt.://aa22222ee11111e0000oooaaa
files//aper/22177
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pp. 210-218. Springer, 2018.
Petar Velicković. Everything is connected: Graph neural networks. Current Opinion in Structural
Biology, 79:102538, 2023.
Clement Vignac and Pascal Frossard. Top-n:
Equivariant set and graph generation without ex- changeability. In International Conference on Learning Representations, 2022. URL https/ //openreview.ro net
www.roorr-dd-ooo-o -et.arrvveee.nnt//Eommm------- IP_jkk
Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang. Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In The Eleventh International Confer- ence on Learning Representations, 2023. URL /ttpps::/ppenrevieww net/ forum?id
UaAD 6WX
Rui Wang, Elyssa Hofgard, Han Gao, Robin Walters, and Tess E. Smidt. Discovering symmetry breaking in physical systems with relaxed group convolution, 2024.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-I1), pp. 681-688. Citeseer, 2011.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: A benchmark for molecular machine learning. Chemical Science, 9, 10 2017. doi: 10.1039/C7SC00664A..
Tian Xie, Xiang Fu, Octavian- Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint arXfv:2110.06197, 2021.
YuQing Xie and Tess Smidt. Equivariant symmetry breaking sets, arXiv preprint 202002240..02681 2024.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL https
//openreviw http::teeeoommmmm1dddddd........
Qi Yan, Zhengyang Liang, Yang Song, Renjie Liao, and Lele Wang. Swingnn: Rethinking permutation invariance in diffusion models for graph generation, 2023.
Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. In International conference on machine learning, pp. 7134-7143. PMLR, 2019.
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi- node representation learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), ), Advances in Neural Information Processing Systems, 2021. URL htt.ttttt///ppeetet
net. /o/ forum d=Hor 9mgBGdss
14

---

Published as a conference paper at ICLR 2025
Yan Zhang, David W Zhang, Simon pncoste-Julien, Gertjan J. Burghouts, and Cees G. M. Snoek. Multiset-equivariant set prediction with approximate implicit differentiation. In International Conference Learning Representations, 2022. URL hattoss//ppeerrree....eae/aaaaa
1d==5k7RR22E:j03
Lingxiao Zhao, Xueying Ding, and Leman Akoglu, Pard: Permutation-invariant autoregressive diffusion for graph generation, 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDICES
Proofs
16
A.1 Proof of Theorem A.1
17
Proof of Theorem
17
A.3 Proof of Corollary A.3
17
Proof of Proposition
18
A.5 Proof of Theerem
19
Generalization benefits of equivariant conditional distributions
19
Implementation inversion kernels
20
D
Breaking symmetry in different groups
22
D.1 Permutation groups
22
D.2 Subgroups of the general linear group
................................
22
D.3 Representation of jointly equivariant functions
22
The inversion kernel injects minimal noise
23
Ising model experimental details
23
Ground state of the anisotropic lsing model
23
F2
Hamiltonian encoding
24
Group action
25
F4
Training setup
25
F55
Energy results
26
G
Graph diffusion experimental details
26
Graph autoencoder experimental details
27
A
PROOFS
Our results hold for any acting properly on X, which includes compact groups and actions on Riemannian manifolds by isometries. For more details, including the relationship of proper actions to needed mcasurability conditions, we refer the reader to Chiu & Bloem- Reddy (2023).
Theorem 3.4 is consequence of the following result, which says can represent any equivariant conditional distribution by randomly canonicalizing some function.
Theorem A.1 (Randomized canonicalization) IP (Y is equivariant if and only if
Y G.S. (x(((X))e) e.s. - X,,)
(5)
for some function o X (0.1) V. independent noise ~ Unif 1), and g|X distributed according to the inversion kernel for some orbit representative map ?
The proof (Appendix A.I follows closely that of Bloem- and Teh, which bootstraps determin- istic symmetry from a canonicalizer to that of (Y X). We analogously bootstrap the distributional
symmetry of IP (ğ X) to that of model " X, for Y, where is arbitrary function.
16

---

Published as a conference paper at ICLR 2025
In addition Theorem 3.4 the above allows derive representation in terms of relaxed equivariant functions as defined by Kaba Ravanbakhsh (2023).
Definition A.2. A function f: + Y is relaxed equivariant if for any x E X and g E G, we have hf(x) f(gz) for some h gG.-
On r with trivial stabilizer, a relaxed equivariant function behaves like a usual equivariant function. On points with non-trivial self-symmetry, relaxed equivariance only enforces symmetry of f "up to elements in the stabilizer," so the output need not have any self-symmetry.
Corollary A.3 (Relaxed equivariance). PP (X is equivariant if and only if
- 9x](X,e)
(6)
for some x x (0, + y relaxed equivariant its first input, with gx XX Unif(Gx)), and (0, 1) independent of X and 9x.
The proof is in Appendix A.3
A.1 PROOF OF THEOREM A.1
Proof. IP (Y is equivariant and only if (Chiu & Bloem-Reddy 2023 Theorem 3)
X) (x))
where ğ] X is distributed according to the inversion kernel associated to 7. By conditional noise
outsourcing (Kallenberg 2021] Proposition 8.20) this is equivalent to there existing a measurable
function ø : x (0, 1) + Y' such that
g yy (()()),c)
(8)
where c ~ Unif(0, 1) is independent of (g. X). Rearranging and noting (x Rig. X gives the result.
A.2 PROOF OF THEOREM 3.4
Proof. The forward implication is clear, letting f(x,g, 0 (r,c) c) with given in the previous theorem. On the other hand consider a function f X' x G x (0, 1) such that hf(x, 9.c) = fhhr, hg, <), and suppose Y a.e. [(X,ğ.c) Write år for a random variable sampled from the inversion kernel conditioned on x, independently from c Unif(0, 1). For any x E X' and h E G we have
(AY E B|X = (hY E BX = = (hf(h-12,0) 2n e) E B).
(9)
By the equivariance of the inversion kernel, -h-1z d h h-1 Applying first this fact and then the equivariance of f we obtain
P Phy E BIh = r) P (h/(h~ 'x, 1r
-.y e) E B) = = Pf(x, On, 6) B) P ( E B x)
(10)
Note that for this direction, we only needed the equivariance of the distribution IP (]] not its restriction to specific coset.
A.3 PROOF OF COROLLARY
Proof. Given from the previous theorem, let f be a relaxed equivariant function such that f()(z)))) = (()())))) (which we can always do).
R.s. (o()(X), = (f()(X).c) = gh -h-
(11)
for some E G (depending on the construction of f) such that hy(X) = X. But g|X ~ Unif(h()(x))) and (X)h Gx so 9h- - ~ Unif(Gx).
17

---

Published as a conference paper at ICLR 2025
Suppose the other hand Y "= 9xf( for relaxed equivariant f. Write O2 for uniform random element G2 independent Unif 1). We then have
IP PhY& BJIX x) P (NY B|X = -- IP (bgn-12/10 (1, E B).
(12)
Since Ih- in gath
POLY E B ( B).
(13)
By the relaxed equivariance of f, for some k- E h GG, the above is equal to
P (xx, B).
(14)
Noting that hk E Ga. we have Ia 9nhk and thus
IP (hY E B|XX x) = PY E B|X
(15)
(The invariance of the distribution of gx X used here, though what is strictly needed is equality in distribution of h k- and 9z.)
A.4 PROOF OF PROPOSITION 5.1
We decompose the forward and reverse implications in the following four statements:
1. If the action of G on Z is free except for a measure zero subset, then equivariance of IP (Y[X) implies that Y 4.5- s(X; e, for an appropriate f and e. Note that equivariance of Z|X is not needed here.
2. If the action of G on Z free except for measure zero subset and IP (ZIX) is equivariant, then Y 4.5. [(x, Z,e) implies equivariance of IP (Y|X).
3. If the action of G on X' Z not free except for measure zero subset, then f (X, E) is not equivalent to equivariance of IP (Y]
4. If (Z is not equivariant, then Y a.S. [XX, 7, is not equivalent to equivariance of P (Y]X)
Proof 7 Let be a group element distributed according to the inversion kernel. Reusing arguments from main results show there exists map t : X' x Z x (0, + G equivariant in the first two inputs such that ; 0.9. t tXX, Z, 17) where 77 ~ Unif 1) is random noise independent of X and Z. By Corollary 3.4 Y fo (X, tXX, Z, n)) eo) for a jointly equivariant fu- The unit interval and unit square both being standard probability spaces, there exists measure preserving bijection <> (1, (o). We thus define f(x, 2,e) ) = fo(X, ((X, (,))
To show the existence of t, one repeats the proof of Theorem A.1 and Corollary 3.4 but applying
Chiu Bloem-Reddy (2023) Theorem 3) in the special case of an essentially free action. In particular, since (X, Z) has trivial stabilizer almost surely.
J T (X, 2) 2),
(16)
where 7(X (x(X), Z) (with slight abuse of notation). The rest of the proof is identical to that the forward direction of Corollary 3.4
Proof 2 The proof is identical to that of the reverse direction of Corollary 3.4
Proof of Suppose G does not act on Z essentially freely. Suppose P (Y|X) is equivariant and
there symmetry breaking with non-zero probability, i.c. P (Gx,z & Gy) > 0. (Such examples are not hard to construct.) Suppose for the sake of contraction that also Y 4.8. f (X, Z) with some f and usual. Curie's Principle gives the contradiction: Gx,z Gy any realization of I7
18

---

Published as a conference paper at ICLR 2025
Proof of We consider non-equivariant PP (Z[X); concretely, suppose Z is constant E Z, and that there exists g E G such that g * 2. Let f : X'' x Z x (0,1) + 3 be the jointly equivariant function f(z e) = x + (so y = X' x 2). Then if Y a_ f(X,,,e),
P (g vlgX = =) P (gf(XX u| = 9 g 11 + gz y P (Y -Y= r)) #1 y) =
(17)
(18)
That is, (Y is not equivariant.
A.5
PROOF OF THEOREM 6.1
The proof of Theorem 6.1 resembles that of Lemma 3.12 in Elesedy
(2023). The latter applies the orthogonal decomposition of Elesedy & Zaidi (2021) in combination with the result of Gloem-RRedee
Tch 2020). Analogously, we combine the orthogonality arguments with our Proposition 5.1
Proof: The assumption that G acts essentially freely on 2 and the equivariance of PP (Y|X), by part of our proof of Proposition 5.1 (Appendix A.4), means that there exists f X 2 x (0, 1) - y jointly equivariant in its first two coordinates such that Y a.5. [x Z, e) for independent noise (ni 1). Defining for each e the function J (I, + l (3r, 2)), the finiteness of | implies fc E (3 x V.Y.P (X, 2)) for almost every e. Note next that the invariance of P (X) and equivariance of XZ imply the invariance of the joint distribution (X, 2). We can therefore apply Lemma 1 of Elesedy & Zaidi (2021) to obtain decomposition f f= f fl into equivariant and equivarriat" components, orthogonal under the inner product (fi, f2) = [ [2 P) (dæ, dz). The risk of ʃ may then be written as
E[Is- 12 ((xx2))) ElIl ft ()) |1xx)) R()) |1
(19)
where we obtain the first expression by conditioning on c, and the equality follows from the orthogo- nality of f and fe. The theorem follows by subtracting R()) from both sides of the equation. 0
B GENERALIZATION BENEFITS OF EQUIVARIANT CONDITIONAL DISTRIBUTIONS
Here, interested in describing the generalization benefits of using equivariant conditional distributions, when the ground truth distribution P P(Y|X) is equivariant. The general outline, which we will fill in below, follows the work of Elesedy & Zaidi 2027 and its straightforward generalization in [Eesedy 2023). We similarly assume that X has a G-invariant distribution, that IP [Y] ) is equivariant, and the group is compact F Then, considering a Hilbert space of conditional distributions (or rather, their unnormalized counterparts), one can show that the equivariant ones form a subspace. The generalization benefits of assuming equivariance can then be expressed in terms of the projection operator onto that subspace.
In order to follow the program above, we will treat conditional distributions as equivariant functions f: X2 + PP))). Furthermore, in order to work in Hilbert space, we restrict ourselves to distributions with a square integrable density with respect to some measure dy on y. We assume the measure is invariant under G- canonical example being y finiteddimensional Euclidean space with Lebesgue measure and G acting orthogonally. Then, we define IP()) C C))()) by
P())
-())=
-W: IR s.t. s.) 2 0, / ((v) (y dy 1,
/ yEy
(ur) dy < <00
(20)
vEy
We will treat densities p(y|z) as members of the Hilbert space H. = 12 (2)))) P). where the data distribution P on X is G-invariant, The inner product between functions f1, E H is
(fr. = - (()()) P(dx) = J / f(00)))2((11)) dy P(dr),
'Our previous results held in the more general case of proper group actions.
19

---

Published as a conference paper at ICLR 2025
where for simplicity use the notation f(y|z) (f()))(v) for even those f E H. which not probability densities.
G' acts on functions e (2)))) and thus P())) (v) (
By the assumption of the G-invariance of dy, the inner product on L2)))) is invariant: (01 (222
1gb1,9b22 Standard arguments from Elesedy 20023 Lemma 3.1) then show that the Reynolds operator R. H - H given by
(Rf(a) g 1 f((r)) X(dg)
(21)
[ ((ulla) X(dg)
(R)()
(22)
is in fact the orthogonal projection onto the subspace of equivariant functions-ia. those f E H such that f(g f(y gz).. For the Reynolds operator to be useful to us, remains check that sends normalized conditional densities p(u (z) to normalized conditional densities (which will then be equivariant). But this clear:
J (RD)(Ul)) dy I le plavlsz) M(dg) dy - J / plavloæ) dy X(dg) 1,
(23)
where at the end we used the invariance of dy, and the fact that p(ylæ) and À are normalized.
We then consider risk under the (2)))) loss. The generalization gap between two conditionals P1,22 P2 E
(pI p2) R(pi) R(p2)
(24)
where R is the risk as measured against the ground truth conditional p",
R(p) Jf. [I(()) p0 p0(( P(dx) J (olule)) -p'()l))) dy P(dx).
(25)
We can rewrite that risk as
R(p) = 11p--"111 = 1p1 - 2(p, '' + 11p"1|2.
(26)
(We subsequently drop the subscripts for conciseness.) The generalization gap between an arbitrary
p H and its equivariant projection p Rp is then given in terms of the orthogonal component
p² p p:
App = || p" 1| - 11
(27)
= = + pー 1- || 22
(28)
| p" 2|2 11-1112- - 11p- p²11 = 11p+11
(29)
where one gets to the the last line by using the orthogonality of p and p.p" E Hg.
C IMPLEMENTATION OF INVERSION KERNELS
We now provide more details on the implementation of inversion kernels. As noted in the main text, general method is provided by the following optimization approach of et al. (2023). Given an energy function E' X' + R, there exists a canonicalization 7 such that the set arg migngeee E(g- -x) GZT(x), so long as the energy function is "non-degenerate". By that, mean that there is unique minimizer 7(x) arg min
E(x') for any orbit Ga. For differ- ent groups, we describe how an energy function E can be parameterized such that the optimization efficient. We also give more details on the non-degeneracy requirement, which is non-trivial to satisfy for some groups, but does not pose issues in practice.
We use to denote the (normalized) Haar measure G.
20

---

Published as a conference paper at ICLR 2025
Discrete translation and rotation groups with equivariant neural networks As shown by Kaba it al. (2023), the energy function E can in general be alternatively represented with an equivariant function. This can be formalized with the following proposition.
Proposition C.I. Suppose 8 X + R is a jointly equivariant function such that s(g, hr) = s(h g. x) Vg, h E G, E X. Then, there is a finction E : xx + R. such that s(g,x) Lig -'(g-1a).
Proof. To prove this, we show that s(g. 2) only depends on '. Consider any two pairs (g.z) and (g', x²) such that g = g'''x'. Then, it must be that there is an h E such that g' hg and x' hr. If we then consider s(g',²²) = s(hg, hx)), using the equivariance condition we obtain sig', x²) = =(g.x).
'x'.
By currying, we can also see the equivariant function as outputting real number for each group element, e.g. s: X' - RC, For finite groups. like discrete transsaanlts groups. the function can be conveniently implemented using equivariant neural network architectures like CNNs (LeCun et al. 1995) and G-CNNs (Cohen & Welling 2016). By averaging the output feature map over channels (but not over fibers), we can obtain a real number for each group element and take the arg max to sample from the inversion kernel. With this parametrization, the non-degeneracy requirement is heuristically expected to be satisfied. This is because for it not to be satisfied, the weights of the neural network would have to "accidentally" result in identical outputs for different group elements.
Symmetric group with sorting The above parametrization in terms of an equivariant function s: X + RC is impractical for the symmetric group S,, since its size grows combinatorially with input size n (here, we assume X' consists of sets of n objects). For this group, there is however a simple and efficient way to canonicalize using sorting. This can also be seen as an optimization procedure as follows.
We first define the energy E(g-tx) [ p², where f / + RR is an Su-equivariant function that scores each element of the and p [n. l..... 11. From equivariance, we have E'g = fr) p" Then, as shown by e.g. Blondel et al. (2020), arg mingeg E(g- = arg mig f(xx pf argsort fz) We can therefore simply take to be any S,-cquivariant neural network, and sort the outputs corresponding to each element of an input set or nodes in graph. When the input has self-symmetries, multiple permutations will sort the input, so one is chosen randomly from that set.
Note that using this method for graphs will not allow us sample from inversion kernel in general, since the non-degeneracy of the corresponding energy function cannot be guaranteed. This problem related the difficulty of canonicalizing graphs (Babai & Luks 1983 McKay & Piperno| 201 4). However, luckily, the fact that we do not sample from an inversion kernel does not restrict our ability to represent arbitrary equivariant conditional distributions. Since we still sample ğ from an equivariant conditional distribution, Proposition 5.1 ensures that the representational power of the method is preserved. However, recall the conjecture that sampling from an equivariant conditional distribution of minimal entropy is best for learning (as noted in the main body, as well as motivating the search for "ideal" SBSs in Xie & Smidt (2024)). From this perspective, the closer a true/powerful graph canonicalization is used, the easier learning may be.
Continuous groups For continuous groups, in principle the energy minimization can be performed with gradient-based methods. It would, however. be impractical require optimization until con- vergence. If this is not the case, or if the energy is degenerate, we do not formally sample from an inversion kernel. However, we still sample ğ from an equivariant conditional distribution, which is compatible with Proposition 5.1 Heuristically, stochastic gradient- descent with a small learning rate is similar to Langevin sampling of the distribution (xp(-EEg-2a)) xT) where T' is related to the stochasticity of the optimization (Welling & Teh 2011). Moreover, some existing canonicalization methods are easily adapted to sampling from equivariant conditional distributions. For example, if one canonicalizes point cloud by defining coordinate axis based on the center of mass and two points of maximal radius, one can randomly sample these points of maximal radius in the case of a tie (which arises under self-symmetry).
21

---

Published as a conference paper at ICLR 2025
D BREAKING SYMMETRY IN DIFFERENT GROUPS
In this appendix, we show how to learn symmetry breaking biases v E V (such that G. is trivial) for different groups. For any group G, the first basic requirement for the group to act faithfully on V. We then want to choose V such that can initialize E V and obtain G. with probability I (with the assumption that v is initialized by sampling from an absolutely continuous distribution).
D.1 PERMUTATION GROUPS
For pernutation groups such as S, and p4m (the symmetry group of an image grid), the group admits faithful representation that maps permutation matrices acting on RR". We therefore choose V R".
In this case, suffices that all the elements of symmetry breaking bias v are different. This is captured by the following proposition.
Proposition D.I. Let G act faithfully by permutation on R" and E R" be such that Vs # 0g for any i + j. Then G is trivial. In addition, the set of y not satisfying this condition of measure zero with respect to the Lebesgue measure.
Proof. The proof of the first part of the proposition is trivial. For any E G. (gr); = ( Since the group action is faithful, for any E G except the identity, there is an E [n]] such that g (1) + i. For this i, Ug (() # Us therefore G. is trivial.
The second part follows the same idea as Proposition of Kaba & Ravanbakhsh (0033) Consider
the hyperplanes in R" defined H, {u E R 12 " Any e not satisfying the condition is an element of S = Hyg. A hyperplane Hep defines an (n 1) -)-dmeensional space in R" Any
subspace of R" of dimension strictly less than n. is of measure zero. The countable union of such subspaces S is also of measure zero.
D.2 SUBGROUPS OF THE GENERAL LINEAR GROUP
We also consider groups that admit faithful representations as subgroups of GL(n), such as O (n) for point clouds and atomic systems. In this case, we can choose the symmetry breaking bias to be linearly independent vectors, so that V RR
Raxe, Assume G acts on V as a product of faithful actions, e.g. gu - [gu'.... . gu") where v² the i-th column vector of If the vectors [v] ...... vn] are linearly independent, then G. is trîvial. In addition, the set of u such that this condition is not satisfied is of measure zero with respect the Lebesgue measure.
Proposition D.2. Let G C GL(n) and
Proof. For the first part, we identify the action of the group and matrix multiplication gv. For any g E G.,. it must be that gr = v. Since the columns of u are linearly independent, u is invertible. We therefore have gov v.- which implies g =
For the second part, the idea is similar to the proof of Proposition D.I above. If two columns of are not linearly independent, it implies that is element of subspace Hij {u E Rax= 3a R s.t. of = =av) for some and j. This is a subspace of measure zero since is of dimension n² n The union of all such subspaces S = Hi also of measure zero since it is countable.
D.3 REPRESENTATION OF JOINTLY EQUIVARIANT FUNCTIONS
Here we show more formally that the encoding of group elements using vectors allows represent any jointly equivariant function.
Proposition D.3. Let f X x G - y be any function jointly equivariant in its arguments, i.e. (ha, hg) = hf(fz g) & G. and let V be a vector space on which G acts. If v V has trivial stabilizer, then there exists an equivariant function fo X x [v] + y such that f(x, g) = fo(x @ gu).
22

---

Published as a conference paper at ICLR 2025
Proof. Consider any jointly equivariant f f: X G + V and suppose E V has trivial stabilizer. We may let fo(r @ u) = f(x,9) where g is the unique group element such that gr = u.
E THE INVERSION KERNEL INJECTS MINIMAL NOISE
We argue that using ø|X in place of independent Z E Z introduces the least amount of noise possible into the functional representation of Proposition 5.1 We will measure "amount of noise" by conditional entropy, assuming G is finite for ease of exposition. entropy is additive. it is minimized if Z is restricted to a single orbit, which (assuming G acts freely) isomorphic to G' itself. We thus consider Z as a G- v-luueed random variable.
a disjoint union of orbits. Since
First notice that if Z is independent of X, then it must be uniform on G to be equivariant. It is easy to show the uniform distribution maximizes entropy. To see inversion kernels minimize it, let If PP (Z is equivariant, then for any h E G. we have x) = p. Thus the support of PP ( = x)) least the size of G,. Entropy will again be minimized if (Z) X x)) indeed has a support of this size. This is exactly the case when Z X in fact distributed like ğ|_X, according to an inversion kernel.
g E G be such that IP (Z g X x)
(Z = hgg
F
ISING MODEL EXPERIMENTAL DETAILS
F.1 GROUND STATE OF THE ANISOTROPIC ISING MODEL
We present here an analytical derivation of the ground states of the anistropic Ising model. This is used obtain the ground-truth values for the average energy in Table and the ground-truth phase diagram in Fig. 5
and
The general form of the Hamiltonian of spin system with binary interactions is given by
H (a) Σ Jijooo
-hhor-
(30)
The family of anisotropic Ising models we consider given by
1(a) = Σ J'a,o Σ J" (FJ),
Σ harn
(31)
(15)
where Egir) indicates that the lattice sites i and j are nearest neighbors in the & direction. We also consider periodic boundary conditions. We can re-express the Hamiltonian using the variables Dg, by, [o,, E [1, - -I], --], taking values on horizontal and vertical interaction edges instead of the lattice sites. The variable encodes the value of the spins and adjacent to the bond. For convenience, we will write by as one-hot vector, with
if bf [1,0,0,0]
[1.
[0 0,
if [1.
bo
(32)
1,0
if
[- 1,
00,0,0,1]
if 6G
[-1, 1,
In terms of these variables, the Hamiltonian becomes
Ju
H(b) = - Σ
by
Σ
Ju
by
(by/ + by)
(33)
jy
2
Ju
For the change of variables correspond valid spin configuration, we additionally need to satisfy the constraints bij.0 = biy.o and b = bij,1 for any i, i.i'...j' Any configuration of variables satisfying such constraint will be an element of the feasible set B3
Finding the ground state therefore corresponds to the optimization problem arg min H(b).
Since the form of the Hamiltonian Eq. (33 is simple sum of non- con-interacting terms, first directly minimize to find the srouts-states without considering the constraint. We will then verify that the minimum lies in the feasible set B
23

---

Published as a conference paper at ICLR 2025
We first we-trite the Hamiltonian in the following way
J + h
JJ + h/2
H(b) = Σ
J²
by
(34)
- Jz
- Ju
J
h
f.
J h./2
We can then minimize each term independently to obtain the solution
0,00
Ja 2 h 1, 2
11
b =
1,0 or 0,
if J < - hb
(35)
if Ja 2 7 h 50
where {,,y). Without loss of generality consider h 0. This leaves us with four possibilities.
1. J 2 Ju - h, The ground states is given by b 11,0,000. which is in and corresponds the ferromagnetic (FM) phase (see Fig. 6a
4'
2. J < 1. 1 - 4. Feasible ground states consists of alternating between bi
[0, 1,0 0J and bij = [0.00 0, 0J. This is indeed in B and corresponds the antiferromagnetic (AFM) phase (see Fig. 6b).
3. J 2 - < - Feasible ground states consists of having byj 0, 0] and alternating between by [0, 0. 0] and by
[0,0, 1,0). This is indeed and
corresponds to the stripes phase (S²) (see Fig. 6c).
4. J J h. This the same as above but with stripes in the x direction.
4'
(a) Ferromagnetic phase
(b) Antiferromagnetic phase
(c) Stripes phase
Figure Illustration of the different ground-statee of the anisotropic Ising model
We can see that the different ground-states are associated with specific types of orders. The type of order of an arbitrary spin configuration a, which is sense its closeness the respective ground-states, can be quantified via order parameters (Beekman et al. 2019
The order parameters are given by
1 OPı = N Σ dj
OAE = N E-1) as
Og
N Σ
(-1)" a
(36)
where j and i are respectively the and positions of spin o, The order parameters take the value 1 if and only if a is the associated ground-state. In addition, Orm + +AAM + Ogv < 1, which means that the orders are mutually exclusive. This allows to draw meaningful phase diagrams like the ones in Fig. 5
F.2 HAMILTONIAN ENCODING
For the encoding the Hamiltonian interaction parameters as input to neural networks, a graph representation would be possible. with interaction parameters Jı, encoded as edge attributes and external field values as node attributes (see Fig. 7a). However, this choice would make it much more challenging the p4m symmetry of the Hamiltonian. We therefore leverage the structure of the
24

---

Published as a conference paper at ICLR 2025
(a) Graph encoding
(b) Image encoding
Figure 7: Data structures for the encoding of the Hamiltonian interaction parameters. J is represented in blue, J in yellow and the external field in green. The lattice sites of the spins are represented with black dots. In the image, gray pixels corresponding the "holes" between edges, are set O.
lattice and adopt an image representation (see Fig. 7b We can then conveniently use G-CNNs and MLPs as prediction networks.
We choose to set the size of the spin grid to 64 x 64, which corresponds to images of size 128 128. also encode the interaction parameters J and the transverse field across two different channels. The dimension of the inputs to the models is therefore [batch, 2, 128, 128]
In intermediary activations the neural network, we always preserve the size of the image. At the output of the network, obtain spin configuration by indexing the image pixels corresponding to lattice sites (with black dots on figure Fig. 7b). The energy is then computed over the lattice sites.
F.3 GROUP ACTION
We explicit here the action of the group påm (the symmetry group of the square lattice) on the Hamiltonian and on spin configurations.
To make the action clearer we unpack lattices indices variable in horizontal and vertical components as i (i i.) For E p4m, the action on spin configuration is given by (ar,)))) (g- The action on the indices permutes them.
The action on the Hamiltonian parameters is similarly given by gf( (i)( hg 1jz . 1a) and
(0(1,,))(( = Jig-11p 1/* 1.0) -o g (z)) -- practice, action by the group on the image encod- ing of the Hamiltonian is simply given by performing the transformation on the image (Fig. 7b).
For any Hamiltonian, acting on both the parameters and the spin configuration preserves the energy:
HaJah (go)
Σ Jig lin.g 12)),19 19 (9 11 120) 1j 15x)
(37)
Σ 11
which we see equals (o) due to the sum over i and j.
For the anisotropic Ising model we consider, the Hamiltonian parameters themselves are self- symmetric under the subgroup pmm, which includes translations, reflections and 2-fold rotations (but not 4-fold). In other words for any g' E pmm, g' J = J and g'h h h. This is easily seen from Fig. 7b the image encoding exhibits a wallpaper pattern with symmetry pmmm.
F.4 TRAINING SETUP
Unsupervised training is performed by having the neural network o J [0. 11aA output the probability that each spin is up by applying a softmax on the last layer; symmetry breaking elements sampled through canonicalization given by G-CNN as in Kaba et al. (203)). When there is
25

---

Published as a conference paper at ICLR 2025
tie in the canonicalization, element of the argmax set is chosen randomly. We use the EquiAdapt library (Mondal et al. 2 to implement the canonicalization. While training. we then compute the expectation value of the spin at each site and treat this as the configuration, using the Hamiltonian as a loss function. At evaluation, we sample spin value for each site using the probability output from the neural network.
We define a training set of Hamiltonian parameters as J = -1 and sampling J ~ Unif (-3,3) 3), h Unif (0, 2). with parameters constant over the lattice. The training set of size 1024. For the validation we also use 1024 samples. We consider Uwo test sets: one in-distribution (ID) test set of 10, 000 regularly sampled Hamiltonian parameter values in the same range. We also consider an out-of sittof--distibutiin (OOD) test set, which is the ID test set, augmented with rotations of the Hamiltonians parameters (J, h). For each test set example, with 0.5 probability we act with the group element corresponding to 90 degree rotation. Note that it is not necessary to consider other augmentations p4m because they will either be self-symmetries the Hamiltonian parameters (be elements of the coset of (J.))) will act in the same way as a 90 degree rotation (be elements of the coset of (J,)))), with g = 90%).
F.5 ENERGY RESULTS
We report energy results in Table 3 We see that for the ID test, the relaxed group convolutions method is slightly better than SymPE. However, for the OOD test set, SymPE becomes significantly better, This is due to the fact that SymPE retains an equivariance inductive bias compared to the relaxed convolutions methods which is not equivariant. Note that even for SymPE, the vanilla G-CNN and results for the OOD test set are slightly worst even if the methods model equivariant conditional distributions. This can be attributed to border effects. For the -ouiaiiivar MLP and the relaxed group convolutions, the decrease in performance is much more significant.
G-CNN+noise,
Table 3: Test set energies of predicted configurations
Method
Energy (ID)
Energy (OOD)
Parameters
Forward time (s)
Random configurations
0.0
0.0
MLP
-1.22
-0.93
3.2M
8 x 10
MLP + aug.
-1.24
-1.24
3.2M
x 10
G-CNN
-0.69
-0.69
397K
0.47
G-CNN noise
-1.28
-1.28
399K
0.49
Relaxed group convolution
-1.49
-1.42
463K
0.47
G-CNN SymPE (Ours)
-1.46
-1.47
468K
0.52
Ground truth
-1.60
-1.60
We also compare the parameter count and forward time (on Nvidia Quadro RTX 8000 GPUs with of the different models. We see that the computational overhead of SymPE is small compared the vanilla G-CNN.
G GRAPH DIFFUSION EXPERIMENTAL DETAILS
Our experimental setup follows exactly that of the original DiGress model, as described in Vignac jet al. (202)), with the same discrete denoising diffusion process and graph transformer architecture.
We used the QM9 dataset of small molecules, incorporating explicit hydrogen atoms, for evaluating the validity and uniqueness of generated molecular graphs along with the negative log-likelihood of the test set. For the MOSES dataset we report the negative likeliood not on the scaffold test set. The graphs were preprocessed similarly to the standard setup DiGress, where node features represent atom types and edge features represent bond types. We incorporated spectral features into the network to improve expressivity, following the methodology outlined the original paper. The positional encodings introduced by SymPE were concatenated to node and edge features during the diffusion process.
the withheld test set,
In the Table 4 report the negative likelihld results for MOSES.
26

---

Published as a conference paper at ICLR 2025
Table Negative log-likel for molecular generation on MOSES
Method
NLL
Parameters
Training time (h)
DiGress
65.9
16.2M
28.
DiGress + SymPE (ours)
30.4
16.3M
34.61
H GRAPH AUTOENCODER EXPERIMENTAL DETAILS
As discussed in the main body, autoencoders with "very symmetric" latent spaces pose a problem for equivariant models. In the graph auto-encoding setup of Satorras et al. (202T). self-symmeery of a graph with adjacency matrix A arises in the form of its automorphism group, (g E S, g A]. For example, the square graph in Figure has C4 automorphism group. When using a permutation- equivariant encoder e, e(A) must therefore also have at least C2 symmetry. Even though the goal of the -qquvvarann decoder d is to satisfy d(e(AA)) = A, which would not seem to explicitly require symmetry- -brakk.i... there is subtle problem: the choice of latent space Z. Precisely, Z does not contain any node-wise featurization with precisely Ci-symmetry: if {21..... ,2, E 2 have -4-symmetry, this implies that 2
24 | More abstractly, there exist graphs A, with stabilizer (automorphism group) GA, such that there is no Z E Z with Gz Ga: only GA Ç Gz. Thus, for these graphs, any equivariant encoder will produce an "overly self-symmetric" latent embedding. To reconstruct A, must therefore break the latent-space symmetry induced by the equivariant encoder
In our experiments, we followed the training hyperparameters of Satorras et al. (02I). but trained for fewer epochs (50) using their "erdos renyinodes 0 25 "ooooo"" dataset and the "AE" architec- ture. Moreover, later compare to breaking symmetries of the embedding (post-encoding) as well. [Satorras et al. (2021) suggest doing this with random noise as initial node features. However, this breaks all permutational symmetry, not just the graph's ' automorphism group. In contrast, our method is still equivariant to permutations of nodes that are automorphically equivalent. Although it is not feasible to exactly sample from the graph's 's automorphism group due to computational constraints, one can still hope to sample from subgroup of Sn, containing the automorphism group (i.e., an equivariant variable Z small support).
follow the setup of Satorras et al.) (2021) and break symmetries at the input, but
To apply SymPE, we must decide how to sample g. We compute nodewise embeddings using one graph convolutional layer, then let ğ be permutation that sorts the scalar embeddings. This method empirically works the best, but we also compare to heuristic without learned parameters based on Laplacian positional encodings. For this heuristic, the fourth largest singular values, P E R and then apply a Icarnable dimensionality reduction R² to obtain the vector y == Pw E RR". ÿ is obtained by sorting y. letting the sorting algorithm break ties, and the symmetry breaking input i = ğe is obtained by correspondingly sorting learned vector E RR. The symmetry breaking input ğv is again obtained by sorting v, letting the sorting algorithm break ties. Although the Laplacian eigenvectors are only determined up to sign (if the singular values are unique), and have further rotational ambiguity under repeated singular values, sampling from the possible space of such spectral encodings should still constitute an equivariant random variable Z.
truncate Laplacian positional encodings to
We also include some additional baselines in Table 5 none of which work as well as our proposed method. In both this table and in the main body, % Error is the same metric as used Satorras et al. (2021).
"The possible stabilizers of Z, the latent space of node-wise featurizations, are groups of the form S,, Siy where i +... + 4g = n. of course, one could also different latent space, such latent space of matrices. However, this may defeat the purpose of leamingg an expressive, dimensiionliiy-reduee latent space.
27

---

Published as a conference paper at ICLR 2025
Table 5: Full results of cross-entropy loss and reconstruction error
Method
BCE
% Error
# Param.
Uniform (embedding)
20.2
5.8
88,043
No SB (without input encoding)
19.03
6.4
88,017
SymPE with Laplacian, only channel
13.2
2.8
88,239
Noise (input)
10.1
2.3
88,017
No SB (with input encoding for fair comparison)
10.0
3.9
101,074
Noise (embedding)
9.7
3.8
101,075
SymPE with Laplacian
7.0
1.5
88,885
SymPE with Laplacian and noise (embedding)
6.7
1.5
88,891
Uniform (input)
5.7
1.3
88,890
Laplacian (directly input)
5.6
0.010
88,885
SymPE with learned features (ours)
3.7
0.078
101,750
28


Reviews:
[{"review_id": "yaUI5tD7zc", "reviewer": "Reviewer_oSLY", "summary": "The paper introduces a new method for breaking symmetries in equivariant neural networks through the probabilistic symmetries concept and randomized canonicalization. The proposed approach is implemented as a sampling from a canonicalization network, with additional positional encoding (SymPE), along with the equivariant network.", "questions": "There are some works on positional encodings for molecules beyond Laplacian, for example (1). Can you comment on that and how it might perform compared to SymPE?  \n\n1. Dwivedi, Vijay Prakash, Luu, Anh Tuan, Laurent, Thomas, Bengio, Yoshua, and Bresson, Xavier. \"Graph Neural Networks with Learnable Structural and Positional Representations.\" 2022", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper shows a theoretical motivation for the proposed method and extends previous works on equivariant distribution to tackle the symmetry breaking case. It also conducts evaluations on three tasks: graph autoencoder (using EGNN), graph generation (on QM9 using DiGress), and ground-states of the Ising model (using G-CNN), with improvement over the equivariant baselines.", "weaknesses": "The limitation of the method, as already mentioned, is that it requires sampling from a canonicalization function in addition to using equivariant networks, which might lead to increased complexity and additional computations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NzKkhc036C", "reviewer": "Reviewer_1aaz", "summary": "The paper proposes to sample a group element probabilistically to break the self-symmetry of an input to an equivariant network. This allows more flexible model outputs with lower symmetry, which is crucial for prediction and generation tasks with self-symmetries. The group element is sampled from an inversion kernel, which can be thought of as shifting the uniform distribution on the stabilizer of a data point $x$ by a learned canonicalization element $g = \\gamma(x)$. The network then takes the randomly sampled $g$ as input in addition to $x$ to break symmetry.", "questions": "* Eq (1): should it be this way or $P(Y | X = g^{-1}x) = P(gY | X = x)$?\n* Why are the results in Table 1 different from those in Satorras et al (Figure 5)? From my understanding, the \"No SB\" row in your table refers to an equivariant GNN without symmetry breaking and should match the EGNN performance reported by Satorras et al. If there are differences in settings that cause the discrepancy, they should be highlighted.\n* In Table 2, apart from NLL, other metrics are more similar across different models. How do you interpret this? Also, DiGress + SymPE (nodes only) has better validity vol. stability scores but is not highlighted. Is this intentional or a mistake?\n* L500: can you specify the action of $g \\in p4m$ on $(J, h, \\sigma)$?\n* L508: What does it mean by \"Hamiltonian parameters are randomly rotated by 90 deg\"?\n* Does your method have an advantage over G-CNN w/o symmetry breaking in predicting ground-states of Ising models partly because the interaction $J$ and the transverse field $h$ are initialized with self-symmetry? Would the results be different (maybe better for the baseline) if $J$ and $h$ are not constant on the lattice?\n* L1259: \"symmetry breaking elements are computed through a deterministic canonicalization\". If the group elements are deterministic, how do you break symmetry?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main idea of this paper is to take a randomly sampled group element as an additional input to the equivariant network to break the self-symmetry of an input. The proposed method is straightforward yet effective, tested on various tasks including graph reconstruction and generation, and the ground-state prediction of Ising models. The authors provide theoretical justifications for the method, explaining the motivation of key design clearly. There are also very helpful discussions on alternative methods to break symmetry, e.g. injecting totally random noise instead of that from a lower entropy.", "weaknesses": "* The analog to the positional encoding in Transformers is a bit confusing to me. In the setup of this paper, we want to break the **self-symmetry** of the data $x$, i.e. $gx=x$. However, in terms of sequence modeling, there is no self-symmetry in the data, i.e. if we permute the words in a sentence it would no longer be the same sentence. It is the symmetry of the **model** that we want to break. While you could still argue that your method also breaks the symmetry of the model in the sense that $f(gx) = gf(x)$ is not always true, in essence your method is still equivariant if you consider the joint action on the sampled group element. Also, the word \"position\" actually has different meaning in these two contexts. The positions in the Transformer PE refer to the indices of words in a sequence, i.e. **in** a single data point. On the other hand, in your method, the positions refer to the group element sampled from the inversion kernel, i.e. the hypothetical position **of** a data point on the group action orbit. I feel this is quite different from its literal meaning in the Transformer PE, and I don't feel this analog provides a good intuition.\n* Regarding the encoding of the group element, I wonder if it's better to define things in the language of group representations, since it's already based on the assumption that $G$ acts freely on $v$ by matrix multiplication. Also, I feel the explanation in the text is good enough and Prop 4.1 does not provide any new information.\n* Consider changing some notations, e.g. in L275 $e \\leftarrow \\tilde g v$, $e$ may be confusing because it can also refer to the identity element of the group.\n* The experiment section could benefit from more clarifications. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vnbnUClSrB", "reviewer": "Reviewer_XGk3", "summary": "The paper addresses the inability of equivariant neural networks to break symmetries in their outputs, which is a limitation for certain tasks. The authors present a theoretical framework for representing equivariant conditional distributions, allowing symmetry breaking via randomized canonicalization. They propose SymPE (Symmetry-breaking Positional Encodings), a method that expands the representational power of equivariant networks while retaining symmetry inductive bias. Experimental results demonstrate that SymPE significantly improves performance across various applications, including diffusion models for graphs, graph autoencoders, and lattice spin system modeling.", "questions": "I have no questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper considers an important but challenging task.\n2. The authors provide SymPE, a method motivated by both theoretical and empirical studies.\n3. Even though the paper has a theoretical contribution, it is not difficult to read.", "weaknesses": "Actually, the paper is solid and interesting. The key limitation admitted by the authors is that the proposed method requires access to a canonicalization method, and more general theoretical results are also desirable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Knt3vOjlNF", "reviewer": "Reviewer_JR9n", "summary": "The paper introduces a method for symmetry breaking in equivariant models, particularly focusing on equivariant GNNs. The motivation and relevance of this work are clear. However, while the idea and motivation are compelling, the paper itself is disorganized, unclear, and lacks important details. As it stands, I cannot recommend its acceptance, and significant revisions will be necessary for it to be considered.", "questions": "The overall structure of the paper is very poor, and I will highlight the most critical issues in each section.\n\n**Contribution**\nIn the contribution section, you need to clearly state your achievements. However, you provided an outline of the paper’s structure instead, which is typically referred to as an arrangement.\n\n**Background and main text**\n1. Are your distributions discrete or continuous? This distinction needs to be clearly stated, as it drastically changes the mathematics. Since you condition on $X=x$, it suggests discreteness, yet many of your sets appear infinite.\n2. What are the input and output random variables, X and Y? What are their input and output spaces? This is very unclear.\n3. There are no meaningful references to figures in the paper.\n4. The figure legends are unclear.\n5. In Equation 2, line 137, Y is a random variable, but your function f maps elements into $Y$. How does that make sense? Also, it’s unclear whether the probability is over X or over epsilon.\n6. In Theorem 3.4, equation 3, it’s not clear what the probability is being taken over.\n7. In the paragraph beginning at line 235, how do you compute this argmin efficiently over infinite groups or over the set of permutations?\n8. In my opinion, Section 5, which extends the discussion to noise, should be moved to the appendix, while expanding your existing work in the main body would be more valuable.\n9. Why is Theorem 6.1 necessary?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The paper presents a solid theoretical foundation on the subject and offers a well-motivated experiments section. The proposed method is supported by both theoretical insights and detailed experimental results.", "weaknesses": "The presentation of the paper is very poor. From the background through to the experiments, the notations and mathematical explanations are unclear throughout.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)