Structuring Representations Using Group Invariants
Authors: Mehran Shakerinava
Arnab Kumar Mondal
Siamak Ravanbakhsh
Venue: NeurIPS 2022 Accept (2022)
Keywords: Equivariance
Invariance
Geometry
Group Theory
Representation learning
Self-supervised learning

Abstract: A finite set of invariants can identify many interesting transformation groups. For example, distances, inner products and angles are preserved by Euclidean, Orthogonal and Conformal transformations, respectively. In an equivariant representation, the group invariants should remain constant on the embedding as we transform the input. This gives a procedure for learning equivariant representations without knowing the possibly nonlinear action of the group in the input space. Rather than enforcing such hard invariance constraints on the latent space, we show how to use invariants for "symmetry regularization" of the latent, while guaranteeing equivariance through other means. We also show the feasibility of learning disentangled representations using this approach and provide favorable qualitative and quantitative results on downstream tasks, including world modeling and reinforcement learning.

Structuring Representations Using Group Invariants
Mehran Shakerinava', Arnab Kumar Mondal', Siamak Ravanbakhsh McGill University and Mila, Montréal, Canada fmehran. shakerinava, arnab. mondal, siamak. @mi la quebec
Abstract
A finite set of invariants can identify many interesting transformation groups. For example, distances, inner products and angles are preserved by Euclidean, Orthog- onal and Conformal transformations, respectively. In an equivariant representation, the group invariants should remain constant the embedding as we transform the input. This gives a procedure for learning equivariant representations without knowing the possibly nonlinear action of the group in the input space. Rather than enforcing such hard invariance constraints on the latent space, we show how to use invariants for "symmetry regularization" of the latent while guaranteeing equivari- ance through other means. We also show the feasibility of learning disentangled representations using this approach and provide favorable qualitative and quanti- tative results on downstream tasks, including world modeling and reinforcement learning.
Introduction
Sample efficient representation learning critical open challenge in deep learning for AI. When we have prior information about transformations that are relevant to a particular domain, building representations that are aware of these transformations can lead better sample efficiency and generalization. One way to use such symmetry priors is to make the network invariant to the given transformations. A generalization of this idea called equivariance, where transforming the input transforms the output in specific way. An equivariant network that makes good predictions for particular input also generalizes all input transformations, making symmetry a useful prior.
While recent years have witnessed a range of exciting equivariant deep models, there are several limitations. First, most equivariant networks constrain the network architecture, often requiring specialized implementations. Moreover, transformations considered by the existing methods are often assumed to be linear in both input and representation space. This is the case for architectures designed for finite permutation groups and continuous Lie groups. Approaches that go beyond linear transformations in the input space often assume access group information i.e., the group member that transforms one input to another is known. This paper introduces a simple approach that addresses all of these limitations.
Our approach uses the invariants a given linear representation of a transformation group. Previ- used connect different geometries, and group theory in Klein's Brlangen
ously invariants program 32). According to this view, geometries are concerned with invariant quantities under certain transformations. For example, Fuclidean geometry is concerned with the length, angle, and parallelism of lines, among others, because Euclidean transformations preserve these. However, moving to the more general and less structured Affine geometry, notions of distance and angle no longer relevant, while parallelism remains an invariant of the geometry. The corresponding symmetry groups are examples of Lie groups that have subgroup relation, E(n) < Af f()), thereby enabling the groups to characterize a hierarchy (or lattice) of different geometries.
These authors contributed equally to this work.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

From this geometric perspective, our proposal in this work is to induce geometry on the embedding and make it equivariant to given group by enforcing the invariants of their defining action. For example, distance is the invariant for Fuclidean geometry, which means all Distance-preserving transformations are Fuclidean. Therefore, to enforce equivariance to the Fuclidean group, it is sufficient to ensure that the embedding of any two data points has the same distance before and after the same transformation of the inputs; see Figure]] [ While this approach uses the defining action of different groups in the embedding space, the same group can have a non-linear and unknown action on the input space. In the pendulum example of Figure I the group E(3) acts on the value of each input image pixel using unknown and -onear action. Moreover, this approach does not require the pairing of group members with transformations, a piece of information that is often unavailable.
Figure I: E(3) eqquivarir embedding for the pendu- lum. The input æ consists of pair images that identify both the angle and the angular velocity of a pendulum. The equivariant embedding learns to encode both: the true angle shown by a change of color and angular velocity using a change of brightness. The two circular ends (black and white) correspond to states of maximum angular veloc- ity in opposite directions. The SymReg objective for the Fuclidean group learns this embedding by preserving the pairwise distance between the codes before (f(z), f(a))) and after (f(tx (g. x)). {tx (g. x)) )) transformations of the input by 1x. Therefore dashed lines have equal lengths. For the pendulum, the transformations are in the form of applying positive negative torque in some range.
(x(a
In the rest of the paper, we arrive at the idea above from a different path: after reviewing related works in Section 2 and providing a background in Section 3 Section 4 observes that equivariance, in its general form, can be a weak inductive bias. This is because having an injective code is sufficient for equivariance "any" transformation group. However, in this manifestation of equivariance, the group action on the embedding can be highly non-linear. Since the simplicity of the action on the embedding seems essential for equivariance to become a useful learning bias, propossss proposes to regularize the group action on the code to make it "simple". This symmeiry regularization (SymReg) objective is group-dependent and the essence of our approach. Enforcing geometric invariants in the latent space is proposed as a symmetry regularization. While we focus on equivariant representation learning through self-supervision, in principle, supervised tasks can also benefit from the proposed SymReg. An important benefit of symmetry-based representation is its ability to produce disentangled representations through group decomposition (27) Section studies disentanglement using SymReg. Section 7 presents a range of experiments to understand its behavior and puts it in the context of comparable baselines.
2 Related Works
Finding effective priors and objectives for deep representation learning is an integral part of the quest for Al 3. Among these priors. learning equivariant deep representations has been the subject of many works over the past decade. Many recent efforts in this direction have focused on the design
of equivariant maps [57] 3
47
34
15
23
[54]
6] where the "linear" action of the group on the 19,
data is known. A particularly relevant cxample here is Villar et al. [54], which uses group invariants to construct equivariant maps where the group acts using its linear defining action in the input space. Due to this constraint, the application of these models has been focused on fixed geometric data such as images [3). sets [60] (5), graphs [39 33). spherical data and the (special) orthogonal
group 14 50 22 the Euclidean group [52 55 24 or other physically motivated groups such as
the Lorentz 4 or Poincare group [54], among others.
In the present work, the group action is unknown and possibly non-linear. Our setup is closer to the body of work on generative representation learning [7
40 in which the (linear) transformation Among these generative coding methods, transforming autoencoder 229 is closely related early work, which in addition to equivariance,
11
applied to the latent space [46] 58 35 16
21

---

seeks to represent the part-whole hierarchy in the data. What additionally contrasts our work with the follow-up works on capsule networks [48 38 is that SymReg is agmostic the choice of architecture and training. We only rely on our objective function to enforce equivariance.
Since we consider learning equivariant representations through self-supervision, exciting recent progress in this area is also quite relevant [25 42 53
61 20 4T. While the use of transforma- tions is prominent in these works, in many settings, the objective encourages invariance certain transformations, making such models useful for invariant downstream tasks such as classification. Similar to many of these methods, we also use transformed pairs to learn representation, with the distinction of learning an equivariant representation. An exception is the recent work of Dangovski al. [7]. which learns an equivariant representation by separating the invariant embedding from the pose, where the relative pose is learned through supervision. Therefore, in that work, in contrast to ours, one needs to know the transformation that maps one input to another. When considering the Euclidean group, SymReg preserves distances in the embedding space under -ineiiner transforma- tions of the input. This embedding should not be confused with isometric embedding [51], where the objective is to maintain the pairwise distances between points in the input and the embedding space.
26
3 Background on Symmetry Transformations
We can think of transformations set of bijective maps on domain X. Since these maps composable, we can identify their compositional structure using an abstract group G. For this reason, such transformations are called group actions. To formally define transformation groups, we first define an abstract group. A group G is set equipped with a binary operation, such that the set is closed under the operation gg' E G Vg, g e G. every g e G has unique inverse such that 9g -1 e, where is the identity element of the group, and the group operations are associative (gg') )g" g(g'g").
G-action on a set X is defined by a function f: G x X + X, which can be thought of as a bijective transformation parameterized by g e G. In order maintain the group structure, the action should satisfy the following two properties: (1) the action of the identity is the identity transformation t(e,)) x;; (2) the composition of two actions is equal the action of the composition of group elements (gg,( = t(gg', .r). The action t is faithful to G if transformations of X using each g G unique i.e., Vg. g Ex X s.t. x(g,x)= t(gg, .x). If a G-action is defined on a X, we call X a G-set, Many groups are defined using their defining action; for example, SO(3) is the group of rotations in 3D space. While this defining action is linear transformation, the same group can act non-linearly on Rº using the action t: S0(3) R" R".
4 Equivariance is Cheap, Actions Matter
A symmetry-basee representation embedding is a function f X - Z such that both X and Z are G-sets, and furthermore, f "knows about" G-actions, in the sense that transformations of the input using tx have the same effect as transformations of the output using some action tz:
(tx x)) tz (g. f(x)) Vg, GxX X
(1)
The following claim shows that despite many efforts in designing equivariant networks, simply asking for the representation to be equivariant is not a strong inductive bias, and we argue that the action matters. Put another way, the strong performance of existing equivariant networks should be attributed to the fact that the group action on the embedding space is simple (linear).
Proposition 4.1. Given transformation group tx G X X, the function f X Z is equivariant representation if Vg 6 G, X
f(x) = f(x') ((xx(g,=)) f(xx
That is, two embeddings are identical iff they are identical for all transformations.
The proof is in the appendix. The condition above is satisfied by all injective functions, indicating that many functions are equivariant to any group.
Corollary 4.2. Any injective function f X - Z is equivariant to any transformation group tx G X X. if we define G action on the embedding space as
tz(g,2) (()x) 2g.zExx
(3)
3

---

The ramification of the results above in what follows is two-fold:
1. While injectivity ensures equivariance, the group action on the embedding, as shown in Equation ( can become highly non-linear. Intuitively, this action recovers r f f=),,, applies the group action r' = 1x (z) in the imput domains and maps back to the embedding space f' to ensure equivariance.
In the following, we push iz towards simple linear G-action through optimization of f. This objective can be interpreted as a symmetry regularization or a symmetry prior (SymReg).
2. Although Corollary 4.2 uses injectivity of f for the entire X, we only need this for the data manifold. In practice, one could enforce injectivity on the training dataset D using a decoder, architectural choices such as momentum encoder [26], or loss functions defined on the training data, such as hinge loss [25] Lninge D) Ex,x'=zeD max (e- [f()) f(z')l .) or other losses that monotonically decrease with distance, such as [(=)) or its logarithm -Io((1)())) f(x')l). In experiments, we use the logarithmic barrier function.
Symmetry Regularization Objectives
In learning equivariant representations, we often do not know the abstract group and how transforms our data, fx We assume that one can pick reasonable abstract group G that "contains" the ground truth abstract group acting on the data- - i.e., action on the input may not be faithful. Our goal is to learn an f: X Z that is equivariant w.r.t. the actions (x lz, where Lx Gx X X unknown and Z is some (simple) G-action on Z of our choosing.
More Informed but Less Practical Setting. In the most informed case, the dataset also contains information about which group member y ε G can be used to transform x to x' -that is, the dataset consists of triples (I, g. = tx (9. 2)). By having access to this information, we can regularize the embedding using the following loss function: LG inforrmdd (f, D) = (xx.g..)) ((f(z))) - tg f())). where is appropriate loss function, such as the square loss. At its minimum, we have f(x:) (z. f()) or ((tx(g.x) = tz (g. f (x)), enforcing equivariance condition of Equa- tion D- However, even if the optimal value is not reached, due to its injectivity, is still G- equivariant, and the the objective above is regularizing the G action on the code. This informed setup used in equivariant contrastive learning of (17). The assumption of having access g is realistic when we know the action Lx, so that we can generate (I,, g, 2t) triplets. Fortunately, using group invariants, we may still learn an equivariant embedding, tied to the dataset.
if we do not have the group information
Here, introduce method for several well-known groups first and then claborate the more general treatment.
Example 1 (Euclidean Group). The defining action of the Euclidean group Enn) is the set of transfor- mations that preserve the Fuclidean distance between any two points in IR. a.k.a. isometries. These transformations are compositions of translations, rotations, and reflections. Since, for the real domain, all Euclidean isometries are linear and belong to F(n), we can enforce the group structure on the em- bedding by ensuring that distances between the embeddings before and after any transformation match. For this, we need the dataset D to be a set of pairs of pairs ((x,I, = fx (g. x)), where x, x' are transformed using the same unknown group member below combined with injection loss are sufficient produce an E'n) rregulreulr embedding:
) (æ', tx (g. x' )))) preservationnn loss
before transformation
distange after the transformation
LE(M) (f.) ((2)) f(x)))
(4)
For example, in the standard RL setup, where we have access to triplets (s, (s,a, s'), can easily form D by unrolling an episode and collecting two different state transitions corresponding to a particular action. In practice, with a finite number of actions, we efficiently generate this dataset by keeping a separate buffer for each action where we store state transitions for that action and sample from that buffer to train the embedding function f. We provide the algorithm in Appendix C.
Example 2 (Orthogonal and Unitary Groups). The defining action of the orthogonal group O(n) preserves the inner product between two vectors. The analogous group in the complex domain is the

---

unitary group, which preserves the complex inner product. Our rermmmtty-regulariaation objective enforces this invariant: Lo( D) Σ {(f(x)) f(=')~ - f(x))' f(x()) )
(
For the unitary group, one additionally needs to embed complex domain Z = C", where the only difference is in the definition of the inner product.
Example 3 (Conformal Group). The invariant of conformal geometry is the angle. In a Euclidean embedding, conformal transformations include a combination of translation, rotation, dilation, and inversion with respect to an 1-sphere. To enforce this group structure, we need triplets of inputs before and after transformation (1), (z', xx)), (x", xx()), so that we can calculate the angle in the embedding. Conformal SymReg objective, which preserves angles, imposes a weaker constraint on the embedding than the distance preservation of the Euclidean group since the latter implies the former. Moreover, has an additional benefit that, compared LF(n). the loss cannot be minimized by simply shrinking the embedding. Therefore in practice, the injection enforcing losses of Section not as crucial when using conformal symmetry regularization.
5.1 General Setting
Given a group G' acting linearly a vector space Z, invariant polynomials associated with this action are those polynomials satisfying ((lz(z,g) g)) = P(z) Vg E G. These polynomials form an algebra studied in the field of invariant theory [56 44 In particular, a relevant problem is the question of whether there exists a finite set of bases for invariant polynomials for a given group representation. This question was one of Hilbert's 23 problems, and it was answered affirmatively by Hilbert himself for linear reductive groups, which includes classical Lie groups (28). Our proposal, in its most general form is to ensure invariance of polynomial bases within the orbits of the latent space before-after transformation of the input.
Some examples of classical Lie groups and their invariants are: volume and orientation preservation by the Special Linear group, where the corresponding invariant polynomial is the determinant; Lorentz and Poincare groups are the analogs of the Orthogonal and Euclidean groups in the Minkowski space respectively, therefore equipped with similar invariants; the Symplectic group preserves another bilinear form. Finite groups also possess invariants. We show this use of invariants for SymReg through the important example of the symmetric group.
Example 4 (Symmetric Group). Symmetric polynomials P(zı,..., zn) that are invariant under all permutations of variables have finite set of elementary bases:
cI = Σ Zzi E2(2)= = Σ 1sysn
2yZks
en (z) = 21.2...
1sjcken
Assuming an n-dimensional embedding (i.e. 2j RR), the corresponding SymReg objective penalizes change in these elementary basis before-after transformation. At its minimum value, this penalty ensures that transformations of the input lead to permutations of the latent dimensions however, with SymReg, this loss is used only to regularize the embedding. An alternative approach to SymReg for finite groups and, in particular, the Symmetric group is discussed in Appendix B.
Choice of Lie group Deciding on Lie group for each application and in particular working with the corresponding invariants can be cumbersome. A simple alternative is to use an E(n) (eqi--ravvr embedding for sufficiently large n. This because Lie groups have isometric Euclidean embedding for sufficiently large n. We demonstrate this in the experiments with SO(3) group in Section 7.1
6 Decomposing the Representation
Higgins et al. 2 suggested a notion of disentangled representation based on decomposition of the abstract group into direct product form G G1 x.. Gt. There are two approaches learning such decomposed representation using SymReg, depending on whether or not we can perform certain types of transformations in isolation. For example, an RL agent may transform its environment through actions like moving single limb that can be performed in isolation. In this case, we call the decomposition active to contrast it with the passive case, where the action of different subgroups always mixed in our dataset.
5

---

Figure 2: Visualization of SymReg's latent pro- jection for the rotating Chair dataset. The chair is rotated in three orthogonal axes from 0 to 2n. The latent embedding for each chair pose pro- jected from 16D embedding space to 2D space for visualization. The colors of the representations are mapped to the chair's is angle of rotation. We notice that the mapping function learned is tinuous with respect the transformations of the object, and it maps the rotations along an axis to a circular manifold. This is true for cach orthogonal axis of rotation. We observe a similar result for any other initial pose for the chair. qualitative comparison with an existing method [25] pro- vided in Appendix D.I.
J:
Active Decomposition. Let ((91......93 gb) E G1 x x... G be a product group, where G, = e,..., .) G) can be identified with normal subgroup of G. In active decomposition, (e,,..,,,, Gi. e,
sub-groups can act in isolation, and therefore we have k types of tuples our dataset DI...., D3 c D. Each subset D; is associated with actions of a subgroup G; using tx ((e...., C.gi, ....,.)..)) 9. 6 G, In this setting, the representation f X + = Z1 x x Zμ can be thought of as separate functions where fi X Z; is equivariant G--action and invariant to all Gy.J actions. This gives the following objective
G (J,
Σ (J, D.)
G.G.. (Ji. D(D),,
(5)
equivariance to G,
invariance to Gj for jai
where Li Df.D) enforces invariance of f to G-transformations in D - e.g. by penalizing I1(x)) ((x((gx)))
Passive Decomposition. When we have no control over transformations, and we are simply given the data, it is still possible to use an abstract group that has product form. Here again, f:X X + Z Zi x x Zu. but the loss function is simply enforced on each block separately i.e. p. (J, D) Σi La, (fi, D), where LG, (Ji, D) is SymReg objective from Section
7 Experiments
We conducted many experiments to qualitatively study the representation learned by SymReg and its ability produce disentangled representation, and quantitatively compare against simple baselines in representation learning and downstream RL tasks. For details on architecture and training, see Appendix G.
7.1 Qualitative Analysis
In this section, we visualize the representation learned for two examples from the Gym environment [5], including the pendulum and the mountain car (see Appendix D.2), followed by an experiment involving rotating object where we know the ideal embedding is the SO(3) manifold. Finally, Figure visualizes a conformal embedding for double-bump world. In most cases, we also visualize a Variational AutoEncoder ( (VAE) [30] embedding for comparison. Our objective here is to visually demonstrate the behavior of SymReg and its remarkable ability to learn an embedding informed by the non-linear transformation of the input.
The Pendulum. For this experiment, the input x is two consecutive frames of the pendulum that have been grayscaled and downsampled to 32 32 pixels. The action space is range of torques

---

that can be applied to the base of the pendulum. We use the action to transform the data. We use the objective of Equation (4) to learn an (equ-vaaivaii representation. To efficiently estimate L we use a mini-batch that consists of 64 randomly sampled observations from the environment and their transformations via three randomly sampled actions (4 x 64 samples in total). The model learns to parameterize the embedding using the angle and the angular momentum of the pendulum from the input data; see Figure | experiments the Mountain Car Environment in Appendix D.2).
In order to compare, we visualize the learned latent of VAE and run similar
Rotating Chair. We consider a 3D chair from ModelNet40 [59 and transform it through the action of the group (3).. The group action on the input is the 2D projection into a 48 x 48 image after the 3D rotation of the chair. While the group of interest is S0(3), we use SymReg loss of Equation (4)
following Section 5.1 We embed the chair in R using SymReg and visualize the latent by rotating the chair along three orthogonal axes and projecting the latent codes into a 2D space. Figure 2 shows three circular latent traversals of SymReg embedding corresponding to rotation around each axis, which is consistent with the structure of the SO(3) manifold. The process of learning the (3(3) manifold is challenging task (see Appendix D.1), and previous works assumed that the group member corresponding to each transformation is given [46 []. In contrast, we only use the observations corresponding similar actions during training and not the group members themselves.
As we see later, this is critical in settings such as RI,, where group information is unavailable.
Conformal Embedding for Double- Bump World. Double-bump world consists of a rectangular bump sig- nal and a triangular bump signal, both cyclically shifted and superimposed. These transformations are given by a pair (A1, A2) which cyclically shifts the rectangular bump by As and the triangular bump by 22. In our experi- ments, the signal length is 64, and the length of the bump is 16. SymReg em- beds to a 4-dimensional conformal space in this example. right) shows the random projection of the embedding, where the colors change as the triangle bump moves. The figure suggests that SymReg can learn to represent data point using the location of two bumps. For comparison, we show the embedding found by VAE and SimCLR 9.
MORKING MONEONN
Figure 3: (left) SimCLR (middle) VAE (right) SymReg embed- dings of double-bump world.
7.2 Experiments on Active and Passive Decomposition
In this section, we first contrast active and passive decomposition in their ability to disentangle the two bumps in the double bump world. We observe that while both can decompose the embedding into a product form SO(2) x SO(2), only active decomposition leads to disentanglement. Finally, we apply active decomposition to the more complex setting of ego-motion, where SymReg can decompose the representation of the agent's 's state into location and orientation.
Decomposition of the Double-Bump World. Here, we compare the active and passive decompo- sition for the same double-bump world. While the ground truth is S0(2) x SO(2), SymReg uses the larger group E(2)= x E(2). In the active case, each subgroup moves one of the bumps, and the loss of Equation 5 is used to learn an embedding for each subgroup. In the passive case, both bumps move randomly. Figure compares comparess the decomposed embedding found in each case. While in both cases, the 50(2)= SO(2) torus is decomposed into a product of circles, only the active case successfully disentangles the two bumps. Note that the color of each point is based on the location of the triangle bump. Our results agree with Caselles-Dupré Dupré et al. [ who claim that learning disentangled representation requires interaction with the environment; see also 43 40). However, we note that while the disentangling of the bump movements does not happen in the passive case, we can still successfully "decompose" the embedding.
Active Decomposition for Ego-Motion. We used modified version of the single-room envi- ronment of MiniWorld [12 for this experiment. The agent is standing in a 3D room containing eight differently colored boxes around the walls. A map of the room can be seen in Figure 5 An
7

---

passive
active
-13
-20
-20
1.0
1.3
100
1.03
0.75
C
L
It
I
Figure 4: Active versus passive decomposition for the double-bump world. Four images the bottom show a pair inputs (æ, x²) and their transformations (1), The figure shows the embedding for two inputs before, (Ir, ) in blue, and after, (I,, in red, the same transformation. This transformation cyclically shifts both the triangle and the square the left, but the amount of translation is larger for the square. In both passive and active decomposition, the Euclidean distance is preserved by the transformation the red points have the same distance from each other the blue points every manifold. In active decomposition (right), one the manifolds encodes the circular translation the triangle bump, while the second one represents the location of the square bump. Various colors indicate the location of the triangle. In the case of passive decomposition (left), since the transformation of individual shapes does not guide the decomposition, the manifolds jointly encode the location of each bump type.
Figure 5: Decomposition of the ego-motion. The dataset contains first-person view of room. Transformations in- clude right-left rotation and The equivariant embedding is produced by active decomposi- tion using these two transformations, where the ring-structured manifold corresponds the rotation action, and the smaller manifold corresponds translations. Color coding shows the ground truth angle of the image. The black square markers show the traversal of the embedding as the agent rotates while standing in the middle of the room. Note that black squares are concentrated in the center of the second manifold.
Map
movement.
observation consists of a first-person view of the room, downsampled to 32 x 32 pixels. The agent rotate left/right move forward/backward. We learn F2(2) F(2) equivariant embedding using the active decomposition objective of Equation (5). Each mini-batch consists of 64 random observations and the result of applying all four actions those states (4 64 samples in total).
Figure 5 visualizes the embedding of the input in two sub-figures, where the more prominent figure shows the embedding corresponding to the rotation action, and the more petite figure (bottom right) shows the embedding corresponding to horward-bacwwan movement. The first figure also shows the first-person view when the agent rotates while standing in the middle of the room. The corresponding markers collapse around the center of the second embedding, demonstrating an intuitive embedding parameterized by rotation angle and location. Walking straight across the room also produces the expected behavior of traversing the second manifold while the rotation angle, for the most part, remains fixed (not shown).
7.3
Quantitative Evaluation in Downstream Tasks
7.3.1
World Modelling
We select the Atari games Pong and Space Invaders as our environments for the world modeling experiments. These environments were previously used by Kipf et al. [3]] to evaluate the Contrastive

---

ENVIRONMENT
METHOD
H@@
MRR
WORLD MODEL(AE)
23.8.
44.7+24
WORLD MODEL((AE)
1.0+
5.1+0.1
ATARI PONG
C-SWM
36.5+ 5.6
56.2+62
OURS
45.2. 3.4
60.2.3.
WORLD MODEL(AE)
40.2+
59.6+
WORLD MODEL(VAE)
1.0=5.3
5.3
SPACE INVADERS
C-SWM
48.5=
66.
OURS
54.2.
68.7.
Table 1: Hits Rank (H@)) and Mean Reciprocal Rank (MRR) different methods.
METHODS
INVERTED PENDULUM
REACHER
SWIMMER
VANILLA
500 150
-11.25
25.6+ 344
AR-DECOUPLED
30- 15
-13.3a
16-
AE-FINETUNED
580s
-11.2a22
262
EN-SSL-DECOUPLLE
100. 17
-15226
12.5
IN-SSL-FINETUNED
550:21
-1224.1
25.9 14.8
EQ-SSL-DECOURE
456x 190
-14.8
18 4,5
EO-SSL-FINETUNED
710 120
-10-26
27
800: 180
-14.5
21=4.1
DEC-SYMRECH SY-EEE-EOUUUULLEE
600. 200
-12.8x27
194
SYMREG-FINETUNED
950. 50
-10134
31.5.39
Table 2: Average reward collected over 10 episodes for various models in Inverted Pendulum, Reacher and Swimmer. We provide the standard errors using 5 random seeds.
Structured World Model (C-SWM). We train the encoder using Euclidean SymReg of Equation 4.. freeze it, and then learn a Multi-Layer Perceptron (MLP) based transition function in the latent space. Following Kipf et al. [31], report Hits at Rank 1 (H@) 1) and Mean Reciprocal Rank (MRR), which are invariant to the embedding scale. These evaluation metrics measure the relative closeness of the following state's representation predicted by the transition model and the representation of the observed next state. We use set of reference state representations to measure the relative closeness (embedding random observations from the experience buffer). Section 7.3.2 reports these measures and shows that a simple transition model learned on top of our embedding outperforms C-SWM in both games. Other reported baselines use AutoEncodcer (AB) and Variational AutoEncoder (VAE) to learn embeddings.
7.3.2 Reinforcement Learning
Next, consider three Mujoco environments: InvertedPendullm, Reacher, and Swimmer from OpenAl Gym 5 and learn directly from the image observations. We compare our model with Auto- Encoder (AE) and Self-supervised Learning (SSL.) based baselines. While AE learns to reconstruct the image observations the states, SSL learns to inject invariance (IN-SSL) or equivariance (EQ- SSL) to agent actions. Given a triplet (s, a, s). IN-SSL maximizes the likelihood of f(s) and f(s') being similar (SimCLR [OO.. EQ-SSL of Dangovski et al. [18), in this context, additionally predicts the action that leads the state transition. We introduce variation, the -iw-dimeniionl embedding is used as substitute for the high- nigh-dimensional input data without further adjustment (-decoupled). The second variation allows for fine-tuning during the reinforcement learning stage (-fine-tuned) We use random policy to collect trajectories for the pre-training and use Proximal Policy Optimization (PPO) 49 algorithm for the downstream task. To evaluate the data efficiency of these models, report the average reward collected over 10 episodes the first 100,000 steps for Reacher and Swimmer and 30,000 steps for Inverted Pendulum in Section 7.3.2 (since Inverted Pendulum generally learns faster, took fewer number of steps.)
variations of each model. In the first

---

We see that out of all the representation learning methods, learned representations of SymReg most adequately capture the structure of the environment in Inverted Pendulum since the RL agent just trained on the fixed representation PPO. Reacher, SymReg, along with other non-gen non-generative models, performs poorly compared to the AE. We believe that this is because the representation is focused on transformations caused by the agent's s actions while details that can be valuable from the reward's perspective - in this case, the small object that the Reacher should reach are ignored. This observation points limitation of all non-generative approaches that fine-tuning can resolve. To further verify this, we combined SymReg with Decoder (Dec-SymReg) and noticed a significant improvement in the performance of the decoupled variation. In Swimmer, again, we see that learning the agent's 's transformations is not enough to get all the reward information as the background movement decides how far the agent has swum. Indeed, allowing the encoder to fine-tune allows the representations to reflect the reward information and improve performance.
outperforms all of them, including vanilla
Conclusion
We proposed learn equivariant representations by learning an injective embedding that regularized towards a simple linear action using group invariants. We demonstrate this to be simple, intuitive, and yet effective approach for representation learning. In the future, we would like to understand data characteristics that motivate the choice of one Lie group over others. We would also like to explore SymReg for the symmetric group and its combination with Fuclidean groups as a way to represent various objects in Euclidean space. We would also like to investigate further the best choice of objectives or mechanism for preventing a representation collapse in conjunction with SymReg.
Acknowledgments
We would like to thank anonymous reviewers for their constructive feedback. This research was in part supported by CIFAR AI chairs program and NSERC Discovery grant. The computational resources were provided by Mila and Compute Canada (now Digital Research Alliance of Canada). As a part of Mila, the authors acknowledge the material support of NVIDIA in the form of computational resources and IDT team and their technical support for maintaining the Mila compute clusters.
References
[1] Anderson Brandon, Hy Truong Son, Kondor Risi. Cormorant: Covariant Molccular Neural Networks // Advances in Neural Information Processing Systems. 32. 2019. 14537-14546.
[2] Anonymous. . Learning Symmetric Representations for Equivariant World Models Submitted to The Tenth International Conference on Learning Representations. 2022. under review.
[3] Bengio Yoshua, Courville Aaron, Vincent Pascal. Representation learning: A review and new perspectives // IEEE transactions on pattern analysis and machine intelligence. 2013. 35, 8. 1798-1829
[4] Bogatskiy Alexander, Anderson Brandon, Offermann Jan, Roussi Marwah, Miller David, Kondor Risi. Lorentz group equivariant neural network for particle physics // International Conference on Machine Learning. 2020. 992-1002.
[5] Brockman Greg, Cheung Vicki, Pettersson Ludwig, Schneider Jonas, Schulman John, Tang Jie, Zaremba Wojciech. Openai gym // arXiv preprint arXiv:1606.01544 2016.
[6] Bronstein Michael M, Bruna Joan, Cohen Taco, Veličković Petar. Geometric deep learning: Grids, groups graphs, geodesics, and gauges // arXiv preprint arXiv:2104.13478. 2021.
[7] Burgess Christopher P. Higgins Irina, Pal Arka, Matthey Loic, Watters Nick, Desjardins Guil- laume, Lerchner Alexander. Understanding disentangling in beta-VAE // arXiv preprint arXiv:1804.03599, 2018
[8] Caselles- Dupré Hugo, Garcia-Orit Michael, Filliat David. Symmetry- based disentangled rep- resentation learning requires interaction with environments arXiv preprint N03i:1994.0022440 2019.
10

---

[9] Chen Ting, Kornblith Simon. Norouzi Mohammad, Hinton Geoffrey. A simple framework for contrastive learning of visual representations International conference on machine learning. 2020. 1597-1607.
[10] Chen Ting, Kornblith Simon, Norouzi Mohammad, Hinton Geoffrey E. A Simple Framework for Contrastive Learning of Visual Representations CoRR. 2020. abs/2002.05709.
[11] Chen Xi, Duan Yan, Houthooft Rein, Schulman John, Sutskever Ilya, Abbeel Pieter. Infogan: Interpretable representation learning by information maximizing generative adversarial nets // arXiv preprint
2016.
[12] Chevalier-Boisverl Maxime. gym-miniworld environment for OpenAl Gym. 2018.
[13] Cohen Taco, Welling Max. Group equivariant convolutional networks // International conference on machine learning. 2016. 2990-2999.
[14] Cohen Taco S., Geiger Mario, Köhler Jonas, Welling Max. Spherical CNNs // International Conference Learning Representations. 2018.
[15] Cohen Taco S, Geiger Mario, Weiler Maurice. A General Theory of Equivariant CNNs on Homogeneous Spaces Advances in Neural Information Processing Systems. 32. 2019. 9145- 9156.
[16] Cohen Taco S, Welling Max. Transformation properties of learned visual representations arXiv preprint
2014.
[17] Dangovski Rumen, Jing Li, Loh Charlotte, Han Seungwook, Srivastava Akash, Cheung Brian, Agrawal Pulkit, Soljačíć Marin. Equivariant Contrastive Learning arXiv preprint arXiv:21 1t::22111.009999 2021.
(18) Dangovski Rumen, Jing Li, Loh Charlotte, Han Seungwook, Srivastava Akash, Cheung Brian, A Agrawal Pulkit, Soljacic Marin. Equivariant Self-Superviee I Learning: Encouraging Equivari- ance in Representations // International Conference Learning Representations. 2022.
[19] Dehmany Nima, Walters Robin, Liu Yanchen, Wang Dashun, Yu Rose. Automatic Symme- try Discovery with Lie Algebra Convolutional Network // Advances in Neural Information Processing Systems. 2021.
[20] Ermolov Aleksandr, Siarohin Aliaksandr, Sangineto Enver, Sebe Nicu. Whitening for self- supervised representation learning // International Conference on Machine Learning. 2021. 3015-3024.
[21] Falorsi Luca, Haan Pim de, Davidson Tim R, De Cao Nicola, Weiler Maurice, Forré Patrick, Cohen Taco S. Explorations in homeomorphic variational auto-encoding arXiv preprint N1t::1807.04688 2018.
[22] Finkelshtein Ben, Baskin Chaim, Maron Haggai, Dym Nadav. A simple and universal rotation equivariant point-cloud network // arXiv preprint arXiv:2203.01216 2022.
[23] Finzi Mare, Welling Max, Wilson Andrew Gordon. Practical Method for Constructing Equiv- ariant Multilayer Perceptrons for Arbitrary Matrix Groups // arXiv preprint arXiv:2104.09559 2021.
[24] Fuchs Fabian, Worrall Daniel, Fischer Volker, Welling Max. Se (3)-transformers: 3d roto- translation equivariant attention networks // Advances in Neural Information Processing Systems. 2020. 33. 1970-1981.
[25] Hadsell Raia, Chopra Sumit, LeCun Yann. Dimensionality reduction by learning an invari- ant mapping 2006 IEEE Computer Society Conference on Computer Vision and Pattern
Recognition (CVPR 06). 2. 2006. 1735-1742.
[26] He Kaiming, Fan Haoqi, Wi Yuxin, Xie Saining, Girshick Ross. Momentum contrast for unsupervised visual representation learning Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. 9729-9738.

---

[27] Higgins Irina, Amos David, Pfau David, Racaniere Sebastien, Matthey Loic, Rezende Danilo, Lerchner Alexander. Towards a definition of disentangled representations // arXiv preprint urXiv:1812.022360 2018.
[28] Hilbert David, David Hilbert. Theory of algebraic invariants. 1993.
[29] Hinton Geoffrey E, Krizhevsky Alex, Wang Sida D. Transforming auto-encoders // International conference on artificial neural networks. 2011. 44-51.
[30] Kingma Diederik P. Welling Max. Auto-encoding variational bayes arXiv preprint arXiv:1312.6144 14. 2013.
[31] Kipf Thomas, Pol Elise van der, Welling Max. Contrastive Learning of Structured World Models // International Conference Learning Representations, 2020.
[32] Klein Felix. comparative review of recent researches in geometry // Bulletin of the American Mathematical Society. 1893. 2, 10. 215-249.
[33] Kondor Risi, Son Hy Truong, Pan Horace, Anderson Brandon, Trivedi Shubhendu. Covariant compositional networks for learning graphs // arXiv preprint arXiv:1801 .01144. 2018.
[34] Kondor Risi, Trivedi Shubhenda. On the generalization of equivariance and convolution in neural networks to the action of compact groups // International Conference on Machine Learning. 2018. 2747-2755.
[35] Kulkarni Tejas D, Whitney Will, Kohli Pushmeet, Tenenbaum Joshua B. Deep convolutional inverse graphics network arXiv preprint arXiv:1503.03167. 2015.
[36] LeCun Yann, Bengio Yoshua, others Convolutional networks for images, speech, and time series The handbook of brain theory and neural networks. 1995. 3361, 10. 1995.
[37] Lenc Karel, Vedaldi Andrea. Learning covariant feature detectors // European conference on computer vision. 2016. 100-117.
[38] Lenssen Jan Eric, Fey Matthias, Libuschewski Pascal. Group Equivariant Capsule Networks Advances in Neural Information Processing Systems. 31. 2018. 8844-8853.
[39] Maron Haggai, Ben-Hamu Heli, Shamir Nadav, Lipman Yaron. Invariant and equivariant graph networks // arXiv preprint arXiv wtt::1812.0090 812.09902. 2018
[40] Mita Graziano, Filippone Maurizio. Michiardi Pietro. An identifiable double VAE for disentan- gled representations // International Conference on Machine Learning. 2021. 7769-7779.
[41] Mondal Arnab Kumar, Jain Vineet, Siddigi Kaleem, Ravanbakhsh Siamak. EqR: Equivariant Representations for Data-Efficient Reinforcement Learning // Proceedings of the 39th Interna- tional Conference on Machine Learning. 162. 17-23 Jul 2022. 15908-15926. (Proceedings of Machine Learning Research).
[42] Oord Aaron van den, Li Yazhe, Vinyals Oriol. Representation learning with contrastive predictive coding // arXiv preprint arXiv:1807.03740 007448.77..8.... 2018
[43] Painter Matthew, Hare Jonathon, Brugel-Bennet/ Adam. Linear disentangled representations and unsupervised action estimation // arXiv preprint arXiv:2008.07922 2020.
[44] Popov Vladimir L, Vinberg Ernest B. Invariant theory // Algebraic geometry IV. 1994. 123-278.
[45] Qi Charles R, Hao, Mo Kaichun, Guibus Leonidas Pointnet: Deep learning on point sets for 3D classification and segmentation // Proceedings of the IEEE conference computer vision and pattern recognition. 2017. 652-660.
[46] Quessard Robin, Barrett Thomas D, Clements William R. Learning group structure and dis- entangled representations of dynamical environments // arXiv preprint arXiv:2002.06991. 2020.
12

---

[47] Ravanbakhsh Siamak, Schneider Jeff. Poczos Barnabas. Equivariance through parameter-sharne sharing // International Conference on Machine Learning. 2017. 2892-2901.
[48] Sabour Sara, Frosst Nicholas, Hinton Geoffrey Dynamic Routing Between Capsules Advances in Neural Information Processing Systems. 30. 2017. 3856-366..
[49] Schulman John, Wolski Filip, Dhariwal Prafulla, Radford Alec, Klimov Oleg. Proximal Policy Optimization Algorithms. 2017.
[50] Shakerinava Mehran, Ravanbakhsh Siamak. Equivariant Networks for Pixelized Spheres International Conference on Machine Learning. 2021. 9477-9488.
[51] Tenenbaum Joshua B, De Silva Vin, Langford John C. global geometric framework for nonlinear dimensionality reduction // science. 2000. 290, 5500. 2319-2323.
[52] Thomas Nathaniel, Smidt Tess, Kearnes Steven, Yang Lusann, Li Li, Kohlhoff Kai, Riley Patrick. Tensor field networks: Rotation-and translation-qquivrraan neural networks for 3D point clouds |/ arXiv preprint arXiv:1882.08221. 2018.
[53] Tian Yonglong, Krishnan Dilip, Isola Phillip. Contrastive multiview coding // arXiv preprint arXiv:1906.05849. 2019.
[54] Villar Soledad, Hogg David, Storey-Fisher Kate, Yao Weichi, Blum-Smith Ben. Scalars are universal: Equivariant machine learning, structured like classical physics Advances in Neural Information Processing Systems. 2021. 34.
[55] Weiler Maurice, Cesa Gabriele. General Equi-iuuivar Steerable CNNs // Advances in Neural Information Processing Systems. 32. 2019.
[56]
Weyl Hermann. The classical groups: their invariants and representations. 1946.
[57] Wood Jeffrey, Shawe-Tayle John. Representation theory and invariant neural networks Discrete applied mathematics. 1996. 69, 1-2. 30.
[58] Worrall Daniel E, Garbin Stephan J. Turmukhambetov Daniyar, Brostow Gabriel J. Interpretable transformations with encoder-decoder networks // Proceedings of the IEEE International Conference Computer Vision. 2017. 5726-5735.
[59] Wu Zhirong, Song Shuran, Khosla Aditya, Fisher, Zhang Linguang, Tang Xianou, Xiao Jianxiong. 3d shapenets: A deep representation for volumetric shapes // Proceedings of the IEEE conference on computer vision and pattern recognition. 2015. 1912-1920.
[60] Zaheer Manzil, Koturr Satwik, Ravanbakhsh Siamak, Poczos Barnabas, Salakhutdinov Russ R, Smola Alexander J. Deep Sets // Advances in Neural Information Processing Systems. 30. 2017. 3391-3401.
[61] Zbontar Jure, Jing Li, Misra Ishan, LeCun Yann, Deny Stéphane Barlow twins: Self-supervised learning via redundancy reduction // arXiv preprint arXiv:2103.0322 2021.
13


Reviews:
[{"review_id": "tgrrv4Md_7D", "reviewer": "Reviewer_tJ3i", "summary": "This paper discusses a self-supervised framework to learn equivariant representations (for a given group) from data. One of the key observations in this paper is that in most existing equivariant architectures, the group action on the embedding space is linear and hence limits the complexity of the architectures. The authors, therefore, propose to allow non-linear group actions and derive a simple relation for any function to be equivariant (Eq 2). The authors then translate this formula into a general recipe and an associated loss function (SymReg) that “regularises” for equivariance. More concretely, the authors translate this to \\emph{preserve} geometric invariants of the respective group action. For eg, euclidean distances for SE(3), angles for conformal transformations, etc. \n\n3 synthetic (Pendulum, chair, double bump world) and 2 real-world applications are shown. The main message in the demonstrations is that the embeddings learned by the proposed self-supervised framework are directly correlated to the actual variables of the manifold (for eg, angle, angular momentum, 3D Euler angles etc) ", "questions": "Questions and Comments\n\n- It is unclear how to deduce equation 4 from equation 2. \n\n- More importantly, how does plainly minimizing equation 4 not descend into something trivial (like all 0s or constant). In the terminology of self-supervised learning, you have positives and negatives. If I interpret equation 4 as a positive, where is the negative? Or is SymReg used in conjunction with another loss? (it doesn’t appear so at least for the synthetic examples). To me, this is a very significant issue and I will wait for the discussion to make an opinion on this before labeling it as a strong drawback. \n\n- (Suggestion) I would recommend more direct and compelling examples from 2D or 3D geometry. For eg, please see - (1.) Pai, Gautam, Aaron Wetzler, and Ron Kimmel. \"Learning Invariant Representations Of Planar Curves.\" (2016). (2.) Basri, Ronen, and David Jacobs. \"Efficient representation of low-dimensional manifolds using deep networks.\" arXiv preprint arXiv:1602.04723 (2016). \n\n\n\n", "limitations": "See weaknesses. As a pre-rebuttal/discussion rating I am inclined to weigh in positively on this paper - for its clarity of discussion and a very interesting message on equivariant actions. However, there is still confusion in some aspects and I will wait for the discussion phase to make a more informed opinion. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\n- I found the articulation and clarity of “Equivariance is cheap actions matter” to be very nice and a good take home message\n- Overall, this paper reads extremely well, and I appreciate the clarity of thought in most commentaries \n- I also like the synthetic demonstrations, (~ sort of) validating the main claims of this paper (although I have some questions below) \n\n\nWeaknesses\n\n\n- Somehow, despite the main message of re-looking at equivariance through non-linear actions, I don’t get the impression at the end of the paper that the problem is solved. For eg, I see no demonstration (experimental, visual or otherwise) of the serious drawbacks of enforcing equivariance through linear actions and the benefits of allowing for non-linear group actions in practice. For eg, it would be a convincing demonstration if - the embeddings from some linear-action equivariant auto encoder are compared to the proposed regularisation and indeed there is a positive difference.\n\n- One confusion that arises from this paper is: in the end what exactly is proposed? Is SymReg a manifold learning paradigm? (that generates more interpretable features?). In this case, it would be a good idea to also show the output of a similar dimensionality reduction scheme like [26]. \n\n- For the synthetic examples, the comparative demonstrations are pretty weak. For one thing, the size of the embeddings in figures 4 and 6 is much too small to be comfortable. it would really be more convincing to see the application of [26] for the experiment in figure 2 side-by-side with the proposed method. \n\n- Some important issues, which I will phrase as questions. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "qfDxcLPwV_u", "reviewer": "Reviewer_44UN", "summary": "The paper advocates the use of invariants for learning symmetry-preserving (equivariant) representations. Rather than building a network that maintains group actions at each stage of processing (as commonly done in approaches such as G-convolutional networks and their steerable descendants) abstractly or concretely, the idea here is to enforce the preservation of invariants as an objective of the learning process (for example, maintaining distances for E(d)). For factored groups, multiple embeddings can be used to capture actions of each factor group, and it is also possible to use less specific invariants to capture subgroups (for example, maintain SO(2)-covariance for groups of discrete rotations).\n\nThe paper applies this idea to a set of example applications and discusses the results and findings.", "questions": "- Is there a generic \"algorithm\" how to apply the ideas of this paper in practice, or is a case-by-case assessment needed to find invariants, objective functions and suitable networks to search for them?\n- The idea of embeddings with invariants seems to be tightly related to metric learning and approaches such as generalized MDS (multi-dimensional scaling). Can one reuse ideas from that field?\n", "limitations": "As stated above, I see some limitations in terms of costs of computing the embedding and the ability to reach the required precision (depending on the application). I think this is worth discussing a bit more in the paper.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I would like to start the discussion of this submission by stating that I found it difficult to understand what the paper is aiming. The observation of Section 4 that equivariance (homomorphism into some group representation on the output side) is almost trivial (only invertibility required) when general group actions (or representations of the group) are allowed is certainly valuable and seems to be a valuable call to caution in terms of how to formally characterize what is actually desired. The proposed solution is to identify invariants and enforce them using soft-constraints in the objective function.\n\nThe criticism of learning equivariant representations by using multiple stages of suitable group representations has targets two problems: First, the representation could theoretically be meaningless, but this does not happen in practice where specific representations are designed the structure of which is well-understood and can be used to reconstruct invariant embeddings. For example, steerable networks for use in areas such as computational physics aim at covariant computations to maintain conservation laws (for example, by learning covariant force-fields or potentials with covariant derivatives); here, the structure of the equivariant map is very important (although this aspect might not have been emphasized explicitly that much). Secondly, such representations might not be suitable for *discovering* symmetry in data, possibly encoded in a highly non-linear way. Here, the usage of invariance-preserving embeddings could indeed be beneficial, as less constraints on the form of the symmetry and group action are needed.\n\nOn the technical side, some problems remain though: The idea of finding embeddings with the same symmetry properties as the input is compelling in its simplicity and easy of use in down-stream applications, and the idea of setting up larger abstract groups the actual symmetry in the data only forms a subgroup of leads to some interesting applications. However, there are difficulties with this approach:\n- Invariants enforce via an objective function are approximate and have to be traded-off with other potential constraints of the embedding (which might be quantified in a different unit of measurement, thereby complicating parameter choice for varying sampling densities or other settings).\n- The invariants have to be derived manually; the paper only gives some examples of well-known cases. From the discussion, it is not clear (and I personally do not know) how to generally map between groups and their invariants.\n- Even if the invariants are enforced as much as possible, the network itself might only provide approximate symmetry if not designed towards equivariance. For some applications (as physical simulations), accuracy is an essential requirement.\n- Learning invariants might be very costly, as the unconstrainted network is much larger: For example, trying to learn the shift-invariance of a CNN by distance constraints to a fully-connected MLP would be very costly (theoretically doable, but not practical). Similar problems could easily occur in other cases such as fin-granular rotations, as the non-symmetric network needs sufficient representational capacity.\n\nMy impression is that the paper proposes a rather elementary mechanism that is hard to implement effectively (for some applications, where precision of symmetry matters) and efficiently (for applications with many degrees of freedom where symmetry is a strong constraint). It remains a bit vague on the implementation side, only showing a collection of examples. That said, some of the examples are quite interesting as they automatically discover relevant structure.\n\nOverall, I am rather uncertain what to recommend. Many of the issues that I have perceived might be a problem of positioning and writing, but I do think that some non-trivial technical problems remain that are not easily solved (and that should be discussed). At the current point, I would give a negative overall assessment for NeurIPS 2022, but I would be happy to change my rating if the further discussion clarified misconceptions on my side.\n\n**Update:** After reading the other reviews and the rebuttal (thanks to the authors for their detailed explanation) I understand the contribution better (non-linear vs. linear embeddings and why it is a good idea to just regularize the learned embedding vs. prescribing one). The issue of approximate regularization (which was also brought up by other reviewers) seems to remain (the method cannot guarantee exact invariants), but this is not a \"show-stopper\" (some applications rely on this, but many do not, and the gained ability to discover new symmetry seems very promising). I have therefore (in line with the other reviewers) updated my rating towards a positive recommendation. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ksp89UN8Syp", "reviewer": "Reviewer_E8cu", "summary": "This paper provides a methodological approach that enforces equivariance to certain to certain groups using only special losses during optimization. Their method is in the context of self-supervised learning. In the sense that similar to transforming auto encoder, in most cases, they have images and their transformed pairs in the batch. Then they constrain the learned embedding by enforcing them to be in a certain space. They do so by enforcing the defining invariant of that space. For example for the case of Euclidean space, the defining invariant is the distance. So they optimize the fact that the distance of embeddings for two images should stay the same before and after a transformation. \nThere are known defining invariants for many groups. Most interestingly they can easily enforce equivariance to Affine transformations by simply forcing the embeddings to be euclidean. \nThey also provide modification that can handle decoupled groups by adding invariance to other groups for each data point of one group.\n\nThere is enough evidence in the experiments that suggests the work delivers what it promises. The visualizations specially show that being Euclidean for SO(3) equivariance makes it not only equivariant but easy to investigate and representations are intuitive. The practical application in RL show that when the equivariance is the key factor adding their loss strongly improves the result. When other factors such as background or details are also important adding a finetuning and generative decoding would relax the model and enables it to encode more aspects of the scene.\n\nTwo aspects that was most impressive were the fact that they only input the images and that there is no architectural constraints or special architectural implementations. The transformation can be even unknown (already exists in the data...) so it's not reliant to only transformations that can be done by our code during data pre-processing. Also, since it is architecture agnostic it can be applied on any network however it is implemented, specialized for a specific processor, etc. \n\nOne aspect that is concerning is that since it is only a matter of optimization, what happens if the loss is competing with other losses. For example if the network is trained for a specific task, such as segmentation, then one needs to balance the factors between the equivariant losses and the final loss. Since it is not architectural, it is not guaranteed that the embeddings are equivariant. If you lower the factor on the proposed loss compared to the task loss so that segmentation results get better, it could just simply not be equivariant and at the test time it won't generalize to new viewpoints.", "questions": "Where is the code? In the questionnaire you have answered yes to the code one, but I couldn't find the url and the zip only contains a pdf.\n\nHow do you project to lower dimensions  for visualization? In the decoupled case of the bumps with the 4 dimensions do the dimensions readily form the circles or do you do any projections? ", "limitations": "This work does not increase the negative social impact of the literature.\n\nWhether their regularizer is applicable to setups with natural images is yet to be seen and it's their current limitation.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The proposed method is theoretically sound. They clearly state their math, ideas, and proofs. The manuscript is well written and easy to follow and understand. The idea is novel and applicable. They provide clear experiments showcasing the strengths of their method in the RL domain.\n\nWeaknesses: The regularizer can be too strict (enforcing distance equivariance for example) for a downstream task. For the RL case they mitigate the issue by adding a generative decoder and finetuning. It is not clear if after finetuning the embedding is still equivariant or not. \nAlso, it is not shown that it is applicable in a natural image setup or tasks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "T7s3uyrf2f7", "reviewer": "Reviewer_oRnK", "summary": "The paper proposes an approach that learns geometric invariants in the embedding space instead of in the input space. The authors point out that since injective functions are equivariant to any group, equivariant networks’ strong performance is a result of simple group actions. The proposed method does not constraint network structures and allows nonlinear or unknown group action on the input space. Experiments show that incorporating the geometric invariant into the loss function enables the neural network to learn meaningful embeddings and allows for decomposing of the symmetry group. Additionally, using the learnt embedding on downstream tasks achieves state-of-the-art performance.", "questions": "- For the world modeling experiment, why did the authors select the two particular experiments? It would help to provide justification that strong performance on the two games indicates that SymReg would perform well on the grid and 3-body physics environment too. \n- From the argument in section 4, it seems that equivariance is still a useful information for non-injective functions such as many neural networks. When the group action is known, will SymReg outperform traditional equivariant models? ", "limitations": "Limitations are listed in the conclusion. The authors also provide further discussion of some limitations (e.g. finite groups) in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Inducing equivariance on the embedding instead of the input is novel. The observation that equivariance can be a weak inductive bias is insightful and the proposed method is well motivate. The distinction from previous works is clearly described in the related work section.\n- The visualizations are very informative and provides strong support for the meaningfulness of the learnt embedding. \n- The experiments are comprehensive, and empirical results support the advantage of the proposed approach compared to baselines. \n\nWeaknesses:\n- Although the paper addresses the problem of complex or unknown group actions in the input space, it still requires knowledge about which group to use on the embedding space. All experiments in the paper are performed on datasets that have physically meaningful symmetries. However, for tasks that do not have an obvious symmetry, it is unclear how to select a group. \n\nSmall typos: line 136 “X x G” should be “G x X” to be consistent with other places; Figure 3 caption “againts”\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept