Unsupervised Learning of Group Invariant and Equivariant Representations
Authors: Robin Winter
Marco Bertolini
Tuan Le
Frank Noe
Djork-Arné Clevert
Venue: NeurIPS 2022 Accept (2022)
Keywords: equivariance
invariance
representation learning
autoencoder
unsupervised learning

Abstract: Equivariant neural networks, whose hidden features transform according to representations of a group $G$ acting on the data, exhibit training efficiency and an improved generalisation performance. In this work, we extend group invariant and equivariant representation learning to the field of unsupervised deep learning.  We propose a general learning strategy based on an encoder-decoder framework in which the latent representation is separated in an invariant term and an equivariant group action component. The key idea is that the network learns to encode and decode data to and from a group-invariant representation by additionally learning to predict the appropriate group action to align input and output pose to solve the reconstruction task.  We derive the necessary conditions on the equivariant encoder, and we present a construction valid for any $G$, both discrete and continuous. We describe explicitly our construction for rotations, translations and permutations. We test the validity and the robustness of our approach in a variety of experiments with diverse data types employing different network architectures.

Unsupervised Learning of Group Invariant and Equivariant Representations
Robin Winter"
Marco Bertolini"
Tuan Le
Bayer AG
Bayer AG
Bayer AG
Freie Universität Berlin
marco. www.bbeetolinnaaaae Saanenibbaarrr.. com
Freie Universität Berlin
robin. whbbtr.intrrrbbeerrr com
tuan www.beee2bbe.... com
Frank Noé
Djork-Arné Clevert
Freie Universität Berlin
Bayer AG
Microsoft Research frank, nwweoooberrueeeeeee
djork-arde arne. www.eeeleeeetbbaee.coomm
Abstract
Equivariant neural networks, whose hidden features transform according to rep- resentations of a group G acting on the data, exhibit training efficiency and an improved generalisation performance. In this work, extend group invariant and equivariant representation learning to the field unsupervised deep learning. We propose general learning strategy based which the latent representation is separated in an invariant term and an equivariant group action component. The key idea is that the network learns encode and decode data to and from a group-invariant representation by additionally learning predict the appropriate group action align input and output pose solve the reconstruction task. We derive the necessary conditions on the equivariant encoder, and we present a construction valid for any G, both discrete and continuous. We describe explicitly our construction for rotations, translations and permutations. We test the validity and the robustness of our approach in variety of experiments with diverse data types employing different network architectures.
encoder-decoder framework in
1
Introduction
An increasing body of work has shown that incorporating knowledge about underlying symmetries in neural networks as inductive bias can drastically improve the performance and reduce the amount of data needed for training Cohen Welling (0166)); Bronstein et al. (2021). For example, the equivari- ant design with respect to the translation symmetry of objects in images proper of convolutional neural networks (CNNs) revolutionized the field of image analysis LeCun et (1955.. Message Passing neural networks, respecting permutation symmetries in graphs, have enabled powerful predictive models on graph-structured data Gilmer al. (0177; Detterrard et al. (2016). Recently, much work has been done utilizing 3D rotation and translation equivariant neural networks for point clouds and volumetric data, showing great success predicting molecular ground state energy levels with high fidelity Miller et al. (2020g Anderson et al. (2019); Klicpera et al. (2020); Schütt al. (2021). Invariant models take advantage the fact that often properties of interest, such as the class label of an object in image the ground state energy of molecule,
invariant certain group actions (e.g., translations or rotations), while the data itself is not (e.g., pixel values, atom coordinates).
There are several approaches to incorporate invariance into the learned representation of neural network. The most common approach consists teaching invariance to the model by data augmenta-
"equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1
Z
8
x
V
g
Z
X
X
Figure 1: a) Schematic of the learning task this work is concerned with. Data points E X are encoded to and decoded from latent space Z. Points in the same orbit in X are mapped to the same point (orbit) & Z X/G. Latent points
mapped to canonical elements E {px (g)x/VV E G}. b) Schematic of our proposed framework with data points ø, encoding function y) decoding function canonical elements i; group function and group action g
tion: during training, the model must learn that group transformation on its input does not affect its label. While this approach can lead to improved generalization performance, it reduces training efficiency and quickly becomes impractical for higher dimensional data Thomas et al. (2018). A group transformations of the Puny et al. (02T). While feasible with finite groups, this method requires, for instance, sampling for infinite groups Lyle et al. (2020). A third approach is to impose invariance as a model architectural design. The simplest option is to restrict the function to be learned to be a composition of symmetric functions only (Schütt et al. (018). Such choice, however, can significantly restrict the functional form of the network. A more expressive variation of this approach consists of an equivariant neural network, followed by a symmetric function. This allows the network to leverage the benefits of invariance while having a larger capacity due to the less restrictive nature of equivariance. In fact, many real-world application, equivariance beneficial if not necessary Smidi (2020); Miller et al. (2020). For example, the interaction of a molecule (per se rotational invariant) with an external magnetic field is an intrinsically equivariant problem.
second technique, known feature averaging, consists of averaging model predictions
All aforementioned considerations require some sort of supervision to extract invariant representations from data. Unsupervised learning of group invariant representations, despite its potential in the field of representation learning, has been impaired by the fact that the representation of the data in general does not manifestly exhibit the group as symmetry. For instance, in the of an encoder-decoder framework in which the bottleneck layer is invariant, the reconstruction is only possible up to group transformation. Nevertheless, the input data typically parametrized in terms of coordinates in some vector space X, and the reconstruction task can only succeed by employing knowledge about the group action on X.
Following this line of thought, this work is concerned with the question: Can we learn to extract both the invariant and the (complementary) equivariant representations of data in unsupervised way?
To this end, we introduce group-invariant representation learning method that encodes data in a group-invariant latent code and group action. By separating the embedding in a group-invariant and a group-equivariant part, we can learn expressive lower-dimensional group-invarian representations utilizing the power of autoencoders (AEs). We can summarize the main contributions of this work as follows:
We introduce novel framework for learning group equivariant representations. Our repre- sentations are by construction separated an invariant and equivariant component.
We characterize the mathematical conditions the group action function component and we propose explicit construction suitable for any group G. To the best of our knowl- edge, this the first method for unsupervised learning of separated invariant-equivariant representations valid for any group.
We show in various experiments the validity and flexibility of our framework by learning representations of diverse data types with different network architectures. We also show that the invariant representations are superiour to the non-invariant counterparts in downstream tasks, and that they can successfully employed transfer learning for molecular property predictions.

---

2 Method
2.1 Background
We begin this section by introducing the basic concepts which will central our work.
A group G is set equipped with an operation (here denoted -) which is associative as well as having an identity element e and inverse elements. In the context of data, we are mainly interested in how groups represent geometric transformations by acting on spaces and, in particular, how they describe the symmetries of an object or of set. In either case, we are interested in how groups act on spaces. This is represented by group action: given a set X and group G, (left) action of G on X is map p: G X - X such that it respects the group property of associativity and identity element. If X is vector space, which we will assume for the remainder of the text, we refer to group actions of the form px G + GL(() as representations of G. where the general linear group of degree n GL(X) is represented by the set of n. x invertible matrices. Given a group action, concept which will play an important role in our discussion is given by the fixed points of such an action. Formally, given a point X and an action (representation) px of G on X, the stabilizer of G with respect to x is the subgroup G2 = {gg E Glpx(g)= = x} C G.
In the context of representation learning, we assume our data to be defined as the space of representation-vaauue
functions on some set V, i..., X = {flf : + W7. For instance, a point cloud in three dimensions can be represented as the set of functions f R + Z2, assigning to every point r E R the value ((r) = 0 (the point is not included in the cloud) or f(r) = (the point is included in the cloud). Representations pV of group G on V can be extended to representations on f. and therefore on X, px G -I. as follows
(1) In what follows, we will then only refer to representations for the space X, implicitly referring to equation ① for mapping back to how the various components transform. A map y + W is said be G-equivariant with respect to the actions (representations) pv, Pw if (ppy(g))) for every g E G and v E V. Note that G-invariance is particular case of the above, where we take PV. PW to be the trivial representations. An element x E X can be described in terms of a G-invariant component and a group element g E G, as follows: let Pinv X + X/G, be an invariant map mapping each element x E X to a corresponding canonical element é in the orbit the quotient space X/G. Then for each E X there exist a g E G such that = px (g) Pim (ar).
(xx( (x) (w(o)f(pv(C ")))).))...
Pw (v)
2.2 Problem Definition
We consider a classical autoencoder framework with encoding function 7) : X + Z and decoding function d : Z + X, mapping between the data domain X, and latent domain Z, minimizing the reconstruction objective d(((r(())), z). with difference measure d (e.g.. Lp norm). As discussed above, we wish to learn the invariant map 17 (fan in the previous paragraph), thus
Property 2.1. The encoding function 17 X + Z is G-invariant, i.e., n(px(g)x) = nイ()) Ve e X, Vg G.
The decoding function 8 maps the G-invariant representation However, as is G-invariant, 5 can at best map n)(x;) E 7 back to an element å E X such that {ex (V) G}. i.e., an element the orbit of x through G. This is depicted in Figure Da Thus, the task of the decoding function ö Z + X is to map encoded elements 2 = n)x)= E Z to an element X such that =9_ E G &ucch that
Z back to the data domain X.
8(n(z)) = å = px (gn)x.
(2)
We call è the canonical element of the decoder 5. We can rewrite the reconstruction objective with G-invariant encoding function r as d(px - ))((((z)),, One of the main results of this work consists in showing that é and 9z can be simultaneously learned by suitable neural network. That is, we have the following property of our learning scheme:
Property 2.2. There exists a learnable function vp> : X - G such that, given suitable n, as described above the relation px (x(())))((((x))
holds for all x E X.
We call any function yi satisfying 2.2p suitable group function. Figure Db describes schematically our proposed framework. what follows, we will first characterize the defining properties of suitable group functions. Subsequently, we will describe our construction, valid for any group G.
3

---

2.3 Predicting Group Actions
In the following we further characterize the properties of vb We begin by stating two key results, while we refer to the Appendix A for the proofs.
Proposition 2.3, Any suitable group function v X - G is G-equivarian at a point I E X up the stabilizer G2 i.e., (px((g)x) g. 5
Proposition 2.4. The image of any suitable group function vp : X - surjective into Gx is the stabilizers of all the points of X.
G Gx
where
Let us briefty discuss example. Suppose X = {r (ro, 02, 23) RAx2 a.; PR=(0e=x/2) To: Zo E R2} and G = SO(2). X describes all collections of vertices of squares centered at the origin of IR22, and it is easy to check that 9x = Zn, generated by 7/2 rotation around the origin. In this case, any such square can brought to any other square (of the same radius) by rotation of an angle 0 < T/2, thus Imv 2 {ge E SO(2) I 0 s T/2 = SO(2)
Combining the two propositions above we have the following
Lemma 2.5. Any suitable group function to is an isomorphism O2 = G/G for any x E X, where
Og C X is the orbit of with respect to G X.
2.4 Proposed Construction
Next, we turn to our proposed construction of class of suitable group functions that satisfy Property 2.2 for any data space X and group G. As we described above, these functions must be learnable.
Property 2.6 (Proposed construction). Withou loss of generality, we write our target function = Eo where μ X - Y is learnable map between the data space X and the embedding space Y, while & : YY + G is a deterministic map. Our construction is furthrr determined by the following properties:
We impose p. X + Y to be G-equivariant, that is, ((px(g))) p (F) all x E X and G.
We ask that Y is homogeneous space, that is, given any element yo Y, every element y E Y can be written as Py (g)90 for some E G.
The map & Y + G is defined as follows: E(y) such that pv (g)(g)) for any chosen point yo Y.
In what follows we will show that our construction satisfies the properties of the previous section. For proofs see Appendix. We begin with the following
Proposition 2.7. Let vp & p be suitable group function and let μ X Y be G-equivariant. Then, G2 = Gp(z) for all x E X.
The result of the above proposition is crucial for our desired decomposition of the learned embedding, as it ensures that no information about the group action on X is lost through the map μc: if group element acts non-trivially in X, it will also act non-trivially in Y.
Proposition 2.8. Given y, yo. the element g such that y py pr (g)yo is unique up to the stabilizer Ggo-
This proposition establishes the equivariant properties of the map E. Finally, we have Proposition 2.9. Let 1b = E where y and & are as described above. Then, a suitable group function.
2.5 Intuition Behind the Proposed Framework
We conclude this rather technical section with a comment on the intuition behind our construction. Assuming for simplicity that the domain set V admits the structure of vector space, Y represents the space spanned by all basis vectors of V. The point yo represent a canonical orientation of such basis, and the clement E(y) g is the group element corresponding basis transformation. As

---

all elements can be expressed in terms of coordinates with respect to given basis, it is natural to consider a canonical basis for all orbits. justifying the assumption of homonogeneity of the space Y.
Further, let us assume that the invariant autoencoder correctly solves its task, & ~ 8(n(2)). Now let & E O₂ such that ã 6(n():), and by definition, &t pv (g)x for some g G. Now, the correct orbit element is identified when (²()) = e, since (xx = g ((]) = g and thus px(9-1)
px ()) æ. Hence, during training & needs learn which orbit elements are decoded as "canonical", i.e., without the need of an additional group transformation. To clarify, here "canonical" does not reflect any specific property of the element, but simply refers to the orientation learned from the decoder during training. In fact, different decoder architectures or initializations will lead to different canonical elements.
Finally, note how the different parts of our proposed framework (17, 6 and v)), as visualized in Figure 1b, can be jointly trained by minimizing the objective
d(px ((x))()))))(((())))))))
which is by construction group invariant, i.e., not susceptible to potential group-related bias in the data (e.g., data that only occurs in certain orientations).
3 Application to Common Groups
In this section we describe how our framework applies to variety of common groups which we will then implement in our experiments. As discussed in Section 2.2 and visualized in Figure [q. the main components of our proposed framework are the encoding function 17. the decoding function and the group function db. As stated in Property 2.1 the only constraint for the encoding function is that it has to be group invariant. This is general straightforward to achieve for different groups as we will demonstrate in Section 5 Our proposed framework does not constrain the decoding function 8 other than that has to map elements from the latent space Z to the data domain X. Hence, can be designed independently of the group of interest. The main challenge is in defining the group function d = f μ such that it satisfies Property 2.2 Following Property 2.6 we now turn to describing our construction of E, μ and Y for variety of common groups.
Orthogonal group SO(2). The Lie group SO(2) is defined as the set of all rotations about the origin in I.2. We take Y to be the circle S c R that is, the space spanned by unit vectors in R². Now, 5 is a homogeneous space: any two points 8o, $1 E S" are related by a rotation. Without loss of generality, we take the reference vector yo be the vector (1,0) S Then given vector y E 5", we can write
yz yy
Ux
Vy
))
(4)
By
thus, the function & S² SO()) is determined by E(v) go such that 0 = arccos(y,. a
Orthogonal group (3). We assume that X has no fixed points, as this is usually the case for generic shapes (point clouds) in R would be tempting to take Y to be the sphere S c R3, that is, the space spanned by unit vectors in R3. While this space is homogeneous, it does not satisfy the condition that the stabilizers of G are trivial. In fact, given any vector y1 E 52, we have Gy = {g SO(3)lg is rotation about y1).
In order to construct a space with the desired property, consider a second vector E Sa orthogonal to V1, 8/2 C vi- Taking Y to be the space spanned by yı, y2 E S is easy to see that now all the stabilizers are trivial. Finally, let y1 x V2 E S², then we construct the rotation matrix
f1.z
y2.z
13.x
(11 y2
(
0 0 Ryyo 1
R
V1.9
V2.9
V3.y
which satisfies
R
V1,a
y2,z
y3,2
Symmetric group Sn. ements of the set M
suitable space Y is the set of ordered collections of unique el- (1.2...., n.. For instance, for n
3. we have Y
5

---

It is trivial to see that the action of the permutation group on the set is free, that is. all the stabilizers are trivial. Explicitely, given any cermutation-equivartii
(11,2,3,
(3. (.)
RR", we obtain an element also obvious that any element in Y can be written as (1 .......n) is, a group element acting on the canonical yo (1, (1, ...,,,..111.
vector
argsort w) E EY. Moreover, it is
(n)) that
Translation group T... Here we take Y IR", which is homogeneous with respect to the translation group. In fact, any vector y E Y can be trivially written as y = y + 0 = y + yo: where the origin of RR. Our group function takes therefore the form {(v) = y-
Euclidean group SE(n). A generic transformation of the Euclidean group on n-dimensional representation E
U + Au bb AE SO(n) , Tn.
(5)
Let P2.............................. (a+1) be collection of n + nndimennional SB(n)-qquivarias vectors, that is, pilpx (g)z) = py i= n. We construct T. (Pg Hh441//lpp Pa+all E S", = 1, where 5th is the unit n-dimensional sphere. These n ortho-normal vectors are translation invariant but rotation equivariant, and are suitable construct the rotation matrix
D2
while the extra vector Vn++1 = Mn++1 can be used to predict the translation action. Putting all together, the space Y described by vectors yr Ua + Dn+1. and yo Ln is the n unit matrix, as
(R + In+1) (i)
To) - Dn+1 (y
yn)
4 Related Work
Group equivariant neural networks. Group equivariant neural networks have shown great success for various groups and data types. There are two main approaches to implement equivariance in layer and, hence, in neural network. The first, and perhaps the most common, imposes equivariance on the space of functions and features learned by the network. Thus, the parameters of the model are constrained to satisfy equivariance Thomas et al. (2018); Weiler & Cesa (2019)); WWiler et al. (2118)); Esteves et al. (020). The disadvantage of this approach consists in the difficulty of designing suitable architectures for all components of the model, transforming correctly under the group action Xu et al. (202T). The second approach to equivariance consists in lifting the map from the space of features to the group G. and equivariance is defined on functions the group itself [Romero & Hoogendoorn (2020): Romero et al. (2020g Hoogeboom et al. (2018). Although this strategy avoids the architectural constraints, applicability is limited to homogeneous spaces Hutchinson al. (201
and involves an increased dimensionality of the feature space, due to the lifting to G. Equivariance has been explored in a variety of architecture and data structures: Convolutional Neural Networks
Cohen & Welling (2016a) 60); WWorrll et al.] (201)) Weiler et al. (2018c); [Bekkers et al. (2018); Thomas
20188c):Bekkerrs
al.
2018):
Dieleman et al. a (2: Kondor & Trivedi ( 201 8); Cohen & Welling (20T6b); Cohen
et al.
2018);
Finzi et al. (2020). Transformers Vaswani et al. 2 Fuchs et al. (2020); Hutchinson
al.
2021:
Romero & Cordonnier 2020). Graph Neural Networks Defferrard al. (2016); Bruna
et al.
2013):
Kipl & Welling (2016 6) Gilmer et al. (201 7); Satorras et al. (202) 1) and Normalizing
Flows Rezende & Mohamed| (2015); Köhler et (219 2020); Boyda et al. (2021) l). These methods
are usually trained in supervised manner and combined with symmetric function (e.g. pooling) to
extract group-invariant representations.
Group equivariant autoencoders. Another line of related work is concerned with group equivariant autoencoders. Such models utilize specific network architectures to encode and decode data in an equivariant way, resulting into equivariant representations only Hinton al. (201T); [Saoour al.) (2017); Kostorek et al. (2019); Guo et al. (219)). Feige (0199) use weak supervision in an AE to extract invariant and equivariant representations. Winter et al. (20TT) implement a permutation- invariant AE to learn graph embeddings, in which the permutation matrix for graph matching is learned during training. In that sense, the present work can be seen as a generalization of their approach for generic data type and any group.

---

Input
-VIZ5XI
3
3 m
333 ?
333 3
3.3
3
1334 3 W
134
3
6
Output
333
m
E
6
9
9
Output
3
w
6
Rotation
Figure 3: Input and predicted output for rotated versions of three MNIST images. Top row shows the input image successively rotated by 45". Middle row shows the decoded (canonical) image and bottom row shows the decoded image after applying the predicted rotation.
Unsupervised invariant representation learning. The field of unsupervised invariant representa- tion learning can be roughly divided into two categories. The first consists learning an approximate group action in order to match the input and the reconstructed data. For instance, Mehr et al. (2018b) propose to encode the input in quotient space, and train the model with a loss that is defined by taking the infimum over the group G. While this is feasible for (small) finite groups, for continuous groups they either have to approximately discretize them or perform a separate optimization of the group action at every back propagation step find the best match. Other work Shu et al. (2018); Koneripalli et al.) (2020) proposes to disentangle the embedding in a shape-like and a deformation component. While this is in spirit with our work, their transformations transformations) and are approximative, that is, the components are not explicitly invariant and equivariant with respect the transformation, respectively.
local (we focus on global
In the case of 2D/3D data, co-alignment of shapes can be used to match the input and the reconstructed shapes. Some approaches are unfeasible Wang et al. (012) they are not compatible with purely unsupervised approach, while other (Aerriou et al. (206); Chaouch & Verroust- Blondet (2008) 2009) leverage symmetry properties of the data and PCA decomposition, exhibiting however limitation regarding scalability. For graphs, the problem of graph matching [Bnnee & Jiang (2000) has been tackled in several works and with different approaches, for instance algorithmically. e.g., Ding et al. (020), or by means of a GNN Li (2019).
On the topic of group theory-based embedding disentanglement, some works are based on the definition of Higgins al. (2018) of a disentangled representations. We refer this as "symmetry- based decomposition", where the various factors in the disentangled representation correspond to the decomposition of symmetry groups acting on the data space. In Plau et al. (2020), the authors show that, with some assumption on the geometry of the underlying data space, is possible to learn to factorize a Lie group from the orbits in data space. The works Hosooya (2019; Keurti et al. (2022), for instance, design unsupervised generative VAES approaches for learning representation corresponding to orthog- onal symmetry actions the data space. In our work, on the other hand, we learn a decomposi- tion into separate group representations. These are all representations of the same group, but act differently on different data space (analogously to different SO(3) representations identified by the angular quantum number | 0, ......
classial
Inveriant
Figure 2: TSNE embedding of the encoded test. dataset for a classical and our proposed SO(2) in- variant autoencoder.
Experiments
In this section we present differnt experiments for the various groups discussed in Section 3
P
5.1 Rotated MNIST
In the first experiment, we train an SO(2)-invariant autoencoder on the original (non-rotated) MNIST dataset and validate the trained model on the rotated MNIST dataset (ref. mni) which consists of randomly rotated versions of the original MNIST dataset. For the functions 1) and & we utilize SO(2)-Steerable Convolutional Neural Networks Weiler & Cesa (2019)). For more details about the
Source code for the different implementations available atlttpss//github http:/thwwbbcmmm//ttttrrrgggggee
7

---

+]
XIRX
(AXA
04
BlAS
07
|MAM
41214 23
13
Figure 4: a) Element wise reconstruction accuracy of our proposed permutation invariant autoencoder (cross) and a classical non-permutation invariant autoencoder (diamond) for different embedding and set sizes. b) Example set x with 100 elements with its canonical reconstruction å and the predicted permutation matrix P (resulting into a perfect reconstruction). One can confirm for oneself that, c.g., rr38] x' z'(0), matching P[38, 0] 1. c) Best viewed in colour. Visualization of the two-dimensional autoencoder for all 5151 sets of 100 elements with 3 different element classes. Each point represents one set, colours represent set compositions (proportion of each element class, independent of the order).
embedding of a permutation-invarit
network architecture and training, we refer Appendix B In Figure we show images in different rotations and the respective reconstructed images by the trained model. The model decodes the different rotated versions of the same image (i.e., elements from the same orbit) the same canonical output orientation (second row in Figure 3). The trained model manages to predict the right rotation matrix (group action) align the decoded image with the input image, resulting in an overall low reconstruction error. Note that the model never saw rotated images during training but still manages to encode and reconstruct them due its inherent equivariant design. We find that the encoded latent representation is indeed rotation invariant (up to machine precision), but only for rotations of an angle n. n E N. For all other rotations, we see slight variations in the latent code, which, however, discretized grid. Still, inspecting the
is to be expected due to interpolation artifacts for rotations 2d-projection of the latent code of our proposed model in Figure / we see distinct clusters for each digit class for the different images from the test dataset, independent of the orientation of the digits in the images. In contrast, the latent code of a classical autoencoder exhibits multiple clusters for different orientations of the same digit class.
5.2
Set of Digits
Next, we train permutation-invarian autoencoder on sets of digits. A set with N digits is represented
by concatenating one-hot vectors of each digit in N D-dimensional matrix, where we take = 10. Notice that this matrix-representation of set is not permutation invariant. We randomly sampled 1 1.000.000 different sets for training and 100.000 for the final evaluation with N = 20, 30, 40, 100, two sets that are the same
respectively, removing all permutation equivariant sets (i.e., there up to a permutation). For comparison, we additionally trained classical non eeermutationiivvarte autoencoder with the same number of parameters and layers as our ionmutttiooniivaiiann version. For more details on the network architecture and training we refer to Appendix C Here, we demonstrate how the separation of the permutation invariinn information of the set (i.e., the composition of the set) from the (irrelevant) inder-information results in a significant reduction of the space needed to encode the set. In Figure 4.. we plot the element-wise reconstruction accuracy of different sized sets for both models for varying embedding (bottleneck) sizes. As the classical autoencoder has to store both the composition of digits in the set (i.e., number of elements for each of the 10 digits classes) as well as matrix representation, the reconstruction accuracy drops for
their order in the permutation--eependen increasing size of the set N for a fixed embedding size. For the same reason, perfect reconstruction accuracy is only achieved if the embedding dimension is at least as large as the number of digits in the set. On the contrary, our proposed permutation invariant autoencoder achieves perfect reconstruction accuracy with significant lower embedding size. Crucially, as no order information has to be stored in the embedding, this embedding size for perfect reconstruction accuracy also stays the same for increasing size N of the set. In Figure 4 we show one example for a set with N 100 digits, with the predicted canonical orbit element z and the predicted permutation matrix. As perhaps expected, the canonical element clusters together digits with same value, while not using the commonly used order of Arabic numerals. This learned order (here (1.9,4.0.336,8,77,.55 stays fixed for the trained network for different inputs but changes upon re-initialization of the network.
8

---

b)
Figure 5: a) Five different Tetris shapes represented by points at the center of the four blocks respectively. Input points, output points and rotated (predicted group action) output points as reconstructed by our proposed SE(3)- and S(N)) autoencoder are visualized. b) Two-dimensional latent space for all Tetris shapes augmented with Gaussian noise (a 0.01). Colors of points match colors of shapes on the right. c) Two molecular conformations and their reconstructions represented as point cloud and ball-and-stick model (left true, right predicted).
In Figure f we show the two-dimensional embedding of a permutation invariant autoencoder trained on set of N 100 elements chosen from D 3 different classes (e.g. digits 0,1,2). As the sets only consists (D+N-1) (102) 5151 elements in the two-dimensional embedding and colour them according to their composition. As proposed auteoncoder only needs store the information about the set composition and not the order, the embedding is perfectly structured with respect to the composition can be seen by the colour gradients in the visualization of the embedding.
different elements (but in different compositions and order) we can visualize the ino
5.3 Point Cloud
Point clouds are common way describe objects in 3D space, such the atom positions of molecule or the surface of an object. As such, they usually adhere to 3D translation and rotation sym- metries and are unordered, i.c., permutation invariant. Hence, investigate in the next experiment combined SE(3)- and S -viivvariant autoencoder for point cloud data. We use the Tetris Shape toy dataset Thomas et al. 2018) which consists of 8 shapes, where each shape includes N 4 points in 3D space, representing the center of each Tetris block. To generate various shapes, we augment the shapes by adding Gaussian noise with 0.01 standard deviation each node's 's position. Different orientations are obtained by rotating the point cloud with a random rotation matrix E SO(3) and further translating all node positions with the same random translation vector R Ta- For additional details on the network architecture and training we refer to Appendix D In Figure 5 we visualize the input points and output points before and after applying the predicted rotation. The model successfully reconstructs the input points with high fidelity (mean squared error of ~ 4 x 10-5) for all shapes and arbitrary translations and rotations. Figure P shows the two-dimensional embed- ding of the trained SE(3)- and S -i-inaari autoencoder. Augmenting the points with random noise results into slight variations in the embedding, while samples of the same Tetris shape class still cluster together. The embedding is invariant with respect to rotations, translation and permutations of the points. Notably, the SE(3)-invariant representations can distinguish the two chiral shapes (compare green and violet coloured shapes in the bottom right of Figure 5b). These two shapes are mirrored versions of themselves and should distinguished in an SE(3) equivariant model. Models that achieve SE(3) invariant representations by restricting themselves to composition of symmetric functions only, such as working solely on distances (e.g. SchNet (Shiit et al. (2018) or angles (e.g.
ANI-1 Smith et al. (207)) between points fail to distinguish these two shapes
Thomas et al. (018))

---

Molecular Conformations. We showcase our learning framework on real-world data by au- toencoding the atom types and geometries of small molecules from the QM9 database Ramakr ishnan al. 2014). We achieved reconstruction RMSE of 0.15 + 0.07 À for atom coordi- nates and perfect atom type accuracy on 5000 unseen test conformations (see Figure P for two examples and Appendix E.2.2 for more reconstruction predictions). Given a point cloud of N nodes, the G = SE(3) x -nvaarian embedding z has to store information about the Carte- sian coordinates P E RON as well as the 5 distinct atom types A E {0, 1) represented as one-hot encodings. The largest molecule in the QM9 database has Nmmx 29 atoms, thus the degrees of freedom of the data space X are 3 .29 5 29 = 12615. Our embeddings com- press this high-dimensional space of molecular conformations into z E Z C R2256 dimensions.
5.4 ShapeNet
We also run experiments on the ShapeNet dataset [hang et al. (2015). We utilized 3D Steerable CNNs proposed by WWiiler et al. (2018b) as equivariant encoder for the 3d voxel input space. We utilized the scalar outputs as rotation-invariant embedding (z) and predict (analogously to our experiments on 3d point clouds) 2 rotation- rotation-equiivaiaan vectors to con- struct a rotation matrix p(g). In Figure I in the Appendix we show example reconstructions of shapes from the SE(3) invariant representations. Similar to MNIST experiment, we compared the resulting embedding space to the embeddings produced by a non-invariant autoencoder model. As the dataset comes in an aligned form (c.g., cars always aligned in the same orientation), we additionally applied random 90 degree rotations to remove this bias (while avoiding interpolation artifacts) when training the non-invariant model. Random rotations also applied to the common test set. In Figure X visualize TSNE projection of the embeddings of both models. We can see a well structured embedding space for our model with distinct clusters for the different shape classes. On the other hand, the embeddings produced by the non-invariant autoencoder is less structured and one can make out different clusters for the same shape label but in different orientations. Moreover, we compared the downstream performance and generalizability of KNN classifier on shape classification, trained on 1000 embeddings and tested on the rest. The classifier based on our inviriiatt embeddings achieved an accuracy of 0.81 while the classifier based on the non-invariant embeddings achieved an accuracy of only 0.63.
Figure 6: TSNE embedding of the encoded dataset for classical autoencoder and our proposed SO(3) invariant autoencoder.
6 Conclusion and Future Work
In this work we proposed a novel unsupervised learning strategy to extract representations from data that are separated in group invariant and equivariant part for any group G. We defined the sufficient conditions for the different parts of our proposed framework, namely the encoder, decoder and group function without further constraining the choice of (G-) specific network architecture. In fact, we demonstrate the validity and flexibility of our proposed framework for diverse data types, groups and network architectures.
To the best of our knowledge, we propose the first general framework for unsupervised learning of separated invariant-equivariant representations valid for any group. Our learning strategy can be applied to any AE framework, including variational AEs. It would be compelling to extend our approach to fully probabilistic approach, where the group action function samples from probability distribution. Such formalism would be relevant in scenarios where some elements of a group orbit occur with different frequencies, enabling this to be reflected in the generation process. For instance, predicting protein-ligand binding sites depends on the molecule's orientation with respect to the protein pocket or cavity. Thus, in generative approach, it would be highly compelling generate group action reflecting a candidate molecule's orientation in addition to a candidate ligand. We plan to return to these generalization and apply our learning strategy to non- con-trivial real- world applications in future work.
Notice that the data space X can be described as the product space between 3N and 145N
10

---

References
Rotated MNIST.
http::/sttee..gogle..comm
iro wwwntreal. catpttrrra....p..lliii.c.c
[Online; accessed 05-Jamuary-2021)
Anderson, B., Hy, T.-S., and Kondor, R. Cormorant: Covariant molecular neural networks. arXiv
preprint arXiv: / 090999000.40 04015, 2019.
Averkiou, M., Kim, V. G., and Mitra, J. Autocorrelation descriptor for efficient coåARalignment of 3d shape collections. Computer Graphics Forum, 35, 2016.
Axelrod, S. and Gomez-Bombarelli, R. GEOM, 2021. URL JNGTDE
Bekkers, E. J., Lafarge, M. W., Veta, M., Eppenhof, K. A., Pluim, J. P., and Duits, R. Roto-translation covariant convolutional networks for medical image analysis. In International conference on medical image computing and computer-assisted intervention, pp. 440-448, Springer, 2018.
Boyda, D., Kanwar, G., Racanière, S., Rezende, D. J., Albergo, M. S., Cranmer, K., Hackett, C., and Shanahan, P. E. Sampling using su (n) gauge equivariant flows. Physical Review D, 103(7): 074504, 2021.
Bronstein, M. M., Bruna, J., Cohen, T., and Veličković, P. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104. 13400. 2021.
Bruna, J., Zaremba, W., Szlam, A., and LeCun, Y. Spectral networks and locally connected networks on graphs. arXiv preprint
2013.
Bunke, H. and Jiang, X. Graph Matching and Similarity, pp. 281-304. Springer US, Boston, MA. 2000. ISBN 978-1-4615-4401-2, doi: //doi.or//10 101.:01100070111111111 161-------15-44---21100
URL https
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. Shapenet: An information-rich 3d model repository. arXiv preprint
arXiv:1512.03311. 2015.
Chaouch, M. and Verroust-Blondet, A. A novel method for alignment of 3d models. 2008 IEEE International Conference on Shape Modeling and Applications, pp. 187-195, 2008
Chaouch, M. and Verroust-Blondet, A. Alignment of 3d models. Graph. Model., 71:63-76, 2009.
Cohen, T. and Welling, M. Group equivariant convolutional networks, In International conference on machine learning, pp. 2990-2999, PMLR, 2016a.
Cohen, T. Geiger, M., and Weiler, M. A general theory of equivariant homogeneous spaces. arXiv preprint arXiv:1811.02017 2018.
Cohen, S. and Welling, M. Steerable cnns. arXiv preprint arXiv: arXiv:/6/2.08499. 2016b.
Defferrard, M., Bresson, X., and Vandergheynst, Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29:3844-385 2016.
Dieleman, S., De Fauw, J., and Kavukcuoglu, K. Exploiting cyclic symmetry in convolutional neural networks. In International conference on machine learning, pp. 1889-1898. PMLR, 2016.
Ding, J., Ma. Z.. Wu, Y.. and Xu. J. Efficient random graph matching via degree profiles. Probability Theory and Related Fields, 179:29-115, 2020.
Esteves, C., Makadia, A., and Daniilidis, K. Spin-weighted spherical cnns. ArXiv, 107/,006.00731, 2020.
Feige, I. Invariant-equivariant representation learning for multi-class data, 2019.

---

Finzi, M., Stanton, S., Izmailov, P., and Wilson, A. G. Generalizing convolutional neural net- works for equivariance lie groups on arbitrary continuous data. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3165-3176. PMLR, 13-18 Jul 2020. URL https://proceeddi
mlr. cetess/v119/fiiiiii
html
Fuchs, F. B., Worrall, D. E., Fischer, V., and Welling, M. Se (3) transfsoooor: 3d roto-translation equivariant attention networks. arXiv preprint arXiv:2006. 10503, 2020.
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quantum chemistry. In International conference on machine learning. pp. 1263-1272, PMLR, 2017.
Grover, A., Wang, E., Zweig, A., and Ermon, S. Stochastic optimization of sorting networks via continuous relaxations. arXiv preprint
2019.
Guo, X., Zhu, E.,, Liu, X., and Yin, J. Affine equivariant autoencoder. In IJCAI, pp. 2413-2419, 2019.
Higgins, I., Amos, D., Pfau, D., Racanière, S., Matthey, L.,, Rezende, D. J., and Lerchner, A. Towards a definition of disentangled representations. ArXiv,
2018.
Hinton, G. E., Krizhevsky, A., and Wang, S. D. Transforming auto-encoders. In International
conference on artificial neural networks, pp. 44-51. Springer, 2011.
Hoogeboom, E., Peters, J. W. T., Cohen, T., and Welling, M. Hexaconv. ArXiv, 2018.
Hosoya, H. Group-based learning of disentangled representations with generalizability for novel contents. In IJCAI, 2019.
Hutchinson, M. J., Le Lan, C., Zaidi, S., Dupont, E., Teh, Y. W., and Kim, H. Lietransformer: Equivariant self-attention for lie groups In International Conference Machine Learning, pp. 4533-4543. PMLR, 2021.
Keurti, H., Pan, H.-R., Besserve, M., Grewe, B. F,, and Scholkopf, B. Homomorphism autoencoder learning group structured representations from observed transitions. ArXiv, abs/2207. 02022207.11067, 2022.
Kipf, N. and Welling, M. Semi supervise classification with graph convolutional networks. arXiv preprint arXiv:1609.02000000 02900, 2016.
Klicpera, J., Groß, J., and Günnemann, S. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.00 00.1030.....................3333 2020.
Köhler, J., Klein, L., and Noé, F. Equivariant flows: sampling configurations for multi-body systems with symmetric energies. arXiv preprint arXiv: 2019.
Köhler, J., Klein, L.., and Noé, F. Equivariant flows: exact likelihood generative learning for symmetric densities. In International Conference Machine Learning, pp- 5361-5370. PMLR, 2020.
Kondor, R. and Trivedi, S. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning, pp. 2747-2755 PMLR, 2018
Koneripalli, K., Lohit, S., Anirudh, R., and Turaga, K. Rate-invariant autoencoding of time-series. ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 3732-3736, 2020.
Kosiorek, R., Sabour, S., Teh, W., and Hinton, G. E. Stacked capsule autoencoders. arXiv 2019.
preprint
LeCun, Y., Bengio, Y., et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 1995.
Li, Y., Gu, C., Dullien, T., Vinyals, O., and Kohli, Graph matching networks for learning the similarity of graph structured objects. ArXiy, hts/1904.12787 2019.
12

---

Lyle, C., van der Wilk, M., Kwiatkowska, M. Z., Gal, Y.. and Bloem- -loem-Reddy, B. On the benefits of invariance in neural networks. ArXiv, abs/2005.00178, 2020.
Mehr, E., Lieutier, A., Bermudez, S., Guitteny, V., Thome, N., and Cord, M. Manifold learning in quotient spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 96759117, 2018a.
Mchr, A., Licutier, A., Bermudez. S., Guitteny, V., Thome, N., and Cord, M. Manifold learning in quotient spaces. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9165-9174, 2018b. doi: 10.1 20.1109/CVP..2088.0000
Miller, B. K., Geiger, M., Smidt, T. E.,, and Noé, F. Relevance of rotationally equivariant convolutions for predicting molecular properties. arXiv preprint wwXiv:200.0886611 2020.
Pfau, D., Higgins I., Botev, A., and Racanière, S. Disentangling by subspace diffusion. ArXiv, abs/2006.12900 12982, 2020.
Prillo, S. and Eisenschlos. J. Softsort: A continuous relaxation for the argsort operator. In Interna- tional Conference on Machine Learning, pp. 7793-7802. PMLR, 2020.
Puny, O., Atzmon, M., Ben-Hamu, H., Misra, I., Grover, A., Smith, E. J., and Lipman, Y. Frame averaging for invariant and equivariant network design, 2021. bttps:t:::/ hnttps://arxivv http://.////xww////////// org/ /bs/ 03336
Ramakrishnan, R., Dral, P. O., Rupp. M., and von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
Rezende, D. and Mohamed, S. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530-1538. PMLR, 2015.
Romero, D. W. and Cordonnier, J.-B. Group equivariant stand-alone self-attention for vision. arXiv preprint arXiv:2010.00977, 2020.
Romero, D. W. and Hoogendoorn, M. Co-attentive equivariant neural networks: Focusing equivari- ance on transformations co-occurring in data. ArXiv, abs/191 2020.
Romero, D. W., Bekkers, E. J., Tomezak, J. M., and Hoogendoorn, M. Attentive group equivariant convolutional networks. ArXiv, abs/2002.038800 2020.
Sabour, S., Frosst, N., and Hinton, G. E. Dynamic routing between capsules. arXiv preprint 17X::::710.09829 2017.
Satorras, V. G., Hoogeboom, E., and Welling, M. E(n) equivariant graph neural networks, 2021.
Schütt, K. T., Sauceda, H. E., Kindermans, P.-J., Tkatchenko, A., and Müller, K.-R. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24): 241722, 2018.
Schütt, K. T., Unke, O. T., and Gastegger, M. Equivariant message passing for the prediction of tensorial properties and molecular spectra, 2021.
Shu, Z., Sahasrabudhe, M., Güler, R. A., Samaras, D., Paragios, N., and Kokkinos, Deforming autoencoders: Unsupervised disentangling of shape and appearance. ArXiv, abs/ 806.06503, 2018.
Smidt, T. Fuclidean symmetry and equivariance in machine learning. ChemRxiv, 2020. doi: 223444/chemmrxi.11935555 12935198.v1.
Smith, S., Isayev, O., and Roitberg, A. E. Ani-1: extensible neural network potential with dft accuracy at force field computational cost. Chemical science, 8(4):3192-322 2017.
Thomas, N., Smidt, T., Kearnes, S., Yang. L., Li, L., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation- and translation-qquivarianl neural networks for 3d point clouds, 2018.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, E.,, and Polosukhin, Attention is all you need. Advances in neural information processing systems, pp- 2017.
13

---

Wang, Y., Asafi. S., van Kaick, O. M., Zhang, H., Cohen- Or, D., and Chen, B. Active co-analysis of set of shapes. ACM Transactions on Graphics (TOG), 31:1 - 10, 2012.
Weiler, M. and Cesa, G. General e(2)-equiaaiaant steerable cnns. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019a. URL httLhttps:/// http::s:s///cceeednnee wwwp/6999944000000000000 Paper. påf
neurips.cc/pappeee /aap:r//200//111//85666777/////1
Weiler, M. and Cesa, G. General -queivairt steerable enns. arXiv preprint arXiv:1911.082211, 2019b.
Weiler, M., Geiger, M., Welling, M., Boomsma, W., and Cohen, T. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In NeurIPS, 2018a.
Weiler, M., Geiger, M., Welling, M., Boomsma, W., and Cohen, 3d steerable enns: Learning
rotationally equivariant features in volumetric data, 2018b.
Weiler, M., Hamprecht, F. A., and Storath, M. Learning steerable filters for rotation equivariant cmns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849-858, 2018c.
Winter, R., Noe, and Clevert, D.-A. Permutation--invarian variational autoencoder for graph-level representation learning. arXiv preprini arXiv: 1ete:2144009956 2021.
Worrall, D. E., Garbin, S. J., Turmukhambetov, D., and Brostow, G. J. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2017.
Xu, J., Kim, H., Rainforth, T., and Teh, Y. W. Group equivariant subsampling, 2021.
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R., and Smola, A. Deep sets. arXiv preprint arXiv: 2017.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, ow they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after unning experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total mount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "vc_MwH93TNK", "reviewer": "Reviewer_FQV4", "summary": "An autoencoder architecture that is based on equivariant and invariant neural layers is proposed. The most important aspect of the architecture is that the encoder stage can map the input to a guaranteed group invariant feature and a group action such that the decoder operates on the group invariant feature and produces the reconstruction of the input in a canonical form. The predicted group action is then applied on this canonical decoder output to reconstruct the input exactly (assuming the autoencoder is trained well). Experiments on images and point clouds are performed for a variety of groups.", "questions": "The authors should discuss what, if any, are the downstream applications of such a method. Is it data compression? Or do they envision the group invariant features learned by their method to be useful for high-level tasks like robust classification?\n\nRelated to reference (a) above, if the eventual goal is to use these group-invariant features for tasks like classification, I wonder why we need to find the group action at all. If we have a group-invariant latent feature, we can use a decoder and a group invariant reconstruction loss for training the autoencoder. \n\nI do hope that the authors address the weaknesses in the paper. Particularly, I would like them to clarify the contributions of the paper better, especially in light of other closely related work already in literature. Experiments need to be improved as well, as I have indicated.", "limitations": "The authors have not discussed the limitations of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The method provides guaranteed invariance to the transformation group.\n2. The method is general enough to apply to both discrete (such as permutations) and continuous (such as rotations and translations) groups, and to a diverse set of domains -- images, point clouds. \n3. Experiments show that the network is indeed able to learn a group invariant and the corresponding group action when trained with the reconstruction loss alone.\n\n\nWeaknesses:\n\n1. Missing related work: I think the main weakness in the paper is that the authors have missed plenty of related work in literature. I give some examples here of works which consider unsupervised learning of group invariant representations:\n\n(a) Manifold Learning in Quotient Spaces: https://openaccess.thecvf.com/content_cvpr_2018/papers/Mehr_Manifold_Learning_in_CVPR_2018_paper.pdf\n\n(b) Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance: https://www.ecva.net/papers/eccv_2018/papers_ECCV/papers/Zhixin_Shu_Deforming_Autoencoders_Unsupervised_ECCV_2018_paper.pdf\n\n(c) Rate-Invariant Autoencoding of Time-Series: https://ieeexplore.ieee.org/abstract/document/9053983\n\nIn (a), group invariant features are learned in an unsupervised fashion, and a group-invariant loss function is used. Similar ideas have been applied for rotation-invariant autoencoders for spherical images as well. In (b) and (c), the invariant feature and the group transformation are disentangled in an autoencoder which is closely related to this submission. In (b) and (c), the ideas are applicable to much larger groups than considered in the present submission as well -- diffeomorphisms -- which I believe cannot be handled by the method presented submission, at least for now. \n\nGiven these works, I think the authors should significantly improve the related work section. They should put this submission in context of such works and better explain the main advantages their method provides.\n\n2. Missing comparison with all these related work: I do think that the papers I have cited and the ideas therein are closely related to the spirit of this work. And thus, the authors should provide some experimental comparisons to one or more of these works.\n\n3. MNIST and Tetris are toy datasets in my opinion. At least some experiments with more complex datasets like CIFAR-100, Patch Camelyon or ShapeNet are needed to better validate the effectiveness of the method. Authors do not provide quantitative measures of reconstruction quality for all experiments, which makes reproducibility hard. Also, non-equivariant baselines and corresponding quantitative evaluation are useful to compare the proposed method and to find out whether indeed the group invariant and equivariant network can be significantly smaller for the same reconstruction quality. \n\nOverall, I think there are too many weaknesses in this submission and am recommending a rejection. \n\n\n## Post author response\n\nI think the authors have addressed all my questions well. They have also addressed the weaknesses pointed out by the other reviewers well. The additional experiments and comparison with the QAE baseline make the contributions of the paper stronger. \n\nI hope the authors will expand their related work section and put the work in context of several other closely related papers (I only mentioned three, but there are others). I don't fully agree that because some of the papers can also model local transformations and this submission can handle only global transformations, that these papers are not closely related. In fact, they point to some limitations of the current work. For example, how can this submission be extended to handle infinite-dimensional groups which are important in image and time series registration? I am now recommending a weak accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pZOP7eIYdfJ", "reviewer": "Reviewer_4UN9", "summary": "Equivariant neural networks have been shown to improve data efficiency and model performance, but require knowledge about the group acting on the data space to be implemented. This work promises to learn the group action from data instead, foregoing the need for ad hoc implementations of the group action on the feature maps / data space. To this end, authors propose learning two separate mappings: (1) from the input sample to a group-invariant canonical representation and (2) from the input sample to a group element / group action. Together, these allow for reconstructing the input sample under its observed pose. The authors show, in a range of experiments, that their framework is able to learn to successfully reconstruct input samples under their respective poses, from an invariant embedding and a learned group action.", "questions": "1. I would like to see the authors address my concern regarding their claim on avoiding the necessity of ad-hoc group-specific implementations. Is it not correct that you need such group-specific implementations of $\\eta, \\mu$ and $\\xi$ in your framework? If so, how does your framework reflect the claim that you are \"learning the group action\"?\n\n2. I don't fully understand the need for learning separate encoders $\\eta$ and $\\mu$. Can you not reuse the same embedding in both cases and to obtain $z$ simply group invariantly project over the equivariant embedding?\n\n3. I'm not completely clear on the need for separating equivariant and invariant representations in the first place, could the authors expand on this a bit? Is it simply because we want to place no constraints on the decoder $\\delta$ that we discard information on group action over $X$ in the $\\eta\\rightarrow\\delta$ pathway? But if we want a latent space structured according to a group action, can we not simply use an equivariant encoder and decoder?\n\n4. Regarding the experiments, in 5.1 you mention that, upon inspection, the encoded latent representation is indeed rotation invariant up to machine error for 90 deg rotations, and shows interpolation artefacts otherwise. But this is just a consequence of the method of interpolation you are using to rotate the images and the encoder (SO(2)-CNN) you are using, and isn't necessarily a result of your framework correct? You use an encoder which is invariant, hence the learned representation is invariant. I also believe the invariance property of your encoder completely explains the rotation invariant structure of your latent space. To me this doesn't seem a result of your framework, but of the (group-specific) encoder you are using?\n\n5. For the set of digits experiment in 5.2, the experimental setup is not clear to me. Is the task permutation invariant or equivariant? Do you only evaluate the presence of digits in the reconstruction or their order as well? If you are only evaluating the set contents, is the equivariant pipeline discarded for this experiment? Or, if you are evaluating their order as well, I think Fig 4a is misleading, as in addition to the embedding your framework needs the permutation matrix for reconstruction (which is of larger dimensionality than the embedding).\n\n6. In experiment 5.3 you investigate a SE(3) and S_N autoencoder. From line 322, it seems you are only evaluating your reconstruction up to to a translation and permutation (you are only predicting a rotation with $\\psi$)? So the autoencoder as a whole is invariant to permutations and translations and only equivariant to rotations, correct? In which case, why didn't you make it invariant to translations and permutations as well? As your autoencoder is invariant to translation, why do you apply translation augmentations to the input?\n\n7. In line 330 you note that your SE(3)-equivariant model (which it is not, it is only SO(3)-equivariant) can distinguish between mirrored chiral shapes, and that SE(3)-invariant models are unable to do so. I believe this is incorrect; reflections are part of the E(3) group, and not of the SE(3) group, hence SE(3)-invariant models should be able to distinguish between these shapes. Could you comment on this?\n\n\n**General post-rebuttal response**\nI would like to thank the authors for their very thorough rebuttal, addressing most of my concerns, and the concerns raised by the other reviewers. I think the work significantly improved in experimental evaluation and comparison. As indicated in my last comments, parts of the motivation and framing of the model proposed by the authors remains misleading / incorrect to me. I would recommend that, were this work to be accepted, the authors review some of the claims they make in the paper regarding the \"learning of a group action\". I feel such claims should be addressed before publication.\n\nThe authors did address many of my questions regarding the broad motivation for their invariant-equivariant approach, but from my point of view, the authors skip a step by discarding fully invariant aggregation of equivariant self-supervised representations. I would recommend the authors empirically or theoretically show that such representations don't work in practice, as motivation for their (arguably more complex) approach. \n\nOn the condition of inclusion of their responses in an updated version of the manuscript, I raise my recommendation to a borderline accept.\n\n", "limitations": "I did not see the authors address any limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality and Significance** The authors propose a sound method for implementing group equivariant auto-encoders. I have some concerns regarding claims the authors make throughout regarding learning the group action. If my understanding is correct, this method only \"learns\" the group action on the data space in the sense that the $\\mu$ operator learns a map from any point in the data space $x \\in X$ into an embedding space $Y$, where this map is required to be equivariant and $Y$ is required to be a homogeneous space of the group. Next, an operator $\\xi$ essentially maps this homogeneous space into the left-regular group action. The whole point of this encoder is to extract pose, but since you presuppose the encoder to be equivariant to the group action, is this not trivial?\n\nSince $\\mu$ is equivariant to the group action it inherently preserves pose, i.e. prop 2.7 is a property of any operator equivariant to the group action, and therefore the group action isn't actually \"learned\"; its structure in the embedding space is given by the equivariant operator. Furthermore, this separation between equivariant and invariant representations hardly seems like a contribution; as you yourself mention, most applications of equivariant neural networks learn equivariant representations which are then projected to an invariant representation. I don't fully understand the need for learning separate encoders $\\eta$ and $\\mu$. Can you not reuse the same embedding in both cases and simply apply a symmetric operation to $\\mu(x)$ obtain to obtain $z$? In fact, for experiment 5.1 you mention that you use the same architecture for both pipelines, why not share the representation?\n\nI would like to see the authors note on how this work relates to Keller & Welling (2021). The methods seem related in that both authors propose learning group equivariant latent representations from data without explicitly encoding the group action.\n\n**Quality and Clarity** The authors explain their derivations in thorough and clear manner. In section 3 I'm left with some questions. [line 186, 187] \"*we turn to describing our construction of $\\xi, \\mu$ and $Y$ for a number of groups*\". For ${\\rm SO(2)}$ you only describe how to construct $Y$ and $\\xi$, what about $\\mu$? For ${\\rm SO(3)}$ you only describe $Y$, what about $\\xi, \\mu$? Same for the rest of the examples. Could you address this?\n\n$\\xi$ needs to be constructed by hand, i.e. a new group requires a new implementation of $\\xi$ AND a new implementation of $\\eta$ and $\\mu$ because of new equivariance constraints. There seems to be no actual learning of the group action and although you claim your method \"avoids the necessity of ad-hoc group-specific implementations\" it does in fact require such group-specific implementations to be implemented itself?\n\nFrom my view, the experiments used by the authors to investigate and evaluate their approach do not adequately show its merits. For details on my concerns, see below questions 4-7. The experiments do not convince me of the need for separate invariant-equivariant terms. To this end, could the authors, in their experimental setups, compare with a fully equivariant encoder-decoder?\n\n*References*\n\nKeller, T. A., & Welling, M. (2021). Topographic vaes learn equivariant capsules. Advances in Neural Information Processing Systems, 34, 28585-28597.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gZFXNuV-DlI", "reviewer": "Reviewer_E8UY", "summary": "Designs an autoencoder that projects each input element into a pair of vectors: one that is group-invariant, and one that represents the group action necessary to recover the original vector from the decoded group-invariant representation.  The group action encoder is designed as a learnable map followed by a deterministic map; the consequent of proposition 2.7 is a sufficient condition to guarantee that this construction results in a suitable group function.  Learnable maps are proposed (and proven suitable) for 2d and 3d rotation groups, permutation groups, and translation groups.", "questions": "The notation is challenging, and its challenges are exacerbated by typos and undefined terms.  For example:\n\nGL(X) is never defined.\n\nOn line 82 of p. 3, the application of f_X to g and x is written as f_X(g)x, but in equation (1), it is written as f(\\rho_V(g^{-1})x) -- I think that in both cases, the x should be inside the parentheses.\n\nA map \\phi is said to be G-equivariant if \\psi(...) = ...\\psi -- either the \\phi should be a \\psi or vice-versa\n\nwe wish to learn the invariant map \\phi_{inv}, thus... --- but \\phi_{inv} is never mentioned in the definition on the next line\n\nJust in general, I didn't understand the purpose of the \\rho_X(g) notation.  If I had to guess, I would say that it takes a general representation, g, of a member of the set G, and converts it into the instantiation of that member that is applicable to external dataset X?  The definition of this notation in the text is self-contradictory and unhelpful: a group action is defined as f:G,X -> X, then a representation is defined as (1) a group action, but also (2) as a function f:X -> GL(X), which seems to be a contradiction.  Neither (1) nor (2) seems to fit with the way \\rho_X(g) is used in the rest of the article.  \n\n", "limitations": "Limitations and ethical implications are not discussed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Experimental examples show that the proposed encoder results in group-invariant representations with very high classification accuracy and very high cluster purity for rotation, permutation, and rotation+permutation+translation groups.\n\nObviously, many other papers have proposed group-equivariant and group-invariant autoencoders.  This paper seems, to me, to go significantly beyond those previous proposals, because the consequent of Proposition 2.7 provides a sufficient condition for the design of an autoencoder with separable group-invariant and group-action representations.\n\nIt's interesting that Prop. 2.7 says that if mu is G-equivariant, then G_x=G_\\mu, but only the consequent is necessary in Prop. 2.9 to prove that \\xi\\circ\\mu is a suitable group function.  As a result of affirming the consequent in this way, we never get a proof that a suitable group function is also G-equivariant.  It seems intuitively to be the case, but it is not proven, nor, perhaps, necessary.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZdQeyp-BVm4", "reviewer": "Reviewer_CwTC", "summary": "This paper proposes an invariant and equivariant unsupervised learning method, i.e., an invariant and equivariant autoencoder.\nWhen considering such a problem, the naive problem is the decoder construction.\nThis is because when the invariant model maps to an intermediate space, the group action is already obvious in that space, and the information about the group has disappeared.\nThe method used in this paper avoids this problem by adding group labels to the intermediate space, and from their experimental results with rotated MNIST and others, it appears that this method indeed achieves a better representation than the usual auto-encoder.", "questions": "See Weaknesses.", "limitations": "Not applicable.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThis paper is based on a natural conception and solves the problem of invariant auto encoders.\n\nWeaknesses\nThe proposed method and solution are too natural to be non-trivial.\nIt would be better if situations in which expressions created in this way would be useful could be shown.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept