Addressing the Topological Defects of Disentanglement
Authors: Diane Bouchacourt
Mark Ibrahim
Stephane Deny
Venue:  (2021)
Keywords: Disentanglement
Equivariance
Topology
Representation theory
Character theory

Abstract: A core challenge in Machine Learning is to disentangle natural factors of variation in data (e.g. object shape vs pose).  A popular approach to disentanglement consists in learning to map each of these factors to distinct subspaces of a model's latent representation. However, this approach has shown limited empirical success to date. Here, we show that this approach to disentanglement introduces topological defects (i.e. discontinuities in the encoder) for a broad family of transformations acting on images ---encompassing simple affine transformations such as rotations and translations.  Moreover, motivated by classical results from group representation theory, we propose an alternative, more flexible approach to disentanglement which relies on distributed equivariant operators, potentially acting on the entire latent space. We theoretically and empirically demonstrate the effectiveness of our approach to disentangle affine transformations. Our work lays a theoretical foundation for the recent success of a new generation of models using distributed operators for disentanglement (see Discussion).

Under review as a conference paper at ICLR 2021
ADDRESSING THE TOPOLOGICAL DEFECTS OF DISEN- TANGLEMENT
Anonymous authors Paper under double-blind review
ABSTRACT
A core challenge in Machine Learning is to disentangle natural factors of vari- ation in data (e.g. object shape vs pose). A popular approach disentangle- ment consists in learning to map each of these factors to distinct subspaces model's latent representation. However, this approach has shown limited em- pirical success to date. Here, we show that this approach to disentanglement introduces topological defects (i.e. discontinuities the encoder) for broad family of transformations acting on images encompassin simple affine trans- formations such as rotations and translations. Moreover, motivated by classical results from group representation theory, we propose an alternative, more flex- ible approach to disentanglement which relies on distributed equivariant oper- ators, potentially acting the entire latent space. We theoretically and em- pirically demonstrate the effectiveness of our approach to disentangle affine transformations. Our work lays theoretical foundation for the recent success of new generation of models using distributed operators for disentanglement (see, discussion). All code is available at htttps://anonynnn horprnnmmmouuuuuuun..... seceeeceeee scciece/ 5laeeeeeer//77722 54dc- fede-ddebccee btw-bbccc-////ffc1111.
INTRODUCTION
Learning disentangled representations is arguably key to build robust, fair, and interpretable MI. systems (Bengio et al. 2013 Lake et al. 2017;; Locatello et 2019a). However, it remains unclear how achieve disentanglement in practice. Current approaches aim map different factors of variations in the data to distinct subspaces of latent representation, but have achieved only limited empirical success (Higgins al.] 20166] Burgess al. (018).. More work on the theoretical foun- dations of disentanglement could provide the key to the development of more successful approaches.
In its original formulation, disentanglement consists in isolating statistically independent fac- tors of variation in data into independent latent dimensions. This perspective has led to a range of theoretical studies investigating the conditions under which these factors are identifiable (Locatello et al., 2019b, Shu et al., 2020 Locatello et al., 2020 Hauberg 2019 Khemakhem et al.) 2020). More recently, Higgins et al. (2018) has proposed alternative perspective connecting disentangle- ment to group theory (see Appendix for a primer on group theory). In this framework, the factors of variation are different subgroups acting the dataset, and the goal is to learn representations where separated subspaces are equivariant to distinct subgroups -a promising formalism since many transformations found in the physical world are captured by group structures (Noether) 1915). However, the fundamental principles for how to design models capable of learning such equivariances remain to be discovered (but see Musslles-Dupré et al. (2019))
Here we attack the problem of disentanglement through the lens of topology (Munkres) 2014). We show that for a very broad class of transformations acting on images -nccopassing all affine transformations (e.g. translations, rotations), an encoder that would map these transformations into dedicated latent subspaces would necessarily be discontinuous. With this assurance, reframe disentanglement by distinguishing its objective from its traditional implementation, resolving the discontinuities of the encoder. Guided by classical results from group representation theory

---

Under review as a conference paper at ICLR 2021
A
X
4
K
7
4
H
4
4
VINE
4
4
4 4
4
4
Y
4
5
5
S
4
4
CC
MAE
5
9
9
4
CN
VAE
Beta VAE
CCI VAIE
VAE
COIVAE
Letarn Dissersion
Latent Dimerci11
Latent Dimension
progral
LEs"
Loo
252
200
72
200
2447
1027
220
2024
2007
324
J
prrinal
OF
30"
72'
100
144"
200"
2167
252"
239
201
Figure 1: Failure modes of common disentanglement approaches. A. Latent traversal best cap- turing rotation for VAE, 3-VAE, and CCI-VAE for rotated MNIST restricted to single digit class ("4"). B. Same as panel A for all 10 MNIST classes. C. Variance of single latents in response to image rotation, averaged over many test images. D. Ranked eigenvalues of the latent covariance ma- trix in response to image rotation, averaged over many test images. E. A supervised disentangling model successfully reconstructs some digits (top) but fails on other examples (bottom). F. Failure cases of the supervised model trained on a dataset of 2000 rotated shapes (see also Fig. 8
Scott & Serre 1996), we then theoretically and empirically demonstrate the capacity of model equipped with distributed equivariant operators in latent space disentangle range of affine image transformations including translations, rotations and combinations thereof.
2 EMPIRICAL LIMITATIONS OF TRADITIONAL DISENTANGLEMENT
In this section empirically explore the limitations of traditional disentanglement approaches, in both unsupervised (variational autoencoder and variants) and supervised settings.
VAE, DAE and CCI-VAE We show that, consistent with results from prior literature, a variational autoencoder model (VAE) and its variants successful at disentangling the factors of variation on simple dataset. We train a VAE, beta- VAE and CCI- VAE (Kingma & Welling 2014
Higgins al. 2016 Burgess et al. 2018) a dataset composed of single class of MNIST digits (the *4s" ), augmented with 10 evenly spaced rotations (all details of the models and datasets are in App. B After training, qualitatively assess the success of the models disentangle the rotation transformation through traditional latent traversals: we feed an image of the test set to the network and obtain its corresponding latent representation. We then sweep range of values for each latent dimension while freezing the other dimensions, obtaining a sequence of image reconstructions for each of these sweeps. We present in Fig. [ examples dimension, selected to be visually closest to a rotation (see Fig. f latent traversals along all other latent dimensions). We find that these models are mostly successful at the task of disentangling rotation for this simple dataset, in the sense that sweep along a single dimension of the latent maps to diverse orientations of the test image.
latent traversals along single latent
We then show that on a slightly richer dataset (MNIST with all digits classes), a VAE model and its variants fail to disentangle shape from pose. We train all three models studied (VAE,
2

---

Under review as a conference paper at ICLR 2021
beta- VAE, CCI-VAE) MNIST augmented with rotation, and find that all these models fail to disentangle rotation from other shape-related factors of variation (see Fig. I for the most visually compelling sweep and Fig. 6 for sweeps along all latent dimensions). We further quantify the failure of disentanglement by measuring the variance along each latent in response to a digit rotation, averaged over many digits (see Fig. [ and details of analysis in App. []. We find that the information about the transformation is distributed across the latents, in contradiction with the conventional notion of disentanglement. One possibility would be that the direction of variance is confined to a subspace, but that this subspace is not aligned with any single latent. In order to discard this possibility, we carry a PCA-based analysis on the latent representation (Fig. IP and App. E and we show that the variance in latent representation corresponding to image rotation is not confined a low-dimensional subspace.
Supervised Disentanglement We further explore the limitations of traditional disentanglement in a supervised framework. We train an autoencoder on pairs of input and target digit images [E) where the target image is rotated version of the input image with discrete rotation angle indexed by an integer value k. The imput image is fed into the encoder to produce latent representation. This latent representation then multiplied by a matrix operator L'k. parameterized by the known transformation parameter k. This matrix operator, which we call the disentangled operator, is composed of a 2-by-2 diagonal block with rotation matrix and an identity matrix along the other dimensions (shown in Fig. (... The disentangled operator (i) is consistent with the cyclic structure of the group of rotations and (ii) only operates on the first two latent dimensions, ensuring all other dimensions are invariant to the application of the operator. The transformed latent is then decoded and compared to the target image using an L2 loss (in addition, the untransformed latent is decoded and compared to the original image for regularization purposes). The only trainable parameters the encoder and decoder weights. We use the same architecture for the encoder and decoder of this for the VAE models in the previous section. This supervised disentanglement
model that model partly succeeds in mapping rotation to a single latent on rotated MNIST (Fig. [ top row). However, there remains some digits for which disentanglement fails (Fig. D bottom row). it-is diffieult to evaluntee theeettttt because MNIST only composed of 10 classes of shapes corresponding to the 10 different digits. To further expose the limitations of this model, we design a custom dataset composed of 2000 simple shapes in all possible orientations. When trained on this extensive dataset, we find that eer the model fails to capture rotations on many shapes. Instead, it replaces the shape of the input image with a mismatched stereotypical shape (Fig. F We reproduce all these results with translation in the appendix (Fig. 8-1
to rotate many different images with MNIST,
In conclusion, find that common disentanglement methods are limited in their ability to disentangle pose from shape in a relatively simple dataset, even with strong supervision (see also Locatello al. (0119b)) We cannot empirically discard the possibility that larger model, trained For longer on even more examples of transformed shapes, could eventually learn to disentangle pose from shape. However, in the next section we will prove, using arguments from topology, that under the current definition of disentanglement, an autoencoder cannot possibly learn perfect disentangled representation for poses and shapes. Sec. 48.4 and Sec__ 4 we will show that another type of model inspired by group representation theory can properly disentangle pose from shape.
3 REFRAMING DISENTANGLEMENT
In this section, we formally prove that traditional disentanglement by a continuous encoder is math- ematically impossible for a large family of transformations, including all affine transformations. We then provide a more flexible definition of disentanglement that does not suffer from the same theoretical issues.
3.1 MATHEMATICAL IMPOSSIBILITY OF DISENTANGLEMENT
We first consider simple example case where disentanglement is impossible. We consider the space of all images of 3 pixels X = R3, and the transformation acting on this space to be the group of integer finite translations, assuming periodic boundary conditions of the image in order to satisfy
3

---

Under review as a conference paper at ICLR 2021
A
Image space X
Latent space Z
Zx
2
J²3
3
#g1
fE
22
22
ZF2
Figure 2: Visual proof of the topological defects of disentanglement. Top left: O1, O2 and O3 three examples of orbits of 3-pixel-images transformed by translation. Bottom: (left) orbits visu- alized in image space (points constitute the orbits, continuous lines are for visualization purposes); (right) orbits in latent space. When projected onto the equivariant subspace Ze (gray dotted lines), all orbits should collapse onto each other. Yet the orbit of a uniformly black image (red dot) con- tains a single point and thus cannot be mapped onto the other orbits. B. Discontinuity of fr around symmetric images. Top: consider an image of an equilateral triangle, with an infinitesimal pertur- bation on one corner (black dot), undergoing rotation (color changes are for visualisation purposes). Bottom: (left) after rotation of 120%, the orbit in image space (here projected onto 3 dimensions for visualisation) almost loops back on itself: (right) in Zg. each angle of rotation corresponds distinct point in space. Therefore, the encoder fE is discontinuous (as shown by the red arrows).
the group axiom of invertibility (Fig. 2A, see App. A for definitions). Given an image, the set of images resulting from the application of all possible translations to this image is called the orbit of this image. We note that the space of images IR3 is composed of an infinite set of disjoint or- bits. Can we find an encoder which maps every point image space X disentangled space Z?
To conform to the conventional definition of disentanglement (Higgins al., 2018) (see App-[ffor a formal definition). Z should be composed of two subspaces, namely (i) an equivariant subspace ZE containing all and only the information about the transformation (i.e. location along the orbit) and (ii) an invariant subspace Zr, invariant to the transformation but containing all other information about the image (i.e. identity of the orbit). Each orbit should thus lie plane parallel to Z (otherwise some information about the transformation would leak into Zz), and all orbits projected onto Zg should map onto each other (otherwise some information about the identity of the orbit would leak into Zg). We now consider the orbit containing the black image [0,0,0]. Since all translations the black image are the black image itself, this orbit contains only one point. And yet, the image of this orbit in Zg should conform the image of other orbits, which generally consist of 3 distinct points. Since a function cannot map a single point to points, an encoder f ensuring disentanglement for all images cannot exist.
Using similar topological arguments, we formally prove the following theorem (App. generalizing the observation above to large family of transformations including translations, rotations and scalings.
Disentanglement into subspaces by a continuous encoder impossible for any finite group acting Euclidean space R.N.
Theorem 1:
3.2 PRACTICAL EXAMPLES OF TOPOLOGICAL DEFECTS
The formal theorem of the previous section does not tell us how hard it would be to approximate disentanglement practice. We show next that a disentangling encoder f would need to be discon- tinuous around all images that present symmetry with respect to the transformation, which makes this function very discontinuous in practice. As an example, we consider the image of an equilateral triangle undergoing rotation (Fig. 2B, color changes are for visualisation purposes). Due to the sym- metries of the triangle, a rotation of 120 of this image returns the image itself. Now we consider the same image with infinitesimal perturbation on one corner of the triangle, breaking the symmetry
4

---

Under review as a conference paper at ICLR 2021
of the image. A rotation of 120 of this perturbed image returns an image that is infinitesimally close to the original image. And yet the equivariant part of the encoder ff (i.e. the projection of f onto the equivariant subspace Zr) should map these two images to disjoint points in the equivariant subspace Zg. in order to properly encode the rotation transformation. Generalizing this argument to that disentangling encoder would be discontinuous in the neighbor- hood of all images that present a symmetry with respect to the transformation to disentangle. This incompatible with most deep learning frameworks, where the encoder is usually neural network implementing a continuous function. We provide formal proof of the discontinuity of fg in App-
all symmetric images,
The invariant encoder fi (i.e. the projection of f onto the invariant subspace Z1) also presents topological defects around symmetric images. We provide both visual proof and a formal proof of these defects in App. C.2
3.3 A MORE MORE FLEXIBLE DEFINITION OF DISENTANGLEMENT
An underlying assumption behind the traditional definition of disentanglement is that the data is naturally acted upon by set of transformations that orthogonal each other, and that modify well-separated aspects of the data samples. However, in many cases this separation between fac- tors of variation of the data is not possible (as also noted by Higgins al. (2018)). We notice that the current definition of disentanglement unnecessarily conflates the objective of isolating factors of variation with the algorithm of mapping these factors into distinet subspaces of the internal repre- sentation. In order to build model that respects the structure of the data and its transformations. the latent space should instead preserve the entunglement between factors of variation that are not independent. Thus, we-
A model equipped with a latent operator is equivariant to a transformation if encoding a sample then applying the latent operator is equivalent to transforming the sample first then encoding, Formally, we say a model fiX + is equivariant to a transformation Ot. with parameter & if for any input
du (x
(1)
where K_ is the sheespace.o. of transformation parameters, The operators 6R and ussssssc. capuure how the space, and representation space respectively. With this view, transformation _o_____a__ acts on the i
we turn to a definition of disentanglement in which the transformations are modelled as distributed operators (i.e. not restricted to a subspace) in the latent space.
Definition 1:
Definition 1. A representation ix disentangled with respect to a set of transformations, there is family of controllable operators, potentially acting on the entire representation, where each operator corresponds to the action of a single transformation and the resulting model is equivariant model
These operators are A representation said to be disentangled with respect to a particular is known operators acting on
of gremp group into this peiissepppeennttionnnneee poientially posteenteeellllyddsssstteeddddd distributed nevoss full latent, eetleeee-eehhe operatee is equivvriiet te-the-actions single single-subrrene caanbrroo..oonrrllln form, thus allowing the user to manipulate the latent representation by applying the operator, This definition, more flexible than traditional disentanglement in the choice of the latent operators, while serving the same objective as traditional disentanglement (isolation and identification of obeys to the same desiderata of identification and isolation of the factors of variations present in the data).
This definitionis in the sense that they have an explicit
3.4 THE SHIFT OPERATOR FOR AFFINE TRANSPORMATIONS
The flexibility of D unlocks powerful toolbox for understanding and building disentangled models...Tansiee Transformations in data_often have additional structure structure_describbng how to_((__)) (oo_((___ndd undo transformation (invertibility), (2) leave sample unchanged (identity), and (3) (3) rearrange parentheess (nnpntttt)_.d______._
for the order transformations are applied (associativity). A collection of transformations with a rule for combining two transformations into another satisfying these th requirements has powe group structure (see App. A for a formal definition). With a group structure, we can decompose transformations into subgroups, describe how to represent transformations as matrices, and build
5

---

Under review as a conference paper at ICLR 2021
flexible disentangled models.
Using classical results from the linear representation of finite groups (Scott & Serre [996), we show Ap- now that a carefully chosen distributed operator in latent space, the shift operator 4/x (shown in Fig. 3/) is linearly isomorphic specific transformations that include integer pixel translations and rotations, With this operator, we can_ learn a latent space equivariant any affine transformation using a simple, linear autoencoder, Consider linear encoder model -
f. If we want W. to be an equivariant invertible linear mapping. W between X and Z,
Eguation Sereatiionnrrriiinn Trwwittes as follows:
W Wor()) On vn W
(2)
where 6b and Lk are the representations of De G. on the image and latent Satentsspaee respectively wwweeee
as defined in App: [ For W. to be equivariant, Equation 2 must be true for every image Is W is invertible, Equation true if and only vk K, the two representations L and @x are isomorphic:
Wk K, K the W
We consider additional properties on d corresponding, to the assumptions (i) that G is cyclic of order K with generator lo and (ii) & is isomorphic to the regular representation of G (see Scott & Serre (1996)) These properties are respected by all cyclic linear transformations of finite order K of the images (see App. A for definitions). Special-eases-e eases of these tetassoorrmtiiins-eeeeeeeee ale anserete-and-eyeee afffee transformations, such as integer pixel translation with periodic boundary conditions, or rotations. practical consequence of this proof is that it is possible learn latent space equivariant to any affine transformation with this operator, using simple linear autoencoder.
Definition and properties of the shift operator Fer-a transformation 91 C such that dx sù where 9u is the generater of Given that the encoder and decoder are linear and invertible, the two representations ø and yt must be isomorphic. Two representations are isomorphic if and only if they have the same character (Scott & Serre 1996 Theorem 4. Corollary (see App A Afror definition of characters). We thus want choose vi such that it preserves the character of the representation corresponding to the action of G the correspooddnngg shift operator the is the enittrnittrryyy shift matrix expenentiated by le. Note that this operator is distributed as acts on the dataset of images. Importantly. we_ will see that our proposed operator needs to be distributed in the sense that it should act on the full latent spaee, contrary to conventttiaal disentanglement inpteas. the proposed shift operator is flexible in the sense that it does not require specific knowledge of which transformation groupis applied to the data (i. e. for example rotations or translations), but only the order of the group and the character of its representation (App. A Furthermore, in App- code.
Let us consider the matrix of, order K C.L.Chh corresponds to a shift of elements in K-dimensional vector by positions. We construct from Mj. the shift operator as representation of the group's action on the latent space._.For For each ls. E its corresponding shift operator is the block diagonal matrix_oo of order N. composed of N repetition of M -
k
0
M
M4
0
:
(4)
4/4 =
M
(5)
M
We show in App. C that the shift operator has the same character as the representation corresponding to the action of G on the dataset of images. Thus, two are isomorphic and using this shift operator ensures that an equivariant invertible linear mapping W exists between -ore image space and the latent space equipped with the shift operator,_ In Appendix C33.2 we alse prepese prepene-eeeemple complex version opetttttosoooo that we can also replace this shift operator by

---

Under review as a conference paper at ICLR 2021
a complex diagonal operator, operator which is more computationally efficient Although these operaters cannot theoretically be mapped to continuous transforms, we note that continuoss transform ean be appreximated using finite-groups, In the remainder of the paper, we denote $1,,,v shift of operator-that opprator--aat-erverpponttt--et Co ot transformmtiions (e. g. rotations)ef order with generator fo order K; such that -=
In the next section, we show how these theoretical results can lead to practical and effective disentangling models for affine transformationssuch as rotations and translations. multiply with the latent...
The shift operator computes shift of the latent space and we use this form of operator to represent any finite cyclic group of affine transformations (ieeeet either rotation, translation in X, translation in y) The role of the encoder is to construct latent space where transformations can be represented as shifts. Importantly, the shift operrtoo does not require knowledge of the transformation in advance, only the cycle order of each group, which is assumption we. will relax, with the weakly supervised version of the shift operator These assumptions allow us to guarantee a linear equivariant model can be learned with pairs of examples (see our training. objectives in App. B Note that the shift operator only handles cyclic groups with finite order (or a product of such groups), In order to tackle continuous transformations. discretisation step could be added and we leave the exploration of this extension for future work.
4 DISTRIBUTED DISENTANGLEMENT IN PRACTICE
Our empirical (Sec. 2) and theoretical (Sec. 3) findings converge to show the difficulties of dis- entangling even simple affine transformations into distinct subspaces. Here show that, using distributed equivariant operators instead, it is practically possible to learn to disentangle these affine transformations, according to our more flexible definition of disentanglement.
4.1 THE SUPERVISED SHIFT OPERATOR MODEL
Guided by our theoretical results, we train a supervised non-variational autoencoder using pairs of samples and their transformed version (with a known transformation indexed by k) using the distributed shift operator from Sec. 3.4 (shown in Fig. (3A) instead of the disentangled operator from Sec. 2. We feed the original sample x to linear invertible (or q _ see App- B.2). encoder that produces latent representation. The latent representation is then multiplied by the shift operator matrix parametrized by k. The transformed latent then decoded and L2 loss between the two reconstructions (xr and its transformed version) and their respective ground-truth images is back poock-prppggted.
quasi-invertible,
As predicted by character theory, our proposed model is able to correctly structure the latent space, such that applying the shift operator to the latent code at test time emulates the learned transformation (see Fig. Aand test MSE reported in T Also consistent with, ,nn the LSBD disentangiement baaggnnnt__mssuuuee (eeasure_from Annnyyous (2021) 1) and reported in App.
Interretttt test MSS for rotations are lower than for translations. We believe this is due to the fact that in the case of translations, changes in the image induced bacieedbyy_eeeeeeeeeee by__accc_hhhh____e____ee transformation are less visually striking than with rotations. Consistent with the theory, the same linear autoencoder equipped with a disentangled operator fails at learning the transformation (Fig. 3B. We reproduce these results for trans in App. E We thus show that, unlike prior approaches, our theoretically motivated model is able to learn to disentangle affine transformations from examples.
4.2 THE WEAKLY SUPERVISED SHIFT OPERATOR MODEL.
Here show that method can also learn to disentangle transformations in weakly supervised setting where the model is not given the transformation parameter between pairs of transformed images (e.g. rotation angle) during training. We consider the case of a single transformation for simplicity. We encode samples by pairs x1, (with 22 a transformed version of x1) into 2 and 22 respectively, and use a phase correlation technique to identify the shift between z and 22, in
qusssuiinveettblee see App.
7

---

Under review as a conference paper at ICLR 2021
A
A
M/A
M
3
1%
cool
aoo
sin
200
0
J
144
LAST
212
OF
12"
146"
140
2107
252
200
224
9
D
+
-
C
D
K
0.72
126
13:09
11A
6
611
15.24
1:12
24.44
E
1
252
P
229
200
23
M
24
Figure 3: Success and flexibiliity of proposed distributed shift operator models: A. Proposed shift operator model successfully learns rotation on simple shapes. B. Disentangled operator fails learn rotation. C. Weakly supervised shift operator model, using the complex version of the shift operator, successfully rotates simple shapes. Note that the model maps ground-truth counter clockwise rotations to clockwise rotations, while respecting the cyclic structure of the group. D. Stacked shift operator model succeeds conjunction of translations. E. Stacked shift operator model succeeds on conjunction of translations and rotations. Numbers above plots indicate rotation angle and/or translation in and y respectively.
Reddy & Chatterji (1996). An L.2 loss applied on reconstructed samples, with the original sam- ple transformed according to all possible ht and weighted according to soft-max scores given by the cross-correlation method (see App. B.3 for details). Here and in the remainder of the experiments, we use the complex version of the shift operator for computational efficiency (shown in Fig. 3P) This weakly supervised version of the model has an extra free parameter which is the number of latent transformations, not known priori. Let us denote Kд this number, which can be different than the ground-truth order of the group K. We explore the effect of different Ky, in App. B.3 The results of this model (with 10 latent transformations K2) are shown in Fig. 3C The weakly super- vised shift operator model works almost as well as its supervised counterpart, and this is confirmed by test MSE (see Table 2). The same model can successfully be trained on MNIST digits (Fig. [4].
4.3 MULTIPLE TRANSFORMATIONS STACKING SHIFT OPERATORS
So far, we have only considered the case of a single type of transformation at time. When working with real images, there are more than one type of transformation jointly acting on the data, for example a rotation followed by a translation. Here we show how we can adapt our proposed shift operator model to the case of multiple transformations.
Stacked shift operator model In the case of multiple transformations, group element is com- position of consecutive single transformations. For example, elements of the Special Euclidean group (i.e. translations and rotations) are composition of the form ay h, where is an ele- ment of the z-translation subgroup, a, of the y-translation subgroup, and h of the image rotation subgroup. Our theory in C.3 ensures that each of these subgroup's action in image space is linearly isomorphic to the repeated regular representation th We can thus match the structure of the learning problem by simply stacking linear layers and operators. We build stacked version of shift operator model. in which we use one complex shift operator for each type of transformation, and apply these operators the latent code in sequential manner, akin to Tai al. (201)). Specifically,
8

---

Under review as a conference paper at ICLR 2021
consider image X that encounters a consecutive set of transformations 9n-9n-1: 0n.1..n....... 91, with 9: E G, Vi. We encode x into 2 with a linear invertible encoder 2 = W z. We then apply the operator the latent space, 1 (91 ), corresponding to the representation of gi on the latent space Z. We then apply a linear layer 1, before using the operator corresponding to G2. The resulting latent code after all operators have been applied is z' 1/vn (g.) ... 1/2 (92) 1122. The transformed latent code x' is fed to the linear decoder, in order to produce a reconstruction that will be compared with the ground truth transformed image x', in the single transformation case.
Translations in X and Y Consider the conjunction of translations in and y axes of the image. This is finite group of 2D translations. This group is a direct product of two eyclic groups, and it is abelian (i.e. commusaative.. We refer the interested reader to App. D.I for details on direct products. To tackle this case with the stacked shift operator model, we first use the shift operator 2,2.A.N corresponding to the translation in then apply linear layer denoted L1. before using the operator Vy. corresponding to the translation in y: 2' = Vy.k L1 Kk,,k 2. We train this stacked model on translated shapes with 5 integer translations in both and y axes (i.e. the group order is 25). Results reported in Fig. 3 show that the stacked shift operator model is able to correctly handle the group of 2D translations.
Translations and rotations We consider a discrete and finite version of the Special Euclidean group, where A is finite group of 2D translations presented in the previous section and H a finite cyclic group of rotations. This group has a semi-direct product structure (see App. D.3 for details) and is coon-commttative, contrary to the 2D translations case. With the stacked shift operators model, we first use the operator Vh.j.N N corresponding to the rotation h, then the one for translation in x. then the one for y- y-translation. The resulting transformed latent code is z' = byy,'' NN L2 L1 1hh.... z. We train this stacked model on discrete rotations followed by integer translations and discrete tations, using 5 integer translations in both x and y axes and 4 rotations. Results reported in Fig. BE and MSE Table show that the model is perfectly able structure the latent space such that the group structure is respected. Additionallyy__igeee reconstructions by the stacked shift operator model in the cases of (i) translation in both x and y axes and (ii) rotations and translations in both axes, In appendix Figure [6] we also explore the case where the order of the group of rotations is 5, breaking the semi-dires product structure (see -
[7 and [8 show_pairs pairs of samples and the
note in Appendix D) and show that the stacked shift operator nonetheless performs with great performance
Insight from representation theory on the structure of hidden layers When dealing with mul- tiple transformations (e.g. rotations and translations). we know the form of the operator for every subgroup (shift operator), but we do not know priori the form of the resulting operator for the entire group. In App. DII and D.4 we derive from representation theory the operator for the entire group in the 2D lannsaation and Special Euclidean group cases and show that they be built from each subgroup's 's operator in a non-trivial way. Importantly, we show that the resulting operator for the discrete finite Special Euclidean case has block matrix form representation based on represen- tations of both translations and rotations. This is expected: this group is non-commutative, so the correct operator cannot be diagonal otherwise two operators corresponding to two elements would commute. Equipped with this theory we can derive insights about the form that intermediate layers should take after training. In particular, we show (App. [.2) that the layer Ln should be block diagonal matrix consisting of repetitions of a permutation matrix that reorders elements in thak z. Similarly, in the case of translations and rotations together, T-2 must reorder &y .N. and T. be the product of two matrices L1 PQ, where Q is N by N block diagonal matrix and P is reordering the rows of the vector Qvh.,,.= = (see App. D.5). In future work, we plan to explore the use of these insights regularize internal layers of stacked shift operator models.
DISCUSSION
Finding representations that are equivariant to transformations present in data is a daunting problem
with no single solution. A large body of work (Cohen et al. 2020 2018 Esteves et al. 2018 Grey- danus et al. 2019 Romero et al. 2020 Finzi et al. 2020 Tai et al. 2019) proposes to hard-code equivariances in the neural architecture, which requires a priori knowledge of the transformations
present in the data. In another line of work, Palorsi et al. (0088) Davidson et al. (208): Falorsi et al.

---

Under review as a conference paper at ICLR 2021
2019) show that the topology of the data manifold should be preserved by the latent representation, but these studies do not address the problem of disentanglement. Higgins et al. (2018) have proposed the framework of disentanglement as equivariance that we build upon here. Our work extends their original contribution in multiple ways. First, we show that traditional disentanglement introduces topological defects (i.e. discontinuities in the encoder), even in the case of simple affine transfor- mations. Second, we conceptually reframe disentanglement, allowing equivariant operators to act on the entire latent space, so as resolve these topological defects. Finally, we show that models equipped with such operators successfully learn to disentangle simple affine transformations.
An important direction for future work will be to expand the reach of the theory to a broader family of transformations. In particular, it is unclear how the proposed approach should be adapted to learn transformations which are not affine or linear in image space, such as local deformations, compositional transformations (acting on different objects present in an image), and out-of plane rotations of objects in images (but see Dupont et al. (2020) for an empirical success using a variant of the shift operator). Another important direction would be to extend the theory and proposed models to continous Lie groups. Moreover, our current implementation of disentanglement relies on some supervision, by including pairs of transformed images(with without knowledge-ett the parameter of the transformation occurring between them). It moreover it would be important to understand how disentangled representations can be leamne without such pairs of transformed (mages-(see Anselmi et al. (2019); /Zou et al. (2020) for relevant work).
Finally, our work lays theoretical foundation for the recent success of a new family of methods that -instead of enforcing disentangied representations to be restricted to distinct subspaces- use operators (hard-coded or learned) acting on the entire latent space (Connor & Rozell 2020 Connor et al., [2020; Dupont et al.] [2020] Giannone et al., [2020] Quessard et al. -n Welling| (Sohl
2020 (see also (Memisevic & H S
et al. (017]) for precursor methods). These methods work well where traditional disentanglement methods fail:
Cohen &
for instance by learning to generate full 360 in-plane rotations of MNIST digits (Connor & Rozell
2020). and even out-of-plane rotations of 3D objects (Dupont et al.] 2020). Unlike our simple theoretically-derrvee models which are are- suited only for alfine transformations,
These methods use distributed operators in combination with non-linear autoencoder architectures, interesting di- rection for future theoretical investigations. Moreover, in the case where the latent operators cannot thsee be determined in advance like in the affine case, these operatorspeeseecceeeooo could be learned like in Connor et
al. A benefit of this approach is that multiple opertoors can be learned in the same subspace, instead of the stacking strategy that we needed use in the case of hard-coded shift operators.
REFERENCES
Anonymous. Quantifying and learning disentangled representations with limited supervision. In Submitted to International Conference on Learning Representations, 2021. URL https:/7
openreview wtteeeewwwwweeeeeeeeeeeeeeeeeeee peet//forum?dd=22--1
NNHj66600 under review.
Fabio Anselmi, Georgios Evangelopoulos, Lorenzo Rosasco, and Tomaso Poggio. Symmetry- adapted representation learning. Pattern Recognition, 86:201-208, February 2019. ISSN 0031- 3203. doi: 10. 1g.10166j.patcog2211..0..00
URL http: / www. wcwencedirect.......cmcm. com/
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1788 2013. Publisher: IEEE
R. Berndt. Representations of Linear Groups: An Introduction Based on Examples from Physics and Number Theory. Vieweg+ Vieweg+Teubne Teubbnr Verlag, 2007. ISBN 9783834894014. URL https: https/ssooooee...ooooooo2.2.
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in S betbbbetaSaaa VAE. arXiv:1804.03500 [cs, star), April 2018. URL.htttp://rrxiv.oor
abs/ /atos//8844 1804. 03599. arXiv: 1804.03599.
Hugo Caselles- Dupré, Michael Garcia Ortiz, and David Filliat. Symmetry- -asee Disentangled Representation Learning requires Interaction with Environments. In H. Wallach, H. Larochelle,
10

---

Under review as a conference paper at ICLR 2021
A. Beygelzimer, F. d' textuuotesingle Alché-Buc, E. Fox, and R. Gareett (cvaan,, in Neu- Curran Associates, Inc., 2019.
ral Information Processing Systems 32, pp.
Taco Cohen and Max Welling. Learning the Irreducible Representations of Commutative Lie Groups. arXiv:1402.4437 [es], May 2014. URL arXiv: 1402,4437.
4437
Taco Cohen, Mario Geiger, and Maurice Weiler. A General Theory of Equivariant CNNs on Ho- mogeneous Spaces. arXiv: 1/811100111 (cs, stat), January 2020. URL http:://txxi...rtttt org/ abs/1811. 02017
arXiv: 1811.02017.
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint arXiv:1801.10100, 2018.
Marissa Connor and Christopher Rozell. Representing Closed Transformation Paths in Encoded Network Latent Space. In AAAI, pp. 3666-3675, 2020.
Marissa C. Connor, Gregory H. Canal, and Christopher J. Rozell. Variational Autoencoder with Learned Latent Structure. arXiv:2006. 10597 [cs, stat), June 2020. URL htttp://arxiv.
org abs/ 2006. 10597
arXiv: 2006.10597.
Tim Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak. Hy- perspherical Variational Auto-Encoders. arXiv:/804.0089 [cs, stat], September 2018. URL
http:://rxiis http:://aaxii.oog/abb/////////// wwwpgg/aa/////// 00891
arXiv: 1804.00891.
Emilien Dupont, Miguel Angel Bautista, Alex Colburn, Aditya Sankar, Carlos Guestrin, Josh Susskind, and Qi Shan. Equivariant Neural Rendering. arXiv preprint arXiv:2006.076300 2020.
Carlos Esteves, Christine Bheenhlla,,e,tee, Xiaowei Zhou, and Kostas Daniilidis. Polar Transformer Networks. 1709 889 [cs]. February 2018. URL http:p//aaaaii..ogg///////1111
01889
arXiv: 1709.01889.
Pavel Etingof, Oleg Golberg, Sebastian Hensel, Tiankai Liu, Alex Schwendner, Dmitry Vaintrob, and Elena Yudovina. Introduction representation theory, 2009. URL http::ttt//xii.....///
abs/ 0827 cite arxiv:0901.088 00827.
Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S. Cohen. Explorations in Homeomorphic Variational Auto-Encoding. arXiv:1807.ne 1807 04689
[cs, stat], July 2018. URL http:://aaxiv.orrgg/////
arXiv: 3807.04689.
Luca Falorsi, Pim Haan, Tim Davidson, and Patrick Forré Reparameterizing Distributions on Lie Groups. abs /1903 02958
1003:990..0955 [cs, math, stat), March 2019. URL htLl:tt//axxv..co
arXiv: 1903.02958.
M. Finzi. S. Stanton, P. Izmailov, and A. G. Wilson. Generalizing Convolutional Networks for Equivariance Lie Groups on Arbitrary Continuous Data. In Proceedings of the International Conference on Machine Vision and Machine Learning. 2020.
Giorgio Giannone, Saeed Saremi, Jonathan Masci, and Christian Osendorfer. No Representation without Transformation. arXiv: 19/2,03845 fes, stat], April 2020. URI. htt.:ttpp///xxii..rggg. bbss/912.. 03845
arXiv: 1912.03845.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pp. 15379-15389, 2019.
Søren Hauberg. Only Bayes should learn a manifold (on the estimation of differential geometric [cs, stat), September 2019. URL httt:://axxiv: arXiv: 1806.04994.
structure from data). 1rX6v org/abs 11t.///06... 04994
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Veta-VAE: Learning Basic Visual Concepts with Constrained Variational Framework. November 2016. URL https: httpas////peereeeeeeeeee
forum? 2orum??1d=Sy2FxU9g11
11

---

Under review as a conference paper at ICLR 2021
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards Definition of Disentangled Representations. arXiv:1812.022. [es, stat], December 2018. URL http://arxxv.or///bb///////// 02230 1812.02230.
arXiv:
Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational Au- toencoders and Nonlinear ICA: A Unifying Framework. arXiv:1907 04489 [cs, stat), February 2020. URL http:://ttaae wwwpxxi..orgggaoo 1907. 04809 arXiv: 1907.04809
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv: arXiv:1312.6114 Jcs, 6114 arXiv: 1312.6114.
star], May 2014. URL
Brenden M. Lake, Tomer Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017. Publisher: Cambridge University Press.
Yann LeCun, Corinna Cortes, and CJ Burges, Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, 2010.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. In Advances in Neural Informa- tion Processing Systems, pp. (4611-14624) 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learn- ing of Disentangled Representations. arXiv:1811. [cs, stat]. June 2019b. URL http:
//arxiv. wwwp///g//bbaaseeee 1111913991111
arXiv: 1811.12359.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-Supervisedr Disenaanleeennnnnnneee Without Compromises. arXiv:2002.088 022888 /cs, stat], June 2020. URL/attp://arxttet
abs 2002. 02886
arXiv: 2002.02886.
Roland Memisevic and Geoffrey E. Hinton. Learning to Represent Spatial Transformations with
Factored Wigher-Order Boltzmann Machines. Neural Computatiion, 1476):14331142, June
2010. ISSN 0899-7667, 1530-888X. doi: 10.11622/ncc22000000-00-99999999
URL https:
www.wwitppp........m..........wt www.miitrees..ommmmm journaal /or http://a...a.a.oooooo///o//////a 2010. 01 09 953
James Raymond Munkres. Topology. Pearson, Harlow, ed edition, 2014. ISBN 978-1-292-02362-
5. OCLC: 904316202.
Emmy Noether. The finiteness theorem for invariants of finite group (translation of Emmy Noether's "Der Endlichkeitsatz der Invarianten endlicher Gruppen" by c. mclarty in 2015). 1503.v/503.0765: [math], March 1915. URL http:://axxis /Btt/ppp///aaa...oo///////////.0
arXiv: 1503.07849.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, AlcheBuc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Robin Quessard, Thomas D. Barrett, and William R. Clements. Learning Group Structure and Disentangled Representations of Dynamical Environments. arXiv preprint arXiv:2002.06991, 2020.
B. S. Reddy and B. N. Chatterji. An fft- based technique for translation, rotation, and scale- inveriiiv image registration. IEEE Transactions Image Processing, 1266:12661127 1996.
12

---

Under review as a conference paper at ICLR 2021
David W. Romero, Erik J. Bekkers, Jakub M. Tomezak, and Mark Hoogendoom. Attentive Group Equivariant Convolutional Networks. arXiv preprint arXiv:2002.03830, 2020.
L.L. Scott and J.P. Serre. Linear Representations of Finite Groups. Graduate Texts in Mathematics.
Springer New York, 1996. ISBN 9780387901909, URL https://boks..oogle..
bbokk?idd-Nf22:55TTOO
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly Supervised
Disentanglement with Guarantees. 1rXiv:190.0997 [cs, stat], April 2020. URL http:
//arxiiv.rrg/abs/11111 09777
arXiv: 1910.09772.
Jascha Sohl-Dickstein, Ching Ming Wang, and Bruno A. Olshausen. An Unsupervised Algorithm For Learning Lie Group Transformations. arXiv:/001.1027 [cs], June 2017. URL http://
arxiv.org/ http//bs/110000 1027 arXiv: 1001.1027.
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant Transformer Networks, Interna- tional Conference on Machine Learning, 2019.
Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: structure for efficient numerical computation. Computing in Science & Engineering, 13(2):22, 2011.
Stéfan van der Walt, Johannes L. Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouillart, Tony Yu, and the scikit-image contributors. scikit- image: image processing in Python. PeerJ, 2:e453, 2014. ISSN 2167-8359. doi: 10.7717/peerj. 453. Utt.httts:////oo org/ 100 771 2tt.10....111111111111 /t1.//peeee.... .53
Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. arXiv preprint arXiv:2007.02933, 2020.
13

---

Under review as a conference paper at ICLR 2021
Appendix
Table of Contents
Prerequisites in Group Theory and Representation Theory
14
B
Experimental Details
15
B.1
Dataset generation
15
B.2
Model architectures and training
B.3
Weakly supervised shilt operator training procedure
17
B.4
Hyper-parameters
18
Reframing Disentanglement: Formal Proots
18
Topological proof against disentanglement
18
Topological defects arise in neighborhood of symmetric images
Character Theory the disentanglement of finite discrete linear transformations
21
D
The Case of Multiple Transformations: Formal Derivations
23
D.1
Translations in both axes as direct product
23
D.2
Insights on the stacked shift operators model for the 2D translation group
24
D.3
Semi-direct product of two groups
25
D.4
Representations of the (discrete finite) Special Euclidean group
26
D.5
Insights the stacked shift operators model for the discrete finite Special Fu-
clidean group
31
D.6
Details of Section D.4
35
E
Additional Results
36
Quantifications
36
Additional analyses
36
A PREREQUISITES IN GROUP THEORY AND REPRESENTATION THEORY
Definition of a group A group is set G together with an operation o x G + G such that respects the following axioms:
Associativity: (gk gw) ) o gk" 9ы o (gk gke") with Okv Ik's 9k" G.
Identity element: there exists an element eg E G such that, for every Or E G,
9 - - P... and eg is unique.
Inverse element: for each 9a E G. there exists an element gk G, denoted g such that 9% 9k
9k eg-
In the paper, for clarity we do write explicitly the operation unless needed.
Finite cyclic groups We will be interested in finite groups, composed of finite number of elements (i.e. the order G).
A cyclic group is a special type group that is generated by single element, called the generator go. such that each element can be obtained by repeatedly applying the group operation to go or its inverse. Every element cyclic group can thus written as g" g Note that every eyclic group is abelian (i.e. its elements commute).
A group G that is both finite and cyclic has finite order K such that of eG.
14

---

Under review as a conference paper at ICLR 2021
Representation and equivariance Informally, for model to be equivariant to group of transformations means that if we encode a sample, and transform the code, we get the same result as encoding the transformed sample. (Higgins et al., 2018) show that disentanglement can be viewed as a special case of equivariance, where the transformation of the code is restricted to a subspace. We provide formal definition of equivariance below after introducing notations.
In the framework of group theory, we consider a linear representation of the group G acting on X (Scott & Serre 1996]: á G - GL(X). Each element of the group On E G is represented by a matrix ((gk) Qk, and Φx a matrix with specific properties:
1. é is homomorphism: ((g:)))) ) ((g))) gks EG.
2. th is invertible and 0/- ( as 60 " = (lantk d(ea) = T. where 7 is the identity matrix in (X).
The set of matrices ob form a linear representation of G and they multiply with the vectors in X as follows:
&x 0x:X(<>>)
s.t. Vo X, (x)
The character of di the function X such that for each 9h G. it returns the trace of d(gs) da. iee. Xo(ax) = Tr(dx). Importantly, the character of a representation completely characterizes the representation up linear isomorphism (i.e. change of basis) (Scott & Serre 1996). The character table of a representation is composed of the values of the character evaluated at each element of the group.
Similarly, we denote linear representation of the action of G onto the latent space Z by G - GL(Z) such that Vk, ((xx)) t/a. While corresponding to the action of the same group element 9b E G. the does not have to the same as de, as it represents the action of G on Z and not on X.
wh Z(= C*) s.t. V2 (n)))) Z.
Note that we consider complex latent space. With these notions, formally the model ʃ X + 7 is equivariant to the group of transformations that acts on the data if, for all elements of the group Is and its action db and thy on the spaces X and Z respectively, we have:
Vo E X, Vk K, (1b(()) = the (f(x))
Finally, a group action on image space R. is said to be affine if it affects the image through an affine change of cooordinates.
B EXPERIMENTAL DETAILS
B.1 DATASET GENERATION
Simple shapes We construct a dataset of randomly generated simple shapes, consisting ran- domly chosen points connected by straight lines with 28x28 pixels. We normalize pixel values to ensure they lie within [o, 1]. For all experiments, we use dataset with 2000 shapes. For each shape, apply 10 counterclock wise rotations by {0", 36", 72" ... 324"} using scikit-image's rotation functionality (see van der Walt et al. (2014)). For translations along the x-axis or y-axis we apply 10 translations using numpy.roll (see Van Der Walt al. 2011). This ensures periodic boundary conditions such that once a pixels is shifted beyond one edge of the frame, it reappears on the other edge. For experiments with supervision involving pairs, we construct every possible combination of pairs, 31, 22 and apply every transformation to both Ii and Ø2. For datasets containing multiple transformations, we first rotate, then translate along the x-axis, then translate along the y-axis. We use 50% train-test split then further split the training set into 20%-80% for validation and training.
MNIST (LeCun et al. 2010) Similar to simple shapes, we construct rotated and translated ver- sions of MNIST by applying the transformations to the original MNIST digits. We normalize
15

---

Under review as a conference paper at ICLR 2021
pixel values to lie between [0, 1]. For supervised experiments, similarly construct every com- bination of pairs ø1, 22 and apply transformations to both ø1 and z2- Since constructing every combination of transformed pairs would lead to many multiples the size of the original MNIST, randomly sample from the training set to match the original numher of samples. We use the full test set augmented with transformations for reporting losses.
B.2 MODEL ARCHITECTURES AND TRAINING
We implement all models using Py PyTorc with the Adam optimizer (Paszke et al. Ba
2019 Kingma &
2014).
Variational Autoencoder and Variants We implement existing state-of-the-ee disentanglement methods 3-VAE and CCI-VAE, which aim to learn factorized representations corresponding to fac- tors of variation in the data. In our case the factors of variation are the image content and the transformation used (rotation or translations). We use the loss from CCI-VA made up of recon- struction mean squared error and Beilback--eibblee divergence scaled by 8:
L=
Σ (r. 4
Cl
where is the number of samples and Kullback-Leibler divergence is estimated as KI(g(=)a)) p(2) 0.5 21 + In(o?) n² of where the latent dimension (see Kingma & Welling (2014)). For a standard VAE, we use C = 0 and 8 = 1.0. For 3-VAE, we sweep over choices of [ with C 0. For CCI-VAE, we sweep over choices of 33 and linearly increase C throughout training from C = to C 36.0.
We use the encoder/decoder architectures from Burgess et al. 2018 comprised of 4 convolutional layers, each with 28 channels, 4x4 kernels, and a stride of 2. This followed by 2 fully 256-unit connected layers. We apply ReLU activation after each layer. The latent distribution is generated from 30 units: 15 units for the mean and 15 units for the log-variance of Gaussian distribution. The decoder is comprised of the same transposed architecture as the encoder with a final sigmoid activation.
Autoencoder with latent operators For the standard autoencoder and autoencoders with the shift/däsentangled latent operators, we use supervised training with pairs (21,32) and transfor- mation parameter & corresponding to the transformation between I2 and Ø2. The loss is the sum of reconstruction losses for æ1 and Zz:
L (mi (D(f(a))))) +
(02, fo(v)2(((
(10)
where is the number of samples and &b the disentangled or shift latent operator. For a standard autoencoder, only the first reconstruction term is present in the loss function.
For the non-linear autoencoders, we use the same architecture above based on CCI-VAE. In the linear case, we use single fully 28x28 connected layer and an 800 dimensional latent space. We use 800 dimensions to approximate the number of pixels (28x28) and ensure that K=10 divides the latent dimension. This is approximation of the correct theoretical operator (which should be invertible) that works well in practice.
Weakly supervised shift operator model We use linear encoders and decoders with 784 di- mensional latent space to match the number of pixels. Indeed the weakly supervised shift operator uses the complex version of the shift operator, so Training is done with 1.2 loss on all possible reconstructions of æ1 (of each training pair i), weighted by scores a,,k Appendix B.3 below gives detailed explanation of the computation of the scores a
perfectly match the size of the image.
m
K
Σ
(11s - fo(f(22)))
0u (o(oss((((((((a))))))))))) /m
(11)
16

---

Under review as a conference paper at ICLR 2021
where is the number of samples. At test-time, we use the transformation with maximum score ag
Stacked shift operator model We use linear encoders and decoders with 784 dimensional latent space to match pixel size, as we use in the stacked model the complex version the shift operator. Intermediate layers L; are invertible linear layers of size 784 as well. Training done with L2 loss on reconstructed samples as in the autoencoder with shift latent operator (see Equation 10]
B.3 WEAKLY SUPERVISED SIIFT OPERATOR TRAINING PROCEDURE
Method We experimentally show in Section 4.2 that the proposed operator the works well in practice. Additionally, we developed a method for inferring the parameter of the trans- formation that needs to act on the sample. We encode samples by pairs 21, 22 (with xę a transformed version of x1) into and respectively, and use a classical phase correlation technique [Reddy & Chatterji 1996) to identify the shift between 21 and 22, described below. Then, use the complex diagonal shift operator parametrized by the inferred transformation parameter k.
Importantly, in the weakly supervised version of the shift operator, the model has extra free parameter which the number of latent transformations, not known a priori. Let us denote K2 this number, which can be different than the ground-truth order of the group K.
To infer the latent transformation that appear between 21 and 22, we compute the cross- power spectrum between the two codes and 22, that are both complex vectors of size N and obtain a complex vector size N. We repeat Kg. times this vector, obtaining a K,, N matrix, of which we compute the inverse Fourier transform. The resulting matrix should have rows that are approximately
except at the row corresponding to the shift between the two images, see Reddy & Chatterji (199)). Thus, compute the mean of the frequencies of the real part of the inverse Fourier result (i.c. the mean over the N values the second dimension). This gives us a Kg -i-dimensional vector, which
vector of scores of each l to be the correct shift between 21 and 22. During training, we compute the soft-max of these scores with a temperature parameter k, this gives Kr weights ag. We transform 21 with all K possible shift operators, decode into K' reconstructions Ih. and weight the mean square error between I2 and each vx by og before back propagatii.. This results, for each samples pair (I1, 22,1), in the loss:
L. Σ (D(
(12)
where m is the number of samples and ai the scores for the pair (2,,), At test-time, we use the transformation with maximum score aj
Dataset
K₁
10
21
Shapes (10,0,0)
0.001 0.0013
0.0005 + 0.0001
Shapes (0,10,0)
0.0097 0.0052
0.0038 = 0.0013
Shapes (0,0,10)
0.0115
= 0.0049
0.005 = 0.0033
MNIST (10,0,0)
0.0035 = 0.0039
0.0074 + 0.0083
Table 1: Comparing test mean square error (MSE) standard deviation of the mean over random seeds for different Kr. Numbers in () refer to the number of rotations, the number of translations on the r-axis, and the number of translations on the y-axis respectively.
Effect of the number of latent transformations In the weakly supervised shift operator model, the number of latent transformations K2, free parameter. Interestingly, when using rotations the best cross-validated number of transformations is 10, which matches the ground-truth order of the group. For translations (either on the x or the y axis), best results are obtained using K₃ = 21 which is larger than the ground-truth order of the group K. Table I compares test MSE for both values of KG.- We think that in the case of translations, changes in the image induced by each shift (each transformation) less visually striking than with rotations, and larger K₁ gives extra
17

---

Under review as a conference paper at ICLR 2021
flexibility to the model identify the group elements and respects the group structure (namely its cyclic aspect).
B.4
General hyper-parameters We sweep across several sets of syper-parameter for our experi- ments. We report results for the model with the lowest validation test loss. avoid over-fitting, results for any given model are also stored during training for the parameters yielding the lowest validation loss.
For experiments with simple shapes we sweep across combinations of
5 seeds: 0, 10, 20, 30, 40
4 batch sizes: 4,8,16,32
learning rates: 0.0005, 0.001
For MNIST, we sweep combinations of
5 seeds: 0, 10, 20, 30, 40
batch sizes: 8,16,32, 64
2 learning rates: 0.0005, 0.001
addition to these general parameters, we also sweep across choices of 3, {4, 10, 100, 1000} and latent dimension {10, 30} for the variational autoencoder models and variants. We repeat all exper- iments across four seeds used to initialize random number generation and weight initialization.
Weakly supervised shift operator hyper-parameters We perform a sweep over hyper- parameters as described above. Additionally for the weakly supervised model, we sweep temperature 7 of the soft-max that shapes the scores of each transformation over values т = (0.01, 0.1, 1.0}, and the number of transformations composing the operator family (i.e. the order of the group) over values 10 and 21, where 10 is the ground-truth order of the group.
Stacked shift operator model parametes We perform sweep over hyper hyper-paramedee parameters as described above. The only exception is that for the case of the Special Euclidean group, we train only for 5 epochs, and try batch sizes (32, 64) for MNIST and {16, 32) for simple shapes, as the number generated samples high. Similarly, for the case of 2D translations, we use batch sizes {16, 32} for simple shapes.
C REFRAMING DISENTANGLEMMENT. FORMAL PROOFS
We consider the action finite group on image space N. Using tools from topology, we show that it is impossible to learn representation which disentangles the action of this group with continuous encoder f.
C.1 TOPOLOGICAL PROOF AGAINST DISENTANGLEMERT
We consider finite group G of cardinal |G] that acts on R Given an image x E R.V an orbit containing that image is given by {g1x, ...,
We consider encoder ʃ R + M that disentangles the group action. The image of f is composed of an equivariant subspace and an invariant subspace. We define fF R - ME the projection of f its equivariant subspace and fi N + M, the projection of f its invariant subspace.
fg is equivariant to the group action. In equations:
Va,, Vg, f'(gg) g5
(13)
18

---

Under review as a conference paper at ICLR 2021
For the disentanglement to be complete, fE should not contain any information about the identity of the orbit the image belongs to. If O1 and O2 are two distinct orbits, we thus have:
Yan E O1, Vv2 E O2, Fg E G, G,Jn(z) (52(22)
(14)
fi is invariant the group action:
Yz, Vg. fr(ge) = f1(2)
(15)
We also assume that the representation contains all the information needed to reconstruct the input image:
Vz1 E O1, Vrz O2. fz(z)) # fr(
(16)
This last assumption corresponds to assuming that every image can be perfectly identified from its latent representation by decoder (i.e. perfect autoencoder). We also assume that both the encoder and decoder are continuous functions. This is a reasonable assumption as most deep network ar- chitectures differentiable and thus continuous. In the language of topology, the encoder f is homeomorphism, a continuous invertible function whose inverse is also continuous function. Also called a topological isomorphism, homeomorphism is a function that preserves all topological properties its input space (see Munkres (2014) p. p.1 Here we prove that f cannot preserve the topology of INN while disentangling the group action of G.
Consider fslo the restriction of fE to a single orbit of an image æ without any particular symmetry. fElo inherits continuity from f, and it can easily be shown that fElo is invertible (otherwise, infor- mation about the transformation on this orbit is irremediably lost, and f can thus not be invertible).
følo is thus also a homeomorphism, and preserves the topology of the orbit in image space, which is set of |G| disconnected points.
By equation 14 we know that restrictions of fE all other orbits have an image contained in the same topological space. As a consequence, the image of fE itself is a set of |G] disconnected points.
Since fE is a projection of f. the image of f should at least be composed of |G| disconnected parts (this follows from the fact that the projection of a connected space cannot be disconnected). However, this is impossible because the domain f R. which connected, and an home- omorphism, thus preserving the connectedness of ReN.
In summary, we have shown that, for topological reasons, continuous invertible encoder cannot possibly disentangle the action of finite group acting on image space R., In the next section, show that topological defects arise in the neighborhood of all images presenting a symmetry with respect to the transformation.
C.2 TOPOLOGICAL DEFECTS ARISE IN THE NEIGHBORHOOD OF SYMMETRIC IMAGES
We proved in the previous section that it is impossible to map a finite group acting on R to a disen- tangled representation with a continuous invertible encoder. In this section, order to gain intuition of why disentanglement is impossible, we show that topological defects appear in the neighborhood of images that present symmetry with respect to the group action.
C.2.1
fE IS NOT CONTINUOUS ABOUT SYMMETRIC IMAGES: FORMAL PROOF
Let's consider an image x, that presents symmetry with respect the group action:
Ag G.g # eG: gT. Fs
(17)
Let's further assume that an infinitesimal perturbation to this image along a direction u breaks the symmetry of the image:
4A < E, := cu, ga' # z'
(18)
Since fø preserves the information about the transformation.
[fm(gx') [x(x')
(19)
19

---

Under review as a conference paper at ICLR 2021
where C' is the smallest distance between two disconnected points of Mg. We assume that the group
action is continuous:
gr' go, + O(c) I, +
(20)
We can rewrite equation 19 as:
[fr(w O(t)) - /n))) + +(+)) > C
(21)
which contradiction with the continuity hypothesis on the encoder. We have thus shown that the equivariant part of the encoder fE presents some discontinuities around all images that present a symmetry. Note that for both rotations and translations, the uniform image is an example of symmetric image with respect to these transformations.
C.2.2 fi IS NOT DIFFERENTIABLE ABOUT SYMMETRIC IMAGES: VISUAL PROOF
As an example, we consider an equilateral triangle which is either perturbed at its top corner, left corner, or both corners (Fig. 4. When perturbed on either one of its corner, the perturbation moves the image to the same orbit, because the triangle perturbed on its right corner is a rotated version of the triangle perturbed on top corner. The gradient of Ja along these two directions at the equilat- eral triangle image should thus be the same (so as not to leak information about the transformation in the invariant subspace Zí). The simultancous perturbation along the two corners moves the image to different orbit, so the gradient of fi along this direction should not be aligned with the previous gradients (so as to preserve all information about identity of the orbit). And yet, if the function fi was differentiable everywhere, this gradient should be linear combination of the former gradients, and thus all three gradients should be collinear. The function fr can thus not be differentiable ev- erywhere. This imperative is incompatible with many deep learning frameworks, where the encoder is implemented by a neural network that is differentiable everywhere (with a notable exception for networks equipped with relu nonlincarities which are differentiable almost everywhere).
I
fI
211
H2
212
Figure 4: Visual proof that the invariant part of the encoder ff cannot be differentiable about sym- metric figures. We assume f, is differentiable and show a contradiction. We consider equilateral triangle which is perturbed at its top corner, left corner, or both corners. When perturbed either one of its corner, the perturbation brings the image to the same orbit, because of the symmetry. In latent space, the perturbation should thus move the latent representation in the same direction. The pertur- bation along the two corners simultaneously brings to image different orbit, and yet, since the perturbation is simple linear combination of the single-corner perturbations, it can only be colinear these perturbations. This collinearity leads the encoder not being injective, and thus loosing information about the identity of the image.
C.2.3 fI NOT DIFFERENTIABLE ABOUT SYMMETRIC IMAGES: FORMAL PROOF
Next, we show that, if we add an extra assumption on the encoder, namely that it is differentiable everywhere, it is also impossible achieve the invariant part of the encoder fi. Note that this extra assumption if true for networks equipped with differentiable non- non-linearities such as tanh or sigmoid, but not for networks equipped with relus.
20

---

Under review as a conference paper at ICLR 2021
Let's consider image 2, presenting symmetry w.r. the group action. We consider perturbations of that image along two distinct directions and gu.
By symmetry, it is easy that:
ffr du
ff
(22)
8gu
As consequence, a perturbation by w' (u+gu) direction:
equal to the perturbation along one or the other
2
0fr Ou'
0fi Ou
0fr 8gu
f
=
Du
(23)
In the general case, x' T2 eu' does not belong to the same orbit as 2, +eu. fI thus losing information about which orbit the perturbed image æ' belongs to, which is in contradiction with the assumption shown in Equation 16
C.3 CHARACTER THEORY OF THE DISENTANGLEMENT OF FINITE DISCRETE LINEAR TRANSFORMATIONS
C.3.1 THE SHIFT OPERATOR
Datasets are structured by group transformations that act on the data samples. Our goal is for the model reflect that structure. Specifically, want model be equivariant the transformations that act on the data. We defined group equivariance in Section
We show here that carefully chosen distributed operator in latent space, - the shift opera- tor-is is linearly isomorphic all cyclic linear transformations of finite order of images (i.e. the group G' of transformations is eyclic and finite and it acts linearly on image space). The practical consequences of this fact is that is possible learn an equivariant mapping to every affine transformation using this operator, using linear invertible encoder and decoder architectures.
Consider linear encoder model f W. If want W to be an equivariant invertible linear mapping W between X and Z, Equation rewrites as follows:
W (x) 4/x (W x) Vz X(= Yk
(24)
where dù and the are the representations of gı E G on the image and latent space respectively, as defined in A
For W be equivariant, Equation 24 must be true for every image æ. As W is invertible, Equation is true if and only if, Vk K, the two representations th and du are isomorphic:
vk K, dx W-1 the W
(25)
We consider additional properties on o corresponding to the assumptions (i) that G is eyclic of order K with generator go and (ii) is isomorphic to the regular representation of G (see Scott & Serre 1996)):
1. ò is cyclic, i.c. such that Φb ob where do 6(go) and thus of = I.
2. The character of ó is s.t. (s(e) = N and (gx) = 0 for + e.
The second property might seem counter-intuitive but it just means that the transformation leaves no pixel unchanged (i.e. permutes all the pixel). In the case of rotations, this is approximately true since only the rotation origin remains in place. The character table of Φ. given the second property, is
9o
g
90
K K-1
N
0
0
0
21

---

Under review as a conference paper at ICLR 2021
We have just seen that if the encoder and decoder are linear and invertible, the two representations and y must be isomorphic. Two representations are isomorphic and only if they have the same character (Scott & Serre 1996, Theorem 4, Corollary 2). We thus want to choose such that it preserves the character of the representation & corresponding to the action of G on the dataset of will see that our proposed operator needs to be distributed in the sense that it should act on the full latent code.
images. Importantly,
Let us consider the matrix of order K |G] that corresponds to a shift of elements in K- dimensional vector by positions.
the permutation corresponding to shift of exponentiated by k. We construct from Mr the shift operator as a representation of the group's action on the latent space. For each Øk G such that gk = go its corresponding shift operator the block diagonal matrix of order N composed of repetition of M
k
0
17
M
2/k =
M
(27)
M =
(26)
M
0
Let us compute the character table of this representation. First, for the identity, is trivial to see that X (e) N. Second, for any 9k * e, we have
Xalgx) Tr(bk)
(28)
since all diagonal elements of Dh will be O. Therefore, the character table of the shift operator is the same as X
K K-1
90
96
90
X
0
Using this shift operator ensures that an equivariant invertible linear mapping W exists between image space and the latent space equipped with the shift operator. Note that the character of the disentangled operator does not match the character table of Φ, and so we verify once again in this linear autoencoder setting that the disentangled operator is unfit to be equivariant to affine transformations of images.
When K does not divide N, we use latent code dimension that is slightly different than N but divisible by K. This is an approximation of the correct theoretical operator and we verify that works well in practice.
In the next Section C.3.2 show that also replace this shift operator by complex diagonal operator, which more computationally efficient to multiply with the latent.
C.3.2 COMPLEX DIAGONAL SHIFT OPERATOR
In order to optimise computational time, can also consider for Mj the following diagonal com- plex matrix:
k
0
M
(29)
K-1
22

---

Under review as a conference paper at ICLR 2021
with exx 2 The shift operator in this case is diagonal matrix, follows:
ME
w K-1
ME
Dk.N
(30)
k-1
K-1
Let us compute the character table this representation. First, for the identity, is trivial to see that X (e) N. Second, for any Ok 7 e, we have
w )m 1- (
(g+) = Tr(Un)
0
(31)
w
since we're assuming N can be divided by K. Again, the character table of the shift operator is the same Xá. Using this operator fastens computation since it requires only multiplying the diagonal values (a vector of size N) with the latent code (a vector of size N as well) instead of doing matrix multiplication. Note that when using this complex version of the shift operator, the encoding and decoding layers of the autoencoder should be complex as well.
as w = C 2
When K does not divide N, we still use latent code of size N and the operator th is diagonal matrix of order N, but the last cycle w.... unfinished (does not go until k-11) The character of this representation is no longer equal to 0 for non-identity elements 9s # e, but equals a small value << N and this approximation works well practice.
D THE CASE OF MULTIPLE TRANSFORMATIONS: FORMAL DERIVATIONS
D.1 TRANSLATIONS IN BOTH AXES AS A DIRECT PRODUCT
To cover the case of 2D translations (acting on both x and y axes of the image), we consider abelian group G that is the direct product of two subgroups G A₂ Ay- Both A, and Ay are normal in G because every subgroup of an abelian group is normal. Moreover, we consider that A₂ and Ay are both cyclic of order K and K' respectively, which the case for integer translation of an image using periodic boundary condition. We denote ax,0 and ay,0 the gen- erators of Ag and Ay. and write each translation
(y, with az, = = ay,k 40o-
We show that the shift operator plicity that both K and K" divide N, and KK' divides N. Following (Scott & Serre (1996). we write group elements g G as I(k,k') (am,k, ky The order of the group is K KK'. If consider the regular representation of 2D translations over the image space, as in Section C.3 its character table is
handle this case, with differences. We assume for sim-
9(0. 9(0.2)
9
X
N
We consider two representations over the latent space: the : A, + GL(CK) is linear represen- tation of A₂ and thy A, - GL(C*') a linear representation of Ay. each of the matrix form described in Section C.3.2 Group theory (Scott Serre 1996) tells us that sb is the tensor product of r/s and n i.e. for (ks, k') E G:
1/ = #(g) = (((az,k, dy.x)) = (the 8 (yy (g, = (ax,)) 6 the (ay,)))))
(32)
Let us consider the two shift operators:
diz = Diag 1, W1, w?,, W/ K- -11k
(33)
dy Diag( W2, w W2
K¹ -13h
(34)
23

---

Under review as a conference paper at ICLR 2021
where wi e a of order KK':
and W2
RT.
Then the tensor product 1/2,k @ Vy writes diagonal matrix
w
(1)
V B. ==
W2
(35)
w
wfwh
k(K -1 (1) uig
w1
The character of t @ Vo is
K'
K
XM 8
W2
nuk
x
41
nk
(36)
n=
n=
If h * 0, Σx- w 1-w4m T-48
0 since
w K
2iak
1 (and similarly for
we
if k' + 0). Hence, for (k, k) + (0,0). Σn-o
wa
wik'
KK- wit
0. For
(k, = (0, 0), (.,)),,,,, 0u KK' Thus, the character table of
the
& My is
9(0.1)
9(0.2)
9(-1IK'-1)
X
KK'
We will use for 2D translations a diagonal operator that is repetition of KK times wa.k @ yy (assuming KK' divides N), denoted v. with is for gk,k" (az,k: ay.
Vor E v.Kk
whl
w
(((K-1) (1 (IK--1)
W
2k. ==
1/ Vu.k
w2k
k(K 1) (((---1)
w1
(37)
Thus, the character table of
e
9(0.1)
)(0.2)
9(K 9(K-1x''11
X
N
We see that & has the same character table & the representation in image space, hence is suited operator to use for this case.
D.2 INSIGHTS ON THE STACKED SHIFT OPERATORS MODEL FOR THE TRANSLATION GROUP
For the case of 2D translations where G A, x Ay, we first use the operator corresponding to ag,ks the translation in v, then intersect linear layer denoted Ia before using the operator corresponding
24

---

Under review as a conference paper at ICLR 2021
to ay.k', the translation in y. When operate the latent code, perform
= y. ,N 22
(38)
where V.N.N is the operator representing the translation in æ. It matrix order N, where aa.. repeated times.
25
3,
w
wFt
k(K-1)
W1
NA.k.N
(39)
2
wf
wpth
k(K-1)
w1
Similarly, the operator corresponding ag is matrix of order N, where by is repeated times
N
K
wider
U
-1K
W2
Vy. ==
(40)
wpw
(K'
W2
We see that in order for the product tyy.", La Ko. when applied to z, to match the result of , applied to z, we need a permutation matrix P that operates on a matrix of order K K' made of blocks of K' times th & and returns matrix of order KK' with:
at the first K' rows and columns c = 1, (K + 1). KK' +1
wi at rows from K' + to 2K and columns = 2, K + 2.... KK' K + 2 etc.
until w ((K-1) at rows from KK' K' KK' and columns K, 2K, KK'
And the layer Lj would be matrix of order N made of this permutation matrix P. repeated in block diagonal form Kk' times, such that (Ny, N L v.,A.N) z Vk.K Z.
D.3 SEMI-DIRECT PRODUCT OF TWO GROUPS
D.3.1 DEFINITION SEMI-DIRECT PRODUCT
semi-direct product G H of two groups and H a group such that:
A is normal in G.
25

---

Under review as a conference paper at ICLR 2021
There is homomorphism f H. - Aut((A where Aut(AA is the group of automorphism A. For a A, denote f(h)a by h(a). In other words, f(h) represents how H acts on A.
The semi-direct product G A H defined to be the product A H with multiplication law
(01, (2), h2) (a1/4 (a2), h2h22
(41)
Note that this enforces that h1 (a1) = h
D.3.2 IRREDUCIBLE REPRESENTATION OF A SEMI-DIRECY PRODUCT
In this section, we also assume A is abelian. One can derive the irreducible representations of G in this case, as explained in SSott & Serre 1996) Section 8.2.
First, consider the irreducible characters of A, they are degree since is abelian and form the character group X = Hom(A, (A, C*). We use X to match (Soot & Serre (1996) notation, but note that X does not denote image space here, but the group of characters of A. acts on this group by:
(x = -
(42)
Second, consider a system of representative of the orbits of H in X. Each element of this system is Xi. G/H. For given Xa. denote H, the stabilizer of Xts i... hx. Xs. This means
1x, (a) = x; aa)) = xi(a), Va
(43)
Third, extend the representations of A representation of G, A H; by setting Xi(ah) = xi(a), h ai, A The X² are also characters of degree of G,
(44)
Fourth, consider the irreducible representations of H,, Scott & Serre (1996) propose to use the irreducible representation of H, Combining with the canonical projection G; - H; we get irreducible representations p of Gs. Irreducible representations G, are obtained by taking the tensor product Xı & p.
Finally, the irreducible representations of G are computed by taking the representation induced by Xi @ p.
Etingof et al. 2009) show that the character of the induced representation is
Xindg, (a. HH
1
Σ
x-((kaa)xxoh1-+111)
(45)
WeHssth-'WWee
D.4 REPRESENTATIONS OF THE (DISCRETE FINITE) SPECIAL EUCLIDEAN GROUP
In this section, we focus on the specific case the semi-direct product G = A x H where = Ag Ag is the group of 2D translations and H the group of rotations. Hence, A is abelian and H is a cyclic group. We will derive the irreducible representations of this group using the method presented in the previous section D.3.2
D.4.1 A NOTE ON THE DISCRETE FINITE SPECTAL EUCLIDEAN GROUP
While the Special Euclidean group respects the structure of semi-direct product in the continuous transformations case, we will consider its discrete and finite version. That is, we consider a finite number of translations and rotations. With integer valued translation, which is of interest when working with images (considering translations of finite number of pixels), we cannot consider all rotations. Indeed for example rotations of 2 break the normal aspect of the subgroup of transla- tions.
Proof. Take the translation element (1, 1) (one pixel translation in x and y respectively), and the rotation h of angle 2 Consider the composition hah -1 applied to point of coordinates i, this gives hah-1(h (i. h(h ((h-'(i,)) ) (1) (i.j) + A(1, (i.j) + (0, v2) and the translation (0, /2) is not integer translation. Thus, A is not normal in G in this case.
26

---

Under review as a conference paper at ICLR 2021
In what follows, consider rotations that preserve the normality of the group of integer 2D trans- lations of the image. Namely, these are rotations of 2m 4 म. 3at and identity and their multiples. Nonetheless, think the approach is insightful and approximate solutions could be found with this method. Furthermore, to ease derivations we consider that both K 2 2 and K' 2 2 are odd (and thus the product K K' is odd), such that stabilizers of the character group of A are either the entire H or only the identity. We leave the exploration of even K and K' for future work.
D.4.2 FINDING THE ORBITS
As we consider integer translations with periodic boundary conditions, characters of 2D com- plex numbers and are function of 2² elements of the 2D discrete and finite translation group of the form a = (z, y) with I = z and vo The characters of this group are evaluated as 12ml 21
kt 91
k')
X(a: .n (a) respectively.
K
K
where 2ı, 1/1 of the form with 21 E 1... K and y1 E E...KK K'
We consider two cases:
1. X0,0-
2. X(21.1) with 21 0 yı # 0.
Note that the total number of orbits of H in X is 1 Ex.ex |Hal where H2 is the stabilizer of Xa- Either Xx = ex is the identity element of X and Hgl H or X* # ex and the only stabilizer ен (as shown in Section D. thus [H = 1. Thus, we have that:
|4|
Σ m.
H = (1x H + (14) 1) x =)= 11
(46)
H
H
[
Hence, there are first case, X(o.o)).
[A/-1 [7]
orbits in case (the total number of orbits minus the orbit considered in the
D.4.3 ACTION OF ON THE CHARACTERS OF
Elements of H acts characters of A as:
1x (a)
X(x1.)) (h (h- ah)
(47)
X(arm)) (a)) (a))
(48)
22 + (ons( -0)a sii(-ssin(--)) 0)y) (son 0)))))
(49)
12n(( cos(0)+ # ai((-))=++ (-in( -i)+ # cos(-0))v)
(50)
12n(( cos(O) # sin( sim 2 com()))))
(51)
X(es cos(8)-v1 sin(0) +1) cos(8)) (a)
(52)
= ())(()))))))
(53)
where @ the angle of rotation of h.
D.4.4 CASE I Xo.0 (ORBIT THE ORIGIN)
A representative is X0.0 1. The stabilizer group H, is the entire H, and the irreducible representations of И are of the form '2* मे for
I.... [ where [H] is the total number of rotations. Thus we use the tensor product e, (0, Φ) = 1 @ 22a मि as representation of G. There are H such irreducible representations of G. all of degree 1, since the group of rotation is abelian.
The resulting representations, corresponding the irreducible get from combining this
27

---

Under review as a conference paper at ICLR 2021
orbit and each one of the irreducible of H can be represented in matrix form:
j
12x TM
p(a, =
(54)
0
m2m
where j such that hg and ho the generator of the group of rotations.
D.4.5 CASE 2 ((mi.))
(AI-1 [8]
OF THEM)
representative can be taken to be Xr X( (I. v), and æı, y1 are now fixed. Its stabilizer group H, is only {eH} (see (Berndt 2007) and proof in Section D.6.1
We now select P an irreducible representation of {eH}. and we select the trivial representa- tion I and combining with the canonical projection step 1 will also be representation of G,. We then take the tensor product Xr @ as a representation of G. = x {ew}- us now derive representation of the entire group A G Indg, (Xr @ 1).
Induced representation Indg, (xr @1 1) First, we need a set of representatives the left coset of G in G/G² = A x ey- The left cosets are defined as (a, h)Gx = {(a, () Cн) Va, E A). There |H] cosets (one per h), which coset Gk. We take as representative (eA,hu) = hs)e,,)(e,,en) E (ex, so the h; are now fixed.
denote (cA, GR.. We take representative 9i for each
The representation (p, W) is induced by (Xr 1, V)
W
P(g.)V
(55)
with Ø: = (eA, h:) and p(g:) is described below.
For each and each Io, there is j(i) E {), and ʃg E G5 such that we have g9: 95(8)
Indeed:
99: (a, h)(ee, hi)
(56)
(a, hhi)
(57)
(eA, (hah))))))))))))
(58)
f30 hh,). In other words, the action of (a, h) on element w W permutes the representatives:
M Σ plo) ((ta))))(((M)>>())))) 1=1
p(g)w= p((a
(59)
Now, we need to find for each element of G, the resulting permutation of the coset representatives. G is generated by E A, h H: any element (a, h) G can be written (a, en (,,,)) h). For (a, ew), we get 95(3) = (EA, h:) 9i- The induced representations are thus:
(
p(a,en) =
Xr (h (a)
(60)
Xr
(a))
An element (a, h) is in the same coset as (eA, h) (a, (cA, (a- h- (es,
(h (a e) Ge
28

---

Under review as a conference paper at ICLR 2021
with permutation. For h) get 95(1) (eA, hha). The resulting induced representations are:
0
0
Xr((hh; (r((hh)-2()))-4 (ea))
()) = 0
(ea))
0
(((hhy--((in)) ( (ex))
(61)
i.e.
0 0 r
p(eA.h) = Ph =
1
(62)
0 ... 1 0
Where the permutation matrix Ps. above represents how h acts on its own group. If hh, h then Ph. has a 1 at the column j of its i-th row. For example, hhim = h1 then j(H) = 1, (1) H| and so for the row 1, there is x- ((a)) = 1 at the |H column as above.
Let us denote for clarity Xr.i Xr (a)). (a)). The representations of (a, h) will be
Xr.1
0
0
0
Xr.1
pla, h) = pla,aaa)))eee en)
Xr.2
Ph
0
Xr.2
(63)
0
Xr. H
0
Xr.1a|
For each orbit, we get one of these (of degree H).
D.4.6 RESULTING REPRESENTATION
The resulting representation will be block diagonal matrix of size [A][H] of the form
H
H copies
Mr, (a)
h))
Mr, (a)
H copies
MR(a)
Ma
(64)
The first H elements on the diagonal correspond to the irreducíble representation of Agg this is the shift operator have been considering the single transformation case. Second, each matrix product M, (a, h)) h)PP. representative r, is repeated H times as it is of degree |H| (see calculation of degree in the Section D.6.2 and corresponds to:
(a) 0
M,(a)= =
xh (a))
(a))
(65)
0
0
Xx
29

---

Under review as a conference paper at ICLR 2021
If assume |||||| divides N, operator Pa,A.N that is the repetition of [ p(a, h) into a matrix of order N:
times
h)
2n T
Mn(a)P,
V.A. ==
p(a, 11)
(66)
12m p
PR
Its character table is:
(eA, en) N
(a),en)
ew)
(4/4), h())
X
0
0
and is the same character as the operator acting image space, as shown with the computation of the character of p(a, h) Section D.4.7
D.4.7 CHARACTER TABLE
For the identity element (н), get for p(a, h) the identity matrix, and its trace is |A|[H]). When h * EH, Ps. has no diagonal element and > 1 e 12 hnj 0. hence the trace of p(a. h) = When + CA, if h j en we are in the previous case. If h eH. PHr is the identity matrix and the representation will be
p(a,ex) (q. =
Mr, (a)
(67)
My,(a)I
MR(a)
The trace of p(a, eH) is
Tr(p(a,en) |||+++10 ||||
Σ
(h, (a))
(68)
k.
as each block-diagonal matrix is repeated ||| times, If we interchange the order of summation we get:
Tr(p(a,en)) = ||| + H Σ
Σ
xr(h
(69)
p=
heE
H +
H
Σ
Σ
hixe (a))
(70)
where hixx, represents the action of h; on Xr- The action of every h; on Xx gives all the elements in the orbit of Xr. so the double results in the sum over all characters X X of A, apart from the orbit of Xo,0 which contains only Xu,0-
Tr(p(a,eH)) = H + H
Σ
x(a)
(71)
XEX
(72)
30

---

Under review as a conference paper at ICLR 2021
The sum of the all the irreducible characters, for eλ is (Scott & Serre 1996
Corollary 2):
ExEx x(a) = 0. Hence,
= Σ Ex(a)= =0 =0 +
Σ
X(a)
(73)
XEX
xEXxXXNo
=1+ Σ
X(a)
(74)
XEX,XX Xo,0
Σ
X(a) -L.
(75)
x
Hence,
Tr(p(a,en) = |l| + [=](-1) =
(76)
Consequently, the character table of p the following:
(C..eH) |
(H)
(02,
(qpa.
Xo
The character of is
| times the character of Xp. i.c.
(eA,eH)
en)
(aI,EH)
(a2, e④)
(aaa, haa) 0
X
N
0
D.5 INSIGHTS ON THE STACKED SHIFT OPERATORS MODEL FOR THE DISCRETE FINITE SPECIAL EUCLIDEAN GROUP
Similarly to 2D translation case, we can use the theoretical form of the representation to use to gain insight on what the intersected layers L; should be for the case of 2D translations in conjunction with rotations. Elements of this group are (a, h) where a is the 2D translation, composed of ar the translation on the r-axis, 0y translation on the y-axis, and h the rotation. Using the stacked shift operators model, we first use the operator corresponding to the rotation, then the one for translation v, then the one for y-translation. The resulting transformed latent code
z' Yy NN L2 LIVE..N
(77)
with 0y =ay described in Equations 39 and
ay.o
0z,k
a=.0
h
hig The operators for translations Da,NNN Uy
40
We repeat them here for clarity.
w
11
W
Vaz.k.k.N
(78)
2.
w
wpk
A(K-1)
W
31

---

Under review as a conference paper at ICLR 2021
1/2
w
b.k.
wik
(1x--11
W2
(79)
(1)
where w1
2im and w2
And Uh is the repetition of the shift operator for the rotation
N times, follows,
vh.j
22x Tे
21 12
e"
TIT
m2 T
(2n (i
(80)
The representations for each transformation Vlrk,N, Vy,K thig.N are linked with the represen- tation that the theory gives us in Equation 66
First, recall that elements of H on character X+ X(aran)
X(m in (h (a)) = (t.(((z..))))(
(81)
Thus when h; spans H in a given matrix M,, we obtain H distinct characters built from i.e.
Xra (a))
M,(a)
0
Xba(r) (a)
0
(82)
0
(a))()((a)
For another matrix Myr, we obtain again H distinct orbits, distinct from the one present in M, (otherwise they would be in the same orbit). We have A-1 "H" representatives (excluding X(0.0)), thus |A| - 1 characters of A evaluated at a are obtained by considering all the matrices M,. The remaining character is X(u.0) 1. Hence, the diagonal matrix p(a, eω) of order |A||/[| contains all the characters of A evaluated at the element each one repeated | times since each matrix M, is repeated| [ times, and the first diagonal elements of p(a, eH) are 1.
So we see that what we need are | times each character of As explained in Section D.4.2 these characters are of the form:
1/2m F1
V1
I1
y1 k)
X(er.)) (a)
K
k+27
K1
k))
i(20
K
i(2n
k
k²) k)
wf w ynk
(83)
32

---

Under review as a conference paper at ICLR 2021
The characters of each translation group Xx1 wi for the z-translation and Xy wi" for the y u-translation, thus:
((x1.01(()) = X(x1.m)) (an,ks (y,k)) = wf w2 -xrr (ax, Xen (ay.kr
(84)
Since & is represented N times in M2KKNN and yy af is represented N K" times in Wy.kr,, we can K. take La reorder Vy, into blocks of K' repeated diagonal elements such that when multiplied with &.N by doing 4yy .N k. we obtain KK [ blocks of size |A], each containing
K'
N
every character of A once. Furthermore, we can see that the resulting matrix by .N L2422,R.N
composed |||| blocks of size || containing every character of A, but repeated | times.
This resulting matrix is then multiplied with the matrix I Linh..NN So let us now turn to the
representation of the rotation h, that is vh,..N- It is composed of the elements that appear in the upper-left diagonal of p(a, h). But we also need to make the permutation matrices Ph appear. Recall that diagonal matrix of order [H] that corresponds to the shift representation of h hg (not N repeated [ times) is
22n प
e
Wh.j
(2n 21
(85)
22x 2f --1
121
we right-multiply the by
1
125
18
1
C
(86)
22 (11-1)
12m (1M R1
181
[8
and left-multiply it by
1
1
12x
12
(87)
12
(18 -1)
12
12
(12) -1111111
[2]]
we obtain matrix that is filled with O except for each row one time the value H at the column e where e such that 1 hhn em, i.e. hhg = hy. this is exactly what Ps, represents, as we defined such that hh; = hy then Pa. has a at the column j of its i-th row. Thus,
1
Ph
B
(88)
[
Consider block diagonal matrix M where the first H elements are all ones (on the diagonal), and then the matrix B is repeated on the block diagonal [A] times.
[ times
M
|
(89)
B
times
B
33

---

Under review as a conference paper at ICLR 2021
Consider M' that is the repetition of |A| times Vh.55
Wh.j (of order {
12 p
e 12x -
M'
(90)
2n T
22m
TH
M' is of order [| Thus,
H elements
12m 1i
MM'C = H
1
I8I 1 e
[8
(A
(91)
Ph
1) times
Ph.
Ph.
which is of order [| Assuming the encoder takes care of the mist-luulipliiationn by C and the scaling by |M] (which can learn to do), we do not write the scaling and right-multiplicttion by C.
The operator Viai
N H
repetitions of tha.j. hence can be seen as N [
repetitions of
M'. Thus, let us denote Q a N by block diagonal matrix form 지 repetitions of M. When multiplying Q with Vn v can get MM repeated | times:
MM
22n T
Ph
QULiNN=
Ph
(92)
MM
i2m P
P
This is the matrix that is multiplied with Vy, L2 However, the order of the rows in the resulting does not correspond to the ordering the characters in Wy. N LEVeNNN If we want to match the operator in Equation 66 such that
Va (Vy.k,, L2 LiVh,,,)2
(93)
We need L1 be the product of two matrices P and Q, such that P reorders the rows of the vector Quh...,2 to match the ordering of the characters in Vy L2W, The result of
(vyy,' L2 PQU
(94)
will be a vector that is a permuted version the vector we would get with ela, and a linear decoder can learn to reorder it if want exactly match v..
34

---

Under review as a conference paper at ICLR 2021
D.6 DETAILS OF SECTION D.4
D.6.1 STABILIZERS OF ORBITS CASE 2
We show here that for Xr X(mr (z. y) with 21 # 0 or y1 # 0 the stabilizer group H; only {en).
Proof. Let us consider the stabilizer group of X(a..n) is composed of the elements h H X(m1-))))) other words, we must have that the character of the rotated vector h((T1. 91)) is the same as the character corresponding to (I1,31) for any Seeing characters 2D complex numbers, must have (æ1, v1) (cos Oz sin fy,, sin 8z1 + cos fy1). i.e. the rotated vector corresponding to the character is equal to itself. As we employ boundary conditions, this is for example the case for A = or 0 - for æ1 K', 2. 91 2, since the inverse of 2
such that for E A, ax) (a) ((((=1)an))())
K K' is itself. But we restrict ourselves and we consider only odd K and K", such that there is no angle such that the result of the rotation of (æ1, V/1) gives the same vector in complex space.
K
D.6.2 DEGREES OF THE REPRESENTATIONS
Case In Case we have H representations of degree 1.
Case 2 Recall that in Case 2, H4 {EH} and we use p 1, thus the character of the induced representation is
XIndg (x-01)
Σ
x-(h(a))xi(en)
(95)
WEHs.LN tMM=
hh' ey means h1 1111 If en H will span the entire H and we have
XIndg (x.8)) (a, eH)
Σ xx(()xx(eH) = [a] WEH
(96)
If h eи then there is no element h' for which this is true. Indeed, otherwise it means h h'h = en leading to contradiction. Thus:
Xindy (x. @1) (a, - (a) if = en
0 h + ee
To obtain the degree of the representation, we calculate the character of the identity element.
Xlndg, (1)
Σ
Xe(ea)x((en)
(97)
WEHsAN tegh'ee
Σ x-(eea)x(ee)= [#)x-(es)x((em) = [exx((e)) WeH
|H (98)
where we use the fact (x)) 1 (A is abelian hence its irreducible characters are of degree 1) and that / ehh' ен true for all h' E H. Hence, the degree of the induced representation is H H)xi(en) = H since (i(en) [ 1.
Thus in case 2 each orbit induces unique induced representation of degree |H]. and we have [A|-1 orbits in the second case. This shows we derived all the irreducible representations of [M]
G, as if we do the sum of the degree squared of irreducible representations we have
|A| -
H
H
H +
||| H H = ||
(99)
H
which correctly equals the order of G, see Scott & Serre (1996 Corollary 2).
35

---

Under review as a conference paper at ICLR 2021
E ADDITIONAL RESULTS
E. QUANTIFICATIONS
Test Mean Squared Error 2 reports test MSE for disentangled, supervised shift and weakly supervised shift operators.
Table 2: Test mean square error (MSE) + standard deviation of the mean over random seeds. Num- bers in () refer to the number of rotations, the number of translations on the z-axis, and the number of translations on the y-axis in this order. The case of multiple transformation does not apply to the weakly supervised shift operator, and we did not experiment on translated MNIST with the weakly shift operator as its performance on translated simple shapes and rotated MNIST were showing its relevance already.
Dataset
Model Shift
Disentangled 0.0208 + 5.2c- 6
Weak. sup. shift (K₃ = 10) 0.001 + 1.3c-3
Shapes (10,0,0)
0.0002 + 1.8e-66 6
Shapes (0,10,0)
0.0352 + 9.0e 6
0.0052 + 9.6e 6
0.0097 + 5 5-
Shapes (0,0,10)
0.0353 + 7.6e-6
0.0052 + 5
0.0115 + 4.9e-3 n/a
Shapes (0,5,5)
n/a
0.0047 + 1.9e-5
Shapes (4,5,5)
n/a
0.0049 = 7
n/a
Shapes (5,5,5)
n/a 0.0660 8.2e 5
0.0021 = 6
n/a 0.0035 + 3.9e-3 n/a n/a n/a
MNIST (10,0,0)
0.0004 + 5.66-6
MNIST (0,10,0)
0.0838 + 3.20- 5
0.0079 + 5.1e-5
MNIST (0,0,10)
0.0857 + 5.1e-5 n/a
0.0062 + 5
MNIST (4,5,5)
0.004 + 4.0e-5
LSBD Disentanglement Measure We compute the LSBD disentanglement metric to quantify how well the shift operator captures the factors of variation compared the disentangled operaaoo (see Anonyymus 2021)) 122D)..Nooo traditional disentanglement metrics are not appropriate as they describe how well factors of variation are restricted to subspaces in contrast to our proposed framework using distributed latent operators. LSBDD, on the other hand, measures how well latent operators capture each factor yariation to quantify disentanglement even for distributed operators. Using LSBD, we quantify the advantage of the shift operator with LSBD of 0.0020 versus the disentangled operator with LSBD of 0.0106 for the models in Fig. A and B
E.2 ADDITIONAL ANALYSES
Latent Variance and PCA Analysis The analysis in Fig. [C and DD quantitatively measures disentangiement in the latent representation for VAE and variants on rotated MNIST. We first apply every transformation vary the transformation. The final figure shows the average variance across all test samples. For the PCA analysis, we seek to determine whether the transformation acts on a subspace of the latent representations. We first compute the ranked eigenvalues of the latent representations of each shape with all transformations applied to the input. We then normalize the ranked eigenvalues by the sum of all eigenvalues to obtain the proportion of variance explained each latent dimension. Finally, we plot the average of the normalized ranked cigenvalues across all test samples.
given shape and compute the variance of the latent representation as we
Additional latent traversals Figures 5and all latent traversals for the model with the best validation loss. We see the success of disentanglement in the case of a single digit and the failure of a latent to capture rotation in the case of multiple digits. We show a more granular traversal with 50
plots per latent for each baseline in [n Figures 9 10 we show comparable results for the case of translations along the x-axis and y-axis.
shows additional results on MNIST. Fig. [4a]shows that the weakly supervised shift operator performs well on Rotated MNIST, and in Fig. 14b we see the stacked shift operator model is able correctly encode the multiple
Additional results for the distributed operator models Fig.
36

---

Under review as a conference paper at ICLR 2021
transformations case MNIST. Fig. 13 shows performance of the weakly supervised shift operator on translated simple shapes (either x or translations). The effect of the number latent trans- formations is explored in App. B.3 It shows the best model is in the case of translation obtained with 21 transformations. Nonetheless, to be able to feed to the model the correct 10 transformations needed in these plots, the reported plots for the model with 10 latent transformations. Fig. [T shows example results for the case of the Special Euclidean group (i.e. rotations in conjunction with translations) in setting where the semi-direct product structure is broken (see Appendix ID.4.I) Here, we used a rotation group of order 5. We see the stacked shift operator model performs well nonetheless.
Figure 5: Single Rotated MNIST Digit Label: Latent traversals for VAE (left), B-VAE (middle), CCI-VAE (right) trained on a single rotated MNIST digit (10 rotations). Latent traversal spans range [-6, 6] for each latent dimension.
37

---

Under review as a conference paper at ICLR 2021
4
2
4
4
4
4
4
4
4
4
4
4
5
5
2
4
5
9
9
Latent traversals for VAE (left), 8-VAE (middle), CCI-VAE (right) trained on all rotated MNIST digits (10 rotations). Latent traversal spans the range [-6, 6] for each latent dimension. Note in this case, the best validation model for VAE contained 30 latent dimensions whereas 3-VAE and CCI-VAE contain 10.
Figure 6: Rotated MNIST:
38

---

Under review as a conference paper at ICLR 2021
(a) Granular latent traversals for VAE trained on all rotated MNIST digits (10 rotations).
(b) Granular latent latent traversals for 3-VAE trained on all rotated MNIST digits rotations).
(c) Granular latent traversals for CCI-VAE trained all rotated MNIST digits (10 rotations).
Figure 7: Granular latent traversals for VAE VAE, CCI-VAE trained on all rotated MNIST digits (10 rotations), Latent traversal spans the range 6-6,) 6] with step size 0.25 yielding 50 plots per latent dimension. Note in this case, the best validation model for VAE contained 30 latent dimensions whereas 3- VAE and CCI-VAE contain
39

---

Under review as a conference paper at ICLR 2021
4-
7
3
DH
IST
TK
DN
(a) Supervised disentangled operator on Rotated
(h) Supervised disentangled operator Rotated
Shapes (10 rotations)
MNIST (10 rotations).
Figure 8: Non-linear disentangled operator with latent rotations
Figure 9: Single MNIST Digit Label Translated along X-Axis: Latent traversals for VAE (left), 3-VAE (middle), CCI-VAE (right) trained on a single MNIST digit translated along the -xis (10 translations). Latent traversal spans the range [-6, 6] for each latent dimension.
40

---

Under review as a conference paper at ICLR 2021
Figure 10: Single MNIST Digit Label Translated along Y-Axis: Latent traversals for VAE (left), 3-VAE (middle), CCI-VAE (right) trained on single MNIST digit translated along the y-axis (10 translations). Latent traversal spans the range [-6, 6] for each latent dimension.
41

---

Under review as a conference paper at ICLR 2021
3
3
3
3
3
5
5
5
3
3
7
3
7
7
9
333
3 3
3
S
S
5
3
3
3
3
3
3
3
3
3 3
8
3
3
3
3
3
3
3
3
3
2
5
3
Q
2
3
5
2
2
8
3
3
3
3
3
3
3
3
3
3
5
3
9
2
3
3
4
6
3
3
3
3
5
3
3
S
5
3
3
3
6
5 5
0
3
!
3
3
3
5
5
3
3
6
6
3
3
3
3
3
3 3
3
3
S
S
3
3
3
3
Figure 11: MNIST Translated along X-Axis: Latent traversals for VAE (left), 8-VAE (middle), CCI-VAE (right) trained on all MNIST digits translated along the x-axis (10 translations). Latent traversal spans the range [-6, for each latent dimension.
42

---

Under review as a conference paper at ICLR 2021
8
8
7
7
2
2
7
8
8
9
7
8
7
8
8
3
3
8
7
7
8
9
9
44
8
Figure 12: MNIST Translated along Y-Axis: Latent traversals for VAE (left), 8-VAE (middle), CCI-VAE (right) trained on all MNIST digits translated along the y-axis (10 translations). Latent traversal spans the range [-6, 6] for each latent dimension.
catchnul
criital
+
(a) Supervised disentangled operator on translated shapes along the x- x-axis (10 translations)
(b) Supervised disentangled operator on translated shapes along y-axis (10 translations).
Figure 13: Non-linear disentangled operator with latent translations
43

---

Under review as a conference paper at ICLR 2021
212
SUT7
19
X
TTV
L
INn
Q
I<e
nP
173
33
++
(b) Supervised shift operator on Rotated- Translated MNIST (4 rotations, x-translations and 5 y-translations).
(a) Weakly supervised shift operator Rotated MNIST (10 rotations).
Figure 14: MNIST additional experiments.
0
Q
9
ergza
15
23
CES SM
0
12
15
ZY
A
PK
P
22
15
27
V
7
(a)
Weakly
supervised shift operator on x-
(b) Weakly supervised shift operator on
y/-
translations (10 translations).
translations (10 translations).
Figure 15: Simple shapes additional experiments.
44
23.35
JUN
XL3
JUR
JUN'
1733
1KC
DR"
12
W
23
2-44
Figure 16: Supervised shift operator Rotated-TTaaslated product structure not respected as rotations angles are j = -I....5 (5 rotations, x- translations and y-translations).
Translated simple shapes when the semi-direct
44

---

Under review as a conference paper at ICLR 2021
Test Reconstruction
x2 from
x1
x2
transformed z1
A
A
7
Figure 17: Pairs of test samples and their reconstructions for the stacked shift model with 5 translations both x and y.
45

---

Under review as a conference paper at ICLR 2021
Test Reconstruction
x2 from transformed z1
x1
x2
1
7
Figure 18: Pairs of test samples and their reconstructions for the stacked shift model with rotations and translations in both and y,
46


Reviews:
[{"review_id": "zJP3HDipisl", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper first shows that existing approaches to latent space disentanglement perform poorly when the latent space topology (usually Euclidean) does not match the actual data topology, using rotation equivariance as an example. This analysis culminates in a general impossibility theorem for this type of disentanglement. The authors then propose a relaxed definition of disentanglement and show that it can be realized by means of a shift operator in latent space. Theoretical and empirical results demonstrate the superiority of the new approach. This is a very interesting idea that represents significant progress in an important problem. \n\nUnfortunately, the current organization of the paper does not work well: the authors devote too much space (half of the paper!) to the explanation of the problem, and too little (barely one page) to its solution. This leaves the reader with many unanswered questions about how the new method works and what its crucial details are. Some of these questions are later dealt with in the appendix, but this is too late.\n\nMy main suggestion for improvement is therefore to move most of section C 3.1 to the main text and allocate the required space by shortening the motivation (up to section 3.2) and possibly the discussion of multiple transformations in section 4.3. Content that would get lost by this change should be moved to the appendix.\n\nMore minor points are:\n* The authors repeatedly refer to \"recent success of ... distributed operators\", but do not cite and discuss any prior work. Please add appropriate references to the introduction or related work.\n* Rotations and translations are continuous transformations, whereas the proposed shift operator is discrete. Does this discretization introduce rounding errors or other artifacts? How many discretization levels are needed, and how can this number be determined? Does discretization have undesirable limitations? Such potential limitations should at least be acknowledged. Ideally, these questions should be investigated experimentally (but this can be left for future work if infeasible in the present paper). \n* In appendix E.1, results for rotations are an order of magnitude better than those for translations. Why is this the case?\n* Figure 3E: It is hard to judge if the results align with the ground truth. Preferably, the ground truth should be displayed for reference.\n* Is it necessary to design the network according to a-priory knowledge of the relevant group transformations, or can this be inferred automatically? For example, what happens if the latent space implements a group that does not correspond to any symmetry in the data?\n\nI'm willing to raise my rating if these points (in particular, the relocation of section C 3.1) are suitably addressed in an updated version of the submission.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VPMXquybssI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the notion of disentanglement in a group representation theoretic setting. Disentangling is sometimes conceptualized as mapping distinct factors (e.g. position / orientation) to distinct subspaces. It is shown theoretically that such a naive notion of disentangling is impossible for topological reasons, and this is confirmed empirically. An alternative definition of disentanglement is given, where instead of confining the effect of each transformation to a subspace, an operator is used that acts on the whole latent space (this operator is chosen as a shift operator, which works for cyclic groups). It is shown empirically that an autoencoder with a shift operator in latent space is better able to learn rotations and translations.\n\nThe paper does a good job explaining why the naive notion of disentangling leads to topological problems, and convincingly backs this up with experiments as well. The insight is not new to me personally, but I can't find a reference that explains it and I think it is not widely understood, so I consider this an important contribution to the (very muddled) discourse on disentangling.\n\nDefinition 1 provides a new definition of disentangling. However, the statement is not very precise, and I am not convinced that it can reasonably be considered as a definition of disentanglement. The definition is: \n\n\"A representation is said to be disentangled with respect to a particular decomposition of a symmetry group into subgroups, if there is a family of known operators acting on this representation, potentially distributed across the full latent, where each operator is equivariant to the action of a single subgroup.\"\n\nBased on the rest of the paper, I think this means that we have for each subgroup G_i an operator phi_i(g) acting on the latent space. The definition does not make it clear that we wish the encoder to be equivariant wrt this operator and some operator acting on the input space, but I will assume that is what is meant (otherwise, having an operator acting on the latent space is a rather vacuous requirement on the encoder/representation). The definition does speak of the operator being equivariant, which I will take to mean that it is a group representation, i.e. phi(gg') = phi(g)phi(g'). The operator being distributed I will take to mean that phi(g) can be any linear map, not necessarily acting trivially on a subspace or being (block-) diagonal / reduced. \n\nThe definition mentions that each subgroup should have its own operator, but since all of them act on the whole subspace this seems to a trivial constraint. Indeed if we have a representation of the whole group acting on the latent space, simply restricting it to each subgroup gives us a representation of the subgroups. I would further note that what is done in practice in the paper is different from this definition, because we have one latent space per operator, not multiple operators acting on the same space.\n\nUnder this interpretation, I don't see how the definition is saying anything else than that the network should be equivariant wrt some representation of the group acting on the input and output space. Although equivariance is a good property for various reasons, it does not seem to me to be reasonable definition of disentangling by itself. Indeed, the identity map satisfies this constraint trivially.\n\nIt may be that I have misunderstood definition 1, but this strengthens the case for making it mathematically precise.\n\nEven if one can question whether Def 1 is a good formalization of disentangling, the paper does show empirically that it is easier to learn an equivariant encoder/decoder when the latent operator is a shift operator or a diagonalized complex version of it, rather than a disentangled operator (with one 2x2 rotation matrix block and an identity block; fig 3b). Although I don't know if these two approaches have been compared before, several older papers consider similar models to the shift operator model. \n\nFor instance, in a sequence of papers Memisevic & Hinton considered factorized RBMs that do something similar. Cohen & Welling described a representation-theoretic version of this model which is very similar what is presented in this paper (at least the linear AE), and also gave a definition of disentangling (under this definition, the complex diagonal shift operator is disentangled while the original shift operator is not). Models with a stack of multiple operators were considered by Sohl-Dickstein et al.\n\nIf one wishes to define a notion of disentangling based on subgroups and representations, it may be worth investigating subgroup adapted / Gelfand-Tsetlin bases.\n\nIn summary, I think this paper contains several interesting observations and results, and I think the general direction is very interesting and deserves further study. However, I'm not convinced that this paper provides a good definition of disentangling, the experiments although convincing and well executed are restricted to simplified domains, and some of the insights / methods presented in the paper are already present in earlier work. Nevertheless I hope the authors will not be discouraged, and continue to work on this important and fundamental problem using the tools of representation theory.\n\nReferences\nMemisevic & Hinton, Learning to Represent Spatial Transformations\nwith Factored Higher-Order Boltzmann Machines, 2010\nSohl-Dickstein, Wang, Olshausen, An unsupervised algorithm for learning Lie group\ntransformations, 2010\nCohen & Welling, Learning the Irreducible Representations of Commutative Lie Groups, 2014\nWakin, Donoho, Choi, Baraniuk, The multiscale structure of non-differentiable image manifolds, 2005\n\n----\nPost-discussion update:\nHaving read the other reviews, author response and updated paper, I still think this paper is borderline. The insight that disentangling transformations as naively defined is impossible for topological reasons is valid and interesting, but seems to have been already observed by others, e.g. Falorsi et al. Nevertheless the paper does a good job explaining this so it could be useful, as some authors seem to not know about this issue. The definition of disentangling still seems a bit vague to me, and I'm not convinced of practical applicability of the proposed method. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "MpCSLT84Yk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors proposed a new way to disentangle affine transformations without topological defects. This paper made several theoretical contributions including a new definition of disentanglement and demonstration of the topological defects in existing disentanglement methods. Experimentally, this paper showed how their proposed shift operator model is powerful when dealing with topological defects.\n\nDisentanglement is a relatively challenging task due to the lack of clear definition and the lack of a robust evaluation method. The authors did a good job providing new theoretical definitions and providing empirical and qualitative results to support their claims. The main weakness of the paper is the lack of quantitative metrics to evaluate their approach and compare with others. In addition, the model doesn’t appear to be very flexible as it requires that the transformation is known in advance.\n\nStrengths:\n+ Overall, the paper is well written and contains a good review of advances in the theory of disentanglement.\n+ The idea of addressing topological defects for disentanglement appears novel. \n+ Using operators on the entire latent space is a new direction for the study of disentanglement. The authors’ viewpoint that “isolating factors of variation” is different from “mapping these factors into distinct subspaces”, and how they propose a new definition based on this viewpoint is interesting.\n\n\nWeaknesses:\n- Lack of quantitative evaluation metrics. The MSE in the appendix is not enough for quantifying disentanglement. \n- Since this paper focuses on disentanglement, at least Factor-VAE, one of the other representative disentanglement VAE models should be considered when doing the model evaluation.\n-  Baseline models should be optimized in a more comprehensive manner (e.g., currently the selection of beta is {4, 10, 100, 1000} and latent dimension is {10, 30}). It’s unclear whether these models have been well optimized, or what measures are used to optimize the models for this task. \n- Because the method requires that the transformation is known in advance, this limits the flexibility of the approach. \n- How different transformations impact each other is not shown experimentally - there is only an example on Fig 3E showing some visual results, but this should be elaborated on further given the goal of the paper.\n\nMinor points:\n- The complex version of the shift operator is used. It would be interesting to show another version and their differences. \n- Latent traversals results appear to be rather sparse. It would be interesting to show how the variation exists inside the model via dense traversals and the computing of generated images variation with different latent traversals.\n- Rotations may be more challenging to learn. 2000 examples may be insufficient for the model to learn this transformation correctly.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HN42rkn06HW", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents the idea that the current formulation of disentangled latent representations of data that have been presented are implausible in the sense that the factors are often not actually independent and cannot be learned or generated as independent. Instead the authors put forth the idea of transformations of data that are equivariant to the latent space representation as a formulation of disentangled factors. The authors use group theoretical constructs such as shift and rotation operators to show that a latent space representation should be equivariant such transformations. In other words, if a latent space representation is rotated, it should still reconstruct correctly, because the reconstruction loss should be trained on a rotated version of the image. \n\nThe key strengths of this paper are the examples that showcase the lack of ability to learn independent latent factors. Figure 1 displays the failure to learn rotation as a factor in the MNIST digit dataset. Figure 2 is even more convincing in that it shows that the orbits of the different factors cannot be mapped to one another and thus cannot be truly independent. \n\nSecond, I believe that the idea that is better stated in the introduction  on how disentanglement can be framed is valuable: “ In this framework, the factors of variation are different subgroups  acting on the dataset, and the goal is to learn representations where separated (of the data) subspaces are equivariant  to distinct subgroups.” Theoretically the authors are proposing an operational view of the latent factors as separate transformations on the data, and the representation as having subspaces equivariant to the transformations. Definition 1 is trying to state the same idea but is much less clear to the average ML reader\n\nMore generally, the authors should work harder to communicate this to the ML audience. The group theoretical background from the appendix should be in the background section, particularly the idea of equivariance and group operations. \n\nThe key weakness is that their new formulation of disentanglement is that it is definitional does not give a plan of how this should be done. Based on their description it seems as if the dataset has to come with a set of known operations on the data (like rotations) that are equivariant. How would such operations be learned de novo from the data? It seems as if the framework requires learning two things separately 1. A latent representation of the data 2. A set of equivariant operations on the data (that are perhaps cyclic generators of an orbit). It is not clear how this would be learned. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EUDs1-oanjh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summarize what the paper claims to contribute.**\nThe authors claims to show that disentanglement into subspaces by a continuous encoder is impossible for any finite group acting on Euclidean space\nThe authors claim to introduce an alternative definition of disentanglement that is more flexible and leads to a \n\n**Strengths:**\nThe authors consider the problem of disentangled representation learning which is of considerable interest to the community\nThe authors approach the problem by imposing structure through their disentangled operators\n\n**Weaknesses:**\nThe reliance of the “impossibility of disentanglement” proof seems to rely heavily on the example of the perturbed triangle. The example and its assumptions seem fairly rigid and unnatural and I am unconvinced this captures the reality of disentangled representation learning with auto-encoding networks.\nThe approach of adding structure by means of a transformation operator was also used in [1,2] which are cited but not compared against. Instead the authors compare against various VAEs which do not impose any external structure which does not seem particularly appropriate.\nIf I understand correctly, the paper seems to be based on a mischaracterization of the arguments in [3]\n\n**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.**\nReject. See weaknesses\n\n**Supporting arguments for your recommendation.**\nWhile the authors tackle an interesting problem and propose an interesting solution, the arguments on which the paper is based seem flawed.\n\n**Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.**\nAs I understand, the argument is against the utility of the *linear* disentangled representation in [3]. The more flexible definition the authors propose seems quite close to *disentangled representation* in [3], please clarify the difference.\nMoreover, it seems the authors suggest the definition of disentangled representations proposed in [3] requires that subspaces corresponding to factors of variation are single dimensional (section 2) which is not the case, please clarify.\nHow does the approach compare against other methods, namely [1,2,4] that use structure to encourage disentangling of the representation?\nSection 2 asserts that the VAE and its variants do not learn disentangled representations and uses PCA to show this is true. I expect that if this same analysis were used in the structured case, a similar result would be found, in particular, since the rotation matrix interacts with multiple dimensions of the latent code. Perhaps my intuition is incorrect, please clarify.\n\n**Provide additional feedback with the aim to improve the paper.**\nPerhaps a rewording could clarify: (Supervised Disentanglement) is composed of a 2x2 diagonal block... →  is a block diagonal matrix with a 2x2 rotation matrix in the upper left block and 1s on the remaining diagonals\n(just after 11) The authors state that most deep networks are differentiable, my understanding is that the common ReLU networks are not differentiable but subdifferentiable\n\n**Possible typos:**\n(VAE, beta-VAE and CCI-VAE) the ”4s” → the ``4s”\n(Dfn of a group; identity element) g_k e_G = e_G g_k = e_G → g_k e_G = e_G g_k = g_k\n\n**Post rebuttal**\nI thank the authors and other reviewers for their comments and discussion. While the direction the authors pursue is of unquestionable merit, I remain unconvinced that the work as it stands is sufficiently impactful for this venue. \n\n[1] Falorsi, Luca, et al. \"Explorations in homeomorphic variational auto-encoding.\" arXiv preprint arXiv:1807.04689 (2018).\n[2] Connor, Marissa, and Christopher Rozell. \"Representing Closed Transformation Paths in Encoded Network Latent Space.\" AAAI. 2020.\n[3] Higgins, Irina, et al. \"Towards a definition of disentangled representations.\" arXiv preprint arXiv:1812.02230 (2018).\n[4] Cohen, Taco, and Max Welling. \"Learning the irreducible representations of commutative lie groups.\" International Conference on Machine Learning. 2014.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline case (quite comparable to the other borderline case in my batch). The paper has received careful reviews and based on my weighting of the different arguments I arrive at an average score between 5.75 and 6.. The authors present some worthwhile ideas related to disentanglement that deserves more attention and that could spark more research in this direction. At the same time, the level of novelty and significance of this work remains a bit limited. Taken together the paper is likely not compelling enough to be among the top papers to be selected for publication at ICLR. 
Decision: Reject