Towards Building A Group-based Unsupervised Representation Disentanglement Framework
Authors: Tao Yang
Xuanchi Ren
Yuwang Wang
Wenjun Zeng
Nanning Zheng
Venue: ICLR 2022 Poster (2022)
Keywords: Disentangled representation learning
Group theory
VAE

Abstract: Disentangled representation learning is one of the major goals of deep learning, and is a key step for achieving explainable and generalizable models. The key idea of the state-of-the-art VAE-based unsupervised representation disentanglement methods is to minimize the total correlation of the joint distribution of the latent variables. However, it has been proved that their goal can not be achieved without introducing other inductive biases. The Group Theory based definition of representation disentanglement mathematically connects the data transformations to the representations using the formalism of group. In this paper, built on the group-based definition and inspired by the \emph{n-th dihedral group}, we first propose a theoretical framework towards achieving unsupervised representation disentanglement. We then propose a model based on existing VAE-based methods to tackle the unsupervised learning problem of the framework. In the theoretical framework, we prove three sufficient conditions on model, group structure, and data respectively in an effort to achieve, in an unsupervised way, disentangled representation per group-based definition. With these conditions, we offer an option, from the perspective of the group-based definition, for the inductive bias that existing VAE-based models lack. Experimentally, we train 1800 models covering the most prominent VAE-based methods on five datasets to verify the effectiveness of our theoretical framework. Compared to the original VAE-based methods, these Groupified VAEs consistently achieve better mean performance with smaller variances.

Published as a conference paper at ICLR 2022
TOWARDS BUILDING A GROUP-BASED UNSUPERVISED REPRESENTATION DISENTANGLEMENT FRAMEWORK
Yang Tao Xuanchi Ren". Yuwang Wang", Wenjun Zeng', Nanning Zheng'
Xi'an Jiaotong University, "HKUST, "Microsoft Research Asia, "EIT
ABSTRACT
Disentangled representation learning is one the major goals of deep learning, and is key step for achieving explainable and generalizable models. A well-defined theoretical guarantee still lacks for the VAE-based unsupervised methods, which are set of popular methods to achieve unsupervised disentanglement. The Group Theory based definition of representation disentanglement mathematically connects the data transformations to the representations using the formalism of group. In this paper, built on the group-based definition and inspired by the n-th dihedral group, first propose theoretical framework towards achieving unsupervised representation disentanglement. We then propose a model, based on existing VAE- based methods, to tackle the unsupervised learning problem of the framework. In the theoretical framework, we prove three sufficient conditions on model, group structure, and data respectively in an effort achieve, in an unsupervised way, disentangled representation per group-based definition. With the first two of the conditions satisfied and necessary condition derived for the third one, we offer additional constraints, from the perspective of the group- hasup definition, for the existing VAF-based models. Experimentally, we train 1800 models covering the most prominent VAE-based methods on five datasets to verify the effectiveness our theoretical framework. Compared to the original VAE-based methods, these Groupified VAEs consistently achieve better mean performance with smaller variances.
INTRODUCTION
Learning independent and semantic representations of which individual dimension has interpretable meaning, usually referred to as disentangled representations learning, is critical for artificial intelli- gence research (Bengio et al.
2013). Such disentangled representations are useful for many tasks:
domain adaptation (Li et al., 2019 Zou et al. (2020). zero-shot learning (Lake et al., (017)) and adversarial attacks (Alemi et al. 2016), etc. Intuitively, a disentangled representation should reflect the factors of variations behind the observed data of the world, and one latent unit is only sensitive to changes of an individual factor.
Due to the facts that obtaining the ground-truth labels requires significant human effort and humans can learn those factors unsupervisedly, unsupervised representation disentanglement draws much attention from researchers recently. A lot of methods are proposed base some intuitions. Most of the state-of-the-art methods (Higgins et al. 2017 Burgess et al. 2018 Kim Mnih 2018) 1018 Chen et al. 2018 Kumar et al. 2017) based on Variational Autoencoder (VAE) (Kingma & Welling 2013).
These methods are fully unsupervised and can be applied to a variety of complex datasets (Lee et al.
2020). However, these methods suffer from the unidentifiability problem (Locatello al. 2019b)
due to a lack of theoretical guarantee. Another stream of works (Chen et al. 2016 Lin et al. 2020
Khrulkov al. 202T Lee et al. (2020) leverage generative adversarial network (GAN) (Goodfellow
ct al. (0014) to achieve disentanglemen but are not interpretable. In general, a well-defined theoretical guarantee is needed for those methods.
The research of symmetry in physics demonstrates that infinitesimal transformations that conform to some symmetry groups on physical objects can reflect their nature (Anderson, 1972, Noether
Work done during internships at Microsoft Research Asia. Corresponding author

---

Published as a conference paper at ICLR 2022
1915).
Recently, inspired by this research on symmetry, Higgins al. (2018) proposed group-based defimition of disentangled representation. They argue that the symmetries, i.c., the transformations that change certain aspects of data and keep other aspects unchanged. ideally reflect the underlying data structure. The group-based definition is a formal and rigorous mathematical definition of faithful and, ideally, interpretable representation of the generative factors of data, which is widely accepted (Grefff itt allll| 2019] Mathicu et al. 2019; Khemakhem et al. 2020). Subsequently, due to the fact that the definition is defined by the world state (i.e., Ground Truth) and based on the assumption (Caselles 2019) that this definition should be useful for downstream tasks such as a Reinforcement Learning, Caselles-Duple et al. (2019). Quessard et al. (020)), Painter et al. (020) propose
Dupré et al.
environment-baset (to provide world state) methods to learn such disentangled representations in Reinforcement Learning settings. These inspire us to ask the following question: how would the definition benefit unsupervised representation disentanglement representation conforming to the definition in the setting of unsupervised representation learning?
and how to learn such a disentangled
In Group Theory] the n-th dihedral group (Judson 2020 is set of all permutations of polygons vertices, forming a permutation group under the operation of composition (Miller 1973). The generators in an n-th dihedral group. i.e., flip and rotation, can be regarded the disentangled factors and also transformations. In this paper, inspired by the n-th dihedral group, we answer the above questions and address the challenge by proposing theoretical framework to make the definition practically applicable for unsupervised representation disentanglement. We then propose a model to tackle the learning problem of the framework and verify its effectiveness. We theoretically prove in Section the three sufficient conditions towards achieving disentangled representation per group- based defiiiton, which are referred to model, group structure, and data constraint, respectively. With these conditions, we offer additional constraints from the perspective of the definition. The additional constraints encourage existing VAE-based models to satisfy the symmetry requirement that comes from the nature of factors. Finally, we provide learning model based on the existing VAE-based methods in an effort to fulfill the three conditions (with the model and group structure constraint and a necessary condition for the data constraint satisfied). As an intuitive understanding, we introduce the additional constraints to reorganize the latent space to restrict its symmetry in an unsupervised way. These additional constraints indeed narrow down the solution space of VAE- based models. Detailed discussion in Sec. 5.4 Our model consistently achieves statistically better performance in prominent metries (higher means and lower variances) than corresponding existing VAE-based models on five datasets, demonstrating that the group-based definition together with our proposed framework further encourages disentanglement.
Our main contributions are summarized as follows:
To our best knowledge, we are the first to provide a theoretical framework to make the formal group-basee mathematical definition of disentanglement practically applicable to unsupervised representation disentanglement.
Our theoretical framework provides additional constraints from the perspective of group- based definition for the existing VAE-based methods.
We propose a learning model of the framework by deriving and integrating additional loss into existing VAE-based models, in an effort to make the learned representation conform to the group-based definition without relying on the environment (as done in Caselles-Dupré et al. (2019): Quessard et al. )2000); Painter et al. (2020))
2 RELATED WORKS
Different definitions have been proposed for disentangled representation (Bengio et al. 2013 Higgins
et al. 2018 Suter et al. 2019). However, only the group based definition proposed by Higgins et al. (2018) focuses on the disentangled representation itself and is mathematically rigorous, which is well accepted (Caselles- Dupré et al. 2019 Quessard et al.) 2020 Painter et al., 2020 Diane Bouchacourt 2021). Nevertheless, HHggins et (2018) do not propose a specific learning method based on their definition. Before this rigorous definition was proposed, there had been some success in identifying generative factors in static datasets (without interaction with environment) e.g., 3-VAE (Higgins et al.
We assume some basic familiarity with the fundamentals of Group Theory and Group Representation Theory. Please refer Appendix A for some basic concepts.
2

---

Published as a conference paper at ICLR 2022
2017),
2018), and FactorVAE Kim & Mnih 2018). More recent works (Srivastava et al. 2020 Shao et al. 2020 Kim et al.] 2019 Lezama
Anneal-VAL (Burgess et al. 2018), ) 3-TCVAE Chen al.
2018 Rezende & Viola 2018 also do not consider the group-based definition. Therefore, how
group-based definition will facilitate these methods is still an open question. Besides, all these works suffer from the unidentifiability problem (Locatello et al. 2019b), which is challenging problem in this literature. From group-based definition, our framework points out that, the unidentifiability problem could be solved once the data constraint satisfied. However, in this work, we can only get a necessary condition for data constraint, and we still can not solve this challenging problem.
As pointed out in Quessard et al. (2020). not straightforward to reconcile the probabilistic infer- ence methods with the group- based definition framework. [Caeells- Dupré et al. (219), Quessard et al. (2020), Painter et al. (2020) leverage the interaction with the environment (assuming is available) as supervision instead of minimizing the total correlation the VAE-based methods do. Consequently, the effectiveness of these methods is limited the datasets with the environment available. Our framework learns a representation conforming to the group-based definition without relying on the environment. Pfau et al. (2020) propose a non-parametric method to unsupervisedly learn linear disentangled planes in data manifold under metric. However, pointed out by the thors, the method does not generalize to held-out data and performs poorly when trying to disentangle directly from pixels.
To summarize, the existing probabilistic inference methods lack theoretical support, while the application scope of existing methods based the group-based mathematical definition Higgins et al. (2018) is very limited. To the best of our knowledge, our work is the first to reconcile the probabilistic generative methods with the inherently deterministic group-based definition framework of Tlggins et al. (2018).
3. THE GROUP-BASED FRAMEWORK FOR UNSUPERVISED REPRESENTATION DISENTANGLEMENT
Our goal is to explore the benefit of the group-based definition for unsupervised representation disentanglement and learn such a disentangled representation. The background of the group-based definition is provided in Section [.1] Section presents the theoretical framework towards achieving unsupervised disentanglement, in which we derive three sufficient conditions on the model, group structure, and data, respectively. The conditions the model and group structure provide additional constraints for the existing VAE-based models.
3.1 GROUP-BASED DEFINITION
We briefly review the group-based definition of disentangled representation Higgins et al. (2018). Considering a group G acting on world state space W (can be understood as ground-truth) of data space O and representation space Z via group action "w and group action iz respectively. For mapping f = bo h. where b and h denote the data generative process and encoding, we state: the mapping is equivariant between the actions on W and Z if
9 fw)) f(g w), Vg E G, Vw W.
(1)
Definition Assume G can be decomposed as G Gi x G2 x... x Gm- The set disentangled with respect to G if: (i) group action of G Z exits. (it) the mapping f is equivariant between the actions on W and Z. (iti) There a decomposition Z Zi x Ze x x Zm such that each Z, is affected only by the corresponding G
challenging to apply the group-based definition to an unsupervised disentanglement setting in practice because the definition refers to the world state space W, the group action of G on W, and mapping which are typically inaccessible in practice. We tackle the challenge by re- fe-framing the definition in new framework in the following section.
3.2 PROPOSED THEORETICAL FRAMEWORK
Since when the representation is disentangled, one latent unit in the representation space is only sensitive to changes of an individual generative factor, we make the following assumptions: G
3

---

Published as a conference paper at ICLR 2022
is direct product of m cyclic groups (as suggested by Higgins et al. (2018) and for simplicity): G = (Z/n2)) = z//nZ x Z/n2 x ... x 2 n2, where n is the assumed total number of possible values for factor and is the total number of factors; we further assume Z is set with the same elements in G. Therefore, the group actions of G on Z can be set to be element-wise addition, i.e., g g 2, V22 Z. 9 E G. For the generator of dimension of G, gi = (0,...; ..... 0), Di only affects the i-th dimension of z by + 2. In addition, the action of each generator on w only affects single dimension of w.
As we can seen from Equation Jabove, the group action is defined on w. which is often not accessible. making it difficult to apply the definition practice. Therefore, for the unsupervised setting, we would like to use permutations on the data space O (which only provides data without labels) to substitute the group actions on W. Specifically, inspired by the n-th dihedral group (Dummit & Foote 1991), we construct permutation group Φ; serving the role of an "agent" of G. The actions of G on W can be performed by Pg E $ on O, which can be formulated as
[ig - w) = h(pp b()) = h19g o), Vw E W.
(2)
where o denotes the data (e.g,, image) corresponding to the world state w through the mapping function b. If the above equation holds, we state that the "agent" permutation group & exists. We first give the conditions for the existence of this "agent" permutation group D. then derive the additional condition to achieve such disentanglement. We accomplish these two objectives in Theorem [ with the proof provided in Appendix B Theorem [ states that a general permutation group & on O can serve as an agent group (agent group exists) if and only if both (i) and (ii) are satisfied. If the agent group exists, and its permutations (actions on O) can be defined by autoencoder-lik model as shown in the equation in (iii), then Z is disentangled with respect to G.
Theorem For the group G (Z/nZ)" permutation group &b on O, representation space Z, a World State space W, and mapping b and h, Equation @ holds if and only if (1) 1b is isomorphic to G, and (ii) For each generator of dimension of G, 9G, there exists a P; E &D = 1, y.... m, such that Pi b(w) = b(g - w. Vi E W, and 4p; is a generator of tb; Further, if Eqquatio P holds and (iii) (Pg b(w) = h (g f(w)) Vu W,, W,Pg E L, then Z is disentangled with respect G. where PP is the corresponding element in & of under the isomorphism.
In Theorem D (i) states that the relation between the elements (i.e., group structure) is preserved between & and G, and we denote it as the group structure constraint; (ii) actually indicates data constraint that all variations in the data can be generated by compositions of some basic permutation generators tii We denote it as the data constraint; (1ii) states that the permutations in the agent group T are defined by encoding, action, and decoding, which is referred to as the model constraint. Note that in Theorem only the data constraint refers to the world state w.
Here is sketch of the proof: data constraint is a special case of Equation 2 for generator, and group structure constraint is a relation-preservngg constraint on compositions of generators, and satisfying both constraints will thus result in that Equation holld for any general element in & and vice versa. Moreover, we derive Equation for disentanglement when combining the model constraint and Equation?? 2
The model constraint specifies the way to permute the data. When the data is permuted, its world state changes. Therefore, how the world states transit between each other is modeled by the model constraint applied on the data. The isomorphism between &b and G ensures that the world state space W and data space O have the same symmetry. In this way, the model applied on the data learns the transition of the world states. Note that we aim to bring this group-based definition, which requires ground truth by default, into the unsupervised setting. Now only the data constraint refers to the world states, and it seems almost impossible to derive a sufficient condition for it without the labels. We thus make a trade-off in which we use a necessary condition in the next section.
4 GROUPIFIED VAE: A LEARNING METHOD OF THE FRAMEWORK
Let's look closer into the three constraints, respectively. Firstly, we consider the model constraint, (Pg 0 h (g h(o)) Yo E 0.9, E ④ which suggests that the action of & on O can be implemented using an autoencoder-like network that performs encoding, action on its representation space, and decoding. Given an autoencoder-like network with encoder h and decoder d, since d
4

---

Published as a conference paper at ICLR 2022
Decode
d
P
Encoder
h
0 =
Decoder
2 + 99
(a)
Figure 1: Overview of the implementation (Groupified VAE). (a) Illustration of permutation group - {p E G) defined on a VAE-based model, where G (2/n2)". The generators iPi; Pj & are permutations on O. Specifically, when optimized, Pi and (i3 are horizontal and vertical movements. (p: defined as the solid orange arrows illustrate: encode an image o to representation 2, perform 17 to get z, add to z, and decode back to the image. This process can be regarded as an exchange of images in dataset (permutation), as the dashed orange arrow shows. These permutations form group 2 (b) The Isomorphism Loss, which guarantees that & is isomorphic to G, includes Abel Loss La constraining the commutativity, and Order Loss C. constraining the cyclicity.
is approximately the inversion of h, the model constraint can be formulated as
h- h(o)) e d(g h(o)), Yo 0,g
together with further implementation of & as described in Section 4.1 the model constraint can be fulfilled. Secondly, The data constraint requires that all variations in the data can be generated by compositions of some basic permutations generators. Previous VAE-based works (Higgins et al.) 2017 Burgess al., 2018 Kim Mnih 2018 Chen al.) 2018: Kumar al.) 2017) aim to generate the data with independent generative factors, which is in line with the data constraint. Intuitively, if the VAE-based model can generate the data from statistical independent basic latent units and each unit corresponds to the basic permutation generator, the data constraint may be satisfied. Based on the intuition above, we prove that the world state is independently sampled per dimension, the minimization of total correlation is necessary condition for the data constrain! (see Appendix E Therefore, we can leverage existing VAE-based models to fulfill the data constraint to some extent for the unsupervised setting. Lastly, satisfy the group structure constraint, we derive self superviddd Isomorphism Loss which can be incorporated into the VAF-based model as described in Section 4.2
4.1 IMPLEMENTATION OF GROUP +
The key is to implement the group actions of G on into the VAE-based models, we need to map the representation = to group that isomorphic to G (cyclic representation space). Therefore, we construct function 7) to achieve this mapping. Moreover, this mapping is required to be differentiable. in order for proppaaaatttio to be adopted for optimization. According to Group Theory. there is an isomorphism between G and the n-th root unity group: (xxp((2rz)//n]] E zz where n, m are the same as in G. Therefore, the representation 2 can mapped to a by the function n as 7(2) exp (2wiz)/m n) (see Figure (a)). However, 7 can not be mapped to directly as it has complex numbers, but we can use Buler's formula: exp ((zizz)nn) = sin((2 -ii cos( (2mz), to map to its real and imaginary part, i.e., vector sin (2T (n) and cos (2T=2/))) The two vectors are concatenated and fed to the decoder.
For case of implementation, the permutation group & can be approximately generated by compositions of generators, i.c., =< P1, P2...... pm >. Recall that the generator P; of group & is defined as P; o o= d(g; h(o)) = d(h(o) + g.), Vo E O, where g. is generator of dimension in G, as shown in Figure (a). For yPi, we implement gi h(()) by adding (without loss of generality) to the i-th dimension of h(o), then make it cyclic by function n, Similarly, for yo add the value of
5

---

Published as a conference paper at ICLR 2022
4.2 IMPLEMENTATION OF THE ISOMORPHISM
In this section, to satisfy the group structure constraint (isomorphism), we derive two equivalent constraints, which are then converted into an Isomorphism Loss Lt. Many groups are uniquely determined by the properties of the generators, c.g., group G =< a, bla² = 6 = e, ab = ba >. In addition, since the group & isomorphic to G, + is also expected to be commutative and eyelic. In light of this, we derive two constraints on generators that are equivalent to the isomorphism condition, as described in Theorem 2
Please refer to Appendix (rrendix(ofor.. the proof.
Theorem 2 The defined permutation group & =< PI..P2........ > is isomorphic to G (Z/nZ)" ifand only if: (i) for generators Pi; Pj E 1,15i, S m, we have Pifj = PPjii, and (ii) Vi: Φ, S m. we have "p" e, where e the identity element of group 4
The first constraint requires the group & to be an abelian group (Judson 2020). Therefore, we denote it as Abel constraint and the loss derived from as the Abel Loss L.- The second a constraint on the order of elements. We thus denote it as the Order constraint and the loss derived from it as the Order Loss C. See Appendix for a more detailed implementation.
Abel Loss. For the Abel constraint: Visi Pi E Φ S i, S we have pigyj = 93 We minimize ||şp (yej o) - (Py (ypt o)]l, Vo E O to meet the Abel constraint, as shown Figure (b). The Abel Loss is the sum of the losses of all combinations of two generators. Denote the set of combinations as = {(t,j) S j,j S m]. The Abel Loss is defined as follows
L.
Σ oEO (iJi)CC
Σ
lle (ws 0) ., (pp:.))
(4)
Order Loss. For the Order constraint: Vy; E D S m, we have yp"" e. where e the identity element in group & (identity mapping). Note that with n times composition of yi, it is difficult for the gradient to back-propagate. We thus use an approximation that uses times of composition instead. When the autoencoder can do the reconstruction well, this approximation holds, see appendix E for details. Similar to Abel Loss, we minimize ||y; (pp _ o) oll, Vo E O to satisfy the Order constraint. The whole process is illustrated in Figure (b). However, the equation is not symmetrical and leads to bias. Therefore, we use the following symmetrical form instead:
C. = Σ Σ
(llee: - (pp (pp-11.0 o) - oll + || (p. oll) -ool)
(5)
OEO DEiIsism
With the above two loss functions optimized, the isomorphism condition is satisfied, which can be illustrated by Theorem Please refer to Appendix D for the proof.
Theorem The following two conditions are equivalent: (1) Vy,, E 1,1 S m. have
Piij = lpsss and Vip E $ i m. we have s² (ii) the Abel Loss function (Equation and the Order Loss function (Fquation are optimized.
Since the Abel Loss and Order Loss are equally important for satisfying the isomorphism condition, we assign equal weight to them. Thus, the Isomorphism Loss L L. C.. With the implementation of group & the model constraint is satisfied. We optimize the Isomorphism Loss to satisfy the group structure constraint. To further satisfy the data constraint to some extent as described in Section 4 we leverage VAE-based models and optimize their original loss (that minimizes the total correlation). denoted as LVAE. AE. Therefore, the Total Loss is L = LV AE + "i Cr, where Y is the weight of Isomorphism Loss. We denote the above VAE-based implementation Groupified VAE.
EXPERIMENTS
We first verify the effectiveness of Groupified VAE quantitatively in learning disentangled representa- tions on several datasets and several VAE-based models. Then, we show its effectiveness qualitatively on two typical datasets. After that, we perform a case study on the dSprites dataset to analyze the effectiveness, and conduct ablation studies on the losses and hyperparameters. For the performance comparison of two downstream tasks (abstract reasoning Vaan Steenkiste et al. (2019) and fairness evaluation Locatello al. (2019a)), and more comprehensive results, please see Appendix

---

Published as a conference paper at ICLR 2022
5.1 DATASETS AND BASELINE METHODS
To evaluate our method, we consider several datasets: dSprites (Higgins et al. 2017). Shapes31) (Kim & Mnih 2018), Cars3D (Reed et al. 2015). and the variants of dSprites introduced by Locatello et al. (Locatello al. 2019b): Color-dSprites and Noisy-dSprites. Please refer to Appendix G for the details of the datasets.
We choose the following four baseline methods as representatives of the existing VAE-based models, which are denoted as Original VAEs. We verify the effectiveness of our implementation based on those methods. 3-VAE (Higgins et al. 2017) introduces a hyperparameter /3 in front of the KL. regularizer of the VAE loss. It constrains the VAE information capacity to learn the most efficient representation. AnnealVAE (Burgess et al. 2018) progressively increases the bottleneck capacity so that the encoder learns new factors of variation while retaining disentanglement in previously learned factors. FactorVAE (Burgess et al. 2018) and 3-TCVAE (Chen et al. 2018)) both penalize the total correlation (Watanabe 1960), but estimate it with adversarial training Nguyen et al. 2010
Sugiyama et al. 2012) and Monte-Car Carlo estimator respectively.
dprits
DCI Groupilied
BetaVAE
MIG Groupitsed
FactorVAE
Original
Original
Grospified
Original
Original
Groupineed
B-VAE
0.23 + 0.10
0.46 + 0.085
0.75 0.083
0.86 0.051
0.14 + 0.057
0.37 : 0.089
0.51 + 0.008
0.63 0.089
AnnealVAE
0.28 4 0.10
0.39 1: 0.056
0.84 :: 0.050
0.87 :: 0.0067
0.10
0.34 +: 0.061
0.70 1 0.094
0.68 :: 0.058
PactorVAR
0.30
0.41 0.074
0.040
0.89 0,020
0.27 + 0.052
0.061
0.74 0.068
0.75 0.075
3-TCVAE
0.065
0.36 0.11
0.96 0.026
0.038
0.17 0.067
0.24 0.083
0.68 0.088
Car33d
DCI Groupified
Besa
MIG
VAc
Original
Original
Groupified
Original
Groupified
Original
Groupifted
8.VAB
0.18 + 0.059
0.24 :: 0.041
0.99
1.c 3
1.0 0.0
0.071 = 0.033
0.11 # 0.002
0.81 = 0.006
0.93 = 0.034
VAE
0.22 0.046
0.25 0.046
0.99
le
0.99 1.5e
0.074 0.016
0.10 0.014
0.82 2: 0.062
0.87 0.028
VAC
0.21
0.054
0.040
0.99
1.0 0.0
0.90 0.039
0.034
B-TCAB
0.24 1 0.049
0.20 0.018
1.0
1 0.0
1.0 0.0
0.30 - 0.021
0.11 0.033
0.88 1 0.040
0.93 0.034
Shapesäd
DCI
BetaVAE
MIG Groupified
Vac
Original
Groupified
Original
Groupified
Original
Original
Groupinee
3-VAE
0.44 2 0.116
0.56 :: 0,10
0.91 = 0.072
0.90 : 0.045
0.28 4: 0.18
0.42 0.15
0.82 0.008
0.82 0.043
AnsealVAR
0.52+ 0.051
0.60 9.078
0.82 0.076
0.89 0.086
0.48 0.047
0.50 0.062
0.75 0.074
0.83 0.066
PactorVAR
0.47+ 0.10
0.49 0.065
0.86 + 0.055
0.80 0.075
0.33
+ 0.13
0.43 + 0.11
0.81 0.056
0.79 0.006
3-TCVAE
0.66 0.10
0.72 0.061
0.97 +: 0.039
0:96 0.042
0.40 0:18
0.47 + 0,090
0.80 + 0.064
0.90 0.046
Table 1: Performance (mean + std) on different datasets and different models with different metrics. We evaluate 3-VAE, VAR, Factor and 3-TCVAE on dSprites, Cars3d, Shapes3d, Noisy- dSprites, and Color-dSprites for 1800 settings. These settings include different random seeds and
hyperparameters, refer Appendix G for the details. We only show the first three datasets here, For more results, please refer to Appendix
5.2 QUANTITATIVE EVALUATIONS
This section performs quantitative evaluations on the datasets and models introduced with different random seeds and different hyperparameters. Then, we evaluate the performance of the Original and Groupified VAES in terms of several popular metrics: Beta VAE score (Higgins et al. 2017). DCI disentanglement Eastwood & Williams (2018) (DCI in short), MIG (Chen et al. 2018)) and
Factor score (Kim & Mnih 2018). We assign three or four hyperparameter settings for each model on each dataset. We run it with ten random seeds for each hyperparameter setting to minimize
the influence of random seeds. Therefore, we totally run ((3 x 10 x 3 + 10 x 3 3) x 2) x 5 = 1800 models. We evaluate each metric's mean and variance for each model on each dataset to demonstrate the efffcctiveness of our method. As shown in Table I these Groupified VAEs have better performance (numbers marked bold in Table I than the Original VAES on almost all the cases.
On Shapes3d, the Groupified VAEs outperform the Original ones on all the metrics except for BetaVAF scores, suggesting some disagreement between Beta VAE scores and other metrics. Similar disagreement is also observed between the variances of MIG and other metrics Cars3d. Note that the qualitative evaluation in Empendix[sshoww that the disentanglement ability of Groupified VAEs is better on Shapes3d and Cars3d.
7

---

Published as a conference paper at ICLR 2022
Groupiñied Original
(a) Original
(b) Groupified
Visual traversal comparison between Original and Groupified 3-TCVAE. The traversal results of Groupified VAEs are less entangled.
Figure 2:
Figure 3: Traversal results of two factors (floor color, scale) of Original and Groupified ,- TCVAE. The traversal results of Groupified VAEs are cyclic.
5.3 QUALITATIVE EVALUATIONS
We qualitatively show the Groupified VAEs achieve better disentanglement than the Original ones. As shown in Figure 2 the traversal results of Groupified 3-TCVAE on Shape3d and Car3d are less entangled. For more qualitative evaluation, please refer to Appendix J To verify that the Groupified VAES learn cyclic representation space (where n = 10), we provide the traversal results [0, 18] with step of 2 for both the Groupified and Original B-TCAE that the traversal results Groupified VAEs are of high quality with period of 10 (equal to n). However, the Original VAEs generate low-quality images without cyclicity. For the comparison of the results on CelebA (real-world datasets), please see appendix
Shape3d in Figure 3 We observe
5.4 VISUALIZATION OF THE LEARNED REPRESENTATION SPACE
To understand how our theoretical framework helps the existing VAE-based models to improve the disentanglement ability, we take dSprites as an example, visualize the learned representation space, and show the typical score distributions of the metrics. First, we visualize the space spanned by the three most dominant factors (x position, position, and scale).
As shown in Figure for more results, please refers to Appendix [.. the spaces learned by the Original VAEs collapse, while the spaces of the Groupified VAEs only bend little bit. The main reason that the Isomorphism Loss, serving as self-supervision signal, suppresses the representation space distortion and encourages the disentanglement of the learned factors. As Figure 4 shows, the Groupified VAEs consistently achieve better mean performance with smaller variances. The isomorphism reduces the search space of the network that the Groupified VAEs converge the ideal disentanglement solution.
83
Originll
Goupfidd
Drwefeer
Dixppter
(a) VAt score
(b) DCI disentanglement
(c) MIG
(d) FactorVAE score
Figure 4: Performance distribution of Original and Groupified AnnealVAE on dSprites (demonstrated by the Violin Plot (Hintze Nelson 1998)). ). Variance is due to different hyperparameters and random seeds. We observe that Groupified Anneal VAE improves the average performance with smaller variance in terms of Beta VAE score (a), DCI disentanglement (b), and MIG (c), and has a comparable mean performance with smaller variance in terms of Factor VAE score (d).
Groupified
Factor Size 10
Original
10
15
w/o Abel
w/o Order
Groupified
DCI 0.27 0.34 0.062
0.38
0.055
0.38
0.064
0.28 0.11
0.34 = 0.056
0.38 0.055
Table Ablation study on the factor size n and Isomorphism Loss. DCI disentanglement listed (mean + std).
8

---

Published as a conference paper at ICLR 2022
Latent space visualization
Labent space visuaization
Space
Lotent space visusltatioo
-15
Boition
>> mossom
(a) C = 10, Groupified
C 20, Groupified
(c) C = 25, Groupified
(d) C 30, Groupified
Latent space sussiizttion
Labent space
Latent
Latent visualizat
(e) C 10, Original
(l) C = 20, Original
(g) C 25, Original
(h) C 30, Original
Figure 5: The representation space spanned by the learned factors by Original (bottom row) and Groupified Anneal (top row). The position of each point is the disentangled representation of the corresponding image. An ideal result is all the points form cube and color variation is continuous. The increase of C' (a hyperparameter of Anneal VAE) results in a collapse of representation space of the Original VAE. The collapse is suppressed by the Isomorphism Loss, which leads to better disentanglement.
5.5 ABLATION STUDY
We perform an ablation study on the assumed total number of possible values for a factor (factor size) n, Abel Loss L. and Order Loss C. We take the AnnealVAE trained on dSprites as an example. We only consider the DCI disentanglement metric here. We investigate the influence of factor size n. Besides, to evaluate the effectiveness of the two constraints, the models with the Abel Loss alone or Order Loss alone added are also evaluated. In this setting, we fix to 10. We compute the mean and variance of the performance for 30 settings of hyperparameters and random seeds. Table Tablee]shon shows that the isomorphism plays a role of cycle consistency in the representation space, leading to better disentanglement. The performance is robust to the factor size n, as the models learn adapt to different in the training process. The models with only the Abel Loss Order Loss applied have improved performance compared to the originals. The former (Abel Loss) performs better than the latter, suggesting that commutativity plays more important role. Note that the number of factors m can be learned and is not a hyperparameter. See Appendix F for details. Y empirically set to 1.
6 CONCLUSION
In this paper, we have opened the possibility of applying group-based definition to unsupervised disentanglement by proposing theoretical framework. The group structure and model constraint in the framework are effective for existing VAE-based unsupervised disentanglement methods. In addition, by establishing the feasibility of learning the representation conforming to the definition in unsupervised settings, we have exhibited the consistently better mean performance with lower variance attributed to the definition. We believe our work constitutes promising step towards unsupervised disentanglement with theoretical guarantee. As to the limitation, we only provide a necessary condition for the data constraint, as a result, we can not address the unidentifiability problem. Tackling the unidentifiability problem with the group-based definition is beyond the scope of this work, we will leave as future work. In addition, a natural extension of our framework is to use lie group Hall (2015) (which is also manifold) extend our framework.

---

Published as a conference paper at ICLR 2022
REFERENCES
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
Philip Anderson. More is different. Science, 177(4047::393336, 1972.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. TPAMI, 35(8):1798-1828, 2013.
Christopher P Burgess, Irina Higgins, Arka Pal. Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerehner. Understanding disentangling in beta -vae. arXiv preprint arXivv1804.03599 2018.
Hugo Caselles-Dupre, Dupréé Michael Garcia Ortiz, and David Filliat. Symmetry-based disentangled representation learning requires interaction with environments. In NeurlPS, pp.
2019.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In NeurIPS, pp. 2610-2622 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ily Sutskever, and Picter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In NeurPIS, 2016.
Stephane Deny Diane Bouchacourt, Mark Ibrahim. Addressing the topological defects of disentan- glement, 2021. URL htttpp://opeeeeee.eeeeeeeeeeeeee http://ffooouummooooooooooorrrr
David S Dummit and Richard M Foote. Abstract algebra, volume 1999, Prentice Hall Englewood Cliffs, NJ, 1991.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In ICLR, 2018.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, 0bss/106.226// 2014. ptctt:://axxi..orr//bbbbe
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv:1903.000 2019.
Brian Hall. Lie groups, Lie algebras, and representations: an elementary introduction, volume 222. Springer, 2015.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations, arXiv preprint 1rX10:181 2018.
Jerry L Hintze and Ray D Nelson. Violin plots: a box plot-density trace synergism. The American Statistician, 52())181-184, 1998.
Thomas W Judson. Abstract algebra: theory and applications. Virginia Commonwealth University Mathematics, 2020.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In AISTATS, pp. 2207-2217, 2020.
Valentin Khrulkov, Leyla Mirvakhabova, Ivan Oseledets, and Artem Babenko. On disentangled representations extracted from pretrained gans, 2021. URL bttps://openreve// net/ FForum?idaai dEVCXX333CCAX
10

---

Published as a conference paper at ICLR 2022
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In ICML, 2018.
Minyoung Kim, Yuting Wang, Pritish Sahu, and Vladimir Pavlovic, Bayes-factor-vac: Hierarchical bayesian deep auto-encoder models for factor disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2979-2987, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan, Variational inference disentangled latent concepts from unlabeled observations. arXiv preprint arXiv: 171100044 2017.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Wonkwang Lee, Donggyun Kim, Seunghoon Hong, and Honglak Lee. High-fidelity synthesis with disentangled representation. In ECCV, 2020.
José Lezama. Overcoming the disentanglement vs reconstruction trade-off via jacobian supervision. In International Conference on Learning Representations, 2018.
Yu-Jhe Li, Ci-Siang Lin, Yan-Bo Lin, and Yu-Chiang Frank Wang. Cross-dataset person re- identification via unsupervised pose disentanglement and adaptation. In ICCV, 2019.
Zinan Lin, Kiran Thekumparampil. Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In ICML, 2020.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. In NeurIPS, pp. 14611-14624, 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In ICML, pp. 4114 4124, 2019b.
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In /CML, pp. 4402-4415 2019.
Willard Miller. Symmetry groups and their applications. Academic Press, 1973.
Alexander McFarlane Mood. Introduction to the Theory of Statistics. McGraw-hill, 1950.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11)55447-5811, 2010.
Emmy Noether. Der endlichkeitssatz der invarianten endlicher gruppen. Mathematische Annalen, 77 (1):89-92, 1915.
Matthew Painter, Adam Prugel-Bennett, and Jonathon Hare. Linear disentangled representations and unsupervised action estimation. NewrIPS, 33, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Veto,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NewrIPS 2017 workshop, 2017.
David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière. Disentangling by subspace diffusion. NeurlPS, 33, 2020.
Robin Quessard, Thomas D Barrett, and William R Clements. Learning group structure and disentan- gled representations of dynamical environments. arXiv preprint arXiv:2002.0699/ 2020.
Scott E. Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. In NeurIPS, 2015.
11

---

Published as a conference paper at ICLR 2022
Danilo Jimenez Rezende and Fabio Viola. Taming arXiv preprint arXhv:1810.00597, 2018.
Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. Controlvae: Controllable variational autoencoder. In International Confer- ence on Machine Learning, pp. 8655-8664, PMLR, 2020.
Akash Srivastava, Yamini Bansal, Yukun Ding, Cole Hurwitz, Kai Xu, Bernhard Egger, prasanna Sattigeri, Josh Tenenbaum, David D Cox, and Dan Gutfreund. Improving the reconstruction of disentangled representation learners via multi-stage modelling. arXiv preprint arXiv:2010.13187, 2020.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the bregman divergence: a unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics, (4(5):009--1044 2012.
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer, Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In ICML, pp. 6056-6065. PMLR, 2019.
Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are dis- entangled representations helpful for abstract visual reasoning? In NeurlPS, pp. 14245-14258, 2019.
Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4(1 ):66-82, 1960.
Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint wrXiv:1901.000/1 2019.
Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative lie group vae for disentanglement learning. arXiv preprint arXiv:2106.03377. 2021.
Yang Zou, Xiaodong Yang, Zhiding Yu, BVK Kumar, and Jan Kautz. Joint disentangling and adaptation for cross-domain person re-identification. In ECCV, 2020.
12

---

Published as a conference paper at ICLR 2022
A
PRELIMINARIES
Group:
A set G together with a binary operation as: o G G - G satisfying the following properties:
Associativity: Va, b, (:s.((oo))
Identity: 3e E G, s.L. Goa
Inverse: Va G, G:0oa =a oa=e.
It is customary to represent group with a set G and the binary operation as pair (G, o). When the binary operation clear, we represent group (G, G and use multiplication to represent the binary operation o, i.e., a o ab, Va, b E G.
Group Action:
Let (G, o) be group and be a set. By the group actions of (G, o) on P, we mean mapping:
P:GXP=P,
(6)
such that
ba G. p-a. -
p. p, where is the identity element of G.
Symmetry Group and Permutation Group:
Let be a nonempty set, the bijections from to itself called Permutations. S(E) denotes the set containing all the permutations Σ. S(E) forms group under the binary operation: composition of functions, which is called Symmetry Group. A
subgroup of (E)) is called Permutation Group.
Abelian Group: the commutative law (Va, a b E G bo a) holds a group (G, o), such a group is called abelian group.
Subgroup: a subset H group is itself group under the operation of G, we say that H is subgroup of G.
3
3
(1432) 0
(13)(24) pp
(1234) 14
4
4
2
4
(23)
(23) = 4.0
(24)
f
(12)(34) fly
(13)
f
Figure 6: 4-th Dihedral Group (D,): The groups of symmetries for square. Note that permutation (123) means 1 2.2 > 3,3
Equivalence Relation; An equivalence relation on set B is subset U C BBx B satisfying: (It is customary to represent (a, b) E U as b)
Reflexive: Va E B, (0, EU.
Symmetric: (a, U = (b.a)EU.
Transitive: (a, U and (b,c) E = (a, EU.
13

---

Published as a conference paper at ICLR 2022
Equivalence Class: Let ~ be equivalence relation set B. We take E B, and then the equivalence class containing is the subset _ {b Clb aa B.
Homomorphism Let (G..) and (H, o) be two groups. A homomorphism f. from G to H, is a mapping G + H, such that f(x y) f(x f(v), Vz, G.
Isomorphism: A homomorphism f G + which is bijective is called an isomorphism. Two groups are said to be isomorphic if there exists isomorphism between them.
Related Groups:
(i) Additive group of integers modulo n: Let denote positive integer. We define equivalence relation on Z as a ~ b + = (mod n) (a and b have the same remainder modulo n)) The relation
1, where represents the equivalent class containing i, i..., i {m Z/m i (mod n)}. Let Z,, {0, I, ................................ I}, and Z,, forms group under the b, denoted by Z/nZZ.
divides Z into n equivalent classes: 0, I..... n2
binary operation: + 5
(it) Group of n-th root unity: Let n denote a positive integer and C, {e - |0 S S 1,a 3 be roots of xx = 1, and then Cn forms group under complex multiplication. For the mapping:
Cn (Zn; +) defined by f(e 2n1e
a, have
f( 2nte
=
(
= =a+b=
2ria
Therefore, a homomorphism, and thus is an isomorphism.
Congruence Class: Let n denote a positive integer, and we define an equivalence relation on Z as + a = b (mod n). The relation divides Z into n equivalent classes: 0, 1......,7 1, which are congruence classes and the elements of the additive group of integers modulo n.
Subgroup Generated by B: If is a nonempty subset of the group (G, o), the defined by B >= {a E Gla a1 02 .... an with either a, E B or a, B} forms a subgroup of G. which is called the subgroup generated by B, e.g.. G =< bla² of Section Here 01,... an are called Generators, e.g., a, in G.
ab ba >> in the toy example
Cyclic Group A group G' is called cyclic if there is an element a E G such that G = {a"In E Z} =< a C>, e.g.g. & nZ. Such an element a is called a generator of G.
Symmetry: A symmetry of geometric figure is a rearrangement of the figure preserving the arrangement of its sides and vertices as well as its distances and angles.
Dihedral Group: The permutation group formed by the symmetries of regular n-sided polygon, denoted by D,,
D4 as an example: an image dataset, the symmetries are analogous to image transformations. We often abbreviate permutation + 2, + 3,3 - 4, 4 + 1 as (1234). The elements of group D4 are shown in Figure 6, from which we know that all of the transformations are compounded by two basic permutations: Please note that f and are analogous to disentangled factors. What's 's more, the Group can be generated by these basic permutations f. with some properties, i.e., D4 =< flrff²
The vertices of square are numbered by (1, 4),, which is analogous to
horizontal flip and rotate 90 degrees clockwise
4 fr r-1 >. The constraints f2 1,r4 1, fr = - are analogous to the group constraints in this paper.
B PROOF FOR THEOREM
In our setting, the group-based definition is equivalent to the equivariant condition y f(w) =f(g w) for the following reason. Since the group actions of G on is the element wise addition, (a) the group action of G' Z exists and (6) group actions of G; only affect Z; (refer to Section 3.2). These two conditions of the group-based definition thus hold.
Proof. In the following, prove in Step that the necessary and sufficient conditions of & playing the "agent" role, i.e., f(g w) = h(pp b(w)), are (i) + isomorphic to G, and (ii) there exist (p; E Φ, s.t. 9p: b(w) b(g. w). i 1. ... We then prove in Step 2 that if equation f(g w) = h(pp -(w)) and the definition of y E + hold, then disentangled per group- based definition.
14

---

Published as a conference paper at ICLR 2022
Step 1: In the following, prove that the necessary and sufficient conditions of fgg = h(p-b(w))
are (t) and (ia) above. Note that this holds even without a specific definition of ④b
=) For general permutation group & on O, assume there exist (i such that y b(w) = blg: w) = ..2...., m. Assume that there exists an isomorphism T : G + L, where group G (Z/nZ)m From Group Theory, we know that there exists 14 E (ut(E) such that the following equation holds, where Aut(Φ) denotes the group of automorphisms of Φ i.e., /a is an isomorphism a 4 + 4. We denote the composition of μ., 7 as o, and have
(g)) = ()(( Pt, 1,2,.... m.
(8)
Since we have yp, b(w) - b(g: w), i = 1,2, ................................ ,,, the following equation holds,
( b(w) = p. b(w) = b(g. w), i = ,,2....,77. m.
(9)
Since both J4, r are isomorphism, o 1 o is also an isomorphism, which indicates that yp; is generator of Φ. For Vip 4, since
bijection, there exists g E G (2/n2) such that (g) = & and g = E kigi, E Z, where 9: = 1.2..... m denotes the generators in G. In order to calculate o b(w), we consider specific example (gg + 92).
((()+ w) (())) ((r()) b(w)) = (g)) (.66)-b((o)) b(9 b(gl -02 w0) b(()) +92) (10)
Therefore, for the general element io, we have
x .- b(w)= o E k.g.) b(w) bl biΣ kig w) = b(g ww)
(11)
In general, we have . b(w) b(g w) and thus we have h o b(g w) f[g w) = =h(p b(w)) when taking / on both sides.
=) Here we prove that (i) G is isomorphic to & and (ii) there exists is such that ;P; b(w) = blg: -w), i,2,,......... m are two necessary conditions of equation f(g. w) = b(p-b((w) = h(pp x)).
We take f on both sides of the equation. For convenience, we rewrite g w, ip b(w) as g(w) and ((b(w)), then we have
g(w) f- o h bow(()))))))))) = bb po
(12)
Note that the notation o here denotes the composition of functions. We define the mapping 7 between and G follows:
T(p) = g = o ob.
(13)
Note that b is a bijection, thus 7 is a bijection. We take ps, E o and we have
T(p: o (p)) b pi ob
= (b o pp ob) o (b- o s0 ob)
(14)
Therefore, is homomorphism and thus is isomorphism. i.e., + is isomorphic to G. Recall that group G has the form of G = (Z/nž)m, where it is direct product of m cyclic groups For Vip E Φ we have
8 b( - [((wx) = b(g(w)).
(15)
1,2..... 1,2, derived from the above equation:
For generators 9: E G. =
m and the corresponding ; r- (g:), we have specific one
PP (bw((())))))) ((]((w) b(g. w))
(16)
Q.E.D.
Step 2: As discussed in Section and Section 3, we define o as y x = p -(w) =
Mxx) = h- fw)), where is the same as in Step 1. Since is a bijection, for Vip $ o (p) uniquely exists, and thus is well-defined. We bring it into f(g w) = h(p b(w)), and derive f(g-w)= 9 f(w), i.e., the representation space Z is disentangled with respect to G. Specifically, for the existing si, i = 1, .,.,, m. from equation g we have o ((p:) gr and - = h h (ww).
Q.E.D.
15

---

Published as a conference paper at ICLR 2022
C PROOF FOR THEOREM 2
In order to easily distinguish a general element and a generator, we use & to stand for the general element in T and 9i stand for generators in t. In addition, for mapping G - Φ, notation ker(T) is generally the inverse image e, i.e., ker(T) = {g E Glr(g) = e), where e is the identity element in Ø.
Proof. Note that Section 3.4 describes the implementation of the elements in +, but there no guaran- tee that + is group. Therefore, here we first prove in Step that set + =< (P1 422, ... 9m 9/49 PypP, P = e, i = 1, ...... m and j 1, ..,,...,77 m > forms a group under the composition of functions, where e satisfies that el;; = Pt. then prove in Step the necessary and sufficient condition for the isomorphism is to meet both the Abel constraint and the Order constraint.
Step To prove set forms group under some operation, we only need verify that the elements of the set satisfy the following three properties: 1. Associativity 2. Identity 3. Inverse.
We first verify the Associativity property: Vé.. P2. (PI E ②, s.t. (0.0) )o: @
Recall that in Section 3.2, the representation space is set with the same elements in G. The group action of G on Z the addition. We have g 2 E Z, Vg E G, E Z, and then we have d(g 2) E O, Vg E G, 2 E Z. Therefore, ;i is permutation: Pi O - O, i.e, Pi E S(O), where S(O) the symmetry group on images set O, then for pic, Pp: PP E 4P,,, P22...... pm J. we have
(p:P;) = p)(()pp))
(17)
the generators of & thus satisfy the Associativity property. However, whether the general elements of satisfy the Associativity property is unknown. We take V d.. th, oi E $, where de = II, v0 ds
II. 1 th = II. o and li, Si, l E Z, and we have
(dee)) )or- = (II +
II
II
II
II
II
p = (1
(18)
Therefore, the set of mappings & satisfies the Associativity property under the composition of functions.
2. We then verify the Identity property: Be O, s.t. Tb cΦ dc Φ. For generators Pi: E fPP.. (2... pm we have pf = EPj = 45- Therefore, for general
elements, we take V v E 4 = II, p and we have:
p
p (ep))
II 4
II
p = II
(19)
((fS)
{(+)}
This states that the identity element Φ is spfh Φ Therefore, the set mappings & satisfies the Identity property.
3. We finally verify the Inverse property: Vó E + _ E G: 00 = 1001=0110 6-
For the generators Pi E f4P1, P2......... pml , we have pt ppt c, S n, E. For any general element & E $, @ = II, ". we have:
II p
II ((ッン))
(1)
e,
(20)
((1
denote II, P (i-k k 6- and have 00-1 Similarly, also have 1 These two equations state that for any general element E t 0 = II, ph we have an inverse element g = II, ,0"" in $ Therefore, the set of mappings satisfies the Inverse property. To summarize, T is a group.
Q.E.D.
Step 2: In this step, prove that the necessary and sufficient condition for the isomorphism
(Z(nZ)" ~e 1p, Pm > is the satisfaction of spi9 = Pype 1,2, m and j = 1. 1, 2,.... m. and w/ e, i 1,2,..., m.
16

---

Published as a conference paper at ICLR 2022
=) Assume the isomorphism is G (Z/nZ)" - + =< PI P2, 9m >, for generators
Pe: Pj E {41. .2,........................... (mm) there exist 9a = =, 0ggi, 9o = -, 3.g. E s.s.t. T(ga) = Gi T(gs) =
Pj. where 96, = 1,2,.... denotes the generators in G, which is the m-dimensional one-hot vector (O..... 1, (.....,0) 0) E G with in position and 0 elsewhere, and 02 B Z. Since G is Abelian group, we have
P:P; ((g))((g)) = ((ags)) = (gg3ga) = ((ga)((ga) Pjps.
(21)
For generators 9; E {4i; ( P2,..........................(.. fm.) we have 9. Ei aigi G,s.t. T(g.) Pis the times composition of itself is
- = =gaa)" = T(nga) = T( E agmgi) = a,ec) = T(eG) =
(22)
where eG, e are identity elements of G and & respectively. In Equation 22 T(eG) e holds because is an isomorphism, and ker(r) = {ea). The sufficiency is proven.
- In the following, we prove that when two conditions are satisfied simultaneously, the mapping we define is an isomorphism. Considering the mapping 7 T - G, defined as
T: Pp: 9s: T: PiPj + gi + 9j-
(23)
For general elements Φe, Ox E D, where Φe = 11, 1/:0000 0g = III pf" and 1, E Z, we have
II o
II p fi
I(++ 8i)g..
(24)
Note that these terms can be merged since Piiy = Pyps. Equation 24 is partitioned into two parts follows
The summation on the right side of
E(4 ((4++i)) + si)g; E'4g l.9. + Σ 8:9 =
II
1 +
II 1 = = ( + T()))
(25)
Consequently, we have (6)6))) r():) T())). which states that 7 is homomorphism. Then, only need to prove that mapping r is bijective. First, we prove t is injective, i.c., (tr(()) = {ea) after that, prove is surjective, i.e., can find the inverse image of any element.
Since is not hard to obtain T(Φ) = T(e Φ) = ((e) + (()), we have r(e) = (O...., 0) = eG. i.e.,
e E < Assume there is Φ1 = II, st. s.l. (db) = EG. we have
T(क1)= Σ 4g. cc L/ (1, are divisible by
II d Il e = (26)
The equation above states that - (e) = {e} (r : Φ + t) and mapping is injective. We take
Ag E g (ki, .i...., kim) E (Z/nZ)" m. and have
0) ..., g = (k ........)))
(O...., Rim
Σ kigi
II
(27)
Hence, we have - (g) = II, o E Φ which indicates that mapping T is surjective. Since mapping is bijective and homomorphism, 7 is an isomorphism.
Q.E.D.
D PROOF FOR THEOREM 3
Proof. the following, we prove that the necessary and sufficient condition for satisfying Abel and Order constraints is that both Abel and Order Loss are minimized.
+) For the condition:
Pr, PPj E Φ < i.j S m, we have Pig; = kp The constraint on any image E O can be obtained by
(o)) (p)()(()) = po(es(o)) ()(()(()) = 0.
(28)
17

---

Published as a conference paper at ICLR 2022
For the set of combinations of factors C {(i,j)] iij m) and the set containing images O we have
f== ===
Σ 11p: (Pj o) Pj (fp o)|l minimized.
(29)
(i.))CC
We obtain the Abel loss. For the Order loss, we first consider n times composition of the same mapping Pi, we have
91 = = 1/ pipp"" =
(30)
Therefore, then have si = 1-111 For single image O, have
pi p)():-((o) (o)) = (. (o)) =0 - (p (o)) =
(31)
Thus, for the factors and the set containing all images O, we have
E Σ l1 ( o) all is minimized.
(32)
DEOuSism
To eliminate the bias of optimization, we optimize the symmetry form of the Order Loss, and we have
L- Σ Σ (lee - o) - + l1+p, (p:: o) oll) is optimized.
(33)
OEOoSiSm
The Order Loss obtained.
When the Abel Loss La is optimized, for v E O, we have
(p(()(()) - ()(((()) = 0 = pp(((()) = 40 (p)()))
(34)
Therefore, for Pe, 42, Φ, i 1.2..... m and 1,2, ...... m. we have Pify = 4y and we obtain the Abel constraint. When the Order Loss Lo is minimized, for Vo E O, we have
( (0)) o = 0 (: (o)) =0
(35)
This implics that
Pi 1;;-1 9i p
(36)
Therefore, we obtain the Order constraint. The Group constraints are satisfied.
18

---

Published as a conference paper at ICLR 2022
E THE DATA CONSTRAINT
In this section, we prove that if cach dimension of the world state is independently sampled (p(w) = Πsp(s;), where denotes the probability mass function (for discrete random variable) probability density function (for continuous random variable)), then the minimization total correlation (p(2) I1.p(z)) ) a necessary condition satisfy the data constraint (see Theorem 4 and Theorem 5 beloow.
Our goal is to learn a representation conforming to the group-based definition of disentanglement with encoder h and decoder d. Then the data constraint be formulated as follows. For generator 9, E G, we have b(g. - w) pp b(w) h h- (gi 2), where = f(w) and is the corresponding permutation gi under the isomorphism between G and 4. We reorganize the formula
and have h o b(g: w) = f(: w) = 91
Theorem Assume that the action of generator gi on z is g. = 9: (element-wist addition), and the action of each generator 9i w only affects single dimension of w (see the assumptions in Section 3.2). Then the equation [(g ww = z, Vi E W is equivalent to: for each = 1,,2,- 1,2,...,y. there exists bijective function 7/4 8.l. 21 = ((w5) for some j.
Proof. =) Without loss of generality, assume 9i only affects the j-th dimension of w. If the equation flg w) = 9: 2, Vw E W holds, it's obvious that, for each i, there exists a bijective function 7/2 s.t. 2; (w)))
+) In the following, we use (g.ww, ww) ((g, )) to denote the i-th dimension of vector 9. w (g.-2). If for = 1...., n. the functions 2 Vi (W2)) hold for some j. by using index permutation j (i) we can rewrite the function as 2; = 7/1 (+)). Therefore, 2 can be formulated as follows
[wi,,... w.) = (2) (,,, .n)) (m1 (ww())) : (w(()()))
(37)
Please note that how the world state transits on dimension dose not affect how disentangled the representation is. Therefore, we define action of generator g: on as follows.
-:-w 9: (Wi.... Wn) (W1,.... ........ (g: w) ..... w,,) (W1, "U......77. 7;' ((g:
wa), (38)
Then, take f on both sides of the above equation, and apply Equation 37 and have
f(g w) f(w....... ((gs .).)...., ,,) = (n (w) ((% ((g. 2)))), Vn (W)()))
(z1,..., (9: z)i,.... zn) 9=
(39)
Q.E.D.
Note that is obvious that the following theorem (Theorem) 5 holds for discrete random variables and bijective functions. However, since the world state space is dense and the encoder and decoder d are differentiable in general, also prove for the case where w and are treated as continuous random variables. Theorem Sstatess states that the minimization of total correlation (p(z) = II,p(2)) is a necessary condition make equations 2; 7/ (()) satisfied, where 1, .., n and 7 (i) an index permutation.
Theorem For the independent random variables W1, W2...., Wn. considering the functions 2; = 8, (w1, W2...., Wx), where i 1,2...., TL and each 6; is bijective, differentiable function. For i 1...., n, if there exists an index permutation π (i) s.t., we have 0z Ows + and 0240)) O. where Owa
1,2..., nu but k + i, then the new random variables 21, 22. En are independent.
Proof. We treat the random variables W1, W.....,, Wn as an n-dimensional random vector W (wh, w.,..., wp). Similarly, we write Z = (2x(1), 2x. (x(()) which is rearranged by index
permutation (i). According to Change of Variable Theorem For Random Vectors (Mood 1950). we have
p(2))((W)= = p 11,p(w,)
(40)
where J(W) is the jacobian matrix of Z w.r.t. W, the (i,j)-th entry of it is 0z.m)
owey Since for each
i 1, 1,.... n, we have azatt) Ow. o 0 and
dzet))
0, where ko = 1,2.... ... but k o i, then the jacobian
19

---

Published as a conference paper at ICLR 2022
matrix can be formulated as follows
(z)
J(W) II,
(41)
Ow.
According to Change of Variable Theorem For Random Variable Mood (1550), we have
d2z(11)
p (())()))
= p( w.;)
(42)
Ow,
Bring Equation 41 and Equation
42
into Equation 40 we have
(1,((s))
11,p(zx())
pl ((
Dw.
ITp((x(()
IT,(2z)
(43)
dze(1)
11,
II:
Ow.
Q.E.D.
Please note that the inverse proposition of the theorem above does not hold. As counterexample, for zero mean independent gaussian random variables W1, W2 with common variance o², new random variables 21 v wf + w (the norm of vector (W1, Wy )), 22 tan (ww2/uu (/1//w1) (the angle between 2z vector (W1, w2) and (0, 1)) are independent (Mood 1950) but Dus + for all i,j E {1. In addition, the sufficiency would be satisfied for some specific settings, e.g., there are only two independent uniformly distributed random variables W1, W2, and the functions 21, 22 linear functions.
Combining Theorems and
have that the minimization of total correlation (p(z) 11,p(z))) is a necessary condition to satisfy the data constraint.
DETAILS OF IMPLEMENTATION
F.
ABEL LOSs DETAILS
As mentioned in the main paper, the Abel Loss of the VAE-based models is as follows:
-== Σ Σ 11p: (4, o) 9,, (yp: (p)),))ll
(44)
oEO (E,))C
where ii (4g (P)()(())) represents the top path of Figure (a), and So
(o)
represents the bottom path of Figure (a). For better optimization, we constrain such consistency on their representation (straight dotted double arrow in Figure (a)) instead of the reconstructed images.
Besides, we constrain the consistency between the representations of intermediate images (curved dotted double arrow in Figure (a)).
+i
Excooder
+uty
Dunde
Densderr
(a)
Abel Loss
(b) Order Loss
Figure 7: Overview the Isomorphism Loss. The Abel Loss and Order Loss constrain the commuta- tivity and cyclicity of permutation group t, respectively. The dotted lines in the figure represents reconstruction loss.
F.2 ORDER Loss DETAILS
As mentioned in the main paper, the Order Loss of VAE-based models is as follows:
C.
Σ Σ (1lp. (4. o) - oll + 1| (p:: o) - oll)
(45)
oE0
20

---

Published as a conference paper at ICLR 2022
where Pi (4p"' o) represents the lower path of Figure (b), and si (p, o) represents the upper path of Figure (b). Similar to the Abel Loss, we do not constrain such consistency on the reconstructed images for better optimization but their representations instead (the long curved dotted line Figure (b)). Besides, we constrain the consistency between the representations of intermediate images (short curved dotted lines in Figure (b)).
F.3 THE NUMBER FACTORS
For the given VAE-based models, dimensional KI. divergence (indicating the meaningful dimensions) increases during the training process. Therefore, we use m dimensions of which the corresponding dimensional KL divergence K L, z T, where T is a hyperparameter, which is empirically set to 30 our experiments.
21

---

Published as a conference paper at ICLR 2022
G DETAILS OF EXPERIMENTS
G.1 DATASET DETAILS
In all the experiments, we resize the images to 64 x 64 resolution. We introduce all the datasets used in our paper in detail.
dSprites| Higgins et al. (2017): :: dSprites contains 737,280 binary 2D shapes (heart, oval and square) images with five ground truth factors: shape (3 values), scale values), orientation (40 values), x-position (32 values), y-position (32 values). Then we introduce the variants of dSprites (Color dSprites and Noisy dSprites) created by Locatello al. Locatello al. (2019b).
Color dSprites, the shapes of the images in dSprites are randomly colored.
Noisy dSprites, the background of the images in dSprites noise.
Shapes3D Kim & Mnih (2018): Shapes3D contains 480,000 images of 3D shapes with 6 ground truth factors: shape values), scale (8 values), orientation (15 values), floor color (10 values), wall color (15 values), object color (10 values).
Cars3D Reed et al. (2015): This dataset consists of 183 car CAD models, each rendered from 24 azimuth directions and 4 elevations.
Encoder
Decoder
Input: 64x64 x number of channels
Input: 10 or 20
4 4 conv, 32 ReLU, stride
FC, 256 ReLU
4 X 4 conv, 32 ReLU, stride 2
FC, 256 ReLU
4 4 conv, 32 ReLU, stride
FC, X 4 32 ReLU
4 x 4 conv, 32 ReLU, stride 2
4 x 4 deconv, 32 ReLU, stride 2
FC 256 ReL.U
deconv, 32 ReLU, stride
FC 256 ReLU
4 4 deconv, 32 ReLU, stride 2
FC 2x 10
4 deconv, number channels, stride 2
Table 3: Architecture the encoder and decoder of VAEs. For Original VAE, the dimension of input of the decoder is 10. For Groupified VAE, the dimension is 20. Note that the number representation dimensions of Groupified VAE is still 10, which is the same as Original VAE, the comparison with Original VAE is fair Watters et 20199)
G.2 ARCHITECTURE FOR ENCODER AND DECODER
We follow Locatello et al. Locatello et al. 2019b) to use the same architecture of VAES in all of the experiments: the activation function used is ReLU except for the last layer of decoder, shown in Table 3 For the details of the Discriminator in FactorVAE, please refer to Table (a) and (c).
Model 3-VAE
Parameter
Value
[10; 20; 30:]
AnnealedVAE
C start
[10; 20; 30:]
[3e4; 4e4; 5e4:]
end
[2e4; 3e4; 4e4:]
FactorVAE 3-TCVAE
[5; 10; 15]
[6; 9; 12]
random seed group
[1: 2; 3; 4: 5; 6; 7; 8; 9:]
[True; False]
Table 4: Hyperparameters and random seeds for every model.
22

---

Published as a conference paper at ICLR 2022
(c) Architecture Dis-
(a) Optimizer for Discriminator
(b) General hyperparameters for VAE
criminator
Parameter
Values
Parameter
Values
Discriminator
Batch size
Batch size
64
FC. 1000 LReLU
Optimizer
Adam
Representation dimension
10
FC, 1000 LReLU
Adam: betal
0,9
Optimizer
Adam
FC, 1000 LReLU
Adam: beta2
0.999
Adam: betal
0.9
FC, 1000 LRcLU
Adam: epsilon
1.0c-8
Adam: beta2
0,999
FC. 1000 LReLU
Adam: learning rate
0.0001
Adam: epsilon
1.0e-8
FC, 1000 LReLU
Adam: learning rate
0.0001
FC,
Decoder type
Bernoulli
Table 5: Shared hyperparameters in all experiments. LReLU stands for leaky ReLU.
G.3 EXPERIMENT SETTINGS
We run using different hy perparameetess and random seeds for every VAE-based model implemented by Pytorch Paszke et al. (2017). As shown in Table for 3- B-V,,, we assign 3 choices for 3 and 10 random seeds for both the Original and Groupified VAEs: 10 x 60 settings for each dataset. assign three choices for C' and 3 choices for the start and end pair, also assign 10 random seeds. summary, for all 5 datasets, run (((3 10 x 2) x 3) + 3 10 2) 1800 models. For other hyperparameters, please refer to Table (b).
Similarly, also assign 60 settings for FactorVAE and 3-TCVAE, For AnnealVAE,,
H RELATION TO SOME PREVIOUS WORKS
H.1 SIAMETRY-BASED DISENTANGLEMENT
Caselles- Dupré al. (0199) argue that the symmetry-based disentanglement requires interaction with the environments. Specifically, for given disentangled representation z w.r.t. a world with some group action on W, there are multiple other worlds (same world states and symmetry) of a static dataset that have different group actions on each dimension of W. For those worlds, the not disentangled, per the group-based definition. However, in our work, we do not assume that a static dataset has world equipped with some specific fixed group action on W in advance, instead we use permutation group as an agent to learn world with proper group actions. For this learned world, there is only one representation that satisfies the definition. Therefore, we do not need an environment to provide group actions on W to determine which world it is. Please our proof in Theorem 1. In the testing phase, since input the images the model directly to derive their representation, the disentanglement of the representation does not rely on how the world state transitions between each other (i.e., result of group actions W). Therefore, framework can learn such a disentangled representation without interaction with the environments.
representation
H.2 LIE GROUP VAE
Zhu et al. (2021) argue that the representation space being a vector space is sub-optimal since requires the model to learn to discard different scales of variations. They propose to use lie group as the representation space instead and use Hessian Penalty to encourage disentanglement. Our framework is complimentary to their proposed method. The lie group representation can be applied to extend our framework. Here we leave it for future work. In addition, our proposed representation mapped by the sine and cosine function (erp((2riz)) /n) for real vector 2) is also lie group.
23

---

Published as a conference paper at ICLR 2022
1 MORE QUANTITATIVE RESULTS AND SCORE DISTRIBUTION
I.1 QUANTITATIVE RESULTS
The performance of the Original and Groupified VAES on all five datasets is shown in Table 6 Our method outperforms the original one on most of the datasets in terms of nearly all the metrics.
dSpetiss
ICI
VAE
MIG
FactorVAH
Original
Groupified
Original
Groupified
Original
Groupiseed
Original
Groupified
8-VAE
0.23 4: 0.10
0.46 1 0.085
0.75 4: 0.083
0.86 =: 0.051
0.14 =: 0.087
0.37 1: 0.089
0.51 1: 0.098
0.63 0.089
AnneulVAE
0.28 + 0.10
0.39 0.056
0.84 + 0.030
0.87 0.0067
0.23 + 0.10
0.34 0.061
0.70 0.094
0.68 0,058
FactorVAE
0.38 0:100
0.41 0.074
0.89 0.040
0.89 = 0.020
0.27 0.092
0.31 0.061
0.74 0.068
0.75 0.075
0:35 0.065
0.36 0.11
0.86 + 0.026
0.861 + 0.038
0.17 0.067
0.24 0.003
0.69 + 0.008
0.70 0.008
Cars3d
DC1 Groepilied
MIG
VAc
Original
Original
Groupilied
Original
Groupitied
Original
Groupilied
8-VAR
0.18 + 0.059
0.24 0.041
0 1.6c 3
1.0 0.0
0.071 + 0.0332
0.11 + 0.032
0.81 + 000066
0.93 + 0.034
0.22 0.046
0.046
0.99 4e
0.99 1.5e
0.074 + 0.016
0.10 0.014
0.82 0.062
0.87 0.028
VactorVAE
0.21
0.040
0.99
1.0 0.0
0.027
0.90
0.034
0.24 2 0.000
0.26 1 0.046
1.0 1 0.0
1.0 0.0
0.10 = 0.021
0.11 1: 0.003
0.88 a: 0.040
0.93 0.034
Noisy
MIG
VActorVAF
dSprits
Original
Groupiñed
Original
Groupified
Original
GrouptSid
Orgiiall
Groupißeed
0.056 +:66+:00 0.018
0.087 + 0.051
0.624 +: 0.090
0.847 = 0.055
0.030 = 0.022
0.085 = 0.065
0.355 =: 0.093
0.407 + 0.071
Anneal VAF
0,533 + 0.013
0.0600 + 0.022
0.631 + 0.096
0.644
0.031
0.335 - 0.027
0.047 0.082
0.434 0.080
0.481 0.087
VadorVAE
0.114
0,062
0.099 0.057
0.682 1 0.081
0.684
0.070
0.077 0.0046
0,066 1 0.046
0.457 0.088
0.468 1 0.098
B-TVVAE
0.081
0.053
0.050
0.040 0:000
0.088
0.353
Shupes3d
BetaVAE
FactorVAE
Original
Groupißed
Original
Groupißed
Original
Groupified
Original
Groupified
(-VAH
0.44 + 0.176
0.56 + 0.10
0.91 0.072
0.90 + 0.045
0:28 0.18
0.42 = 0.15
0.82 - -1.088
0.82 0.043
Ann
0.52 0.051
0.60 + 0.078
0.82 0.076
0.89 0.086
0.48 0.047
0.50 0,052
0.75 0.074
0.83 0.066
VAC
0.47 0.10
0.49 0.065
0.86 0.055
0.80 :: 0.075
0.33 0:13
0.43 0.11
0.81 0.056
0.79 0.066
--TCVAE
0.66 2 0.10
0.72 a 0.061
0.97 0.039
0.96 2 0.012
0.40 4 0.18
0.47 1 0.090
0.59 0.064
0,90 0.046
Color
DCI
Vei
MIG
FactoVVEB
dSprits
Original
Groupised
Ongiinal
Groupified
Original
Groupified
Original
Groupited
BeiVVA
0.174 + 0.097
0.328
0:130
0 0.798-2 + 0.094
0.844
0.050
0.103 + 0
0.243 + 0.118
0.501 =0. 0.148
0.648 + 0.092
Anneal VAF
0.268
0.337 0.114
0.008
0.856 0.031
0.084
0.252 0.104
0.718
0.034
FactorVAE
0.294
0.101
0.322 0.104
0.961 0.008
0.862 0.029
0.203
0.236 0.091
0.729
0.068
0.730 0,080
8-TCVAB
0.338 + 0.052
0.305 0.082
0.876 :: 0.024
0.681 0.031
0.160 : 0.040
0.269 0.000
0.711
0.086
0.788 0.050
Table 6: Performance (mean + std) on different datasets and different models with different metrics. We evaluate 3-VAE, VARe Factor and 3-TCVAF on dSprites, Cars3d, Shapes3d, Noisy- dSprites, and Color-dSprites for 1800 settings. These settings include different random seeds and hyperparameters.
I.2 ABSTRACT REASONING & FAIRNESS
Abstruct reasoning?
Unfaimess scores.
As pointed out by Locatello et al. Locatello et al. (2019b), the disentangled representation' ' down- stream tasks should also be verified. There- fore, we verify the effectiveness of the repre- sentations learned by the Groupified VAEs on Shapes3d in two downstream tasks: abstract rea- soning Van Steenkiste et al.] (2019) and fairness evaluation Locatello et al. 2019a). As Table 7 shows, the performance of the abstract reasoning models fine-tuned on the representation learned by the Groupified Factor is better than the original ones. In terms of fairness evaluation, we can observe that the unfairness scores of the representation learned by the Groupified FactorVAEs are lower than the Original ones.
Original Groupified
0.948 + 0.031
0.023 + 0.007
0.954 1 0.028
0.018 1 0.008
Table 7: Downstream task performance on the models trained on the representation learned by original and groupified FactorVAE.
I.3 SCORE DISTRIBUTION
The detailed distribution of the performance is shown in this section (demonstrated by the Violin Plot Hintze & Nelson (1998). The performance distributions on dSprits, Car3d, Noisy dSprites,
24

---

Published as a conference paper at ICLR 2022
Color-dSprites, and Shapes3d are shown in Figure Figure respectively.
Figure
10
Figure
11 and Figure
12
1.4 COMPARISON WITH METHOD INTERACTION WITH THE ENVIRONMENT
Our work considers an unsupervised setting, which is more practical one. To understand the price to pay, we compare our unsupervised Groupified models to the methods that use the interaction with the environment as supervision. Here we provide the comparison between RGIVAE (Panter et
2020) and our Groupified 3- 8-TCVAE on dSprites, Shapes3D,, and Color dSprites as shown in Table g
Datasets
DCI
BetaVAE
MIG
Factor VAE
RG:VAE
Groupited
RG/VAE
Groupified
RGIVAE
Groupified
RG:VAE
Grouptied
dSprites
Shapes3D
0.83 1. 0.056
0.72 1 0.061
1.00 1 0.000
0.96 a 0.042
0.25 à 0.031
0.47
0.090
0.98 4: 0.032
0.90 4: 0.046
Color dSprites
0.11 0.002
0.40 0.082
0.294
0.031
0.028
0.27 0.090
0.300
0.79 0.050
Table 8:
Performance (mean + variance) different datasets of RGIVAE and Groupified 3-TCVAE with different metrics. These settings include different random seeds and hyperparameters.
Since there are no results reported on Shapes3D and Color dSprites, we conduct experiments on these two datasets with the official implementation using the recommended syper-parameent addition, the results reported in Painter et al. (2020) are of models trained with 16 latent units and 3 random seeds. We also conduct experiments on dSprites with 10 latent units and 10 random seeds, which is our setting. From Table we observe that there is still gap between RGrVAE and Groupified 3-TCVAE, especially on Shapes3D. However, the latent learned by RG+VAE is not as pure as Groupified 3-TCVAE (lower MIG). Additionally, RGrVAE performs poorly because factor (color) is not modeled in the environment of Color dSprites.
In
I.5 COMPARISON WITH CONTROLVAE
In this section, we provide a comparison between Original and Groupified ControlVAE (Shao et al. 2020). For VAn, we use the official implementation and follow the default setting,
maa 25. We follow Locatello al. (2019b) to set the hy hyperparameter interval to 10. For other
parameters, we follow Shao et al. (2020). The results dSprites, Shapes3D, and Color dSprites
presented in Table 9
DCI
BetaVAE
FactorVAB
Original
Groupified
Original
Groupified
Original
Groupified
Original
Groupified
dSprites
0.31 0.003
0.46 0.115
0.83
0.084
0.081
0.16 0.082
0.27 0.123
0.62 4: 0.087
0.74 i 0.110
Shapes3D
0.59 0.144
0.85 0.165
0.86
0.142
0.97 : 0.149
0.21 0.212
0.72 0.151
0.59 a: 0.174
0.58 i 0.171
Color Color dSprites
0.47 0.111
0.54 0.055
0.94
4: 0.045
0.98 + 0.024
0.28 4 0.086
0.35 0.055
0.80 4 0.070
0.84 4 0.022
Table Performance (mean variance) on different datasets of ControlVAE and Groupified Con- troIVAE with different metries. These settings include different random seeds and hyperparameters.
From Table 9
we observe that our method consistently improved the performance of ControlVAE under the same hyper-parameters, especially on Shapes3D.
1.6 DETAILED RESULTS UNDER DIFFERENT PAPEERPPRAMETERS
In order to provide more convincing comparison, we compare our method and the original one at different levels regularization parameters. We take 3-TCVAE as an example. As Table 10 TT and
12 show, our method consistently improves the performance of the original methods on most of the metrics, especially on MIG and DCI.
attos: /e
25

---

Published as a conference paper at ICLR 2022
Regulize strength
BetaVAE
MIG
FactorVAB
Original
Groupified
Original
Groupified
Original
Groupitied
Original
Groupified
0.022
0,390
0.105
9=9
0.37 + 0.051
0.36 = 0. 103
0.85
0.028
0.87
0.026
0.17 # 0.053
0.24 = 0.097
0.63 2 0.092
0.70 2: 0.002
0.35
0.057
0.38 0.114
0.87
0.024
0.86
0.051
0.20 1. 0.071
0.30 0.088
0.71 0,084
0.71 0.104
Table 10: Performance (mean + variance) on dSprites of Original and Groupified 8-TCVAE with different metrics. The results under different regularize strength are reported.
Regulize strength
DCI
BetaVAE
MIG
FaciorVAE
Original
Groupilled
Original
Groupiiid
Original
Groupified
Original
Groupilied
8=6
0.56 0.080
0.68 = 0.071
0.98 0.028
0.97 0.036
0.31 +: 0.123
0.42 : 0.098
0.88 0.046
0.90 0.052
8=9
0.07 + 0.089
0.76 = 0.048
0.97 0.087
0.96 0.048
0.38 + 0.196
0.51 + 0.084
0.90 0,078
0.89 0.015
0.053
0.96 0.044
0.059
Table 11: Performance (mean + variance) on Shapes3D of Original and Groupified 3-TCVAE with different metrics, The results under different regularize strength are reported.
Regulize
DCT
BetaVAE
MIG
FactorVAR
Original
Groupified
Original
Groupified
Original
Groupifedd
Original
Groupified
8=6
0.35 0.051
0.43 e 0.073
0.80
0.009
0.89
0.016
0.17 1. 0.038
0.29 1 0.073
0.73 0.084
0.82 1 0.027
0.104
3= 12
0.31 +: 0.061
0.35 = 0.081
0.86 0.081
0.87 0.020
0.16 0.011
0.21 + 0.083
0.66 2 0.060
0.75 = 0.051
Table 12: Performance (mean + variance) on Color dSprites of Original and Groupified 3-TCVAE with different metrics. The results under different regularize strength reported.
B-VAE
0,
0.2
01
Graathe
Oreecifod
Modris
Dese
(a) VAt score
(b) DCI disentanglement
(c) MIG
(d) FactorVAE score
0.
Anneal
I
0.2
6:75
Madic
Malth
(e) BetaVVE score
disentanglement
MIG
(h) FactorVAI score
Factor VAE
0.0
Galo
2 6.4
0:5
Origial
Vodks
Madich
Modih
Midill
(i) VetaVAE score
ii) DCI disentanglement
(k) MIG
(I) FactorVAB score
B-TCVAE
0.4
0.2
0.9
6:09
Grearie
Deseiieed
Droed
Drsecifict
VAI score
DCI disentanglement
MIG
(p) FactorVAI
Figure 8: Performance distribution dSprites. Variance is due different hyperparameters and random seeds. We consider four metrics: BetaVAE score, DCI disentanglement, MIG, and FactorVAF score. We observe that Groupified VAEs outperform the original ones.
26

---

Published as a conference paper at ICLR 2022
1.00
3 3-VAE
0,003
eub
0.7
0.7
Madhh
Soathh
(a) VAt score
(b) DCI disentanglement
(c) MIG
(d) FactorVAE score
AnnealVAR
1000
640
011
0:00
600
0.t0
Bubt
0.70
Staces
Mokes
(e) Beta score
DCI disentanglement
(g) MIG
(h) FactorVAE score
Factor
0405
03959
010
BetaVVE score
DCI disentanglement
MIG
FactorVAB score
8 3-TCVAE
6335
3.907
$375
1,00
0.5
Grauelid
VAI score
DCI disentanglement
MIG
(p) FactorVAE
Figure 9: Performance distribution Cars3d. Variance is due different hyperparameters and random seeds. We observe that Groupified models outperform the Original ones.
27

---

Published as a conference paper at ICLR 2022
[ VAE
ba
05
870
015
53
64
6m
BetaVAE
DCT disentanglement
MIG
(d) FactorVAE
Anneal VAB
0.10
0100
!
6400
0:30
Oreachee
Dreashhed
Drearhod
BetaVVEE
DCI disentanglement
MIG
(h) FactorVAE
FactorVAB
6.
82
15
$
0.30
0.4
6.0
BA0
Posset
Model
Modett
(i) BetaVVE score
(j) DCI disentanglement
(k) MIG
(1) Factor VAE score
--TCVAE
0:0
015
8399
0:0
6539
000
6129
Pokk
Riddi
(m) BeIaVAE score
(n) DCI disentanglement
(0) MIG
(p) FactorVAI score
Figure 10: Performance distribution on Noisy dSprites. Variance is due different hyperparameters and random seeds. We observe that Groupified VAEs outperform the Original ones.
28

---

Published as a conference paper at ICLR 2022
8 DAVAE
0.7
2.72
6.0
Orearkee
Groepiiig
(a) BetaVVA score
(h) DCT disentanglement
(c) MIG
(d) FactorVAR score
Anneal VAE
0.3
0.2
02
0.1
Madics
(e) Beta score
DCI disentanglement
(g) MIG
(h) FactorVAE score
Factor tArE
0.05
$300
83875
040-
0.5
Oriphell
Oreed
Orignall
Groud
oeggial
Srocd
Maaics
Micic
(i) BetaVVAE score
ii) DCT disentanglement
(k) MIG
(l) Factor VAE score
B-TCVAE
8875
0:25
mnn
Oriinal
CoapFel
Drien
(m) BAta score
(n) DCI disentanglement
(0) MIG
(p) FactorVAR score
Figure 11: Performance distribution on Color dSprites. Variance is due to different hyperparameters and random seeds. We observe that Groupified VAEs outperform the Original ones.
29

---

Published as a conference paper at ICLR 2022
3-VAE
E:00
0.3
8.00
0.2
Genasrfes
Orchrl
Creasied
03
Orthol
Drosi Fied
Groep
(a) Beta score
(b) DCT disentanglement
(c) MIG
(d) FactorVAR score
3.00
Anneal
0:00
0:50
8:60
045
042
BetaVAR
DCT disentanglement
MIG
(h) FactorVAR
FactorVAB
0
Maits
Maaics
Madih
(i) BetaVAE score
ii) DCI disentanglement
(k) MIG
(1) FactorVAL score
3-TCVAB
1891
$359
0.00
Orchel
GrouSFed
Ortioll
Drsccfed
Seeaa
(m) VAta score
(n) DCI disentanglement
(0) MIG
(p) FactorVAR score
Figure 12: Performance distribution on Shapes3d. Variance is due to different hyperparameters and random seeds. We observe that Groupified models outperform the Original ones.
30

---

Published as a conference paper at ICLR 2022
J MORE QUALITATIVE RESULTS
We evaluate our methods qualitatively on two typical datasets: Cars3d and Shapes3d. We visualize the traversal results of the Original and Groupified FactorVAE and 3-TCVAE. For every factor, traverse five randomly sampled representations. As shown in Figure 13 and Figure [4] the traversal results the Groupified FactorVAE and 3-TCVAE show that these models learn less entangled representations on Shapes3D (e.g., Orientation of FactorVAE and Scale and Shape of 3-TCVAE).
Similarly, as shown in Figure 15 and Figure 16 the Groupified FactorVAEs and 3-TCVAE achieve better disentanglement ability on Car3d (e.g.. Rotation of FactorVAB and Yaw of (3TCVVA).
For the real-world dataset, we show the qualitative comparison of the Groupified and original AnnealVAE trained on CelebA. As sig for most of the factors, our model can extract cleaner semanties than the original model. For example, the hair color entangled with the face shape in the original model, but cleaner in the Groupified model.
Original
Groupified
(a) Orientation (b) Wall color (c) Object color (d) Floor color
(c) Scale
(f) Shape
Figure 13: Learned latent variables using Original and Groupified FactorVAE on Shapes3d dataset.
The traversal range is (-2, 2).
Original
Groupified
(a) Orientation (b) Wall color (c) Object color (d) Floor color
(e) Scale
Shape
Figure 14: Learned latent variables using Original and Groupified 3-TCVAE Shapes3d dataset.
The traversal range is (-2, 2).
31

---

Published as a conference paper at ICLR 2022
Original
Groupified
(a) Yaw
(b) Color
(c)
Whiten
(d) Rotation
Figure 15: Learned latent variables using Original and Groupified FactorVAE on Car3d dataset. The traversal range is (-2, 2).
Original
Groupified
(a) Azimuth
(b) Yaw
(c) Color
(d) Whiten
(c) Rotation
Figure 16: Learned latent variables using Original and Groupified 3-TCVAE on Car3d dataset. The
traversal range is (-2,22 2).
(a) Original
(b) Groupified
hair length
gender skin color
azimuth
hair style
skin color2
face shape
smile glasses
Figure 17: Learned latent variables using Original and Groupified AnnealVAB. on CeleBa dataset.
The traversal range is (-2, 2). The factors learned by Groupified model are less entangled.
32

---

Published as a conference paper at ICLR 2022
K MEANINGFUL DIMENSION VISUALIZATIONS
When we assign some dimensions to Isomorphism Loss for Groupified AnnealVABS, e.g., the first-5 dimensions, have an interesting observation that the assigned dimensions are meaningful in Groupified AnnealVAEss. The KL divergence increases continuously on these assigned dimensions after the Isomorphism Loss is applied to them. Note that the KL divergence loss AnnealVAB indicates the amount of information encoded. As Figure I dimensions increases the beginning of training, which means Isomorphism Loss results in that the assigned dimensions become meaningful. Finally, the assigned first-five dimensions learn to encode the semantics of position, y position, scale, and orientations. The possible reason that the latent factors are learned and disentangled in the assigned dimensions due to the punishment of the Isomorphism Loss. To illustrate that controllable dimensions in Groupified AnnealVAE are not exception, provide more visualizations. The results of covering all hyperparameters settings with two random seeds are shown in Figure 18 to Figure 26 suggesting that their dimensions are controllable.
the KL divergence of assigned
leurooo Eo veedie decagne prisssg
divergenca dimaniirss
:
13.00
30007
(a) Dimensional KL. training (b) Dimensional KL. in 3e5 steps
(c) Images traversal of dimensions
HL diergaace all dimansiors
9008
D60C
15000
20008
2500l
3000
(d) Dimensional KL in training
(e) Dimensional KL in 3e5 steps (f) Images traversal of dimensions
Figure 18: Mcaningful dimensions visualization for C' = 10, end = 30000 (different random seeds). The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a). The KL. divergences in different dimensions are different after training (b). As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which are results of run with different random seed.
33

---

Published as a conference paper at ICLR 2022
divergenco of dimmesoons
1998
(a) Dimensional KL in training
(b) Dimensional
KL. in 4e5 steps
(c) Images traversal of dimensions
Linwvioe wite EI dhergeare triiiing proosss
dhergesce al dimiskors
:
1955
(d) Dimensional KL in training
(e) Dimensional
KL in 4e5 steps (I) Images traversal of dimensions
Figure 19: Meaningful dimensions visualization for C' 10, end = 40000 (different random seeds). The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a). The KL divergences in different dimensions are different amounts after training (b). As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So as (d), (e), and (f), which are results of run with a different random seed.
divergencc all dimemaars
:
(a) Dimensional KL in training
(b) Dimensional KL in 5e5 steps
(c) Images traversal of dimensions
Eimevco E1 dieegence proes
dheergesce el dimistoos
:
19000
(d) Dimensional KL in training
(e) Dimensional KL in 5c5 steps (I) Images traversal of dimensions
Figure 20: Meaningful dimensions visualization for C 10, end = 50000 (different random seeds). The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a). The KL divergences in different dimensions are different after training (b). As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which are results of run with different random seed.
34

---

Published as a conference paper at ICLR 2022
devergence of dimmesoons
10309
(a) Dimensional KL in training
(b) Dimensional
KL in 3e5 steps
(c) Images traversal of dimensions
Erteeiing Linwaizr wie dongess vp treanng sirees
dhergesce al dimiskors
:
10h09
15003
(d) Dimensional KL in training
(e) Dimensional KL in 3e5 steps (I) Images traversal of dimensions
Figure 21: Meaningful dimensions visualization for C' 20, end = 30000 (different random seeds). The KL divergences of the target dimension (0-4 dimension) increase one by one during training (a). The KL divergences in different dimensions are different amounts after training (b). As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So as (d), (e), and (f), which have different random seed.
divergence all dimewaors
:
26.8
(a) Dimensional KL in training
(b) Dimensional KL in 4e5 steps
(c) Images traversal of dimensions
Emmrnirr Nowe doargese treinng mrrees
dhveggesee el diwistoos
30033 2018 2919932023
(d) Dimensional KL in training
(e) Dimensional KL in 4e5 steps (I) Images traversal of dimensions
Figure 22: Meaningful dimensions visualization for C 20, end 40000 (different random seeds). The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a). The KL divergences in different dimensions are different after training (b). As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which are results of run with different random seed.
35

---

Published as a conference paper at ICLR 2022
B:2waing Eioenerr Aita diwggsse treiing prrees
diveggenc all dimansiors
(a) Dimensional KL in training
(b) Dimensional
KL in 5e5 steps
(c) Images traversal of dimensions
frennng
HL devegence in of dimiestrss
:
30001
1100
(d) Dimensional KL in training
(c) Dimensional KL in 5e5 steps (f) Images traversal of dimensions
Figure 23: Meaningful dimensions visualization for C = 20, end = 50000 (different random seeds). The KL divergences of the target dimension (0-4 dimension) increase one by one during training (a). The KL divergences in different dimensions are different after training (b). As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which are results of run with different random seed.
divergnnce training procoss
diverpence dimensioos
draw1
di
dmap
3000
10000
15020
20000
25320 32000
Traaiin staes
(a) Dimensional KL in training
(b) Dimensional KL in 3e5 steps (c) Images traversal of dimensions
mension wize diveggence training process
Ki diverpence all dimensions
drows
6m-4
dma5
Toainn
(d) Dimensional KL in training
(e) Dimensional KL in 3e5 steps
(f) Images traversal of dimensions
Figure 24: Mcaningful dimensions visualization for C' = 30, end = 30000 (different random seeds). The KL, divergences the target dimension (0-4 dimension) increase one by one during training (a). The KL divergences in different dimensions are different after training (b). As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which are results of run with different random seed.
36

---

Published as a conference paper at ICLR 2022
Dimension wite dieegence traning process
KL divergence all dimonsioos
drte2
5000 1000010300000000
50000 19000 15003
600g
Thaiiing alssa
(a) Dimensional KL in training
(b) Dimensional KL in 4c5 steps
(c) Images traversal of dimensions
EL d-derrrueed in allormrrsian
4056 1056r
19000
30033
9076
10300
a000
(d) Dimensional KL in training
(c) Dimensional KL in 4c5 steps (f) Images traversal of dimensions
Figure 25: Meaningful dimensions visualization for C' 30, end = 40000 (different random seeds). The KL divergences the target dimension (0-4 dimension) increase one by one during training (a). The KL divergences in different dimensions are different after training (b). As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (c), and (f), which results of run with different random seed.
Shay doevede who dheege ie mano
RL. dedooriuuuddd in zll iilddieecions
(a) Dimensional KL. in training (b) Dimensional KI. in Se5 steps
(c) Images traversal of dimensions
clowwie darge treens
RL d--ddrouued in diceeersaa
:
3008
23002
4000
(d) Dimensional KL in training
(e) Dimensional KL in Se5 steps (f) Images traversal of dimensions
Figure 26: Mcaningful dimensions visualization for C' = 30, end = 50000 (different random seeds). The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a). The KL divergences in different dimensions are different after training (b). As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which are results of run with different random seed.
37

---

Published as a conference paper at ICLR 2022
L MORE REPRESENTATION SPACE VISUALIZATIONS
To illustrate that the Isomorphism Loss suppresses the representation space collapse VAE-based methods is not an exception, we provide more visualizations of the representation space visualization of Groupified VAEs. The Original and Groupified 3-VAEs, Anneal FactorVAEs, and 3- TCVAEs are shown in Figure 27 Figure 28 Figure 29 and Figure 30 respectively, which implies that the representation space of Groupified VAEs are organized better than the original ones.
(a) 8 10, Groupified
(b) B 10, Original
(c) B 10, Groupified
(d) 3 10, Original
(e) B = 20, Groupified
(1) B = 20, Original
(g) 3 = 20, Groupified
(b) 3 = 20, Original
(i B 30, Groupified
① 3 = 30, Original
(k) 8 = 30, Groupified
(1) | 30, Original
The representation space span by the Groupified and Original 3AVAE We train the models with the same hyperparameter but different random seeds for different runs. The 3D location of each point is the disentangled representation of the corresponding image. Moreover, an ideal result that the points form a cube, and color variation continuous. Higher hyper-parameter 3 results in the collapse of representation space. The collapse is suppressed by the Isomorphism Loss, which leads to better disentanglement.
Figure 27:
38

---

Published as a conference paper at ICLR 2022
(a) C' = 10, Groupified
(b) C = 10, Original
(c) C = 10, Groupified
(d) C = 10, Original
(e) C' 20, Groupified
(f) C' 20, Original
g C 20, Groupified
(h) C 20, Original
(i 30, Groupified
( 30, Original
(k) C = 30, Groupified
(1)
30, Original
Figure 28: The representation space span by the Groupified and Original AnnealVAE. We train the models with the same hyperparameter but different random seeds for different runs. The 3D location of each point is the disentangled representation of the corresponding image. Moreover, an ideal result is that all the points form cube, and color variation is continuous. Higher parem-parameter C' results the collapse of representation space. The collapse suppressed by the Isomorphism Loss, which leads to better disentanglement.
39

---

Published as a conference paper at ICLR 2022
(a) = 5, Groupified
(b) = 5, Original
(c) 5, Groupified
(d) = 5, Original
(c) 7 10, Groupified
( 10, Original
7= 10, Groupitied
(h) 7 10, Original
15, Groupified
( 7 15, Original
(k) 7 15, Groupified
(1) 7 15, Original
The representation space span by the Groupified and Original Fac.orVAE. We train the models with the same hyperparameter but different random seeds for different runs. The 3D location of each point is the disentangled representation of the corresponding image. Moreover, an ideal result is that all the points form a cube, and color variation is continuous. The collapse is suppressed by the Isomorphism Loss, which leads to better disentanglement.
Figure 29:
40

---

Published as a conference paper at ICLR 2022
(a) 3 6, Groupified
(b) B 6, Original
(c) B = 6, Groupified
(d) B 6, Original
(e) Groupified
(f) 9, Original
(g) Groupified
(h) Original
(i B 12, Groupified
( 12, Original
(k) 8 = 12, Groupitied
(1 = 12, Original
Figure 30: The representation space span by the Groupified and Original 3-TCVAE We train the models with the same hyperparameter but different random seeds for different runs. The 3D location of each point is the disentangled representation of the corresponding image. Moreover, an ideal result is that all the points form cube, and color variation is continuous. Higher hyper- parameer C results the collapse of representation space. The collapse suppressed by the Isomorphism Loss, which leads to better disentanglement.
41


Reviews:
[{"review_id": "ts_zv_9FZUy", "reviewer": "Reviewer_M8Aa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The mathematical part of the paper and experiments appear sound.\n- The isomorphism loss is to my knowledge novel, and seems to bring performance gain across datasets and models.\n\nWeaknesses:\n-  The paper is at best ambiguous and at worst misleading concerning whether or not it claims to solve the problem of unsupervised representation disentanglement\n    - *\"we offer an option [...] for the inductive bias that existing VAE-based models lack\"*\n    - *\"Finally, we provide a learning model based on the existing VAE-based methods in an effort to fulfill the three conditions.\"*\n    - *\"we are the first to provide a theoretical framework to make the formal group-based mathematical definition of disentanglement practically applicable to unsupervised representation disentanglement.\"*\n   - *\"making the learned representation conform to the group- based definition without relying on the environment\"* \n\n   But the paper never theoretically shows that the data constraint is actually satisfied, and instead relies on a necessary condition of the data constraint. This greatly undermines the theoretical contribution of the paper.\n- I find the mathematical writing to be overall quite obfuscated, which complicates assessment of the theoretical contribution.\n\nAs actionnable feedback:\n- I would recommend to be more open about which theoretical guarantees the proposed framework actually provides.\n- Could you please give intuitive explanations of the role of the isomorphism loss, its novelty and comparison to related work, and why it brings this performance gain? ", "summary_of_the_paper": "The paper proposes a theoretical framework for unsupervised representation disentanglement (in the sense of Higgins et al.) based on three constraints (group structure, data and model). It further describes a learning method for satisfying these constraints, and experimentally shows some performance gain on traditional disentanglement metrics. \n\nMy intuitive understanding of the constraints is a follows:\n- the model constraint is satisfied by construction of the agent group, whose structure is transferred from the latent space by the model.\n- the group structure constraint ensures that this transfer is an isomorphism, and thus that the agent group is actually a group.\n- the data constraint ensures that the agent group actually matches the ground truth generative factors. \n\nPlease clarify if one of my statements is incorrect.", "summary_of_the_review": "I find this paper to be overselling the importance of its theoretical contribution, and will therefore recommend rejection. I am looking forward to discussion with the authors, and will reconsider my rating if the authors respond to the two points listed as actionnable feedback in my main review. In particular I'd like to hear more about the isomorphism loss. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jqUCcSVuzEG", "reviewer": "Reviewer_AKvy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThis paper appears to be the first in bringing and realizing the concept of group-based disentanglement in unsupervised VAE. The method was rigorously motivated and backed by theoretical analysis. The experiments were relatively thorough in terms of the datasets, metrics, and comparison disentanglement VAE methods considered. The improvement of metrics, while moderate to marginal in some cases, were consistent across most metrics and datasets.\n\nThe visuals of the cyclic representation shown in section 5.2 and Fig 5 are especially interesting, as well the latent space shown in Fig 3 demonstrating the benefit of the isomorphism loss.\n\nWeakness:\n\nSome of the metrics seem to be weaker than usually reported in the comparison methods. E.g., The MIG metrics reported for dSprites seemed to be rather low for each of the \"original\" VAE methods compared to those seen in literature (e.g., Fig 3 in [1]). Please clarify, especially since the margin of improvements in many metrics were less than 1 standard deviation of the statistics.\n\nWhile it is appreciated that the presented work considers a \"unsupervised\" learning of representations in comparison to existing works that uses the interaction with the environment as supervision to adopt the group-based definition, it would be still desirable to see a comparison of performance to understand what is the price to pay to go from a supervised to unsupervised setting. This is somewhat related to the comment above regarding avoiding using weak baselines.\n\nIn the final manuscript, please adjust the figures and tables such that they 1) appear in the order they're referred to in the text and 2) are spaced as close to the text description as possible. Right now Fig 4 and Fig 5 are referred to first, before Fig 3 and 2 are discussed in the text.\n", "summary_of_the_paper": "This paper presents an unsupervised approach to achieve group-based disentangled representation learning, as opposed to existing environment-based approaches. A theoretical framework for the group-based VAE was proposed, and implementations of the group and isomorphism reported. Experiments were performed on four common benchmark datasets, with comparisons to several representative VAE-based disentanglement framework (beta-vae, annealVAE, factorVAE, beta-TCVAE), demonstrating the groupified VAE achieves better disentanglement in many metrics. Qualitative evaluations demonstrate the cyclic representation space learned by the groupified VAE.", "summary_of_the_review": "This is an overall well written manuscript describing a novel method backed by in-depth theoretical analysis. The experiments were well designed and evaluations relatively thorough. There are some questions that can be addressed and improvements that can be made to the current manuscript, but overall I consider this to be an interesting work that will have good discussion potential for the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "L8MHP_cTued", "reviewer": "Reviewer_YTd1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The authors provide a theoretical framework on the learning of group-based disentangled representation, which is a step forward to the more general guaranteed disentanglement learning.\n2. The experimental results validate the disentanglement effect of the 'Groupify' operation on multiple base models and synthetic datasets.\n3. The proposed technique is compatible with all the information-theory-based baseline models. \n\nWeaknesses:\n1. The introduction of $\\Phi$ group is unnecessarily complex. Do the authors mean the $\\Phi$ group is a direct product of $m$ cyclic groups of order n (or even simpler, $m$-dim Torus)? Is each cyclic subgroup represented by $(sin((2\\pi z))/n), cos((2\\pi z)/n))$, with z being the output of the VAE encoder? The generator of each subgroup is the action of rotating each circle by step $2\\pi/n$? I see no necessity of incorporating more concepts from Group Theory like 'congruence class modulo n', 'n-th root unity group', 'additive group of integers modulon'. To me they are all the same in the context of this paper.\n2. The mention of 'n-th dihedral group' is not informative, or even misleading. Dihedral groups are generated by only two subgroups, rotation and flip, which is not very related to the group used in this method (a product of m cyclic groups of the same order n).\n3. The Abel loss and Order loss can only encourage the representation to maintain a group structure of m-dim torus. However, this does not solve the unidentifiability problem that motivates this method (the impossibility of disentanglement mentioned in the abstract, introduction section, and the paragraph before Sec. 4.1) because the ground-truth correspondence between the group action on the world space and its action effect on the image space is still unknown. The real problem here is not how to learn a group that is isomorphic to an m-dim cyclic group, but how to make sure the learned cyclic subgroups correspond to the interpretable variations shown in the image space. This paper proposes a method to learn a cyclic-group representation, which is valuable by itself in some way, but it still falls into the unidentifiability problem proposed in Locatello et al. 2019b.\n4. I don't think $\\phi\\phi^{-1}=e$ can replace $\\phi\\phi^{n-1}=e$ in the Order loss. $\\phi\\phi^{n-1}=e$ ensures the cyclic structure of order n, but $\\phi\\phi^{-1}=e$ holds in any group with arbitrary order. The equation $\\phi^{-1}=\\phi^{n-1}$ holds when the cyclic structure has **already been learned**, but $\\phi\\phi^{-1}=e$ alone cannot help the learning of a cyclic structure.\n5. It looks like the proposed VAE model is a special case of the Commutative Lie Group VAE proposed in https://arxiv.org/abs/2106.03375 with the learned group representation being replaced by a pre-defined direct product of cyclic groups. They should be more clearly discussed and compared in the main paper.\n6. Currently only the synthetic datasets provided in the open-sourced disentanglement library (https://github.com/google-research/disentanglement_lib) are used in the experiments. More results on some real-world datasets like CelebA should be provided to validate the model's generalization ability.\n7. The paper only shows the effectiveness of the method as an incremental technique to improve the existing disentanglement models, but does not show its stand-alone performance. If the proposed constraints are effective enough, they should outperform the existing SOTA models by being directly added to a vanilla VAE (as all the baseline models do). ", "summary_of_the_paper": "This paper provides a theoretical framework on the learning of group-based disentanglement representations. It proposes a method to learn a cyclic group representation with the Abel loss and Order loss based on VAEs. Experiments validate the effectiveness of the proposed method on improving existing disentanglement VAE models.", "summary_of_the_review": "I believe the theoretical contribution of this paper is valuable to the group-based disentanglement learning (or general disentanglement learning) community, but I still have concerns listed in the main review section. I currently rate this paper as weak reject, but I will be glad to improve my score if my concerns are properly solved.\n\n======\n\nAfter rebuttal:\nThanks for the authors' response to my comments. I think a large part of my concerns has been addressed. I will increase the score to 6 weak accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GqC25fhEjkn", "reviewer": "Reviewer_PA1y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Proposed a new theoretical framework for achieving unsupervised representation disentanglement based on $n$-th dihedral group.\n2. Provided theoretical analysis for the proposed framework via model and group structure\n3. Conducted extensive experiments to verify the proposed framework\n\nCons:\n1. Missing some latest works on disentanglement. The related work section is a bit weak since it only discussed the VAE models before the year 2019. In fact, there are a lot of related works about disentangled representation learning after 2019, such as TamingVAE, ControlVAE, and other works [1,2,3,4].\n2. Please also compare the proposed framework with the baselines in recent work as mentioned above.\n\n3. In Table 1, for dSprites dataset, the MIG score is much lower than the result illustrated in the literature $\\beta$-TCVAE and ControlVAE. Please clarify and explain it. \n\n[1] Srivastava, Akash, Yamini Bansal, Yukun Ding, Cole Hurwitz, Kai Xu, Bernhard Egger, Prasanna Sattigeri, Josh Tenenbaum, David D. Cox, and Dan Gutfreund. \"Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modelling.\" arXiv preprint arXiv:2010.13187 (2020). \n\n[2] Shao, H., Yao, S., Sun, D., Zhang, A., Liu, S., Liu, D., ... & Abdelzaher, T. (2020, November). ControlVAE: Controllable variational autoencoder. In International Conference on Machine Learning (pp. 8655-8664). PMLR.\n\n[3] Kim, M., Wang, Y., Sahu, P., & Pavlovic, V. (2019). Bayes-factor-vae: Hierarchical bayesian deep auto-encoder models for factor disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2979-2987).\n\n[4] Lezama, J. (2018, September). Overcoming the disentanglement vs reconstruction trade-off via jacobian supervision. In International Conference on Learning Representations.", "summary_of_the_paper": "This paper proposed a new theoretical framework for achieving unsupervised representation disentanglement based on $n$-th dihedral group. Further, it proved three sufficient conditions on the model, group structure, and data respectively. Evaluations on multiple benchmark datasets demonstrate the proposed framework achieves better mean performance with smaller variances compared to some VAE models.", "summary_of_the_review": "This paper proposed a new theoretical framework for achieving unsupervised representation disentanglement. My concern is that this work only compares it with the baselines before 2018, so it is hard to say whether it outperforms the recent baselines. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission provides a theoretical framework on the learning of group-based disentanglement representations and proposes a novel method to learn such representations.

The reviewers appreciated the novel perspective of the paper in introducing the concept of group-based disentanglement in unsupervised VAE. Furthermore, the approach was considered to be soundly theoretically motivated and experiments to be extensive. There was a lively discussion between reviewers and authors about certain ambiguities in the manuscript; however, they seem to have been largely resolved to the reviewers' satisfaction.

While there was a reviewer with a very low confidence recommending rejection, this paper brings an indisputably interesting novel perspective to the learning of unsupervised representations and I thus recommend acceptance.
Decision: Accept (Poster)