Year: 2021

Abstract:
Disentangled representation learning is one of the major goals of deep learning, and is a key step for achieving explainable and generalizable models. A well-defined theoretical guarantee still lacks for the VAE-based unsupervised methods, which are a set of popular methods to achieve unsupervised disentanglement. The Group Theory based definition of representation disentanglement mathematically connects the data transformations to the representations using the formalism of group. In this paper, built on the group-based definition and inspired by the n-th dihedral group, we first propose a theoretical framework towards achieving unsupervised representation disentanglement. We then propose a model, based on existing VAEbased methods, to tackle the unsupervised learning problem of the framework. In the theoretical framework, we prove three sufficient conditions on model, group structure, and data respectively in an effort to achieve, in an unsupervised way, disentangled representation per group-based definition. With the first two of the conditions satisfied and a necessary condition derived for the third one, we offer additional constraints, from the perspective of the group-based definition, for the existing VAE-based models. Experimentally, we train 1800 models covering the most prominent VAE-based methods on five datasets to verify the effectiveness of our theoretical framework. Compared to the original VAE-based methods, these Groupified VAEs consistently achieve better mean performance with smaller variances.

heading: 1 INTRODUCTION
text: Learning independent and semantic representations of which individual dimension has interpretable meaning, usually referred to as disentangled representations learning, is critical for artificial intelligence research (Bengio et al., 2013). Such disentangled representations are useful for many tasks: domain adaptation (Li et al., 2019; Zou et al., 2020), zero-shot learning (Lake et al., 2017), and adversarial attacks (Alemi et al., 2016), etc. Intuitively, a disentangled representation should reflect the factors of variations behind the observed data of the world, and one latent unit is only sensitive to changes of an individual factor.
Due to the facts that obtaining the ground-truth labels requires significant human effort and humans can learn those factors unsupervisedly, unsupervised representation disentanglement draws much attention from researchers recently. A lot of methods are proposed base on some intuitions. Most of the state-of-the-art methods (Higgins et al., 2017; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017) are based on Variational Autoencoder (VAE) (Kingma & Welling, 2013). These methods are fully unsupervised and can be applied to a variety of complex datasets (Lee et al., 2020). However, these methods suffer from the unidentifiability problem (Locatello et al., 2019b) due to a lack of theoretical guarantee. Another stream of works (Chen et al., 2016; Lin et al., 2020; Khrulkov et al., 2021; Lee et al., 2020) leverage generative adversarial network (GAN) (Goodfellow et al., 2014) to achieve disentanglement but are not interpretable. In general, a well-defined theoretical guarantee is needed for those methods.
The research of symmetry in physics demonstrates that infinitesimal transformations that conform to some symmetry groups on physical objects can reflect their nature (Anderson, 1972; Noether,
1915). Recently, inspired by this research on symmetry, Higgins et al. (2018) proposed a group-based definition of disentangled representation. They argue that the symmetries, i.e., the transformations that change certain aspects of data and keep other aspects unchanged, ideally reflect the underlying data structure. The group-based definition is a formal and rigorous mathematical definition of faithful and, ideally, interpretable representation of the generative factors of data, which is widely accepted (Greff et al., 2019; Mathieu et al., 2019; Khemakhem et al., 2020). Subsequently, due to the fact that the definition is defined by the world state (i.e., Ground Truth) and based on the assumption (CasellesDupré et al., 2019) that this definition should be useful for downstream tasks such as a Reinforcement Learning, Caselles-Dupré et al. (2019), Quessard et al. (2020), Painter et al. (2020) propose environment-based (to provide world state) methods to learn such disentangled representations in Reinforcement Learning settings. These inspire us to ask the following question: how would the definition benefit unsupervised representation disentanglement, and how to learn such a disentangled representation conforming to the definition in the setting of unsupervised representation learning?
In Group Theory, the n-th dihedral group1 (Judson, 2020) is a set of all permutations of polygons vertices, forming a permutation group under the operation of composition (Miller, 1973). The generators in an n-th dihedral group, i.e., flip and rotation, can be regarded as the disentangled factors and also transformations. In this paper, inspired by the n-th dihedral group, we answer the above questions and address the challenge by proposing a theoretical framework to make the definition practically applicable for unsupervised representation disentanglement. We then propose a model to tackle the learning problem of the framework and verify its effectiveness. We theoretically prove in Section 3.2 the three sufficient conditions towards achieving disentangled representation per groupbased definition, which are referred to as model, group structure, and data constraint, respectively. With these conditions, we offer additional constraints from the perspective of the definition. The additional constraints encourage existing VAE-based models to satisfy the symmetry requirement that comes from the nature of factors. Finally, we provide a learning model based on the existing VAE-based methods in an effort to fulfill the three conditions (with the model and group structure constraint and a necessary condition for the data constraint satisfied). As an intuitive understanding, we introduce the additional constraints to reorganize the latent space to restrict its symmetry in an unsupervised way. These additional constraints indeed narrow down the solution space of VAEbased models. Detailed discussion in Sec. 5.4. Our model consistently achieves statistically better performance in prominent metrics (higher means and lower variances) than corresponding existing VAE-based models on five datasets, demonstrating that the group-based definition together with our proposed framework further encourages disentanglement.
Our main contributions are summarized as follows:
• To our best knowledge, we are the first to provide a theoretical framework to make the formal group-based mathematical definition of disentanglement practically applicable to unsupervised representation disentanglement.
• Our theoretical framework provides additional constraints from the perspective of groupbased definition for the existing VAE-based methods.
• We propose a learning model of the framework by deriving and integrating additional loss into existing VAE-based models, in an effort to make the learned representation conform to the group-based definition without relying on the environment (as done in Caselles-Dupré et al. (2019); Quessard et al. (2020); Painter et al. (2020)).

heading: 2 RELATED WORKS
text: Different definitions have been proposed for disentangled representation (Bengio et al., 2013; Higgins et al., 2018; Suter et al., 2019). However, only the group-based definition proposed by Higgins et al. (2018) focuses on the disentangled representation itself and is mathematically rigorous, which is well accepted (Caselles-Dupré et al., 2019; Quessard et al., 2020; Painter et al., 2020; Diane Bouchacourt, 2021). Nevertheless, Higgins et al. (2018) do not propose a specific learning method based on their definition. Before this rigorous definition was proposed, there had been some success in identifying generative factors in static datasets (without interaction with environment), such as β-VAE (Higgins
1We assume some basic familiarity with the fundamentals of Group Theory and Group Representation Theory. Please refer to Appendix A for some basic concepts.
et al., 2017), Anneal-VAE (Burgess et al., 2018), β-TCVAE (Chen et al., 2018), and FactorVAE (Kim & Mnih, 2018). More recent works (Srivastava et al., 2020; Shao et al., 2020; Kim et al., 2019; Lezama, 2018; Rezende & Viola, 2018) also do not consider the group-based definition. Therefore, how group-based definition will facilitate these methods is still an open question. Besides, all these works suffer from the unidentifiability problem (Locatello et al., 2019b), which is a challenging problem in this literature. From group-based definition, our framework points out that, the unidentifiability problem could be solved once the data constraint is satisfied. However, in this work, we can only get a necessary condition for data constraint, and we still can not solve this challenging problem.
As pointed out in Quessard et al. (2020), it is not straightforward to reconcile the probabilistic inference methods with the group-based definition framework. Caselles-Dupré et al. (2019), Quessard et al. (2020), Painter et al. (2020) leverage the interaction with the environment (assuming it is available) as supervision instead of minimizing the total correlation as the VAE-based methods do. Consequently, the effectiveness of these methods is limited to the datasets with the environment available. Our framework learns a representation conforming to the group-based definition without relying on the environment. Pfau et al. (2020) propose a non-parametric method to unsupervisedly learn linear disentangled planes in data manifold under a metric. However, as pointed out by the authors, the method does not generalize to held-out data and performs poorly when trying to disentangle directly from pixels.
To summarize, the existing probabilistic inference methods lack theoretical support, while the application scope of existing methods based on the group-based mathematical definition Higgins et al. (2018) is very limited. To the best of our knowledge, our work is the first to reconcile the probabilistic generative methods with the inherently deterministic group-based definition framework of Higgins et al. (2018).

heading: 3 THE GROUP-BASED FRAMEWORK FOR UNSUPERVISED REPRESENTATION DISENTANGLEMENT
text: Our goal is to explore the benefit of the group-based definition for unsupervised representation disentanglement and learn such a disentangled representation. The background of the group-based definition is provided in Section 3.1. Section 3.2 presents the theoretical framework towards achieving unsupervised disentanglement, in which we derive three sufficient conditions on the model, group structure, and data, respectively. The conditions on the model and group structure provide additional constraints for the existing VAE-based models.

heading: 3.1 GROUP-BASED DEFINITION
text: We briefly review the group-based definition of disentangled representation Higgins et al. (2018). Considering a group G acting on world state space W (can be understood as ground-truth) of data space O and representation space Z via group action ·W and group action ·Z respectively. For a mapping f = b ◦ h, where b and h denote the data generative process and encoding, we state: the mapping f is equivariant between the actions on W and Z if
g · f(w) = f(g · w), ∀g ∈ G, ∀w ∈W. (1)
Definition 1 Assume G can be decomposed as G = G1×G2× · · ·×Gm. The set Z is disentangled with respect to G if: (i) group action of G on Z exits. (ii) the mapping f is equivariant between the actions on W and Z. (iii) There is a decomposition Z = Z1 × Z2 × · · · × Zm such that each Zi is affected only by the corresponding Gi.
It is challenging to apply the group-based definition to an unsupervised disentanglement setting in practice because the definition refers to the world state space W , the group action of G on W , and mapping b which are typically inaccessible in practice. We tackle the challenge by re-framing the definition in a new framework in the following section.

heading: 3.2 PROPOSED THEORETICAL FRAMEWORK
text: Since when the representation is disentangled, one latent unit in the representation space is only sensitive to changes of an individual generative factor, we make the following assumptions: G
is a direct product of m cyclic groups (as suggested by Higgins et al. (2018) and for simplicity): G = (Z/nZ)m = Z/nZ× Z/nZ× · · · × Z/nZ, where n is the assumed total number of possible values for a factor and m is the total number of factors; we further assume Z is a set with the same elements in G. Therefore, the group actions of G on Z can be set to be element-wise addition, i.e., g · z = g + z,∀z ∈ Z, g ∈ G. For the generator of dimension i of G, gi = (0, . . . , 1, . . . , 0), gi only affects the i-th dimension of z by gi + z. In addition, the action of each generator gi on w only affects a single dimension of w.
As we can seen from Equation 1 above, the group action is defined on w, which is often not accessible, making it difficult to apply the definition in practice. Therefore, for the unsupervised setting, we would like to use permutations on the data space O (which only provides data without labels) to substitute the group actions on W . Specifically, inspired by the n-th dihedral group (Dummit & Foote, 1991), we construct a permutation group Φ, serving the role of an “agent” of G. The actions of G on W can be performed by ϕg ∈ Φ on O, which can be formulated as
f(g · w) = h(ϕg · b(w)) = h(ϕg · o),∀w ∈W, g ∈ G, (2)
where o denotes the data (e.g., image) corresponding to the world state w through the mapping function b. If the above equation holds, we state that the “agent” permutation group Φ exists. We first give the conditions for the existence of this “agent” permutation group Φ, then derive the additional condition to achieve such disentanglement. We accomplish these two objectives in Theorem 1 with the proof provided in Appendix B. Theorem 1 states that a general permutation group Φ on O can serve as an agent group (agent group exists) if and only if both (i) and (ii) are satisfied. If the agent group exists, and its permutations (actions on O) can be defined by an autoencoder-like model as shown in the equation in (iii), then Z is disentangled with respect to G.
Theorem 1 For the group G = (Z/nZ)m, a permutation group Φ on O, a representation space Z, a World State space W , and mapping b and h, Equation 2 holds if and only if (i) Φ is isomorphic to G, and (ii) For each generator of dimension i of G, gi, there exists a ϕi ∈ Φ, i = 1, . . . ,m, such that ϕi · b(w) = b(gi · w), ∀w ∈ W , and ϕi is a generator of Φ; Further, if Equation 2 holds and (iii) ϕg · b(w) = h−1(g · f(w)) ∀w ∈W,ϕg ∈ Φ, then Z is disentangled with respect to G, where ϕg is the corresponding element in Φ of g under the isomorphism.
In Theorem 1, (i) states that the relation between the elements (i.e., group structure) is preserved between Φ and G, and we denote it as the group structure constraint; (ii) actually indicates a data constraint that all variations in the data can be generated by compositions of some basic permutation generators {ϕi}i=1,...,m. We denote it as the data constraint; (iii) states that the permutations in the agent group Φ are defined by encoding, action, and decoding, which is referred to as the model constraint. Note that in Theorem 1, only the data constraint refers to the world state w.
Here is a sketch of the proof: data constraint is a special case of Equation 2 for a generator, and group structure constraint is a relation-preserving constraint on compositions of generators, and satisfying both constraints will thus result in that Equation 2 holds for any general element in Φ, and vice versa. Moreover, we can derive Equation 1 for disentanglement when combining the model constraint and Equation 2.
The model constraint specifies the way to permute the data. When the data is permuted, its world state changes. Therefore, how the world states transit between each other is modeled by the model constraint applied on the data. The isomorphism between Φ and G ensures that the world state space W and data space O have the same symmetry. In this way, the model applied on the data learns the transition of the world states. Note that we aim to bring this group-based definition, which requires ground truth by default, into the unsupervised setting. Now only the data constraint refers to the world states, and it seems almost impossible to derive a sufficient condition for it without the labels. We thus make a trade-off in which we use a necessary condition in the next section.

heading: 4 GROUPIFIED VAE: A LEARNING METHOD OF THE FRAMEWORK
text: Let’s look closer into the three constraints, respectively. Firstly, we consider the model constraint, ϕg · o = h−1(g · h(o)) ∀o ∈ O,ϕg ∈ Φ, which suggests that the action of Φ on O can be implemented using an autoencoder-like network that performs encoding, action on its representation space, and decoding. Given an autoencoder-like network with an encoder h and a decoder d, since d
is approximately the inversion of h, the model constraint can be formulated as
ϕg · o = h−1(g · h(o)) =∆ d(g · h(o)),∀o ∈ O, g ∈ G, (3)
together with further implementation of Φ as described in Section 4.1, the model constraint can be fulfilled. Secondly, The data constraint requires that all variations in the data can be generated by compositions of some basic permutations generators. Previous VAE-based works (Higgins et al., 2017; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017) aim to generate the data with independent generative factors, which is in line with the data constraint. Intuitively, if the VAE-based model can generate the data from statistical independent basic latent units and each unit corresponds to the basic permutation generator, the data constraint may be satisfied. Based on the intuition above, we prove that if the world state is independently sampled per dimension, the minimization of total correlation is a necessary condition for the data constraint (see Appendix E). Therefore, we can leverage existing VAE-based models to fulfill the data constraint to some extent for the unsupervised setting. Lastly, to satisfy the group structure constraint, we derive a self-supervised Isomorphism Loss which can be incorporated into the VAE-based model as described in Section 4.2.

heading: 4.1 IMPLEMENTATION OF GROUP Φ
text: The key is to implement the group actions of G on Z into the VAE-based models, we need to map the representation z to a group that is isomorphic to G (cyclic representation space). Therefore, we construct a function η to achieve this mapping. Moreover, this mapping is required to be differentiable, in order for back-propagation to be adopted for optimization. According to Group Theory, there is an isomorphism between G and the n-th root unity group: {exp((2πiz)/n)|z ∈ Zm}, where n,m are the same as in G. Therefore, the representation z can be mapped to z by the function η as z = η(z) = exp((2πiz)/n) (see Figure 1 (a)). However, z can not be mapped to directly as it has complex numbers, but we can use Euler’s formula: exp((2πiz)/n) = sin((2πz)/n)+i cos((2πz)/n) to map z to its real and imaginary part, i.e., vector sin((2πz)/n) and cos((2πz)/n). The two vectors are concatenated and fed to the decoder.
For ease of implementation, the permutation group Φ can be approximately generated by compositions of generators, i.e., Φ =< ϕ1, ϕ2, . . . , ϕm >. Recall that the generator ϕi of group Φ is defined as ϕi · o = d(gi · h(o)) = d(h(o) + gi),∀o ∈ O, where gi is generator of dimension i in G, as shown in Figure 1 (a). For ϕi, we implement gi · h(o) by adding 1 (without loss of generality) to the i-th dimension of h(o), then make it cyclic by function η. Similarly, for ϕ−1i , we add the value of n− 1.

heading: 4.2 IMPLEMENTATION OF THE ISOMORPHISM
text: In this section, to satisfy the group structure constraint (isomorphism), we derive two equivalent constraints, which are then converted into an Isomorphism Loss LI . Many groups are uniquely determined by the properties of the generators, e.g., group G =< a, b|a2 = b2 = e, ab = ba >. In addition, since the group Φ is isomorphic to G, Φ is also expected to be commutative and cyclic. In light of this, we derive two constraints on generators that are equivalent to the isomorphism condition, as described in Theorem 2. Please refer to Appendix C for the proof.
Theorem 2 The defined permutation group Φ =< ϕ1, ϕ2, . . . , ϕm > is isomorphic to G = (Z/nZ)m if and only if: (i) for ∀ generators ϕi, ϕj ∈ Φ, 1 ≤ i, j ≤ m, we have ϕiϕj = ϕjϕi, and (ii) ∀ϕi ∈ Φ, 1 ≤ i ≤ m, we have ϕni = e, where e is the identity element of group Φ.
The first constraint requires the group Φ to be an abelian group (Judson, 2020). Therefore, we denote it as Abel constraint and the loss derived from it as the Abel Loss La. The second is a constraint on the order of elements. We thus denote it as the Order constraint and the loss derived from it as the Order Loss Lo. See Appendix F for a more detailed implementation. Abel Loss. For the Abel constraint: ∀ϕi, ϕj ∈ Φ, 1 ≤ i, j ≤ m, we have ϕiϕj = ϕjϕi. We minimize ‖ϕi · (ϕj · o)− ϕj · (ϕi · o)‖,∀o ∈ O to meet the Abel constraint, as shown in Figure 1 (b). The Abel Loss is the sum of the losses of all combinations of two generators. Denote the set of combinations as C = {(i, j)|1 ≤ i, j ≤ m}. The Abel Loss is defined as follows
La = ∑ o∈O ∑ (i,j)∈C ‖ϕi · (ϕj · o)− ϕj · (ϕi · o)‖. (4)
Order Loss. For the Order constraint: ∀ϕi ∈ Φ, 1 ≤ i ≤ m, we have ϕni = e, where e is the identity element in group Φ (identity mapping). Note that with n times composition of ϕi, it is difficult for the gradient to back-propagate. We thus use an approximation that uses 2 times of composition instead. When the autoencoder can do the reconstruction well, this approximation holds, see appendix E for details. Similar to Abel Loss, we minimize ‖ϕi · (ϕn−1i · o) − o‖,∀o ∈ O to satisfy the Order constraint. The whole process is illustrated in Figure 1 (b). However, the equation is not symmetrical and leads to bias. Therefore, we use the following symmetrical form instead:
Lo = ∑ o∈O ∑ 1≤i≤m ( ‖ϕi · (ϕn−1i · o)− o‖+ ‖ϕ n−1 i · (ϕi · o)− o‖ ) . (5)
With the above two loss functions optimized, the isomorphism condition is satisfied, which can be illustrated by Theorem 3. Please refer to Appendix D for the proof.
Theorem 3 The following two conditions are equivalent: (i) ∀ϕi, ϕj ∈ Φ, 1 ≤ i, j ≤ m, we have ϕiϕj = ϕjϕi and ∀ϕi ∈ Φ, 1 ≤ i ≤ m, we have ϕni = e (ii) the Abel Loss function (Equation 4) and the Order Loss function (Equation 5) are optimized.
Since the Abel Loss and Order Loss are equally important for satisfying the isomorphism condition, we assign equal weight to them. Thus, the Isomorphism Loss is LI = Lo + La. With the implementation of group Φ, the model constraint is satisfied. We optimize the Isomorphism Loss to satisfy the group structure constraint. To further satisfy the data constraint to some extent as described in Section 4, we leverage VAE-based models and optimize their original loss (that minimizes the total correlation), denoted as LV AE . Therefore, the Total Loss is L = LV AE + γILI , where γI is the weight of Isomorphism Loss. We denote the above VAE-based implementation as Groupified VAE.

heading: 5 EXPERIMENTS
text: We first verify the effectiveness of Groupified VAE quantitatively in learning disentangled representations on several datasets and several VAE-based models. Then, we show its effectiveness qualitatively on two typical datasets. After that, we perform a case study on the dSprites dataset to analyze the effectiveness, and conduct ablation studies on the losses and hyperparameters. For the performance comparison of two downstream tasks (abstract reasoning Van Steenkiste et al. (2019) and fairness evaluation Locatello et al. (2019a)), and more comprehensive results, please see Appendix I.

heading: 5.1 DATASETS AND BASELINE METHODS
text: To evaluate our method, we consider several datasets: dSprites (Higgins et al., 2017), Shapes3D (Kim & Mnih, 2018), Cars3D (Reed et al., 2015), and the variants of dSprites introduced by Locatello et al. (Locatello et al., 2019b): Color-dSprites and Noisy-dSprites. Please refer to Appendix G for the details of the datasets.
We choose the following four baseline methods as representatives of the existing VAE-based models, which are denoted as Original VAEs. We verify the effectiveness of our implementation based on those methods. β-VAE (Higgins et al., 2017) introduces a hyperparameter β in front of the KL regularizer of the VAE loss. It constrains the VAE information capacity to learn the most efficient representation. AnnealVAE (Burgess et al., 2018) progressively increases the bottleneck capacity so that the encoder learns new factors of variation while retaining disentanglement in previously learned factors. FactorVAE (Burgess et al., 2018) and β-TCVAE (Chen et al., 2018) both penalize the total correlation (Watanabe, 1960), but estimate it with adversarial training (Nguyen et al., 2010; Sugiyama et al., 2012) and Monte-Carlo estimator respectively.

heading: 5.2 QUANTITATIVE EVALUATIONS
text: This section performs quantitative evaluations on the datasets and models introduced with different random seeds and different hyperparameters. Then, we evaluate the performance of the Original and Groupified VAEs in terms of several popular metrics: BetaVAE score (Higgins et al., 2017), DCI disentanglement Eastwood & Williams (2018) (DCI in short), MIG (Chen et al., 2018), and FactorVAE score (Kim & Mnih, 2018). We assign three or four hyperparameter settings for each model on each dataset. We run it with ten random seeds for each hyperparameter setting to minimize the influence of random seeds. Therefore, we totally run ((3× 10× 3 + 10× 3× 3)× 2)× 5 = 1800 models. We evaluate each metric’s mean and variance for each model on each dataset to demonstrate the effectiveness of our method. As shown in Table 1, these Groupified VAEs have better performance (numbers marked bold in Table 1) than the Original VAEs on almost all the cases.
On Shapes3d, the Groupified VAEs outperform the Original ones on all the metrics except for BetaVAE scores, suggesting some disagreement between BetaVAE scores and other metrics. Similar disagreement is also observed between the variances of MIG and other metrics on Cars3d. Note that the qualitative evaluation in Appendix J shows that the disentanglement ability of Groupified VAEs is better on Shapes3d and Cars3d.

heading: 5.3 QUALITATIVE EVALUATIONS
text: We qualitatively show the Groupified VAEs achieve better disentanglement than the Original ones. As shown in Figure 2, the traversal results of Groupified β-TCVAE on Shape3d and Car3d are less entangled. For more qualitative evaluation, please refer to Appendix J. To verify that the Groupified VAEs learn a cyclic representation space (where n = 10), we provide the traversal results of [0,18] with a step of 2 for both the Groupified and Original β-TCVAE on Shape3d in Figure 3. We observe that the traversal results of Groupified VAEs are of high quality with a period of 10 (equal to n). However, the Original VAEs generate low-quality images without cyclicity. For the comparison of the results on CelebA (real-world datasets), please see appendix J.

heading: 5.4 VISUALIZATION OF THE LEARNED REPRESENTATION SPACE
text: To understand how our theoretical framework helps the existing VAE-based models to improve the disentanglement ability, we take dSprites as an example, visualize the learned representation space, and show the typical score distributions of the metrics. First, we visualize the space spanned by the three most dominant factors (x position, y position, and scale).
As shown in Figure 5 (for more results, please refers to Appendix L), the spaces learned by the Original VAEs collapse, while the spaces of the Groupified VAEs only bend a little bit. The main reason is that the Isomorphism Loss, serving as a self-supervision signal, suppresses the representation space distortion and encourages the disentanglement of the learned factors. As Figure 4 shows, the Groupified VAEs consistently achieve better mean performance with smaller variances. The isomorphism reduces the search space of the network so that the Groupified VAEs converge to the ideal disentanglement solution.

heading: 5.5 ABLATION STUDY
text: We perform an ablation study on the assumed total number of possible values for a factor (factor size) n, Abel Loss La, and Order Loss Lo. We take the AnnealVAE trained on dSprites as an example. We only consider the DCI disentanglement metric here. We investigate the influence of factor size n. Besides, to evaluate the effectiveness of the two constraints, the models with the Abel Loss alone or Order Loss alone added are also evaluated. In this setting, we fix n to 10. We compute the mean and variance of the performance for 30 settings of hyperparameters and random seeds. Table 2 shows that the isomorphism plays a role of cycle consistency in the representation space, leading to better disentanglement. The performance is robust to the factor size n, as the models learn to adapt to different n in the training process. The models with only the Abel Loss or Order Loss applied have improved performance compared to the originals. The former (Abel Loss) performs better than the latter, suggesting that commutativity plays a more important role. Note that the number of factors m can be learned and is not a hyperparameter. See Appendix F for details. γI is empirically set to 1.

heading: 6 CONCLUSION
text: In this paper, we have opened the possibility of applying group-based definition to unsupervised disentanglement by proposing a theoretical framework. The group structure and model constraint in the framework are effective for existing VAE-based unsupervised disentanglement methods. In addition, by establishing the feasibility of learning the representation conforming to the definition in unsupervised settings, we have exhibited the consistently better mean performance with lower variance attributed to the definition. We believe our work constitutes a promising step towards unsupervised disentanglement with theoretical guarantee. As to the limitation, we only provide a necessary condition for the data constraint, as a result, we can not address the unidentifiability problem. Tackling the unidentifiability problem with the group-based definition is beyond the scope of this work, we will leave it as future work. In addition, a natural extension of our framework is to use lie group Hall (2015) (which is also a manifold) to extend our framework.

heading: A PRELIMINARIES
text: Group: A set G together with a binary operation ◦ as: ◦ : G × G → G satisfying the following properties:
• Associativity: ∀ a, b, c ∈ G, s.t.(a ◦ b) ◦ c = a ◦ (b ◦ c). • Identity: ∃ e ∈ G, s.t. ∀ a ∈ G, e ◦ a = a ◦ e = a. • Inverse: ∀ a ∈ G,∃ a−1 ∈ G : a ◦ a−1 = a−1 ◦ a = e.
It is customary to represent a group with a set G and the binary operation ◦ as a pair (G, ◦). When the binary operation is clear, we represent group (G, ◦) as G and use multiplication to represent the binary operation ◦, i.e., a ◦ b = ab, ∀ a, b ∈ G. Group Action: Let (G, ◦) be a group and P be a set. By the group actions of (G, ◦) on P , we mean a mapping:
·P : G× P → P, (6) such that
• ∀a, b ∈ G, p ∈ P, (a ◦ b) · p = a · (b · p). • e · p = p, where e is the identity element of G.
Symmetry Group and Permutation Group: Let Σ be a nonempty set, the bijections from Σ to itself are called Permutations. S(Σ) denotes the set containing all the permutations on Σ. S(Σ) forms a group under the binary operation: composition of functions, which is called Symmetry Group. A subgroup of S(Σ) is called Permutation Group.
Abelian Group: If the commutative law (∀ a, b ∈ G, a ◦ b = b ◦ a) holds in a group (G, ◦), such a group is called an abelian group.
Subgroup: If a subset H of a group G is itself a group under the operation of G, we say that H is a subgroup of G.
Equivalence Relation: An equivalence relation on a set B is a subset U ⊂ B ×B satisfying: (It is customary to represent (a, b) ∈ U as a ∼ b)
• Reflexive: ∀ a ∈ B, (a, a) ∈ U . • Symmetric: (a, b) ∈ U ⇐⇒ (b, a) ∈ U . • Transitive: (a, b) ∈ U and (b, c) ∈ U ⇒ (a, c) ∈ U .
Equivalence Class: Let ∼ be an equivalence relation on a set B. We take an a ∈ B, and then the equivalence class containing a is the subset a =∆ {b ∈ C|b ∼ a} ⊂ B. Homomorphism Let (G, ·) and (H, ◦) be two groups. A homomorphism f , from G to H , is a mapping f : G→ H , such that f(x · y) = f(x) ◦ f(y), ∀x, y ∈ G. Isomorphism: A homomorphism f : G → H which is bijective is called an isomorphism. Two groups are said to be isomorphic if there exists an isomorphism between them.
Related Groups: (i) Additive group of integers modulo n: Let n denote a positive integer. We define an equivalence relation on Z as a ∼ b⇔ a = b (mod n) (a and b have the same remainder modulo n). The relation divides Z into n equivalent classes: 0, 1, . . . , n− 1, where i represents the equivalent class containing i, i.e., i = {m ∈ Z|m = i (mod n)}. Let Zn = {0, 1, . . . , n− 1}, and Zn forms a group under the binary operation: a+ b = a+ b, denoted by Z/nZ. (ii) Group of n-th root unity: Let n denote a positive integer and Cn = {e 2πia n |0 ≤ a ≤ n− 1, a ∈ Z} be n roots of xn = 1, and then Cn forms a group under complex multiplication. For the mapping: f : Cn → (Zn,+) defined by f(e 2πia n ) = a, we have
f ( e 2πia n · e 2πibn ) = f ( e 2πi(a+b) n ) = a+ b = a+ b = f ( e 2πia n ) · f ( e 2πib n ) (7)
Therefore, f is a homomorphism, and thus is an isomorphism.
Congruence Class: Let n denote a positive integer, and we define an equivalence relation on Z as a ∼ b ⇔ a = b (mod n). The relation divides Z into n equivalent classes: 0, 1, . . . , n− 1, which are congruence classes and the elements of the additive group of integers modulo n.
Subgroup Generated by B: If B is a nonempty subset of the group (G, ◦), the set defined by < B >= {a ∈ G|a = a1 ◦ a2 ◦ · · · ◦ an with either ai ∈ B or a−1i ∈ B} forms a subgroup of G, which is called the subgroup generated by B, e.g., G =< a, b|a2 = e, ab = ba > in the toy example of Section 1. Here a1, . . . an are called Generators, e.g., a, b in G.
Cyclic Group A group G is called cyclic if there is an element a ∈ G such that G = {an|n ∈ Z} =< a >, e.g., Z/nZ. Such an element a is called a generator of G.
Symmetry: A symmetry of a geometric figure is a rearrangement of the figure preserving the arrangement of its sides and vertices as well as its distances and angles.
Dihedral Group: The permutation group formed by the symmetries of a regular n-sided polygon, denoted by Dn.
D4 as an example: The vertices of a square are numbered by {1, 2, 3, 4}, which is analogous to an image dataset, the symmetries are analogous to image transformations. We often abbreviate permutation 1→ 2, 2→ 3, 3→ 4, 4→ 1 as (1234). The elements of group D4 are shown in Figure 6, from which we know that all of the transformations are compounded by two basic permutations: horizontal flip f and rotate 90 degrees clockwise r. Please note that f and r are analogous to disentangled factors. What’s more, the Group can be generated by these basic permutations f, r with some properties, i.e., D4 =< f, r|f2 = 1, r4 = 1, fr = r−1f >. The constraints f2 = 1, r4 = 1, fr = r−1f are analogous to the group constraints in this paper.

heading: B PROOF FOR THEOREM 1
text: In our setting, the group-based definition is equivalent to the equivariant condition g ·f(w) = f(g ·w) for the following reason. Since the group actions of G on Z is the element wise addition, (a) the group action of G on Z exists and (b) group actions of Gi only affect Zi (refer to Section 3.2). These two conditions of the group-based definition thus hold.
Proof. In the following, we prove in Step 1 that the necessary and sufficient conditions of Φ playing the “agent” role , i.e., f(g · w) = h(ϕ · b(w)), are (i) Φ is isomorphic to G, and (ii) there exist ϕi ∈ Φ, s.t. ϕi · b(w) = b(gi · w), i = 1, 2, . . . ,m. We then prove in Step 2 that if equation f(g · w) = h(ϕ · b(w)) and the definition of ϕ ∈ Φ hold, then Z is disentangled per group-based definition.
Step 1: In the following, we prove that the necessary and sufficient conditions of f(g·w) = h(ϕ·b(w)) are (i) and (ii) above. Note that this holds even without a specific definition of Φ.
⇒) For a general permutation group Φ on O, we assume there exist ϕi such that ϕi · b(w) = b(gi · w), i = 1, 2, . . . ,m. Assume that there exists an isomorphism τ : G → Φ, where group G = (Z/nZ)m. From Group Theory, we know that there exists µ ∈ Aut(Φ) such that the following equation holds, where Aut(Φ) denotes the group of automorphisms of Φ, i.e., µ is an isomorphism µ : Φ→ Φ. We denote the composition of µ, τ as σ, and have
σ(gi) = µ(τ(gi)) = ϕi, i = 1, 2, . . . ,m. (8)
Since we have ϕi · b(w) = b(gi · w), i = 1, 2, . . . ,m, the following equation holds,
σ(gi) · b(w) = ϕi · b(w) = b(gi · w), i = 1, 2, . . . ,m. (9)
Since both µ, τ are isomorphism, σ = µ ◦ τ is also an isomorphism, which indicates that ϕi is a generator of Φ. For ∀ϕ ∈ Φ, since σ is a bijection, there exists g ∈ G = (Z/nZ)m, such that σ(g) = ϕ and g = ∑ i kigi, ki ∈ Z, where gi, i = 1, 2, . . . ,m denotes the generators in G. In order to calculate ϕ · b(w), we consider a specific example σ(g1 + g2).
σ(g1+g2)·b(w) = σ(g1)·(σ(g2)·b(w)) = σ(g1)·b(g2 ·w) = b(g1 ·(g2 ·w)) = b((g1+g2)·w). (10)
Therefore, for the general element ϕ, we have ϕ · b(w) = σ( ∑ i kigi) · b(w) = b( ∑ i kigi · w) = b(g · w). (11)
In general, we have ϕ · b(w) = b(g · w) and thus we have h ◦ b(g · w) = f(g · w) = h(ϕ · b(w)) when taking h on both sides.
⇐) Here we prove that (i)G is isomorphic to Φ and (ii) there exists ϕi such that ϕi · b(w) = b(gi ·w), i = 1, 2, . . . ,m are two necessary conditions of equation f(g ·w) = h(ϕ ·b(w)) = h(ϕ ·x). We take f−1 on both sides of the equation. For convenience, we rewrite g · w,ϕ · b(w) as g(w) and ϕ(b(w)), then we have
g(w) = f−1 ◦ h ◦ ϕ ◦ b(w) = b−1 ◦ ϕ ◦ b(w). (12)
Note that the notation ◦ here denotes the composition of functions. We define the mapping τ between Φ and G as follows: τ(ϕ) = g = b−1 ◦ ϕ ◦ b. (13) Note that b is a bijection, thus τ is a bijection. We take ϕi, ϕj ∈ Φ and we have
τ(ϕi ◦ ϕj) = b−1 ◦ ϕi ◦ ϕj ◦ b = (b−1 ◦ ϕi ◦ b) ◦ (b−1 ◦ ϕj ◦ b) = τ(ϕi) ◦ τ(ϕj).
(14)
Therefore, τ is a homomorphism and thus is an isomorphism. i.e., Φ is isomorphic to G. Recall that group G has the form of G = (Z/nZ)m, where it is a direct product of m cyclic groups. For ∀ϕ ∈ Φ, we have ϕ · b(w) = h−1 ◦ f(g(w)) = b(g(w)). (15) For generators gi ∈ G, i = 1, 2, . . . ,m and the corresponding ϕi = τ−1(gi), we have a specific one derived from the above equation:
ϕi · b(w) = b(gi(w)) = b(gi · w). (16)

heading: Q.E.D.
text: Step 2: As discussed in Section 1 and Section 3, we define ϕ as ϕ · x = ϕ · b(w) = h−1(σ−1(ϕ) · h(x)) = h−1(g · f(w)), where σ is the same as in Step 1. Since σ is a bijection, for ∀ϕ ∈ Φ, σ−1(ϕ) uniquely exists, and thus ϕ is well-defined. We bring it into f(g · w) = h(ϕ · b(w)), and derive f(g ·w) = g ·f(w), i.e., the representation space Z is disentangled with respect toG. Specifically, for the existing ϕi, i = 1, 2, . . . ,m, from equation 8, we have σ−1(ϕi) = gi and ϕi ·x = h−1(gi ·f(w)). Q.E.D.

heading: C PROOF FOR THEOREM 2
text: In order to easily distinguish a general element and a generator, we use φ to stand for the general element in Φ and ϕi to stand for generators in Φ. In addition, for mapping τ : G → Φ, notation ker(τ) is generally the inverse image of e, i.e., ker(τ) = {g ∈ G|τ(g) = e}, where e is the identity element in Φ.
Proof. Note that Section 3.4 describes the implementation of the elements in Φ, but there is no guarantee that Φ is a group. Therefore, here we first prove in Step 1 that set Φ =< ϕ1, ϕ2, . . . , ϕm|ϕiϕj = ϕjϕi, ϕ n i = e, i = 1, 2, . . . ,m and j = 1, 2, . . . ,m > forms a group under the composition of functions, where e satisfies that eϕi = ϕi, then prove in Step 2 the necessary and sufficient condition for the isomorphism is to meet both the Abel constraint and the Order constraint.
Step 1: To prove a set forms a group under some operation, we only need to verify that the elements of the set satisfy the following three properties: 1. Associativity 2. Identity 3. Inverse.
1. We first verify the Associativity property: ∀φs, φt, ϕl ∈ Φ, s.t. (φsφt)φl = φs(φtφl). Recall that in Section 3.2, the representation space is a set with the same elements in G. The group action of G on Z is the addition. We have g · z ∈ Z,∀g ∈ G, z ∈ Z, and then we have d(g · z) ∈ O,∀g ∈ G, z ∈ Z. Therefore, ϕi is a permutation: ϕi : O → O, i.e., ϕi ∈ S(O), where S(O) is the symmetry group on images set O, then for ∀ϕi, ϕj , ϕk ∈ {ϕ1, ϕ2, . . . ϕm}, we have
(ϕiϕj)ϕk = ϕi(ϕjϕk), (17)
the generators of Φ thus satisfy the Associativity property. However, whether the general elements of Φ satisfy the Associativity property is unknown. We take ∀φs, φt, φl ∈ Φ, where φt = ∏ i ϕ ti i , φs =∏
i ϕ si i , φl = ∏ i ϕ li i and ti, si, li ∈ Z, and we have
(φtφs)φl = (∏ i ϕtii ∏ i ϕsii )∏ i ϕlii = ∏ i ϕtii (∏ i ϕsii ∏ i ϕlii ) = φt(φsφl). (18)
Therefore, the set of mappings Φ satisfies the Associativity property under the composition of functions.
2. We then verify the Identity property: ∃ e ∈ Φ, s.t. ∀φ ∈ Φ, eφ = φe = φ. For generators ϕi, ϕj ∈ {ϕ1, ϕ2, . . . ϕm} , we have ϕni ϕj = eϕj = ϕj . Therefore, for general elements, we take ∀φ ∈ Φ, φ = ∏ i ϕ ki i , and we have:
ϕni φ = e ∏ i ϕkii = (eϕj)ϕ kj−1 j ∏ i{i 6=j} ϕkii = ϕ kj j ∏ i{i6=j} ϕkii = ∏ i ϕkii = φ. (19)
This states that the identity element of Φ is e = ϕni ∈ Φ. Therefore, the set of mappings Φ satisfies the Identity property.
3. We finally verify the Inverse property: ∀φ ∈ Φ,∃φ−1 ∈ G : φφ−1 = φ−1φ = e.
For the generators ϕi ∈ {ϕ1, ϕ2, . . . ϕm}, we have ϕki ϕ n−k i = ϕ n i = e, 1 ≤ k ≤ n, k ∈ Z. For any general element φ ∈ Φ, φ = ∏ i ϕ ki i , we have:
φ ∏ i ϕn−kii = ∏ i{i 6=j} ϕkii (ϕ kj j ϕ n−kj j ) ∏ i{i 6=j} ϕn−kii = · · · = e, (20)
we denote ∏ i ϕ n−ki i as φ
−1 and have φφ−1 = e. Similarly, we also have φ−1φ = e. These two equations state that for any general element φ ∈ Φ, φ = ∏ i ϕ ki i , we have an inverse element
φ−1 = ∏ i ϕ n−ki i in Φ. Therefore, the set of mappings Φ satisfies the Inverse property.
To summarize, Φ is a group.

heading: Q.E.D.
text: Step 2: In this step, we prove that the necessary and sufficient condition for the isomorphism (Z/nZ)m ∼< ϕ1, ϕ2, . . . , ϕm > is the satisfaction of ϕiϕj = ϕjϕi, i = 1, 2, . . . ,m and j = 1, 2, . . . ,m, and ϕni = e, i = 1, 2, . . . ,m.
⇒) Assume the isomorphism is τ : G = (Z/nZ)m → Φ =< ϕ1, ϕ2, . . . , ϕm >, for generators ϕi, ϕj ∈ {ϕ1, ϕ2, . . . ϕm}, there exist gα = ∑ i αigi, gβ = ∑ i βigi ∈ G, s.t. τ(gα) = ϕi, τ(gβ) = ϕj , where gi, i = 1, 2, . . . ,m denotes the generators in G, which is the m-dimensional one-hot vector (0, . . . , 1, . . . , 0) ∈ G with 1 in position i and 0 elsewhere, and αi, βi ∈ Z. Since G is an Abelian group, we have
ϕiϕj = τ(gα)τ(gβ) = τ(gαgβ) = τ(gβgα) = τ(gβ)τ(gα) = ϕjϕi. (21) For generators ϕi ∈ {ϕ1, ϕ2, . . . ϕm}, we have gα = ∑ i αigi ∈ G, s.t. τ(gα) = ϕi, the n times composition of itself is
ϕni = τ(gα) n = τ(ngα) = τ( ∑ i αingi) = τ( ∑ i αieG) = τ(eG) = e, (22)
where eG, e are identity elements of G and Φ respectively. In Equation 22, τ(eG) = e holds because τ is an isomorphism, and ker(τ) = {eG}. The sufficiency is proven. ⇐) In the following, we prove that when two conditions are satisfied simultaneously, the mapping τ we define is an isomorphism. Considering the mapping τ : Φ→ G, defined as{
τ : ϕi 7→ gi; τ : ϕiϕj 7→ gi + gj .
(23)
For general elements φt, φs ∈ Φ, where φt = ∏ i ϕ ti i , φs = ∏ i ϕ si i and ti, si ∈ Z, we have
τ(φtφs) = τ (∏ i ϕtii ∏ i ϕsii ) = τ (∏ i ϕti+sii ) = ∑ i (ti + si)gi. (24)
Note that these terms can be merged since ϕiϕj = ϕjϕi. The summation on the right side of Equation 24 is partitioned into two parts as follows∑
i (ti + si)gi = ∑ i tigi + ∑ i sigi = τ (∏ i ϕtii ) + τ (∏ i ϕsii ) = τ(φt) + τ(φs). (25)
Consequently, we have τ(φtφs) = τ(φt) + τ(φs), which states that τ is a homomorphism. Then, we only need to prove that mapping τ is bijective. First, we prove τ is injective, i.e., ker(τ) = {eG}, after that, we prove τ is surjective, i.e., one can find the inverse image of any element.
Since it is not hard to obtain τ(φ) = τ(e · φ) = τ(e) + τ(φ), we have τ(e) = (0, . . . , 0) = eG, i.e., e ∈ τ−1(eG). Assume there is φl = ∏ i ϕ li i , s.t. τ(φl) = eG, we have
τ(φl) = ∑ i ligi = eG ⇒ li|n (li are divisible by n)⇒ φl = ∏ i ϕlii = ∏ i e = e. (26)
The equation above states that τ−1(eG) = {e} (τ : Φ → Φ) and mapping τ is injective. We take ∀ g ∈ G, g = (k1, k1, . . . , km) ∈ (Z/nZ)m, and have
g = (k1, . . . , 0) + · · ·+ (0, . . . , km) = ∑ i kigi = τ (∏ i ϕkii ) . (27)
Hence, we have τ−1(g) = ∏ i ϕ ki i ∈ Φ, which indicates that mapping τ is surjective. Since mapping τ is bijective and homomorphism, τ is an isomorphism.

heading: Q.E.D.

heading: D PROOF FOR THEOREM 3
text: Proof. In the following, we prove that the necessary and sufficient condition for satisfying Abel and Order constraints is that both Abel and Order Loss are minimized.
⇒) For the condition: ∀ϕi, ϕj ∈ Φ, 0 ≤ i, j ≤ m, we have ϕiϕj = ϕjϕi. The constraint on any image o ∈ O can be obtained by
ϕi(ϕj(o)) = ϕj(ϕi(o))⇒ ϕi(ϕj(o))− ϕj(ϕi(o)) = 0. (28)
For the set of combinations of factors C = {(i, j)|1 ≤ i, j ≤ m} and the set containing images O, we have La = ∑ o∈O ∑ (i,j)∈C ‖ϕi · (ϕj · o)− ϕj · (ϕi · o)‖ is minimized. (29)
We obtain the Abel loss. For the Order loss, we first consider n times composition of the same mapping ϕi, we have
ϕni = e⇒ ϕn−1i ϕi = ϕiϕ n−1 i = e. (30)
Therefore, we then have ϕ−1i = ϕ n−1 i . For a single image o ∈ O, we have
ϕi(ϕ n−1 i (o)) = ϕi(ϕ −1 i (o)) = o⇒ ϕi(ϕ −1 i (o))− o = 0. (31)
Thus, for the set of factors and the set containing all images O, we have∑ o∈O ∑ 0≤i≤m ‖ϕi · (ϕ−1i · o)− o‖ is minimized. (32)
To eliminate the bias of optimization, we optimize the symmetry form of the Order Loss, and we have Lo = ∑ o∈O ∑ 0≤i≤m (‖ϕi · (ϕ−1i · o)− o‖+ ‖ϕ −1 i · (ϕi · o)− o‖) is optimized. (33)
The Order Loss is obtained.
⇐)When the Abel Loss La is optimized, for ∀ o ∈ O, we have
ϕi(ϕj(o))− ϕj(ϕi(o)) = 0⇒ ϕi(ϕ(o)) = ϕj(ϕi(o)). (34)
Therefore, for ∀ϕi, ϕj ∈ Φ, i = 1, 2, . . . ,m and j = 1, 2, . . . ,m, we have ϕiϕj = ϕjϕi, and we obtain the Abel constraint. When the Order Loss Lo is minimized, for ∀ o ∈ O, we have
ϕi(ϕ −1 i (o))− o = 0⇒ ϕi(ϕ −1 i (o)) = o. (35)
This implies that ϕni = ϕi ◦ ϕn−1i = ϕi ◦ ϕ −1 i = e. (36)
Therefore, we obtain the Order constraint. The Group constraints are satisfied.

heading: E THE DATA CONSTRAINT
text: In this section, we prove that if each dimension of the world state is independently sampled (p(w) = Πip(wi), where p denotes the probability mass function (for discrete random variable) or probability density function (for continuous random variable)), then the minimization of total correlation (p(z) = Πip(zi)) is a necessary condition to satisfy the data constraint (see Theorem 4 and Theorem 5 below).
Our goal is to learn a representation z conforming to the group-based definition of disentanglement with an encoder h and a decoder d. Then the data constraint can be formulated as follows. For a generator gi ∈ G, we have b(gi · w) = ϕi · b(w) = h−1(gi · z), where z = f(w) and ϕi is the corresponding permutation of gi under the isomorphism betweenG and Φ. We reorganize the formula and have h ◦ b(gi · w) = f(gi · w) = gi · z.
Theorem 4 Assume that the action of generator gi on z is gi · z = z + gi (element-wise addition), and the action of each generator gi on w only affects a single dimension of w (see the assumptions in Section 3.2). Then the equation f(gi ·w) = gi · z, ∀w ∈W is equivalent to: for each i = 1, 2, . . . , n, there exists a bijective function γi s.t. zi = γi(wj) for some j.
Proof. ⇒) Without loss of generality, we assume gi only affects the j-th dimension of w. If the equation f(gi · w) = gi · z, ∀w ∈ W holds, it’s obvious that, for each i, there exists a bijective function γi s.t. zi = γi(wj).
⇐) In the following, we use (gi ·w)i ((gi ·z)i) to denote the i-th dimension of vector gi ·w (gi ·z). If for i = 1, . . . , n, the functions zi = γi(wj) hold for some j, by using an index permutation j = π(i) we can rewrite the function as zi = γi(wπ(i)). Therefore, z can be formulated as follows
f(w1, . . . , wn) = z = (z1, . . . , zn) = (γ1(wπ(1)), . . . , γn(wπ(n))). (37)
Please note that how the world state transits on dimension i dose not affect how disentangled the representation is. Therefore, we define action of generator gi on w as follows.
gi ·w = gi · (w1, . . . , wn) = (w1, . . . , (gi ·w)j , . . . , wn) = (w1, . . . , γ−1i ((gi ·z)i), . . . , wn), (38) Then, we take f on both sides of the above equation, and apply Equation 37, and have
f(gi · w) = f(w1, . . . , γ−1i ((gi · z)i), . . . , wn) = (γ1(wπ(1)), . . . , γi(γ −1 i ((gi · z)i)), . . . , γn(wπ(n)))
= (z1, . . . , (gi · z)i, . . . , zn) = gi · z (39)

heading: Q.E.D.
text: Note that it is obvious that the following theorem (Theorem 5) holds for discrete random variables and bijective functions. However, since the world state space is dense and the encoder h and decoder d are differentiable in general, we also prove for the case where w and z are treated as continuous random variables. Theorem 5 states that the minimization of total correlation (p(z) = Πip(zi)) is a necessary condition to make equations zi = γi(wπ(i)) satisfied, where i = 1, 2, . . . , n and π(i) is an index permutation.
Theorem 5 For the independent random variables w1, w2, . . . , wn, considering the functions zi = δi(w1, w2, . . . , wn), where i = 1, 2, . . . , n and each δi is a bijective, differentiable function. For i = 1, . . . , n, if there exists an index permutation π(i) s.t., we have ∂zπ(i)∂wi 6= 0 and ∂zπ(i) ∂wk
= 0, where k = 1, 2 . . . , n but k 6= i, then the new random variables z1, z2, . . . , zn are independent.
Proof. We treat the random variables w1, w2, . . . , wn as an n-dimensional random vector W = (w1, w2, . . . , wn). Similarly, we write Z = (zπ(1), zπ(2), . . . , zπ(n)) , which is rearranged by index permutation π(i). According to Change of Variable Theorem For Random Vectors (Mood, 1950), we have
p(Z)|J(W )| = p(W ) = Πip(wi) (40)
where J(W ) is the jacobian matrix of Z w.r.t. W , the (i, j)-th entry of it is ∂zπ(i)∂wj . Since for each i = 1, . . . , n, we have ∂zπ(i)∂wi 6= 0 and ∂zπ(i) ∂wk = 0, where k = 1, 2 . . . , n but k 6= i, then the jacobian
matrix can be formulated as follows
J(W ) = Πi ∂zπ(i)
∂wi (41)
According to Change of Variable Theorem For Random Variable ( Mood (1950)), we have
p(zπ(i))| ∂zπ(i)
∂wi | = p(wi) (42)
Bring Equation 41 and Equation 42 into Equation 40, we have
p(Z) = Πip(wi)
Πi| ∂zπ(i) ∂wi |
= Πip(zπ(i))Πi| ∂zπ(i) ∂wi |
Πi| ∂zπ(i) ∂wi |
= Πip(zπ(i)) = Πip(zi) (43)

heading: Q.E.D.
text: Please note that the inverse proposition of the theorem above does not hold. As a counterexample, for zero mean independent gaussian random variables w1, w2 with common variance σ2, new random variables z1 = √ w21 + w 2 2 (the norm of vector (w1, w2)), z2 = tan
−1(w2/w1) (the angle between vector (w1, w2) and (0, 1)) are independent (Mood, 1950) but ∂zi∂wj 6= 0 for all i, j ∈ {1, 2}. In addition, the sufficiency would be satisfied for some specific settings, e.g., there are only two independent uniformly distributed random variables w1, w2, and the functions z1, z2 are linear functions.
Combining Theorems 4 and 5, we have that the minimization of total correlation (p(z) = Πip(zi)) is a necessary condition to satisfy the data constraint.

heading: F DETAILS OF IMPLEMENTATION

heading: F.1 ABEL LOSS DETAILS
text: As mentioned in the main paper, the Abel Loss of the VAE-based models is as follows: La = ∑ o∈O ∑ (i,j)∈C ‖ϕi · (ϕj · o)− ϕj · (ϕi · o)‖, (44)
where ϕi · (ϕj · o) = ϕi(ϕj(o)) represents the top path of Figure 7 (a), and ϕj · (ϕi · o) = ϕj(ϕi(o)) represents the bottom path of Figure 7 (a). For better optimization, we constrain such consistency on their representation (straight dotted double arrow in Figure 7 (a)) instead of the reconstructed images. Besides, we constrain the consistency between the representations of intermediate images (curved dotted double arrow in Figure 7 (a)).

heading: F.2 ORDER LOSS DETAILS
text: As mentioned in the main paper, the Order Loss of VAE-based models is as follows: Lo = ∑ o∈O ∑ i∈I ( ‖ϕi · (ϕ−1i · o)− o‖+ ‖ϕ −1 i · (ϕi · o)− o‖ ) , (45)
where ϕi · (ϕ−1i · o) represents the lower path of Figure 7 (b), and ϕ −1 i · (ϕi · o) represents the upper path of Figure 7 (b). Similar to the Abel Loss, we do not constrain such consistency on the reconstructed images for better optimization but on their representations instead (the long curved dotted line in Figure 7 (b)). Besides, we constrain the consistency between the representations of intermediate images (short curved dotted lines in Figure 7 (b)).
F.3 THE NUMBER OF FACTORS m
For the given VAE-based models, dimensional KL divergence (indicating the meaningful dimensions) increases during the training process. Therefore, we use m dimensions of which the corresponding dimensional KL divergence KLi ≥ T , where T is a hyperparameter, which is empirically set to 30 in our experiments.

heading: G DETAILS OF EXPERIMENTS

heading: G.1 DATASET DETAILS
text: In all the experiments, we resize the images to 64× 64 resolution. We introduce all the datasets used in our paper in detail.
dSprites Higgins et al. (2017): dSprites contains 737,280 binary 2D shapes (heart, oval and square) images with five ground truth factors: shape (3 values), scale (6 values), orientation (40 values), x-position (32 values), y-position (32 values). Then we introduce the variants of dSprites (Color dSprites and Noisy dSprites) created by Locatello et al. Locatello et al. (2019b).
Color dSprites, the shapes of the images in dSprites are randomly colored.
Noisy dSprites, the background of the images in dSprites is noise.
Shapes3D Kim & Mnih (2018): Shapes3D contains 480,000 images of 3D shapes with 6 ground truth factors: shape (4 values), scale (8 values), orientation (15 values), floor color (10 values), wall color (15 values), object color (10 values).
Cars3D Reed et al. (2015): This dataset consists of 183 car CAD models, each rendered from 24 azimuth directions and 4 elevations.

heading: G.2 ARCHITECTURE FOR ENCODER AND DECODER
text: We follow Locatello et al. Locatello et al. (2019b) to use the same architecture of VAEs in all of the experiments: the activation function used is ReLU except for the last layer of decoder, as shown in Table 3. For the details of the Discriminator in FactorVAE, please refer to Table 5 (a) and (c).

heading: G.3 EXPERIMENT SETTINGS
text: We run using different hyperparameters and random seeds for every VAE-based model implemented by Pytorch Paszke et al. (2017). As shown in Table 4, for β-VAE, we assign 3 choices for β and 10 random seeds for both the Original and Groupified VAEs: 3× 10× 2 = 60 settings for each dataset. Similarly, we also assign 60 settings for FactorVAE and β-TCVAE. For AnnealVAE, we assign three choices for C and 3 choices for the start and end pair, also assign 10 random seeds. In summary, for all 5 datasets, we run (((3 × 10 × 2) × 3) + 3 × 3 × 10 × 2) × 5 = 1800 models. For other hyperparameters, please refer to Table 5 (b).

heading: H RELATION TO SOME PREVIOUS WORKS

heading: H.1 SYMMETRY-BASED DISENTANGLEMENT
text: Caselles-Dupré et al. (2019) argue that the symmetry-based disentanglement requires interaction with the environments. Specifically, for a given disentangled representation z w.r.t. a world with some group action on W , there are multiple other worlds (same world states and symmetry) of a static dataset that have different group actions on each dimension of W . For those worlds, the representation z is not disentangled, per the group-based definition. However, in our work, we do not assume that a static dataset has a world equipped with some specific fixed group action on W in advance, instead we use permutation group Φ as an agent to learn a world with proper group actions. For this learned world, there is only one representation that satisfies the definition. Therefore, we do not need an environment to provide group actions on W to determine which world it is. Please see our proof in Theorem 1. In the testing phase, since we input the images to the model directly to derive their representation, the disentanglement of the representation does not rely on how the world state transitions between each other (i.e., as a result of group actions on W ). Therefore, our framework can learn such a disentangled representation without interaction with the environments.

heading: H.2 LIE GROUP VAE
text: Zhu et al. (2021) argue that the representation space being a vector space is sub-optimal since it requires the model to learn to discard different scales of variations. They propose to use lie group as the representation space instead and use Hessian Penalty to encourage disentanglement. Our framework is complimentary to their proposed method. The lie group representation can be applied to extend our framework. Here we leave it for future work. In addition, our proposed representation mapped by the sine and cosine function (exp((2πiz)/n) for a real vector z) is also a lie group.

heading: I MORE QUANTITATIVE RESULTS AND SCORE DISTRIBUTION

heading: I.1 QUANTITATIVE RESULTS
text: The performance of the Original and Groupified VAEs on all five datasets is shown in Table 6. Our method outperforms the original one on most of the datasets in terms of nearly all the metrics.
I.2 ABSTRACT REASONING & FAIRNESS Abstract reasoning↑ Unfairness scores↓
As pointed out by Locatello et al. Locatello et al. (2019b), the disentangled representation’s downstream tasks should also be verified. Therefore, we verify the effectiveness of the representations learned by the Groupified VAEs on Shapes3d in two downstream tasks: abstract rea-
soning Van Steenkiste et al. (2019) and fairness evaluation Locatello et al. (2019a). As Table 7 shows, the performance of the abstract reasoning models fine-tuned on the representation learned by the Groupified FactorVAEs is better than the original ones. In terms of fairness evaluation, we can observe that the unfairness scores of the representation learned by the Groupified FactorVAEs are lower than the Original ones.

heading: I.3 SCORE DISTRIBUTION
text: The detailed distribution of the performance is shown in this section (demonstrated by the Violin Plot Hintze & Nelson (1998)). The performance distributions on dSprits, Car3d, Noisy dSprites,
Color-dSprites, and Shapes3d are shown in Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12, respectively.

heading: J MORE QUALITATIVE RESULTS
text: We evaluate our methods qualitatively on two typical datasets: Cars3d and Shapes3d. We visualize the traversal results of the Original and Groupified FactorVAE and β-TCVAE. For every factor, we traverse five randomly sampled representations. As shown in Figure 13 and Figure 14, the traversal results of the Groupified FactorVAE and β-TCVAE show that these models learn less entangled representations on Shapes3D (e.g., Orientation of FactorVAE and Scale and Shape of β-TCVAE).
Similarly, as shown in Figure 15 and Figure 16, the Groupified FactorVAE and β-TCVAE achieve better disentanglement ability on Car3d (e.g., Rotation of FactorVAE and Yaw of β-TCVAE).
For the real-world dataset, we show the qualitative comparison of the Groupified and original AnnealVAE trained on CelebA. As Figure 17 shows, for most of the factors, our model can extract cleaner semantics than the original model. For example, the hair color is entangled with the face shape in the original model, but cleaner in the Groupified model.

heading: K MEANINGFUL DIMENSION VISUALIZATIONS
text: When we assign some dimensions to Isomorphism Loss for Groupified AnnealVAEs, e.g., the first-5 dimensions, we have an interesting observation that the assigned dimensions are meaningful in Groupified AnnealVAEs. The KL divergence increases continuously on these assigned dimensions after the Isomorphism Loss is applied to them. Note that the KL divergence loss in AnnealVAE indicates the amount of information encoded. As Figure 18 shows, the KL divergence of assigned dimensions increases at the beginning of training, which means Isomorphism Loss results in that the assigned dimensions become meaningful. Finally, the assigned first-five dimensions learn to encode the semantics of x position, y position, scale, and orientations. The possible reason is that the latent factors are learned and disentangled in the assigned dimensions due to the punishment of the Isomorphism Loss. To illustrate that controllable dimensions in Groupified AnnealVAE are not an exception, we provide more visualizations. The results of covering all hyperparameters settings with two random seeds are shown in Figure 18 to Figure 26, suggesting that their dimensions are controllable.

heading: L MORE REPRESENTATION SPACE VISUALIZATIONS
text: To illustrate that the Isomorphism Loss suppresses the representation space collapse in VAE-based methods is not an exception, we provide more visualizations of the representation space visualization of Groupified VAEs. The Original and Groupified β-VAEs, AnnealVAEs, FactorVAEs, and βTCVAEs are shown in Figure 27, Figure 28, Figure 29, and Figure 30, respectively, which implies that the representation space of Groupified VAEs are organized better than the original ones.

Review summaries:
This paper provides a theoretical framework on the learning of group-based disentanglement representations. It proposes a method to learn a cyclic group representation with the Abel loss and Order loss based on VAEs. Experiments validate the effectiveness of the proposed method on improving existing disentanglement VAE models.

The paper proposes a theoretical framework for unsupervised representation disentanglement (in the sense of Higgins et al.) based on three constraints (group structure, data and model). It further describes a learning method for satisfying these constraints, and experimentally shows some performance gain on traditional disentanglement metrics.   My intuitive understanding of the constraints is a follows: - the model constraint is satisfied by construction of the agent group, whose structure is transferred from the latent space by the model. - the group structure constraint ensures that this transfer is an isomorphism, and thus that the agent group is actually a group. - the data constraint ensures that the agent group actually matches the ground truth generative factors.   Please clarify if one of my statements is incorrect.

This paper presents an unsupervised approach to achieve group-based disentangled representation learning, as opposed to existing environment-based approaches. A theoretical framework for the group-based VAE was proposed, and implementations of the group and isomorphism reported. Experiments were performed on four common benchmark datasets, with comparisons to several representative VAE-based disentanglement framework (beta-vae, annealVAE, factorVAE, beta-TCVAE), demonstrating the groupified VAE achieves better disentanglement in many metrics. Qualitative evaluations demonstrate the cyclic representation space learned by the groupified VAE.

This paper proposed a new theoretical framework for achieving unsupervised representation disentanglement based on $n$-th dihedral group. Further, it proved three sufficient conditions on the model, group structure, and data respectively. Evaluations on multiple benchmark datasets demonstrate the proposed framework achieves better mean performance with smaller variances compared to some VAE models.