Towards Building A Group-based Unsupervised Representation Disentanglement Framework

1 INTRODUCTION . Learning independent and semantic representations of which individual dimension has interpretable meaning , usually referred to as disentangled representations learning , is critical for artificial intelligence research ( Bengio et al. , 2013 ) . Such disentangled representations are useful for many tasks : domain adaptation ( Li et al. , 2019 ; Zou et al. , 2020 ) , zero-shot learning ( Lake et al. , 2017 ) , and adversarial attacks ( Alemi et al. , 2016 ) , etc . Intuitively , a disentangled representation should reflect the factors of variations behind the observed data of the world , and one latent unit is only sensitive to changes of an individual factor . Due to the facts that obtaining the ground-truth labels requires significant human effort and humans can learn those factors unsupervisedly , unsupervised representation disentanglement draws much attention from researchers recently . A lot of methods are proposed base on some intuitions . Most of the state-of-the-art methods ( Higgins et al. , 2017 ; Burgess et al. , 2018 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Kumar et al. , 2017 ) are based on Variational Autoencoder ( VAE ) ( Kingma & Welling , 2013 ) . These methods are fully unsupervised and can be applied to a variety of complex datasets ( Lee et al. , 2020 ) . However , these methods suffer from the unidentifiability problem ( Locatello et al. , 2019b ) due to a lack of theoretical guarantee . Another stream of works ( Chen et al. , 2016 ; Lin et al. , 2020 ; Khrulkov et al. , 2021 ; Lee et al. , 2020 ) leverage generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) to achieve disentanglement but are not interpretable . In general , a well-defined theoretical guarantee is needed for those methods . The research of symmetry in physics demonstrates that infinitesimal transformations that conform to some symmetry groups on physical objects can reflect their nature ( Anderson , 1972 ; Noether , 1915 ) . Recently , inspired by this research on symmetry , Higgins et al . ( 2018 ) proposed a group-based definition of disentangled representation . They argue that the symmetries , i.e. , the transformations that change certain aspects of data and keep other aspects unchanged , ideally reflect the underlying data structure . The group-based definition is a formal and rigorous mathematical definition of faithful and , ideally , interpretable representation of the generative factors of data , which is widely accepted ( Greff et al. , 2019 ; Mathieu et al. , 2019 ; Khemakhem et al. , 2020 ) . Subsequently , due to the fact that the definition is defined by the world state ( i.e. , Ground Truth ) and based on the assumption ( CasellesDupré et al. , 2019 ) that this definition should be useful for downstream tasks such as a Reinforcement Learning , Caselles-Dupré et al . ( 2019 ) , Quessard et al . ( 2020 ) , Painter et al . ( 2020 ) propose environment-based ( to provide world state ) methods to learn such disentangled representations in Reinforcement Learning settings . These inspire us to ask the following question : how would the definition benefit unsupervised representation disentanglement , and how to learn such a disentangled representation conforming to the definition in the setting of unsupervised representation learning ? In Group Theory , the n-th dihedral group1 ( Judson , 2020 ) is a set of all permutations of polygons vertices , forming a permutation group under the operation of composition ( Miller , 1973 ) . The generators in an n-th dihedral group , i.e. , flip and rotation , can be regarded as the disentangled factors and also transformations . In this paper , inspired by the n-th dihedral group , we answer the above questions and address the challenge by proposing a theoretical framework to make the definition practically applicable for unsupervised representation disentanglement . We then propose a model to tackle the learning problem of the framework and verify its effectiveness . We theoretically prove in Section 3.2 the three sufficient conditions towards achieving disentangled representation per groupbased definition , which are referred to as model , group structure , and data constraint , respectively . With these conditions , we offer additional constraints from the perspective of the definition . The additional constraints encourage existing VAE-based models to satisfy the symmetry requirement that comes from the nature of factors . Finally , we provide a learning model based on the existing VAE-based methods in an effort to fulfill the three conditions ( with the model and group structure constraint and a necessary condition for the data constraint satisfied ) . As an intuitive understanding , we introduce the additional constraints to reorganize the latent space to restrict its symmetry in an unsupervised way . These additional constraints indeed narrow down the solution space of VAEbased models . Detailed discussion in Sec . 5.4 . Our model consistently achieves statistically better performance in prominent metrics ( higher means and lower variances ) than corresponding existing VAE-based models on five datasets , demonstrating that the group-based definition together with our proposed framework further encourages disentanglement . Our main contributions are summarized as follows : • To our best knowledge , we are the first to provide a theoretical framework to make the formal group-based mathematical definition of disentanglement practically applicable to unsupervised representation disentanglement . • Our theoretical framework provides additional constraints from the perspective of groupbased definition for the existing VAE-based methods . • We propose a learning model of the framework by deriving and integrating additional loss into existing VAE-based models , in an effort to make the learned representation conform to the group-based definition without relying on the environment ( as done in Caselles-Dupré et al . ( 2019 ) ; Quessard et al . ( 2020 ) ; Painter et al . ( 2020 ) ) . 2 RELATED WORKS . Different definitions have been proposed for disentangled representation ( Bengio et al. , 2013 ; Higgins et al. , 2018 ; Suter et al. , 2019 ) . However , only the group-based definition proposed by Higgins et al . ( 2018 ) focuses on the disentangled representation itself and is mathematically rigorous , which is well accepted ( Caselles-Dupré et al. , 2019 ; Quessard et al. , 2020 ; Painter et al. , 2020 ; Diane Bouchacourt , 2021 ) . Nevertheless , Higgins et al . ( 2018 ) do not propose a specific learning method based on their definition . Before this rigorous definition was proposed , there had been some success in identifying generative factors in static datasets ( without interaction with environment ) , such as β-VAE ( Higgins 1We assume some basic familiarity with the fundamentals of Group Theory and Group Representation Theory . Please refer to Appendix A for some basic concepts . et al. , 2017 ) , Anneal-VAE ( Burgess et al. , 2018 ) , β-TCVAE ( Chen et al. , 2018 ) , and FactorVAE ( Kim & Mnih , 2018 ) . More recent works ( Srivastava et al. , 2020 ; Shao et al. , 2020 ; Kim et al. , 2019 ; Lezama , 2018 ; Rezende & Viola , 2018 ) also do not consider the group-based definition . Therefore , how group-based definition will facilitate these methods is still an open question . Besides , all these works suffer from the unidentifiability problem ( Locatello et al. , 2019b ) , which is a challenging problem in this literature . From group-based definition , our framework points out that , the unidentifiability problem could be solved once the data constraint is satisfied . However , in this work , we can only get a necessary condition for data constraint , and we still can not solve this challenging problem . As pointed out in Quessard et al . ( 2020 ) , it is not straightforward to reconcile the probabilistic inference methods with the group-based definition framework . Caselles-Dupré et al . ( 2019 ) , Quessard et al . ( 2020 ) , Painter et al . ( 2020 ) leverage the interaction with the environment ( assuming it is available ) as supervision instead of minimizing the total correlation as the VAE-based methods do . Consequently , the effectiveness of these methods is limited to the datasets with the environment available . Our framework learns a representation conforming to the group-based definition without relying on the environment . Pfau et al . ( 2020 ) propose a non-parametric method to unsupervisedly learn linear disentangled planes in data manifold under a metric . However , as pointed out by the authors , the method does not generalize to held-out data and performs poorly when trying to disentangle directly from pixels . To summarize , the existing probabilistic inference methods lack theoretical support , while the application scope of existing methods based on the group-based mathematical definition Higgins et al . ( 2018 ) is very limited . To the best of our knowledge , our work is the first to reconcile the probabilistic generative methods with the inherently deterministic group-based definition framework of Higgins et al . ( 2018 ) . 3 THE GROUP-BASED FRAMEWORK FOR UNSUPERVISED REPRESENTATION DISENTANGLEMENT . Our goal is to explore the benefit of the group-based definition for unsupervised representation disentanglement and learn such a disentangled representation . The background of the group-based definition is provided in Section 3.1 . Section 3.2 presents the theoretical framework towards achieving unsupervised disentanglement , in which we derive three sufficient conditions on the model , group structure , and data , respectively . The conditions on the model and group structure provide additional constraints for the existing VAE-based models . 3.1 GROUP-BASED DEFINITION . We briefly review the group-based definition of disentangled representation Higgins et al . ( 2018 ) . Considering a group G acting on world state space W ( can be understood as ground-truth ) of data space O and representation space Z via group action ·W and group action ·Z respectively . For a mapping f = b ◦ h , where b and h denote the data generative process and encoding , we state : the mapping f is equivariant between the actions on W and Z if g · f ( w ) = f ( g · w ) , ∀g ∈ G , ∀w ∈W . ( 1 ) Definition 1 Assume G can be decomposed as G = G1×G2× · · ·×Gm . The set Z is disentangled with respect to G if : ( i ) group action of G on Z exits . ( ii ) the mapping f is equivariant between the actions on W and Z . ( iii ) There is a decomposition Z = Z1 × Z2 × · · · × Zm such that each Zi is affected only by the corresponding Gi . It is challenging to apply the group-based definition to an unsupervised disentanglement setting in practice because the definition refers to the world state space W , the group action of G on W , and mapping b which are typically inaccessible in practice . We tackle the challenge by re-framing the definition in a new framework in the following section . 3.2 PROPOSED THEORETICAL FRAMEWORK . Since when the representation is disentangled , one latent unit in the representation space is only sensitive to changes of an individual generative factor , we make the following assumptions : G is a direct product of m cyclic groups ( as suggested by Higgins et al . ( 2018 ) and for simplicity ) : G = ( Z/nZ ) m = Z/nZ× Z/nZ× · · · × Z/nZ , where n is the assumed total number of possible values for a factor and m is the total number of factors ; we further assume Z is a set with the same elements in G. Therefore , the group actions of G on Z can be set to be element-wise addition , i.e. , g · z = g + z , ∀z ∈ Z , g ∈ G. For the generator of dimension i of G , gi = ( 0 , . . . , 1 , . . . , 0 ) , gi only affects the i-th dimension of z by gi + z . In addition , the action of each generator gi on w only affects a single dimension of w. As we can seen from Equation 1 above , the group action is defined on w , which is often not accessible , making it difficult to apply the definition in practice . Therefore , for the unsupervised setting , we would like to use permutations on the data space O ( which only provides data without labels ) to substitute the group actions on W . Specifically , inspired by the n-th dihedral group ( Dummit & Foote , 1991 ) , we construct a permutation group Φ , serving the role of an “ agent ” of G. The actions of G on W can be performed by ϕg ∈ Φ on O , which can be formulated as f ( g · w ) = h ( ϕg · b ( w ) ) = h ( ϕg · o ) , ∀w ∈W , g ∈ G , ( 2 ) where o denotes the data ( e.g. , image ) corresponding to the world state w through the mapping function b . If the above equation holds , we state that the “ agent ” permutation group Φ exists . We first give the conditions for the existence of this “ agent ” permutation group Φ , then derive the additional condition to achieve such disentanglement . We accomplish these two objectives in Theorem 1 with the proof provided in Appendix B. Theorem 1 states that a general permutation group Φ on O can serve as an agent group ( agent group exists ) if and only if both ( i ) and ( ii ) are satisfied . If the agent group exists , and its permutations ( actions on O ) can be defined by an autoencoder-like model as shown in the equation in ( iii ) , then Z is disentangled with respect to G. Theorem 1 For the group G = ( Z/nZ ) m , a permutation group Φ on O , a representation space Z , a World State space W , and mapping b and h , Equation 2 holds if and only if ( i ) Φ is isomorphic to G , and ( ii ) For each generator of dimension i of G , gi , there exists a ϕi ∈ Φ , i = 1 , . . . , m , such that ϕi · b ( w ) = b ( gi · w ) , ∀w ∈ W , and ϕi is a generator of Φ ; Further , if Equation 2 holds and ( iii ) ϕg · b ( w ) = h−1 ( g · f ( w ) ) ∀w ∈W , ϕg ∈ Φ , then Z is disentangled with respect to G , where ϕg is the corresponding element in Φ of g under the isomorphism . In Theorem 1 , ( i ) states that the relation between the elements ( i.e. , group structure ) is preserved between Φ and G , and we denote it as the group structure constraint ; ( ii ) actually indicates a data constraint that all variations in the data can be generated by compositions of some basic permutation generators { ϕi } i=1 , ... , m. We denote it as the data constraint ; ( iii ) states that the permutations in the agent group Φ are defined by encoding , action , and decoding , which is referred to as the model constraint . Note that in Theorem 1 , only the data constraint refers to the world state w. Here is a sketch of the proof : data constraint is a special case of Equation 2 for a generator , and group structure constraint is a relation-preserving constraint on compositions of generators , and satisfying both constraints will thus result in that Equation 2 holds for any general element in Φ , and vice versa . Moreover , we can derive Equation 1 for disentanglement when combining the model constraint and Equation 2 . The model constraint specifies the way to permute the data . When the data is permuted , its world state changes . Therefore , how the world states transit between each other is modeled by the model constraint applied on the data . The isomorphism between Φ and G ensures that the world state space W and data space O have the same symmetry . In this way , the model applied on the data learns the transition of the world states . Note that we aim to bring this group-based definition , which requires ground truth by default , into the unsupervised setting . Now only the data constraint refers to the world states , and it seems almost impossible to derive a sufficient condition for it without the labels . We thus make a trade-off in which we use a necessary condition in the next section .

Summary: This paper proposed a new theoretical framework for achieving unsupervised representation disentanglement based on $n$-th dihedral group. Further, it proved three sufficient conditions on the model, group structure, and data respectively. Evaluations on multiple benchmark datasets demonstrate the proposed framework achieves better mean performance with smaller variances compared to some VAE models.