Towards Building A Group-based Unsupervised Representation Disentanglement Framework

1 INTRODUCTION . Learning independent and semantic representations of which individual dimension has interpretable meaning , usually referred to as disentangled representations learning , is critical for artificial intelligence research ( Bengio et al. , 2013 ) . Such disentangled representations are useful for many tasks : domain adaptation ( Li et al. , 2019 ; Zou et al. , 2020 ) , zero-shot learning ( Lake et al. , 2017 ) , and adversarial attacks ( Alemi et al. , 2016 ) , etc . Intuitively , a disentangled representation should reflect the factors of variations behind the observed data of the world , and one latent unit is only sensitive to changes of an individual factor . Due to the facts that obtaining the ground-truth labels requires significant human effort and humans can learn those factors unsupervisedly , unsupervised representation disentanglement draws much attention from researchers recently . A lot of methods are proposed base on some intuitions . Most of the state-of-the-art methods ( Higgins et al. , 2017 ; Burgess et al. , 2018 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Kumar et al. , 2017 ) are based on Variational Autoencoder ( VAE ) ( Kingma & Welling , 2013 ) . These methods are fully unsupervised and can be applied to a variety of complex datasets ( Lee et al. , 2020 ) . However , these methods suffer from the unidentifiability problem ( Locatello et al. , 2019b ) due to a lack of theoretical guarantee . Another stream of works ( Chen et al. , 2016 ; Lin et al. , 2020 ; Khrulkov et al. , 2021 ; Lee et al. , 2020 ) leverage generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) to achieve disentanglement but are not interpretable . In general , a well-defined theoretical guarantee is needed for those methods . The research of symmetry in physics demonstrates that infinitesimal transformations that conform to some symmetry groups on physical objects can reflect their nature ( Anderson , 1972 ; Noether , 1915 ) . Recently , inspired by this research on symmetry , Higgins et al . ( 2018 ) proposed a group-based definition of disentangled representation . They argue that the symmetries , i.e. , the transformations that change certain aspects of data and keep other aspects unchanged , ideally reflect the underlying data structure . The group-based definition is a formal and rigorous mathematical definition of faithful and , ideally , interpretable representation of the generative factors of data , which is widely accepted ( Greff et al. , 2019 ; Mathieu et al. , 2019 ; Khemakhem et al. , 2020 ) . Subsequently , due to the fact that the definition is defined by the world state ( i.e. , Ground Truth ) and based on the assumption ( CasellesDupré et al. , 2019 ) that this definition should be useful for downstream tasks such as a Reinforcement Learning , Caselles-Dupré et al . ( 2019 ) , Quessard et al . ( 2020 ) , Painter et al . ( 2020 ) propose environment-based ( to provide world state ) methods to learn such disentangled representations in Reinforcement Learning settings . These inspire us to ask the following question : how would the definition benefit unsupervised representation disentanglement , and how to learn such a disentangled representation conforming to the definition in the setting of unsupervised representation learning ? In Group Theory , the n-th dihedral group1 ( Judson , 2020 ) is a set of all permutations of polygons vertices , forming a permutation group under the operation of composition ( Miller , 1973 ) . The generators in an n-th dihedral group , i.e. , flip and rotation , can be regarded as the disentangled factors and also transformations . In this paper , inspired by the n-th dihedral group , we answer the above questions and address the challenge by proposing a theoretical framework to make the definition practically applicable for unsupervised representation disentanglement . We then propose a model to tackle the learning problem of the framework and verify its effectiveness . We theoretically prove in Section 3.2 the three sufficient conditions towards achieving disentangled representation per groupbased definition , which are referred to as model , group structure , and data constraint , respectively . With these conditions , we offer additional constraints from the perspective of the definition . The additional constraints encourage existing VAE-based models to satisfy the symmetry requirement that comes from the nature of factors . Finally , we provide a learning model based on the existing VAE-based methods in an effort to fulfill the three conditions ( with the model and group structure constraint and a necessary condition for the data constraint satisfied ) . As an intuitive understanding , we introduce the additional constraints to reorganize the latent space to restrict its symmetry in an unsupervised way . These additional constraints indeed narrow down the solution space of VAEbased models . Detailed discussion in Sec . 5.4 . Our model consistently achieves statistically better performance in prominent metrics ( higher means and lower variances ) than corresponding existing VAE-based models on five datasets , demonstrating that the group-based definition together with our proposed framework further encourages disentanglement . Our main contributions are summarized as follows : • To our best knowledge , we are the first to provide a theoretical framework to make the formal group-based mathematical definition of disentanglement practically applicable to unsupervised representation disentanglement . • Our theoretical framework provides additional constraints from the perspective of groupbased definition for the existing VAE-based methods . • We propose a learning model of the framework by deriving and integrating additional loss into existing VAE-based models , in an effort to make the learned representation conform to the group-based definition without relying on the environment ( as done in Caselles-Dupré et al . ( 2019 ) ; Quessard et al . ( 2020 ) ; Painter et al . ( 2020 ) ) . 2 RELATED WORKS . Different definitions have been proposed for disentangled representation ( Bengio et al. , 2013 ; Higgins et al. , 2018 ; Suter et al. , 2019 ) . However , only the group-based definition proposed by Higgins et al . ( 2018 ) focuses on the disentangled representation itself and is mathematically rigorous , which is well accepted ( Caselles-Dupré et al. , 2019 ; Quessard et al. , 2020 ; Painter et al. , 2020 ; Diane Bouchacourt , 2021 ) . Nevertheless , Higgins et al . ( 2018 ) do not propose a specific learning method based on their definition . Before this rigorous definition was proposed , there had been some success in identifying generative factors in static datasets ( without interaction with environment ) , such as β-VAE ( Higgins 1We assume some basic familiarity with the fundamentals of Group Theory and Group Representation Theory . Please refer to Appendix A for some basic concepts . et al. , 2017 ) , Anneal-VAE ( Burgess et al. , 2018 ) , β-TCVAE ( Chen et al. , 2018 ) , and FactorVAE ( Kim & Mnih , 2018 ) . More recent works ( Srivastava et al. , 2020 ; Shao et al. , 2020 ; Kim et al. , 2019 ; Lezama , 2018 ; Rezende & Viola , 2018 ) also do not consider the group-based definition . Therefore , how group-based definition will facilitate these methods is still an open question . Besides , all these works suffer from the unidentifiability problem ( Locatello et al. , 2019b ) , which is a challenging problem in this literature . From group-based definition , our framework points out that , the unidentifiability problem could be solved once the data constraint is satisfied . However , in this work , we can only get a necessary condition for data constraint , and we still can not solve this challenging problem . As pointed out in Quessard et al . ( 2020 ) , it is not straightforward to reconcile the probabilistic inference methods with the group-based definition framework . Caselles-Dupré et al . ( 2019 ) , Quessard et al . ( 2020 ) , Painter et al . ( 2020 ) leverage the interaction with the environment ( assuming it is available ) as supervision instead of minimizing the total correlation as the VAE-based methods do . Consequently , the effectiveness of these methods is limited to the datasets with the environment available . Our framework learns a representation conforming to the group-based definition without relying on the environment . Pfau et al . ( 2020 ) propose a non-parametric method to unsupervisedly learn linear disentangled planes in data manifold under a metric . However , as pointed out by the authors , the method does not generalize to held-out data and performs poorly when trying to disentangle directly from pixels . To summarize , the existing probabilistic inference methods lack theoretical support , while the application scope of existing methods based on the group-based mathematical definition Higgins et al . ( 2018 ) is very limited . To the best of our knowledge , our work is the first to reconcile the probabilistic generative methods with the inherently deterministic group-based definition framework of Higgins et al . ( 2018 ) . 3 THE GROUP-BASED FRAMEWORK FOR UNSUPERVISED REPRESENTATION DISENTANGLEMENT . Our goal is to explore the benefit of the group-based definition for unsupervised representation disentanglement and learn such a disentangled representation . The background of the group-based definition is provided in Section 3.1 . Section 3.2 presents the theoretical framework towards achieving unsupervised disentanglement , in which we derive three sufficient conditions on the model , group structure , and data , respectively . The conditions on the model and group structure provide additional constraints for the existing VAE-based models . 3.1 GROUP-BASED DEFINITION . We briefly review the group-based definition of disentangled representation Higgins et al . ( 2018 ) . Considering a group G acting on world state space W ( can be understood as ground-truth ) of data space O and representation space Z via group action ·W and group action ·Z respectively . For a mapping f = b ◦ h , where b and h denote the data generative process and encoding , we state : the mapping f is equivariant between the actions on W and Z if g · f ( w ) = f ( g · w ) , ∀g ∈ G , ∀w ∈W . ( 1 ) Definition 1 Assume G can be decomposed as G = G1×G2× · · ·×Gm . The set Z is disentangled with respect to G if : ( i ) group action of G on Z exits . ( ii ) the mapping f is equivariant between the actions on W and Z . ( iii ) There is a decomposition Z = Z1 × Z2 × · · · × Zm such that each Zi is affected only by the corresponding Gi . It is challenging to apply the group-based definition to an unsupervised disentanglement setting in practice because the definition refers to the world state space W , the group action of G on W , and mapping b which are typically inaccessible in practice . We tackle the challenge by re-framing the definition in a new framework in the following section . 3.2 PROPOSED THEORETICAL FRAMEWORK . Since when the representation is disentangled , one latent unit in the representation space is only sensitive to changes of an individual generative factor , we make the following assumptions : G is a direct product of m cyclic groups ( as suggested by Higgins et al . ( 2018 ) and for simplicity ) : G = ( Z/nZ ) m = Z/nZ× Z/nZ× · · · × Z/nZ , where n is the assumed total number of possible values for a factor and m is the total number of factors ; we further assume Z is a set with the same elements in G. Therefore , the group actions of G on Z can be set to be element-wise addition , i.e. , g · z = g + z , ∀z ∈ Z , g ∈ G. For the generator of dimension i of G , gi = ( 0 , . . . , 1 , . . . , 0 ) , gi only affects the i-th dimension of z by gi + z . In addition , the action of each generator gi on w only affects a single dimension of w. As we can seen from Equation 1 above , the group action is defined on w , which is often not accessible , making it difficult to apply the definition in practice . Therefore , for the unsupervised setting , we would like to use permutations on the data space O ( which only provides data without labels ) to substitute the group actions on W . Specifically , inspired by the n-th dihedral group ( Dummit & Foote , 1991 ) , we construct a permutation group Φ , serving the role of an “ agent ” of G. The actions of G on W can be performed by ϕg ∈ Φ on O , which can be formulated as f ( g · w ) = h ( ϕg · b ( w ) ) = h ( ϕg · o ) , ∀w ∈W , g ∈ G , ( 2 ) where o denotes the data ( e.g. , image ) corresponding to the world state w through the mapping function b . If the above equation holds , we state that the “ agent ” permutation group Φ exists . We first give the conditions for the existence of this “ agent ” permutation group Φ , then derive the additional condition to achieve such disentanglement . We accomplish these two objectives in Theorem 1 with the proof provided in Appendix B. Theorem 1 states that a general permutation group Φ on O can serve as an agent group ( agent group exists ) if and only if both ( i ) and ( ii ) are satisfied . If the agent group exists , and its permutations ( actions on O ) can be defined by an autoencoder-like model as shown in the equation in ( iii ) , then Z is disentangled with respect to G. Theorem 1 For the group G = ( Z/nZ ) m , a permutation group Φ on O , a representation space Z , a World State space W , and mapping b and h , Equation 2 holds if and only if ( i ) Φ is isomorphic to G , and ( ii ) For each generator of dimension i of G , gi , there exists a ϕi ∈ Φ , i = 1 , . . . , m , such that ϕi · b ( w ) = b ( gi · w ) , ∀w ∈ W , and ϕi is a generator of Φ ; Further , if Equation 2 holds and ( iii ) ϕg · b ( w ) = h−1 ( g · f ( w ) ) ∀w ∈W , ϕg ∈ Φ , then Z is disentangled with respect to G , where ϕg is the corresponding element in Φ of g under the isomorphism . In Theorem 1 , ( i ) states that the relation between the elements ( i.e. , group structure ) is preserved between Φ and G , and we denote it as the group structure constraint ; ( ii ) actually indicates a data constraint that all variations in the data can be generated by compositions of some basic permutation generators { ϕi } i=1 , ... , m. We denote it as the data constraint ; ( iii ) states that the permutations in the agent group Φ are defined by encoding , action , and decoding , which is referred to as the model constraint . Note that in Theorem 1 , only the data constraint refers to the world state w. Here is a sketch of the proof : data constraint is a special case of Equation 2 for a generator , and group structure constraint is a relation-preserving constraint on compositions of generators , and satisfying both constraints will thus result in that Equation 2 holds for any general element in Φ , and vice versa . Moreover , we can derive Equation 1 for disentanglement when combining the model constraint and Equation 2 . The model constraint specifies the way to permute the data . When the data is permuted , its world state changes . Therefore , how the world states transit between each other is modeled by the model constraint applied on the data . The isomorphism between Φ and G ensures that the world state space W and data space O have the same symmetry . In this way , the model applied on the data learns the transition of the world states . Note that we aim to bring this group-based definition , which requires ground truth by default , into the unsupervised setting . Now only the data constraint refers to the world states , and it seems almost impossible to derive a sufficient condition for it without the labels . We thus make a trade-off in which we use a necessary condition in the next section .

Summary: The paper proposes a theoretical framework for unsupervised representation disentanglement (in the sense of Higgins et al.) based on three constraints (group structure, data and model). It further describes a learning method for satisfying these constraints, and experimentally shows some performance gain on traditional disentanglement metrics.   My intuitive understanding of the constraints is a follows: - the model constraint is satisfied by construction of the agent group, whose structure is transferred from the latent space by the model. - the group structure constraint ensures that this transfer is an isomorphism, and thus that the agent group is actually a group. - the data constraint ensures that the agent group actually matches the ground truth generative factors.   Please clarify if one of my statements is incorrect.