Quantifying and Learning Disentangled Representations with Limited Supervision

1 INTRODUCTION . Disentangled representation learning aims to create low-dimensional representations of data that separate the underlying explanatory factors of variation in data . These representations provide an interpretable ( Sarhan et al. , 2019 ) and useful tool for various purposes , such as noise removal ( Lopez et al. , 2018 ) , continuous learning ( Achille et al. , 2018 ) , and visual reasoning ( van Steenkiste et al. , 2019 ) . However , there is no consensus about the exact properties that characterize a disentangled representation . Higgins et al . ( 2018 ) provide a formal definition for Symmetry-Based Disentangled ( SBD ) and Linearly SBD ( LSBD ) data representations , building upon the idea that representations should reflect the underlying structure of the data . In particular , they argue that variability in the data comes from transformations in the real world from which the data is observed . Having a formal definition of disentanglement can serve as a paradigm for the evaluation of disentangled representations . Although several methods have been proposed to learn SBD or LSBD representations , none of them provide a clear metric for quantifying the level of disentanglement in these representations . Quessard et al . ( 2020 ) introduce a loss term that measures the complexity of the transformations acting on their learned representations based on the number of parameters needed , but this term does not directly characterize disentanglement . Caselles-Dupré et al . ( 2019 ) only evaluate the performance of their learned representations when used in a particular downstream task . Moreover , existing methods require information about the transformation relationships among data points for the entire training dataset . This information is used to produce models that enforce the properties of SBD or LSBD representations and can be considered as a form of supervision . For example , this supervision can consist of the parameters of the transformation that connects a pair of data points , such as a rotation angle . Obtaining this supervision on the transformations for a dataset can be an expensive task that requires expert knowledge . In this work we focus on characterizing and quantifying LSBD and developing a method capable of obtaining LSBD representations by using a limited amount of supervision on the transformation properties of a dataset . The main contributions of this paper are : 1 . An easy-to-compute metric to quantify LSBD given certain assumptions ( see Section 4 ) , which acts as an upper bound to a more general metric ( derived in Appendix D ) . 2 . A partially supervised method to obtain LSBD representations that , during training , can also use data without supervision on the transformation relationships . 2 RELATED WORK . The concept of disentanglement comes from the intuition that data can be described in terms of a set of independent explanatory factors that constitute the variability of the data . In probabilistic modeling , these factors are interpreted as random independent unobserved latent variables that condition the data generation process ( Kulkarni et al. , 2015 ; Higgins et al. , 2016 ; Chen et al. , 2016 ; 2018 ) . Nevertheless , there is no consensus about the exact properties that disentangled representations should have . Evidence of this is the wide range of metrics used to characterize such representations ( Locatello et al. , 2018 ) . However , a reasonable expectation is that such representations should capture and separate the variations and properties of the data . Recent work has turned the attention to capturing not only the independent explanatory factors of a dataset , but also the transformations that those factors undergo in data generation . Examples of methods that attempt to produce representations with similar transformation properties to those of the explanatory factors of data are ( Cohen & Welling , 2015 ; Worrall et al. , 2017 ; Sosnovik et al. , 2019 ) . Transformations of the real world determine the variability of data and its structure . These socalled symmetry transformations have long been studied in Physics ( Gross , 1996 ) and have been formalized with group theory . Such transformations often affect only a subset of the properties that describe the real world and leave the rest invariant . The connection between disentanglement and symmetries has recently been formalized by Higgins et al . ( 2018 ) into the definitions of Symmetry-Based Disentangled ( SBD ) and Linearly SBD ( LSBD ) representations . Quessard et al . ( 2020 ) ; Caselles-Dupré et al . ( 2019 ) propose methods to obtain such SBD and/or LSBD representations , but their methods require supervision on the transformation relationships among datapoints for the entire training dataset . Moreover , there is no clear metric for the level of disentanglement in their methods . 3 SYMMETRY-BASED DISENTANGLEMENT . Higgins et al . ( 2018 ) provide a formal description of disentanglement that connects the symmetry transformations affecting the real world ( from which data is generated ) to the internal representations of a model . The definitions are grounded in concepts from group theory , for a more detailed description of these concepts please refer to Appendix A . These definitions assume the following setting . W is the set of possible world states , with underlying symmetry transformations that are described by a group G and its action · : G ×W → W on W . In particular , G can be decomposed as the direct product of K groups G = G1 × . . . × GK . Data is obtained via an observation function b : W → X that maps world states to observations in a data space X . A model ’ s internal representation of data is modeled with the inference function h : X → Z that maps data to the representation spaceZ . Together , the observation and the inference constitute the model ’ s internal representation of the real world f : W → Z with f ( w ) = h ◦ b ( w ) . The definitions for Symmetry-Based Disentangled ( SBD ) and Linearly SBD ( LSBD ) representations formalize the requirement that a model ’ s internal representation f : W → Z should reflect and disentangle the transformation properties of the real world . The definition of SBD can be found in Appendix B . In particular , our work focuses on LSBD representations in which the transformation properties of the model ’ s internal representations should be linear . The exact definition is as follows : Linearly Symmetry-Based Disentangled ( LSBD ) Representations A model ’ s internal representation f : W → Z , where Z is a vector space , is LSBD with respect to the group decomposition G = G1 × . . .×GK if • there is a decomposition of the representation space Z = Z1 ⊕ . . . ⊕ ZK into K vector subspaces , • there are group representations for each subgroup in the corresponding vector subspace ρk : Gk → GL ( Zk ) , k ∈ { 1 , . . . , K } • the group representation ρ : G→ GL ( Z ) for G in Z acts on Z as ρ ( g ) · z = ( ρ1 ( g1 ) · z1 , . . . , ρK ( gK ) · zK ) , ( 1 ) for g = ( g1 , . . . , gK ) ∈ G and z = ( z1 , . . . , zK ) ∈ Z with gk ∈ Gk and zk ∈ Zk . • the map f is equivariant with respect to the actions of G on W and Z , i.e . , for all w ∈W and g ∈ G it holds that f ( g · w ) = ρ ( g ) · f ( w ) . Symmetry-based disentanglement with respect to the data spaceX We do not directly observe W , only data observations in X , thus it is most practical to evaluate ( L ) SBD with respect to the inference map h : X → Z . However , the definitions for SBD and LSBD representations are given with respect to the world states representation map f : W → Z . If the action of G on W is regular and b is injective though , a representation f is ( L ) SBD if the inference h restricted to all possible observations is also ( L ) SBD . In this case , we can evaluate the same conditions for ( L ) SBD only for the function h. For a more detailed description on these conditions consult Appendix C . 4 ASSUMPTIONS . In this paper we assume a setting where an underlying group structure and its action on the data space and representation space are provided as expert knowledge , which allows us to provide a metric that quantifies Linear Symmetry-Based Disentanglement ( LSBD ) and a method to achieve it . Although these assumptions limit the settings in which our disentanglement metric and method can be used , we believe it is valuable because ( 1 ) it provides a good basis for future research into more general metrics and methods by making clear which assumptions should be relaxed , ( 2 ) it gives good insights into the effect of the level of supervision on transformations ( as shown by our results in Section 7 ) , and ( 3 ) there are many practical scenarios where having this type of expert knowledge is a reasonable assumption . More concretely , we make the following assumptions about the available information about the data and its transformations : 1 . Let W be the set of real world states . Let X be the data space , where observations are obtained . Let Z be the representation space where we wish to model this data ; Z is a vector space with given basis . 2 . We have a Lie group G and a regular action of G on W that describes the underlying symmetries ofW . The group decomposes as the direct product of groupsG = G1 × . . .×GK . 3 . The observation function b : W → X is injective . Therefore , we can define a group action forG on the image of b , i.e . · : G×b ( W ) → b ( W ) as g ·x = b ( g ·b−1 ( x ) ) ( see Appendix C for more details ) . This action is also regular . Practically , this implies that each observation is a transformed version of another observation , and this transformation is unique . 4 . We have an action · : G × Z → Z that is linear and has a known representation ρ : G → GL ( Z ) , i.e . for any g ∈ G we can determine ρ ( g ) . 5 . The representation ρ : G → GL ( Z ) is linearly disentangled , i.e . there is a known decomposition Z = Z1 ⊕ . . . ⊕ Zk such that ρ ( g ) · z = ( ρ1 ( g1 ) · z1 , . . . , ρK ( gK ) · zK ) , with gk ∈ Gk , zk ∈ Zk for k = 1 , . . . , K. 6 . There exists an embedded submanifold ZG ⊆ Z such that the action ofG on Z restricted to ZG is regular , and ZG is invariant under the action . Only ZG will be used as the codomain of the inference map , h : X → ZG . Example : the special orthogonal group SO ( 2 ) of 2D rotations Let ’ s look at an example that satisfies the assumptions above and can reasonably occur in practice . Consider a windmill that can be rotated around a fixed vertical axis , with blades that can rotate independently of the windmill ’ s orientation . The dataset consists of observations ( i.e . 2D images ) of this windmill , where the only variability is in the rotations of the windmill and of its blades , all other aspects ( e.g . camera angle , lighting conditions , object position , etc . ) are fixed . Thus , the variability in this dataset can be described as two independent 2D rotations , see Figure 1 . We can satisfy the assumptions above as follows : 1 . The world states W can be described by the angle of the entire windmill and the angle of the blades , these angles are independent of each other . The data space X is the space of images , i.e . arrays of pixel values . A suitable representation space Z will become apparent soon . 2 . It is well-known that 2D rotations can be described by the special orthogonal group SO ( 2 ) , so the transformations underlying this dataset can be described with the group decomposition G = G1 × G2 = SO ( 2 ) × SO ( 2 ) . The corresponding action is indeed regular ; any two world states are connected with one unique transformation ( i.e . pair of rotations ) .1 3 . If the windmill looks different from each angle , and the blades are always visible , then the observation function b : W → X is indeed injective . 4 . Each subgroup Gk = SO ( 2 ) has a representation ρk : SO ( 2 ) → GL ( R2 ) as rotation matrices for vectors in R2 , so we can use Z = Z1 ⊕ Z2 = R2 ⊕ R2 as the representation space , and the direct sum ρ = ρ1 ⊕ ρ2 as the group representation . 5 . Defined in this way , the representation ρ is linearly disentangled . 6 . The actions of the subgroups Gi on the subspaces Zi are not regular , but they become regular when restricted to the unit circle or 1-sphere S1 = { z ∈ R2 : ‖z‖ = 1 } , which is a submanifold of Zi = R2 and invariant under the action . Thus , we can use ZG = S1 × S1 as the codomain of the inference map h , to obtain a regular group action of G on ZG . 1Note that for a windmill with e.g . 4 identical blades , a rotation of π 2 radians corresponds to the identity transformation , and angles can be defined mod π 2 .

Summary: This paper follows on the work of Higgins et al. 2018 that used linear symmetry based disentangled (LSBD) representations where real-world transformations provide some structure in the data that can be leveraged. The main contribution of this paper is to provide a metric to measure the quality of disentanglement in the learned representation. The paper makes some assumptions on the samples in the dataset and assumes that there are some group actions that relate one data point to another. Under this assumption, the paper derives a simple and easy-to-compute disentanglement measure. Second, this paper shows a method that can work with partial supervision to learn disentangled representation. This is done by using some data with supervision on the underlying transformation between the data points, and another set of data points where no labeling information is given. The proposed method uses a diffusion variational autoencoder.