Quantifying and Learning Linear Symmetry-Based Disentanglement

The definition of Linear Symmetry-Based Disentanglement ( LSBD ) formalizes1 the notion of linearly disentangled representations , but there is currently no metric2 to quantify LSBD . Such a metric is crucial to evaluate LSBD methods and to3 compare to previous understandings of disentanglement . We propose DLSBD , a4 mathematically sound metric to quantify LSBD , and provide a practical implemen-5 tation . Furthermore , from this metric we derive LSBD-VAE , a semi-supervised6 method to learn LSBD representations . We demonstrate the utility of our metric7 by showing that ( 1 ) common VAE-based disentanglement methods don ’ t learn8 LSBD representations , ( 2 ) LSBD-VAE as well as other recent methods can learn9 LSBD representations , needing only limited supervision on transformations , and10 ( 3 ) various desirable properties expressed by existing disentanglement metrics are11 also achieved by LSBD representations.12 1 Introduction13 Learning low-dimensional representations that disentangle the underlying factors of variation in data14 is considered an important step towards interpretable machine learning with good generalization . To15 address the fact that there is no consensus on what disentanglement entails and how to formalize it,16 Higgins et al . ( 2018 ) propose a formal definition for Linear Symmetry-Based Disentanglement , or17 LSBD , arguing that underlying real-world symmetries give exploitable structure to data.18 However , there is currently no metric to quantify LSBD . Such a metric is crucial to properly evaluate19 methods aiming to learn LSBD representations and to relate LSBD to previous definitions of disentan-20 glement . Although previous works have evaluated LSBD by measuring performance on downstream21 tasks ( Caselles-Dupré et al. , 2019 ) or by measuring specific traits related to LSBD ( Painter et al.,22 2020 ; Quessard et al. , 2020 ) , none of these evaluation methods directly quantify LSBD according to23 its well-formalized definition.24 We propose DLSBD , a well-formalized and generally applicable metric that quantifies the level of25 LSBD in learned data representations . We show an intuitive justification of this metric , as well26 as its theoretical derivation . We also provide a practical implementation to compute DLSBD for27 common symmetry groups . Furthermore , we show that our metric formulation can be used to derive28 a semi-supervised method to learn LSBD representations , which we call LSBD-VAE . To make29 LSBD-VAE more widely applicable , we also demonstrate how to disentangle symmetric properties30 from other non-symmetric properties , and how to quantify this disentanglement with DLSBD.31 We show the utility of DLSBD by quantifying LSBD in a number of settings , for a variety of datasets32 with underlying SO ( 2 ) symmetries and other non-symmetric properties . First , we evaluate common33 VAE-based disentanglement methods and show that most don ’ t learn LSBD representations . Second,34 we evaluate LSBD-VAE and other recent methods that specifically target LSBD , showing that they35 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . can obtain much better DLSBD scores while needing only limited supervision on transformations.36 Third , we compare DLSBD with existing disentanglement metrics , showing that various desirable37 properties expressed with these metrics are also achieved by LSBD representations.38 2 Related Work39 Plenty of works have focused on learning and quantifying disentangled representations recently , but40 research has shown that there is little consensus about the exact definition of disentanglement and41 methods often do not achieve it as well as they proclaim ( Locatello et al. , 2019 ) . To introduce some42 much-needed formalization , Higgins et al . ( 2018 ) proposed to define disentanglement with respect43 to symmetry transformations acting on the data . They used group theory to provide two formal44 definitions , which we refer to as ( Linear ) Symmetry-Based Disentanglement , or ( L ) SBD . In this45 paper we focus only on LSBD , not SBD.46 Several methods have been proposed to learn LSBD representations ( Caselles-Dupré et al. , 2019 ; 47 Painter et al. , 2020 ; Quessard et al. , 2020 ) . These methods also learn to represent the transformations48 acting on the input data , assuming various levels of supervision on these transformations . Other49 methods have previously focused on capturing transformations of the data outside the context of50 disentanglement as well ( Cohen and Welling , 2015 ; Sosnovik et al. , 2019 ; Worrall et al. , 2017 ) .51 3 Linear Symmetry-Based Disentanglement52 Higgins et al . ( 2018 ) provide a formal definition of linear disentanglement that connects symmetry53 transformations affecting the real world ( from which data is observed ) to the internal representations54 of a model . The definition is grounded in concepts from group theory , we provide a more detailed55 description of these concepts in the Supplementary Material.56 The definition1 considers a group G of symmetry transformations acting on the data space X through57 the group action · : G ×X → X . In particular , G can be decomposed as the direct product of K58 groups G = G1 × . . .×GK . A model ’ s internal representation of data is modeled with the encoding59 function h : X → Z that maps data to the embedding space Z . The definition for Linearly Symmetry-60 Based Disentangled ( LSBD ) representations then formalizes the requirement that a model ’ s encoding61 h should reflect and disentangle the transformation properties of the data , and that the transformation62 properties of the model ’ s encoding should be linear . The exact definition is as follows:63 Definition : Linear Symmetry-Based Disentanglement ( LSBD ) A model ’ s encoding map h :64 X → Z , where Z is a vector space , is LSBD with respect to the group decomposition G =65 G1 × . . .×GK if66 1. there is a decomposition of the embedding space Z = Z1 ⊕ . . . ⊕ ZK into K vector67 subspaces,68 2. there are group representations for each subgroup in the corresponding vector subspace69 ρk : Gk → GL ( Zk ) , k ∈ { 1 , . . . , K } 70 3. the group representation ρ : G→ GL ( Z ) acts on Z as71 ρ ( g ) · z = ( ρ1 ( g1 ) · z1 , . . . , ρK ( gK ) · zK ) , ( 1 ) for g = ( g1 , . . . , gK ) ∈ G and z = ( z1 , . . . , zK ) ∈ Z with gk ∈ Gk and zk ∈ Zk.72 4. the map h is equivariant with respect to the actions of G on X and Z , i.e . , for all x ∈ X73 and g ∈ G it holds that h ( g · x ) = ρ ( g ) · h ( x ) .74 Furthermore , we say that a group representation ρ is linearly disentangled with respect to the group75 decomposition G = G1 × . . .×GK if it satisfies criteria 1 to 3 from the LSBD definition above.76 1The original definition actually considers an additional set of world states W , but our definition is more practical and can be shown to be the same under mild conditions , see Supplementary Material . 4 Quantifying LSBD : DLSBD77 4.1 Intuition : Measuring Equivariance with Dispersion78 Figure 1 : A dataset of images from a rotating object expressed in terms of the group G = SO ( 2 ) acting on a base image x0 . It is possible to quantify the level of LSBD of an encoding map h by measuring its equivariance with respect to a group representation ρ . Since all data has been generated from x0 , equivariance can be measured as the dispersion of the points { ρ ( g−1n ) · h ( xn ) } Nn=1 . To motivate our metric , let ’ s first assume a set-79 ting in which a suitable linearly disentangled80 group representation ρ is known . Let ’ s further81 assume that the dataset of observations can be82 expressed with respect toG acting on some base83 point x0 ∈ X , i.e . { xn } Nn=1 = { gn · x0 } Nn=1.84 Formally , this assumes that the action of G on85 X is regular . In this case , we can use the inverse86 group elements g−1n to transform each data point87 toward the base point x0 , i.e.88 x0 = g −1 1 · x1 = . . . = g −1 N · xN . ( 2 ) Since ρ is linearly disentangled , we only need to89 measure the equivariance of the encoding map90 h to quantify LSBD . Equivariance is achieved91 when h ( g · x ) = ρ ( g ) · h ( x ) , for all g ∈ G , x ∈92 X . Given the dataset described above , we can93 check this property for x ∈ { xn } Nn=1 and g ∈94 { gn } Nn=1.2 In particular , from Equation ( 2 ) we95 can see that we have equivariance if96 h ( x0 ) = ρ ( g −1 1 ) · h ( x1 ) = . . . = ρ ( g −1 N ) · h ( xN ) . ( 3 ) This not only characterizes perfect equivariance , but also allows for an efficient way to quantify how97 close we are to true equivariance , by measuring the dispersion of the points { ρ ( g−1n ) · h ( xn ) } Nn=1.398 Given a suitable norm ‖ · ‖Z in Z , we can thus quantify LSBD in this setting as99 1 N N∑ n=1 ∥∥∥∥∥ρ ( g−1n ) · h ( xn ) − 1N N∑ n′=1 ρ ( g−1n′ ) · h ( xn′ ) ∥∥∥∥∥ 2 Z , ( 4 ) i.e . we compute the mean of { ρ ( g−1n ) · h ( xn ) } Nn=1 and use the average squared distance to this mean100 for points in { ρ ( g−1n ) · h ( xn ) } Nn=1 as our LSBD metric , see Figure 1.101 However , this formulation requires knowing the right linearly disentangled group representation and102 a suitable norm in Z . Moreover , it implicitly assumes a uniform probability measure over the group103 elements { gn } Nn=1 . In the next section we formulate our metric for a more general setting.104 4.2 DLSBD : A Metric for LSBD105 Generalizing the ideas from the previous section with concepts from measure theory , we propose a106 metric to measure the level of LSBD of any encoding h : X → Z given a data probability measure µ107 on X , provided that µ can be written as the pushforward GX ( · , x0 ) # ν of some probability measure108 ν on G by the function GX ( · , x0 ) for some base point x0 . More formally,109 µ ( A ) = GX ( · , x0 ) # ν ( A ) = ν ( { g ∈ G | GX ( g , x0 ) ∈ A } ) , ( 5 ) for Borel subsets A ⊂ X . Note that this is only possible if the action GX is transitive.110 For example , the situation of a dataset with N datapoints { xn } Nn=1 = { gn · x0 } Nn=1 corresponds to111 the case in which ν and µ are empirical measures on the group G and data space X , respectively:112 ν : = 1 N N∑ i=1 δgi , µ : = 1 N N∑ i=1 δxi . ( 6 ) 2Note that { gn } Nn=1 can be used to describe all known group transformations between elements in the dataset by means of composition and inverses , since xi = gi · ( g−1j · xj ) . Thus it suffices to check equivariance for these N group transformations . 3Note that we do not actually need to know x0 nor h ( x0 ) . We define the metric DLSBD for an encoding h and a measure µ as113 DLSBD : = inf ρ∈P ( G , Z ) ∫ G ∥∥ρ ( g ) −1 · h ( g · x0 ) −Mρ , h , x0∥∥2ρ , h , µ dν ( g ) , with Mρ , h , x0 = ∫ G ρ ( g′ ) −1 · h ( g′ · x0 ) dν ( g′ ) , ( 7 ) where the norm ‖·‖ρ , h , µ is a Hilbert-space norm depending on the representation ρ , the encoding map114 h : X → Z , and the data measure µ . More details of this norm can be found in the Supplementary115 Material . Moreover , P ( G , Z ) denotes the set of linearly disentangled representations of G in Z.116 Lower values of DLSBD indicate better disentanglement , zero being optimal.117 4.3 Practical Computation of DLSBD118 There are two main challenges for computing the metric of Equation ( 7 ) . First , to calculate the119 integrals in the formula , all possible datapoints that can be expressed as g · x0 with g ∈ G =120 G1 × · · · × GK must be available . Second , the infimum of the integrals over all possible linearly121 disentangled representations must be estimated . This requires finding the possible invariant subspaces122 Z = Z1⊕· · ·⊕ZK induced by the encoding h over which the group representations are disentangled.123 We present a practical implementation of an upper bound to DLSBD for an encoding function h given124 a dataset X generated by some known group transformations . In particular , this approximation of125 DLSBD is designed for a group decomposition G = G1 × · · · ×GK where each Gk = SO ( Dk ) with126 k ∈ { 1 , . . . , K } the group of rotations in Dk dimensions . This implementation approximates the127 integrals of Equation ( 7 ) by using the empirical distribution of X . The invariant subspaces of Z to the128 subgroup actions are found by applying a suitable change of basis . In the new basis , the disentangled129 group representations are expressed in a parametric form whose parameters are optimized to find the130 tightest bound to DLSBD . See Figure 2 for an intuitive description of the process.131 Assume there is a datasetX that can be modeled in terms of the group decompositionG = G1×· · ·Gk.132 For each Gk subgroup there is a set of known group elements Gk ⊆ Gk uniformly sampled such133 that the dataset is described in terms of all elements in G = G1 × · · · × GK and a base point x0 as134 X = { ( g1 , . . . , gK ) · x0|gk ∈ Gk , k ∈ { 1 , . . . , K } } .135 For each subgroup Gk we construct a set of encoded data Zk ⊆ Z whose variability should only de-136 pend on the action of Gk . The set Zk is given by Zk = { zk ( g1 , . . . , gK ) |gj ∈ Gj , j ∈ { 1 , . . . , K } } ,137 in which138 zk ( g1 , . . . , gK ) = h ( ( g1 , . . . , gK ) · x0 ) − 1 |Gk| ∑ g′∈Gk h ( ( g1 , . . . , gk−1 , g ′ , gk+1 , . . . , gK ) · x0 ) . ( 8 ) Similar to ( Cohen and Welling , 2014 ) , we find a suitable change of basis that exposes the invariant139 subspace Zk corresponding to the k-th subgroup Gk . The new basis is obtained from the eigenvectors140 resulting from applying Principal Component Analysis ( PCA ) to Zk . Each element in Zk is projected141 into the first Dk eigenvectors . The new set is denoted as Z ′k ⊆ RDk with elements z′k ( g1 , . . . , gK ) ⊆142 RDk that are the projected versions of zk ( g1 , . . . , gK ) .143 ( Quessard et al. , 2020 ) describes how one could parameterize the subgroup representations of SO ( Dk ) 144 for arbitrary Dk but here we will focus on Gk = SO ( 2 ) . In this case , we can parameterize each145 subgroup representation in terms of a single integer parameter ω ∈ Z as ρk , ω ( gk ) corresponding146 to a 2 × 2 rotation matrix whose angle of rotation is ω multiplied by the known angle associated147 to the group element gk ∈ Gk = SO ( 2 ) . For this subgroup we can approximate the Mρ , h , x0 from148 Equation ( 7 ) as Mk , ω given by149 Mk , ω = 1 |G| ∑ ( g1 , ... , gK ) ∈G ρk , ω ( g −1 k ) · z ′ ( g1 , . . . , gK ) . ( 9 ) Similar to Equation ( 7 ) we would like to find the optimal ρk , ω that minimizes the integral over the150 group representations . We can define a parameter search space Ω ⊆ Z , e.g . Ω = [ −10 , 10 ] for finding151 the optimal ω ∈ Ω that minimizes the dispersion , this is expressed in the following equation152 D ( k ) LSBD = min ω∈Ω 1 |G| ∑ ( g1 , ... , gK ) ∈G ‖ρk , ω ( g−1k ) · z ′ ( g1 , . . . , gK ) −Mk , ω‖2 . ( 10 ) EachD ( k ) LSBD measures the degree of equivariance of the projected embeddings for each k-th subgroup153 corresponding to the best fitting group representation . The upper bound to the metric is finally obtained154 by averaging across all subgroups DLSBD ≤ 1K ∑K k=1D ( k ) LSBD.155 5 Learning LSBD Representations : LSBD-VAE156 In this section we present LSBD-VAE , a semi-supervised VAE-based method to learn LSBD repre-157 sentations . The main idea is to train an unsupervised Variational Autoencoder ( VAE ) ( Kingma and158 Welling , 2014 ; Rezende et al. , 2014 ) with a suitable latent space topology , and use our metric as an159 additional loss term for batches of transformation-labeled data.160 Assumptions LSBD-VAE requires some knowledge about the group structure G that is to be161 disentangled . Concretely , the group and its decomposition G = G1 × . . .×GK should be known,162 as well as a suitable linearly disentangled group representation ρ : G→ GL ( Z ) and a latent space163 Z = Z1 ⊕ . . . ⊕ ZK . Moreover , we assume there exists an embedded submanifold ZG ⊆ Z such164 that the action of G on Z restricted to ZG is regular , and ZG is invariant under the action . Only ZG165 will then be used as the codomain for the encoding map , h : X → ZG.166 We demonstrate the assumptions above for the common group structure G = SO ( 2 ) × SO ( 2 ) . For167 the group representation ρ = ρ1 ⊕ ρ2 , with Z = R2 ⊕ R2 , we can use rotation matrices in R2 for168 ρ1 and ρ2 . We can then use 1-spheres S1 = { z ∈ R2 : ‖z‖ = 1 } for the embedded submanifold:169 ZG = S 1 × S1 . In this case , the action of G on Z restricted to ZG is indeed regular , and ZG is170 invariant under the action.171 Unsupervised Learning on Latent Manifold To learn encodings only on the latent manifold172 ZG , we use a Diffusion Variational Autoencoder ( ∆VAE ) ( Perez Rey et al. , 2020 ) . ∆VAEs can173 use any closed Riemannian manifold embedded in a Euclidean space as a latent space ( or latent174 manifold ) , provided that a certain projection function from the Euclidean embedding space into the175 latent manifold is known and the scalar curvature of the manifold is available . The ∆VAE uses176 a parametric family of posterior approximates obtained from a diffusion process over the latent177 manifold . To estimate the intractable terms of the negative ELBO , the reparameterization trick is178 implemented via a random walk.179 In the case of S1 as a latent ( sub ) manifold , we consider R2 as the Euclidean embedding space , and180 the projection function4 Π : R2 → S1 normalizes points in the embedding space : Π ( z ) = z/|z| . The181 scalar curvature of S1 is 0.182 4This projection function is not defined for z = 0 , but this value does not occur in practice . Semi-Supervised Learning with Transformation Labels Caselles-Dupré et al . ( 2019 ) proved183 that LSBD representations can not be inferred from a training set of unlabeled observations , but that184 access to the transformations between data points is needed . They therefore use a training set of185 observation pairs with a given transformation between them.186 However , we posit that only a limited amount of supervision is sufficient . Since obtaining supervision187 on transformations is typically more expensive than obtaining unsupervised observations , it is188 desirable to limit the amount of supervision needed.189 Therefore , we augment the un-190 supervised ∆VAE with a super-191 vised method that makes use of192 transformation-labeled batches , i.e.193 batches { xm } Mm=1 such that xm =194 gm · x1 for m = 2 , . . . , M , where195 the transformations gm ( and thus196 their group representations ρ ( gm ) ) are197 known and are referred to as transfor-198 mation labels . The simplified version199 of the metric from Equation ( 4 ) can200 then be used for each batch as an ad-201 ditional loss term ( with x0 = x1 ) , as202 it is differentiable under the assump-203 tions described above ( using the Eu-204 clidean norm ) .205 We make a small adjustment to Equation ( 4 ) for the purpose of our method , since the mean computed206 there does not typically lie on the latent manifold ZG . Thus , we use the projection Π from the ∆VAE207 to project the mean onto ZG . Writing the encodings as zm : = h ( xm ) , the additional loss term for a208 transformation-labeled batch { xm } Mm=1 then becomes209 LLSBD = 1 M M∑ m=1 ∥∥∥∥∥ρ ( g−1m ) · zm −Π ( 1 M M∑ m=1 ρ ( g−1m ) · zm ) ∥∥∥∥∥ 2 , ( 11 ) where g1 = e , the group identity.210 Moreover , instead of feeding the encodings zm to the decoder , we use ρ ( gm ) · z , where z =211 Π ( 1 M ∑M m=1 ρ ( g −1 m ) · zm ) . This encourages the decoder to follow the required group structure.212 This only affects the reconstruction loss component of the ∆VAE.213 Figure 3 illustrates the supervised part of our method for a transformation-labeled batch { xm } Mm=1.214 The loss function is the regular ELBO ( but with adjusted decoder input as described above ) as used215 in ∆VAE plus an additional term γ · LLSBD , where γ is a weight hyperparameter to control the216 influence of the supervised loss component . By alternating unsupervised and supervised training217 ( using the same encoder and decoder ) , we have a method that makes use of both unlabeled and218 transformation-labeled observations.219 6 Experimental Setup220 We evaluate the disentanglement of several models on three different image datasets ( Square , Arrow,221 and Airplane ) with a known group decomposition G = SO ( 2 ) × SO ( 2 ) describing the underlying222 transformations . For each subgroup a fixed number of |Gk| = 64 with k ∈ { 1 , 2 } transformations is223 selected . The datasets exemplify different group actions of SO ( 2 ) : periodic translations , in-plane224 rotations , out-of-plane rotations , and periodic hue-shifts.225 In real settings , not all variability in the data can be modelled by the actions of a group . Therefore,226 we also evaluate the same models on two datasets ModelNet40 ( Wu et al. , 2014 ) and COIL-100227 ( Nene et al. , 1996 ) that consist of images from various objects ( i.e . non-symmetric variation ) under228 known out-of-plane rotations ( SO ( 2 ) symmetries ) . In many settings it is easy to obtain labels for229 such rotations , e.g . when the camera or object angle is controlled by an agent . See Figure 4 for230 examples of the datasets . For more details , see the Supplementary Material.231 For the Square , Arrow , and Airplane datasets we test LSBD-VAE with transformation-labeled batches232 of size M = 2 . More specifically , for each experiment we randomly select L disjoint pairs of data233 points , and label the transformation between the data points in each pair . We vary the number of234 labeled pairs L from 0 ( corresponding to a ∆VAE ) to N/2 ( in which case each data point is involved235 in exactly one labeled pair ) . We set the weight γ of the supervised loss component to γ = 100236 for all experiments . We choose M = 2 for our experiments since it is the most limited setting for237 LSBD-VAE . Higher values of M would provide stronger supervision , so successful results with238 M = 2 imply that good results can also be achieved for higher values of M ( but not necessarily vice239 versa ) .240 For the COIL-100 and ModelNet40 datasets , we train LSBD-VAE on batches containing images of241 one particular object from all different angles ( 72 and 64 for COIL-100 and ModelNet40 , respectively ) .242 Each batch is labelled with transformations ( g1 , e ) , . . . , ( gM , e ) , where gm represent rotations , and243 the unit transformation e indicates that the object is unchanged . To represent the rotations we use244 a S1 latent space as in ∆VAE , whereas for the object identity we use a 5-dimensional Euclidean245 space with standard Gaussian prior as in regular VAEs . LSBD is measured as the disentanglement of246 rotations in the latent space . For these experiments we used γ = 1.247 We furthermore test a number of known disentanglement methods for comparison , including tra-248 ditional disentanglement methods as well as methods focusing on LSBD . In particular , we use249 disentanglement_lib ( Locatello et al. , 2019 ) to train a regular VAE ( Kingma and Welling , 2014 ; 250 Rezende et al. , 2014 ) , β-VAE ( Higgins et al. , 2017 ) , CC-VAE ( Burgess et al. , 2018 ) , FactorVAE251 ( Kim and Mnih , 2018 ) , and DIP-VAE-I/II ( Kumar et al. , 2018 ) . Furthermore we evaluate the method252 from Quessard et al . ( 2020 ) that focuses on LSBD . We also tested ForwardVAE ( Caselles-Dupré253 et al. , 2019 ) , but show only limited results since we were not able to reproduce any reasonable results254 for our datasets.255 We use encodings from all these methods to evaluate DLSBD , as well as common traditional disentan-256 glement metrics from disentanglement_lib : Beta ( Higgins et al. , 2017 ) , Factor ( Kim and Mnih,257 2018 ) , SAP ( Kumar et al. , 2018 ) , DCI Disentanglement ( Eastwood and Williams , 2018 ) , Mutual258 Information Gap ( MIG ) ( Chen et al. , 2018 ) , and Modularity ( MOD ) ( Ridgeway and Mozer , 2018 ) .259 More information about the architecture , epochs and hyperparameters can be found in the Supplemen-260 tary Material . For the traditional disentanglement methods trained on Square , Arrow and Airplane261 datasets the latent spaces have 4 dimensions , since these are the minimum number of dimensions262 necessary to learn LSBD representations for an underlying SO ( 2 ) × SO ( 2 ) symmetry group , see263 ( Higgins et al. , 2018 ; Caselles-Dupré et al. , 2019 ) . For COIL-100 and ModelNet40 we use latent264 spaces with 7 dimensions for a fair comparison with the LSBD-VAE method.265 7 Results : Evaluating LSBD with DLSBD266 We now highlight four key observations from our experimental results . In particular , we differentiate267 between the methods ( VAE , β-VAE , CC-VAE , FACTOR , DIP-I , DIP-II ) and metrics ( BETA,268 FACTOR , SAP , DCI , MIG , MOD ) that approach disentanglement in the traditional sense , and269 methods ( ∆VAE , QUESSARD , LSBD-VAE ) and metric ( DLSBD ) that focus specifically on LSBD.270 The full quantitative results can be found in the Supplementary Material.271 7.1 Standard Disentanglement Methods Don ’ t Learn LSBD Representations272 Figure 5 summarizes the DLSBD scores ( lower is better ) for all methods on all datasets . Bars show273 the mean scores over 10 runs for each method , the vertical lines represent standard deviations.274 LSBD-VAE/L indicates our method trained on L labelled pairs ( LSBD-VAE/0 corresponds to the275 unsupervised ∆VAE ) , LSBD-VAE/full indicates our method trained on batches containing a single276 object in all known transformations ( for datasets with non-symmetric variation ) . Note that LSBD-277 VAE obtained very good scores ( nearly 0 ) on the Arrow and Pixel datasets , hence the missing278 bars.279

Summary: Summary:  This paper aims to operationalize the symmetry-based disentanglement idea proposed in Higgins et al., 2018, and proposes a novel loss function that can be used to both evaluate and learn disentangled representations (including learning from datasets where only a subset of samples are fully labeled). The paper includes some experimental validation of the proposed ideas on synthetic datasets. It also includes cross-comparisons between both the LSBD-based metric and method to state of the art disentangling metrics and methods, which demonstrate the complementary aspects of the LSBD method (e.g., its ability to capture multi-dimensional disentangled factors, for example rotations in a euclidean plane).  Overall, the paper is clearly written and makes a positive contribution in pushing for disentangling metrics that are motivated by a novel formalism. The impact of this paper is slightly limited due to the strength of supervision required (full supervision of all factors of variation) for implementing both the D_LSBD metric and the loss.