Theory and Evaluation Metrics for Learning Disentangled Representations

1 INTRODUCTION . Disentanglement learning holds the key for understanding the world from observations , transferring knowledge across different tasks and domains , generating novel designs , and learning compositional concepts ( Bengio et al. , 2013 ; Higgins et al. , 2017b ; Lake et al. , 2017 ; Peters et al. , 2017 ; Schmidhuber , 1992 ) . Assuming the observation x is generated from latent factors z via p ( x|z ) , the goal of disentanglement learning is to correctly uncover a set of independent factors { zi } that give rise to the observation . While there has been a considerable progress in recent years , common assumptions about disentangled representations appear to be inadequate ( Locatello et al. , 2019 ) . Unsupervised disentangling methods are highly desirable as they assume no prior knowledge about the ground truth factors . These methods typically impose constraints to encourage independence among latent variables . Examples of constraints include forcing the variational posterior q ( z|x ) to be similar to a factorial p ( z ) ( Burgess et al. , 2018 ; Higgins et al. , 2017a ) , forcing the variational aggregated prior q ( z ) to be similar to the prior p ( z ) ( Makhzani et al. , 2015 ) , adding total correlation loss ( Kim & Mnih , 2018 ) , forcing the covariance matrix of q ( z ) to be close to the identity matrix ( Kumar et al. , 2017 ) , and using a kernel-based measure of independence ( Lopez et al. , 2018 ) . However , it remains unclear how the independence constraint affects other properties of representation . Indeed , more independence may lead to higher reconstruction error in some models ( Higgins et al. , 2017a ; Kim & Mnih , 2018 ) . Worse still , the independent representations may mismatch human ’ s predefined concepts ( Locatello et al. , 2019 ) . This suggests that supervised methods – which associate a representation ( or a group of representations ) zi with a particular ground truth factor yk – may be more adequate . However , most supervised methods have only been shown to perform well on toy datasets ( Harsh Jha et al. , 2018 ; Kulkarni et al. , 2015 ; Mathieu et al. , 2016 ) in which data are generated from multiplicative combination of the ground truth factors . It is still unclear about their performance on real datasets . We believe that there are at least two major reasons for the current unsatisfying state of disentanglement learning : i ) the lack of a formal notion of disentangled representations to support the design of proper objective functions ( Tschannen et al. , 2018 ; Locatello et al. , 2019 ) , and ii ) the lack of robust evaluation metrics to enable a fair comparison between models , regardless of their architectures or design purposes . To that end , we contribute by formally characterizing disentangled representations along three dimensions , namely informativeness , separability and interpretability , drawing from concepts in information theory ( Section 2 ) . We then design robust quantitative metrics for these properties and argue that an ideal method for disentanglement learning should achieve high performance on these metrics ( Section 3 ) . We run a series of experiments to demonstrate how to compare different models using our proposed metrics , showing that the quantitative results provided by these metrics are consistent with visual results ( Section 4 ) . In the process , we gain important insights about some well-known disentanglement learning methods namely FactorVAE ( Kim & Mnih , 2018 ) , β-VAE ( Higgins et al. , 2017a ) , and AAE ( Makhzani et al. , 2015 ) . 2 RETHINKING DISENTANGLEMENT . Inspired by ( Bengio et al. , 2013 ; Ridgeway , 2016 ) , we adopt the notion of disentangled representation learning as “ a process of decorrelating information in the data into separate informative representations , each of which corresponds to a concept defined by humans ” . This suggests three important properties of a disentangled representation : informativeness , separability and interpretability , which we quantify as follows : Informativeness We formulate the informativeness of a particular representation ( or a group of representations ) zi w.r.t . the data x as the mutual information between zi and x : I ( x , zi ) = ∫ x ∫ z pD ( x ) q ( zi|x ) log q ( zi|x ) q ( zi ) dz dx ( 1 ) where q ( zi ) = ∫ x pD ( x ) q ( zi|x ) dx . In order to represent the data faithfully , a representation zi should be informative of x , meaning I ( x , zi ) should be large . Because I ( x , zi ) = H ( zi ) −H ( zi|x ) , a large value of I ( x , zi ) means that H ( zi|x ) ≈ 0 given that H ( zi ) can be chosen to be relatively fixed . In other words , if zi is informative w.r.t . x , q ( zi|x ) usually has small variance . It is important to note that I ( x , zi ) in Eq . 1 is defined on the variational encoder q ( zi|x ) , and does not require a decoder . It implies that we do not need to minimize the reconstruction error over x ( e.g. , in VAEs ) to increase the informativeness of a particular zi . Separability and Independence Two representations zi , zj are separable w.r.t . the data x if they do not share common information about x , which can be formulated as follows : I ( x , zi , zj ) = 0 ( 2 ) where I ( x , zi , zj ) denotes the multivariate mutual information ( McGill , 1954 ) between x , zi and zj . I ( x , zi , zj ) can be decomposed into standard bivariate mutual information terms as follows : I ( x , zi , zj ) = I ( x , zi ) + I ( x , zj ) − I ( x , ( zi , zj ) ) = I ( zi , zj ) − I ( zi , zj |x ) I ( x , zi , zj ) can be either positive or negative . It is positive if zi and zj contain redundant information about x . The meaning of a negative I ( x , zi , zj ) remains elusive ( Bell , 2003 ) . Achieving separability with respect to x does not guarantee that zi and zj are separable in general . zi and zj are fully separable or statistically independent if and only if : I ( zi , zj ) = 0 ( 3 ) If we have access to all representations z , we can generally say that a representation zi is fully separable ( from other representations z6=i ) if and only if I ( zi , z6=i ) = 0 . Note that there is a trade-off between informativeness , independence and the number of latent variables which we discuss in Appdx . A.7 . Interpretability Obtaining informative and independent representations does not guarantee interpretability by human ( Locatello et al. , 2019 ) . We argue that in order to achieve interpretability , we should provide models with a set of predefined concepts y . In this case , a representation zi is interpretable with respect to yk if it only contains information about yk ( given that zi is separable from all other z6=i and all yk are distinct ) . Full interpretability can be formulated as follows : I ( zi , yk ) = H ( zi ) = H ( yk ) ( 4 ) Eq . 4 is equivalent to the condition that zi is an invertible function of yk . If we want zi to generalize beyond the observed yk ( i.e. , H ( zi ) > H ( yk ) ) , we can change the condition in Eq . 4 into : I ( zi , yk ) = H ( yk ) or H ( yk|zi ) = 0 ( 5 ) which suggests that the model should accurately predict yk given zi . If zi satisfies the condition in Eq . 5 , it is said to be partially interpretable w.r.t yk . In real data , underlying factors of variation are usually correlated . For example , men usually have beard and short hair . Therefore , it is very difficult to match independent latent variables to different ground truth factors at the same time . We believe that in order to achieve good interpretability , we should isolate the factors and learn one at a time . 2.1 AN INFORMATION-THEORETIC DEFINITION OF DISENTANGLED REPRESENTATIONS . Given a dataset D = { xi } Ni=1 , where each data point x is associated with a set of K labeled factors of variation y = { y1 , ... , yK } . Assume that there exists a mapping from x to m groups of latent representations z = { z1 , z2 , ... , zm } which follows the distribution q ( z|x ) . Denoting q ( zi|x ) = ∑ z6=i q ( z|x ) and q ( zi ) = EpD ( x ) [ q ( zi|x ) ] . We define disentangled representations for unsupervised cases as follows : Definition 1 ( Unsupervised ) . A representation or a group of representations zi is said to be “ fully disentangled ” w.r.t a ground truth factor yk if zi is fully separable ( from z6=i ) and zi is fully interpretable w.r.t yk . Mathematically , this can be written as : I ( zi , z6=i ) = 0 and I ( zi , yk ) = H ( zi , yk ) ( 6 ) The definition of disentangled representations for supervised cases is similar as above except that now we model q ( z|x , y ) instead of q ( z|x ) and q ( z ) = ∑ x , y pD ( x , y ) q ( z|x , y ) . Recently , there have been several works ( Eastwood & Williams , 2018 ; Higgins et al. , 2018 ; Ridgeway & Mozer , 2018 ) that attempted to define disentangled representations . Higgin et . al . ( Higgins et al. , 2018 ) proposed a definition based on group theory ( Cohen & Welling , 2014 ) which is ( informally ) stated as follows : “ A representation z is disentangled w.r.t a particular subgroup yk ( from a symmetry group y = { yk } Kk=1 ) if z can be decomposed into different subspaces { zi } H i=1 in which the subspace zi should be independent of all other representation subspaces z6=i , and zi should only be affected by the action of a single subgroup yk and not by other subgroups y 6=k. ” . Their definition shares similar observation as ours . However , it is less convenient for designing models and metrics than our information-theoretic definition . Eastwood et . al . ( Eastwood & Williams , 2018 ) did not provide any explicit definition of disentangled representations but characterizing them along three dimensions namely “ disentanglement ” , “ compactness ” , and “ informativeness ” ( between z any yk ) . A high “ disentanglement ” score ( ≈ 1 ) for zi indicates that it captures at most one factor , let ’ s say yk . A high “ completeness ” score ( ≈ 1 ) for yk indicates that it is captured by at most one latent zj and j is likely to be i . A high “ informativeness ” score1 for yk indicates that all information of yk is captured by the representations z . Intuitively , when all the three notions achieve optimal values , there should be only a single representation zi that captures all information of the factor yk but no information from other factors y6=k . However , even in that case , zi is still not fully interpretable w.r.t yk since zi may contain some information in x that does not appear in yk . This makes their notions only applicable to toy datasets on which we know that the data x are only generated from predefined ground truth factors y = { yk } Kk=1 . Our definition can handle the situation where we only know some but not all factors of variation in the data . The notions in ( Ridgeway & Mozer , 2018 ) follow those in ( Eastwood & Williams , 2018 ) , hence , suffer from the same disadvantage . 1In ( Eastwood & Williams , 2018 ) , the authors consider the prediction error of yk given z instead . High “ informativeness ” score means this error should be close to 0 .

Summary: The paper presents a new set of metrics for evaluating disentangled representations in both supervised and unsupervised settings. Disentangled representations are evaluated along three dimensions: informativeness, separability, and interpretability. While previous work offers metrics for similar dimensions (e.g., (Eastwood & Williams, 2018)), the paper suggests that the metrics of the submission are superior to (Eastwood & Williams, 2018), based on a comparison between FactorVAE and Beta-VAE.