Evaluating Disentanglement of Structured Latent Representations

1 INTRODUCTION . A salient challenge in generative modeling is the ability to decompose the representation of images and scenes into distinct objects that are represented separately and then combined together . Indeed , the capacity to reason about objects and their relations is a central aspect of human intelligence ( Spelke et al. , 1992 ) , which can be used in conjunction with graph neural networks or symbolic solvers to enable relational reasoning . For instance , ( Wang et al. , 2018 ) exploit robot body structure to obtain competitive performance and transferability when learning to walk or swim in the Gym environments . ( Yi et al. , 2019 ) perform state-of-the-art visual reasoning by using an object based representation in combination with a symbolic model . In the last years , several models have been proposed to learn compositional representations in an unsupervised fashion : TAGGER ( Greff et al. , 2016 ) , NEM ( Greff et al. , 2017 ) , R-NEM ( Van Steenkiste et al. , 2018 ) , MONet ( Burgess et al. , 2019 ) , IODINE ( Greff et al. , 2019 ) , GENESIS ( Engelcke et al. , 2019 ) , Slot Attention ( Locatello et al. , 2020 ) , MulMON ( Li et al. , 2020 ) , SPACE ( Lin et al. , 2020 ) . They jointly learn to represent individual objects and to segment the image into meaningful components , the latter often being called `` perceptual grouping '' . These models share a number of common principles : ( i ) Splitting the latent representation into several slots that are meant to contain the representation of an individual object . ( ii ) Inside each slot , encoding information about both object position and appearance . ( iii ) Maintaining a symmetry between slots in order to respect the permutation invariance of objects composition . These mechanisms are intuitively illustrated in Figure 1 . To compare and select models , it is indispensable to have robust disentanglement metrics . At the level of individual factors of variations , a representation is said to be disentangled when information about the different factors is separated between different latent dimensions ( Bengio et al. , 2013 ; Locatello et al. , 2019 ) . At object-level , disentanglement measures the degree of object separation between slots . However , all existing metrics ( Higgins et al. , 2016 ; Chen et al. , 2018 ; Ridgeway and Mozer , 2018 ; Eastwood and Williams , 2018 ; Kumar et al. , 2017 ) are limited to the individual case , which disregards representation structure . To cite Kim and Mnih ( 2018 ) about the FactorVAE metric : The definition of disentanglement we use [ ... ] is clearly a simplistic one . It does not allow correlations among the factors or hierarchies over them . Thus this definition seems more suited to synthetic data with independent factors of variation than to most realistic datasets . As a result , prior work has restricted to measuring the degree of object separation via pixel-level segmentation metrics . Most considered is the Adjusted Rand ( ARI ) Index ( Rand , 1971 ; Greff et al. , 2019 ) , where image segmentation is viewed as a cluster assignment for pixels . Other metrics such as Segmentation Covering ( mSC ) ( Arbelaez et al. , 2010 ) have been introduced to penalize oversegmentation of objects . A fundamental limitation is that they do not evaluate directly the quality of the representation , but instead consider a visual proxy of object separation . This results in problematic dependence on the quality of the inferred segmentation masks , a problem first identified by Greff et al . ( 2019 ) for IODINE , and confirmed in our experimental study . To address these limitations , we introduce the first metric for evaluating disentanglement at individual hierarchy levels of a structured latent representation . Applied to object-centric generative models , this offers a systematic , unified approach to evaluating ( i ) object separation between latent slots ( ii ) disentanglement of object properties inside individual slots ( iii ) disentanglement of intrinsic and extrinsic object properties . We theoretically show that our framework gives stronger guarantees of representation quality than previous disentanglement metrics . Thus , it can safely substitute to them . We experimentally demonstrate the applicability of our metric to three architectures : MONet , GENESIS and IODINE . The results confirm issues with pixel-level segmentation metrics , and offer valuable insight about the representation learned by these models . Finally , as a core technical component , we present the first representation probing algorithm handling slot permutation invariance . 2 BACKGROUND . 2.1 DISENTANGLEMENT CRITERIA . There exists an extensive literature discussing notions of disentanglement , accounting for all the different definitions is outside the scope of this paper . We chose to focus on the three criteria formalized by Eastwood and Williams ( 2018 ) , which stand out because of their clarity and simplicity . Disentanglement is the degree to which a representation separates the underlying factors of variation , with each latent variable capturing at most one generative factor . Completeness is the degree to which each underlying factor is captured by a single latent variable . Informativeness is the amount of information that a representation captures about the underlying factors of variation . Similarly to prior work , the word disentanglement is also used as a generic term that simultaneously refers to these three criteria . It should be clear depending on the context whether it is meant as general or specific . 2.2 THE DCI FRAMEWORK . For brevity reasons , we only describe the DCI metrics of Eastwood and Williams ( 2018 ) , which are most closely related to our work . The supplementary material provides a comprehensive overview of alternative metrics . Consider a dataset X composed of n observations x1 , . . . , xn , which we assume are generated by combining F underlying factors of variation . The value of the different factors for observation xl are denoted vl1 , . . . , v l F . Suppose we have learned a representation z = ( z1 , . . . , zL ) from this dataset . The DCI metric is based on the affinity matrix R = ( Ri , j ) , where Ri , j measures the relative importance of latent zi in predicting the value of factor vj . Supposing appropriate normalization for R , disentanglement is measured as the weighted average of the matrix ’ row entropy . Conversely , completeness is measured as the weighted average of column entropy . Informativeness is measured as the normalized error of the predictor used to obtain the matrix R . 3 EVALUATING STRUCTURED DISENTANGLEMENT . 3.1 contains a high level description of the goals steering our framework . Our metric is formally described in 3.2 . In 3.3 , we present theoretical results showing that our framework can safely substitute to DCI since it provides stronger guarantees that the selected model is correctly disentangled . 3.1 PRESENTATION OF THE FRAMEWORK . Prior work has focused on measuring disentanglement at the level of individual factors of variation and latent dimensions ( that we will refer to as global or unstructured disentanglement ) . In the case of compositional representations with several objects slots , additional properties are desirable : 1 . Object-level disentanglement : Objects and latent slots should have a one-to-one matching . That is , changing one object should lead to changes in a single slot , and vice-versa . 2 . Slot disentanglement : Inside a given slot , properties of the object must be disentangled . 3 . Slot symmetry : The latent dimension responsible for a given factor ( e.g. , color ) should be invariant across slots . This means that all slots should have the same inner structure . Our structured disentanglement metric allows to evaluate all these criteria within a single unified framework . Similarly to DCI , it is based on the affinity matrix ( Rτ̂ , τ ) , where Rτ̂ , τ measures the relative importance of latent zτ̂ in predicting the value of factor vτ . The key novelty compared to DCI is that we propose to measure disentanglement with respect to arbitrary projections of the matrix R. Intuitively , projections correspond to a marginalization operation that selects a subset of hierarchy levels and discards the others . The coefficients Rτ̂ , τ that are projected together are summed to create group affinities . Figure 2 gives an intuitive illustration of this process on a toy example . With object-centric representations , projecting at the object/slot level allows to study the relation of objects and slots without taking their internal structure into consideration . Ultimately this permits to evaluate our object-level disentanglement criterion . Projecting at property level allows to study the internal slot representation independently of the object , for evaluation of both internal slot disentanglement and slot symmetry , with a single metric . The identity projection conserving all levels measures flat ( DCI ) disentanglement . This generalizes to arbitrary hierarchies in the representation , such as disentanglement of position and appearance , or intrinsic and extrinsic object properties . 3.2 MATHEMATICAL DEFINITION . To formalize our framework most conveniently , we propose to view the affinity matrix R = ( Rτ̂ , τ ) as a joint random variable ( X , Y ) , where X is a random latent dimension , and Y a random factor . Supposing that R is normalized to have sum one , this means P [ X = τ̂ , Y = τ ] = Rτ̂ , τ . This point of view , which is only implicit in prior work , allows to formalize the projections of R as a coupled marginalization operation ( ρ ( X ) , ρ ( Y ) ) , where ρ ( α1 , . . . , αh ) = ( αe1 , . . . , αel ) selects a subset of hierarchy levels . This yields a concise and elegant mathematical framework , where we can build on standard information-theoretic identities to derive theoretical properties . In the following , HU ( A|B ) denotes the conditional entropy of A with respect to B ( detailed definitions in the supplementary ) . Our disentanglement criteria are the following : Completeness with respect to projection ρ measures how well the projected factors are captured by a coherent group of latents , by measuring column entropy in the projected space . It is measured as C ( ρ ) = 1−HU ( ρ ( X ) |ρ ( Y ) ) , where U = |ρ ( T̂ ) | is the number of groups of latents in the projection . Disentanglement with respect to projection ρ measures to what extent a group of latent variables influences a coherent subset of factors , by measuring projected row entropy . It is measured as D ( ρ ) = 1−HV ( ρ ( Y ) |ρ ( X ) ) , where V = |ρ ( T ) | is the number of groups of factors in the projection . Informativeness does not depend on the projection . It is defined as the normalized error of a low-capacity model f that learns to predict the factor values v from the latents z , i.e . I = ‖f ( z ) − v‖2 ‖v‖2 . The changing log bases U and V aim at ensuring normalization between 0 and 1 . 3.3 ESTABLISHING TRUST IN OUR FRAMEWORK : A THEORETICAL ANALYSIS . Our disentanglement D ( ρ ) and completeness C ( ρ ) metrics depend on the subset of hierarchy levels contained in the projection ρ . We theoretically analyze the influence of the projection choice . It is especially interesting to study the behavior when distinct subsets of hierarchy levels are combined . Our key results are the following : Theorem 1 shows that our framework contains DCI as a special case , when the identity projection is chosen . Theorem 2 shows that the metrics associated to the identity projection can be decomposed in terms of one dimensional projections along a single hierarchy level . Together , these results formally show that one dimensional projections provide a sound substitute for prior unstructured metrics . This is very useful to build trust in our framework . Theorem 1 ( Relation with DCI ) . The DCI metrics of Eastwood and Williams ( 2018 ) are a special case of our framework , with the identity projection conserving all hierarchy levels . Theorem 2 uses the intuitive notion of decomposition of a projection . The projection ρ is said to be decomposed into ρ1 , . . . ρk if the set of hierarchy levels selected by ρ is the disjunct union of the set of levels selected by ρ1 , . . . ρk . In the case of object-centric latent representations , the identity projection ρid that keeps all hierarchy levels { object , property } can be decomposed as ρobject considering only { object } and ρproperty considering only { property } . Theorem 2 ( Decomposition of a projection ) . Consider a decomposition of the projection ρ ( with L latent groups ) into disjunct projections ρ1 , . . . ρk ( with respectively L1 , . . . , Lk latent groups ) . The following lower bound for the joint completeness ( resp . disentanglement ) holds 1− k + k∑ s=1 C ( ρs ) ≤ 1− k∑ s=1 1− C ( ρs ) logLs ( L ) ≤ C ( ρ ) . Suppose that all one dimensional projections verify C ( ρs ) ≥ 1 − , where ≥ 0 . We obtain the lower bound C ( ρid ) ≥ ( 1− k ) + k ( 1− ) = 1− k for the identity projection . For object-centric representations , this implies that when object-level completeness and property completeness are both perfect ( that is , = 0 ) , then DCI completeness is also perfect . The same works for disentanglement . The supplementary materials contains detailed proofs , a matching upper bound , as well as an explicit formula in the special case k = 2 .

Summary: This paper introduces a new metric for evaluating disentanglement of latent representations, which -- unlike prior work -- supports object-centric, structured latent spaces with a set of latent variables (as opposed to a single global latent variable). While prior work on object-centric generative models primarily evaluated their ability to decompose visual scenes into spatial regions corresponding to individual objects, this novel metric allows for systematic evaluation of disentanglement of learned object properties within and across "object slots" (i.e. individual latent variables) in these models. The paper further introduces a representation probing algorithm using an EM-style iterative procedure, which is invariant to slot permutation. The metric is experimentally demonstrated on standard unsupervised scene decomposition datasets and models.