Evaluating Disentanglement of Structured Latent Representations

1 INTRODUCTION . A salient challenge in generative modeling is the ability to decompose the representation of images and scenes into distinct objects that are represented separately and then combined together . Indeed , the capacity to reason about objects and their relations is a central aspect of human intelligence ( Spelke et al. , 1992 ) , which can be used in conjunction with graph neural networks or symbolic solvers to enable relational reasoning . For instance , ( Wang et al. , 2018 ) exploit robot body structure to obtain competitive performance and transferability when learning to walk or swim in the Gym environments . ( Yi et al. , 2019 ) perform state-of-the-art visual reasoning by using an object based representation in combination with a symbolic model . In the last years , several models have been proposed to learn compositional representations in an unsupervised fashion : TAGGER ( Greff et al. , 2016 ) , NEM ( Greff et al. , 2017 ) , R-NEM ( Van Steenkiste et al. , 2018 ) , MONet ( Burgess et al. , 2019 ) , IODINE ( Greff et al. , 2019 ) , GENESIS ( Engelcke et al. , 2019 ) , Slot Attention ( Locatello et al. , 2020 ) , MulMON ( Li et al. , 2020 ) , SPACE ( Lin et al. , 2020 ) . They jointly learn to represent individual objects and to segment the image into meaningful components , the latter often being called `` perceptual grouping '' . These models share a number of common principles : ( i ) Splitting the latent representation into several slots that are meant to contain the representation of an individual object . ( ii ) Inside each slot , encoding information about both object position and appearance . ( iii ) Maintaining a symmetry between slots in order to respect the permutation invariance of objects composition . These mechanisms are intuitively illustrated in Figure 1 . To compare and select models , it is indispensable to have robust disentanglement metrics . At the level of individual factors of variations , a representation is said to be disentangled when information about the different factors is separated between different latent dimensions ( Bengio et al. , 2013 ; Locatello et al. , 2019 ) . At object-level , disentanglement measures the degree of object separation between slots . However , all existing metrics ( Higgins et al. , 2016 ; Chen et al. , 2018 ; Ridgeway and Mozer , 2018 ; Eastwood and Williams , 2018 ; Kumar et al. , 2017 ) are limited to the individual case , which disregards representation structure . To cite Kim and Mnih ( 2018 ) about the FactorVAE metric : The definition of disentanglement we use [ ... ] is clearly a simplistic one . It does not allow correlations among the factors or hierarchies over them . Thus this definition seems more suited to synthetic data with independent factors of variation than to most realistic datasets . As a result , prior work has restricted to measuring the degree of object separation via pixel-level segmentation metrics . Most considered is the Adjusted Rand ( ARI ) Index ( Rand , 1971 ; Greff et al. , 2019 ) , where image segmentation is viewed as a cluster assignment for pixels . Other metrics such as Segmentation Covering ( mSC ) ( Arbelaez et al. , 2010 ) have been introduced to penalize oversegmentation of objects . A fundamental limitation is that they do not evaluate directly the quality of the representation , but instead consider a visual proxy of object separation . This results in problematic dependence on the quality of the inferred segmentation masks , a problem first identified by Greff et al . ( 2019 ) for IODINE , and confirmed in our experimental study . To address these limitations , we introduce the first metric for evaluating disentanglement at individual hierarchy levels of a structured latent representation . Applied to object-centric generative models , this offers a systematic , unified approach to evaluating ( i ) object separation between latent slots ( ii ) disentanglement of object properties inside individual slots ( iii ) disentanglement of intrinsic and extrinsic object properties . We theoretically show that our framework gives stronger guarantees of representation quality than previous disentanglement metrics . Thus , it can safely substitute to them . We experimentally demonstrate the applicability of our metric to three architectures : MONet , GENESIS and IODINE . The results confirm issues with pixel-level segmentation metrics , and offer valuable insight about the representation learned by these models . Finally , as a core technical component , we present the first representation probing algorithm handling slot permutation invariance . 2 BACKGROUND . 2.1 DISENTANGLEMENT CRITERIA . There exists an extensive literature discussing notions of disentanglement , accounting for all the different definitions is outside the scope of this paper . We chose to focus on the three criteria formalized by Eastwood and Williams ( 2018 ) , which stand out because of their clarity and simplicity . Disentanglement is the degree to which a representation separates the underlying factors of variation , with each latent variable capturing at most one generative factor . Completeness is the degree to which each underlying factor is captured by a single latent variable . Informativeness is the amount of information that a representation captures about the underlying factors of variation . Similarly to prior work , the word disentanglement is also used as a generic term that simultaneously refers to these three criteria . It should be clear depending on the context whether it is meant as general or specific . 2.2 THE DCI FRAMEWORK . For brevity reasons , we only describe the DCI metrics of Eastwood and Williams ( 2018 ) , which are most closely related to our work . The supplementary material provides a comprehensive overview of alternative metrics . Consider a dataset X composed of n observations x1 , . . . , xn , which we assume are generated by combining F underlying factors of variation . The value of the different factors for observation xl are denoted vl1 , . . . , v l F . Suppose we have learned a representation z = ( z1 , . . . , zL ) from this dataset . The DCI metric is based on the affinity matrix R = ( Ri , j ) , where Ri , j measures the relative importance of latent zi in predicting the value of factor vj . Supposing appropriate normalization for R , disentanglement is measured as the weighted average of the matrix ’ row entropy . Conversely , completeness is measured as the weighted average of column entropy . Informativeness is measured as the normalized error of the predictor used to obtain the matrix R . 3 EVALUATING STRUCTURED DISENTANGLEMENT . 3.1 contains a high level description of the goals steering our framework . Our metric is formally described in 3.2 . In 3.3 , we present theoretical results showing that our framework can safely substitute to DCI since it provides stronger guarantees that the selected model is correctly disentangled . 3.1 PRESENTATION OF THE FRAMEWORK . Prior work has focused on measuring disentanglement at the level of individual factors of variation and latent dimensions ( that we will refer to as global or unstructured disentanglement ) . In the case of compositional representations with several objects slots , additional properties are desirable : 1 . Object-level disentanglement : Objects and latent slots should have a one-to-one matching . That is , changing one object should lead to changes in a single slot , and vice-versa . 2 . Slot disentanglement : Inside a given slot , properties of the object must be disentangled . 3 . Slot symmetry : The latent dimension responsible for a given factor ( e.g. , color ) should be invariant across slots . This means that all slots should have the same inner structure . Our structured disentanglement metric allows to evaluate all these criteria within a single unified framework . Similarly to DCI , it is based on the affinity matrix ( Rτ̂ , τ ) , where Rτ̂ , τ measures the relative importance of latent zτ̂ in predicting the value of factor vτ . The key novelty compared to DCI is that we propose to measure disentanglement with respect to arbitrary projections of the matrix R. Intuitively , projections correspond to a marginalization operation that selects a subset of hierarchy levels and discards the others . The coefficients Rτ̂ , τ that are projected together are summed to create group affinities . Figure 2 gives an intuitive illustration of this process on a toy example . With object-centric representations , projecting at the object/slot level allows to study the relation of objects and slots without taking their internal structure into consideration . Ultimately this permits to evaluate our object-level disentanglement criterion . Projecting at property level allows to study the internal slot representation independently of the object , for evaluation of both internal slot disentanglement and slot symmetry , with a single metric . The identity projection conserving all levels measures flat ( DCI ) disentanglement . This generalizes to arbitrary hierarchies in the representation , such as disentanglement of position and appearance , or intrinsic and extrinsic object properties . 3.2 MATHEMATICAL DEFINITION . To formalize our framework most conveniently , we propose to view the affinity matrix R = ( Rτ̂ , τ ) as a joint random variable ( X , Y ) , where X is a random latent dimension , and Y a random factor . Supposing that R is normalized to have sum one , this means P [ X = τ̂ , Y = τ ] = Rτ̂ , τ . This point of view , which is only implicit in prior work , allows to formalize the projections of R as a coupled marginalization operation ( ρ ( X ) , ρ ( Y ) ) , where ρ ( α1 , . . . , αh ) = ( αe1 , . . . , αel ) selects a subset of hierarchy levels . This yields a concise and elegant mathematical framework , where we can build on standard information-theoretic identities to derive theoretical properties . In the following , HU ( A|B ) denotes the conditional entropy of A with respect to B ( detailed definitions in the supplementary ) . Our disentanglement criteria are the following : Completeness with respect to projection ρ measures how well the projected factors are captured by a coherent group of latents , by measuring column entropy in the projected space . It is measured as C ( ρ ) = 1−HU ( ρ ( X ) |ρ ( Y ) ) , where U = |ρ ( T̂ ) | is the number of groups of latents in the projection . Disentanglement with respect to projection ρ measures to what extent a group of latent variables influences a coherent subset of factors , by measuring projected row entropy . It is measured as D ( ρ ) = 1−HV ( ρ ( Y ) |ρ ( X ) ) , where V = |ρ ( T ) | is the number of groups of factors in the projection . Informativeness does not depend on the projection . It is defined as the normalized error of a low-capacity model f that learns to predict the factor values v from the latents z , i.e . I = ‖f ( z ) − v‖2 ‖v‖2 . The changing log bases U and V aim at ensuring normalization between 0 and 1 . 3.3 ESTABLISHING TRUST IN OUR FRAMEWORK : A THEORETICAL ANALYSIS . Our disentanglement D ( ρ ) and completeness C ( ρ ) metrics depend on the subset of hierarchy levels contained in the projection ρ . We theoretically analyze the influence of the projection choice . It is especially interesting to study the behavior when distinct subsets of hierarchy levels are combined . Our key results are the following : Theorem 1 shows that our framework contains DCI as a special case , when the identity projection is chosen . Theorem 2 shows that the metrics associated to the identity projection can be decomposed in terms of one dimensional projections along a single hierarchy level . Together , these results formally show that one dimensional projections provide a sound substitute for prior unstructured metrics . This is very useful to build trust in our framework . Theorem 1 ( Relation with DCI ) . The DCI metrics of Eastwood and Williams ( 2018 ) are a special case of our framework , with the identity projection conserving all hierarchy levels . Theorem 2 uses the intuitive notion of decomposition of a projection . The projection ρ is said to be decomposed into ρ1 , . . . ρk if the set of hierarchy levels selected by ρ is the disjunct union of the set of levels selected by ρ1 , . . . ρk . In the case of object-centric latent representations , the identity projection ρid that keeps all hierarchy levels { object , property } can be decomposed as ρobject considering only { object } and ρproperty considering only { property } . Theorem 2 ( Decomposition of a projection ) . Consider a decomposition of the projection ρ ( with L latent groups ) into disjunct projections ρ1 , . . . ρk ( with respectively L1 , . . . , Lk latent groups ) . The following lower bound for the joint completeness ( resp . disentanglement ) holds 1− k + k∑ s=1 C ( ρs ) ≤ 1− k∑ s=1 1− C ( ρs ) logLs ( L ) ≤ C ( ρ ) . Suppose that all one dimensional projections verify C ( ρs ) ≥ 1 − , where ≥ 0 . We obtain the lower bound C ( ρid ) ≥ ( 1− k ) + k ( 1− ) = 1− k for the identity projection . For object-centric representations , this implies that when object-level completeness and property completeness are both perfect ( that is , = 0 ) , then DCI completeness is also perfect . The same works for disentanglement . The supplementary materials contains detailed proofs , a matching upper bound , as well as an explicit formula in the special case k = 2 .

Summary: This paper proposes a new method for evaluating the disentanglement of object-centric representations. It can be seen as generalization of the DCI metrics of Eastwood et al. (2018) to object-centric representations, permitting the evaluation of both inter- and intra-object disentanglement. The two main contributions are: (i) proposing projection/marginalization operations on the matrix R in order to select an abstraction level (e.g. object or property level, as nicely illustrated in Figure 2); and (ii) an EM-based object-matching or "representation probing" algorithm which matches latent object slots with underlying GT objects (specifically, it optimizes the latent slot permutation to maximize the accuracy of the linear predictor f:z → v).