Evaluating Disentanglement of Structured Representations
Authors: Raphaël Dang-Nhu
Venue: ICLR 2022 Poster (2022)

Abstract: We introduce the first metric for evaluating disentanglement at individual hierarchy levels of a structured latent representation. Applied to object-centric generative models, this offers a systematic, unified approach to evaluating (i) object separation between latent slots (ii) disentanglement of object properties inside individual slots (iii) disentanglement of intrinsic and extrinsic object properties. We theoretically show that our framework gives stronger guarantees of selecting a good model than previous disentanglement metrics. Experimentally, we demonstrate that viewing object compositionality as a disentanglement problem addresses several issues with prior visual metrics of object separation. As a core technical component, we present the first representation probing algorithm handling slot permutation invariance.

EVALUATING DISENTANGLEMENT OF STRUCTURED REPRESENTATIONS
Raphaël Dang-Nhu
ABSTRACT
We introduce the first metric for evaluating disentanglement individual hierarchy levels of a structured latent representation. Applied to object-centric generative models, this offers a systematic, unified approach to evaluating (i) object separation between latent slots (ii) disentanglement of object properties inside individual slots (iii) disentanglement of intrinsic and extrinsic object properties. We theoretically show that for structured representations, our framework gives stronger guarantees of selecting good model than previous disentanglement metrics. Experimentally, we demonstrate that viewing object compositionality as disentanglement problem addresses several issues with prior visual metrics of object separation. As a core technical component, we present the first representation probing algorithm handling slot permutation invariance.
INTRODUCTION
A salient challenge in generative modeling is the ability to decompose the representation of images and scenes into distinct objects that are represented separately and then combined together. Indeed, the capacity to reason about objects and their relations is a central aspect of human intelligence (Spelke) cet al. 1992). which can be used in conjunction with graph neural networks or symbolic solvers to enable relational reasoning. For instance, (Wang et al. 2018) exploit robot body structure to obtain competitive performance and transferability when learning to walk or swim in the Gym environments. (Yi al. 2019) perform state-of-the-arr visual reasoning by using an object based representation in combination with a symbolic model. In the last years, several models have been proposed to learn compositional representations in an unsupervised fashion: TAGGER (Greff et al. 2016). NEM (Greff
ct al. 2017). R-NEM (Van Steenkiste al. 2018), MONet (Burgess et al.) 2019 IODINE (Greff
ctt al. 2019). GENESIS (Engelcke et al. 2019). Slot Attention (Locatello et al. 2020), MuIMON (Li
ctt al. 2020 SPACE (in et al.] 2020 They jointly learn to represent individual objects and to
segment the image into meaningful components, the latter often being called "perceptual grouping".
These models share number of common principles: (i) Splitting the latent representation into several slots that are meant to contain the representation of an individual object. (ii) Inside each slot, encoding information about both object position and appearance. (iii) Maintaining symmetry between slots in order to respect the permutation invariance of objects composition. These mechanisms are intuitively illustrated in Figure
To compare and select models, it is indispensable to have robust disentanglement metrics. At the level of individual factors of variations, a representation is said to be disentangled when information about the different factors is separated between different latent dimensions (Bengio et al., 2013 Locatello et al., 2019). At object-level, disentanglement measures the degree of object separation
between slots. However, all existing metrics (Higgins al. 2016 Chen et al.) 2018 Ridgeway and
Mozer 2018: Eastwood and Williams) 2018 Kumar et al. 2017) are limited to the individual case,
which disregards representation structure. To cite Kim and Mnih (008)) about the FactorVAB metric:
The definition of disentanglement we use [..]] is clearly a simplistic one. It does not allow correlations among the factors or hierarchies over them. Thus this definition seems more suited to synthetic data with independent factors of variation than most realistic datasets.

---

Ohject-level disentunglement (ours)
Visual object separation (ARI)
Lalent
Decoder
Input image
Reconstructed
Latent slot
Decoder
Composition
module
Encoder
Latent
Decoder
Latent slot
Decoder
Decoders have
shared weights
Figure 1: A compositional latent representation is composed of several slots. Each slot generates part the image. Then, the different parts are composed together. Pixel-level metrics measure object separation between slots at visual level, while our framework operates purely in latent space.
As a result, prior work has restricted to measuring the degree of object separation via pixel-level segmentation metrics. Most considered is the Adjusted Rand (ARI) Index (Rand) 1971; Greff et al. 2019), where image segmentation viewed as a cluster assignment for pixels. Other metrics such as Segmentation Covering (mSC) (Arbelaez et al. (010)) have been introduced to penalize over- segmentation of objects. fundamental limitation that they do evaluate directly the quality of the representation, but instead consider visual proxy of object separation. This results in problematic dependence on the quality of the inferred segmentation masks, problem first identified by Grefff ct al. (2019) for IODINE, and confirmed in our experimental study.
To address these limitations, we introduce the first metric for evaluating disentanglement at individual hierarchy levels of structured latent representation. Applied to object-centric generative models, this offers a systematic, unified approach to evaluating (i) object separation between latent slots (ii) disentanglement of object properties inside individual slots (iii) disentanglement of intrinsic and extrinsic object properties. We theoretically show that our framework gives stronger guarantees of representation quality than previous disentanglement metrics. Thus. it can safely substitute to them. We experimentally demonstrate the applicability of our metric to three architectures: MONet, GENESIS and IODINE. The results confirm issues with pixel-level segmentation metrics, and offer valuable insight about the representation learned by these models. Finally, as core technical component, we present the first representation probing algorithm handling slot permutation invariance.
BACKGROUND
2.1 DISENTANGLEMENT CRITERIA
There exists an extensive literature discussing notions of disentanglement, accounting for all the different definitions outside the scope of this paper. We chose focus on the three criteria formalized by Eastwood and Williams (2018), which stand out because of their clarity and simplicity. Disentanglement is the degree to which a representation separates the underlying factors of variation, with each latent variable capturing at most one generative factor. Completeness is the degree to which each underlying factor is captured by a single latent variable. Informativeness is the amount of information that representation captures about the underlying factors of variation. Similarly prior work, the word disentanglement is also used as a generic term that simultancously refers to these three criteria. It should be clear depending the context whether it is meant general specific.
2

---

2.2 THE DCI FRAMEWORK
For brevity reasons, we only describe the DCI metrics of Eastwood and Williams (0018), which are most closely related to our work. The supplementary material provides a comprehensive overview of alternative metries. Consider a dataset X' composed of observations x ..... xx", which we assume are generated by combining underlying factors of variation. The value of the different factors for observation æf are denoted of..... up. Suppose we have learned a representation
(21:...,222) =1)
from this dataset. The DCI metric based on the affinity matrix the relative importance of latent 2; in predicting the value of factor vj- Supposing appropriate normalization for R, disentanglement is measured as the weighted average the matrix' row entropy. Conversely, completeness is measured as the weighted average of column entropy. Informativeness is measured as the normalized error the predictor used to obtain the matrix R.
), where R., measures
3
EVALUATING STRUCTURED DISENTANGLEMENT
contains a high level description of the goals steering our framework. Our metric is formally present theoretical results showing that our framework safely substitute to DCI since it provides stronger guarantees that the selected model is correctly disentangled.
3.1
described in 3.2
3.1
PRESENTATION OF THE FRAMEWORK
Prior work has focused on measuring disentanglement at the level of individual factors of variation and latent dimensions (that will refer global or unstructured disentanglement), In the case of compositional representations with several objects slots, additional properties are desirable:
1. Object-level disentanglement: Objects and latent slots should have a one-to-one matching. That is, changing one object should lead to changes in single slot, and vere-versa.
2. Slot disentanglement: Inside given slot, properties of the object must be disentangled.
3. Slot symmetry: The latent dimension responsible for given factor (e.g., color) should be invariant across slots. This means that all slots should have the same inner structure.
Our structured disentanglement metric allows to evaluate all these criteria within single unified framework. Similarly to DCI, it is based on the affinity matrix (R+,)), where R,,- measures the relative importance of latent 2ạ predicting the value of factor er. The key novelty compared to DCI is that we propose to measure disentanglement with respect arbitrary projections of the matrix R. Intuitively, projections correspond to a marginalization operation that selects a subset of hierarchy levels and discards the others. The coefficients R that are projected together summed to create group affinities. Figure 2 gives an intuitive illustration of this process on a toy example.
With object-centric representations, projecting the object/slot level allows to study the relation of objects and slots without taking their internal structure into consideration. Ultimately this permits to evaluate our object-level disentanglement criterion. Projecting at property level allows to study the internal slot representation independently of the object, for evaluation of both internal slot disentanglement and slot symmetry, with single metric. The identity projection conserving all levels measures flat (DCI) disentanglement. This generalizes to arbitrary hicrarchies in the representation, such as disentanglement of position and appearance, or intrinsic and extrinsic object properties.
3.2 MATHEMATICAL DEFINITION
To formalize our framework most conveniently, we propose to view the affinity matrix R= (R+,7) as joint random variable (X, Y), where X is random latent dimension, and Y random factor. Supposing that R is normalized to have sum one, this means P = f, r] = Rg, This point of view, which is only implicit in prior work, allows to formalize the projections of It as coupled marginalization operation (p(X), p(Y)), where p(a1...., an) = (ae....... ......,e)) selects a subset of hierarchy levels. This yields concise and elegant mathematical framework, where we can build on standard information- iiformatioon-hhoretiic identities to derive theoretical properties. In the following, Hy (A) B) denotes the conditional entropy of A with respect to B (detailed definitions in the supplementary).
3

---

Factors of variation
Identity projection
Object
Object
Projection on dimension/property
Color
Size
Color
Size
Color
Size
Dim
Ri
R
R1.4
Ra.1
R1,
Uua Ri
Slot
Dim 1
Na.1 + R3.3
Hg.
Rs,4
Dim
Ra
R2
Rg.s
Ra,4
Latents
Ra.1
Ra.2
Ra.3
Ra.4
R211 Reat R4.1 Ra.
Ra.2 R2st
Dim
Raa2 R4,
Slot
R4,1
R1,2
R4.3
Ra.4
Dim 2
Dim
Projection slot/object
Unstructured (DCI)
Property-level completeness
Object
Object
completeness
R1.1 + 2
R + RIA R2 + Ro
Object-lovel disentanglement
Ro.1 R2
Ra.1 + Ro.2
Raa + Ra R4.3 + R4,4
R4.1 R4
Slot
Figure 2: Overview of projections on toy example of object-centric representation. The affinity scores are marginalized according to the selected hierarchy levels. In projected space, row entropy measures disentanglement, while column entropy measures completeness. The measure of entropy involves renormalization by row (for completeness) or column (for disentanglement) The affinity scores R computed in all-pairs manner. A mapping can be obtained (e.g. from object to slot) by looking for the maximum each column.
Our disentanglement criteria are the following:
Completeness with respect to projection p measures how well the projected factors are captured by coherent group of latents, by measuring column entropy in the projected space. It is measured as
C(p) = Hy (p()))
where U = |p(()) is the number of groups of latents in the projection.
Disentanglement with respect to projection p measures to what extent group of latent variables influences a coherent subset of factors, by measuring projected row entropy. It is measured as
D(p) = Hy (p()))((X))
where V (T) is the number of groups of factors in the projection.
Informativeness does depend the projection. It is defined the normalized enror of low-capacity model ff that learns to predict the factor values from the latents 2, i.c.
[5(x) vll2
The changing log bases and aim at ensuring normalization between and
4

---

Theorem (Decomposition of projection). Consider a decomposition of the projection (with L 3.3 ESTABLISHING TRUST OUR FRAMEWORK: A THEORETICAL ANALYSIS
Our disentanglement (()) and completeness C(p) metrics depend on the subset of hierarchy levels contained in the projection p. We theoretically analyze the influence of the projection choice. It is especially interesting to study the behavior when distinct subsets of hierarchy levels are combined. Our key results are the following: Theorem [rremm[[show shows that our framework contains DCI as a special case, when the identity projection is chosen. Theorem 2 shows that the metrics associated to the identity projection can be decomposed in terms of one dimensional projections along single hierarchy level. Together, these results formally show that one dimensional projections provide sound substitute for prior unstructured metrics. This is very useful to build trust in our framework.
Theorem (Relation with DCI). The DCI metrics of Eastwood and Williams (2018) are special case of our framework, with the identity projection conserving all hierarchy levels.
Theorem uses the intuitive notion of decomposition of projection. The projection p is said to be decomposed into Pl. Pk if the set of hierarchy levels selected by is the disjunct union of the set of levels selected by Pl.... Pk- In the case of object-centric latent representations, the identity projection Pia that keeps all hierarchy levels {object, property) } can be decomposed as Pobject considering only {object) and Peropenty considering only (property).
latent groups) into disjunct projections pı.... (with respectively U.,..., LL latent groups). The following lower bound for the joint completeness (resp. disentanglement) holds
C(ps)
1- k
C(p.)
S (p).
logr (L)
Suppose that all one dimensional projections verify C(ps)= 2 -, where 2 0. We obtain the lower bound ((paa) 2 (1 k) - k(1 -) 1 ke for the identity projection. For object-centric representations, this implies that when object-level completeness and property completeness are both perfect (that is, c = O), then DCI completeness is also perfect. The same works for disentanglement The supplementary materials contains detailed proofs, matching upper bound, well as explicit formula in the special case = 2.
4 PERMUTATION INVARIANT REPRESENTATION PROBING
Representation probing aims at quantifying the information present each latent dimension, to obtain the relative importance Re,r of latent 2; in predicting factor vr. Traditional probing methods either use regression feature importance or mutual information to obtain R (see the supplementary). However, these techniques do not account for the permutation invariance of object-centric representa- tions, in which slot reordering leaves the generated image unchanged. To address this, we propose novel formulation as permutation invariant feature importance problem. As contribution of this paper, we present an efficient EM-like algorithm to solve this task in a tractable way. Traditional representation probing fits a regressor f to predict the factors from latents z, solving
technical
arg min El 10--((=))1
10
Then, the matrix R is extracted from the feature importances of f. In contrast, formulation of permutation invariant representation probing jointly optimizes on permutations (Fi,..., nr) over groups of latent dimensions (for instance slots), allowing to account for slot permutation invariance of the representation, thus solving
arg min
To address the combinatorial explosion created by the joint optimization, we chose an EM-like approach and iteratively optimize and the permutations (Tı,..., TT)) while keeping the other fixed (Algorithm D) This method yields satisfying approximate solutions provided a good initialization (See Figure 5 and the supplementary).
5

---

Algorithm Permutation-invaaiian representation probing
Input: Latent codes 2² .....,.." and factor values v² ...., for input images
for i = 1 to jiters do
# STEP
Fit predictor fi to features 21, z" and targets v², v²,...,'".
STEP
for to do
min f- arg min 1|0 f((T(22))|1 ( ranges on all slot permutations)
min
Fit final to features z , and targets v² ,,.., U Obtain feature importances from finll.
2n
EXPERIMENTAL EVALUATION
We compare three different architectures with public Pytorch implementations: MONet, GENESIS and IODINE, which we believe representative object-centric representations, We evaluate two variants of MONet: the first one, denoted "MONet (Att.)" follows the original design in Burgess et al. 2 and uses the masks inferred by the attention network the reconstruction. The second, denoted as "MONet (Dec.)" is variant that instead uses the decoded masks. For all models, we perform an ablation of the disentanglement regularization in the training loss. Ablated models trained with pure reconstruction loss, denoted (R). All final feature importances obtained with random forests.
Models
Datasets We evaluate all models CLEVR6 (Johnson al. 2017 and Multi-dSprites (Matthey et al., 2017 Burgess et al. 2019). with the exception of IODINE that we restricted to Multi-dSprites for computational reasons, as CLEVR6 requires week on 8 V100 GPUs per training. These datasets are the most common benchmarks for learning object-centric latent representations. Both contain images generated compositionally. Multi-dSprites has 2D shapes on a variable color background, while CLEVR6 is composed of realistically rendered 3D scenes. We slightly modified CLEVR6 to ensure that all objects are visible the cropped image. contains our experimental results at both object and property-level. compared to pixel-level segmentation metrics (ARI and mSC). We highlight some of these numbers Figure 3 Figure ssowws the different projections the affinity matrix R Hinton diagrams. Figure visualizes the slot permutation learned by permutation invariant probing algorithm.
b)
Full loss
Full loss
recenstruction
reconstruction
20
lett-eeell.ee disentanglemt
Objecct-teel Uisentanglern
30
MesET
(00N4F
rpedi
82
82
ARI
(CO
MUNET [Dec) GERESIS
IODNE
MoNE (822
MUNET I0e0 GERESS
IODME
MUMES (611)
80
95
130
110
ARI
Figure 3: a) Ablation of disentanglement regularization tends to decrease disentanglement at object- level (Multi-dSpprtes), b) Ablation of disentanglement regularization tends to increase pixel-level segmentation (measured with ARI Multi-dSprites). e) Object-level disentanglement ARI for different models trained on Multi-dSprites. We observe a negative correlation between both metrics.

---

Multi-dSprites GENESIS
MONet (ATL))
MONet (Rec.)
SLOTS
SLOTS
(9911
IOIIIE
SLOTS
Oqpans
LATENTS
Figure 4: Projections of the aftinity matrix Multi dSpri-dSprites for IODINE, GENESIS and the two variants of MONet, Hinton diagrams: The white is proportional coefficient value.
Before Algorithm
Algorithmm
VODINE
InaUT
Sko Slcc. SMC Stat
Stoc
SkY 0 Stc... SWC Sh1 SIE.
Figure 5: Effect of EM probing (Alg. on slot ordering for a group of similar inputs. Each row corresponds to the decomposition of a given input between slots. a) is without E.M probing, b) with EM-probing. The decomposition compared observe that EM-probing assigns objects to slot in way that is always consistent to the reference input. That is, similar objects are matched to similar slots.
reference decomposition (on the first row). We
Nonet (95s
Table 1: Comparison of the permutation obtained with EM probing vs obtained with IoU matching (multi- dSprites, averaged across models). We report (i) per- centage of exactly matching permutations (ii) average mismatch between both permutations.
57
Nond (Des)
lodine
54
Figure 6: Convergence of the EM probing algorithm, for the different models (Multi- dSprites). Values can be directly pared with Table 2 as the final predictor has higher capacity.
% EXACTLY MATCHING
# AVERAGE MISMATCHI
MEASURED
62 %
1.08
BEST POSSIBLE
WORST POSSIBLE
0 %
CHANCE
0.83 %
7

---

5.1 RELATION TO PIXEL-LEVEL SEGMENTATION METRICS.
Our object-level metrics have the same goal as visual metrics such as ARI and mSC: quantifying object separation between slots. Therefore, we observe general correlation of all metrics, with low values at initialization and improvement throughout training. The main difference is that ARI and mSC evaluate separation at visual level, while our framework purely operates in latent space, measuring the repartition of information (see Figure[]. Consequently, ARI and mSC tend to favor sharp object segmentation masks, while our framework is focused on easy extraction of information from the latent space. This fundamental difference leads to specific regimes of negative correlation. Figure c) shows that ARI and our object-level metric give very different rankings of trained model performance for Multi-dSprits sharp segmentation masks identified by Greff et al. (2019). This particularly visible for IODINE, which achieves good disentanglement despite its low ARI.
suggesting that our framework addresses the dependence of ARI on
5.2 INFLUENCE OF THE DISENTANGLEMENT REGULARIZATION
Figure 3 a) and b) shows an ablation study of the disentanglement regularization. The ablated models are trained with pure reconstruction loss, without regularization of the latent space. For our disentanglement metrics, we observe consistent negative impact. This is consistent with observations for unstructured representations (Eastwood and Williams 2018) 20 Locatello et al., 2019). On the contrary, the ablation tends to improve pixel-level segmentation. This would imply that visual object separation is only caused by architectural inductive biases, which clarifies the impact of disentanglement regularization in prior work.
5.3 INSIGHTS ABOUT THE DIFFERENT MODELS
Qualitatively, visual inspection of the different projections of matrix R in Figures 4 and 11 shows near one-to-one mapping between slots and objects, except some redundancy for the background slot, which is excluded from our metrics. Our quantitative results in Table T confirm a near perfect object disentanglement of up to 90% for Multi-dSprites. For CLEVR6 we consider the value of 50% to be decent, because the logarithmic scale of the conditional entropy tends to create a strong shift towards O. To give a simple reference, if each object is equally influenced by 2 slots among 6, the expected value is - logs () 57%. Table B additionally shows that (i) the robust performance, of GENESIS supports the generalization of its separate mask encoding process and GECO optimizer. (ii) that IODINE is able get close to the performance of MONet, despite less stable training process for IODINE which resulted in an outlier with bad performance. (iii) MONet (Att.) obtains better visual separation metrics, while MONet (Dec.) has better disentanglement. This is because using the decoded masks in MONet (Dec.) increases the pressure to accurately encode the masks in the latent representation. On the contrary, MONet (Att.) does not use the decoded masks for reconstruction. and thus does not need to encode them very accurately.
5.4 EXTENSION TO OTHER HIBRARCHY LEVELS
Table 2: Projection along third hierarchy level (GENESIS, MdSprites).
To demonstrate generalization to more than two hierar- chy levels, we evaluate disentanglement of intrinsic and extrinsic object properties. This specifically applies to GENESIS, for which each slot divided in mask latent and component latent. Intuitively, intrinsic properties (such as shape and color) are related to the nature of the object, whereas extrinsic properties (such as position and orientation) are contextual to the scene. The projection along this third hierarchy level is given in Table 2 for GENESIS trained on Multi-dSprites. Numbers show that extrinsic properties are successfully captured by the mask latent. However, intrinsic properties are not satisfyingly separated. This is because information about object shape is contained both in the segmentation mask and in the generated image of the object. We believe that recent architectures (Nguyen- Seuuyen-Phuoc et al. 2020 Ehrhardt et al. 2020) performing object composition in 3D scene space might solve this problem.
EXTRINSIC
INTRINSIC
MASK LATENT COMPONENT LATENT
0.2
0.32
8

---

Table 3: Experimental results (stderr. over three seeds, values in %). Models with (R) have no disentanglement regularization. ARI/mSC is slightly higher (resp. lower) than prior work for Multi-dSSritee (resp. C because of minor differences in the data generation process. At object-level, we observe that disentanglement and completeness match closely, but this is not the case at property-level, due to the asymmetry in the number of latent factors and slot dimensions.
OBJECT-LEVEL
PROPERTY-LEVEL
(1)
(t)
(1s.(1)
COMP.(1)
INF.(L)
ART(1)
(f)
(ATT)
MONET (ATT) (R)
60 +
60 =
29 +
52 =
36 =
98 + 0.1
86 +
MONET (DBC.)
59 66
74 5
80 1
68 1
MDSPR.
MONET (DDc.) (R)
0.3
0.2
35
53
32 0.8
0.2
69 +
GENESIS
90 + 0.8
91 + 0.6
72 + 0.5
60 = 0.4
24 24 0.4
85
70 + 0.3
GENESIS (R)
3
65
65
42 1111
0.5
96 1 0.6
84 + 0.7
IODINE
7
72
37
62
37 6
8
71 +
IODINE (R)
35 1:
36
17
46 1
61
75 15.14
65 1:
MONET (ATT.
46 06 5
48 +
22: 4
51 3
55
98 + 0.4
68 +
MONET (ATT) (R)
1:
41
13155
44 1
61
93 + 0.4
65 5
MONET (DDC.)
6
48 18
21
50 4
55
90
65
CLEVR6
MONET (DEC.) (R)
10
41 0.2
0.6
44 0.1
60 1 0.6
92
69 =
GENESIS
50
52
47
39 1
47 47 0.9
91 0.4
65.3
GENESIS (R)
43
21
21
92
60
5.5 RE-EXAMINING THE MODEL SELECTION PROCESS
These divergences between visual metrics and our framework lead us to reconsider prior model selection processes, which were primarily centered on ARI. To give three striking examples (i) the recent Slot Attention (Locatello et al. 2020) architecture does not use disentanglement regularization. In light of our ablation study, we believe that this potentially harmful for disentanglement (Section 5.2p. (ii) Burgess al. (2019) and Engelcke et al. (2019) chose to privilege MOnet (Att.) which obtains lower disentanglement that MONet (Dec.) (Section (3.3) (iii) most prior work chose a 2D segmentation mask approach that does not satisfyingly disentangle intrinsic and extrinsic object properties (.4)) We believe that all related existing experimental studies would benefit from the perspective offered by our framework.
RELATED WORK
Structured disentanglement metrics. Most similar to our work is the slot compactness metric of Racah and Chandar (2020), also based aggregation of feature importances measure object separation between slots. However, it only operates at one hierarchy level and does not handle slot permutation invariance, which is essential to obtain meaningful results. Also related is the hierarchical disentanglement benchmark of Ross and Doshi Velez (2021). whose ability to learn the hierarchy levels in the representation, remarkable, but is unfortunately limited in its applicability to toy datasets. Finally, |Esmaeili et al.] (2019) present a structured variational loss encouraging disentanglement group-level. There a high-level connection with our work, but the objective ultimately different. [Esmailli et al. (2019) evaluate their structured variational loss with unstructured disentanglement metrics.
CONCLUSION
Our framework for evaluating disentanglement of structured latent representations addresses number of issues with prior visual segmentation metrics. We hope that will be helpful for validation and selection of future object-centric representations. Besides, took great in not making any domain specific assumption, and believe that the principles discussed here apply to any kind of structured generative modelling.

---

ACKNOWLEDGEMENTS
This work was granted access to the HPC resources of IDRIS under the allocation 1020-AD011012135 made by GENCI. We would like to thank Frederik Benzing, Kalina Petrova, Asier Mujika, and Wouter Tonnon for helpful discussions. This work constitutes the public version of Raphaël Dang-Nhu's Master Thesis ETH Zürich.
REFERENCES
Arbelaez, P., Maire, M.. Fowlkes, C., and Malik, (2010). Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898 -166.
Bengio, Y., Courville, A., and Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1798-117888
Burgess, C. Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A. (2019). Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 1901.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020). End-to-end object detection with transformers. arXiv preprint arXiv:2005.128:
Chen, R. T., Li, X., Grosse, R. B., and Duvenaud, D. K. (2018). Isolating sources of disentanglement in variational autoencoders. Advances in Nearul Information Processing Systems, pages 2610- 100 2620.
Cover, T. M. (1999). Elements of information theory. John Wiley & Sons.
Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 3-.1)
Eastwood, C. and Williams, K. (2018), A framework for the quantitative evaluation disentangled representations.
Ehrhardt, S., Groth, O., Monszpart, A., Engelcke, M., Posner, I., Mitra, N., and Vedaldi, A. (2020). Relate: Physically plausible multi-object scene synthesis using structured latent spaces. arXiv preprint arXiv:2007.01272.
Engelcke, M., Kosiorek, A. R., Jones, O.P. P., and Posner, I. (2019). Genesis: and sampling with object-centric latent representations. arXiv preprint arXiv: /907. 13052.
Generative scene inference
Esmacili, B., Wu, H., Jain, S., Bozkurt, A., Siddharth, N., Paige, B., Brooks, D. H., Dy, J., and Meent, J.WW. (2019). Structured disentangled representations. In The 22nd International Conference on PMLR.
Artificial Intelligence and Statistics, pages
Greff, K., Kaufman, R. L., Kabra, R., Watters, N., Burgess, C.,, Zoran, D., Matthey, L., Botvinick, M., and Lerchner, A. (2019). Multi-object representation learning with iterative variational inference. arXiv preprint Xiv: 1903.0045.
Greff, K., Rasmus, A., Berglund, M., Hao, T., Valpola, H., and Schmidhuber, J. (2016). Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems, pages 4484 4492.
Greff, K., Van Steenkiste, S., and Schmidhuber, J. (2017). Neural expectation maximization. In Advances in Neural Information Processing Systems, pages 6691-6701.
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2016). beta-vae: Learning basic visual concepts with constrained variational framework.
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Lawrence Zimick, C., and Girshick, R. (2017). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2910.
10

---

Kim, H. and Mnih, A. (2018). Disentangling by factorising. arXiv preprint arXiv:1802.05983.
Kumar, A., Sattigeri, P., and Balakrishnan, (2017). Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00644
Li, N., Fisher, R., et (2020). Learning object-centric representations of multi-object scenes from multiple views. Advances in Neural Information Processing Systems, 33.
Lin, Z., Wu, Y.-F, Peri, S. V., Sun, W., Singh, G., Deng, Jiang, J., and Ahn, S. (2020). Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint ar arXiv:2201.000 022407.
Locatello, F,, Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 41244.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. (2020). Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33.
Matthey, L., Higgins, I., Hassabis, D., and Lerchner, A. (2017). dsprites: Disentanglement testing sprites dataset. URL https://github canllceemmiddspppr---dddlleeeeee
on: 2018-05-08).
Mena, G., Belanger, D., Linderman, S., and Snoek, J. (2018). Learning latent permutations with gumbel-sinkhorn networks. arXiv preprint arXiv:1802.088
Nguyen-Phuoe, T., Li, C., Theis, L., Richardt, C., and Yang. Y.-L. (2019). Hologan: Unsupervised learning of 3d representations from natural images. In Proceedings of the IEEE International Conference on Computer Vision, pages 7588-7597.
Nguyen-Phuoc, T., Richardt, C., Mai, L., Yang, Y.-L., and Mitra, N. (2020). Blockgan: Learning 3d object-aware scene representations from unlabelled images. arXiv preprint arXiv:2002.08988
Niemeyer, M. and Geiger, A. (2020). Giraffe: Representing scenes as compositional generative neural feature fields. arXiv preprint arX11:2011.12100.
Racah, E. and Chandar, S. (2020). Slot contrastive networks: A contrastive approach for representing objects. arXiv preprint arXiv:2007.0929:
Rand, W. M. (1971). Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 666336):846885.
Rezende, D. J. and Viola, F. (2018). Generalized elbo with constrained optimization, geco. In Workshop Bayesian Deep Learning, NeurlPS.
Ridgeway, K. and Mozer, M. C. (2018). Learning deep disentangled embeddings with the f-statistic loss. In Advances in Neural Information Processing Systems, pages 185-194.
Ross, and Doshi-Veles (2021). Benchmarks, algorithms, and metrics for hierarchical disentanglement, arXiv preprint
Spelke, E. S., Breinlinger. K.. Macomber, J., and Jacobson, K. (1992). Origins of knowledge. Psychological review, (005
Van Steenkiste, S., Chang, M., Greff, K., and Schmidhuber, J. (2018). Relational neural expec- tation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.10353
van Steenkiste, S., Kurach, K., and Gelly, S. (2018). A case for object compositionality in deep generative models of images. arXiv preprint arXiv:1810.10340.
Wang, T., Liao, R., Ba, J., and Fidler, S. (2018). Nervenet: Learning structured policy with graph neural networks. In International Conference Learning Representations.
11

---

Yi, K., Gan, C., Y., Kohli, P., Wu, J., Torralba, A., and Tenenbaum, B. (2019). Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442.
Yu, D., Kolbæk, M.. Tan, Z.-H., and Jensen, J. (2017). Permutation invariant training of deep models for speaker-independent multi-talker speech separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 241-245. IEEE.
Zaidi, J., Boilard, J., Gagnon, G., and Carbonneau, M.-A. (2020). Measuring disentanglement: A review of metrics. arXiv preprint arXiv:0012.09276.
12

---

A ADDITIONAL RELATED WORK
A.1 OBJECT-CENTRIC REPRESENTATION LEARNING
Many models have been developed to perform unsupervised perceptual grouping and learn composi- tional representations. A first line of work (TAGGER (Greff et al. 2016), NEM (Greff et al. 2017).
R-NEM (Van Steenkiste et al. 2018)) has focused on adapting traditional Expectation Maximization
(EM) (Dempster al. 1977) methods for data clustering to differentiable neural setting. Nonethe- less, despite strong theoretical foundations, these models do not scale to more complex multi-objjct
datasets such Multi-dSprites (Burgess al. 20T9) and CLEVR (Johnson et al. (007)). More
recent efforts (MONet (Burgess et al. 2019)) IODINE (Greff et al. 2019), GENESIS (Engeleke et al.
2019
Slot Attention (Locatello et al. 2020) have focused on application to these datasets.
These architectures differ by how exactly they segment the image between the different slots of the representation. MONet is based on a recurrent attention network that repeatedly separates parts of the image until is totally decomposed. GENESIS was designed address a key limitation of MONet, namely that it does not learn a latent representation for the segmentation, which prevents principled sampling of novel scenes. With GENESIS, segmentation masks are separately encoded, and an autoregressive prior on the mask latents is enforced. IODINE uses different strategy building upon the framework of iterative amortized inference, where an initial arbitrary guess for the posterior is progressively refined. However, this iterative process is very computationally expensive. Slot Attention, as its name indicates, introduces an attention based iterative encoder that is much more computationally efficient than Iodine. The SPACE model (Lin et al. 2020) combines the strength of spatial attention with scene mixture models to further improve applicability. Finally, MulMON Li et al. 2020) extends object-centric representations to a setting with multi-view supervision.
In a parallel line of work, object-comppsitional generationis believed to be promising inductive bias for GAN-like models (van Steenkiste et al., 2018). Obtaining good representations and pro- ducing realistic samples are connected challenges, which both require to develop suitable decoder architectures. Very recent work has been focusing on integrating object compositionality into the
GAN framework (Nguyen-Phuod Phuoc al., 2020; Ehrhardt et al. 2020 Niemeyer and Geiger 2020).
The question of disentanglement is also present in this context (Nguyen-Phuoe et al. 2019 but it remains a secondary objective.
A.2 TRADITIONAL DISENTANGLEMENT METRICS
Several metrics Zaidi al. 2020) have been developed to evaluate the quality of learned representa- tions and allow comparison between models. We identify two main categories:
Classifier-based metrics The first group of metrics is based on fixing the value of a factor of variation and generating several samples sharing this value. Intuitively, if the factors of variation are satisfyingly disentangled in the representation, it should be possible to predict which factor was fixed from the different latents, Inside this category, the metric differ in how they exactly identify the factor. BetaVAE (Higgins et al.) 2016) uses a linear classifier to predict the factor index, while Factor uses majority vote and takes the empirical variances as input, with the goal of addressing several robustness issues. Despite these implementation differences, Locatello et al. (2019) have empirically observed that both metries are strongly correlated across a wide range of tasks and models.
Affinity-based metrics The second category of metrics is based on measuring the affinity between each factor of variation and each latent variable, and evaluating how close these are to a one-to-one mapping between factors and variables. Different ways of quantifying affinity have been proposed: the Mutual Information Gap (Chen et al. (0018) and Modularity (Ridgeway and Mozer 2018) measure mutual information between factors and variables. The SAP score (Kumar et al. 2017 leverages the linear coefficient of determination R² obtained when regressing the factor from the latent. Finally, the DCI metric is based on measuring regression feature importance, using either Lasso or random forests. These metrics also differ on how they exactly assess the separation of factors in the latent representation. The Mutual Information Gap and the SAP score measure the gap between the two most relevant variables for factor. Modularity alternatively suggests to quantify the distance to an
13

---

ideal affinity template. Finally, the DCI metric uses the entropy of the normalized feature importances as an indicator entanglement.
In developing metric, have chosen to privilege the affinity-based approach for reasons. First, it does not require the ability to fix one factor while generating samples, contrary to BetaVAE and FactorVAE. Second, the notion of affinity generalizes in very flexible way to group of factors and latent variables: it is sufficient to sum the scores of all pairs of factors and variables in the groups. We use regression feature importance as way of measuring affinity. Moreover, we privilege the entropy measure of separation over affinity-gay gap methods. Indeed, we believe that entropy captures more information about the repartition of information as it takes into account all coefficients rather than just the top two.
A.3 PERMUTATION INVARIANT LEARNING
Developing algorithms that preserve permutationninnaaiinnc no doubt major challenge for the machine learning community: similar problematics appear in a variety of applications, ranging from speaker separation in the cocktail- party problem (Yu et al. 2017) to object detection losses (Carion et al. 2020). Compared to the representation probing of IODINE, the Hungarian matching in Slot Attention, and the loU matching of MulMON (Li et al. 2020). the fundamental novelty or Algorithm 1 is that it obtains the optimal alignment directly at latent code level (that is. feature level), which is agnostic to the input type. In contrast, the alignment in IODINE and MulMON operates at image level using the segmentation mask, and the Hungarian matching in Slot Attention aligns factor predictions and labels. Besides, the matching approaches used in lodine, Slot Attention and MulMON only identify the most important slot for each target, but the DCI framework requires an importance score for each slot/object pair. For future work, it would be interesting to compare EM-probing with Sinkhorn based approaches for learning latent permutations (e.g. Mena et (2018)).
B PROOFS AND ADDITIONAL THEORETICAL RESULTS
B. BACKGROUND IN INFORMATION THEORY
The description of our metric leverages concepts originating from the field of information- theory [over (1999). In this section, we recall some central definitions and results that we will use in the rest of the manuscript. In all of the following, X, Y and Z denote three discrete random variables defined on common probability space. We denote T1...... .z (resp //... .i..... 3m and 21....: 2n) the potential outcomes of X (resp. Y and Z). K is positive real number. For brevity reasons, we denote P[a:] for P|X- zi]. an similarly for Y and Z and the joint random variables.
Definition 1. The entropy of X in base K quantifies the amount of uncertainty in the potential outcomes of X. is defined as
Hc = Elloo - P(a.)]
=
logx (P(z,)).
i=1
Definition 2.
The conditional entropy of Y given X in base K quantifies the amount of information needed to describe the outcome of given than the value of X known. is defined as
= -
log (Plus|z:)).
bid
Definition 3.
The mutual information of X and Y in base K is measure of the amount of information obtained about one of the two variables by observing the other. It is defined as
P|zn Vel.
Ix(X;Y)= P(p.. Mil log
P(
Definition 4. The conditional mutual information Ix(X; YIZ) is, in base K, the expected value of the mutual information of X and Y conditioned the value of Z. Formally.
P(ra, Mj, #k.
Tx [x(X;Y|2)
Σ
SP(s. Uj 2x] logx
([ Ply 2k
14

---

The following Lemmas standard results of information theory. As these well-known, did not provide proof here and we refer to textbooks such as (over (1999).
Lemma I (Change of base). Let K be positive real number. The change of base formula for entropy is
(x(X)
H (X)
logk1 K
Similarly, we have for conditional entropy that
(X)(X = H (V log K
Lemma 2 (Subadditivity of entropy). Let Xi.... Xn be arbitrary discrete random variables. We have
Hg Xn)
(x():).
t=1
The same holds for conditional entropy, i.e.
Hx X S Σ (x)))
Lemma (Nonnegativity of mutual information). The following holds
(x Hx Hx(X)Y) I (X;
Conditioned third variable, this generalizes to
Hx (X H 1x (X; 2| 2
Lemma (Relation of joint entropy to individual entropies). Let X XI.... Xn be n arbitrary discrete random variables. We have
Xnn ) 2 max H Hx(X,) (Xi).
1=1
The same holds for conditional entropy, i.e.
Hx Hx(X, (X1, (.I......IIY)) máx H (X,
i=1
Lemma 5 (Joint conditional entropy). The following holds
H ( = (X + Ixx (X(
B.2 FORMALIZATION OF THE STRUCTURED SETTING
The specificity of our setting compared the DCI metric is that the structured organization of latent variables and factors of variations can not be accurately described by scalar indexes. We propose to account for this structured organization via tuple indexes, in which each element of the tuple is responsible for a hierarchy level. In general, the structure over factors of variation expresses the goal of metric (e.g. measuring object separation), while the structure over latent dimensions depends the model architecture.
Formally, both structures be defined relations. Consider attributes A1,..., An representing levels of hierarchy. Each attribute is associated with a set of possible values (a domain) for factors domp (Az) and for latents dom;, (A,). These domains constrain the possible values for the tuple indexes, such that the set of tuples T describing the factors of variation a subset of II domp(A,)
and the set of tuples 7 describing the latent dimensions is subset of I[ domn (A4). Thus, the factor values for sample can be denoted (vF)reT, and the representation z can be written z¹ (#:) We also denote [T| F and | L.
15

---

Toy example we consider data generating process with two objects having two properties each. The first attribute A is responsible for the object hierarchy level, while Ag is responsible for the property level. The structure over factors is formally defined
=
{(bbeec 1, color), (object 1, size), (object 2, color), (object 2, size)}
Supposing that there 2 slots with dimensions each, the structure latent dimensions is defined as
-= {(slot 1, dim 1), (slot dim 2), (slot 2, dim 1), (slot 2, dim 2))
B.3 PROOFS OF MAIN PAPER
Theorem 1. With the total projection - (1.....m). our metric captures the disentanglement and completeness metrics of the DCI framework (Eastwood and Williams 2018).
Proof. We will only present the proof for the completeness metric, as the situation for the disentan- glement metric is exactly symmetric, when switching X and Y and rows and columns. With the
total projection i = (I.....n). we have ps (Y) Y, ps (X) X, U (p:(T) = 17| = L. and V (p: = [T|
Therefore, completeness with respect to the total projection is defined
C(i) = H7 (XIY).
According to Definition
we have
Hg (x
Σ P[X = P, 1 f/Y = =1]
FET,FET
Σ TET
PLY = Σ
P[f]T] (P(f)7)).
FET
Now, let us observe that the conditional probability P[+[T] exactly the term Pr,r of the DCI framework that denotes the "probability" of latent & being important to predict factor Consequently, the conditional entropy can be rewritten as follows
Hy (X = Σ P[Y = =]] Σ Pr.r logl (Pr.) Σ P -(H
TET
fET
TET
This yields
-1 - Σ - TET
Σ P
TET
We observe that - is exactly the completeness score in capturing factor defined the DCI framework. Besides, P| = r] can be rewritten as
Σter Ri
PlY r| = Σ PlY x
E R
*et
VETJET
which is exactly the relative generative factor importance used by Eastwood and Williams to construct weighted average expressing overall completeness. This indicates that our probabilistic view of the affinity matrix and metric naturally captures all components of the DCI framework, including the final weighted average step.
For the next theorem, formally define the meaning of decomposition of projections in Definition S Note that the formalism is slightly different as our original notations were simplified for the mai paper. Despite differences in notation, union and decomposition of projections totally equivalent.
16

---

Definition 5 (Union of projections). Consider k disjunct projections i' = -ei, .h.......,hh.........
(ei...., ( } of the generative model. The union of these projections defined
It is projection of size 1 +... 1x-
Theorem 2. [Lower bound] Consider k disjunct projections ii, i of the relations. Let us suppose that i² ,....,"" j have respectively Li,..., L* groups of latents and F*, ...... F*h groups of factors. Moreover, assume that the joint projection = U* - 1
groups of latents and F groups of factors. The following lower bound for the joint completeness holds
1-
C(i logr (L)
1 1- Σ C(1)
E
C
1
Similarly, we have for the disentanglement metric
D(1)) #=1 log (F)
1 1-k Σ D(E) 1
Σ
D
Proof. We only detail the proof for the completeness metric since both cases are exactly symmetric. C(i) defined as follows
C(i) 1 Hy (p.(X)lp:
Now, let us observe that the joint projection p; and the concatenation of the different projections I Pr are similar up to the permutation of the dimensions that originates from sorting the merged index sequences. This permutation has no influence on the conditional entropy since it defined as joint expectation on 7 T. Therefore we have that
C(i) = - Hr II
(X)
PM
II
(1)
According to Lemma
2 we have the following inequality for the joint conditional entropy
k
& H
k pr (x))
Hy
I
pv X)I II (I)
(X
ps (x)
s=1
According to Lemma
we also know that
Hy Per (X)) (X) II
Hy (X)lx
Applying Lemma
Lo change base, we obtain
Hr. (X) HIs (pr. (X)(o)))+)
1 C(i*) log (T.)
logi, (T.)
Together with Equation
1 this yields
C(i) #=1 log (L)
SC C
1
s=1
Since logz~(L) and C(i*) 2 0, we finally get
1 C(ia)
1- k Σ C(i)) <
log,, (L)
which concludes the proof.
17

---

B.4 ADDITIONAL RESULTS
Theorem 3 describes an upper bound for the joint metric attempting to match Theorem However, this upper bound comes in weaker form which is totally controllable. Intuitively, this due the fact that our metrics convey strictly more information that the unstructured DCI framework.
Theorem 3. [Upper bound] Consider k disjunct projections it i of the relations. Let us suppose that i' ,'..., have respectively L',.... L'...., L groups of latents and F",... ...,, F* groups of factors. Moreover, assume that the joint projection = U*=1 i has groups of latents and F groups of factors. The The following upper bound for the completeness metric holds
(-c(it)
C(i)
<1
max Isask
Ag
logr. L
where
Ag = 1t
pr UU (x))
capa
Similarly, we have for the disentanglement metric that
1 D(i*) B
D
1
logr, F 11 max 1seSk
where
B, = IF pi- (rr(Y); (X))os-
Proof. We only prove the bound for the completeness metric, the case of disentanglement is exactly symmetric. C(t) is defined as follows
=(i)=1- = Hr (p((X))))))))
Similar to previous proof, we observe that the joint projection p; and the concatenation of different projections II Pi- are similar up to the permutation of the dimensions that originates from sorting the merged index sequences. This permutation has influence on the conditional entropy since is defined joint expectation on T T. Therefore have that
C(i) = - Hr
II P (x) s=1
II pa pa(())
(2)
1=1
According to Lemma
we have the following inequality for the joint conditional entropy
k
k
pr. (X) pa po
II pr (X) II po)
max Hy
H
t=1
1=1
According to Lemma
we also know that
& Hi pr II
pa (x)
= Hr (pr - As
where
A 41 po(X)i (
t
Applying Lemma] to change base, we obtain
1- 1(*))
Hr. pi (X) II
As.
log To
1=1
Together with Equation 2 this yields
1 -(1)
C(i) <
max ISo5k
AA
log L
18

---

Interprefation of the upper bound The goal of Theorem is to the individual projections to obtain an upper bound for the joint disentanglement (resp. completeness). Intuitively, the meaning of the bound
(1-C(1)) log L
C(i) <
max IseSk
is that the joint completeness can not be better than any of the individual completeness. However, this bound is weaker than Theorem eeooremmbecause because of two main restrictions. First, it harder get rid of the logz, (I. term as I. is greater than I,". One possibillity is to notice that I, S T," ..... I. If we make the additional assumption that I," ... It,, then we obtain log,,- (I.) S k. which gives us
1 min ssak
(C(i) + +A,) A.)
C(i) k
This first assumption can be considered reasonable. But more importantly, we notice an additional interaction term A, in the minimum that is based on mutual information between the different projections. This term can not be removed as it non-negative. The intuition for this conditional mutual information is that it measures the dependence of the different projections. It is 0 exactly when pt (X) and UUge** (Y) are independent conditioned on pa (Y). Assuming that this is case,
we obtain the simplest inequality
C( ISe5k
c(i) S
min
However, there absolutely no guarantee that this assumption is valid. conclude, the upper bound weaker than Theorem 2 because it depends the additional terms A, and B, which account for the degree of dependence between projections and can not be casily controlled. This can be seen as a confirmation that metrics convey strictly more information that the unstructured DCI framework.
In the following, we study the specific case where there are only two levels of hierarchy (k: = 2). In actually derive exact formula rather two matching bounds.
this case,
Theorem 4 (Case k 2). Consider 2 disjunct projections i' and i2 with respectively L and L groups of latents, and F F2 groups of factors. Assume that the joint projection U has groups of latents and F groups of factors. For the completeness metric, the following identity holds
C(i) = 1 1-C(i)) 102(1)
log.: log
L1PPP (X): (Y))p;;({))
Ilpp (X); p:a (Y)lp; (Y))
Ip (X): pa (x);
pa (Y),
In a symmetric way, for the disentanglement metric,
D-D(i') 1- D(i2) D(1) log, (F) logr=(F) (F)
+Ir(pn (X))pxxx
+
(X))
px))
Proof. Again, we only prove the result for the completeness metric as disentanglement is exactly
symmetric. In the case where & = 2, the joint completeness is defined as follows
C(i)
H, (P)))))))) = - H, (pa (1)
With Lemma 5 we can rewrite this last expression
C(i) = 1 - Hr (po (X) ())) - Hz (X=(()))
1 (po (X); pr (X) pr)))
Applying Lemma 3 twice, we obtain
Hy (pa (Y), Hy (X))) Iz (1000((0)0))))00))))))))))
19

---

and H (X)(X))p (Y), p²)())
He (p=(()) - (7))) (M)p())))))))))
Using Lemma T to change base for the two conditional entropies gives us the final equation
C(i) = 1 -(i) 1 1- C(12)
(L) (L)
Ielpo (X))::((())))(()))))
+ (X)))) (X))+
+ Leaa (x)
which concludes the proof.
L
C REPRODUCIBILITY
C.I PERMUTATION INVARIANT FEATURE PROBING
Complementary to permutation invariant probing algorithm, use specific algorithm for generation of the evaluation dataset. It consists in dividing the evaluation dataset into groups of inputs with similar factor values. First, this helps for the initialization of our EM-like approach, the object-to-slot assignment exhibits less variation for similar images. Second, this allows to visually verify that the learned permutation is consistent across inputs (Figures and (9.. Finally, breaks the symmetry between objects, allowing for more meaningful visualizations of the projected latent space. To obtain a global metric, we average the values across the different groups. The method for generating the evaluation dataset is summarized in Algorithm 2 Exact hyperparameters are given below, and an ablation study can found Section D
Algorithm Generation of the evaluation dataset for g to u_nggggg groups do # Generate initial factor values for the group for f in factors do
Sample initial value inilial[J' for factor f
for i = 1 to n samples do values locally
# Samplee
for f in factors do
Sample value image_il [i[[[] near initial[s]
Generate image from factor values image_
C.2 MODELS
MONet We train MONet exactly as in Burgess et al. (2019), except that we set og = 0.1 and Tbg 0.06 which was shown yield better results (Greff et al., 2019). Besides, use GECO (Rezende and Viola 2018) to automatically tune the / parameter during training. The reconstruction constraint was manually set to ensure satisfying visual results (-1.78 for Multi-dSprites, -1755 for CLEVR6). We found it especially important for the beginning of the training to set minimum value of for S.. Failure do so resulted in frequent local optima for the representation in which the network was stuck. We used the implementation provided by the authors of Engelcke et al. (201)) Note that contrary to the latter, we do not include the GECO framework. We use the value of 0.5 given the authors of MONet. The parameter controls mask reconstruction loss, therefore we did not set
in MONet (R).
We follow the training process described in (Engelcke et al. 2019) and use the imple- mentation provided by the authors, under GNU General Public License. CLEVR6 is not considered in this paper: we used the same parameters as for Multi-dSprites and changed the reconstruction objective to 114228.
GENESIS
20

---

We use the parameters described in (Greff et al. 2019), and a third-party implementation For Multi-dSprites, we had to increase the a parameter from 0.1 to 0.14 in order to obtain satisfying results with the slight variations in our dataset.
IODINE
We train all models for 200 epochs. This corresponds to approximately 300 000 updates on Multi- dSprites and 450 000 updates on CLEVR6. Models were trained with one to four V100 GPUs. The training times ranges from a few hours to 1.5 days.
C.3 TRAINING DATASETS
Multi-dSprites For training, we the dataset and preprocessing code from Engeleke et al. (2019).
Note that the datasets contains images composed of to 4 objects, contrary to (Greff et al. 2019)
that goes up to 5 objects. This explains the better segmentation values in our setting compared to the latter.
CLEVR6 We generate the training dataset similarly as in Greff et al. (019), with the modification that we add an additional constraint that all objects have be visible inside the cropped 192x 192 image. This modification important for our metric. It tends to generate slightly denser scenes than in previous work. This might partly explain why we get slightly worse ARI for MONet compared to previous work (0.94 against 0.96). Scenes have at most 6 objects.
C.4 EVALUATION
Evaluation datasets The evaluation datasets sampled in the same way, except that restrict to images with 4 objects for Multi-dSprites and images with exactly 6 objects for CLEVR6. We sample 10 local groups (see Algorithm 2) for Multi-dSprites and 5 for CLEVR6. Each group has 5000 samples, with a 4000/500/500 split for fitting, validation and evaluation of the factor predictor. In Table 7 give the list of factors for each individual object in both datasets, with the possible values for each factor.
Factor prediction For the temporary predictors in Algorithm (inside the loop), we use linear model with Ridge regularization. For the final predictor, we use random forest with 10 trees, and a maximum depth of 15. This because the random forest obtains better predictions, while the linear model permits faster iterations. The number of iterations of the loop is set to 100 for Multi-dSprites. For computational reasons, we reduced this number to 20 for CLEVR6. This reduction does not harm performance as the vast majority of permutations happen in the first iterations. Most factors continuous ordinal: for these, encode factor prediction factor in CLEVR6 has only two classes and we follow the encoding used in sklearn Ridge Classifier. On both datasets, the shape factor has three classes: we generalize the Ridge Classifier encoding and encode the classes as -1, O and -1. We match the classes to these values in order to minimize prediction error.
regression task. The material
Max tree depth We tried different max tree depths (5, 10, 15, 20, 25, 30) on a validation set and observed that (i) value of 15 generally gives better validation performance than 5 and 10 (ii) values > 15 do not significantly improve validation performance and get very slow to This is why we chose to always go with 15. Note that this optimal value might be related to the number of samples used to fit the predictor (currently 4000). Note that the validation was done globally, and not per factor.
Metric In order to filter out noise in the feature importances, we set to 0 to all the relative importance coefficients that are less than 3% of the column maximum in absolute value. In our evaluation of object-level disentanglement, related. For Multi-dSprites, the background slot is identified as the one with higher importance in predicting background color. For CLEVR6, there is no background color, but the background is always reconstructed by the first slot in the benchmarked models. The global informativeness metric an unweighted average over all factors.
remove the factors and latent dimensions that are background
githuh.com/ztiuaan-niiOODD provided license.
21

---

Table 4: Ablation study of Algorithms and for Multi-dSprites. We report mean and std error over three random seeds. All values are in %. 100 is the best possible score and the worst, except for the informativeness metric where 0 is best. All models are trained with full loss.
OBJECT-LEVEL
MODEL
DIS. (f)
COMP. (f)
MONET (ATT.)
52 = 1
52 +
MONET (ATT.) (WITHOUT ALG
30 2
30
MONET (ATT.) (WITHOUT ALG 2)
57 $7-11
57
MONET (ATT.) (WITHOUT ALG 1 AND 2)
22 1 0.5
22 12:40 0.5
MONET (REC.)
75
7
MONET (REC.) (WITHOUT ALG
3
40
MONET (REC.) (WITHOUT ALG 2)
61
61+4
MONET (REC.) (WITHOUT ALG AND 2)
24 2:
24
GENESIS
90 0.8
91 91::00 0.6
GENESIS (WITHOUT ALG
0.1
9.1
GENESIS (WITHOUT ALG 2)
6.4
61
GENESIS (WITHOUT AL. AND
0.3
33 0.3
IODINE
70
72 = 10
IODINE ALG
32:22
33:22
IODINE (WITHOUT ALG 2)
66
66:1
TODINE (WITHOUT ALG AND 2)
26 0.9
26 0.9
Table 5: Informativeness and Completeness per object property for all models trained with full loss on Multi-dSprites.
COMPLETENESS (t)
INFORMATIVENESS (1)
(A) (DEC)
(A) (DEC)
CHANNEL
71
94
16
G CHANNEL
90
68
81
89
10
17
B CHANNEL
90
77
86
96
9
16
4
5
SCALE
32
49
55
38
53
41
ORIENTATION
24
48
36
45
31
65
97
62
Y
33
63
39
67
35
48
74
34
Table Informativeness and Completeness per object property for all models trained with full loss on CLEVR6.
COMPLETENESS (t)
INFORMATIVENESS (1)
GENESIS
MONET (ATT.)
MONET (REC.)
GENESIS
MONET (ATT.)
MONET (REC.)
R CHANNEL
54
47
47
47
57
61
B CHANNEL
61
54
46
38
47
58
SHAPE
27
39
64
66
SIZE
49
64
63
24
28
22
ROTATION
NA
36
106
107
27
54
41
Y
31
61
65
30
39
32
22

---

a) Before Algorithm 1
b) After Algorithm 1
MONet
IODINE
GENESIS
Input
Slot
Slot 1 Slot Slot
Slot
Slot
Slet 1 Slot Slot
Slot
Figure 7: Visualization of the effect of Algorithm 1 on slot ordering for a group of similar inputs.
23

---

a) Before Algorithm 1
b) After Algorithm 1
MONet
IODINE
GENESIS
Input
Slot
Slot 1 Slot Slot
Slot
Slot
Slet 1 Slot Slot
Slot
Figure 8: Visualization of the effect of Algorithm 1 on slot ordering for a group of similar inputs.
24

---

a) Before Algorithm 1
b) After Algorithm 1
9<n
MONet
IODINE
GENESIS
Input
Slot
Slot
Slot
Slot Slot
Slot Slet 1
Slot
Slot
Slot
Figure 9: Visualization of the effect of Algorithm 1 on slot ordering for a group of similar inputs.
25

---

Multi-dSprites GENESIS
MONet (Att.)
MONet (Rec.)
SLOTS
SLOTS
SLOTS
0qpe22
Oijects
Clper
Osject
Objece
Cojict
LATENTS
LATENTS
Sh600
51.07 SCRE ORRE
IODINE
SLOTS
LATENTS
Obeei44
LATENTS
Scafe: CH.int
Shape SOle OPATS
Figure 10: Projections of the affinity matrix at object-level and property-level for all four models trained with full loss on Multi-dSprites. (Hinton diagram)
26

---

CLEVR6
MONet (Att.)
GENESIS
SLOTS
OURCE DEBISS SEACE
LATENTS
SNSpeRENT Sm fanaion
MONet (Rec.)
LATENTS
OMier 0Rler1
OUIITS (Rlect
LATENTS
Figure 11: Projections of the affinity matrix object-level and property-level for all three models trained with fullloss on CLEVR6 (Hinton diagram).
27

---

Table 7: Description of the factors of variation for each individual object in both datasets. For each factor, we give the set of possible values, as well as the exact locality constraints used in Algorithm 2. a denotes the initial value sampled in Algorithm When x: has values in an array, we denote i the index such that = afi]. The Table indicates whether each of the considered factors is intrinsic or extrinsic. The question of whether to consider object size as intrinsic can be debated, however this does not change the results of our experiments.
DATASET FACTORS OF VARIATION
TYPE
POSSIBLE VALUES
LOCAL CONSTRAINT (ALG. 2)
COLOR (R CHANNEL)
INTRINSIC
[0, 63, 127, 191, 255]
[a]i 1]] afa + 1]]
COLOR (G CHANNEL)
INTRINSIC
[0, 63, 127, 191, 255] [0, 127, 191, 253]
[u]i 1], a[J alt 1]]
COLOR (B CHANNEL)
INTRINSIC
[a|i 11, a[j) afs 1]]
SHAPE
INTRINSIC
(circle, square, heart)
(cirele,squaeee heart)
MDSPR
SCALE
INTRINSIC
[o, 1, 5,
[u|s - a[j] alt
ORIENTATION
EXTRINSIC
[0, 1....... 30,
[a[i
3) ali 2I..... 2/ a[i
X COORD.
EXTRINSIC
[0, ......... 30, 31)
[ali 2)..... oft
Y COORD,
EXTRINSIC
[0. 1, 30,31 31
[a(t 2). olt 211
COLOR (R CHANNEL)
INTRINSIC
[0,0.2, 0, 0,6,6, 0.8.
[o]i 11, a[1 alt
1]]
COLOR (G CHANNEL)
INTRINSIC
0., 0.0
[a/i 11: a|: als 1]]
COLOR (B CHANNEL)
INTRINSIC
[0, 0.2, 0, 4.0.6 {eube, pphere,cylindee (rubbor, metal)
[a|t - 11, a[]] als 1]]
SHAPE
INTRINSIC
{cube, sphere, cylinder) (rubber, metal)
CLEVR6
INTRINSIC
SIZE
INTRINSIC
[0.35, 0.7]
[0.35, 0.7]
ROTATION
EXTRINSIC
[0,360]
360] 0.7, 0.7 1 3-3,3] -
X COORD.
EXTRINSIC
[-3,3]
Y COORD.
EXTRINSIC
3]
0.7, 0.7 n [-3,3]
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 VISUALIZATION OF THE PERMUTATION INFERRED BY OUR FEATURE IMPORTANCE ALGORITHM
In this Section, inspect the slot ordering inferred by Algorithm by visualizing the masked reconstruction for each slot. Note that Algorithm does not make use of these reconstructions. In we compare slot ordering before (on the left) and after (on the right) applying the algorithm, for two groups of similar images (generated following Algorithm 2). We include all the considered architectures, trained on Multi-dSprites with variational loss.
Figures g and 2
On the left, we observe that the slot ordering inferred by the models is not consistent across similar images. This effect is particularly visible for IODINE, despite the fact that we set the model internal noise to deterministic value (following observations in Greff et al. (2019)). The background slot is not even deterministic. This unpredictable ordering is extremely detrimental to the accuracy of our metric. Applying our metric to IODINE with this initial ordering yielded extremely poor values which do not do justice to the good representation learned by this model.
On the right, we see that Algorithm successfully learns consistent slot ordering, and puts matching objects at the same position. We notice almost perfect alignment, except for some failure cases with IODINE, where the background slot is switched with missing object. We uncertain to the reasons for this failure, but has limited impact on final performance as we remove the background factors in the object-level projection of the affinity matrix.
D.2 VISUALIZATIONS OF THE LATENT SPACE
In Figures 10 and II we show visualizations of the projected latent space for the different architec- tures (trained with variational loss) for group of inputs in both evaluation datasets. The qualitative inspection of the object-level projection is consistent with the numerical comparison: GENESIS is visually more disentangled, while MONet and IODINE achieve inferior but still satisfying disentan- glement. On Multi -Sptiid, the under sedder-prffrmance of the MONet (Att.) variant is also visible. On Multi-dSprites, we observe that the GENESIS background slot also contains information about all
28

---

objects. This is not surprising the background mask is in object masks. However, this phenomenon is less marked with other architectures. We think that the specific mask encoding of GENESIS somehow strengthens this duplication of information between background and object slot. Note that this does not harm performance as we removed the background slot in the final metric. We believe that this post-processing is fair because of the expected duplication effect described above.
negative of the different
Qualitative comparison of the property-level projections is harder due the higher number of dimensions. Still, some observations can be made. Among object properties, color consistently obtain the best separation the representation. On the GENESIS model, also notice clear separation between the last 16 dimensions that correspond to the component latent and the rest of the slot. Color almost exclusively encoded by the component latent.
D.3 DECOMPOSITION OF THE INFORMATIVENESS AND COMPLETENESS PER FACTOR
In Table decompose informativeness of the different models per property. Note that the completeness results do not necessarily average to the global object-level completeness as our metric involves weighted average. On Multi-dSprites, mostly comes from the group of extrinsic factors that are related to the segmentation mask. This would support the hypothesis that the innovative mask encoding of GENESIS is least partly responsible for the performance increase. However, this effect is less clear on CLEVR6.
observe that the superior results of GENESIS
On CLEVR6, we notice particularly bad metries for the rotation property. This is not surprising as the rotation parameter is completely useless for two of the three shapes (sphere and cylinder) and redundant for the last one (cube). Because of this bad identifiability and affinity scores thresholding. we can not compute rotation completeness for GENESIS. This does not impact the final metric due to the weighting scheme of our metric. Concerning color channels, we note very different behavior between Multi-dSprites and CLEVR6. We believe that this is due to particularity of CLEVR6 which that the set color is restricted in the training dataset.
D.4 ABLATION OF ALGORIHTS AND 2
In Table compare the object-level values given metric with and without Algorithms 1 and 2. We observe a significant performance drop when abalating these algorithms. This is consistent with visual inspection of the slot ordering.
29


Reviews:
[{"review_id": "lxzVqgPcske", "reviewer": "Reviewer_ERyE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Compositional disentanglement is necessary when there are multiple objects of interest in an image. This paper addresses the problem of evaluating models that learn compositional representations.\n+ Permutation-invariant representation probing is a necessary contribution that extends the mapping between latents and generative factors to the object-centric case.\n+ The experiments performed and results displayed are comprehensive and provide a holistic view of the metric within existing methods.\n\nWeaknesses:\n- It would be better to include some real-world examples of the need for compositional disentanglement and the motivation for more than one slot (i.e., global disentanglement).\n- How does this study of compositional disentanglement address spurious correlations in the data? Especially, if object and color are spuriously correlated, can ‘slot disentanglement’ identify such spurious correlations?\n- Object-level disentanglement says that there should be a one-to-one mapping between slots and objects. However, the example given in Figure 2 has two slots for each object. It would be better to explicitly state how the matrix is reduced to the mapping in empirical studies.\n- Usually, it makes sense to visualize each row of affinity matrix R to sum to 1. However, in section 3.2, it is mentioned that R can be normalized to make the sum of entries of R equal to 1. Is this a valid way of normalizing R? If yes, explain it clearly as this idea is being used in proofs of theorems.\n- In the experimental results, the way we find the dimensions to project on is not clear. I think this is also important in identifying the invariant permutations of latent codes.\n- The paper states: “ablation tends to improve pixel-level segmentation. This would imply that visual object separation is directly caused by architectural inductive biases, which puts into question the implicit beliefs of prior work.” Isn’t this expected as the ablation performed removes regularization (needed for disentanglement) and the subsequent improvement is merely proof that the object separation works?\n\nMinor Comments:\nTheorem 1 in main paper is referred to as Theorem 3 in Appendix\n", "summary_of_the_paper": "This paper proposes a metric for evaluating disentanglement in compositional representation learning upending existing global disentanglement metrics, that disregard any representational structure. The proposed metric is based on the projections of the affinity matrix proposed earlier in the DCI metric. The paper also proposes an EM-like permutation-invariant formulation for obtaining the relative importance of a latent in predicting a generative factor. This allows for slots to be permuted wlog. Empirical studies are performed to study the disentanglement of existing compositional representation learners.", "summary_of_the_review": "This paper addresses a timely problem and the proposed evaluation metric and representation probing methods are sound. However, this paper needs to explain some points more clearly to better understand the work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JO5Ib9bqw7V", "reviewer": "Reviewer_ZLzf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**: Valuable generalization of the DCI metrics of Eastwood et al. (2018) to object-centric representations. Both contributions (the projections and the object-matching algorithm) are novel, interesting, and of use to the object-centric representation-learning community.\n\n**Weaknesses**: The clarity of writing, framing of the claims, and experimental evaluation are weak. These need to be improved in order to determine the strength of the proposed metrics. See points below.\n\n**Clarity**: While Fig. 2 and Alg 1. nicely convey the main contributions, the clarity of writing could be improved as the authors tend to: i) dump in results and expect the reader to analyse them themselves (rather than taking the reader through the results with good explanations); and ii) often make statements without direct pointers to evidence to back up the statements. E.g.:\n- i) *Figures 4 and 5:* what are we supposed to take from these?\n- ii) *Sec. 5.3:* many statements made without evidence. E.g. why are the results \"satisfying\" in Section 5.3 -- disentanglement, completeness and informativeness seem poor? Where can we see the “visually satisfying results”?\n\n**Claims/framing**: *“stronger guarantees […] than previous disentanglement metrics”*. Theorem 1 shows that DCI is a special case of the proposed metric, while Theorem 2 shows that when the proposed metric is perfect then so too is DCI. If DCI is also perfect, then surely these guarantees also apply to DCI? How are your guarantees *stronger*?\n\n**Experimental evaluation**:\n- *Representation-probing alg.*: Figure 5 seems to be the only evaluation, and it is not clear how to read this figure -- what is a “consistent slot ordering”? A better evaluation would really help here, e.g.:\n    - Compare with the matchings obtained from segmentation masks.\n    - Show the standard error over seeds *for the same representations/trained method* (to decouple the methods’ variability from the metric’s variability). The authors note that their framework exhibits higher variation than existing metrics -- do they attribute this to the probing algorithm? Or to their metric better-evaluating unstable methods? It would be nice to isolate these sources of variation to truly evaluate the proposed framework.\n\n- *Disentanglement vs. informativeness*: Figure 3 and Table 2 tell a consistent story: models trained with a regularization loss that encourages disentanglement do indeed achieve better disentanglement scores with the proposed metric. However, as demonstrated in [1, Fig 6, right], [2, Table 1b], and this paper (Figure 3c), disentanglement regularization often comes at the cost of reconstruction/informativeness. Why is it then that, in Table 2, informativeness is consistently better for models *with* the disentanglement regularization? This seems very counter-intuitive as the unregularized model should capture at least as much information about the underlying factors, and raises questions about the results obtained.\n- *Predictor f*: What was used in the experiments -- a linear or nonlinear predictor? E.g. lasso[2], random forests[2], or gradient-boosted trees[3]? This is very important information and seems to be missing from the main paper.\n\n**Additional comments/questions**: \n- Slot symmetry (Section 3.1, point 3): is this ever explicitly evaluated (i.e. separately to slot disentanglement)? The authors claim all 3 criteria can be evaluated by their metric -- do they mean implicitly (i.e. if slot symmetry is poor, so too will be their metric scores)?\n- [4] also evaluate intra-object disentanglement (see Section 5.3) by first matching latent slots to GT objects using IoU (see Section 5.1). Your probing algorithm is more general and elegant, but it is worth mentioning this more manual approach to intra-object disentanglement in e.g. the related work section.\n- “The variational loss” (Section 5.2) is quite cryptic -- I’d suggest something like “disentanglement regularization”\n- Line 165: did the authors mean permutations over latent *slots* rather than latent *codes*?\n\n\n\n\n[1] Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017). \\beta-VAE: Learning basic visual concepts with a constrained variational framework. In *International Conference on Learning Representations*.\n\n[2] Eastwood, C. and Williams, C. K. I. (2018). A framework for the quantitative evaluation of disentangled representations. In *International Conference on Learning Representations*.\n\n[3] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2020). A sober look at the unsupervised learning of disentangled representations and their evaluation. *Journal of Machine Learning Research*, 21(209):1–62.\n\n[4] Li, N., Eastwood, C. and Fisher, R. (2020). Learning object-centric representations of multi-object scenes from multiple views. In *Advances in Neural Information Processing Systems*.\n\n", "summary_of_the_paper": "This paper proposes a new method for evaluating the disentanglement of object-centric representations. It can be seen as generalization of the DCI metrics of Eastwood et al. (2018) to object-centric representations, permitting the evaluation of both inter- and intra-object disentanglement. The two main contributions are: (i) proposing projection/marginalization operations on the matrix R in order to select an abstraction level (e.g. object or property level, as nicely illustrated in Figure 2); and (ii) an EM-based object-matching or \"representation probing\" algorithm which matches latent object slots with underlying GT objects (specifically, it optimizes the latent slot permutation to maximize the accuracy of the linear predictor f:z → v). ", "summary_of_the_review": "Interesting and novel ideas, but the often unclear writing, questionable claims/framing and weak experimental evaluation leave me unconvinced of their efficacy. I believe this paper is currently just below the acceptance threshold, but am eager to improve my score if the authors address the aforementioned concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5rMEuYA0wwV", "reviewer": "Reviewer_uWyU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written and of overall high quality. The problem addressed in this paper -- evaluation of object-centric generative models -- is of high relevance to the field, especially given the limited choice of metrics available at present and the unsuitability of standard disentanglement metrics for multi-object scenes. \n\nThe metric introduced in this paper is simple, novel, and of high significance. The experiments are insightful and allow for discussion/evaluation of prior work under a new light -- in the context of feature disentanglement. Especially the insight that regularizers on the latent space (as used in variational objectives) appear to have a positive impact on disentanglement scores but a negative impact on (visual) scene decomposition scores (e.g. segmentation mask ARI) is interesting and will likely have an impact on future work in this area.\n\nIn its present form, I still have two major concerns with the paper:\n\n* I am unsure about the significance of the core technical contribution of this work (i.e., the representation probing algorithm to handle slot permutation invariance). It is unclear to me why earlier approaches, such as training a per-slot probe using Hungarian matching w.r.t. a set of target variables (as done in Slot Attention; Locatello et al., NeurIPS 2020), are unfit for this task. Appendix A.2 explains that the probing/matching approach used in Slot Attention does \"not permit to identify which latent slot was used for the prediction, which is essential to measure disentanglement\". This statement is unclear to me, as the Hungarian matching algorithm returns a permutation matrix which does indeed provide 1-to-1 correspondence between slots and targets, i.e. we can directly read off which slot was used for the prediction. It is possible that I misunderstand the argument by the authors, and I would appreciate it if this could be clarified.\n\n* The images in Figure 1 appear to have been taken/copied from Figure 20 in the Slot Attention paper. Copying parts of figures without referencing the original work is not good scholarship and should be fixed. Since these are visualizations of model predictions, it should be fine to include them here without asking for explicit permission by the authors, but the source should at least be acknowledged in the figure caption.\n\nOther comments:\n* It would be good to include y-axis labels in Figure 4 (or a more descriptive annotation). In its current form, it is unclear whether the x- or the y-axis represents latent slots (vs. ground truth objects).\n* This sentence is (grammatically) unclear (Section 3): \"[...] since it provides stronger guarantees than the selected model is correctly disentangled\".\n", "summary_of_the_paper": "This paper introduces a new metric for evaluating disentanglement of latent representations, which -- unlike prior work -- supports object-centric, structured latent spaces with a set of latent variables (as opposed to a single global latent variable). While prior work on object-centric generative models primarily evaluated their ability to decompose visual scenes into spatial regions corresponding to individual objects, this novel metric allows for systematic evaluation of disentanglement of learned object properties within and across \"object slots\" (i.e. individual latent variables) in these models. The paper further introduces a representation probing algorithm using an EM-style iterative procedure, which is invariant to slot permutation. The metric is experimentally demonstrated on standard unsupervised scene decomposition datasets and models.", "summary_of_the_review": "Overall, this paper presents a significant advance in evaluation of object-centric generative models and provides a sound new metric for this task. Despite my concerns mentioned above, I believe this paper can be accepted (although it is borderline).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new metric for disentanglement of learned representations, extending a prominent framework (DCI) to support object-centric structured representations.

The reviewers agree on the importance of the question and find the metric a valuable contribution for addressing this problem. In the discussion, the reviewers identified some clarity issues that the authors have improved, leading to an overall much better writeup, as well as some deeper evaluation of learned matching agreements. The main remaining points that could be improved are
 - making the results more robust with thorough hyperparam tuning
 - connecting to other methods for inducing soft / probabilistic matchings, such as Sinkhorn or smooth&sparse optimal transport.

Please consider switching to the Times font as recommended by the ICLR style guide.
Decision: Accept (Poster)