Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a metric for evaluating disentanglement in compositional representation learning upending existing global disentanglement metrics, that disregard any representational structure. The proposed metric is based on the projections of the affinity matrix proposed earlier in the DCI metric. The paper also proposes an EM-like permutation-invariant formulation for obtaining the relative importance of a latent in predicting a generative factor. This allows for slots to be permuted wlog. Empirical studies are performed to study the disentanglement of existing compositional representation learners.
main_review: Strengths:
+ Compositional disentanglement is necessary when there are multiple objects of interest in an image. This paper addresses the problem of evaluating models that learn compositional representations.
+ Permutation-invariant representation probing is a necessary contribution that extends the mapping between latents and generative factors to the object-centric case.
+ The experiments performed and results displayed are comprehensive and provide a holistic view of the metric within existing methods.

Weaknesses:
- It would be better to include some real-world examples of the need for compositional disentanglement and the motivation for more than one slot (i.e., global disentanglement).
- How does this study of compositional disentanglement address spurious correlations in the data? Especially, if object and color are spuriously correlated, can ‘slot disentanglement’ identify such spurious correlations?
- Object-level disentanglement says that there should be a one-to-one mapping between slots and objects. However, the example given in Figure 2 has two slots for each object. It would be better to explicitly state how the matrix is reduced to the mapping in empirical studies.
- Usually, it makes sense to visualize each row of affinity matrix R to sum to 1. However, in section 3.2, it is mentioned that R can be normalized to make the sum of entries of R equal to 1. Is this a valid way of normalizing R? If yes, explain it clearly as this idea is being used in proofs of theorems.
- In the experimental results, the way we find the dimensions to project on is not clear. I think this is also important in identifying the invariant permutations of latent codes.
- The paper states: “ablation tends to improve pixel-level segmentation. This would imply that visual object separation is directly caused by architectural inductive biases, which puts into question the implicit beliefs of prior work.” Isn’t this expected as the ablation performed removes regularization (needed for disentanglement) and the subsequent improvement is merely proof that the object separation works?

Minor Comments:
Theorem 1 in main paper is referred to as Theorem 3 in Appendix

summary_of_the_review: This paper addresses a timely problem and the proposed evaluation metric and representation probing methods are sound. However, this paper needs to explain some points more clearly to better understand the work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new metric for evaluating disentanglement of latent representations, which -- unlike prior work -- supports object-centric, structured latent spaces with a set of latent variables (as opposed to a single global latent variable). While prior work on object-centric generative models primarily evaluated their ability to decompose visual scenes into spatial regions corresponding to individual objects, this novel metric allows for systematic evaluation of disentanglement of learned object properties within and across "object slots" (i.e. individual latent variables) in these models. The paper further introduces a representation probing algorithm using an EM-style iterative procedure, which is invariant to slot permutation. The metric is experimentally demonstrated on standard unsupervised scene decomposition datasets and models.
main_review: This paper is well-written and of overall high quality. The problem addressed in this paper -- evaluation of object-centric generative models -- is of high relevance to the field, especially given the limited choice of metrics available at present and the unsuitability of standard disentanglement metrics for multi-object scenes. 

The metric introduced in this paper is simple, novel, and of high significance. The experiments are insightful and allow for discussion/evaluation of prior work under a new light -- in the context of feature disentanglement. Especially the insight that regularizers on the latent space (as used in variational objectives) appear to have a positive impact on disentanglement scores but a negative impact on (visual) scene decomposition scores (e.g. segmentation mask ARI) is interesting and will likely have an impact on future work in this area.

In its present form, I still have two major concerns with the paper:

* I am unsure about the significance of the core technical contribution of this work (i.e., the representation probing algorithm to handle slot permutation invariance). It is unclear to me why earlier approaches, such as training a per-slot probe using Hungarian matching w.r.t. a set of target variables (as done in Slot Attention; Locatello et al., NeurIPS 2020), are unfit for this task. Appendix A.2 explains that the probing/matching approach used in Slot Attention does "not permit to identify which latent slot was used for the prediction, which is essential to measure disentanglement". This statement is unclear to me, as the Hungarian matching algorithm returns a permutation matrix which does indeed provide 1-to-1 correspondence between slots and targets, i.e. we can directly read off which slot was used for the prediction. It is possible that I misunderstand the argument by the authors, and I would appreciate it if this could be clarified.

* The images in Figure 1 appear to have been taken/copied from Figure 20 in the Slot Attention paper. Copying parts of figures without referencing the original work is not good scholarship and should be fixed. Since these are visualizations of model predictions, it should be fine to include them here without asking for explicit permission by the authors, but the source should at least be acknowledged in the figure caption.

Other comments:
* It would be good to include y-axis labels in Figure 4 (or a more descriptive annotation). In its current form, it is unclear whether the x- or the y-axis represents latent slots (vs. ground truth objects).
* This sentence is (grammatically) unclear (Section 3): "[...] since it provides stronger guarantees than the selected model is correctly disentangled".

summary_of_the_review: Overall, this paper presents a significant advance in evaluation of object-centric generative models and provides a sound new metric for this task. Despite my concerns mentioned above, I believe this paper can be accepted (although it is borderline).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method for evaluating the disentanglement of object-centric representations. It can be seen as generalization of the DCI metrics of Eastwood et al. (2018) to object-centric representations, permitting the evaluation of both inter- and intra-object disentanglement. The two main contributions are: (i) proposing projection/marginalization operations on the matrix R in order to select an abstraction level (e.g. object or property level, as nicely illustrated in Figure 2); and (ii) an EM-based object-matching or "representation probing" algorithm which matches latent object slots with underlying GT objects (specifically, it optimizes the latent slot permutation to maximize the accuracy of the linear predictor f:z → v). 
main_review: **Strengths**: Valuable generalization of the DCI metrics of Eastwood et al. (2018) to object-centric representations. Both contributions (the projections and the object-matching algorithm) are novel, interesting, and of use to the object-centric representation-learning community.

**Weaknesses**: The clarity of writing, framing of the claims, and experimental evaluation are weak. These need to be improved in order to determine the strength of the proposed metrics. See points below.

**Clarity**: While Fig. 2 and Alg 1. nicely convey the main contributions, the clarity of writing could be improved as the authors tend to: i) dump in results and expect the reader to analyse them themselves (rather than taking the reader through the results with good explanations); and ii) often make statements without direct pointers to evidence to back up the statements. E.g.:
- i) *Figures 4 and 5:* what are we supposed to take from these?
- ii) *Sec. 5.3:* many statements made without evidence. E.g. why are the results "satisfying" in Section 5.3 -- disentanglement, completeness and informativeness seem poor? Where can we see the “visually satisfying results”?

**Claims/framing**: *“stronger guarantees […] than previous disentanglement metrics”*. Theorem 1 shows that DCI is a special case of the proposed metric, while Theorem 2 shows that when the proposed metric is perfect then so too is DCI. If DCI is also perfect, then surely these guarantees also apply to DCI? How are your guarantees *stronger*?

**Experimental evaluation**:
- *Representation-probing alg.*: Figure 5 seems to be the only evaluation, and it is not clear how to read this figure -- what is a “consistent slot ordering”? A better evaluation would really help here, e.g.:
    - Compare with the matchings obtained from segmentation masks.
    - Show the standard error over seeds *for the same representations/trained method* (to decouple the methods’ variability from the metric’s variability). The authors note that their framework exhibits higher variation than existing metrics -- do they attribute this to the probing algorithm? Or to their metric better-evaluating unstable methods? It would be nice to isolate these sources of variation to truly evaluate the proposed framework.

- *Disentanglement vs. informativeness*: Figure 3 and Table 2 tell a consistent story: models trained with a regularization loss that encourages disentanglement do indeed achieve better disentanglement scores with the proposed metric. However, as demonstrated in [1, Fig 6, right], [2, Table 1b], and this paper (Figure 3c), disentanglement regularization often comes at the cost of reconstruction/informativeness. Why is it then that, in Table 2, informativeness is consistently better for models *with* the disentanglement regularization? This seems very counter-intuitive as the unregularized model should capture at least as much information about the underlying factors, and raises questions about the results obtained.
- *Predictor f*: What was used in the experiments -- a linear or nonlinear predictor? E.g. lasso[2], random forests[2], or gradient-boosted trees[3]? This is very important information and seems to be missing from the main paper.

**Additional comments/questions**: 
- Slot symmetry (Section 3.1, point 3): is this ever explicitly evaluated (i.e. separately to slot disentanglement)? The authors claim all 3 criteria can be evaluated by their metric -- do they mean implicitly (i.e. if slot symmetry is poor, so too will be their metric scores)?
- [4] also evaluate intra-object disentanglement (see Section 5.3) by first matching latent slots to GT objects using IoU (see Section 5.1). Your probing algorithm is more general and elegant, but it is worth mentioning this more manual approach to intra-object disentanglement in e.g. the related work section.
- “The variational loss” (Section 5.2) is quite cryptic -- I’d suggest something like “disentanglement regularization”
- Line 165: did the authors mean permutations over latent *slots* rather than latent *codes*?




[1] Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017). \beta-VAE: Learning basic visual concepts with a constrained variational framework. In *International Conference on Learning Representations*.

[2] Eastwood, C. and Williams, C. K. I. (2018). A framework for the quantitative evaluation of disentangled representations. In *International Conference on Learning Representations*.

[3] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2020). A sober look at the unsupervised learning of disentangled representations and their evaluation. *Journal of Machine Learning Research*, 21(209):1–62.

[4] Li, N., Eastwood, C. and Fisher, R. (2020). Learning object-centric representations of multi-object scenes from multiple views. In *Advances in Neural Information Processing Systems*.


summary_of_the_review: Interesting and novel ideas, but the often unclear writing, questionable claims/framing and weak experimental evaluation leave me unconvinced of their efficacy. I believe this paper is currently just below the acceptance threshold, but am eager to improve my score if the authors address the aforementioned concerns.
review_title: null

Meta-review: This paper presents a new metric for disentanglement of learned representations, extending a prominent framework (DCI) to support object-centric structured representations.

The reviewers agree on the importance of the question and find the metric a valuable contribution for addressing this problem. In the discussion, the reviewers identified some clarity issues that the authors have improved, leading to an overall much better writeup, as well as some deeper evaluation of learned matching agreements. The main remaining points that could be improved are
 - making the results more robust with thorough hyperparam tuning
 - connecting to other methods for inducing soft / probabilistic matchings, such as Sinkhorn or smooth&sparse optimal transport.

Please consider switching to the Times font as recommended by the ICLR style guide.
Decision: Accept (Poster)