Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposed to combine the idea of slot-attention and vector quantization to learn discrete object-centric representation of visual scenes. The proposed model utilizes slot-attention to decompose the image into a set of object-centric slots, and then transform each inferred slot into a few concatenated vectors by vector quantization from a learned codebook. Both set prediction and object discovery tasks are evaluated to show the effectiveness of the proposed model, while particular efforts are made to show the ability to discover disentangled subspace in the latent space partitioned by the discrete quantization.
strength_and_weaknesses: Strengths:
- The proposed work seems to be a reasonable extension of the slot-attention model, and the empirical results on set prediction show some extent of improvements over the baseline.

Weaknesses:
- The writing is generally not as clear as it could be. The most important part of the paper should be Sec. 2.3, however, it's unnecessarily hard to follow, and Figure 1 is not very helpful here. The notations of introduced variables are not introduced in a more natural way, for example, the $e_j^i$ term is not described in a clear way in its first appearance.
- In Sec 2.5, the authors propose to use beta-VAE style loss to encourage the latent space to be disentangled. However, there's no mention and discussion on the weighting effect, i.e. $\beta>1$, which plays a crucial role.
- In object discovery tasks, only qualitative results are provided, and no quantitative evaluation is included, which is not convincing enough, especially when only CLEVR dataset is considered.
clarity,_quality,_novelty_and_reproducibility: Please see above.
summary_of_the_review: This work proposes an approach for learning discrete object-centric representation by combining existing ideas, I believe more thorough evaluation and better clarity are needed in this reviewing cycle.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper combines Slot Attention with VQ-VAE (although in a stepwise training fashion) and shows results on CLEVR and on set prediction tasks.

strength_and_weaknesses: 1. The presentation of the model as 2 independent tasks and entirely different methods is not very helpful. I would have expected the model to be trained on image reconstruction only, and demonstrate that one can make use of the discrete representations to perform the set prediction post-hoc.
   1. Figure 1 shows the setup as one single model diagram, even though they are done entirely separately and in different training stages.
   2. Section 1 and 2 are not very clear in presenting this choice.
   3. The Set Prediction task is not presented well enough. What are these thresholds?
   4. Overall I found the paper hard to follow, even though the idea is rather simple.
2. Combining VQ-VAE with SlotAttention is a good idea, and I think some of the choices they made in how to do so makes sense, however this should be done end to end and with less assumptions to be really impactful.
   1. What happens when you learn everything together? In particular if this was made to work when trying to do reconstruction, this would be a valuable piece of research.
   2. The number of categorical distributions being fixed to the number of generative factors, and the number of categories to be equal to the number of values per factor is too much supervision. 
      1. What happens if you use more?
   3. The fact that continuous variables are handled independently and entirely differently from categorical ones is too much supervision.
      1. What happens if you just use VQ for everything? Obviously you would need to use many for this to make sense.
3. The proposed disentanglement metrics were confusing and I would have assumed that computing the discrete mutual information would have directly done the same?
   1. Can you comment on how these differ?
   2. Figure 3 and 4 were not very clear to me, and feel like they belong to the Appendix? A table could replace Figure 4 and be more informative.
4. Nits:
   1. The math in section 2.3 uses cross products instead of dot products. This should be changed.
   2. I could not see an Appendix with details of the architecture and training setup. There are not enough details about the model in the main text to reproduce this work.

clarity,_quality,_novelty_and_reproducibility: * The paper was not extremely clear, and several sections were quite hard to follow.
* As explained above, I do not think the results and the way the model is presented is of the standard expected by ICLR in this current draft.
* I have not seen a model combining SlotAttention with VQ-VAE yet, so the work presented appears novel to me AFAIK.
summary_of_the_review: Overall, I think this paper tries to do something interesting, but it makes several arbitrary and limiting choices, which seriously hinder the usefulness of the model, for example the decision to train SlotAttention first, then the VQs second. It also assumes quite a lot of knowledge about the task in various places, which reduces its use as a real unsupervised method. Finally, the current presentation is not as clear as it could be and I found it hard to follow. 

Hence in this current form I do not believe this work reaches the standard expected by ICLR.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This work proposes a version of Slot Attention with vector-quantized representations, focussing on object- as well as feature-level disentanglement. The paper also proposes a pair of techniques (DQCF-micro and DQCF-macro) to look at disentanglement when generative factors are encoded as vectors.
strength_and_weaknesses: Pros:
- The set prediction results show a marginal improvement over Slot Attention.
- Code is available in the supplementary materials.

Cons:
- The results are pretty basic on CLEVR alone. Only a single example image is used throughout the paper (for the object discovery and disentanglement results). There are no additional images in the Supplementary material.
- The proposed evaluation for disentanglement (DQCF-micro and DQCF-macro) is purely qualitative and involves inspecting histograms (for DQCF-mico, this could be over several latent spaces). Surely deviation from uniform could be quantified as a metric?
- The paper is not clear on several technical aspects (see questions below). It reads like an early draft at the moment.
clarity,_quality,_novelty_and_reproducibility: The writing reads like an early draft. Please see the following comments and suggestions:

1. Inline citations are weirdly done. Section 6, “Object discovery” is impossible to read because of recurring citations. You don’t need to cite a model more than once in the same in the same paragraph.
2. Section 2.3: 
    - Why not denote each latent space as $L_k$ if you’re using K to denote the number of latent spaces? 
    - You don’t seem to set the number of embeddings $n_{L_i}$ differently across the latent spaces. So why not drop the ${L_i}$ subscript and simply use $n$? This would also be consistent with your equations.
    - Matrix multiplication is not denoted using $\times$. It would be sufficient to say $S’ = MS$. Likewise for $sim^i$.
    - It is unusual to write $S^D = [e^i, …, e^K]$. Generally this would be written $S^D = [e^1, …, e^K]$.
3. Section 2.4 typo: “MPL” -> “MLP”?
4. Section 2.5: you introduce q(L_i | S’) for the first time here. Is it just a categorical distribution over n embeddings parameterized by ${y^i_1, …, y^i_n}$?
5. Section 5 typo: “qualitative” -> “quantitative”?
6. Please make sure all equations have a number.
summary_of_the_review: The paper is certainly not ready for publication. The results need fleshing out and the writing needs a few more iterations.

Here are some questions to help the paper in future iterations:
1. __End-to-end__? Figure 1 shows the “three stages” of the VQ-SA pipeline. Could you confirm whether the stages are run one after the other (freezing weights as you go), or if the pipeline is run end to end?
2. __Prior work__: This isn’t the first work that uses discrete representations with Slot Attention. How does this work relate to Singh et al. 2021 where they also use a discrete VAE?
3. __Traversals__: Figure 2: what exactly do you mean by manipulating a particular attribute in your model? Since each slot representation $S^D$ is a weighted sum of embeddings (concatenated across latent spaces), I don’t understand how you can manipulate the weighted sum. Do you replace the weights $y^i$ with a one-hot lookup?
4. __KL term__: I assume Figure 4 corresponds to the model trained “without KL-term” (shown in Figure 5). Could you also share DQCF-macro results for the model which achieves better disentanglement (“with KL-term”)? What is the effect of boosting the weight of the KL loss?
5. __Multiple latent spaces__: In Table 2, it appears that using a single latent space with 32 embeddings yields better set prediction performance than 4 different latent spaces. Why is that? Is disentanglement the only rationale of using multiple latent spaces?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to use a latent space to quantize object-centric representations learned with slot attention for better disentanglement. The authors adopted a similar idea of VQ-VAE and initialized learnable codebooks for each slot representation generated from slot attention to obtain the vector-quantized representation of slot representations.  With experiments, the authors show that their methods can outperform slot attention on set prediction task in CLEVR.
strength_and_weaknesses: [+] The idea of vector quantizing slot representations for better disentangling representations is new.

[+] The resulting model does show better performance compared with the vanilla slot attention model.

[-] The major concern of this paper lies in the justification of claims in this paper and experiments. The authors are motivated to perform vector quantization to better learn disentangled object-centric representations. However, they only tested the resulting model on the CLEVR dataset on the set-prediction task which shows the limited significance of the design. The disentangling mechanism (KL) is also not quantitatively evaluated in ablative studies. This makes the overall claims of the proposed VQ-SA not fully addressed and justified.

[-] The authors might want to elaborate more on disentangled representations as it is a critical factor in this paper and show their significance. The current sec.4 does not fully show the uniqueness of learning discrete spaces and does not make a direct comparison of disentanglement with ones that do not leverage a quantization module. The description of DQCF-micro and DQCF-macro is also a bit hard to follow in the text, especially given notations are not properly defined before the illustrations.
clarity,_quality,_novelty_and_reproducibility: The paper lacks clarity in several key illustrations of designs (e.g. definition of DFCQ) and might cause problems in understanding. The idea of quantizing slot representations is new however the current experimental results show the limited significance of the learned VQ-SA as a model for better disentanglement in object-centric learning. Codes are provided, however, with limited description on implementation details in the text.
summary_of_the_review: Given that the current experimental results can not fully justify the claims of the authors, I'm recommending rejection and suggest the authors design better tasks/settings for illustrating the effectiveness of the proposed VQ-SA, not only from a qualitative perspective on the analysis of latent spaces. The authors might also want to step further from CLEVR to more complex datasets (both real and synthetic) for evaluating the learned representations.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces Vector Quantized Slot Attention (VQ-SA), a method that applies vector quantization on the learned slots of Slot Attention [1]. VQ-SA is empirically evaluated on supervised set prediction and unsupervised object discovery on synthetic multi-object datasets such as CLEVR.

The idea of combining vector quantization with Slot Attention to address shortcomings of the module is interesting and the authors show some quantitative as well as qualitative improvements, but as all the reviewers point out, the paper in its current form has significant weaknesses: the claims of the paper are not sufficiently experimentally verified (only on very simple synthetic datasets with, for some claims, solely qualitative results), several technical aspects are unclear, and the writing/presentation could be improved.

Given the above limitations, which still hold after the author rebuttal, I recommend rejecting this paper in its current form.

[1] Locatello et al., Object-Centric Learning with Slot Attention (NeurIPS 2020)
justification_for_why_not_higher_score: The authors have improved the clarity of the paper in their revision, but the main concerns around experimental validation remain. Overall, even though the idea is interesting and warrants further exploration, the paper in its current form is not ready for publication.
Decision: Reject