Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new unsupervised transformer architecture for learning disentangled visual concepts. Main contributions include the novel transformer-based architecture to represent images as a set of tokens, each reflecting a visual concept; and a concept disentangling loss, which asks the model to predict mutated concept tokens. Experiments are conducted on Shapes3D, MPI3D, Cars3D, which demonstrate better disentangling and scene decomposition capabilities of the propose model compared with other existing works. When using CLIP encoder as image tokenizer, the framework also enables language-aligned disentanglement. 
strengths_and_weaknesses: S1: The propose architecture seems principle, and as the paper shows can be used in combination with different model architectures.

S2: The concept disentangling loss is beneficial to further learn disentangled visual representation, without additional manual annotations.

S3: Quantitative and qualitative results are decent in demonstrating the quality of the obtained disentangled representation on the commonly used datasets.

S4: The presentation is smooth and easy to follow. 

W1: Some parts of the writing may be under-supported by evidence. See Questions.

W2: Disentanglement results are mostly on simulated datasets. It is not completely clear how the proposed method would transfer to real-world datasets, such as KITTI. The results on CelebA in the supplementary is not always promising, with attributes tangled in some cases.

W3: The design may seem not very well motivated. Contributions are mostly on the architecture side, with no strong intuition why or whether such a design leads to superior performance in general.
questions: 1. From Table 2, how L_dis will affect the result if AE / VQ-VAE are pre-trained? 
2. From Table 2, are the sensitivity results (batch sizes, token numbers) dependent on the datasets? For example, I would expect more complex scenarios are more vulnerable with respect to the number of tokens.
3. Has transformer-based architecture used previously for learning disentangled visual representation? What are the main points to highlight in VCT in comparison to them?
limitations: The key limitation is the lack of convincing results on real-world scenarios, considering the conclusion are most experimental rather theoretical. In addition, it is not always clear why such an architecture leads to better disentangled representation in general, due to the lacking in a good motivation.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an unsupervised framework called Visual Concepts Tokenization (VCT) to extract visual concepts from concrete pixels for tackling disentangled representation learning and scene decomposition. VCT adopts a cross-attention-based tokenizer to abstract visual information into concept tokens. In addition, it also utilizes a concept disentangling loss based on the visual concept token manipulation to ensure the exclusivity of different tokens. The authors conduct extensive experiments to demonstrate the superiority of VCT against SOTA methods on multiple popular disentanglement benchmarks. Furthermore, the authors perform many ablation studies to verify the design choices of cross-attention only concept tokenizer and concept disentangling loss. Even trained without any dedicated design for scene decomposition tasks, VCT can still well decompose a scene into object-level visual representations.
strengths_and_weaknesses: Strengths:
1. The paper is well-written and easy to follow. The proposed VCT approach performs well on disentangled representation learning and scene decomposition tasks. It achieves SOTA performances on three datasets and beats all four types of baseline methods (VAE-based, GAN-based, pre-trained GAN-based, concept-based) by a relatively large margin.
2. As shown in the quantitative results in Figure 3, 4, and 5, VCT performs pretty well on decomposing, reconstructing, and manipulating the learned visual concepts, which demonstrates great model interpretability.
3. Even not trained with explicit object mask annotations, VCT can still abstract high-level visual representations and achieve scene decomposition by representing each object as a concept token.
4. Extensive ablation studies are conducted to confirm the design choices of image tokenizer, concept disentangling loss, concept tokenizer, and concept detokenizer. Another big advantage of VCT is that it is robust to hyper-parameter variations.

Weaknesses:
1. Except for the face dataset CeleBA, the paper conducts all experiments on synthesized datasets. CeleBA is also with limited scene variations. It is doubtful whether the proposed VCT can generalize well to real-world images such as those in ImageNet or MSCOCO.
2. From the qualitative results presented in the Appendix, for predefined $M$ visual concept tokens, only a proportion of them have explicit meanings after training. Moreover, it is unclear in the paper how to decide which learned concept token corresponds to which factor of variation.
3. The default setting used the image tokenizer and detokenizer from a pre-trained VQ-VAE. However, VQ-VAE might already learn meaningful quantized representations that correspond to visual concepts via the vector-quantization layer.
4. Section 4.3 and Section 4.4 are not well presented. The descriptions of the decomposition and recombination processes are hard to understand. Meanwhile, the caption of Figure 6 seems not to match its content, especially for Figure 6 (b). I suggest the authors further improve the presentation for scene decomposition experiments.
questions: 1. Can VCT generalize well to real-world image datasets?
2. How to decide which learned concept token corresponds to which factor of variation?
limitations: No obvious negative societal impact was found.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new transformer-based method for learning a set of prototypes that describe different visual concepts. It leverages the cross-attention/self-attention mechanism to model the interactions between visual inputs and trainable prototypes, and presents a concept disentangling loss to encourage the learning of their correlations. Experimental results on several 3D datasets demonstrate the usefulness of the proposed method.
strengths_and_weaknesses: This paper has the following strengths:
+ It is an interesting idea to leverage the attention mechanism for decomposing visual inputs into unique concepts. 

+ The proposed method is able to learn visual decomposition in an unsupervised manner.

+ The paper provides extensive ablation study and analysis with different settings, and the concept disentanglement based on language is also interesting.

However, there are also weaknesses that can not be overlooked:
- The paper only carries out experiments on synthetic datasets, and the image size is typically very small (i.e., 64x64). It is unclear how well the method could generalize to broader scenarios, for example, prior studies (e.g., [20, 37]) typically consider naturalistic images with more complicated visual scenes. 


- A major claim of the method is to learn disentangled prototypes that correspond to unique concepts. The paper mainly demonstrates the effectiveness of disentanglement based on information-based metrics, and it is difficult to validate if the disentangled representations are truly associated with independent visual concepts.

- Besides the targeted concept prototypes, the method also learns a set of dataset-specific prototypes Y. This is relatively counterintuitive, considering that the visual concepts are supposed to represent a diverse range of visual scenes. What do these dataset prototypes learn? And how important are they to the overall method?

- In addition to removing/adding concepts or swapping concepts between images, is it possible to utilize the proposed method for image editing? I am aware of the qualitative results shown in the supplementary materials, however, they are either limited to specific domains (synthetic data) or not very impressive (causing distortion or no significant effects on facial images).

questions: (1) Please consider experiments with data from broader domains, and also validate the effectiveness of the method via applications such as image editing. 

(2) Please justify the unique concepts learned from the proposed method, and the techniques for attributing prototypes to concepts.

(3) Please provide the details about the mechanism and impacts of the dataset-specific prototypes.

limitations: I did not find significant negative societal impacts in this work.
review_title: null

Meta-review: metareview: This paper proposes an unsupervised transformer based framework called Visual Concepts Tokenization (VCT) to extract visual concepts from concrete pixels for tackling disentangled representation learning and scene decomposition. Experiments on several popular datasets validated the effectiveness of VCT on the tasks of disentangled representation learning and scene decomposition in which VCT outperforms the previous works significantly. Reviewers generally agree the proposed VCT framework is novel and the empirical results are promising (though the results on real-world images seem not as strong as the synthesized data and clearly there is a room to improve on the real-world image data). Authors did a great rebuttal job in making extensive efforts to make revision and give comprehensive answers in response to the reviewers's concerns. Overall, this is a solid paper that has enough contributions to the disentangled representation learning topic and thus is recommended to accept. 
confidence: Less certain
Decision: Accept