Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A self-supervised training over video is proposed for learning compositional scene representations. The paper explores the approach on the CATER and ACRE datasets (video versions of CLEVR), and show its benefits in the domain of visual reasoning and question answering over these videos.  
strength_and_weaknesses: **Strengths**: 
* **Self-supervised**: The approach is self-supervised and so doesn’t require annotations like alternative approaches for building compositional scene representations such as object detectors etc.
* **dynamics for object-centered representations**: The approach encourages temporal dynamics to be captured in the representations, by using a self-supervised approach over videos. This is an important and still underexplored signal in the domain of self-supervised compositional learning that could greatly help discover objects. 
* **Compositional Generalization**: The approach explores and show good results on compositional and systematic generalization, which gives good indication that the representations learned indeed disentangle the dimensions needed for the downstream reasoning tasks.
* **Extensive set of experiments on CLEVR-based data**: see quality point in the question below.

**Weaknesses**:
* **Experiments on synthetic datasets only**: The model is explored on CLEVR-based datasets only. While CLEVR-based data is definitely a great and suitable for exploring the presented approach, I would strongly encourage the authors to explore the approach either on new synthetic multi-object datasets with greater realism and diversity.

clarity,_quality,_novelty_and_reproducibility: **Quality**: The paper has concrete motivation (the benefits of self-supervised approaches compared to stronger object-attribute annotations and the potential of temporal dynamics to encourage object discovery) and explores an important domain. While it would be good to experiments on more datasets, the ones presented are extensive, and explore multiple axes of comspositional generalization, performance on downstream reasoning tasks, and sensitivity to variations and ablations. 

**Clarity**: The paper is well-written, the idea is clearly presented and it is accompanied with useful diagrams and visualizations. 

**Novelty**: 
The novelty of the idea is a bit limited, as there was a prior paper called ALOE (Attention over learned object embeddings enables complex visual reasoning) that, like the submitted paper, also used a self-supervised approach with a masked loss over frames and a transformer architecture, and it explores it over both on CATER and ACRE as done here as well as CLEVERER. 

There is a difference between the approaches where ALOE uses representations extracted from Monet while the approach here uses ViT, but from prior experience with working on the CLEVR dataset in particular, I think prior models shown with attention they can in a quite straightforward manner manage to pinpoint specific objects from the raw 2d image representation, when coupled with the right losses and overall model and settings, so the new component in the paper doesn’t substantially contribute on addressing an unsolved technical challenge. 

**ViT for object-centric on CLEVR vs. real-world**: This is especially straightforward given that the paper explores the model over the CLEVR dataset, since  they use ViT tokens to represent slots, and in CLEVR with choosing the right resolution, they could correspond approximately 1:1 to CLEVR objects – since they cover local convex regions, compared e.g. to more complicated objects or to non-object real-world regions such as sky etc.  

**ViT vs Monet/sparser approaches**: It’s unclear to me whether ViT would the ideal approach in discovering objects on more general data, or since considering each slot on a dense 2D map to be an “object” leads to too many false-positive objects, and may struggle to find good correspondences on other datasets. This is a disadvantage compared to Monet used in ALOE, and there are also other object-centered models that discover sparser more compact representations such as Slot Attention, SAVI (extending slots attention to video and explores more diverse video datasets and also a bit of real-world robotic data), and the GroupViT model – which works very well on diverse real-world images.  

**Claim correctness in introduction**: Furthermore, I find the conceptual distinction the paper tries to make in the introduction between ALOE which uses Monet vs this paper which uses ViT not compelling: the paper says the former is based on pre-processed extracted objects while the latter train the ViT together with the model, but Monet is an unsupervised approach too and one could think of training it together with the frame prediction explored in ALOE. 

**Results**: The quantitative results of the paper are in line with prior works but not surpassing them and in particular not surpassing ALOE. Combined together with the novelty issue compared to ALOE that’s a main weakness of the paper. 

summary_of_the_review: The paper shows a new approach in a growing relatively under-explored domain. Multiple experiments are presented on two datasets, the paper is easy to follow and has good visualizations, but the main issues are 1) the bit limited novelty of the idea in light of the ALOE paper that explored self-supervised transformer based approaches for both the dataset explored here and an additional one (see details above), and 2) The fact that the paper doesn’t explore other datasets beyond the CLEVR family. I therefore overall recommend weak rejection at this point and recommend exploring how to extend the approach for additional datasets, which may lead also to new modeling ideas that could strengthen the technical contribution of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an end-to-end implicit symbolic representation learning framework for visual reasoning tasks. It wisely adopts slot tokens for its bottleneck information properties, masked autoencoding objective, and transformers to learn implicit representations in a self-supervised way. The learned implicit representation can later be applied to solve specific visual reasoning tasks with proper head fine-tuning with target data. Such a learning framework has a very advantage: it does not require specific object abstraction (e.g., detection mask) and thus serves more general purposes. Results on two common benchmarks (CATER and ACRE) show consistent improvements achieved by the proposed framework. 
strength_and_weaknesses: The studied direction is important, and the proposed method took a further step toward a general solution to visual reasoning. Figure 1 clearly demonstrates the difference and advantages between the proposed method and the previous approaches: the proposed framework can perform visual reasoning with neural networks and non-predefined implicit tokens. All the used components: slot token, MAE objective, and transformers (encoder, temporal) are reasonable. Overall the paper is easy to follow.

Extensive experiments are conducted in CATER and ACRE datasets and show the proposed framework consistently outperforms the previous approaches. Ablation studies are conducted thoroughly around masking ratio, total frame numbers, context frame numbers, slot token numbers, slot pool layer, and slot pool methods.

Many analyses and explanations in the current manuscript are intuitive and without supporting empirical evidence. For example, in "Number of Slot Tokens" section, experiments found 1 works the best in the CATER benchmark, and the author explained why: "the model need only maintain an accurate prediction of where the snitch actually or potentially is". If we add golden snitches up to 2/3/4 (if possible, or in other sims), will the best performance achieve by 2/3/4 implicit slot accordingly? or the best performance is still achieved by 1 slot.

What will the performance change if we set the slot token number to 100? will the proposed framework lose the "representational bottleneck" properties and lead to drastic performance drops? 

In the transfer learning setting, the multi-class classification formulation for the goal of both CATER and ACRE tasks encodes strong human priors. The tested visual reasoning tasks are actually solved with specific designs for specific tasks in the end, while the studied implicit representations are uniform, and thus, the contribution did not weak a lot.
clarity,_quality,_novelty_and_reproducibility: Both the clarity and quality are good. The proposed framework may bring some fresh air to the community. The reproducibility is upon the code and pre-trained models released. 
summary_of_the_review: In general, the reviewer believes the proposed approach takes a step towards visual reasoning with deep learning, while the analysis is intuitive and probably lacks some empirical analysis for strong support.

--------------
After discussing with AC and all other reviewers, we generally believe the current manuscript lacks some thorough analysis of the proposed framework, especially for 1 slot token works pretty well in many scenarios. I encourage the author to take all the points raised in the review in their revision and also keep improving the performance in various benchmarks. Reviewer ejDU lists the key points we discussed in our meeting, and hope them can help contribute to a better revision. 

I appreciate the efforts made by the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for compositional visual reasoning in videos, based on recent advances in self-supervised pretraining. The method first trains a spacial-temporal transformer that reconstructs the video frames under the masked autoencoder paradigm, then performs reasoning via transfer learning. The authors claim and demonstrate via extensive experimental study on the CATER and ACRE datasets, that a generalizable compact representation is learnt, superior to other approaches based on object-centric representation. Ablation studies are conducted.
strength_and_weaknesses: Strength:

1. This paper proposes a powerful end-to-end network for compositional visual reasoning, and achieved nice results on two datasets that were regarded as (and designed to be) challenging for end-to-end systems. 

2. The paper also demonstrates self-supervised pre-training can lead to useful representations for visual reasoning, which sheds light to further research.

3. The paper is well written and backed by rich experimental results.

Weakness

1. The paper claims that the model learns "implicit symbolic representation" without a clear definition and elaboration. What is the difference between "symbolic representation" vs. a regular latent representation learned by other self-supervised model? Moreover, there isn't sufficient study on the representation itself other than transfer learning results to back these claims. How does the same architecture perform without the representation?

2. The experimental results have not established, decisive evidence that end-to-end method is superior to object-centric representations. Specifically, under same supervision, the model does not significantly outperform ALOE and ALOE++. The paper also does not provide evaluation on the CLEVRER dataset, a video reasoning benchmark where object ALOE performs nicely.

3. Minor: The data flow in figures goes upwards instead of downwards, which might cause unnecessary confusions to readers.
clarity,_quality,_novelty_and_reproducibility: The paper is well written with sufficient discussion on related works, even though the presentation of figures can be improved. The authors claim that code will be released upon acceptance of the manuscript.
summary_of_the_review: This paper is an interesting paper that applies state-of-the-art self-supervised pretraining to compositional visual reasoning. Despite the reasonable results achieved on two challenging datasets, it is unfortunate that the paper does not provide a deeper look into the learned representation in order to back the claimed novelty. There is definitely a lot of room for improvement here. My score is borderline reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the framework "Implicit Symbolic Concept Learner" (IS-CL), a transformer-based architecture that is capable of reasoning about videos. The framework follows a pretraining-transfer-learning pipeline. That is, the model is first pretrained on a collection of unlabelled videos (by the MAE objective), and then finetuned on the target task (e.g., predicting the location of a certain object).
strength_and_weaknesses: The paper presentation is clear and well-connected with existing works such as NS-CL and ALOE (Figure 1). There are a few places where the authors connect their work with Slot Attention, which has greatly helped me to understand the connections and differences between the proposed framework and others. The proposed framework is also relatively straightforward, with clear motivation. Experimental results have shown the success of the proposed framework. Furthermore, the ablation studies seem very adequate.

Talking about weaknesses, I will start with a few conceptual ones.

First, the idea of "concept learning," following the notations from Mao et al, primarily refers to the correspondence between linguistic units (words, phrases) and visual representations (e.g., red, cubic, left-of, etc.). This seems to be different from the definition of "concepts" or "implicit symbolic concept" in the paper, which, in my understanding, refers to "implicitly defined objects."

Second, when we talk about "visual concepts,", they are usually more "abstract" than "pixel reconstruction." However, since the overall training paradigm (pretraining time) is to reconstruct the pixels, I don't think there is evidence that the model is capable of discovering "symbolic concepts," e.g., colors, shapes, etc., from the pertaining. With all that, I am wondering if the title/model should be better phrased as "implicit object learning" or similar phrases?

Here're a few technical questions.

First, I am wondering why the authors have not tried visual question-answering benchmarks such as CLEVRER, especially given that based on the ablation study (Table 2a), CATER does not require an "implicit symbolic" representation (slot =1 then basically it's just a per-image representation). It seems that the framework can be easily extended to that, as in ALOE. And I think extending the framework to that can significantly improve the completeness of the paper.

Second, in Table 4, it seems that the model still underperforms several baselines. Although authors may argue that those better-performing algorithms are "object-centric" I think it is still completely reasonable to compare IS-CL with ALOE, because ALOE also does not use any object-detection labels during training. Again, I think adding a new benchmark that showcases the advantage of the proposed method will be ideal.

Third, more of a clarification question, regarding the comparison between VideoMAE and IS-CL. Is the only difference that you are using a "single-token" embedding for each frame whereas VideoMAE uses all visual tokens? Because these two models have very similar training objectives and similar architectures.

Fourth, is there any way that we can visualize the learned "implicit slot tokens?" For example, can you visualize the attention maps?

Finally, you mentioned that you have changed the slot-attention-style encoding with a customized transformer-style encoding. Do you have any ablation studies on that?

Clairification questions:

1. Page 5:  allowing the slots to attend not only to raw visual inputs, but also to the encoded patch-level representation. Can you be more specific about this?
clarity,_quality,_novelty_and_reproducibility: The paper is well presented and has novelty over existing works.
summary_of_the_review: Overall I think this paper is a good contribution to the community. However, there are still a few issues that can be addressed by the authors before publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a method for learning compositional scene representations from unlabeled video using an approach inspired by masked autoencoders (MAE). The resulting representations can then be fine-tuned for downstream visual reasoning tasks. The paper presents extensive experiments on the CATER and ACRE datasets, and is generally well-written. Some of the initial issues raised by reviewers were resolved by the author responses, including results on more datasets (authors added results on CLEVERER), several additional ablations (suggested by Reviewer YXhg), more experiments on what the learned representations capture (experiments added to supplementary), and a more precise meaning of “implicit symbolic representations.”

After the author's responses, the reviewers gave borderline ratings to the paper (with ejDU leaning slightly toward acceptance, Cwxs and LKo8 slightly favoring rejection, and only YXhg in favor of acceptance). In order to come to a decision on this paper, the AC and reviewers met to discuss it.

From the discussion, it became clear that there are several key issues with the paper that had not been fully addressed in the author’s responses. First is the performance relative to ALOE. Both the proposed method (IS-CL) and ALOE receive the same supervision: unlabeled videos and input/output pairs for downstream tasks, and no ground-truth object locations, program execution traces, or other intermediate supervision. However IS-CL demonstrates mixed performance relative to ALOE, outperforming ALOE on ACRE but underperforming it on CATER and CLEVERER. Second is the distinction between the “implicit symbols” that IS-CL claims to learn, and the object-centric representation used by ALOE; the authors argue that these are distinct, but were not able to convincingly demonstrate that IS-CL actually learns non-object-centric representations; indeed in the discussion with Reviewer ejDU it became clear that IS-CL primarily learns to attend to objects. Third, the datasets used could have been more convincing; in their initial review, Reviewer Cwxs suggested showing results on more realistic, non-CLEVR-like datasets, and felt that this would make the results much more convincing; fpr example, comparing to the experimental setups of e.g. SAVi, SAVi++, or GroupVIT would have made the paper much stronger.

While Reviewers ejDU and YXhg gave positive ratings to the paper following the author feedback, in the discussion both agreed that the above were critical weaknesses, and neither was willing to argue that the paper should be accepted. In the end all reviewers agreed that the paper is not ready for publication in its current form.

justification_for_why_not_higher_score: The paper shares some technical similarities with prior work ALOE, does not uniformly improve upon ALOE in quantitative performance, and does not convincingly argue that the "implicit symbolic" representations learned by the method are different from object-centric representations used in prior work.
Decision: Reject