Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper learns the object-centric representation for videos by extending the previous static slot attention framework with two new considerations, 1. optical flow for temporal modeling and 2. using simple objects' location cues for better segmentation or tracking. Experiments are conducted on CATER, MOVi and MOVi++. It show SAVi has better performance than its baselines especially when using the objects' cues during training.
main_review: 1. The weakly-supervised setting seems to be new and has not been used in the previous literacture, which adopts abstract hints of the first frame as input and achieves better performance.

2. The idea to use optical flow as supervision signal seems to be new and has not been done before.

3. The writing overall is easy to follow.
summary_of_the_review: The reviewer has some concerns on the novelty of the paper.

1. The idea of using optical flow to help weakly-supervised or unsuperised object learning seems to be straight forward. Previous work like PSGNet[A] also adopts motions and depth as supervision signal and compare their performance with MoNet. It will be necessary to compare and disucss the PSGNet in the related work.

2. The paper only compares with  Slot attention or MoNet in unsupervised object segmentation. Object-centric representation for video learning and reasoning has been widely studied in previous frameworks like [A, B, C, D]. Note that previous work like ALOE [B] and VRDP [C] has also been using Slot attention or MoNet for unsupervised video proposal segmentation. It will be interesting to replace the Slot attention or MoNet with the proposed segmentation model and see their performance on the CLEVRER[E] dataset. 

[A]. Bear D M, Fan C, Mrowca D, et al. Learning physical graph representations from visual scenes[J]. arXiv preprint arXiv:2006.12373, 2020.

[B]. Attention over learned object embeddings enables complex visual reasoning. David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, Matt Botvinick, Arxiv, 2021.

[C]. Chen Z, Mao J, Wu J, et al. Grounding physical concepts of objects and events through dynamic visual reasoning[J]. arXiv 2021.

[D]. Ding M, Chen Z, Du T, et al. Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language[J]. arXiv 2021.

[E]. A]. Yi K, Gan C, Li Y, et al. Clevrer: Collision events for video representation and reasoning[J], ICLR 2020.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an object-centric method for video data, inspired from the slot-attention architecture. To enhance the learning (and to allow the users to specify the kind of entities that they are interested in), the slots are initialized from the first frame either a) random; b) with learnable initialisation; c) as the center of the bounding boxes; d) as bounding boxes or e) as segmentation maps.  The method uses a slot-attention mechanism to correct the slots from each frame, with query consisting in the current prediction of the slots, and keys and values consisting in some features extracted from the frame + GRU on each node, for temporal aggregation.  Then they apply a self-attention layer to propagate the information between slots. Finally, the model is optimise to predict either the optical flow or the rgb content.

The paper presents several ablations: using different types of conditioning, different levels of supervision (with or without optical flow or using estimated unsup optical flow instead of the real one); or by varying the amount of noise they inject in the initial positions of the slots (in the conditioned case).

The method is tested on CATER, MOVi and MOVi++ to quantify the quality of video decomposition and they also experiment with different types of generalization scenarios: various number of frames, new objects, new background, etc.
main_review: **Pros:**

- Adapting Slot Attention to produce coherent object-centric representations in video instead of static images is interesting and can lead to further development in the video-processing field. Moreover the idea is simple and sound.
- I enjoy the experiments showing the transition of results/empirical findings from simple to more complicated scenarios and also from synthetic to more realistic ones. The detailed ablations (built incrementally) offer useful insight for what the current object-centric video models require in order to perform well on more challenging scenarios.
- The experiments where the method is conditioned on a subset of objects (Appendix) set by the user are really nice and open a broad range of applications for the proposed method
- Overall the paper is well written and easy to follow

**Cons:**

- When measuring FG-ARI for the frame-level methods (Slot Att and MONet in Table1) how is the matching between the consecutives frames computed? I agree with the authors that the low score could be due to the lack of temporal consistency, but I think that a basic matching algorithm such as Hungarian matching + pairwise similarity to reorder the slots in a more appropriate way could represent a better baseline.
- It would be nice to see the results mentioned in the last paragraph of the Section 4.2 (Unsupervised settings on MOVi and MOVi++) in a table.
- In the paper, it is mentioned that the comparison with original CRW or T-VOS is not entirely fair since they used different backbone and operate on higher-resolution frames. I agree with that. However, why did the authors decide to retrain those methods instead of using the better backbone/setup for their method? I agree that the results should improve in that situation, but it’s not necessarily a sure fact.
summary_of_the_review: Overall, I enjoy the idea presented in the paper and I consider the empirical part interesting enough to recommend the acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a sequential extension of Slot Attention to tackle the problem of unsupervised / weakly supervised multi-object segmentation and tracking in video data. The method demonstrates successful segmentation and tracking for synthetic video data on unsupervised object representation learning.

main_review: Strengths:
1) Overall, this paper is well written, and the technical details are easy to follow. 
2) The main idea of learning object representations and physical dynamics from videos is interesting.
3) I found the anecdotal evidence for segmenting and tracking corresponding parts of objects very interesting, opening the door for more hierarchical concepts of objects using self/semi-supervised approaches.

Weaknesses:

**Contributions.** [1] already proposed a Slot Attention model based on optical flow for segmenting a single object. Although the author’s method supports multi-object environments while [1] is not, this paper still did not evaluate the proposed approach on real-world data (unlike [1]), which is a concern to me. 
	
Few works have already shown that using learnable query vectors instead of Gaussian-initialized slots helps the slots to learn on a unique embedding. [1] shows that “learnable queries play a similar role as soft clustering, i.e., assigning each pixel to one of the motion groups.” [2] (and others) shows that 2D positional encoding and box query embeddings are essential for initializing the queries for other tasks. To summarize, these works share the same concept that initializing the slots with location or motion embeddings could play a significant role in different downstream tasks.

[1] Self-supervised video object segmentation by motion grouping, ICCV 2021.

[2] TubeR: Tube-Transformer for Action Detection.


**Real-world Video Data.** As far as I see it, multi-object segmentation and tracking are essential for real-life datasets, and I am concerned that this approach could be relevant only for synthetic datasets. The main issue with the Slot attention model is the inability to capture natural texture/background and/or camera movement, and thus I believe this work will not be able to generalize in the real world.


**Technical Novelty.** The proposed approach is heavily based on the Slot Attention [2] model. I could not find an apparent and exciting technical novelty that could be interesting for other domains or different tasks. I believe the extension for the video domain could reach the bar of the conference quality, but still, I expect the authors to bring something new to the table. 

Can the authors highlight 2-3 points of technical or architectural modeling that are unique and different from their perspective from [1] or [2]?

[1] Self-supervised video object segmentation by motion grouping, ICCV 2021.

[2] Object-centric learning with slot attention, NeurIPS 2020.


**Relation to Prior Work.** There are some object-centric approaches that use object-centric representations for video understanding and might be worth considering citing them, such as:

[*] Compositional Video Synthesis with Action Graphs, ICML 2021.

[**] Spatio-Temporal Action Graph Network, ICCVW2019.






summary_of_the_review: My main concern is that I cannot see how the proposed approach can generalize to new domains and tasks. Overall, I like this paper and its contribution, but I think the authors should clarify their contributions and explain how the video slot attention could be leveraged in other domains and tasks. Otherwise, it feels like a perfect model for the CATER/ CLEVRER/ MOVi/ MOVi++ datasets, but the full potential is not entirely clear. 

I am open to the authors' feedback and other reviewers' opinions.


After Rebuttal
------------------


After reading the authors' feedback and other reviewers' opinions, I would like to thank the authors for their rebuttal.

The rebuttal addresses most of my concerns. I am leaning towards acceptance of the paper since it maintains the high bar of the conference quality. I vote for 6.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the problem of learning object-centric representations from videos. Different from existing methods that mostly model this problem under a pure unsupervised setting by reconstructing video frames, this work introduces two improvements. The first is offering weak "hints" (can be seen as additional inputs) during training, which can avoid falling into a trivial solution like previous unsupervised approaches always did. The "hints" here can be in form of pixel-wise masks, bounding boxes, or even as simple as centers of mass.  The second improvement regards the supervision signals. Instead of reconstructing raw pixels, they propose to predict optical flows. In easy scenes the optical flows are actually similar to pixel-wise masks: values inside an object are prone to be consistent. Therefore the learning is eased, and then making it possible to train on more realistic datasets. Extensive experiments are conducted to validate the effectiveness of these two improvements. 
main_review: I like this paper overall, and believe that the two improvements would bring new knowledge to the object-centric learning community.

Strengths
1. Previous object-centric learning methods are mostly unsupervised. The visual results they generate on simple datasets like CATER are quite promising, but in fact, they often fail in outputting meaningful object segmentations. For instance, an object can be represented by two slots while we only want a single one. I think this can hardly be well-addressed under the pure unsupervised setting. To this end, this work shows giving a simple initialization of objects of interest leading to less ambiguous representations of object slots, and I think this is a promising direction for object-centric learning.
2. Using optical flows as supervision makes it possible to perform object-centric learning on more realistic datasets. In such a dataset, the texture and color of objects are more diverse and the previous RGB reconstruction becomes an inappropriate pre-text task: We do not need to remember every detail of the objects' appearance in order to track them. Instead, predicting optical flows is an easier and more beneficial pre-text task.
3. I really like the experimental setups and results. The extensive experiments successfully convince me about the effectiveness of the two improvements proposed.

Weakness:
1. I think the biggest limitation is that using optical flow as supervision cannot handle static objects. However, this is clearly discussed in the limitation section, and I agree that addressing this problem is out of the scope of this paper.

Minor concerns:
1. Section 2,  "Inspired by ... ordinary differential equations", missing references.
2. In the legend of Figure 1, the text "Gated Recurrent Units" is not aligned.
3. Section 2, the Encoder paragraph, the notation is not clear. What is N and D_{enc}?
4. Section 2, the Slot Initialization paragraph, "we set the conditional input to a fixed value indicating padding": what does "indicating padding" mean here?



summary_of_the_review: This paper introduces two improvements upon existing object-centric learning methods: offering additional object-level "hints" as inputs and using motion cues (optical flows) as pre-text supervision. The idea is good and I believe will be beneficial to the object-centric learning community. The writing is good, easy to follow. Experimental results are sufficient to support their claims.
review_title: null

Meta-review: This work proposes a new framework that can learn the object-centric representation for video. The authors did a good job during rebuttal and turned one slightly negative reviewer into positive ones. The final scores are 6,6,8,8. AC agrees that this work is very interesting and deserves to be published on ICLR. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are also encouraged to make other necessary changes.
Decision: Accept (Poster)