Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces an approach for learning to perform high-quality long-term video prediction via an object-centric latent bottleneck. The approach hinges on two key ideas. The first is to use a pre-trained temporal object discovery method to extract a sequence of aligned slots. The second is to use an autoregressive Transformer to rollout future slots conditioned on the sequence of aligned past slots. This enables the model to combine spatial and (multi-step) temporal information to improve the long-term accuracy of future slot rollouts. Extensive experiments exploring design choices, video prediction quality, and object-centric reasoning capabilities validate the efficacy of the approach.
strength_and_weaknesses: Strengths
=====
- The paper makes progress towards answering the difficult question of how to design an effective unsupervised approach for long-term video prediction capable of dealing with complex object-centric dynamics.
- The dynamics model has an elegant and simple design as a basic autoregressive transformer architecture. It appears to be simple enough that it is likely to be useful as a starting point for future methods that tackle more advanced tasks. For example, the dynamics module is shown to be capable of integrating with both SAVi and STEVE.
- The experiments are thorough; the paper uses proper baselines and metrics, multiple relevant multi-object video environments, and validates the design choices with strong empirical results.
- The paper provides a “bonus” insight I found particularly intriguing --- that a strong decoder (VQ-VAE) is not sufficient for learning complex multi-object dynamics.

Weaknesses
=====
I found just a few minor weaknesses to point out:
- It was a bit odd that OCVT was discussed as the most relevant closest work, yet results for this baseline were not provided. If I understand correctly from the appendix related work section, it consistently underperformed G-SWM on all benchmarks? Even so, it would be good to include results for OCVT.
- The simplicity of the approach hinges on the fact that all objects are visible at time step 1 and no new objects appear. This is a fairly important topic to discuss in the paper as this is limiting in terms of applying this approach to real-world video. What would it take to support handling occlusion and appearing/disappearing objects within the burn-in frames?
- Another weakness of the approach is that it is not trained end-to-end with the object discovery module which makes the training process cumbersome, and does not allow the object discovery to take advantage of long-term motion cues.

clarity,_quality,_novelty_and_reproducibility: - I believe the combination of the two ideas of 1) extracting aligned slots from video with a pre-trained object discovery module and 2) treating the problem as a sequence modeling task produces a novel approach for the task of object-centric video prediction
- I think the reproducibility of this work overall is relatively low because of the amount of compute required (4 GPUs in less than 5 days). I would encourage the authors to release trained model weights when open-sourcing their code. However, the authors make a good effort to provide extensive experiment details in the appendix and the overall method is fairly simple + builds on top of existing methods (e.g., SAVi). 
- Overall the paper is very well written. Moving the limitations section from the appendix to the main text is important, as well as expanding it to discuss, e.g., handling occlusion, + object appearance/disappearance within the burn-in frames, and deterministic vs. stochastic future prediction.
- Adding a description for SAVi, STEVE, and Aloe in the main text would help the readability of the paper. Also, I was missing the definition of the AUCCESS metric in the main text.
summary_of_the_review: Overall, I believe this paper contributes valuable insights as well as a useful method for the considered problem. I believe it is ready for publication in its current state.

---
Update after rebuttal: Maintaining my initial positive stance on the paper and recommend acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a transformer-based dynamics model that can be trained on top of pre-trained slot representation of a video. The model first pre-trains SAVi/STEVE to obtain slots from video frames. It then trains an auto-regressive transformer to learn to predict future slots given the past slots.
strength_and_weaknesses: ### Pros

1. The model is novel as it makes use of slot-based representations of SAVi/STEVE and models complex long-term spatiotemporal dependencies using a transformer for predicting the next-step slots.
2. Shows good future predictions. Shows benefits on downstream tasks such as VQA and action planning. Also shows useful ablations.
3. The model is simple and effective and thus has the potential to become the go-to model for further exploration in this line.
4. Has a nice property that the future slot predictions preserve the original order of the slots.
5. An interesting observation is that even though the predicted frames of STEVE have low visual quality, the slot representations were still useful to learn future prediction at the latent level and do well on the VQA downstream task.
6. VQ-former is said to provide poor dynamics suggesting that object-centric representation can be key for good learning of dynamics.
7. Writing is quite clear.

### Weaknesses/Questions

1. I understand that the positional encodings are duplicated over all $N$ slots for a given time-step $t$. However, I am wondering how the positional encodings $P_t$ are set for a given $t$? Are these sinusoidal embeddings in terms of $t$? If so, then would the model generalize in predicting much longer time horizons than those seen in training? Also, are the episodes randomly cropped to make fixed-length clips before training, or are full episodes used directly during training? 
2. Currently, the fact that VQFormer is not performing well is only mentioned in the text and the evaluation metrics (SSIM/PSNR) were said to be not ideal by the authors. So, it would be good to show some videos of VQFormer not performing well in dynamics prediction relative to SlotFormer. This would better highlight that object-centric slots are indeed important. Also, showing some generated videos of SlotFormer for all the datasets can be helpful to see the visual quality of the generations.
3. How beneficial is it to perform explicit roll-outs for the downstream tasks? For instance, in VQA, what would be the performance if the downstream model receives only the burn-in frames without doing the explicit future roll-out? It may be good to show a comparison. A similar question may be asked about the PHYRE action-planning task i.e. what if the slots from burn-in frames are directly used to predict the task-success score? 

### Minor Comments/Questions

1. In the ablation, the effect of burn-in and the roll-out length may be shown as a line plot.
2. Have authors tested action-conditioned generation in SlotFormer and how was it implemented?
clarity,_quality,_novelty_and_reproducibility: - To my knowledge, the work is novel. The closest work is OCVT. However, Slotformer has important advantages over OCVT. OCVT builds on the SPACE family of object representations while SlotFormer builds on the slot-attention family of object representations — the former requiring heavy priors while the latter is more general and has also been shown to work well on visually complex scenes e.g. in SAVi/STEVE. Furthermore, OCVT seems to require explicit Hungarian alignment to compute the prediction loss while Slotformer naturally enjoys aligned slots from SAVi/STEVE.
- The paper is clear and high-quality.
- Authors also promise to release the code.
summary_of_the_review: I think the paper can be accepted in its current form. However, clarifying/answering some of the above questions would be good and I would be happy to raise the score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a core module to model the interaction of frames and predict the consequent states in an object-centric way. Mainly built on the SAVI [1], the proposed module operates on the objects of burn-in frames and generates future rollout autoregressively. In specific, it adopts layers of transformer block added with positional embedding for retaining permutation equivariance to model the object-dynamic representations. Thorough experiments demonstrate that the approach achieves competitive performance on not only video prediction tasks but also some reasoning tasks (action planning and visual question answering).

[1] Conditional Object-Centric Learning from Video. ICLR 2022.
strength_and_weaknesses: Strength:
+ The idea is neat and simple. It fully takes the advantage of the previous object-centric model (i.e., SAVI).
+ The experiment results and visualization strongly show the effectiveness of the method. For example, the emerging property of long-term modeling is well supported by the plot in Figures 2 and 4.
+ The paper writing is easy to follow. 

Weakness:
- The novelty of the architecture is limited. The solely new module in the paper on top of the SAVI is the Transformer blocks. Also, I am confused about why this proposed module excels in long-term modeling. The intuition here is not strong. Does the temporal-dependent position embedding work? Or, does the transformer architecture itself help the long-term modeling? If that is the case, what is the result without naive positional embedding (when the RNN-based model is adopted)? Maybe the authors would be better to highlight it more in the paper. 
- I am confused why would "error accumulation issue" benefit the model. Is there any related work to analyze the phenomenon (the error propagation is crucial in the sequential/autoregressive model)? I don't think it can be treated as common sense for the readers and requires a more reasonable explanation. 
clarity,_quality,_novelty_and_reproducibility: The paper is clear and the supplementary material provides detailed implementation details, which I believe reduces the difficulty of reproducibility. The paper is not novel in terms of the model design whereas it is novel in the configuration of the experiments. 
summary_of_the_review: Overall, the paper shows a promising application for slot attention in object-dynamic modeling. However, some intuition the paper tries to convey is not straightforward. For example, the long-term property. Which design of the model makes it excel in long-term prediction? Also, the technical novelty is limited in terms of the architecture. Thus, my initial recommendation is borderline reject.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a Transformer-based, object-centric autoregressive model for consistent future prediction. The method first learn to extract object-centric representations via pretraining. By using SAVi/STEVE, these slots extracted across multiple frames contain alignment information. Then, it trains a Transformer-based autoregressive model that takes the object slots as input. The experiment is comprehensive and demonstrate good performance compared to the previous methods. 

Strengths. First of all, the paper is very clear and well written. The experiment is very comprehensive. The experiment results are also strong. The idea of the proposed method is not very novel but the simple-and-well-working can be considered also a strong merit that can cover the weakness of the idea novelty. 

The main weakness is the novelty of the idea. But as I mentioned, I believe the paper still contains other important contribution to the community. The model may be trained end-to-end but I agree that it can be a future work or not that easy. 
justification_for_why_not_higher_score: The main idea of the proposed model is not very novel. 
justification_for_why_not_lower_score: The paper is very well written, the proposed model is simple (in a good sense) and works well. The experiments are very comprehensive.
Decision: Accept: poster