Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This work attempts to learn improved slot-based object representations and video segmentations with the help of the SLATE/Image GPT [2] decoder. The model is evaluated on several datasets included ones generated by the authors. 
strengths_and_weaknesses: The results have dubious merit, the claims are poorly supported, and the writing needs a lot of improvement. There is little innovation in experimental methodology. On the plus side, the paper conducts a wide evaluation over several datasets, but the central claim that STEVE works on naturalistic images is not convincing at all. It is also unclear whether STEVE is any different than SAVi with a SLATE/Image GPT [2] decoder.
questions: **Claims**:
1. Page 2, line 87: “The emergence of object-centric slots is strongly dependent on what decoder is used.” Could you please provide supporting evidence for this.
2. Page 3, line 92. “The reconstruction objective is taken to be the squared error between the input and the reconstructed image.” This is not always the case and I would strongly caution against this over-simplification. Typically you would use the mixture likelihood $p(x) = \sum_n m_n \odot p(x; \hat{x_n}, \sigma)$ rather than an MSE wrt to $\hat{x} = \sum_n m_n \odot \hat{x_n}$, which is a lossy point estimate that defeats the purpose of a mixture model.
3. Page 9, line 209: “Therefore, in all the comparisons, the decoding masks gave an advantage to the baselines but not to our model.” This claim is far from obvious. If SAVi’s decoder masks are better than its own attention masks, that is due to either the smoothing effect of the decoder or representation quality/information bottleneck in SAVi specifically. A model with a different loss/architecture/set of hyperparameters may behave entirely differently. For example: if we shrank the decoder size (number of layers/parameters) to make it underfit, the decoded segmentation performance will likely undershoot the attention mask performance. Hence, whether STEVE is at an advantage or disadvantage cannot be ascertained by looking at SAVi's performance alone.

**Methodology**:
1. Why not use an existing natural image/video dataset? It is impossible to judge the quality of Traffic and Aquarium especially given the segmentations were derived by manual annotation. 
2. OP3 is not the best choice of baseline. Its performance is not meaningfully different from SAVi’s (except on MOVi-E where it suffers from not being able to handle camera movement). Its inclusion does not shed any light on STEVE’s performance. Could you consider at least an improved version of OP3 (e.g. PARTS [1])?
3. Figure 3: “We separated the visualization of the foreground segments to make the result easier to interpret by applying a threshold on the segment area.” This makes the use of the ARI-FG metric somewhat disingenuous. It is an acceptable measure so long as the discarded background pixels correspond to a single or separately modeled (e.g. in MONet or SPACE respectively) component. Can you reliably extract the background components from STEVE? Is your threshold fine-tuned per dataset? Do you use the same threshold across the baseline models?

**Results**:
1. Figures 3 and 4: In both CATERTex and MOVi-Tex, the model tends to split the textured background into multiple components. But it tends to produce a solid background on both Traffic and Aquarium data. Could you explain this difference in behavior, despite the obviously textured background in the natural datasets?
2. movi-d-002.gif: the falling object takes at least 4 different slots over time (green, blue, yellow, pink). Surely this is far from desirable? There’s no camera motion in this video. For human eyes, the moving object is trivially trackable.
3. catertex-001.gif: there’s a single slot (pink) for multiple object shadows, even when the objects have different dynamics. If the model took its cues from object dynamics (rather than colors), surely you would expect each object’s shadow to be associated with that object’s slot?

**Writing**:
1. There’s a lot of repetition between the abstract and introduction. Moreover, sentences like the following (Page 2, line 59) are superfluous in a technical submission to a research conference: “From these experiments, we discover important new knowledge to share with the community.”
3. Page 3, line 124: could you please clarify the dimensionality of each $z_{t, l}$? Are they one-hot vectors indexing into the discrete codebook? There’s other model details which would be useful to include in the main paper.
4. Figure 1: what are the error bars computed on?
5. Could you please cite [2] which is clearly the inspiration for the decoder architecture? It is not sufficient to cite SLATE alone.

**References**:

[1]  Zoran, D., Kabra, R., Lerchner, A., & Rezende, D. J. (2021). PARTS: Unsupervised segmentation with slots, attention and independence maximization. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ (pp. 10439-10447).

[2] Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D. &amp; Sutskever, I.. (2020). Generative Pretraining From Pixels. _Proceedings of the 37th International Conference on Machine Learning<_, in _Proceedings of Machine Learning Research_ 119:1691-1703 Available from https://proceedings.mlr.press/v119/chen20s.html.

limitations: There was no attempt to explain the limitations of the model.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Learning the compositional structure (e.g. scene graph) in dynamic visual scenes without labels (e.g. object masks) has proven challenging.  Slot-based models leveraging motion cues have recently made progress in learning to represent, segment, and track objects without direct supervision. However, these methods only work on synthetic data and fails on complex real-world multi-object videos.  This paper proposes STEVE, an unsupervised model for object-centric learning in videos. The paper clams that it made a significant progress by demonstrating its effectiveness on various complex and naturalistic videos unprecedented. This is enabled by a simple architecture that uses a transformer-based image decoder conditioned on slots and the learning objective is simply to reconstruct the observation.  Experiment results on several videos show improvements compared to the previous state-of-the-art, SAVi, SLATE and OP3.
strengths_and_weaknesses: Strengths

The paper proposes a a simple architecture that uses a transformer-based image decoder conditioned on slots and the learning objective is simply to reconstruct the observation. The transformer-based image decoder uses the one proposed in prior work, SLATE (Singh et al., 2022).

Weaknesses

Despite the claims on its effectiveness on complex and naturalistic videos, the datasets seem to be very limiting.  6 of them are procedurally generated datasets, CATER (Girdhar & Ramanan, 2020), CATERTex, MOVi-Solid, MOVi-Tex, MOVi-D, and MOVi-E (Greff et al., 2022). 2 natural datasets are only collected for this paper: Traffic and Aquarium. The paper does not provide details on the complexity and diversity of these datasets. 

The paper seems to lack novelty as the design seems to be a straightforward combination of (1) a CNN-based image encoder, (2) a recurrent slot encoder that updates slots temporally with recurrent neural networks (RNNs), and (3) the slot-transformer decoder of SLATE.
questions: The paper uses discrete VAE encoder to discretize each frame as a sequence of discrete tokens. The authors may want to try Vision-Transformer-based VQGAN (ViT-VQGAN) which has found successes in Parti (Scaling Autoregressive Models for Content-Rich Text-to-Image Generation, https://arxiv.org/abs/2206.10789).

The paper should provide more details on the two natural datasets. Furthermore, it should consider evaluating on existing benchmarks, e.g. Waymo Open Dataset.

The paper should provide more insights and characterize on when STEVE works and when it fails. Identify failure cases is important for future work.

limitations: The paper does not address its limitations. A section clearly articulating the technical limitations is important.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new method for unsupervised object discovery and tracking in videos called STEVE. This model combines an encoding backbone, a recurrent slot encoder and a transformer-based decoder to achieve impressive object discovery results on complex, textured multi-object datasets. Additionally, the model provides useful insights into its predecessor, SLATE, which worked on static images.

strengths_and_weaknesses: # Strengths

S.1) The writing style is great, and the paper is a pleasure to read. It is easy to follow and contains a lot of useful insights. In particular, I enjoyed the summarizing statements at the end of many paragraphs in the experiments section.

S.2) The proposed method is comparatively simple, yet very effective. It achieves impressive object discovery performance, especially on textured multi-object datasets.

S.3) The paper provides extensive and insightful experiments across 8 datasets, 5 of which are introduced by the paper itself. 


# Weaknesses

W.1) The description of the model in l.119-128 was somewhat unclear to me. Is the dVAE an entirely separate model from the slot-encoder, transformer-decoder architecture? How exactly do the dVAE and transformer decoder interact, and why? Additionally, since this is heavily based on the SLATE model, it would be nice to explicitly mention the architectural differences between the two models.

# Others
The references need some clean-up: there are several papers that appear twice in the list.
questions: Q.1) From Figure 1, it becomes apparent that the model improves in performance when a couple of initial frames have been processed. Does this effect still hold when objects move very fast in between frames? Is there some kind of maximum speed of objects between frames that the model can handle?

Q.2) SLATE's big selling point was to be able to generate new multi-object images. Since STEVE is using a very similar architecture, would it be possible to generate multi-object videos with this model as well?

limitations: Limitations are adequately addressed in the paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: -- Post rebuttal update ---
The rebuttal addresses my concerns with new experiments and a new limitations section in the manuscript. I continue to support the acceptance of this paper. Rating: 8.

-- End of post rebuttal update---

This paper proposes a model, called STEVE, for object centric learning for video data. STEVE is a combination of a recurrent slot encoder (like SAVi) and the SLATE slot-transformer decoder. The combination works surprisingly well on relatively complex synthetic data and even in some real world video datasets. STEVE demonstrates better object centric decomposition when compared against baselines on the FG-ARI metric. It also generalizes OOD to unseen number of objects and unseen materials.
strengths_and_weaknesses: ### Strengths

- The proposed method is simply yet very effectively. STEVE does not use extra annotations in the first frame, nor does it need optical flow or other signal in its loss function. Purely using dVAE token reconstruction as the loss function (indireclty just pixel reconstruction), STEVE is able to decompose textured videos, handle static objects (MOVi-D dataset), and moving camera (MOVi-E dataset).

- The method has been tested on a wide variety of datasets. This gives more weight to the claims. The comparisons show that the SLATE decoder is significantly better than a spatial broadcast decoder in both per frame and video metrics. The paper thus also provides insights about the SLATE decoder. SLATE itself is very powerful for object centric learning.

- The analysis covers obvious concerns such as using encoder attention maps as opposed to decoder attention maps, and relevant ablations  such as dVAE granularity are included.

### Weaknesses
* Minor issues:
  - Missing legend in figure 3(a), please note how many objects were used in the in-distribution and OOD settings.

  - If possible include STEVE in figure 5(a). What's the performance of STEVE relative to STEVE-Diagnostic?

  - I think you mean "convolution transpose" not "Deconvolution" in page 9. Please describe the exact architecture of this convolution transpose network. The spatial broadcast decoder also does some convolution transpose operations. How is this decoder different?

  - An explicit big-O analysis would be great in the "computational requirements" paragraph.

* A model figure is missing. I have read SAVi paper so I understand this method but people new to STEVE would likely benefit from a nice figure.

* FG-ARI ignores background pixels. This does not penalize leaking of object segments into the background. Relying only on this metric is dangerous. Is it possible to report readout based metrics in these datasets? For instance the slot attention paper reported property prediction for CLEVR.

* It is still not clear why the transformer decoder, which mixes information from all the slots in the decoding step unlike the mixture decoder which is applied to each slot independently, still leads to object decompositions? Does one need to control transformer capacity to make this work? What is the most sensitive hyper-parameter here?
questions: No major limitations and questions. The paper is a solid submission. I have discussed questions and concerns in the previous section itself.
limitations: Societal impact has been discussed.

Limitations can be elaborated further. For instance, the memory complexity of SLATE has been discussed in the paper; but the time complexity of sampling an image has not been discussed. Since transformer decoders need auto-regressive token-by-token generation they tend to be very slow during inference (not a problem during training). Maybe this becomes a big issue when running this at high resolution (512x512 or 1080p)?
review_title: null

Meta-review: metareview: The paper proposes a method for unsupervised learning of objects from videos. In particular, the proposed approach combines two existing ideas - a recurrent slot-based architecture (like SAVi) and an autoregressive image decoder (like SLATE). The methods is thoroughly evaluated and shown to outperform the relevant baselines.

After considering the authors' feedback and extensive discussions, the reviewers' opinions of the paper are still mixed: two strongly positive, one neutral leaning positive, and one strongly negative. The key strengths and weaknesses that were pointed out are as follows:
Strengths:
1. Simple and efficient model
2. Convincing experimental results on 8 datasets, including 2 real-world
3. Insightful analysis and ablation studies

Weaknesses:
1. Lack of novelty
2. No experiments on more complex real-world datasets
3. FG-ARI is not a perfect evaluation metric
4. Lack of insight from the experiments

Note that some of the mentioned weaknesses are in conflict with strengths - for instance S1 with W1, S2 with W2, S2/3 with W3/4. Taking all of the above into account, I believe the paper presents a simple yet efficient combination of two existing methods and evaluated it thoroughly, in line with (and actually somewhat above) what is commonly done for unsupervised object learning papers - leading to good performance and interesting insights. I believe the simplicity of the method is valuable and should not be critiqued as lack of innovation, and the provided experiments are already on more complex datasets than commonly used in the field. 

Therefore at this point I recommend acceptance, but encourage the authors to take the reviewers' comments to heart (whenever possible) and adjust the paper accordingly.
confidence: Certain
Decision: Accept