Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a method for learning object-centric representations from single 2D images without supervision. Unlike previous work that learns by reconstructing the current frame, the proposed method learns by predicting object movement in future frames. Specifically, the model is trained from triplets of 2D images that are taken consecutively from a 3D scene by a moving camera. The camera intrinsics and movement are known to the model. At test time, the model can be applied to single images for object segmentation and localization. The experiments are done on a custom dataset featuring complex object texture and background. The model shows comparable or slightly better segmentation performance than SlotAttention.
strength_and_weaknesses: - Strengths
    - The problem setting is interesting and close to how humans perceive objects.
    - The custom dataset seems challenging and can be valuable to the community.
- Weaknesses
    - The empirical results are a bit discouraging. The proposed method (with access to camera intrinsics and multiple frames during training) is similar to or only slightly better than SlotAttention (with access to only static images) in terms of segmentation performance. The benefit of predicting object movement is not well demonstrated by these results.
    - It seems the paper aims to tackle (1) complex object texture and (2) object movement in 3D. However, the paper fails to compare with or discuss the related work in these two aspects.
        - Some recent work (e.g., [GENESIS-v2](https://proceedings.neurips.cc/paper/2021/hash/43ec517d68b6edd3015b3edc9a11367b-Abstract.html) and [SLATE](https://openreview.net/forum?id=h0OYV0We3oh)) tackles complex texture from single 2D images without predicting object movement. The authors did try to train GENESIS-v2 on their dataset, but failed to get good results. Perhaps the authors could try to make some version of the dataset that is similar to those used in GENESIS-v2, and show that predicting object movement leads to better segmentation than using static images alone.
        - Quite a few papers tackle object-centric 3D representations (e.g., [O3V](https://proceedings.neurips.cc/paper/2020/hash/20125fd9b2d43e340a35fb0278da235d-Abstract.html), [ROOTS](https://www.jmlr.org/papers/v22/20-1176.html), [Crawford & Pineau](https://github.com/oolworkshop/oolworkshop.github.io/blob/master/pdf/OOL_19.pdf), [ObSuRF](https://arxiv.org/abs/2104.01148), etc). In principle, these methods could be applied to the dataset in this paper. In fact, the representations they learn are more comprehensive than this paper (because they capture 3D appearance) while given the same input (2D image + camera parameters).
        - While not strictly necessary, the paper could also discuss object-centric representation learning from 2D videos (e.g., [SQAIR](https://arxiv.org/abs/1806.01794), [SCALOR](https://arxiv.org/abs/1910.02384), etc). They treat object movement in 2D, but the representation learning also benefits from predicting future frames.
    - The presentation of the paper can be improved. The paper describes the method at very detailed level, without explaining the high-level intuition/motivation behind the key design choices, making it hard for the reader to follow. In particular, I have the following questions:
        - Why use discrete yaw angles instead of continuous values?
        - Why does the imagination network take original image as input rather than the inferred object latents (mentioned in Section 2.3)?
        - Why use a Von Mises prior distribution for angular velocity?
        - When doing the soft-matching, why do you only use a subset of the latent instead of the full latent?
        - Background velocity is set to zero (mentioned in Section 2.4.1). Do you assume static background?
clarity,_quality,_novelty_and_reproducibility: - Clarity: can be improved.
- Quality: results do not seem significant, and the paper fails to discuss a few related works.
- Novelty: the problem setting seems interesting but not entirely novel. The paper fails to highlight its novelty compared to related work.
- Reproducibility: the paper provides a lot of technical detail, but could be more organized.
summary_of_the_review: I recommend reject for the paper in its current form, due to potentially limited significance, insufficient discussion of related work, and clarity issues.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper learns to discover objects by video prediction. The method called OPPOLE extracts each frame into object states(inferred object 3D position and 1D pose), and background. The method learns the decomposition in an unsupervised method that renders the object representation into image frames and compares it with GT RGB images and inferred depth. Several additional regularization terms are proposed to avoid the degenerate solutions. The method is compared with per-frame unsupervised object discovery baselines including slot-attention and monet on their proposed synthetic room dataset. 

strength_and_weaknesses: Strength:
+ The model use motion cues (wo explicit supervision) to supervise the object discovery tasks and the object notion emerge via bottle-neck object representation. 
+ The only supervision is RGB images and camera pose and it is interesting to see depth emerging from unsupervised training. 
+ the paper is written clearly and is easy to follow.

Weakness:
- The biggest claimed contribution is that they combine predictive learning with explicit modeling 3D motion. However, the idea of explicitly modeling 3D is not extremely novel e.g. (Henderson & Lamp NeurIPS 2020, Anciukevicius et al NeurIPS 2022). I do not see a fundamental difference between them.  Although [A] Henderson & Lampert, NeurIPS 20 cannot generalize to a single frame, I wonder what happens if both models are inferred from videos – it may favor [A] as OPPLE is a by-frame prediction, yet it helps to understand what is new in OPPLE. 
Another important basleines to claim the contribution is to model motions in 2D instead of 3D, but still, use predictive learning to train. 
- More ablations are needed to understand how each component contributes to the tasks. Many designs of the method appear rather arbitrary to me, e.g. imagination network chooses not to include geometry even though the warping network does, the effect of regularization terms in training. 
- The comparisons are shown on the synthetic dataset generated by their own, which looks like a variant of ROOM datasets with more diverse textures and a fixed number of objects. However, I wonder why ROOM/CLEVER benchmark cannot be evaluated. Although the baselines will improve bc of less diverse textures, according to the authors, the proposed method should also work on that dataset. I encourage authors to report results on more widely adopted datasets in addition to their own datasets. This would calibrate their method better.
- The proposed method does not handle various number of objects like the baselines do.

clarity,_quality,_novelty_and_reproducibility: The method is clearly presented and written but the key contribution and takeaways are a bit unclear. 
Quality: the method shows some improvement over baselines on their proposed dataset and it is understandable. see above
Novelty: Some module designs are sort of novel e.g. jointly infer depth, matching process between two object sets.  But unfortunately, I have a hard time summarizing "what I've learned from the paper".  details see above. 
The author promise to release code for reproducibility. 
summary_of_the_review: Overall, the method is clearly presented and written but the key contribution and takeaways are a bit unclear. The empirical results are relatively weak in the sense that 1) the results are conducted on their own datasets and 2) more ablations need to be done in order to understand each components.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Note I have reviewed an earlier version of this paper for last year's conference. I have copied some of the notes from my earlier review but I have read the new version carefully and made sure that I only copied parts that hasn't changed in the new version. Overall, the earlier paper and the current version overlap quite significantly (especially in terms of experimental results).

This paper presents an unsupervised object-centric scene representation technique that can decompose a scene into multiple objects (and segment the scene) and infer their 3D locations and pose. The overall setup is very similar to earlier models like MONet but this model works on sequences of images, more precisely on 3 consecutive images. It uses the first two images to infer the 3D position and pose of objects and combining this with known camera motion tries to predict the last (third) image. Then it uses an optical flow based method to warp the image at time t using the predicted object location/pose/depth to predict (some of) the pixels in image at time t+1. The model is trained to minimize the difference between this prediction and the true next image.

In more detail, the object extraction network outputs the location and pose of each object. And a separate depth perception network outputs the depth for each pixel in the image. The location and pose of objects are used to estimate the velocity of each object (e.g., by subtracting the position at t-1 from position at t. note this requires matching each object at time t-1 to object in time t, which they do using a soft-matching approach). These along with the depth information are then used to warp the image at t to predict pixels in image at time t+1. This is possible only for a subset of the pixels so for the rest, they use a separate "imagination" network that takes in object information and predicts the color/depth and object masks at t+1. The predictions from warping and imagination network are then combined to form the final predicted color and depth images.

To train the model, they require images and camera motion, and use a combination of losses: reconstruction loss on predicted and ground truth image, self-supervised losses on object location, pose, and depth.

strength_and_weaknesses: Overall, as I have said last time, I think this is an interesting paper. The optical flow based warping to predict some subset of the pixels in the next timestep is a nice idea and is perhaps the main novelty of this paper. (However, some recent work has used a similar idea [1].)

I think the writing has improved significantly in the newer version; I found the txt much easier to follow (of course this probably has something to do with the fact that I'm reading paper for a second time.)

I think the main concern with the paper, again like last time, is limited experimental evaluation. The model is evaluated only on a single, rather simple dataset (that was generated by the authors). I would at least have expected a comparison on some more or less standard datasets like CLEVR etc., at least on some of the datasets that other competing models were evaluated on. I understand that the dataset generated by authors have textures, which other techniques find difficult to handle, and their technique can deal with better. But I'd like to see that the proposed technique still works in the "simpler" setting that competing methods can handle.

Other notes

- There are better models than MONet and Slot Attention in this literature like PARTS, SLATE etc. It'd be nice to compare to these as well.
- The authors mention O3V as a similar technique. Why don't you compare against it as well?
- Did the authors think about applying this technique to longer sequences? Longer sequences may make it more challenging to apply the technique but will enable model to learn since longer sequences should have more information.
- There are no std deviations in table 1, which makes it difficult to how whether the performance differences are meaningful.
- Slot attn 128 does worse than smaller slot attn. Do the authors why this is the case? I'd have expected to work at least as well. However, in terms of IoU, it does better. Again, having std dev. here would help to understand what is going on.
- Typos:
  - pg 4., top: modificaiton
  - pg 5., bottom: We -> we
  - pg 8., top: ground-true -> ground-truth

[1] The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos, https://arxiv.org/abs/2111.06394
clarity,_quality,_novelty_and_reproducibility: I think the paper is relatively clear and well-written. Many in-line equations make it difficult to follow the text sometimes but overall it is not a huge problem I think. In terms of novelty, given that other work has used a similar warping idea to predict images ([1]), the novelty is limited but still I think the overall technique is original.

summary_of_the_review: Overall, I think the paper is interesting, but unfortunately the empirical evaluation is very limited and this makes it difficult to evaluate the full merit of the proposed approach.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper:   The paper considers video-based object-centric learning tasks.
  The proposed model achieves object-centric learning by predicting per-object motion and future frame observation.
  Experiments show that the proposed method outperforms a collection of single-frame baselines.

strength_and_weaknesses:   - Strength
    - This work exploits the heuristic that parts belonging to the same object should move in a consistent way.
    - The proposed model can predict depth.
    - Compared with baselines that cannot efficiently use motion information, the proposed method can detect objects with non-trivial 
texture.
    - The polar angle prediction results clearly demonstrate the potential of this method,
  - Weakness
    - While the qualitative results shown in figure 2 is much better than various baselines, the huge gap is not reflected in the quantitative results.
    - The FG-ARI and IoU are low in number. This score roughly indicates that the model is completely not unusable even on the simply simulated dataset.
    - If I am not mistaken, slot attention for video (SAVi) supports fully unsupervised training. I am not sure why SAVi is not included as a baseline.
    - Do all objects in the training set move all the time?
    - Even though the traditional dataset (especially [1]) is not challenging enough to show off the benefits of the proposed model, it can still provide valuable insights to the research community.
    - Since the loss comes from the image reconstruction loss, how do you deal with textureless regions (which can probably explain the hole in object centers)?


[1] Object-Centric Representation Learning with Generative Spatial-Temporal Factorization

clarity,_quality,_novelty_and_reproducibility:   - The paper is generally easy to follow.
  - The scale of the y-axis of figure 3 A B C should be aligned for easier comparison.
summary_of_the_review:   The method adopted by the paper is solid and has potential.
  However, I believe the performance can be improved.
  This work is also related to warping-based depth estimation literature [2][3] (on real-world datasets), which may provide hints on how to improve performance. 
   Priors can be used to regularize the behavior.
   For example:

   - Encourage points to remain static (to better segment objects from the ground).
   - Encourage near regions to have similar motion (to reduce over-segmentation within each object).
   - Bidirectional wrapping
   - Refine the mask across multiple transitions

[2] The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth

[3] Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The key idea in the paper is to disentangle objects from videos by proposing a temporally structured reconstruction loss that leverages known camera extrinsics and predicted flow/depth. The inductive bias of spatio temporal smoothness is used to predict next frame given past two frames. Experiments are evaluated on synthetic datasets. 

Although the problem is quite interesting and unsolved in general, the paper needs a lot more work before it is ready for publication. There was consensus amongst reviewers that the experimental validation is limited. During the rebuttal process there were major revisions to some of the empirical numbers (.e.g IOU scores) without much context and explanation -- it needs further review iterations and checking to ensure that all experiments are sound and valid. 


justification_for_why_not_higher_score: The experiments are lacking, there were too many choices and changes made that are left unexplained and the type of datasets investigated are limited. Papers need major revision and review before it is ready for publication.
Decision: Reject