Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper provides a multi-stage solution to unsupervised, frame-wise segmentation in videos.
The common fate heuristic is used to provide initial object detections and segmentations.
Then VAE based model is used to refine the initial results.
A new simulated dataset is proposed.

Results show that the proposed method successfully segments out moving objects and is highly scalable.
main_review: Strengths:
1. I appreciate the effort of building a new dataset.
2. Compared with other methods, the scalability of the proposed method is indeed strong.
3. I am convinced that the direct application of "the common fate heuristic" is novel and is of great potential.
4. While most related work can only generate masks for occluded (partial) objects, the proposed method can generate masks for full objects occluded or not.
5. The ability to sample novel scenes is impressive.

Weakness:
I apologize if some of my concerns are already detailed in the paper and I fail to pick them out.
1. Discussion of related works.
    While the author did a comprehensive literature review, I am not convinced by some of the claims.
    1): I am not sure why "the spatial mixture model does not match the true, underlying scene generation process".
         For example, SPACE employs a depth prediction to predict the relative depth between objects.
         Then the depth information is integrated into the mask value.
         SPACE effectively transforms a 2D segmentation task into 2.5D.
    2): "End-to-End object-centric approaches are difficult to train."
         Without further explanation, I am not sure what "difficult to train" means under the context.
    3): "Full scene models trained in an end-to-end fashion are not shown to scale to a more realistic dataset yet."
         As demonstrated by the paper SCALOR: GENERATIVE WORLD MODELS WITH SCALABLE OBJECT REPRESENTATIONS (which is already cited by the author), such method can be applied to challenging cases.

2. Dataset
The dataset proposed is indeed interesting and challenging.
My only question is that is it true that all foreground objects are moving all the time and the backgrounds do not have any moving parts?

3. Baseline
It is surprising that no baseline model is evaluated on the same dataset.
I understand that there are not many works targeting exactly the same setup, but works like SCALOR or SPACE can also be used as a baseline for numerical results.
After all, being able to segment without "the common fate" heuristic and a rule based object proposal stage seems like an advanrages to me.
To show that the loss function proposed in this work is better, the author can modify SPACE to take bounding boxes proposed by the rule based motion segmentation pipeline. This should be an easy modification.
If under this modification the performance of SPACE is better than before but still behind this paper, then the method propsed in this work is truely great.
Similar modification can be done to SCALOR as well I think.

summary_of_the_review: The results provided in the paper is impressive.
But the authors argue multiple times that the method proposed in this work is better than end-to-end generative model without enough justifiation.
Thus I cannot say that this paper is above accept threshold at current stage.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces an object-centric generative model for visual scenes. The model decouples the problem into three tasks: 1) modelling the  2D appearance and shape of individual objects with a variational auto encoder; 2) same thing for background; 3) sampling the position, size and appearance of individual objects (i.e. scene composition) conditioned on the background. To my understanding, these three components are trained independently.

Contributions:
1) Decoupling these three tasks allows for a more interpretable representation compared to existing end-to-end methods, which allows for better interactions with users (e.g. change the number of objects or their position) 
2) This decoupling potentially makes the third task easier, because it only needs to learn relationships between positions and size given a latent representation (as opposed to learning everything jointly).
3) Training the tasks independently requires a way to obtain "ground-truth" object segmentations - the paper achieves this automatically from videos using motion segmentation with good results
main_review: I found the paper quite useful and interesting. I find it useful because the decomposition allows for better ablative analysis of the model performance compared to previous end-to-end approaches, and because it opens up additional practical applications, such as given user direct and interpretable control on the output generated by the model (e.g. manually input number of objects or an individual object position). I find the results interesting, particularly the thorough ablative analysis: comparing training on ground truth segmentation vs training on motion segmentation.

The paper has some weaknesses, e..g. some claims not thoroughly backed up, lack of clarity on some details. I list them here in detail, from more important to less important: 
1) The paper strongly couples two concepts: a) the choice of using a multi-stage learning approach rather than end-to-end; and b) the use of motion segmentation to enable training this model. In practice, it seems to me that these two things are not really that coupled, but rather motion segmentation is just a way to automatically generate the training data that the model needs, because unlike end-to-end method, this submission needs explicit object and segmentation masks (if I understand correctly). One thing I would like the authors to highlight more  is that, due to this, a limitation of this method is that it can be trained only on video data and not on static images, unless some form of segmentation ground truth is available or there is a way to automatically generate it -  this is mentioned in the conclusions, but I encourage making it clear earlier in the paper
2) It is unclear if the parameters of the scene models learnt entirely independently from the foreground and background model, i.e. by learning the distribution of the latent vectors produced by the VAEs conditioned on the background? Or is there some gradient propagating between these steps
3) von kugelgen et al (2020) also provide a controllable way to change the way the model generates the scenes (e.g. number and type of objects). Can you highlight the main differences? Is it mostly that in this submissions there is also control on scale and position?
4) Results on the fishbowl dataset look quite impressive, less on the car dataset (I suggest adding some visuals from this dataset and of visual comparisons to existing methods in the main body of the paper). It would be interesting to see this method run on other datasets like the Atari dataset or the 3D room dataset from the SPACE paper, to further test generalisation of the multi-stage approach in other domains - do ground truth segmentations exist to test there? 
5) Understanding the performance of this methods on generating new scenes compared to existing work is quite challenging because it is done mostly qualitatively, although I recognise that is a challenge of the domain and existing methods also seems to suffer from this
summary_of_the_review: While I do not see a significant amount of novel technical contributions, the proposed paradigm of breaking down scene generative models into multiple stages of learning that allows encoding prior structure and human-interpretable elements is valuable. It can help generate theoretical insights, and the ablative analysis provided here is a promising first step. It also has the potential to be practically useful, by allowing users more fine-grained controls on the parameters of the generated scenes.

There are some claims with minor issues, and some lack of clarity in some sections (see weaknesses in the review section above). All in all, I think the paper is above the acceptance threshold, and addressing these weaknesses would make me lean towards a clearer accept
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes an object-centric generative model. It consists of motion segmentation, object model, background model, and scene model. The object model is trained to reconstruct an object as if it is not occluded. The authors have introduced a new dataset, called Fishbowl, which provides inmodal and amodal segmentation masks of objects. 
main_review: Object-centric generative models have gained relatively less attentions compared to the normal generation models. The authors have generated a new dataset for this research area. However, I have many concerns that make me hesitant to accept this paper.

FIrst of all, the dataset is far from the realistic setting. It is related to missing literature search. There are many datasets for amodal segmentation. For example, https://openaccess.thecvf.com/content_CVPR_2019/papers/Hu_SAIL-VOS_Semantic_Amodal_Instance_Level_Video_Object_Segmentation_-_A_CVPR_2019_paper.pdf <- this paper basically proposes a more realistic dataset, which provides the same information (amodal masks). I assume the authors have not looked into this literature. There are many other amodal datasets, including KINS and COCO-A.

My second point is that the proposed model has not been evaluated well. I have gone through the paper as well as the supplementary, but there was no appropriate comparison to the previous works. It's hard to believe all the conventional algorithms fail at generating object-centric images.
summary_of_the_review: Because of the above two major weaknesses, I'm skeptical of accepting this paper. I am open to change my rating depends on the rebuttal. For now, my initial rating is reject.
review_title: null

Meta-review: This paper studies the challenging problem of object-centric generation of visual scenes. While the paper has some novel ideas that make it interesting, its (quantitative and qualitative) comparison with existing methods is currently premature to allow drawing conclusions with sufficient evidence.

Instead of claiming that existing models cannot do well for the more realistic datasets mentioned by reviewer dAqW, it would be more convincing to conduct a comprehensive experimental study by comparing the proposed method with existing methods on a range of datasets, from simple ones to more realistic ones. The synthetic Fishbowl dataset introduced in this paper can be one of them.

Moreover, the clarity of the paper could be improved to make it appeal better to the readers.

All three reviewers engaged actively in discussions (both including and not including the authors). Although one reviewer recommends 6 (weak accept), the reviewer also shares some of the concerns of the other reviewers. As it stands, the paper is not ready for acceptance. If the comments and suggestions are incorporated to revise the paper, it will have potential to be a good paper for future submission.
Decision: Reject