Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents an empirical evaluation of a number of recent models for unsupervised object-based video modelling. Five different models are evaluated on three (partially novel) benchmarks, providing a unifying perspective on the relative performance of these models. Several common issues are identified and highlighted using challenge datasets: The reliance on color as a cue for object segmentation, occlusion, object size, and change in object appearance. The paper concludes with several ideas for alleviating these issues.

Strengths:
 1. The paper represents a much needed comparison of several related models which have previously not been evaluated on common benchmarks. Given the rapidly increasing number of competing models in this space, I believe analysis papers like this one serve an important role.
 2. The paper highlights important weaknesses of unsupervised object models, such as the overreliance on color or the difficulties with handling occlusion. While I believe that some of these weaknesses were already known to the people working with these methods, they have not always been formally documented in the respective publications. This paper rectifies this, and also provides guidance as to the relative vulnerability of the different methods.
 3. The methodology is convincing and thorough. Architecture and hyperparameter choices are clearly documented in the appendix, and the datasets have been published. 

Weaknesses:
1. As an analysis paper, this publication does not provide specific new technical contributions to the issues it is evaluating. 
2. The results are not entirely conclusive, in that there is no clear best model, and their relative quality varies with datasets and metrics. That said, some things can be very clearly observed, for instance the importance of object size for the performance of TBA.

Overall, the paper serves an important role in consolidating the ecosystem of unsupervised object representations. Given the increasing need for such analysis papers, and the competent execution, I recommend acceptance.
review_title: A timely and well-executed evaluation of existing methods

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper is actually very well written and tries to answer the question of how various unsupervised learning of object-centric representations to on controlled tasks (synthetic datasets). The positives:
1. Designs a benchmark of three datasets of varying complexity
2. Compare a single image model and four video models (total five) 
3. Defined clear metrics that are around precision, detection, segmentation, tracking 

The cons to the paper:
1. All of the datasets are synthetic and it would have been good to at least pick a real world dataset and confirm the conclusions stand
2. Since, the core goal of the paper is not novelty but a better understanding of various models I would have liked for the discussion to have some clear conclusions and better structure (use X in scenario Y, Model Z needs to be extended for scenarios Y etc.). I feel this section was short and mostly verbose without a clear conclusion
3. Instead of focusing on such a comprehensive set of things - 3 datasets, five models and multiple metrics it would have been better if authors did some minor extensions of the models and showcase novel directions. But, instead the paper is mostly an understand only work and i worry that the conclusions don't necessarily give clear future directions for other researchers to build on.

Given the cons and specifically on not a clear actionable suggestion on how to improve models and no analysis beyond synthetic datasets I am leaning towards a rating of below acceptance threshold.
review_title: The paper provides a framework to compare five models on four synthetic datasets for object detection/segmentation and tracking .

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a benchmark for the evaluation of unsupervised learning of object-centric representation. The benchmark consists of three datasets, multi-object tracking metrics and of the evaluation of four methods. The proposed dataset consists of three sets of video sequences, procedurally generated, which are either generated from slight variations of existing works (Sprites-MOT) or on the basis of existing datasets (dSpirites, Video Object Room). For evaluation, authors propose to use a slight variation of the protocol of the MOT challenge for evaluation (with the addition of a Mostly Detected measure which does not penalize ID switches). As part of the  paper, they also evaluate and discuss the performances of four object-centric representation models, one of them (Video MONet) being an extension of an existing approach, proposed as part of this paper, and the remaining being state of the art approaches for the task.

**Paper Strengths**
- Although I am not familiar with unsupervised learning of object-centric representation, I like the idea of proposing a common protocol for evaluation. Potentially, this can be useful for the community and can help to create a common ground for evaluation.
- The benchmark is comprehensive, in that it contains both data and evaluation measures. The adoption of MOT metrics also appears to be a reasonable choice.
- The comparison between different models is interesting, and authors have also added a custom designed novel method (Video MONet). They investigate a set of challenging scenarios and carry out out-of-distribution tests.
- The paper is a pleasure to read, and authors have also made a good effort in writing a clear and comprehensive supplemental.

**Paper Weaknesses**
My main concern about the paper is the lack of novelty. While I realize that the objective of the paper is to create a common evaluation background, I fail to see a sufficient level of quality and of novelty. In particular:
- the dataset associated with the benchmark are mostly based on existing works - they might be appropriate for evaluating this task, but the level of contribution is a bit limited;
- on the metrics, the only contribution is to suggest using MOT metrics, which are again pre-existing;
- the experimental and the insights it gives, again, can foster the community towards better model, but it's not a sufficient contribution for ICLR in my view.

Overall, I think the paper could be a nice contribution to the literature on unsupervised learning of object-centric representation, but it lacks sufficient contribution for ICLR, in my view. I would therefore suggest to reject the paper. 
review_title: A nice benchmark, but not enough

final_score: rating: 5: Marginally below acceptance threshold
review_content: Overall, this paper is interesting in setting up a benchmark for unsupervised object representations which is a very important problem in computer vision, reinforcement learning, etc. But the paper contains a very small amount of information to make use of the benchmark. Following are my comments:

1. The paper needs to clearly set the contributions. The video datasets are from self-made simulations or taken from other sources because the appendix cites many references for each part of the dataset. 

2. The appendices are linked in the paper but given in the supplementary. It is good to combine both in one paper.  The appendix doesn't require information of existing methods and the current version contains of appendix contain mostly these explanations. 

3. Apart from these, the paper is well written and useful in the community. But the empirical evaluation is not convincing. There could be many baseline approaches for this, for example, paper [1] and the methods it is comparing with. This is my major concern on the paper and if stated well, the recommendation can change. 

[1] SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition, ICLR 2020



review_title: The paper provides a benchmark for unsupervised object representations with 5 state-of-the-art methods and 3 simulation datasets. 

Meta-review: This paper received 4 reviews with mixed initial ratings: 7, 5, 4, 5. The main concerns of R1, R2 and R4, who gave unfavorable scores, included: lack of methodological novelty (analysis-only paper), absence of experiments on real data (3 synthetic-only benchmarks), missing baselines and an overall inconclusive discussion. At the same time R5 notes that the offered fair comparison between SOTA methods was indeed "much needed", and the paper can "serve an important role" in guiding future developments in the community. In response to that, the authors submitted a new revision and provided detailed answers to each of the reviews separately. R1, R2 and R4 did not participate in the discussion, and R5 stayed with the positive rating.
AC agrees with R5 that the provided analysis is insightful, and the effort put into organizing the research community around a single set of benchmarks and metrics is indeed valuable. However, given a simplistic nature of the proposed datasets and lack of other methodological contributions, the submission is not meeting the acceptance bar for ICLR. After discussion with PCs, the final recommendation is to reject.
Decision: Reject