Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper keys in on something that is widely suspected, but has not been rigorously investigated in the unsupervised object segmentation / discovery literature: the fact that nearly all methods do well on "simple" synthetic datasets but not on complex, real-world datasets that are the subject of most supervised object segmentation work.

In particular, this work makes two contributions:

1. It tests several of the better and more popular unsupervised object discovery methods on a range of datasets, spanning simple/synthetic to complex/real-world, and (maybe for the first time) actually quantifies the performance on the harder datasets using standard metrics. (For some reason, a lot of the unsupervised object discovery literature uses the non-standard foreground-ARI metric, which the authors here correctly note will make most models look far better quantitatively than they are qualitatively.) 

In this experiment, the authors find -- unsurprisingly -- that the models they test fail catastrophically on real-world data.

2. The paper creates new datasets in which the real-world datasets are "ablated" to alter object- and scene-level factors, which the authors hypothesize could explain the difference in model performance on synthetic vs real-world data. The authors then test this by training/testing the models on the new "ablated" datasets. When some or all of the factors that distinguish the real-world data are ablated, the unsupervised models can actually perform quite well.
strengths_and_weaknesses: Strengths:

1. Actually quantifying the unsupervised model performance on harder data was necessary, even if the results are unsurprising. This is an important contribution to the literature as it makes explicit something that many people "knew about," but could not point to concretely.

2. The dataset ablation experiments are very creative and the results are interesting. I was surprised that relatively simple factors could explain the gap in performance between synthetic and real datasets, although looking at the "ablated" images in Figure 9 it makes more sense: the fully-ablated datasets really do look much more like the synthetic datasets in terms of object and scene complexity and distribution.

3. The ideas and experiments are well-motivated and clearly explained.

Weaknesses:

1. It's hard to draw conclusions about what the *models* are missing because most of them have very similar design principles. The authors don't comment much on which models seem to handle which dataset factors better than others. 

2. The authors suggest that models based on different principles/inductive biases might be needed to handle the complexity of real-world data, especially ones based on learning from object motion. There is some recent work in this direction (e.g. https://arxiv.org/abs/2205.08515 and https://arxiv.org/abs/2110.06562 and https://proceedings.neurips.cc/paper/2020/file/4324e8d0d37b110ee1a4f1633ac52df5-Paper.pdf). This may have been too recent for the authors to know about at the time of writing (and to test) but it's important to point out that there are ideas for dealing with these issues and some progress.

3. There is no comparison to supervised methods with the ablated (or original) datasets. This is important because it's unclear whether supervised methods perform better on real data due only to their receiving supervision or also because they have better architectures for the segmentation task. (Some standard architecture, like Mask-RCNN, could be useful here.)

4. Conversely, an important control would be to somehow provide supervision to one or more of the object discovery models. This would again help to dissociate the effects of their having a worse architecture from the effects of their using weak learning principles.
questions: Everything in the paper was clear. If the authors could address the weaknesses above, I think this would be a valuable contribution to the literature even though it does not introduce a new algorithm.
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Many object-centric methods have been proposed during the last years that learn to
segment scenes and represent the objects individually. These models however don't scale
to realistic data yet. In the present study, the authors seek to understand which
factors contributing to the complexity of real world images have a high influence on
model performance. To that end, the authors train 4 object-centric models on different
variants of several datasets in which the complexity factors are manipulated
individually. The authors find that both object-level and scene-level factors contribute
to the difficulties that recent models have on natural data and conclude that essential
inductive biases are missing in recent models.
strengths_and_weaknesses: Many papers exist that propose new object-centric models, but only few that analyse the
strenghts and weaknesses of the existing methods in greater detail. The problem that
recent models don't scale to realistic data is well known but not solved yet. Studies
that test models in settings with controlled complexity to better understand these
shortcomings are therefore of great interest for the field. A particular strength of the
paper is the very systematic way of varying the several complexity factors for creating
the dataset veriants.

A weakness of the paper in my view is the presentation of the setup and the results.
Naturally, a study as the one presented in this paper has a certain complexity and
produces many subresults. Presenting those in a way that makes it easier to grasp the
major results would support the paper by making it more accessible. For example:
- For each factor, showing samples with different complexity values would make it
  easier to understand the individual factors and the complexity ranges oberved in
  the datasets used.
- The results of the different variants of a single dataset are divided over many
  figures. Comparing the performances of each model for the differently ablated datasets
  is a key endeavor of the paper, the respective results are however not presented
  jointly. Grouping the results differently might improve the paper in this respect,
  e.g. by having a figure with performances as bar plots, that has models on the x-axis
  and all variants of a single dataset on the y-axis.
questions: - How does the present study compare to other studies anlysing object-centric models? To
  what degree do your results confirm or reject previous results? The paper should
  contain a related work section adressing these questions. Other studies that should
  be mentioned are, e.g., [Karazija et al. 2021](http://arxiv.org/abs/2111.10265) and
  [Papa et al. 2022](http://arxiv.org/abs/2204.08479).
limitations: Limitations are not explicitely discussed in the paper, and the paper would be
strengthened by extending the discussion in that respect.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents an analysis of current unsupervised object-centric learning methods, focussing on shortcomings on real-world datasets. Since the performance of these methods on real images is very low, the paper changes the datasets using GT information, effectively simplifying the images. Analyzing the performance gain of specific alterations (e.g., uniform color, convex shape, etc.) give hints on which factors are especially difficult for current methods.
strengths_and_weaknesses: Strengths
The paper shows how existing datasets can be altered to analyze the performance of current methods. 

The experiments are methodical and the structure of the paper shows the findings in a logical order. 

The changes in the datasets are motivated by intuitive Gestalt principles, resulting in findings that can potentially help the community move towards real-world application of these methods. 

Weaknesses

Comparison and discussion of similar work: 
Karazija L, Laina I, Rupprecht C. ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation, NeurIPS Datasets&Benchmarks 2021.
Proposes a benchmark and an analysis of a similar set of methods. The motivation for introducing a new dataset is based on the same intuition: current real-world datasets are too difficult. Additionally it comes to some of the same conclusions (e.g., texture vs. lighting vs. plain colors).
Weis MA, Chitta K, Sharma Y, Brendel W, Bethge M, Geiger A, Ecker AS. Benchmarking Unsupervised Object Representations for Video Sequences. J. Mach. Learn. Res.. 2021
Analysis of similar concepts for video segmentation methods with a focus on occlusions. Many of the investigated methods are video extensions of the approaches in this paper.

Discussion of the differences between methods: It could be interesting to also analyze the differences between methods. For example AIR does not seem to be affected by any of the changes and does not perform well, whereas MONet  benefits from almost all changes (except YCB-S+C+U – why?) Since methods differ substantially, specific insights can be very valuable to understand what to improve.

Clarity of presentation: The results could be presented in a more comprehensive way. I found myself often confused about what to compare to what. Examples:
- Color ablation datasets have the suffix S, whereas shape ablation has the suffix C.
- To understand the changes in Figure 4 one has to compare the second half of each plot to the second half of the plots in Fig 3. Additionally, the order in Fig 3 and 4 is different.
- To understand the results in Fig 5,7,8 (which are bar plots from 0-1) one has to look up the numerical values in Tab.3 (which are numbers from 0-100). 
This makes it often very difficult to follow the reasoning in the paper and obfuscates the message.

questions: The following questions are mainly derived from the weaknesses above. 
1. How does the work position itself into related literature on benchmarking unsupervised object centric learning methods? Which findings are similar and which differ from previous work?

2. Are there findings about individual methods or approaches that generalize across the experiments? E.g., do VAE based methods generally perform better on texture-less scenes? Why?

3. Is there a way to present the results more clearly? For example by showing deltas in performance, in the same plot and without mixing bar charts and tables.

4. As all methods almost completely fail on original real-world data, are there any substantial findings that are directly related to real images? (Other than: it does not work)

5. What is the range of performance that can be expected across runs of a method? ClevrTex shows high std. dev. results for some methods, indicating that performance can strongly vary across runs.

6. Will the datasets, code, and models be published to help future methods improve and analyze their performance?


limitations: While the paper analyses the limitations of existing methods, the limitations of the presented approach are not discussed sufficiently. What insights cannot be gained from the proposed approach? What is the limit of the statements that can be derived with such a methodology? 
Societal impact is low and adequatly discussed.
review_title: null

Meta-review: metareview: This paper conducts a systematic evaluation of existing unsupervised object segmentation methods on real-world images. Using ablated real-world datasets, the authors identify factors causing the failures of existing methods on real images. 

All three reviewers find the study valuable and creative. This type of studies should be encouraged. 

To address Reviewer 5Tue's concerns, the authors added an additional section (Section 4.5) to analyze the sensitivity of different models on different dataset factors. They also added comparison results. Reviewer Zitt felt that the clarity of the paper has been clearly improved. This reviewer was convinced that this study is valuable for the field. Reviewer 5Tue felt that the changes to the figures and tables improve the readability and the additional experiments are convincing. This reviewer did not find any unadressed major concerns.


confidence: Certain
Decision: Accept