Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images
Authors: Yafei YANG
Bo Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Computer Vision
Image Segmentation
Unsupervised Learning

Abstract: In this paper, we study the problem of unsupervised object segmentation from single images. We do not introduce a new algorithm, but systematically investigate the effectiveness of existing unsupervised models on challenging real-world images. We firstly introduce four complexity factors to quantitatively measure the distributions of object- and scene-level biases in appearance and geometry for datasets with human annotations. With the aid of these factors, we empirically find that, not surprisingly, existing unsupervised models catastrophically fail to segment generic objects in real-world images, although they can easily achieve excellent performance on numerous simple synthetic datasets, due to the vast gap in objectness biases between synthetic and real images. By conducting extensive experiments on multiple groups of ablated real-world datasets, we ultimately find that the key factors underlying the colossal failure of existing unsupervised models on real-world images are the challenging distributions of object- and scene-level biases in appearance and geometry. Because of this, the inductive biases introduced in existing unsupervised models can hardly capture the diverse object distributions. Our research results suggest that future work should exploit more explicit objectness biases in the network design. 

Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images
Yafei Yang
Bo Yang
vLAR Group, The Hong Kong Polytechnic University
wtw.anneeeelpt......mmmm polyu. bo. bo.yangdpolyu edu.
Abstract
In this paper, we study the problem of unsupervised object segmentation from single images. We do not introduce new algorithm, but systematically investi- gate the effectiveness of existing unsupervised models on challenging real-world images. We firstly introduce four complexity factors to quantitatively measure the distributions of object- and scene-level biases appearance and geometry for datasets with human annotations. With the aid of these factors, we empirically find that, not surprisingly, existing unsupervised models catastrophically fail segment generic objects in real-world images, although they can easily achieve excellent performance on numerous simple synthctic datasets, due to the vast gap in objectness biases between synthetic and real images. By conducting extensive experiments on multiple groups of ablated real-world datasets, we ultimately find that the key factors underlying the colossal failure of existing unsupervised models on woal-world images are the challenging distributions of object- and scene-level biases in appearance and geometry. Because of this, the inductive biases introduced in existing unsupervised models can hardly capture the diverse object distribu- tions. Our research results suggest that future work should exploit more explicit objectness biases in the network design.
Introduction
The capability of automatically identifying individual objects from complex visual observations is central aspect of human intelligence [53]. It serves the key building block for higher-level cognition tasks such as planning and reasoning [28]. last years, a plethora of models have been proposed segment objects from single static images in unsupervised fashion: from the early AIR [22] and MONet [8] to the recent SPACE [39], SlotAtt [40], GENESIS-V2 [20], etc. They jointly learn represent and segment multiple objects from single image, without needing any human annotations in training. This process is often called perceptual grouping/binding or object- centric learning. These methods and their variants have achieved impressive segmentation results on numerous synthetic scene datasets such as dSprites [42] and CLEVR [33]. Such advances come with great expectations that the unsupervised techniques would likely close the gap with fully-supervised methods for real-world visual understanding. However, few work has systematically investigated the true potential of the emerging unsupervised object segmentation models on complex real-world images such as COCO dataset [38]. This naturally raises an essential question:
Is it promising or even possible to segment generic objects from real-world single images using (existing) unsupervised methods?
What is an object? To answer the above question involves another fundamental question: what an object? Exactly 100 years ago in Gestalt psychology, Wertheimer [61] first introduced a set of Gestalt principles such as proximity, similarity and continuation to heuristically define visual
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Single
Images
Predicted
Object
Masks
Figure 1: The failure of SlotAut [40] on three real-world images (right-hand side), although it can perfectly segment simple objects on three synthetic images (left-hand side).
data as objects. However, these factors are highly subjective, whilst the real-world generic objects are far more complex with extremely diverse appearances and shapes. Therefore, is practically impossible to quantitatively define what is an object, i.e., the objectness, from visual inputs (e.g., set of image pixels). Nevertheless, to thoroughly understand whether unsupervised methods can truly learn objectness akin to the psychological process of humans, it is vital to investigate the underlying factors that potentially facilitate or otherwise hinder the ability of unsupervised models. In this regard, by drawing on Gestalt principles, we instead define series of new factors to quantitatively measure the complexity of objects and scenes in Section 2. By taking into account both the appearance and geometry of objects and scenes, our complexity factors explicitly assess the difficulty of segmenting a specific object. For example, a chair with colorful textures tend to have higher complexity than single-color ball for unsupervised methods. With the aid of these factors, we extensively study whether and how existing unsupervised models can discover objects in Section 4.
What is the problem of unsupervised object segmentation from single images? A large number of models [63] aim tackle the problem of unsupervised object segmentation from single images. They share several key problem settings: 1) all training images do not have any human annotations; 2) every single image has multiple objects; 3) each image treated as a static data point without any dynamic or temporal information; 4) all models are trained from scratch without requiring any pretrained networks on additional datasets. Ultimately, the goal of these models is to segment all individual objects accurate the ground truth human annotations. In this paper, we regard these settings as the basic and necessary part of unsupervised object segmentation from single images, and empirically evaluate how successfully the existing models can exhibit real- world images.
Contributions and findings. This paper addresses the essential question regarding the potential of unsupervised segmentation of generic objects from real-world single images. Our contributions are:
We firstly introduce 4 complexity factors to quantitatively measure the difficulty of objects and scenes. These factors are key to investigate the true potential of existing unsupervised models.
We extensively evaluate current unsupervised approaches in large-scale experimental study. We implement 4 representative methods and train more than 130 models on curated datasets from scratch. The datasets, code and pretrained models are available https //gi thub. com/ vLAR- www.grrrouuuuuusssee....
We analyze our experimental results and find that: 1) existing unsupervised object segmentation models cannot discover generic objects from single real- sollworrd images, although they can achieve outstanding performance on synthetic datasets, as qualitatively illustrated in Figure 1; 2) the challenging distributions of both object- and scene-level biases in appearance and geometry from real-world images are the key factors incurring the failure of existing models; 3) the inductive biases introduced in existing unsupervised models are fundamentally not matched with the objectness biases exhibited in real-world images, and therefore fail to discover the real objectness.
Related Work. Recently, ClevrTex [35] and the concurrent work [43] also study unsupervised object segmentation single images. Through evaluation (complex) synthetic datasets only, both works focus on benchmarking the effectiveness of particular network designs of baselines. By comparison, paper aims to explore what and how the objectness distribution gaps between synthetic and real-world images incur the failure of existing models. The recent work [60] which investigates video object discovery is orthogonal
work as the motion signals do not exist in single images.
Scope of this research. This paper does not investigate unsupervised object discovery on saliency maps [59]. static multi-views or dynamic videos [63]. Recent methods [10; 30] requiring pretrained models on monolithic object images such as ImageNet [47] are not evaluated as well.

---

2 Complexity Factors
As illustrated in the top row of Figure 2, individual object, represented by a set of color pixels painted within a mask, can vary significantly given different types of appearance and geometric shape. A specific scene, represented by a set of objects placed within an image, can also differ vastly given different types of relative appearance and geometric layout between objects, as illustrated the bottom row. Unarguably, such variation and complexity of appearance and geometry in both object level and scene level directly affects human's 's ability to precisely separate all objects. Naturally, the performance of unsupervised segmentation models are also expected to be influenced by the variation. In this regard, carefully define the following two groups factors to quantitatively describe the complexity of different datasets.
Object #1
Object #2
Object #3
Scene #1 Figure 2: Complexity in appearance and geometry for objects and scenes.
Scene #2
Scene #3
2.1 Object-level Complexity Factors
As to a specific object, all its information can be described by appearance and geometry. Therefore we define the below two factors to measure the complexity of appearance and geometry respectively. Notably, both factors are nicely invariant to the object scale.
This factor aims to calculate how frequently the appearance changes within the object mask. particular, given the RGB image and mask an object, we firstly convert RGB into grayscale and then apply Sobel filter [51] to compute the gradients horizontally and vertically for each pixel within the mask. The final gradient value obtained by averaging out all object pixels. Note that, the object boundary pixels removed to avoid the interference of background. Numerically, the higher this factor is, the more complex texture and/or lighting effect the object has, and therefore it is likely harder segment.
Object Color Gradient:
Object Shape Concavity: This factor is designed evaluate how irregular the object boundary is. Particularly, given an object (binary) mask, denoted as M Maj E IP H we firstly find the smallest convex polygon mask (M cur E "2""> xW) that surrounds the object mask using an existing algorithm [19], and then the object shape concavity value is computed as: Σ M abj/ Σ M eaz- Clearly, the higher this factor is, the more irregular object shape is, and segmentation is more tricky.
2.2 Scene-level Complexity Factors
As specific image, in addition the object-level complexity, the spatial and appearance relation- ships between all objects can also incur extra difficulty for segmentation. We define the following two factors quantify the complexity of relative appearance and geometry between objects in image.
Inter-object Color Similarity: This factor intends to assess the appearance similarity between all objects the same image. Specifically, we firstly calculate the average color for each object, and then compute the pair-wise Euclidean distances of object colors, obtaining K x K matrix where represents the object number. The average color distance is calculated by averaging the matrix excluding diagonal entries, and the final inter-object color similarity is computed as: 1-average color distance/ (255 x v3). Intuitively, the higher this factor is, the more similar all objects appear be, the less distinctive each object is, and it is harder to separate each object.
Inter-object Shape Variation: This factor aims to measure the relative geometry diversity between all objects in the image. We firstly calculate the diagonal length of bounding box for each object, and then compute the pair-wise absolute differences for all object diagonal lengths, obtaining a K K matrix. The final inter-object shape variation is the average of the matrix excluding diagonal entries. The higher this factor, the objects within an image have more diverse and imbalanced sizes, and therefore segmenting both gigantic and tiny objects likely more challenging.
By capturing the appearance and geometry in both object and scene levels, the four factors are designed to quantify the complexity of objects and images. For illustration, Figure shows sample images for the four factors at different values. The higher the values, the more complex the objects

---

Object Color Gradient
Object Shape Concavity
object- level
0,000
0.166
0,518
0.802
0,020
0.121
0.526
0.750
Inter-object Color Similaaity
Inter-obiject Shape Variation
scene level
0,787
0,005
0.105
0.257
0.565
Figure Sample objects and scenes for the four factors at different complexity values. All complexity values are normalized to the range of [0, 1].
and scenes. In fact, these factors are carefully selected from more than candidates because they are empirically more suitable to differentiate the gaps between synthetic and real-world images, and they eventually serve as key indicators to diagnose existing unsupervised models in Section Calculation details of the four factors and other candidates are in appendix.
3 Experimental Design
3.1 Considered Methods
A range of works have explored unsupervised object segmentation in recent years. They typically formulated as (variational) autoencoders (AE/VAE) [36] or generative adversarial networks (GAN) [24]. GAN based models [13; 3; 7; 56; 5; 58; 1] are usually limited to identifying single foreground object and can hardly discover multiple objects due to the training instabilities, therefore not consid- ered in this paper. As shown in Table 1, the majority of existing models are based on AE/VAE and can be generally divided into two groups according to the object representation:
Each object is represented by explicit factors such as size, position, ap- pearance, etc., and the whole image is spatial organization of multiple objects. Basically, such representation explicitly enforces objects to be bounded within particular regions.
Fastd models:
Layer-based models: Each object is represented by an image layer, i.e., binary mask, and the whole image is a spatial mixture of multiple object layers. Intuitively, this representation does not have strict spatial constrains, and instead more flexible to cluster similar pixels as objects.
In order decompose the input images into objects, these approaches introduce different types of network architecture, loss functions, and regularization terms as inductive biases. These biases broadly include: variational encoding which encourages the disentanglement of latent variables; 2) iterative inference which likely ends up with better scene representations over occlusions; 3) object relationship regularization such depth estimation and autoregressive prior which aims capturing the dependency of multiple objects; and many other biases. With different combinations of these biases, many methods have shown outstanding performance in synthetic datasets. Among them, we select 4 representative models for our investigation: 1) AIR [22]. 2) MONet [8]. 3) IODINE [25], and 4) SlotAtt [40]. We also add the fully-supervised Mask R-CNN [29] as an additional baseline for comprehensive comparison. Implementation details are provided in appendix.
3.2 Considered Datasets
We consider two groups of datasets for extensive benchmarking and analysis: 1) three commonly-used synthetic datasets: dSprites [42], Tetris [34] and CLEVR [33], 2) three real-world datasets: YCB (9), ScanNet [17], and COCO [38]. representing the small-scale, indoor- and outdoor-level real scenes respectively. Naturally, objects and scenes in different datasets tend to have very different types of biases. For example, the objects in dSprites tend to have the single-color bias, while COCO does not. Generally, the object-level biases can be divided as: 1) appearance biases including different textures and lighting effects, and 2) geometry biases including the object shape and occlusions. Similarly, the scene-level biases include: 1) appearance biases such as the color similarity between all objects, and 2) geometry biases such as the diversity of object shapes. introduced in Section 2 are designed to well capture these biases. Table qualitatively summarizes the biases of selected datasets. We may hypothesize that the large gaps of biases between synthetic and real-world datasets would have huge impact on the effectiveness of cxisting models.
fact, our complexity factors

---

Table 1: Existing unsupervised models for object segmentation on single images. Each model
includes different inductive biases, such as variational autoencoding (VAE), iterative inference (Iter), object relationship regularization (Rel), etc.
Inductive Biases
Inductive Biases VAE
Factor-based Models
Layer-based Models
VAE
Iter Rel
Iter
Rel
CST-VAE [31]
ICLRW'I
Tagger [26]
NIPS 16
AIR [22] NIPS' 16
RC [26]
ICLRW" 16
SPAIR [16]
19
NEM [27]
NIPS'I7 17
SuPAIR [54]
ICMI'19
MONet [8]
arXiv' 19
GMTO [64]
ICML'19
10DINE [25]
ICMIL119
ASR [62] NeurlPS' 19
ECON [57]
ICLRW'20
SPACE [39]
ICTR
GENESIS [21]
ICTR
GNM [32]
NeurlPS'20
SlotAtt [40]
NeurlPS' 20
SPLIT [11]
arXiv'20
GENESIS-V -2 [20]
NeurlPS'21 21
OCIC [2]
arXiv'20
R-MONet [50]
arXiv'21
GSGN [18]
ICLR'21
CAE [41] arXiv'22
Table 2: The object- and scene-level biases in appearance and geometry of the considered datasets.
Synthetic Datasets
Real-world Datasets
dSprites [42]
Tetris [34]
CLEVR [33]
YCB [9]
ScanNet [17] COCO [38]
Object-level Blases
Texture: simple
simple simple
diverse real
simple real
diverse
Appearance
Lighting:
no
synthetic
real
Shape: simple
simple simple
simple diverse diverse
Geometry
Occlusion: minor no minor
severe severe severe
Scene-level Biases high
Appearance Similarity:
low low high
high
high
Geometry Diversity:
low
low
low
high
high high
To guarantee the fairness and consistency of all experiments, we carefully prepare all six datasets using the following same protocols. Preparation details for each dataset are provided in appendix.
All images are rerendered or cropped with the same resolution of 128 128.
Each image has about 2 to 6 solid objects with blank background.
Each dataset has about 10000 images for training. 2000 images for testing.
3.3 Considered Metrics
Having the six representative datasets and four existing unsupervised methods hand, we choose the following metrics to evaluate the object segmentation performance: 1) AP score which is widely [23], 2) PQ score which is used measure non-overlap [37], and 3) Precision and Recall scores. A predicted mask is considered correct if its loU against a ground truth mask is above 0.5. All objects are treated as a single class. The blank background is not taken into account for fair comparison. To compute AP, we simply treat the mean value of the soft object mask as the object confidence score. Note that, the alternative metrics ARI [45] and segmentation covering (SC) [4] are not considered they can be easily saturated.
used for object detection and segmentation
panoptic segmentation
4 Key Experimental Results
4.1 Can current unsupervised models succeed on real-world datasets?
First of all, we evaluate all baselines on our six datasets separately. In particular, we train each model from scratch on each dataset separately. For fair evaluation, we carefully tune the hyperparameters of each model on every dataset and fully optimize the networks until convergence. Figure 4 compares the quantitative results. It can be seen that all methods demonstrate satisfactory segmentation results on synthetic datasets, especially the recent strong baselines IODINE and SlotAtt. However, not surprisingly, all unsupervised methods fail catastrophically the three real-world datasets.
5

---

xPQ
CLAiE
Tare
CLSVe
Tral
CLAVA
ycb
(TAWR
AIR
MONer
IODINE
(b)
SlotAnr
(e) MaskRCNN
Figure 4: Quantitative results of object segmentation from the five methods on six datasets.
Preliminary Diagnosis: In order to diagnose the colossal failure, we hypothesize that is because of the huge gaps in objectness biases between two types of datasets. In this regard, we quantitatively compute the distributions of our four complexity factors on the six datasets. In particular, the two object-level factors, L.e., Object Color Gradient and Object Shape Concavity, are computed for each object of the six training splits. The two scene- factors, iee,, Inter-object Color Similarity and Inter-object Shape Variation, computed for each image of the six training splits.
From the distributions shown in the top row of Figure 5, we can see that, 1) for the two object-level factors (Subfigs 1-a and 1-c), the three syntheric datasets tend to have extremely lower scores than the real-world datasets, which means that the synthetic objects are more likely have uniform colors and convex shapes; 2) for the scene-level factors (Subfigs 1-b and 1-d), the images in synthetic datasets tend to include less similar objects in terms of color, which means that multiple objects in real-world scenes are less distinctive in appearance. In addition, the multiple objects in synthetic scenes tend to have similar sizes, whereas real-world scenes usually have diverse object scales in single images. To validate whether these distribution biases are the true reasons incurring the failure, we conduct extensive ablative experiments in Sections 4.2, 4.3 and 4.4.
Object Color Gradient Inter-object Color Similarity Object Shape Concavity Inter-object Shape Variation
original
65.
65
6d
dSprtbe
Karri
CLAVE
Scantiet
coCo
dSprttos
Fattls
CLEVE
ten
Seambiet
coco
dSperted
Ratra
CLEVE PCB
Scanfiat
coco
dSprites
CLEVE
(1e) Scaudiet
VCB
coco
(I-a)
(1-b)
(1-c)
object-
level
ablation
aSpetter Tatth
CLeva PoR.
dSpritee
Tatri
CLEVE
Ruyy Beatbbet
YoR
dSpritis
Pheris
CLAYR
Soantees
coco.6
dSpuiin
hett
CLEVE
Somtorr
(2g)
scene- level
ablation
aSpettec
Talds
CLEVE
TORT
Santter
T
asputtes
Rerts
cLeVa
ToBT
Scantie
GocO-T
diprrttes
Tebr
CLEYA
168U
dSpoitis
Phote
CLavR
(3 YCBU
Stantiat cooD-D
(3aa
(3-b)
color ablations
shape ablations
Figure 5: Distributions of the four complexity factors. The top row shows the distributions of original synthetic/real-wolt datasets in Sec 4.1. The 2nd row shows the distributions of object-level ablated datasets in Sec 4.2. The 3rd row shows distributions of scene-level ablated datasets in Sec 4.3.
4.2 How do object-level factors affect current models?
In this section, we aim to verify to what extent the distributions of object-level factors affect the segmentation performance. In particular, we conduct the following three ablative experiments.
Ablation of Object Color Gradient: For each object of the three real-world datasets, we only replace all pixel colors by its average color rgb value within each object mask, without touching the object shapes. In this way, the color gradients of each object are totally erased, thus removing the potential impact of Object Color Gradient. The three ablated datasets are: YCB-C/ ScanNet-C/ COCO-C. Ablation of Object Shape Concavity: For each object the real-world datasets, we find the smallest convex hull [19] for its object mask and then fill the empty pixels by shifting the original object

---

StotArt
*A - Prr Ree
10056
7596
50%6
25%
TeU C+S+T CAS-U CAS-THU
Stamter
C
U
T+U
CASHT
C+SHU
C+S-T+U
coco
C+S
0
TaU C+S+T C+S+U C+S+T4U
10DINE
50%
916
Tat CAS+T
CASAU CAS-T+U
Somneo
T
U
T4U
C+S+T
C+S+U
CASHT4U
coco
Gas
T4U CAS+T C+SAU
CASHTHU
MONer
100%
75%
0
5
C45
T
T+U
CSS+T
CASHU CAS-T+U
ScanNer
Cts
I
U
T+U
Co6+r
C+S4U CASOT+U
coco
T+U
CAS+T
CASHU CISHT4U
AIR
759
999
06
U
T+U C+S+T C-S- CAS-TaU
Scanoo
THU CASTT
CASul CSS-Tal
coco
Gas
U
T+U CASAT CAS+U CAS
Figure 7: Quantitative results of baselines on the three real-world datasets and their variants. The letters C/S/C+S represent the three ablated datasets in Sec 4.2: T/U/T+U represent the three ablated datasets in Sec 4.3; C+S+T/C+S+U/C+++TT
represent the three ablated datasets in Sec 4.4.
pixels. Basically, this ablation aims to only reduce the irregularity of object shapes. yet retaining the distributions of color gradients. The ablated datasets are: YCB-S/ ScanNet-S / COCO-S.
Ablation of both Object Color Gradient and Shape Concavity: We simply combine the above two ablation for each real-world object, getting datasets: YCB-C+S/ ScanNet-C+S COCO-C+S.
For illustration, the 2nd row of Figure 6 shows example images of three ablated datasets: YCB-C YCB-S YCB-C+S.
In the 2nd row of Figure (Subfigs 2-a/2-b), we calculate new distributions of both Object Color Gradient and Inter-object Color Similarity on the datasets YCB-C/ NetnNet-C/ COCO-C. We can see that the object-level gradients become all zeros in Subfig 2-a, even simpler than three synthetic datasets. Yet, the distributions of Inter-object Color Similarity are almost the same as original YCB / ScanNet COCO, i.e., Subfig 2-b is similar to 1-b.
YCB
YCB-C
YCB-S
YCB-C+S
Similarly, the 2nd row of of Figure 5 (Subfigs 2-c/2-d) shows that the distributions of Object Shape Concavity of ablated datasets now become similar to the synthetic datasets, while the distribu- tions of Inter-object Shape Variation keep the same, i.e., Subfig 2-d is similar to Subfig 1-d. Note that, for the ablation of both Object Color Gradient and Shape Concavity, the distributions will be the same as shown in the 2nd row of Figure Having the three groups of object-level ablated real- world datasets, we then evaluate segmentation performance of the baselines separately.
YCB-T
YCB-U
YCB-T+U
YCB-C+S-T YCB-C4SHU YCB-C4S4T4U Figure 6: Sample images with different ablations. The top row shows an original image from YCB dataset. The 2nd row shows examples from three ab- lated datasets in Sec 4.2. The 3rd row presents examples from three ablated datasets in Sec 4.3. The last row shows examples from three ablated datasets in Sec 4.4
Brief Analysis: As shown in Figures 7 & 9, we can see that: 1) Once the pixels real-world objects are replaced by its mean color, i.e., without any color gradients, the object segmentation performance has been significantly improved for almost all meth- ods. 2) Reducing the irregularity of real-world objects can also improve the object segmentation, although not significantly. 3) Overall, these results show that existing methods are more likely to learn the objectness represented by uniform colors and/or reg- ular objects. However, comparing with Figure 4, the results of
7

---

current ablated datasets in Figure 7 still lag behind the synthetic datasets. This means that there must be some other factors that also potentially affect the object segmentation of existing models. More results are in appendix.
4.3 How do scene-level factors affect current models?
In this section, we turn to investigate what extent the distributions of scene-level factors affect the segmentation performance. Particularly, we conduct the following three ablative experiments.
Ablation of Inter-object Color Similarity; In each image of the three real-world datasets, we replace all object textures by a set of new distinctive textures from the existing DTD database [15], as shown in Figure 8. In this way, the multiple objects look more distinctive in appearance, while the per-object texture gradients are roughly preserved. The ablated datasets are denoted: YCB-T/ ScanNet-T/ COCO-T.
Ablation of Inter-object Shape Variation: In each image of the real-world datasets, we normalize the scales of multiple objects by shrinking or expanding the diagonal length of their bounding boxes, such that the new object sizes tend to be uniform. For each object, its shape and texture linearly scaled up or down. Basically, this aims to remove the diversity of object sizes within single images. The ablated datasets are denoted as: YCB-U ScanNet-U/ COCO-U.
Ablation of both Inter-object Color Similarity and Shape Variation: We simply combine the above two ablation strategies for each weal-world image. Ablation details are in appendix. The ablated datasets are denoted as: YCB-T+U/ ScanNet-T+U / COCO-T+U.
For illustration, the 3rd row of Figure 6 shows example images of three ablated datasets: YCB-T YCB-U/ YCB- T+U.
In the 3rd row of Figure 5 (Subfigs 3-a/3-b), we calculate new distributions of both Object Color Gradient and Inter- object Color Similarity on the ablated datasets YCB-T / ScanNet-T/ COCO-T. We can see that the distributions of scene-level color similarity become more similar to synthetic datasets in Subfig 3-b, while the distributions of object color gradients are still similar as original real-world datasets YCB ScanNet / COCO, i.e., Subfig 3-a is similar 1-a.
Figure 8: Six selected texture images from DTD [15].
Similarly, the 3rd row of Figure 5 shows that the distributions of Inter-object Shape Variation of ablated datasets now become similar to the synthetic datasets in Subfig 3-d, whereas the distributions of Object Shape Concavity are still the same as original real-world datasets YCB ScanNet / COCO, i.e., Subfig 3-c similar to Subfig 1-c. Note that, for the ablation of both Inter-object Color Similarity and Shape Variation, the distributions will be the same as the 3rd row of Figure 5. Having these three groups of scene-level ablated real-world datasets, we then separately evaluate each baseline.
As shown in Figures 7 & 9, we can see that: 1) Once the textures of real-world images are replaced by more distinctive textures, i.e., with lower similarity between object appearance, the segmentation performance has been surprisingly boosted remarkably for almost all methods. 2) Normalizing object sizes over images can also reasonably improve the segmentation performance. 3) Overall, these results clearly show that existing unsupervised models significantly favor objectness with distinctive appearance in single images. However, compared with Figure 4, the results on current scene-level ablated datasets are still inferior to synthetic datasets, meaning that the scene-level factors alone are not enough to explain the performance gap. More results are in appendix.
Brief Analysis:
4.4 How do object- and scene-level factors jointly affect current models?
In this section, we aim to study how the object- and scene-level factors jointly affect the segmentation performance. In particular, we conduct the following three ablative experiments.
Ablation of Object Color Gradient, Object Shape Concavity and Inter-object Color Similarity: In each image of the three real-world datasets, we replace the object color by averaging all pixels of the distinctive texture, and also replace the object shape with a simple convex hull. The ablated datasets are denoted: YCB-C+S+T/ ScanNet-C+S+T/ COCO-C+S+T.
8

---

Ablation of Object Color Gradient, Object Shape Concavity and Inter-object Shape Variation: each image of the three real-world datasets, we replace the object color by averaging own texture, and modify the object shape as convex hull following by size normalization. The ablated datasets are denoted as: YCB-C+S+U ScanNet-C+S+U COCO-C+S+U.
Ablation of all four factors: We aggressively combine all four ablation strategies and we get COC.N//CCOOCCCCC
datasets: CEB-C+S+T+UJ
For illustration, the 4th of Figure shows example images of three ablated datasets: YCB-C+S+T / YCB-C+S+U YCB-C+S+T+U.
Since these ablations conducted independently. the new distributions of four complexity factors on current jointly ablated datasets are exactly the same as the second and third rows of Figure 5.
As shown in Figure 7, we can see that: 1) Combining the two object-level factors and either of scene-level factors ablate the real-world datasets, the segmentation performance can be improved as we expect, especially for IODINE and SlotAtt. 2) If the challenging real-world objects and images are ablated in both object- and scene-level, the segmentation performance of all unsupervised models achieves the same level with three synthetic datasets shown in Figure 4, 3) Overall, these three groups of experiments demonstrate that the colossal failure of unsupervised models on real-world images involves both object- and scene-level dataset biases. More experiments and results are in appendix.
Brief Analysis:
Inter-object
Inter-objeet Shupe
Object
Ablation
Object Shape Concavity
Similarity Ablation
Viriition Ablation
Original Images
Ablated Images
IODINE
SkoAn
Masks
YCB-C
ScanNet-C COCO-C
YCB-S ScanNer-8 COCO-S
YOR-T
ScanNes-T COCO-T
YCB-U
ScanNer-Ui COCO-1
Figure 9: Qualitative results of four representative methods on multiple ablation datasets in Sec 4.2
and Sec 4.3.
4.5
Why do current unsupervised models fail on real-world datasets?
As demonstrated in Sections 4.2/4.3/4.4. once the complexity factors are removed from the challenging real-world datasets, existing unsupervised models can perform excellent as on the synthetic datasets, as qualitatively illustrated in Figure 10. From this, we can safely conclude that the inductive biases designed in existing unsupervised models are far from able to match with and fully capture the true and complex objectness biases exhibited in real-world images. Nevertheless, from Figure 7, each baseline tends to favor different objectness biases. In particular,
AIR [22]: As factor based model, AIR has strong -ootiall-occtityy bias. Despite its poor seg- mentation performance across all datasets, there is a notable improvement when inter-object shape variation is ablated from real-world datasets (U / T+U / C+S+U / C+S+T+U). More convinc- ingly, even when all other three factors are ablated (C+S+T), can be hardly improved. These observations show that object shape variation is a significant factor for AIR.

---

Original Images
Ablated Images
GT Masks
AIR
MONet
IODINE
SlotAtt
YCB-
C+S+T+U
ScanNet- C+S+T+U
COCO-
C+S+T+U
Figure 10: Qualitative results on fully ablated real-world datasets (YCB-C+S+T+U / ScanNet- C+S+T+U COCO-C+S+T+U).
MONet [8]: MONet is more sensitive to color-related factors than shape-related factors. The abla- tions of object color gradient and inter-object color similarity significantly improve its performance, while ablations of object shape concavity and inter-object shape variation make little differences. For the color-related factors, the scene-level one is more important than the object-level factor. From this, we can see that MONet has a strong dependency on color. Similar colors tend to be grouped together while different colors separated apart. Furthermore, the ablation on object color gradient alleviates over-segmentation whereas the ablation on inter-object color similarity alleviates under-segmentation. We conjecture that seder-segmentttion can be a more severe issue for MONet on real-world datasets, leading to a larger sensitivity on the scene-level color factor.
IODINE [25]: IODINE also has heavy dependency on both object- and scene-level color-related factors. However, different from MONet, the ablation on object color gradient brings better performance than inter-object color similarity. We speculate shape latent alleviates under-segmentatiinnn by biasing towards more regular shapes. In this way, over-segmentation is the key issue, making the object color gradient a dominant factor.
because the regularization on
SlotAtt [40]: The ablations on all four factors increase the performance of SlotAtt at different levels, among which object- and scene-level color-related factors are more significant. We conjecture that it is because the feature embeddings used by Slot Attention module are learnt from both pixel colors and coordinates, which contributes its sensitivity to both shape and color factors.
Conclusions
We systematically show that existing unsupervised methods are practically impossible to segment generic objects from single real-world images, and investigate the underlying factors that incur the catastrophic failure. With the aid of our carefully designed four object- and scene-level complexity factors, we conduct extensive experiments on multiple groups of ablated real-world objects and images, and safely conclude that the distributions of both object- and scene-level biases in appearance and geometry of real-world datasets are particularly diverse and indiscriminative, such that current unsupervised models cannot segment real objects. Based on this finding, we suggest directions for future study: 1) To exploit more discriminative objectness biases such as object motions which expressively describe the ownership of visual pixels as recently explored in [55; 12; 6] for 2D images and in [52] for 3D point clouds. 2) To leverage pretrained features from single-object- dominant datasets which explicitly regard each image an object as recently studied [10; 30], although such settings are no longer purely unsupervised.
main
Broader Impact:
One limitation of this work is the lack of study about foreground- Barground-bbackground segmentation, but our work can still have a positive impact for researchers to design more practically useful models. A potential negative impact could be using our findings attack existing models.
Acknovledgements: This work was partially supported by Shenzhen Science and Technology Innovation Commission
10

---

References
R. Ahdal, P. Zhu, N. Mitra, and P. Wonka. Labcls4Free:: Unsupervised Segmentation using StyleGAN. ICCV, 2021.
[2] T. Anciukevicius, C. H. Lampert, and P. Henderson. Object-Cenne Centric Image Generation with Factored Depths, Locations, and Appearances. arXiv:2004.00642 2020. 5
[3] R. Arandjelovic and A. Zisserman. Object Discovery with a Copy-Pasting GAN. arXiv:1905.11369 2019.
[4] P. Arheláez, M. Maire, C. Fowlkes, and Malik, Contour Detection and Hierarchical Image Segmentation. TPAMI, 33(5):898-16,, 1.
[5] S. Azadi, D. Pathak, Ebrahimi, and T. Darrell. Compositional GAN: Learning Image- Conditional Binary Composition. IJC,, 2020, 4
[6] D. M. Bear, C. Fan, D. Mrowca, Y. Li, S. Alter, A. Nayebi, J. Schwartz, L. Fei-Fei, J. Wu, J. Tenenbaum, and D. I.. Yamins. Learning Physical Graph Representations from Visual Scenes. NeurIPS, 2020. 10
[7] A. Bielski and P. Favaro. Emergence of object segmentation in perturbed generative models. NeurlPS, 2019. 4
[8] C. P. Burgess, I.. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner. MONet: Unsupervised Scene Decomposition and Representation. 1xXiv:190.113900 2019. 1, 4, 10, 18, 20, 21, 22, 23, 24, 26, 27
[9] B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar. Yale-CMMU-Brrkeley dataset for robotic manipulation research. IJRR, 36(3):261 268, 2017. 4,5,
[10] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging Properties in SuperSupe Vision Transformers. ICCV, 2021. 2, 10
[11] R. Charakom, Y. Thawornattana, S. Itthipuripat. N. Pawlowski. and N. Dilokthanakul. An Explicit Local and Global Representation Disentanglement Framework with Applications Deep Clustering and Unsupervised Object Detection. arXi::2001.00 089577 2020. 5
[12] Н. Chen, R. Venkatesh, Y. Friedman, J. Wu, Tenenbaum, D. L. K. Yamins, and D. M. Bear. supervised Segmentation in Real-World Images via Spelke Object Inference. arXiv:2205.088 08515, 2022. 10
[13] M. Chen, T. Artières, and L. Denoyer. Unsupervised object segmentation by redrawing. NeurlPS, 2019. 4
[14] B. Cheng, R. Girshick, Dollár, C. Berg, and Kirillov. Boundary IoU: Improving Object-Centric Image Segmentation Evaluation. CVPR, 2021. 15, 17
[15] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing Textures in the Wild. CVPR, 2014. 8, 23
[16] E. Crawford and Pineau. Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks. AAAI, 2019. 5
[17] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. ScanNet: Richly-
3D Reconstructions of Indoor Scenes. CVPR, 2017. 4, 5, 19
[18] F. Deng, 7. Zhi, D. Lee, and S. Ahn. Generative Scene Graph Networks. ICLR, 2021. 5
[19] S. Eddins. Binary image convex hull. hull-al hall-algorithm-notes notes// 2011. 3, 6, 15, 24
[20] M. Engelcke, O. P. Jones, and I. Posner. GENESIS-VE Inferring Unordered Object Representations without Iterative Refinement. NewrIPS, 2021. 1,5
[21] M. Engelcke, A. R. Kosiorek, O. P. Jones, and Posner. GENESIS: Generative Scene Inference and Sampling with Object-Centrie Latent Representations. ICLR, 2020. 5,
[22] S. M. A. Eslami, N. Heess, T. Weber, Y. Tassa, K. Kavukcuoglu, and G. E. Hinton. Attend, Infer, Repeat: Fast Seene Understanding with Generative Models. NIPS, 2016. 1, 4, 5, 9, 18, 20, 22, 23, 24, 26, 27
[23] M. Everingham. S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge: A Retrospective. IJC,, 111:98 136, 2015.
11

---

[24] I. J. Goodfellow, J. Pouget-Abadie. M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Nets. NIPS, 2014.
[25] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and
A. Lerchner. Multi-ohjeet representation learning with iterative variational inference. ICMI., 2019. 4,
10, 18, 20, 21, 22, 23, 24, 26, 27
[26] K. Greff, A. Rasmus, M. Berglund, perceptual grouping. NIPS, 2016. 5
Hao, J. Schmidhuber, and H. Valpola. Tagger: Deep unsupervised
[27] K. Greff, S. Van Steenkiste, and Schmidhuber. Neural Expectation Maximization. NIPS, 2017. 5
[28] K. Greff, van Steenkiste, and Schmidhuber. On the binding problem artificial neural networks. arXiv:2012.05208 2020.
[29] K. He. Gkioxari, Dollar, and R. Girshick. Mask R-CNN. (CCV,2017 18, 21
[30] J. Hénaff, S. Koppula, E. Shelhamer, Zoran, A. Jacgle, A. Zisserman, J. Carreira, and Arandjelovié.
Object discovery and representation networks. ECCV, 2022. 2, 10
[31] Huang K. Murphy. Efficient inference in occlusion-awand aware generative models of images. ICLR
Workshops, 2016. 5
[32] J. Jiang and S. Ahn. Generative neurosymbolic machines. NeurIPS, 2020.
[33] Johnson, Hariharan, van der Maaten. Fei-Fei, L. Zitnick, R. Girshick. CLEVR: Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. CVPR, 2017. 1,
18, 19
[34] R. Kabra, C. Burgess, L. Matthey, R. L. Kaufman, K. Greff, M. Reynolds, and A. Lerehner. Multi-Ohject Datasets.
2019. 4,5, 19
[35] L. Karazija, I. Laina, and C. Rupprecht. ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi- Object Segmentation. NeurIPS, 2021. 2
[36] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. ICLR, 2014. 4
[37] A. Kirillov, K. Hc, R. Girshick, C. Rother, and P. Dollár. Panoptic Segmentation. CVPR, 2019. 5
[38] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common Objects in Context. ECCV, 2014. 5,
[391 Z. Lin, Y.FF. Wu, S. V. Peri, W. Sun, G. Singh, F. Deng, J. Jiang, and S. Ahn. SPACE: Unsupervised Object- OriecttOrreeeedd Scene Representation via Spatial Attention and Decomposition. ICLR, 2020. 1,5
[40] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and
T. Kipf. Object-Centric Learning with Slot Attention. NeurlPS, 2020. 1, 1, 4, 5, 10, 18, 20, 22, 23, 24, 26. 27
[41] S. Löwe, P. Lippe, M. Rudolph, and M. Welling. Complex-Valued Autoencoders for Object Discovery. arXiv:2204.02077 2022. 5
[42] L. Mathhe, Higgins, Hassabis, A. Lerchner. dSprites Disentanglement testing Sprites dataset. http::/ggthub.co//eeepnnnt htwp:d/eepnnnndddpptts--daeteeee 2017. 1,4, 5, 19
[43] S. Papa, O. Winther, and A. Dittadi. Inductive Biases for Object-Centric Representations in Presence
of Complex Textures. XXXx:2204.084777 2022. 2
[44] D. D. Polsby and R. D. Popper. The third criterion: Compactness as procedural safeguard against partisan
gerrymandering. Yale L. & Pol'y Rev., 9, 1991. 16
[45] W. M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(336)886 850, 1971.
[46] D. J. Rezende and F. Viola. Taming vacs. arXiv:1810.00597. 2018. 18
[47] O. Russakovsky, J. Deng. H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang. A. Karpathy, A. Khosla, M. Bern- stein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015.
12

---

[48] J. E. Schwartzberg, Reapportiommnt,, gerrymanders, and the notion of compactness. Minn. L. Rev. 50:443,
1965. 16
[49] C. E. Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379- 423, 1948. 16
[50] Shengxin Qian. R-MONet: Region-Based Unsupervised Scene Decomposition and Representation via Consistency of Object Representations. OpenReview.Ner, 2021. 5
[51] I. Sohel and G. Feldman. A 3x3 Isotropic Gradient Operator for Image Processing. Pottern Classification and Scene Analywis, pages 271-272, 1973. 3, 15
[52] Z. Song and B. Yang. OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds. NeurIPS, 2022. 10
[53] E. S. Spelke, K. Breinlinger, J. Macomber, and K. Jacobson. Origins of knowledge. Psychological Review, 99(4))605-632 1992. 1
[54] K. Stelzner, R. Peharz, and K. Kersting. Faster attend-infer-repeat with tractable probabilistic models. ICML., 2019.
[55] M. Tangemann, S. Schneider, J. von Kügelgen, F. Locatello, P. Gehler, T. Brox, M. Kümmerer, M. Bethge, and B. Schölkopf. Unsupervised Object Learning via Common Fate. arXiv:2110.06562, 2021. 10
[56] S. van Steenkiste, K. Kurach, J. Schmidhuber, and S. Gelly. Investigating object compositionality in Generative Adversarial Networks. Neural Networks, 2020.
[57] J. Kügelgenn I. Ustyuzhaninov, P. Gehler, M. Bethge, and B. Schölkopf. Towards causal generative scene models via competition of experts. ICLR Workshops, 2020.
[58] A. Voynov, S. Morozov, and A. Babenko. Object Segmentation Without Labels with Scale Generative Models. ICML, 2021. 4
[59] W. Wang, Q Lai. H. Fu, J. Shen, H. Ling, and R. Yang. Salient Object Detection in the Deep Learning Era: An In-Depth Survey. TPAMI, 2021. 2
[60] M. A. Weis, K. Chitta, Y. Sharma, W. Brendel, M. Bethge, A. Geiger, and A. S. Ecker. Benchmarking Unsupervised Object Representations for Video Sequences. JMLR, 2021.
[61] M. Wertheimer. Untersuchungen Lehre yon der Gestalt. Psychologische Forschung. 1923.
[62] K. Xu, C. Li, J. Zhu, and B. Zhang. Multi-object generation with amortized structural regularization. NeurIPS, 2019.
[63] J. Yuan. T. Chen, B. Li, and X. Xue. Compositional Scene Representation Learning via Reconstruction: A Survey, 2rXiv:220..07135, 2022.
[64] J. Yuan, B. Li, and X. Xue. Generative modeling of infinite occluded objects for compositional scene representation. ICML, 2019. 5
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(C) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] See Contribu- tions and finding in Section 1.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Implementation Details in Appendix.
(c) Did report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] In the tables for qunatitative results, we report standard deviation of performances over three runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Implementation Details in Appendix.
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material URL? [Yes]
See Contributions and finding in Section
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "bZ6WMyf2NLd", "reviewer": "Reviewer_fjR6", "summary": "This paper keys in on something that is widely suspected, but has not been rigorously investigated in the unsupervised object segmentation / discovery literature: the fact that nearly all methods do well on \"simple\" synthetic datasets but not on complex, real-world datasets that are the subject of most supervised object segmentation work.\n\nIn particular, this work makes two contributions:\n\n1. It tests several of the better and more popular unsupervised object discovery methods on a range of datasets, spanning simple/synthetic to complex/real-world, and (maybe for the first time) actually quantifies the performance on the harder datasets using standard metrics. (For some reason, a lot of the unsupervised object discovery literature uses the non-standard foreground-ARI metric, which the authors here correctly note will make most models look far better quantitatively than they are qualitatively.) \n\nIn this experiment, the authors find -- unsurprisingly -- that the models they test fail catastrophically on real-world data.\n\n2. The paper creates new datasets in which the real-world datasets are \"ablated\" to alter object- and scene-level factors, which the authors hypothesize could explain the difference in model performance on synthetic vs real-world data. The authors then test this by training/testing the models on the new \"ablated\" datasets. When some or all of the factors that distinguish the real-world data are ablated, the unsupervised models can actually perform quite well.", "questions": "Everything in the paper was clear. If the authors could address the weaknesses above, I think this would be a valuable contribution to the literature even though it does not introduce a new algorithm.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Actually quantifying the unsupervised model performance on harder data was necessary, even if the results are unsurprising. This is an important contribution to the literature as it makes explicit something that many people \"knew about,\" but could not point to concretely.\n\n2. The dataset ablation experiments are very creative and the results are interesting. I was surprised that relatively simple factors could explain the gap in performance between synthetic and real datasets, although looking at the \"ablated\" images in Figure 9 it makes more sense: the fully-ablated datasets really do look much more like the synthetic datasets in terms of object and scene complexity and distribution.\n\n3. The ideas and experiments are well-motivated and clearly explained.\n\nWeaknesses:\n\n1. It's hard to draw conclusions about what the *models* are missing because most of them have very similar design principles. The authors don't comment much on which models seem to handle which dataset factors better than others. \n\n2. The authors suggest that models based on different principles/inductive biases might be needed to handle the complexity of real-world data, especially ones based on learning from object motion. There is some recent work in this direction (e.g. https://arxiv.org/abs/2205.08515 and https://arxiv.org/abs/2110.06562 and https://proceedings.neurips.cc/paper/2020/file/4324e8d0d37b110ee1a4f1633ac52df5-Paper.pdf). This may have been too recent for the authors to know about at the time of writing (and to test) but it's important to point out that there are ideas for dealing with these issues and some progress.\n\n3. There is no comparison to supervised methods with the ablated (or original) datasets. This is important because it's unclear whether supervised methods perform better on real data due only to their receiving supervision or also because they have better architectures for the segmentation task. (Some standard architecture, like Mask-RCNN, could be useful here.)\n\n4. Conversely, an important control would be to somehow provide supervision to one or more of the object discovery models. This would again help to dissociate the effects of their having a worse architecture from the effects of their using weak learning principles.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YrwBrpkasnO", "reviewer": "Reviewer_Zitt", "summary": "Many object-centric methods have been proposed during the last years that learn to\nsegment scenes and represent the objects individually. These models however don't scale\nto realistic data yet. In the present study, the authors seek to understand which\nfactors contributing to the complexity of real world images have a high influence on\nmodel performance. To that end, the authors train 4 object-centric models on different\nvariants of several datasets in which the complexity factors are manipulated\nindividually. The authors find that both object-level and scene-level factors contribute\nto the difficulties that recent models have on natural data and conclude that essential\ninductive biases are missing in recent models.", "questions": "- How does the present study compare to other studies anlysing object-centric models? To\n  what degree do your results confirm or reject previous results? The paper should\n  contain a related work section adressing these questions. Other studies that should\n  be mentioned are, e.g., [Karazija et al. 2021](http://arxiv.org/abs/2111.10265) and\n  [Papa et al. 2022](http://arxiv.org/abs/2204.08479).", "limitations": "Limitations are not explicitely discussed in the paper, and the paper would be\nstrengthened by extending the discussion in that respect.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Many papers exist that propose new object-centric models, but only few that analyse the\nstrenghts and weaknesses of the existing methods in greater detail. The problem that\nrecent models don't scale to realistic data is well known but not solved yet. Studies\nthat test models in settings with controlled complexity to better understand these\nshortcomings are therefore of great interest for the field. A particular strength of the\npaper is the very systematic way of varying the several complexity factors for creating\nthe dataset veriants.\n\nA weakness of the paper in my view is the presentation of the setup and the results.\nNaturally, a study as the one presented in this paper has a certain complexity and\nproduces many subresults. Presenting those in a way that makes it easier to grasp the\nmajor results would support the paper by making it more accessible. For example:\n- For each factor, showing samples with different complexity values would make it\n  easier to understand the individual factors and the complexity ranges oberved in\n  the datasets used.\n- The results of the different variants of a single dataset are divided over many\n  figures. Comparing the performances of each model for the differently ablated datasets\n  is a key endeavor of the paper, the respective results are however not presented\n  jointly. Grouping the results differently might improve the paper in this respect,\n  e.g. by having a figure with performances as bar plots, that has models on the x-axis\n  and all variants of a single dataset on the y-axis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NaB5imi9p3e", "reviewer": "Reviewer_5Tue", "summary": "The paper presents an analysis of current unsupervised object-centric learning methods, focussing on shortcomings on real-world datasets. Since the performance of these methods on real images is very low, the paper changes the datasets using GT information, effectively simplifying the images. Analyzing the performance gain of specific alterations (e.g., uniform color, convex shape, etc.) give hints on which factors are especially difficult for current methods.", "questions": "The following questions are mainly derived from the weaknesses above. \n1. How does the work position itself into related literature on benchmarking unsupervised object centric learning methods? Which findings are similar and which differ from previous work?\n\n2. Are there findings about individual methods or approaches that generalize across the experiments? E.g., do VAE based methods generally perform better on texture-less scenes? Why?\n\n3. Is there a way to present the results more clearly? For example by showing deltas in performance, in the same plot and without mixing bar charts and tables.\n\n4. As all methods almost completely fail on original real-world data, are there any substantial findings that are directly related to real images? (Other than: it does not work)\n\n5. What is the range of performance that can be expected across runs of a method? ClevrTex shows high std. dev. results for some methods, indicating that performance can strongly vary across runs.\n\n6. Will the datasets, code, and models be published to help future methods improve and analyze their performance?\n\n", "limitations": "While the paper analyses the limitations of existing methods, the limitations of the presented approach are not discussed sufficiently. What insights cannot be gained from the proposed approach? What is the limit of the statements that can be derived with such a methodology? \nSocietal impact is low and adequatly discussed. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe paper shows how existing datasets can be altered to analyze the performance of current methods. \n\nThe experiments are methodical and the structure of the paper shows the findings in a logical order. \n\nThe changes in the datasets are motivated by intuitive Gestalt principles, resulting in findings that can potentially help the community move towards real-world application of these methods. \n\nWeaknesses\n\nComparison and discussion of similar work: \nKarazija L, Laina I, Rupprecht C. ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation, NeurIPS Datasets&Benchmarks 2021.\nProposes a benchmark and an analysis of a similar set of methods. The motivation for introducing a new dataset is based on the same intuition: current real-world datasets are too difficult. Additionally it comes to some of the same conclusions (e.g., texture vs. lighting vs. plain colors).\nWeis MA, Chitta K, Sharma Y, Brendel W, Bethge M, Geiger A, Ecker AS. Benchmarking Unsupervised Object Representations for Video Sequences. J. Mach. Learn. Res.. 2021\nAnalysis of similar concepts for video segmentation methods with a focus on occlusions. Many of the investigated methods are video extensions of the approaches in this paper.\n\nDiscussion of the differences between methods: It could be interesting to also analyze the differences between methods. For example AIR does not seem to be affected by any of the changes and does not perform well, whereas MONet  benefits from almost all changes (except YCB-S+C+U – why?) Since methods differ substantially, specific insights can be very valuable to understand what to improve.\n\nClarity of presentation: The results could be presented in a more comprehensive way. I found myself often confused about what to compare to what. Examples:\n- Color ablation datasets have the suffix S, whereas shape ablation has the suffix C.\n- To understand the changes in Figure 4 one has to compare the second half of each plot to the second half of the plots in Fig 3. Additionally, the order in Fig 3 and 4 is different.\n- To understand the results in Fig 5,7,8 (which are bar plots from 0-1) one has to look up the numerical values in Tab.3 (which are numbers from 0-100). \nThis makes it often very difficult to follow the reasoning in the paper and obfuscates the message.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept