Multi-Object Representation Learning via Feature Connectivity and Object-Centric Regularization
Authors: Alex Foo
Wynne Hsu
Mong-Li Lee
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Object-Centric Learning
Multi-Object Representation Learning
TL;DR: We leverage on feature connectivity to perform fine-grained object discovery and design two object-centric regularization terms as an alternative to the pixel-based reconstruction loss used in state-of-the-art methods.

Abstract: Discovering object-centric representations from images has the potential to greatly improve the robustness, sample efficiency and interpretability of machine learning algorithms. Current works on multi-object images typically follow a generative approach that optimizes for input reconstruction and fail to scale to real-world datasets despite significant increases in model capacity. We address this limitation by proposing a novel method that leverages feature connectivity to cluster neighboring pixels likely to belong to the same object. We further design two object-centric regularization terms to refine object representations in the latent space, enabling our approach to scale to complex real-world images. Experimental results on simulated, real-world, complex texture and common object images demonstrate a substantial improvement in the quality of discovered objects compared to state-of-the-art methods, as well as the sample efficiency and generalizability of our approach. We also show that the discovered object-centric representations can accurately predict key object properties in downstream tasks, highlighting the potential of our method to advance the field of multi-object representation learning.

Multi-Object Representation Learning via Feature Connectivity and Object-Centris Regularization
Alex Foo
Wynne Hsu
Mong Lee
School of Computing
National University of Singapore
{alexfoo, whsu, leeml |acomp....m...m nus edu.sseddu.... 8g
Abstract
Discovering object-centric representations from images has the potential to greatly improve the robustness, sample efficiency and interpretability of machine learning algorithms. Current works on multi-object images typically follow generative approach that optimizes for input reconstruction and fail scale to real-world datasets despite significant increases in model capacity. We address this limitation by proposing a novel method that leverages feature connectivity to cluster neighbor- ing pixels likely to belong to the same object. We further design two object-centric regularization terms to refine object representations in the latent space, enabling our approach scale to complex real-world images. Experimental results on simulated, worlwo complex texture and common object images demonstrate substan- tial improvement in the quality of discovered objects compared to state-of-the-art methods, as well as the sample efficiency and generalizability of our approach. We also show that the discovered object-centric representations can accurately predict key object properties downstream tasks, highlighting the potential of our method to advance the field of multi-object representation learning.
Introduction
Human understanding of the world relies on objects compositional building blocks 24], and emulating this through object-centric representations can improve robustness, sample efficiency, generalization to out-of-domain distributions, and interpretability of machine learning algorithms [8 9]. Recent work utilizes a generative approach, optimizing pixel- based reconstruction to learn object-centric representations as it prioritizes pixel accuracy over object discovery and functional feature extraction 30 This may lead to the failure of discovering objects shape, or boundaries between overlapping objects
35
iK]
45
46 51 This approach has limitations
35 obtaining useful object features, such as position, 23 Additionally, pixel-based reconstruction tends waste model capacity less important visual features, such as complex backgrounds making scaling these methods to real-world images a challenge.
To address the fundamental limitations of pixel-based reconstruction, propose framework that leverages feature connectivity and design two object-centric regularization terms to directly refine object representations, ensuring sufficient separation and high disentanglement between dimensions. Our method utilizes visual connectedness principles [38], where similar pixels that are connected should belong to the same object, to guide object discovery. The regularization terms promote disentangled representations and prevent sub-optimal clustering.
We demonstrate that our approach outperforms state-of-the-art methods in discovering multiple objects from simulated, real-world, complex texture and common object images in a fine-grained manner without supervision. The proposed solution attains sample efficiency and is generalizable to out-of-domain images. The learned object representations also accurately predict key object
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

properties in downstream tasks. Our contributions include: (1) a framework that leverages feature connectivity for fine-grained object discovery, (2) introduction of object-centric regularization terms as an alternative to pixel-based reconstruction loss, (3) experimental validation of our solution's s superior performance, and (4) demonstration of the usefulness of discovered object representations in downstream tasks.
Related Work
Numerous works have demonstrated remarkable success in segmenting real-world images. Most of these works focus semantic segmentation and object detection by utilizing supervised signals [19 [.. In contrast, unsupervised approaches like SLIC [ employ modified k-means algorithm to cluster pixels into superpixels, similar to Felzenszwalb's s algorithm [[5 that relies hand-crafted features for clustering. However, these methods do not focus on learning useful representations for the segmented components.
Various unsupervised methods for learning object-centric representations have been proposed, and can be categorized into three main approaches: spatial attention, sequential attention, and iterative attention. Spatial attention approach utilizes spatial transformer networks [2 to crop out rectangular
regions from image and extract object attributes such position and scale [44] Kll 33 They
rely on a fixed-size sampling grid which may not be suitable for scenes with varying object sizes, and may compromise training when the sampling grid does not overlap with any object.
The sequential attention approach uses RNN-baaed models such as MONet [ and GENESIS to sequentially attend to different regions in an image. These methods employ a deterministic network to perform the attention process, which allows them to capture and represent objects in the scene. However, these methods may neglect smaller objects as they tend to produce a weaker signal during the attention process. This can lead to incomplete or biased representations of scenes with objects of varying sizes. To overcome this, GENESIS-V2 [ perform attention randomly.
uses stochastic stick-breaking process to
In the iterative attention approach, set of object representations is randomly initialized and then iteratively refined to bind these objects to different regions of an image. IODINE [T7] is model that can discover objects with disentangled representations. However, it requires long training times and many samples. Slot Attention
35 introduces competition among the object representations by utilizing cross-attention along the object dimension. While this method is fast, versatile and can be
extended to handle videos (29). it may fail to discover objects when the training set is diverse, and the resulting representations are highly entangled.
EfficientMORL (T)) SLATE 45
SysBinder [4 and BO-QSA are recent developments in the iterative attention approach aimed at addressing some limitations of carlier methods like Slot Attention. EfficientMORL presents an hierarchical variational autoencoder and lightweight iterative refinement network to increase efficiency without sacrificing representation quality. SLATE increases the non-linear interaction between the slots Slot Attention with an autoregressive decoder that conditioned the slots, resulting in improved reconstructions and object-centric representations. SysBinder enhances the slots of Slot Attention with factor representations called block-slots, which provides within-slot disentanglement between learned factors. BO-QSA initializes the slots of Slot Attention as learnable embeddings instead of sampling from a Gaussian distribution and uses bi-level optimization, resulting in more stable training. Despite the advancements, one drawback remains: the number of clusters are fixed a priori which limits the applicability in real-world scenarios where the number of objects or clusters is not known beforehand.
3 Methodology
Our proposed method OC-Net designed to extract objects in an image without relying on labeled data specifying the number of objects present in the image. By not requiring the number of objects to be specified beforehand, OC-Net can generalize better to real-world scenes with varying numbers of objects and handle more complex scenarios.
Figure T shows the main components of OC-Net. The main idea behind OC-Net is to learn pixel embeddings that can be clustered to discover objects and their respective object masks and object

---

Emboddings
Retearking
Inpat
Leyerfamm Pasiiinn Erc.
Entargleneent
Computing
Objest Nask
Loss Cervi
Sampling
Clodseiig
Object
Rear Repressntation
Object Oi
Extracting
Separation Loss Loup
Entarggeneet
Compriing seature connectivity
1kkg
Clustering
Aveeaging
Objest
Repressstalinn
Object O
Listracking
Soparation
Lass
Entanggenentt Loss C.cs
Computting
Objest Mask masr
Sampling
connertivity
Clusteria
Avrezginx
Object Represontation
Object Ost
Figure 1: Overview of OC-Net.
representations. This is achieved by passing the input image through to obtain a set of N pixel embeddings P {PI...., .NN) of D dimensions each. We leverage on feature connectivity and iteratively cluster the embeddings of neighbouring pixels based on the likelihood that they belong to the same object. The output is set of objects O {O1,.... O } where each object a set of pixel embeddings. We derive the object mask m, of each object by setting the pixel corresponding to the embedding in Oj:
convolutional layer
[1 if P, E EO otherwise
my
0
where my[i] denotes the th pixel value and i [N.....NN.
With this, we obtain the matrix of object representations Z = [z,...... Zwl where each z is the sum of extracted mask information and the average of the pixel embeddings in Oj:
z | = (A my )d
+
Σ
[:[d]
(2)
10g
P:EO,
where zy 2y denotes the th value of vector Zj, d E (l,...,,) is the mask transformation matrix.
3.1
Object Discovery
The object discovery process iteratively clusters the pixel embeddings based on their feature connec- tivity and similarity. LayerNorm 2 is applied to normalize all pixel embeddings, and positional encodings added to the pixel embeddings. The neighbors of pixel embedding p the set of embeddings of the neighbours in the input image. We use Dijkstra's 's algorithm to compute the shortest distance of sampled pixel embedding to all other embeddings follows.
Let Uf be the set of pixel embeddings that have not been assigned to an object yet. We uniformly E U. The distance from p. to itself is set to zero, and the distance
sample pixel embedding all the other pixels is set to infinity. We select an unvisited pixel embedding Pm that has the minimum distance to p;- Let Pı: be the embedding of neighbour of the pixel corresponding to Pm- We compute the distance between a pair of neighhouring pixels as the similarity between their corresponding embeddings given by:
sim(pan Pa)
Σ d=1
Pm [d] P2(d)2
(3)

---

where Pm[d] denotes the dth value of the embedding Pm-
If the distance between P; and P& is shorter through pm: we update the shortest distance accordingly. This ensures that we consider the most efficient path between pixel embeddings, leading to better object discovery. We mark Pm visited and consider be part of the same object as p, according to threshold. The process is repeated for the next unvisited pixel embedding until all pixels have been visited.
3.2 Object-Centric Regularization
We design two object-centric regularization terms Seep and Lent to improve the quality the learned object representations for downstream generalization and object discovery. Given the matrix of object representations = [z....., M corresponding to the object training samples y.) we quantify downstream generalization performance with the prediction error:
8z
I z, -vl
where W is the minimum-now mnrm norm solution of the downstream predictor.
Since labels y; are unknown in the unsupervised setting, Oz cannot be directly minimized. Theorem minimize upper bound of 8z by using the projection matrix Pz. Proof of the theorem is provided in the supplementary material.
3.1 below shows that
Theorem 3.1. Let Y be the matrix of training sample labels and Pz be the projection matrix of Z:
Pz=I-2 2T Z
(5)
where is the identity matrix, (t) is the pseudoinverse and Σz = ZZT is the unnormalized covariance matrix Z. Let ||-I|p be the Frobenius norm. Then, the the following relation holds:
8z S Pz |F Y||F
(6)
Since Y in Equation 6 is unknown but fixed, we can minimize óz by minimizing |Pz||F.
From the definition of Pz in Equation 5 |Pz|| minimized when the rank of Σz is maximized We achieve this by maximizing the diagonal entries of Σz with a separation term Lep while simultaneously minimizing its off-diagonal entries with an entanglement term Cents in effect regularizing Σz to be diagonal matrix with maximum number of nonzero entries.
44 3.
Maximizing the diagonal entries of Ez via Leep consequently maximizes the distance between object representations in the latent space. This encourages the model to learn distinct and non-overlapping object representations. Expanding the representation space also ensures that objects with varying features and properties can be accurately represented and distinguished from one another, enhancing both downstream generalization and fine- giee-grained object discovery. We define Leep as:
Csep
Σ
max(0, Voa TT)
D
d=l
where aa is the variance of the tt dimension across the vectors 21," , and 7 a small constant to maintain numerical stability.
Z.M
The entanglement term Lent minimizes the off-diagonal entries Σz and consequently minimizes the correlation between dimensions in the latent space Z. thereby achieves more disentangled object representations. Such representations are easier manipulate and analyze, as each dimension captures a distinct object property, such as position, scale, or color. Lent is defined as follows:
Lent Dx (M -
(8)
Performance Study
We conduct experiments to evaluate the performance of OC-Net in terms of quality, sample efficiency and generalizability. We use diverse range of datasets to demonstrate effectiveness across various scenarios:

---

Table 1: Summary of dataset characteristics
Dataset
Type
Ground Truth
Image Size
# Samples
SSptiedSprites
Simulated
Pixel Mask
x 64
Tetrominoes-NM
Simulated
Pixel Mask
35 x 35
1M
SVHN
Real- World
Bounding Box
Varied
530K
IDRiD
Real-Wordh World
Pixel Mask
4288 x 2848
81
CLEVRTEX
Complex Texture
Pixel Mask
128 x 128
50K
CLEVRTEX-OOD
Complex Texture
Pixel Mask
128 x 128
10K
Flowers
Common Object
Pixel Mask
128 x 128
7K
Birds
Common Object
Pixel Mask
128 x 128
11K
COCO
Common Object
Pixel Mask
128 x 128
12K
1. Simulated datasets Multi-dSprites 2 and Tetrominoes-Ne NM. The former consists of multi- ple oval, heart or square-shaped sprites with some occlusions, while the latter is subset of the original Tetrominoes dataset 23 where images whose ground truth segmentation
requires knowledge of the object shapes filtered out.
2. Real-world multi-object datasets SVHN [36] and IDRiD 40. SVHN consists of street view images of house numbers while IDRiD is the Indian Diabetic Retinopathy Image Segmentation Dataset.
3. Complex texture datasets CLEVRTEX 26 and CLEVRTEX-OOD. CLEVRTEX features scenes with diverse shapes, textures and photo-mapped materials while CLEVRTEX-OOD is the CLEVRTEX out-of-distrbution test set with 25 new materials and 4 new shapes.
4. Common object datasets Flowers 37 Birds 48 and COCO (50) The Flowers dataset features 17 diverse flower classes with large variations viewpoint, scale, illumination and background. Birds is the most widely-used CUB-200-2011 dataset for fine-grained visual categorization. COCO is the variant of the Microsoft Common Objects in Context dataset used for large-scale object segmentation [2]
Table shows the dataset characteristics. Following dSprites, Tetrominoes-NM and SVHN for training and hold out the next 320 samples for testing. For IDRiD, we split this dataset into 54 images for training and 27 images for testing. For CLEVRTEX, use the first 40K samples for training and last 5K samples for testing. For CLEVRTEX-OOD. we use 10K samples for testing. For Flowers, we use the first 6K samples for training and last 1K samples for testing. For Birds, we use the first 10K samples for training and last IK samples for testing. For COCO, we use the first 10K samples for training and last 2K samples for testing.
35 |ОI, we use the first 60K samples in Multi-
We compare OC-Net with SLIC M. Felzenszwalb [T] Slot Attention 35 EfficientMORL (II] GENESIS-VE V2 [R. SLATE 45), SysBinder 46 and BO-QSA 22 We train OC-Net for 1000 iterations with a batch size of 64 using Adam 28 with learning rate of x 10-3, We carried out an initial experiment to choose the clustering threshold. The results show that the value can range from 0.2 2.0 without affecting the performance of OC-Net. As such, set the threshold to 0.7 so that two pixels will belong to the same object if their normalized feature similarity is more than 50%. If pixel is assigned multiple objects, assign it to the mask of the first object in that list and ignore its membership in other objects. Training on 64-by-64 images from Multi-dSprites on a single V100 GPU with 32GB of RAM takes about 10 minutes.
For all methods, we set the maximum number of foreground objects to 6 and 4 for Multi-dSprites and Tetrominoes respectively. Training is carried out for 300,000 iterations with a batch size of 64, using the Adam optimizer with a base learning rate of 4 10-4 We set the size of the latent space to be D = 64 for all models. For SVHN and COCO, the number of objects is set to 6. For IDRiD, the number of objects is set to 20 and we train them for 100,000 iterations. For CLEVRTEX and CLEVRTEX-OOD the number of objects is set 11. For Flowers and Birds, the number of objects set to 2.
We use the Adjusted Rand Index (ARI) to measure the quality of objects discovered 20 The ARI is a measure of similarity between two data clusterings that takes into account the permutation-invaria nature of the predicted segmentation masks and their corresponding ground-truth masks. We also
5

---

use the Dice similarity coefficient, along with the ooneecctiooovve-nniiiii (IoU) between the best matching object masks X and Y follows:
X ny X + Y : IoU(X,Y) = IXUY
XnnY
Dice(X,Y)
(9)
where X the set of object pixels extracted and Y the set of annotated object pixels in the ground truth. We compute the mean Dice and the mean loU scores, denoted as mDice and mloU respectively, by averaging the individual Dice and IoU scores across matches. For the complex textures dataset, background discovery is included in the computation of the scores.
4.1 Experiments on Quality of Discovered Objects
We first evaluate the ability of OC-Net to discover objects from images with multiple objects. Table Ma) shows the average ARI, mDice and mloU scores based on the discovered foreground objects in the simulated datasets after runs. We observe that OC-Net outperforms all other methods by a large margin in Multi-dSprites, and even achieves perfect score for all Tetrominoes-Ne test samples.
Table b)) shows the results on the real-world multi-object image datasets. For SVHN, OC-Net outperforms all methods, even when the ground truth is provided in the form of bounding boxes. This implies that we need expand the discovered object masks into their corresponding bounding boxes which are often rough fits, and is the reason for the close difference in mDice and mloU scores between OC-Net and BO-QSA. For IDRiD, which contains multiple small objects, OC-Net significantly improves the ARI scores and more than doubles the mDice and mloU scores over all methods, demonstrating its robustness challenging object discovery tasks.
Table c)) shows the results on CLEVRTEX and CLEVRTEX- -ODD, which contains complex textured objects and backgrounds. Here, OC-Net again shows superior performance in all metrics, illustrating its capability to effectively segment complex objects. Although general decrease in performance observed across all methods in the CLEVRTEX-OOD test set, likely due to a change in data distribution, OC-Net's performance drop is slight and it still outperforms the closest baseline.
Finally, Table 2d) d) shows the results on Flowers, Birds and COCO common object datasets. Here, OC-Net again shows superior performance in all metrics, illustrating its capability to effectively segment commonly seen natural objects. Notably, OC-Net outperforms all other methods by large margin in COCO, demonstrating its robustness in handling objects with highly varied appearances.
Figure 2 visualizes the objects discovered by the various methods for sample images. OC-Net is able to Identify large and small objects in Multi-dSprites even when these objects are significantly occluded. Moreover, in the Tetrominoes-Nii dataset, despite the presence of shadow effects that often confuse existing methods, OC-Net still manages to separate each tile. For SVHN, only OC-Net is able to segment the character objects out in fineggrained manner. EfficientMORL tend to group all the characters together while the other methods segment the objects in a coarse-grained manner. For IDRiD, OC-Net is able segment out the optic disc and small lesions which other methods fail to discover. For CLEVRTEX and CLEVRTEX-OOS objects from the complex-textured backgrounds in a fine-grained manner. Finally, for Flowers, Birds, and COCO, only OC-Net is able to segment out the complex-shape and multi-part objects from the backgrounds in fine-grained manner.
OC-Net is able to segment out the various
4.2 Experiments on Sample Efficiency
One obstacle to unsupervised object discovery is the availability of a sufficiently large number of suitable training samples. Sample efficiency refers to a model's ability learn effectively from relatively small number of examples. Figure Bl shows the mloU scores as we decrease the number of training samples Multi-dSprites. SVHN and CLEVRTEX. OC-Net able to achieve near-optimal performance even with significantly smaller training set (1,000 samples) compared to all the other methods. The high sample efficiency of OC-Net reduces the need for large, potentially costly or dfficult-to-obtil datasets. This makes OC-Net more practical solution for real-world applications.

---

Table 2: Evaluation scores for the discovered foreground objects.
(a) Simulated datasets
Multi-dSprites
Tetrominoes-NM
Method
ARI
mDice
mloU
ARI
mDice
mloU
SLIC 67.911..0
78.510.00
70.810.00
53.0.10.0
66.110.00
53.610.00
Felzenszwalb
97.4+#.0
98.6=0.0
95.0++.0
95.0=0.0
98.0=0.0
96.9+0.0
Slot Attention
91.3=1..3
45.7=007
32.610.6
99.8
41.5=0.8
26.6200.7
EfficientMORL
85.2:1..5
30.1+1.3
19.5#1.1
99.0#1.7
42.5+2.3
27.6=t1.9
GENESIS-V
85.0++1.3
81.5:1.99
72.2+11.4
97.6++0.5
47.1=1.11
31.0++0.8
SLATE 89.
82.5++0.9
72.6+
84.5+
57.8++0.9
44.3+-0.8
SysBinder
72.3+112
30.6+1111
19.6+1.0
90.7+11.7
41.8+199
27.0++..7
BO-QSA
90.411111
91.1111
88.0112
99.3110.3
40.9+14
25.8111.2
OC-Net
99.811.00
99.510.0
99.1110.0
100.010.00
100.0:1.0
100.0
(b) Real-world datasets
SVHN
IDRiD
Method
ARI
mDice
mloU
ARI
mDice
mloU
SLIC
5.3+0..0
50.1
34.5=0.0
32.2+0.0
12.7+0.0
8.8+0.0
Felzenszwalb
31.7=0.0
51.6+0.0
39.8=0.0
14.7+0.0
19.0+0.0
15.4+0.0
Slot Attention
38.9=1.5
51.7+
36.7=1.7
28.7+1.1
8.6=1.7
5,0+1.6
EfficienMMRL.
32.2:11.7
49.2112.0
34.0.1..8
16.81:1.5
11.122.7
7.0
GENESIS-V2
28.6.11.4
60.8
45.9.11.4
18.3111.
8.8.1.99
5.4.1..6
SLATE
21.2:11.2
57.0.11.3
41.7-11.5
35.6:12.1
8.1
4.7411.8
SysBinder
15.8=116
1.9
34.1=188
25.22:13
16.6#1.7
11.1118
BO-QSA
24.3+112
62.0#1.6
48.3#1.3
27.742.0
7.0
1.5
OC-Net
39.7-0.1
64.6110.1
49.
39.010.4
38.1+0.2
31.2:10.2
(c) Complex textures dataset
CLEVRTEX
CLEVRTEX-OOD
Method
ART
mDiie
mloU
ART
mDice
mloU
SLIC
27.410.0
20,010.0
13.0=0.0
25.810.0
21.710.0
14.010.0
Felzenszwalb
57.310.0
33.6.10.0
26.8.1.0
44.6110.0
29.610.0
23.4:0.00
Slot Attention
58.6=1.6
35.0+1.6
26.7-1.5
51.3+1199
34.1#1.4
25.1+1..
EfficientMORL
59.5+1.77
37.7+1.5
31.1=1.4
53.9+2.5
32.2+2.4
25.3+228
GENESIS-V V2
65.6=1.8
36.9+1.4
30.4+1.4
67.6+1.6
34.2+1.5
27.6+1.9
SLATE
57.5
33.3.11.6
24.4.11.5
$6.6.11.3
34.7122.1
25.3111.
SysBinder
61.4
31.3.1155
23.1:11.4
61.0.12.4
32.3+12.0
23.8
BO-QSA
7..9-1.9
42.9111.8
34.711..7
66.1111.
42.8.1144
33.9
OC-Net
70.7-1.9
45.1
37.5=0.7
69.810.8
43.5:10.7
35.01106
(d) Common objects datasets
Flowers
Birds
COCO
Method
Dice
IoU
Dice
ToU
mDice
mIoU
SLIC
30.5+0.00
18.4+00.0
33.1++0.0
20.3++0.0
36.2++0.0
24.4++0.0
Felzenszwalb
43.7=0.0
30.410.0
34.3-1.0
23.0:10.0
36.610.0
27.110.0
Slot Attention
43.0=11.5
28.6:11.2
42.9-12.0
27.91118
24.8122.0
15.0
EfficientMORL
59.512211
45.2:12.2
44.0-11.9
30.811.8
28.41:233
18.912.1
GENESIS- V2
63.7=2.22
50,2+2.2
41.4+1.7
27.7+1.5
25.1+2.1
16.1+1.7
SLATE
55.6+1.2
40.8+1..8
39.5
25.9+
37.0+1.9
24.4+11.
SysBinder
45.0=1.8
30.8+1.6
33.7+11.
21.1+2.0
18.4+1.6
10.7+1.4
BO-QSA
65.8.11.9
$1.7119
44.6.11.7
30.3.11.5
34.911..
23.6-10.9
OC-Net
67.2.10.2
54.4.002
47.8.10.2
33,5.10.2
48.2.10.2
35.6-10.2

---

(a) Multi-dSprites
Inrage
Felzenenwath
GENESES-V2
SLATE
Sydlinder
B0-QSA
OC-Ner
(b) Tetrominoes-NM
Image
Ground Touth
Pelzenscwalb
Diat Atteedion
GINESIS-U -2
SLTTE
Sydtieder
B0-C24
5CC
O
[C
5CC
SVHN
efficietMeR:
(d) IDRiD
Image
SLATE
Systlinder
OC-Net
(e) CLEVRTEX
Image
Ground Truth
SLIC
Pelsenacwall
Attention
EthoantMOR:
GENESIS
SLATE
Systinder
80- Q4A
OC-Ne
(f) CLEVRTEX-OOD
Flowers
Image
Groand Truth
SLIC
Pelbensweeb
Skvi Attention
EfficietMMB
GENESES 1/2
SLATE
Systiinder
80 0SA
oc-Net
(h) Birds
Image
Ground Trush
SLUC
rébansovalb
Attention
GENESIS-UV
SLATE
Systinder
B0-95A
oc-Net
(i) COCO
Figure 2: Visualization of discovered objects.
4.3 Experiments on Model Generalizability
Ideally, an unsupervised object discovery model should trained understand common visual appearances such the difference between foreground vs background, to discover objects from

---

m1U
Slot Attention
EfficienMMORL
GENESIS-V2
SLATE
SysBinder
BO-OSA
DC-Net
55K
40X
20H
10C
50K
306
10K
8 Training Samples
Training Samples
I1 Training Sampless
(a) Multi-dSprites
(b) SVHN
(c) CLEVRTEX
Figure 3: mloU scores vs decreasing number of training samples.
Table mloU scores for model generalizability after training on Multi-dSprites.
Method
Tetrominoes-NM
SVHN
IDRiD
CLEVRTEX
CLEVRTEX-OOD
Slot Attention
21.833.5
19.5+3.8
7.5
12.2#2.2
12.342.2
EfficientMORL
21.2:13.8
23.4+2.8
6.5+2.66
12.7+322
15.2+2.0
GENESIS
42.9+-4.9
31.1+2.8
8.5+2.44
21.9+
21.3+22.5
SLATE
11,4++.6
21.1+2.0
10.0+1.7
12.7+22
12.9+
SysBinder
28.5+
23.8+1.11
13.9+1.8
10.6+1.5
11,6+1.9
BO-QSA
4188118
24.311..0
4.011..5
24.411.4
22.8
OC-Net
100.0110.0
47.5
29.1
31.710.6
31.3
out-of-domain images. In this set of experiments, we compare the generalization ability of OC-Net with the baselines by training the models Multi-dSprites and testing them the other datasets.
Table Shsowws the results. For Tetrominoes-NM. there is a decrease performance for all methods while OC-Net still obtains perfect score. For SVHN, CLEVRTEX and CLEVRTEX-OOD. the performance all models decrease due to the shift in data distribution. However, OC-Net experiences the smallest drop in performance and still significantly surpasses the best performing method. For IDRiD, the methods show improvement in performance. One possible reason is that the larger training set in Multi-dSprice enables the circular shape of the optic disc be better segmented. Despite this, OC-Net remains the top performer.
4.4
Ablation Studies
Next, we examine the effect of feature connectivity and regularization terms Cxep and Cent on the performance of OC-Net. We implemented three variants of OC-Net: (a) w/o connectivity. Here, we do not require that path should exist between i and & when computing dist i, k); (b) w/o Lent- This network is trained without the entanglement regularization term; (c) w/o Leep- The separation regularization term is not used in the training of OC-Net.
Table D shows the mIoU scores for all the datasets. Without feature connectivity, observe a drop in performance across datasets since is common for images to have multiple identical objects, that is, same color and shape. As such, OC-Net w/o connectivity tend to cluster these blocks single object.
Removing the entanglement term (OC-Net w/o Lene) leads to a slight decrease in the performance as object representations may still separated even when the dimensions are entangled. The largest performance drop in all datasets is seen when the object representation separation term is removed (OC-Net w/o Esep), indicating the importance of having well-separated object representation space for effective object discovery.

---

Table 4: mloU scores for variants of OC-Net.
Method
Multi-dSprites
Tetro-NM
SVHN
IDRiD
CTEX
CTEX-OOD
OC-Net
99.1
100.0:0.0
49.9
31.21:.2
37.5:1.77
35.010..6
w/o connectivity
98.8+0.1
89.2#0.1
36.620.1
17.3+0.1
20.3=0.9
17.7=0.99
wlo Cent
98.7+0.3
99.6+0.1
48.8+0.11
25.8++022
26.4+00.8
18.9+0.44
w/o Leep
49.3++0.3
51.2+0.2
35.0++0.2
4.0+0..1
18.5++0.6
11,0++0.4
5 Prediction based on Learned Object Representation
One characteristic of an effective object-centric representation is ability to encode object properties such as color, position and shape 122 In this section, we show that the learned object representations are disentangled and can be used to predict the values of these properties.
Given the representations and their corresponding ground truth values of a target object property. employ them features to train gradient boosted tree (GBT) [T6]. To evaluate how well the properties of unseen objects are predicted by the GBT, we use the coefficient of determination R² 4 We perform an experiment using the learned object representations from the simulated datasets to predict the properties of object such as color, position and shape. We use the mloU score to match the discovered object to the ground truth object. We split the 320 test images equally into two sets, one for training the GBT model for each object property, and the other for evaluation.
Table shows the average R scores of the GBT models the evaluation set. The GBT models trained with OC-Net representations achieved the highest I22 scores compared to the models trained using the representations from other methods. This suggests that the object representations learned by OC-Net are effective in encoding the object properties.
Table 5: R scores for object property prediction on simulated datasets
Multi-dSprites
Tetrominoes NM
Method
Color
Position
Shape
Color
Position
Shape
Slot Attention
72.2:112
96.8
38.21:0.0
86.5
98.710.6
36.31:.00
EfficientMORI.
86.5+6.2
95.8+-0.1
61.7++0.0
94.9+3322
97.9--0.7
68.5+0.0
GENESIS-VE
78.1+7.5
97.1++0.7
75.8+00.0
88.145.88
94.6+2.66
37.9+0.0
SLATE
87.5++0.7
90.6+4.44
31.7++0.0
85.5+3399
89.6+00.7
10.5+0.0
SysBinder
73.6.1..0
69.31334
33.3:1..0
97.910.6
77.8.2.77
19,910.0
BO-QSA
96.3.11.6
97.410.11
75.2.10.0
98.110.7
98.910022
52.511.0
OC-Net
98.0
98.310.11
78.1
100.0.10.0
99.4
98.710000
6 Conclusion
In this work, we have described a framework called OC-Net that learns object-centric representations in fine-grained manner without supervision. OC-Net leverages on feature connectivity and two new regularization terms to learn disentangled representations and to ensure the representations of different objects are well-sepel separated. From the results of experiments conducted on simulated. real-world, complex texture and common object images, we have demonstrated the superior quality of the object representations over current state-of-the-art, Moreover, we have highlighted the sample efficiency and generalizability of OC-Net. Finally, we have shown how the discovered object representations can used to predict object properties in downstream task, indicating its potential use for other computer vision applications where samples and ground truth labels are limited.
There still obstacles that have to be overcome for successful application of framework the full visual complexity of the real world. A natural next step would be to extend OC-Net to handle real-world scenes containing objects with more complex part-whole hierarchies. It also promising to explore explicit representation of the discovered objects into a dictionary of prototypes to better handle occlusion between objects. Lastly, real-world scenes with multiple objects is still of higher visual complexity than the datasets considered here and reliably bridging this gap is an open problem.
10

---

Acknowledgments and Disclosure of Funding
This research is supported by the National Research Foundation Singapore under its Al Singapore Programme (Award Number: AISG-GGC-2019-001-2AA)
References
[1] Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., and Süsstrunk, S. Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Puttern Analysis and Machine Intelligence, 34(1 (4(11))2274-2282, 2012. doi: 10. 10.1109/PPMM..202.11
[2] Ba, L.,, Kiros, J. R., and Hinton, G. E. Layer normalization. Advances in NIPS 2016 Deep Learning Symposium, 2016.
[3] Bardes, A., Ponce, J., and LeCun, Y. Vicreg: Yarianceiivvarinne-ccvaaas regularization for self-supervised learning. In The Tenth International Conference on Learning Representations, ICLR, 2022.
[4] Bear, D., Fan, C., Mrowca, D., Li, Y., Alter, S., Nayebi, A., Schwartz, J., Fei-Fei, L. F., Wu, J., Tenenbaum, J., et al. Learning physical graph representations from visual scenes. Advances in Neural Information Processing Systems, 33:6027-6039 2020.
[5] Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M. M., and Lerchner, A. Monet: Unsupervised scene decomposition and representation. CoRR, 1901. 11390, 2019.
[6] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229, Springer, 2020.
[7] Caron, M., Touvron, H., Misra, I,, Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in sele-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp.
2021.
[8] Crawford, and Pineau, J. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3412-3420, 2019.
[9] Dittadi, A., Papa. S. S., Vita, M. D., Schölkopf, B., Winther, O., and Locatello, F. Generalization and robustness implications object-centric learning. International Conference on Machine Learning, ICML, 2022.
[10] Eastwood, C. and Williams, C. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
[11] Emami, P., He, P., Ranka, S., and Rangarajan, A. Efficient iterative amortized inference for learning symmetric and disentangled multi-object representations. In International Conference on Machine Learning, pp. 2970-2981 2021.
[12] Engeleke, M., Kosiorek, A. R., Jones, O. P., and Posner, I. GENESIS: generative scene inference and sampling with object-centric latent representations. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
[13] Engelcke, M., Parker Jones, O., and Posner, Genesis-v2: Inferring unordered object represen- tations without iterative refinement. Advances Neural Information Processing Systems, 34: 8094, 2021.
[14] Eslami, S., Heess, N., Weber, T., Tassa, Y.. Szepesvari, D., Hinton, G. E.., et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, 29, 2016.
[15] Felzenszwalb, and Huttenlocher, P. Efficient graph-based image segmentation. Interna- tional journal of computer vision, 1972)1167-181, 2004.

---

[16] Friedman, J. H. Greedy function approximation: gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
[17] Greff, K., Kaufman, R. L., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L.,, Botvinick, M., and Lerchner, A. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2424-2433, 2019.
[18] Greff, K., Van Steenkiste, S., and Schmidhuber. J. On the binding problem in artificial neural networks. arXiv preprini arXiv:2012.05200, 2020.
[19] He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
[20] Hubert, L. and Arabie, P. Comparing partitions. Journal of classification, 2(1):193-213; 1985.
[21] Jaderberg, M., Simonyan, K., Zisserman, A., et al. Spatial transformer networks, Advances in neural information processing systems, 28, 2015.
[22] Jaa, B., Liu, Y., and Huang, S. Improving object-centric learning with query optimization. In The Eleventh International Conference on Learning Representations, 2023.
[23] Kabra, R., Burgess, C., Matthey, L., Kaufman, R. L., Greff, K., Reynolds, M., and Lerchner, A.
Multi-object datasets.
2019.
[24] Kahneman, D., Treisman, A., and Gibbs, B. J. The reviewing of object files: Object-specific integration of information. Cognitive psychology, 24(2):175-219, 1992.
[25] Karazija, L., Laina, I., and Rupprecht, C. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. In Proceedings of the Neural Information Processing Systems, 2021.
[26] Karazija, L.,, Laina, I., and Rupprecht, C. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. In Vanschoren, and Yeung. S. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Curran, 2021.
[27] Kim, J., Choi, J., Choi, H.-J., and Kim, S. Shepherding slots objects: Towards stable and robust object-centric learning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19198-192077 2023.
[28] Kingma Diederik, P. and Adam, J. B. A method for stochastic optimization. International Conference Learning Representations ICLR, 2015.
[29] Kipf, T., Elsayed, F., Mahendran, A., Stone, A., Sabour, S., Heigold, G., Jonschkowski, R., Dosovitskiy, A., and Greff, K. Conditional Object-Centric Learning from Video. In International Conference on Learning Representations (ICI.), 2022.
[30] Kipf, T. N.. van der Pol, E... and Welling, M. Contrastive learning of structured world models. In 8th International Conference on Learning Representations, ICLR, 2020.
[31] Kosiorek, A., Kim, H., Teh, Y. W., and Posner, I. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systemx, 31, 2018.
[32] Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D.,, Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
[33] Lin, Z., Wu, Y., Peri, S. V., Sun, W., Singh, G., Deng, F., Jiang, J., and Ahn, S. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
[34] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference machine learning, pp. 4114-4124, PMLR, 2019.
12

---

[35] Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:11525-11534 2020.
[36] Netzer, Y., Wang. T., Coates, A., Bissacco, A.. Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
[37] Nilsback, M.-E. and Zisserman, A. Delving deeper into the whorl of flower segmentation. Image and Vision Compuling, 28(6): 1049-1062, 2010.
[38] Palmer, S. and Rock, I. Rethinking perceptual organization: The role of uniform connectedness. Psychonomic bulletin & review, 1(1):29-55, 1994.
[39] Paszke, A. a. Pytorch: An imperative style, high- performeree deep learning library. Advances in Neural Information Processing Systems, 2019.
[40] Porwal, P., Pachade. S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., and Meriaudeau, Indian diabetic retinopathy image dataset (idrid), 2018.
[41] Rezende, D. J. and Viola, F. Taming vaes. ArXiv, abs/1810.005997, 2018.
[42] Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-634, 2021. doi: 10.1 1A.10099/ROC..221..3300
[43] Shi, J. and Malik, J. Normalized cuts and image segmentation. IEEE Transactions pattern analysis and machine intelligence, 22(8):888-90) 900, 2000.
[44] Shwartz-Ziv, R., Balestriero, R. Kawaguchi, K., Rudner, T. G., and LeCun, regularization. arXiv
informatiin--heerrttee perspective preprint arXiv: 2023.
iettane-invariance-coo
[45] Singh, G., Deng, F,, and Ahn, S. Illiterate dall-e learns to compose. In International Conference on Learning Representations, 2021.
[46] Singh, G., Kim, Y., and Ahn, S. Neural systematic binder. In The Eleventh International Conference on Learning Representations, 2023.
[47] Wang, X., Girdhar, R., Yu, S. X., and Misra, I. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3124-3134, 2023.
[48] Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie. S., and Perona, P. Caltech- ucsd birds 200. 2010.
[49] Wright, S. Correlation and causation. Journal of agricultural research, 1921.
[50] Yang, Y. and Yang, B. Promising or elusive? unsupervised object segmentation from real-world single images. Advances Neural Information Processing Systems, 35:4722-4733 2022.
[51] Yuan, J., Chen, T., Li, B., and Xue, Compositional scene representation learning via reconstruction: A survey, arXiv preprint arXiv:2202.071355 2022.
13


Reviews:
[{"review_id": "pyXCa3ElWD", "reviewer": "Reviewer_TJh8", "summary": "The paper discusses a novel method for learning object-centric representations from images, the new approach uses feature connectivity to group together pixels likely to be part of the same object and employs two object-centric regularization terms for refining the representations. \n\nThe method has been tested on various types of images and has significant improvements in object discovery quality, sample efficiency, and generalizability compared to prior work. It has the potential to improve many downstream tasks in predicting object properties. But it also has limitations to handle objects with complex part-whole hierarchies and object occlusions.", "questions": "How does the proposed object entanglement compare to the attention mechanism?", "limitations": "The authors mentioned the method has limitations to handle objects with complex part-whole hierarchies and object occlusions. It would still be interesting to see how the method works on more realistic images, such as a picture of a complex kitchen scene, with part-whole complex cabinetry and a human holding objects and having big object occlusions.", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Contributed a novel method for learning object-centric representations from images, the Object Discovery process and Object-Centric Regularization are carefully designed for object-centric learning.\n\n2. Comprehensive mathematical proof and reasoning for the proposed embeddings and regularization terms.\n\n3. Comprehensive comparison between the proposed methods and prior work, and on various datasets.\n\n4. Promising results on sample efficiency, accuracy, and generalizability compared to prior work.\n\n5. Paper writing is clear and succinct.", "weaknesses": "1. Lack of example on incorporating the method to an actual downstream task, and showing the advantage of the method using the proposed terms.\n\n2. Lack of results on more challenging datasets with complex scenes, object types, object shapes, and object occlusions.\n\n3. Designed for object-centric, but as mentioned in the limitations, it has limitations for objects with complex part-whole hierarchies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and for the encouraging comments. We provide some responses to the questions as follows:\n\n**Q1: Lack of results on more challenging datasets with complex scenes, object types, object shapes, and object occlusions.**\n\nWe have conducted additional experiments on the Birds, COCO and Flowers datasets with more complex scenes as suggested. We have also added two more recent baselines BO-QSA [21] and SLASH [1a] for comparison. The table below shows the results:\n\n|  | Flowers | Flowers | Birds | Birds | COCO | COCO |\n|---|---|---|---|---|---|---|\n| Metric | Dice | IoU | Dice| IoU | mDice| mIoU |\n| SLASH | 39.91 $\\pm$1.48 | 25.98 $\\pm$1.15 | 43.45$\\pm$1.10 | 28.24$\\pm$1.08 | 22.34$\\pm$1.10 | 13.37$\\pm$1.07 | \n| BO-QSA | 65.77 $\\pm$1.90 | 51.70 $\\pm$1.93 | 44.61$\\pm$1.65 | 30.29$\\pm$1.50 | 34.90$\\pm$1.14 | 23.57$\\pm$0.94 | \n| OC-Net | **67.21 $\\pm$0.21** | **54.42 $\\pm$0.23** | **47.80$\\pm$0.19** | **33.50$\\pm$0.17** | **48.18$\\pm$0.17** | **35.58$\\pm$0.15** | \n\nThe objects discovered for the various methods on sample images can be viewed in the PDF attached in our general response to all reviewers. \n\n**Q2: The authors mentioned the method has limitations to handle objects with complex part-whole hierarchies and object occlusions. It would still be interesting to see how the method works on more realistic images, such as a picture of a complex kitchen scene, with part-whole complex cabinetry and a human holding objects and having big object occlusions.**\n\nWe will include some of these samples in the camera ready paper.\n\n**Q3: How does the proposed object entanglement compare to the attention mechanism?**\n\nThe Slot Attention baseline [1b] used in the paper relies on an adapted version of the attention mechanism to perform object-centric learning. This attention mechanism does not consider feature connectivity whereas our method does, enabling better fine-grained object discovery and sample efficiency.\n\n[1a] Kim, Jinwoo, et al., \"Shepherding Slots to Objects: Towards Stable and Robust Object-Centric Learning\", CVPR 2023.\n\n[1b] Locatello, Francesco, et al., \"Object-Centric Learning with Slot Attention\", NeurIPS 2020.\n"}, {"review_id": "78s7fhmtaS", "reviewer": "Reviewer_GCAa", "summary": "The paper presents a method for learning pixel representations for multi-object segmentation. The method works by projecting pixels using $1 \\times 1$ convolution into a feature space. The graph is defined on top of pixels using 8-connectivity. A modified Dijkstra's algorithm assigns pixels to objects based on a distance threshold, where the distance between pixels is measured as the Euclidean distance between normalised feature embeddings. Object representation is formed by averaging the pixel embeddings and adding a learnable mask projection. Two losses are used to learn the pixel embeddings, which aim to increase the diagonal and minimise the off-diagonal entries of the object-feature covariance matrix. The method is evaluated on a series of 2D sprite datasets, SVHN and IDRiD datasets and CLEVRTEX, where the method shows strong segmentation performance.", "questions": "\nIs it correct to say that the LayerNorm is applied _without_ learned affine transformation as outputs of this layer (or functions thereof) are not included in the loss calculation? If it is, how is it trained as the membership of pixel to object set does not appear to be differentiable? \n\nHow is the Figure 2 visualisation obtained? It seems that the object discovery procedure requires pixels to be connected, however, there are several \"floating\" pixels coloured the same colour as the objects.\n\nOn L161, $\\epsilon$ is described as being set such that the normalised similarity should be 50%. How are the similarities normalised?\n", "limitations": "\nThere seems to be an unstated assumption that objects need to be up to a certain size. I.e. the threshold implicitly defines a maximum distance between the sampled initial pixel and the furthest pixel that can be included. For example, considering an image of solid color, the resulting pixels should have the same features (close to zero due to LayerNorm) save for the added positional embedding. Thus, only pixels $\\epsilon$ from the starting pixel will be added to the object set $\\mathcal{O}_c$.\n\nIt also seems that the assignment of the pixels to a particular object might be heavily dependent on the sampled initial pixel. That is, a pixel j might be included if the initial sampled pixel i is close to it, but not if it is further away as it cannot satisfy the condition on Line 21 of Algorithm 1 due to too many hops. It will then become a part of a different object. Or conversely, pixels of two different objects from a boundary would be included in the same object if one was sampled as initial pixels. Training would further enforce this. Considering the situation above, if the initial sampled pixel was in the corner, would this not potentially create several components?\n\nSimilarly, could it be that the model cannot handle an object split into two regions by another object, such as e.g. a dog behind a fence post?\n", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "\nThe presented formulation is quite simple. This leads to short training time and high-sample efficiency.\n\nThe method shows strong performance on benchmarks to the point of saturating or \"solving\" simpler cases.\n\nThe writing is clear.\n", "weaknesses": "\nThe description of the method is lacking critical information and details. \n - In particular, it is not clear how final masks are obtained. Description of the method in Algorithm 1 suggests that the object sets $\\mathcal{O}_c$ are not disjoint and may contain the same pixels as other objects (a consequence of revisiting all pixels starting from the empty set (Lines 11 and 24 of the algorithm)). How are the masks obtained in that case, e.g. for evaluation?\n - It is also not clear why the object discovery procedure converges, as it relies upon having learned an already appropriate pixel embedding to always guarantee that the condition in line 21 of the algorithm can be satisfied. What happens if that is not the case?\n\nAs there is no information exchange between pixels in the same object, it is not clear whether semantic or instance information can be learned and extracted. This is somewhat evidenced by the model not performing well on Tetrominoes in the case where the model is required to learn object shapes. As evaluation is limited to simpler or simulated datasets, it is not clear whether this would scale to complex real-world data. Given performance on simpler datasets, it would interesting to show the method working on Birds, COCO, and Flowers in comparison to prior work [21, A].\n\nThere are related works that explore image segmentation by modelling a graph on top of learned or hand-crafted features [B, C, D]. It would be beneficial to discuss and compare to such works, e.g. MaskCut method from [C].\n\nComparison to prior work is limited to older methods, whereas newer methods are not compared against. In particular, a comparison to [21, A, E] is needed given the strong performance on some of the same datasets. Furthermore, methods that were previously shown to obtain good results on CLEVRTEX are not included [E, F, G] in comparisons. Additionally, the object property prediction is limited to simple, nearly monochrome datasets whereas prior work [41] have considered more complex cases in CLEVRTEX already. \n\n\n### References\n\n[A] Seitzer et al. \"Bridging the Gap to Real-World Object-Centric Learning\"\n\n[B] Wang et al. \"Self-supervised Transformers for Unsupervised Object Discovery using Normalized Cut\"\n\n[C] Want et al. \"Cut and learn for unsupervised object detection and instance segmentation\"\n\n[D] Shi and Malik \"Normalized cuts and image segmentation\"\n\n[E] Sauvalle and Fortelle \"Unsupervised multi-object segmentation using attention and soft-argmax\"\n\n[F] Jiang and Ahn \"Generative Neurosymbolic Machines\"\n\n[G] Monnier et al. \"Unsupervised layered image decomposition into object prototypes\"\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and for the extensive pointers to related works. We provide some responses as follows:\n\n**Q1: Algorithm 1 suggests that the object sets are not disjoint and may contain the same pixels as other objects. How are the masks obtained in that case, e.g. for evaluation?**\n\nIf a pixel is assigned to multiple objects, we assign it to the mask of the first object in that list and ignore its membership in other objects.\n\n**Q2: It is not clear why the object discovery procedure converges, as it relies upon having learned an already appropriate pixel embedding to always guarantee that the condition in line 21 of the algorithm can be satisfied. What happens if that is not the case?**\n\nThere is always at least one pixel (the starting pixel $\\mathbf{p}_i$) that satisfies the condition in line 21, thus guaranteeing that the algorithm will terminate.\n\n**Q3: As there is no information exchange between pixels in the same object, it is not clear whether semantic information can be learned and extracted.**\n\nRelational information between pixels in the same object are included in object representations in the form of positional encodings and the processed object mask (Eqn. 2). This enables our model to outperform other baselines in downstream property prediction, demonstrating that semantic information can be extracted.\n\n**Q4: Given performance on simpler datasets, it would interesting to show the method working on Birds, COCO, and Flowers**\n\nWe have conducted additional experiments for further evaluation on Flowers, Birds, COCO as suggested. We have also added two more recent baselines BO-QSA [21] and SLASH [5b] for comparison. The results in the table below shows that OC-Net is robust and remains the top performer:\n\n|  | Flowers | Flowers | Birds | Birds | COCO | COCO |\n|---|---|---|---|---|---|---|\n| Metric | Dice | IoU | Dice| IoU | mDice| mIoU |\n| SLASH | 39.9$\\pm$1.5 | 26.0$\\pm$1.2 | 43.5$\\pm$1.1 | 28.2$\\pm$1.1 | 22.3$\\pm$1.1 | 13.4$\\pm$1.1 | \n| BO-QSA | 65.8$\\pm$1.9 | 51.7$\\pm$1.9 | 44.6$\\pm$1.7 | 30.3$\\pm$1.5 | 34.9$\\pm$1.1 | 23.6$\\pm$0.9 | \n| OC-Net | **67.2$\\pm$0.2** | **54.4$\\pm$0.2** | **47.8$\\pm$0.2** | **33.5$\\pm$0.2** | **48.2$\\pm$0.2** | **35.6$\\pm$0.2** | \n\nThe objects discovered for the various methods on sample images can be viewed in the PDF attached in our general response to all reviewers. We see that OC-Net is able to segment out the objects in a fine-grained manner.\n\n**Q5: There are related works that explore image segmentation by modelling a graph on top of learned or hand-crafted features [B, C, D]. It would be beneficial to discuss and compare to such works.**\n\nDINOSAUR [A], TokenCut [B], MaskCut [C] belong to a class of methods that rely on the DINO model [2a] which was pre-trained on the ImageNet dataset. For fair comparison, we excluded methods that rely on pre-trained models and trained all methods from random initialization. Nonetheless, we will include a section to compare these works in the updated paper as suggested.\n\nWe have carried out additional experiments for comparison with Ncut [D] as suggested. The mIoU results in our general response (3) show that OC-Net outperforms other graph-based methods.\n\n**Q6: A comparison to [21,A,E] is needed given the strong performance on some of the same datasets. Furthermore, methods that were previously shown to obtain good results on CLEVRTEX are not included [E,F,G] in comparisons.**\n\nWe conduct additional comparison with most recently published BO-QSA [21] which attain scores that outperform or are comparable with AST-Seg [E], GNM [F], DTI [G]. The mIoU results in our general response (4) shows that OC-Net is superior.\n\n**Q7: The object property prediction is limited to simple, nearly monochrome datasets whereas prior work have considered more complex cases in CLEVRTEX already.**\n\nWe conducted additional experiments to further evaluate the object property prediction on CLEVRTEX as suggested. Results are in our general response (2).\n\n**Q8: Is it correct to say that the LayerNorm is applied without learned affine transformation as outputs of this layer (or functions thereof) are not included in the loss calculation?**\n\nYes, LayerNorm is applied without affine transformation. Membership of pixel to object set is used to control the flow of gradients such that gradients only flow towards the pixel embeddings which belong to each object. \n\n**Q9: It seems that the object discovery procedure requires pixels to be connected, however, there are several \"floating\" pixels coloured the same colour as the objects.**\n\nAs clarified in Q1, pixels may be assigned to multiple objects and during evaluation, we assign such pixels to the mask of the first object. This may result in some floating pixels. \n\n**Q10: $\\epsilon$ is described as being set such that the normalised similarity should be 50\\%. How are the similarities normalised?**\n\nWe normalise all similarity values to be between $0$ and $1$ by taking the negative exponent. Details have been provided in Appendix A.3.\n\n**Q11: There seems to be an unstated assumption that objects need to be up to a certain size. It also seems that the assignment of the pixels to a particular object might be heavily dependent on the sampled initial pixel.**\n\nObjects can include any pixel as long as there exists a shortest path distance that is less than $\\epsilon$ from that pixel to the sampled pixel. We have found that our trained models are robust to size and sampling sequence as demonstrated by $\\epsilon$ being able to take a wide range of values without affecting performance (see Appendix A.3).\n\n**Q12: Could it be that the model cannot handle an object split into two regions by another object?**\n\nYes, as stated in the limitations on occlusion, if an object is split into two disjoint regions by another object, the model would consider them as separate objects and output three object masks.\n\n[2a] Caron et al. \"Emerging properties in self-supervised vision transformers\", ICCV 2021."}, {"review_id": "5dXb8VddM4", "reviewer": "Reviewer_T4Hn", "summary": "The paper presents a novel approach to object-centric (multi-object) representation learning. In contrast to conventional sequential or iterative attention-based methods, the proposed method employs a pixel-wise shortest distance-based clustering technique. Moreover, the paper emphasizes the use of regularization-based representation learning as an alternative to traditional reconstruction-based approaches. The inclusion of theoretical proofs in the appendix, combined with extensive empirical studies, serves to substantiate the paper's claims and highlight the superior and robust performance achieved by the proposed methodology.", "questions": "1. In terms of the training schedule, it is necessary to provide a clear description regarding any distinction between \"steps\" (L157) and \"iterations\" (L165). This will help readers better understand the experimental details and avoid any potential confusion.\n1. In the figures depicting qualitative results (Figure 2, 6-13), there seems to be a discrepancy in how the authors classify backgrounds and visualize them as black. While the majority of backgrounds are depicted as black, there are cases where backgrounds are shown in different colors. This raises a question about whether there is any meaningful distinction between black-colored backgrounds and those with different colors. Clarification is needed to understand the criteria used for classifying and representing backgrounds in the visualizations.\n1. The paper utilizes a simplistic approach for positional encoding, which differs from other possible options such as the soft positional embedding used in Slot Attention [1] or fixed sinusoidal positional encoding. Since the chosen method deviates from the conventional approaches, it is important to provide a clear explanation regarding the design and rationale behind the positional encoding used in the paper. This will help readers understand the unique choices made and the implications of the selected approach for the overall methodology.\n\n\n[1] Locatello, Francesco, et al., \"Object-Centric Learning with Slot Attention\", NeurIPS 2020.", "limitations": "1. To facilitate a comprehensive and fair comparison, it is crucial to include an analysis of the model's speed in the paper. Given that the proposed method involves iterative updates of pixel distances, which may pose challenges for parallelization, there could potentially be a significant speed sacrifice compared to other methods. However, the paper currently lacks any analysis or discussion regarding the speed performance of the proposed method. \n1. To enhance the persuasiveness of the paper, it would be beneficial to include some more experiments, such as a) recent studies such as ISA [1] and SLASH [2] for the object discovery task, and b) property prediction results over the complex datasets such as CLEVRTEX, in the final version.\n1. The explanation regarding the mask transformation matrix (denoted as A) lacks clarity. It is crucial to provide additional details regarding the initialization and training of the matrix, going beyond the information provided in section A.4. Furthermore, the purpose and necessity of the matrix should be clearly articulated, supported by a thorough description of the procedural intricacies and empirical studies demonstrating its effectiveness.\n\n\n[1] Biza, Ondrej, et al., \"Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames\", ICML 2023.\n\n[2] Kim, Jinwoo, et al., \"Shepherding Slots to Objects: Towards Stable and Robust Object-Centric Learning\", CVPR 2023.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written with clear structures, logical flow, and informative figures that effectively convey the research findings.\n1. The proposed method introduces novel concepts, such as feature connectivity and regularization-based training losses, which lead to impressive results, not only just high score but also in terms of the sample efficiency, on six diverse datasets. These datasets encompass synthetic (simulated), real-world, and complex textures, and the evaluation covers two fundamental tasks in object-centric learning: object discovery and property prediction.\n1. The paper offers theoretical support through a mathematical proof presented in the appendix. This proof establishes an upper bound for the downstream generalization error, enhancing our understanding of the proposed method's performance and its capacity to generalize effectively in downstream tasks.", "weaknesses": "1. The primary concern raised by the reviewer revolves around the scalability of the proposed method to other datasets.\n   - The reviewer questions the difference between using the original RGB input image (with proper normalization) and the feature map generated by the single 1x1 convolution layer where the single 1x1 convolution layer will act as an injective function from a pixel to a representation vector. It is important to address this aspect and provide a clear distinction between the two approaches.\n   - The minimal receptive field (1x1) of the encoder (a single-layer 1x1 convolution) limits the ability to consider local or global context during clustering. This issue is not adequately resolved by the empirical studies since the datasets used lack objects with diverse colors or shapes.\n   - The reviewer acknowledges that the positional encoding helps to compensate for this limitation. However, there is a concern about the naive utilization of positional encoding and its ability to capture complex object shapes. Additionally, there is a worry that the resulting clusters might primarily focus on object colors without incorporating positional information.\n   - In contrast to the reviewer's concerns, the authors claim in section G.1 that the 1x1 convolution contributes to fine-grained object-centric learning. This conflict needs to be addressed and supported by the authors' analysis and empirical evidence.\n   - The reviewer suggests conducting empirical studies on additional datasets with more complex object shapes and color combinations, such as PTR [1], MSN [2], and MOVi [3], as demonstrated in previous works [4,5].\n   - Additionally, considering the results obtained by other encoders used in previous methods, such as Slot Attention [6], would further support the authors' claims.\n\n\n2. Another concern pertains to the rigidity of the feature aggregation process due to the use of a hard clustering algorithm. This rigidity may hinder the proposed method's ability to perform well on downstream tasks and more complex real-world datasets.\n\n[1] Hong, Yining, et al., \"PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning\", NeurIPS 2021. \n\n[2] Stelzner, Karl, et al., \"Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation\", ICLR 2022.\n\n[3] Greff, Klaus, et al., \"Kubric: A Scalable Dataset Generator\", CVPR 2022.\n\n[4] Biza, Ondrej, et al., \"Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames\", ICML 2023.\n\n[5] Kim, Jinwoo, et al., \"Shepherding Slots to Objects: Towards Stable and Robust Object-Centric Learning\", CVPR 2023.\n\n[6] Locatello, Francesco, et al., \"Object-Centric Learning with Slot Attention\", NeurIPS 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and for the insightful comments especially on scalability. We provide some responses to the questions as follows:\n\n**Q1: The reviewer questions the difference between using the original RGB input image (with proper normalization) and the feature map generated by the single 1x1 convolution layer**\n\nUsing RGB values would treat each image on its own, while applying a convolutional layer enables the method to learn appropriate feature maps based on a dataset of images in order to optimize for downstream generalization.\n\n**Q2: The minimal receptive field of the encoder limits the ability to consider local or global context during clustering. This issue is not adequately resolved by the empirical studies since the datasets used lack objects with diverse colors or shapes. The reviewer suggests conducting empirical studies on additional datasets with more complex object shapes and color combinations such as PTR, MSN, and MOVi, as demonstrated in previous works.**\n\nWe have conducted new experiments to further evaluation on additional Flowers, Birds, COCO [3a] datasets which contain objects that are more complex or comparable to PTR, MSN and MOVi. We have also added two more recent baselines BO-QSA [21] and SLASH for comparison as suggested. The results in the table below shows that OC-Net is robust and remains the top performer:\n\n|  | Flowers | Flowers | Birds | Birds | COCO | COCO |\n|---|---|---|---|---|---|---|\n| Metric | Dice | IoU | Dice| IoU | mDice| mIoU |\n| SLASH | 39.91 $\\pm$1.48 | 25.98 $\\pm$1.15 | 43.45$\\pm$1.10 | 28.24$\\pm$1.08 | 22.34$\\pm$1.10 | 13.37$\\pm$1.07 | \n| BO-QSA | 65.77 $\\pm$1.90 | 51.70 $\\pm$1.93 | 44.61$\\pm$1.65 | 30.29$\\pm$1.50 | 34.90$\\pm$1.14 | 23.57$\\pm$0.94 | \n| OC-Net | **67.21 $\\pm$0.21** | **54.42 $\\pm$0.23** | **47.80$\\pm$0.19** | **33.50$\\pm$0.17** | **48.18$\\pm$0.17** | **35.58$\\pm$0.15** | \n\nThe objects discovered for the various methods on sample images can be viewed in the PDF attached in our general response to all reviewers. We see that OC-Net is able to segment out the objects in a fine-grained manner.\n\n**Q3: Another concern pertains to the rigidity of the feature aggregation process due to the use of a hard clustering algorithm. This rigidity may hinder the proposed method's ability to perform well on downstream tasks and more complex real-world datasets.**\n\nOur proposed method can be modified to use soft clustering by simply changing the inclusion criteria (Algorithm 1, line 21) and the terminating condition of the object discovery process (Algorithm 1, line 26). \n\n**Q4: In terms of the training schedule, it is necessary to provide a clear description regarding any distinction between \"steps\" and \"iterations\".**\n\nWe will change \"steps\" to \"iterations\" to avoid confusion.\n\n**Q5: In the figures depicting qualitative results (Figure 2, 6-13), there seems to be a discrepancy in how the authors classify backgrounds and visualize them as black.**\n\nFollowing past work such as Slot Attention [3b], we assign the best-matching regions produced by the various methods as the same color as the objects in the ground truths. Our method does not separately model backgrounds and treats all background of different appearances the same. \n\n**Q6: The paper utilizes a simplistic approach for positional encoding, which differs from other possible options such as the soft positional embedding used in Slot Attention or fixed sinusoidal positional encoding.**\n\nWe found that using linear positional encodings help attain better performance especially for downstream property prediction. We have conducted additional experiments to compare alternative positional encodings as suggested and show the results on Multi-dSprites below: \n\n| Property | Color | Position | Shape |\n|---|---|---|---|\n| Sinusoidal | 97.8$\\pm$0.8 | 75.2$\\pm$4.8 | 43.7$\\pm$0.0 | \n| Linear (Ours) | **98.0$\\pm$0.6** | **98.3$\\pm$0.1** | **78.1$\\pm$0.0** | \n\n**Q7: To facilitate a comprehensive and fair comparison, it is crucial to include an analysis of the model's speed in the paper.**\n\nWe implemented a novel batch-wise parallel version of Dijkstra's algorithm to ensure fast training and evaluation of our model. The code has been provided in the supplementary material. We will include an analysis of the model's speed in comparison to all other methods in the camera ready paper. \n\n**Q8: To enhance the persuasiveness of the paper, it would be beneficial to include some more experiments, such as a) recent studies such as ISA [1] and SLASH [2] for the object discovery task, and b) property prediction results over the complex datasets such as CLEVRTEX, in the final version.**\n\nCode for ISA is currently unavailable, hence we refer to the table in Q2 for additional experiments with SLASH for object discovery on new datasets. We also show the updated results of property prediction with CLEVRTEX below:\n\n| Property | Position | Shape |\n|---|---|---|\n| Slot Attention | 47.5$\\pm$19.6 | 30.6 |\n| EfficientMORL | 21.8$\\pm$2.0 | 18.5 | \n| GENESIS-V2 | 79.8$\\pm$8.0 | 35.2 | \n| SLATE | 62.0$\\pm$8.0 | 30.5 | \n| SysBinder | 38.2$\\pm$3.1 | 29.6 |\n| BO-QSA | 66.5$\\pm$0.3 | 28.7 | \n| OC-Net | **80.7$\\pm$2.4** | **36.1** |\n\n**Q9: The explanation regarding the mask transformation matrix (denoted as A) lacks clarity. It is crucial to provide additional details regarding the initialization and training of the matrix, going beyond the information provided in section A.4.**\n\nWe will include additional description and further studies for the mask transformation matrix in the camera ready paper.\n\n[3a] Yang, Yafei, and Bo Yang. \"Promising or elusive? unsupervised object segmentation from real-world single images.\", NeurIPS 2022.\n\n[3b] Locatello, Francesco, et al., \"Object-Centric Learning with Slot Attention\", NeurIPS 2020.\n"}, {"review_id": "C6yurIyl6A", "reviewer": "Reviewer_2W4b", "summary": "This paper presents a novel approach towards an unsupervised object-centric representation learning framework. Diverging from previous methodologies that prioritize an input reconstruction scheme, this method emphasizes feature connectivity to cluster neighboring pixels, employing two object-centric regularization losses. The underlying insight for these two losses is the learning of a distinct feature of each object while repelling the object feature between objects by constraining the covariance matrix of the predicted label. To validate their approach, the authors execute their experiments on a variety of image types, ranging from simulated and real-world images to those with complex textures. The results reveal a marked performance improvement when compared to alternative methods, demonstrating the efficacy of the proposed framework.", "questions": "1. Beyond the L2 distance employed for feature embeddings, exploring the impact of other choices for feature embeddings on model performance and threshold selection could be beneficial. Various methods such as cosine similarity or Mahalanobis distance, among others, could potentially impact the model performance and threshold choice differently.\n\n2. The mask transformation matrix A's initialization strategy is an area that requires further investigation. An exploration into whether a more intricate network structure could enhance the performance would provide more insight into the system's capabilities and potential limitations. This exploration could open up new possibilities for performance optimization.", "limitations": "See weaknesses and questions.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The innovative approach proposed in this paper provides food for thought. It suggests that the customary input reconstruction paradigm utilized in prior methods can be substituted with the two object-centric regularization terms introduced in this study. The basic concept underpinning these two loss functions is simple, yet it proves to be highly efficacious.\n\n2. The experimental outcomes presented in the paper are encouraging. The proposed framework demonstrates significant improvement when compared with baseline methods. Moreover, the authors effectively showcase the applicability of the proposed design even when the scale of the dataset diminishes. An ablation study further underscores the relevance and importance of the proposed design in this context.", "weaknesses": "1. The scalability of this method to real-world scenes of higher complexity remains a point of ambiguity. Further testing and discussion are necessary to ascertain the effectiveness of this approach in more complex, real-world scenarios.\n2. Here is one typo: Line 113, to be.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and for the encouraging comments. We provide some responses to the questions as follows:\n\n**Q1: The scalability of this method to real-world scenes of higher complexity remains a point of ambiguity. Further testing and discussion are necessary to ascertain the effectiveness of this approach in more complex, real-world scenarios.**\n\nWe have conducted new experiments for further evaluation on additional Flowers, Birds, COCO [4a] datasets which contain objects that have more diverse colors and shapes. We have also added two more recent baselines BO-QSA [21] and SLASH [4b] for comparison as suggested. The results in the table below shows that OC-Net is robust and remains the top performer:\n\n|  | Flowers | Flowers | Birds | Birds | COCO | COCO |\n|---|---|---|---|---|---|---|\n| Metric | Dice | IoU | Dice| IoU | mDice| mIoU |\n| SLASH | 39.91 $\\pm$1.48 | 25.98 $\\pm$1.15 | 43.45$\\pm$1.10 | 28.24$\\pm$1.08 | 22.34$\\pm$1.10 | 13.37$\\pm$1.07 | \n| BO-QSA | 65.77 $\\pm$1.90 | 51.70 $\\pm$1.93 | 44.61$\\pm$1.65 | 30.29$\\pm$1.50 | 34.90$\\pm$1.14 | 23.57$\\pm$0.94 | \n| OC-Net | **67.21 $\\pm$0.21** | **54.42 $\\pm$0.23** | **47.80$\\pm$0.19** | **33.50$\\pm$0.17** | **48.18$\\pm$0.17** | **35.58$\\pm$0.15** | \n\nThe objects discovered for the various methods on sample images can be viewed in the PDF attached in our general response to all reviewers. We see that OC-Net is able to segment out the objects in a fine-grained manner.\n\n**Q2: Beyond the L2 distance employed for feature embeddings, exploring the impact of other choices for feature embeddings on model performance and threshold selection could be beneficial. Various methods such as cosine similarity or Mahalanobis distance, among others, could potentially impact the model performance and threshold choice differently.**\n\nWe will include these comparisons in the camera ready paper.\n\n**Q3: The mask transformation matrix A's initialization strategy is an area that requires further investigation. An exploration into whether a more intricate network structure could enhance the performance would provide more insight into the system's capabilities and potential limitations. This exploration could open up new possibilities for performance optimization.**\n\nWe will include additional description and further studies for the mask transformation matrix in the camera ready paper. \n\n[4a] Yang, Yafei, and Bo Yang. \"Promising or elusive? unsupervised object segmentation from real-world single images.\", NeurIPS 2022.\n\n[4b] Kim, Jinwoo, et al., \"Shepherding Slots to Objects: Towards Stable and Robust Object-Centric Learning\", CVPR 2023."}]

Author rebuttal:
We would like to thank the reviewers for the helpful feedback and insightful comments.

We are encouraged that our contribution of an unsupervised object discovery network which leverages on feature connectivity and designed object-centric regularization terms is described as "*novel*" (TJh8, T4Hn), "*innovative*" (2W4b), "*simple*" (GCAa, 2W4b) and "*yet it proves to be highly efficacious.*" (2W4b).  

All reviewers highlight the "*sample efficiency*" (TJh8, T4Hn, GCAa) and "*applicability of the proposed design even when the scale of the dataset diminishes.*" (2W4b). We are also heartened that our efforts to design a solution based on theoretical grounds has been described as "*comprehensive*" (TJh8), "*enhancing our understanding of the proposed method's performance and its capacity to generalize effectively in downstream tasks*" (T4Hn). Finally, reviewers comment that our experiments "*on six diverse datasets*" (TJh8) are "*comprehensive*" (Tjh8), "*encouraging*" (2W4b) and "*shows strong performance on benchmarks to the point of saturating or 'solving' simpler cases*" (GCAa).

We further thank reviewers for the very constructive feedback which help us refine our work. In response to this feedback, we have conducted new experiments which we summarize below:

**1. Application of OC-Net to more complex scenes**

We have conducted new experiments for further evaluation on additional Flowers, Birds, COCO [5a] datasets which contain objects that have more diverse colors and shapes. We have also added two more recent baselines BO-QSA [21] and SLASH [5b] for comparison as suggested. The results in the table below shows that OC-Net is robust and remains the top performer:

|  | Flowers | Flowers | Birds | Birds | COCO | COCO |
|---|---|---|---|---|---|---|
| Metric | Dice | IoU | Dice| IoU | mDice| mIoU |
| SLASH | 39.91$\pm$1.48 | 25.98$\pm$1.15 | 43.45$\pm$1.10 | 28.24$\pm$1.08 | 22.34$\pm$1.10 | 13.37$\pm$1.07 | 
| BO-QSA | 65.77$\pm$1.90 | 51.70$\pm$1.93 | 44.61$\pm$1.65 | 30.29$\pm$1.50 | 34.90$\pm$1.14 | 23.57$\pm$0.94 | 
| OC-Net | **67.21$\pm$0.21** | **54.42$\pm$0.23** | **47.80$\pm$0.19** | **33.50$\pm$0.17** | **48.18$\pm$0.17** | **35.58$\pm$0.15** | 

The objects discovered for the various methods on sample images can be viewed in the attached PDF. We see that OC-Net is able to segment out the objects in a fine-grained manner.

**2. Object property prediction evaluation on CLEVRTEX**

We conducted additional experiments to further evaluate the object property prediction on CLEVRTEX as suggested. The table below shows the results:

| Property | Position | Shape |
|---|---|---|
| Slot Attention | 47.5$\pm$19.6 | 30.6 |
| EfficientMORL | 21.8$\pm$2.0 | 18.5 |
| GENESIS-V2 | 79.8$\pm$8.0 | 35.2 |
| SLATE | 62.0$\pm$8.0 | 30.5 |
| SysBinder | 38.2$\pm$3.1 | 29.6 |
| BO-QSA | 66.5$\pm$0.3 | 28.7 |
| OC-Net | **80.7$\pm$2.4** | **36.1** |

**3. Additional comparison with new unsupervised segmentation algorithm**

We have carried out additional experiments for comparison with Ncut [5c] as suggested. The mIoU results in the table below show that OC-Net significantly outperforms other graph-based methods:

| Method | Multi-dSprites | Tetrominoes | SVHN | IDRiD | CLEVRTEX | CLEVRTEX-OOD |
|---|---|---|---|---|---|---|
| Felzenszwalb | 95.0$\pm$0.0 | 96.9$\pm$0.0 | 39.8$\pm$0.0 | 15.4$\pm$0.0 | 26.8$\pm$0.0 | 23.4$\pm$0.0 |
| Ncut | 58.9$\pm$0.0 | 57.4$\pm$0.0 | 32.2$\pm$0.0 | 4.3$\pm$0.0 | 22.9$\pm$0.0 | 18.6$\pm$0.0 |
| OC-Net | **99.1$\pm$0.0** | **100.0$\pm$0.0** | **49.9$\pm$0.1** | **31.2$\pm$0.2** | **34.4$\pm$0.9** | **32.3$\pm$0.5** |

**4. Additional comparison with new state-of-the-art model**

We conduct additional comparison with the most recently published BO-QSA model [21]. The mIoU results in the table below shows that OC-Net is superior:

| Method | Multi-dSprites | Tetrominoes | SVHN | IDRiD | CLEVRTEX | CLEVRTEX-OOD |
|---|---|---|---|---|---|---|
| BO-QSA | 88.0$\pm$1.2 | 25.8$\pm$1.2 | 48.3$\pm$1.3 | 4.5$\pm$1.7 | 31.9$\pm$1.7 | 31.2$\pm$1.0 |
| OC-Net | **99.1$\pm$0.0** | **100.0$\pm$0.0** | **49.9$\pm$0.1** | **31.2$\pm$0.2** | **34.4$\pm$0.9** | **32.3$\pm$0.5** |

[5a] Yang, Yafei, and Bo Yang. "Promising or elusive? unsupervised object segmentation from real-world single images.", NeurIPS 2022.

[5b] Kim, Jinwoo, et al., "Shepherding Slots to Objects: Towards Stable and Robust Object-Centric Learning", CVPR 2023.

[5c] Shi, Jianbo, and Jitendra Malik. "Normalized cuts and image segmentation." IEEE Transactions on pattern analysis and machine intelligence, 2000.


Meta-review: This submission has received four reviews that recommend acceptance. The authors provided extensive answers to the reviewers in several rounds of discussion and provided new evidence. All reviewers wrote that their questions has been answered and resolved to the extend possible in this format and recommend acceptance. 

Since all reviewers appreciate the approach as innovative and novel and find the experiments supportive of the claims we recommend a spotlight presentation for this paper. This paper extends on object centric learning with an approach to connect features and the results surpass prior work on relevant datasets. This will be of interest to the object-centric researchers but appears to be a good showcase to present to the wider NeurIPS community.
Decision: Accept (spotlight)