SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the task, each pixel of an image has to be classified into one of the pre-defined object categories. To obtain the category labels for each predicted mask, we perform the following inference steps. (a) we obtain a set of object-centric representations  $\hat{s}$  for each image. (b) We compute all the object features by taking matrix multiplication between  $M_{\hat{s},x}$  and  $x$  from all the images and then perform k-means clustering on these feature vectors, in which the number of clusters is the number of semantic categories of the benchmark. (c) A binary matching algorithm is used to match our clustered categories with the ground truth by maximizing mIoU. (d) Each pixel on a given image can be assigned to the predicted class label corresponding to the binded slot basis.
A4 Extended Comparison and Discussion with DINOSAUR
Pascal VOC 2012	COCO 2017	MOVi-C	MOVi-E
Original DINOSAUR	24.6 ± 0.2	40.5 ± 0.0	67.2 ± 0.3	64.7 ± 0.7
Re-implemented DINOSAUR	27.5 ± 0.2	36.2 ± 0.7	64.0 ± 0.5	62.4 ± 0.7
Cyclic walks (ours)	29.6 ± 0.8	39.7 ± 0.8	67.6 ± 0.3	64.3 ± 0.7
Table A2: Comparison of the original DINOSAUR and our re-implementation on Object discovery. We report the foreground adjusted rand index (FG-ARI).  $(\pm)$  stands for the standard deviations.
Note that the results of DINOSAUR reported in our paper deviate slightly from the results reported in the original DINOSAUR paper [37]. This is because the code for DINOSAUR was not available at the point of submission. Hence, we strictly followed the model specifications in the original paper, re-implemented DINOSAUR on our own, and re
ported the results of our re-implemented version. These results include the performances of our re-implemented DINOSAUR on Birds, Dogs, Cars, and Flowers datasets, which were missing in the original DINOSAUR paper. We released our re-implementation code of all the baselines at link.
In Table A2, we copied the exact same results in the original DINOSAUR paper. For easy comparison, we also copied the results of our re-implemented DINOSAUR and our method from our paper. The ARI-FG metric performances in (mean+/- standard deviation) are reported. From the results, we observed that our method outperformed the re-implemented DINOSAUR and performed competitively well as the original DINOSAUR in all the experiments. Note that in comparison to DINOSAUR, our
method does not require decoders for reconstruction and hence, our method has fewer parameters, less GPU memory usage, converges faster during training, and achieves faster inference speed as indicated in Sec 5.5.
A5 Additional Visualization
(a) Birds.
(b) Cars.
(c) Dogs.
Figure A2: Additional visualization of the predicted foreground masks in the unsupervised foreground extraction task on (a) Birds, (b) Cars, (c) Dogs, and (d) Flowers datasets. The design conventions follow Figure 3(a).
(d) Flowers
A5.1 Additional Visualization Results of the Unsupervised Foreground Extraction
Method	FG-ARI
Slot Attention	59.2
SLATE	61.5
DINOSAUR	64.9
Invariant Slot Attention	91.3
Cyclic walks (ours)	67.4
Table A3: Results of object discovery on ClevrTex dataset. We report the foreground adjusted rand index (FG-ARI) and the best is in bold.
We provide additional visualization results of the unsupervised foreground extraction task on the Birds, Cars, Dogs, and Flowers datasets in Figure A2. The same result analysis from Section 5.1 can be applied here.
A5.2 Additional Positive and Failure Cases of the Unsupervised Object Discovery
In the unsupervised object discovery task, we provide additional positive samples of the predicted object masks in Figure A3 and negative samples in Figure A4. As discussed in Section 5.2, our method consistently predicts semantic regions de
spite zero annotated ground truth masks provided during training.
However, we also notice that our method as well as other methods are not perfect in some cases, especially when the number of slot bases is larger than the number of semantic objects in the scene. For example, in Row 1 of Figure A4, our method correctly segments trees, dogs, and grass, but incorrectly segments the edges of the dogs. In contrast, other methods output completely random segmented masks, carrying no semantic information whatsoever. Our method produces more "reasonable" negatively segmented masks compared with other methods, which suggests that the slot bases trained with our contrastive walks are capable of capturing distinct semantic representations and meanwhile, taking into account the holistic view of the scene.
A5.3 Visualization Results on MOVi
We provide the visualization results of the Movi-C and Movi-E datasets in Fig A6. We found that our method predicts reasonable semantic segmentation masks in complex scenes containing multiple
Figure A3: Additional successful samples of the predicted object masks in the unsupervised object discovery task on Pascal VOC 2012. The design conventions follow Figure 3(b).
Figure A4: Failure cases of the predicted object masks in the unsupervised object discovery task on Pascal VOC 2012 dataset. From left to right, we show the original image (Col.1), the 4 predicted masks of our cyclic walks corresponding to the 4 slots (Col.2), the 4 masks for DINOSAUR (4 slots, Col.3), the 6 masks for BO-QSA (6 slots, Col.4), and the ground truth object masks (Col.5). Each color indicates a predicted mask from a slot (Col.2-4).
Figure A5: Visualization of instance segmentation masks predicted by our cyclic walks on Pascal VOC 2012. From left to right, we show the original image (Col.1), the predicted object instance masks (Col.2) by our default method (4 slots), the masks of our default method after applying Agglomerative Clustering [33] (Col.3, 4 clusters) and the ground truth semantic masks (Col.4). Each color indicates a predicted mask from a slot (Col.2) or a predicted mask from a cluster (Col.3).
objects (e.g. small forks are successfully segmented in Column 1, Row 3 in the cluttered scene). A similar analysis of visualization results from Fig 3(b) can be applied here for Movi-C and Movi-E.
A5.4 Visualization Results of Instance Segmentation
Traditional object-centric representation learning methods have mainly focused on unsupervised semantic segmentation, such as DINOSAUR [37] and SlotCon [44]. Following this line of research, our method was originally designed for semantic segmentation as well. However, there is a new possibility of applying these methods in instance segmentation tasks. We made the first attempt in this direction and provided visualization results of predicted instance segmentation in Fig A5.
Inspired by CutLer [41], we extract corresponding slot representations for all the image patches and apply Agglomerative Clustering [33] to generate instance segmentation masks. Specifically, Agglomerative Clustering performs self-supervised clustering based on a distance matrix. The distance matrix takes account of both predicted object categories by our default method and the position at each pixel.
From the visualization results in Fig A5, we observed that our method produces visually reasonable instance masks after applying post-processing steps introduced in the previous paragraph. We also noticed several challenging cases where our method failed to separate object instances located close to one another (e.g. five sheep in Row 4, Fig A5). In the future, we will rigorously and quantitatively benchmark unsupervised instance segmentation models. We will also improve the designs of slot-attention modules to learn to predict instance segmentation tasks in an end-to-end fashion.
A6 Experiments and Results Analysis on ClevrTex
We conducted experiments on the ClevrTex dataset and evaluated all baseline methods as well as our method. The results are shown in Table A3. In terms of ARI-FG, our method  $(67.4\%)$  outperforms Slot-Attention  $(59.2\%)$ , SLATE  $(61.5\%)$ , and DINOSAUR  $(64.9\%)$ . Consistent with our results in the paper, the experimental results suggest that our method is superior at learning object-centric representations from complex scenes without reconstructions. It is worth noting that when combined
Figure A6: Visualization of predicted semantic segmentation masks of our cyclic walks on MOVi-C and MOVi-E. In the object discovery task, we present the predicted semantic segmentation masks on (a) MOVi-C and (b) MOVi-E. For each example, from left to right, we show the original image (Col.1), the predicted semantic segmentation masks (Col.2) of our cyclic walks (11 slots for MOVi-C and 24 slots for MOVi-E), and the ground truth semantic segmentation masks (Col.3). Each color indicates a predicted semantic segmentation mask from a slot (Col.2).
Figure A7: Schematic illustration on how to apply the learned object-centric representations in downstream tasks. A slot assignment mask is obtained by calculating the pairwise similarity between the features and the slot representations and then taking the corresponding slot indices with the maximum similarity score. After that, the mask can be applied to downstream tasks such as foreground extraction, object discovery, and semantic segmentation.
with a deeper CNN backbone, Slot Attention can achieve significantly higher performance: [6] reports the performance of Slot Attention on CLEVRTex to be  $91.3\%$  in FG-ARI when combined with a ResNet encoder. Compared with Slot Attention (ResNet), our method uses the frozen DINO encoder, pre-trained on naturalistic images (ImageNet). This might lead to poor feature extraction in CLEVRTex due to domain shifts between real-world and synthetic image datasets. As our experiment has shown that a good extractor is essential for our method to work well, our method can be sensitive to domain shifts if the feature extractor has not been fine-tuned on the images from the current dataset.
As shown in many existing works [29, 38] as well as our work, the original slot attention model always performs poorly. To mitigate this issue, the Invariant Slot Attention paper [6] introduces an interesting method based on slot-centric reference frames. Orthogonal to the proposed invariant slot
attention method, we introduced the cyclic walks between slots and feature maps as the supervision signals to enhance the quality of the learned slot representations. Our method does not require decoders, compared with the invariant slot method.
A7 Limitations and Future Work
In this section, we discuss two limitations of our method. First, our model is only suitable for a fixed number of slots. When the number of slots exceeds the number of objects, our method will pay attention to unnecessary edge details. To overcome this limitation, coming up with a mechanism to merge slots or filter slots is an interesting research direction. Second, our model can not achieve instance segmentation in an end-to-end manner during the inference. How to make slots distinguish individual instances of the same category remains a challenging and practical research topic.
In addition to the future work mentioned in the main text, we would also like to look into object-centric representation learning from videos. Since videos carry temporal correlations and coherences in time, the temporal information is important for learning objects from motions. Moreover, our current work has been focusing on unsupervised foreground extraction, object discovery, and semantic segmentation tasks. In the future, we plan to explore the possibility of adapting our method and benchmarking all unsupervised object-centric learning models in high-level cognitive tasks, such as unsupervised visual question answering.