SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(mIoU) [22] and Dice [22]. The mIoU measures the overlap between predicted masks and the ground truth. Dice is similar to mIoU but replaces the union of two sets with the sum of the number of their elements and doubles the intersection. We used publicly available implementations of Slot-Attention [29] and SLATE [38] from [22] and re-implemented DINOSAUR [37] by ourselves. Note that the results of re-implemented DINOSAUR deviate slightly from the original results in [37]. We provided further comparisons and discussed such discrepancy in Appendix A4. However, none of these alter the conclusions of this paper. Following the work of BO-QSA [22], we include Stanford Dogs [23], Stanford Cars [25], CUB 200 Birds [43], and Flowers [32] as benchmark datasets.
Results and Analysis. The results in mIoU and Dice are presented in Table 1. Our method achieved the best performance on the Birds, Dogs, and Cars datasets and performed competitively well as BO-QSA on the Flowers dataset. Slot-attention, SLATE, and DINOSAUR use the pixel-level, feature-level, and token-level reconstruction losses from object-centric representations as supervisory signals. The inferior performance of slot attention over SLATE and DINOSAUR implies that pixel-level reconstructions focus excessively on unnecessary details, impairing the object-centric learning performance. We also note that DINOSAUR outperforms Slot-Attention and SLATE by a significant margin. This aligns with the previous findings that freezing pre-trained feature extractors facilitates object-centric learning. In addition to reconstructions, BO-QSA imposes extra constraints on learnable slot queries with bi-level optimizations, which leads to better performances. This emphasizes the necessity of searching for better supervisory signals to regularize object-centric learning beyond reconstructions. Indeed, even without any forms of reconstruction, we introduce cyclic walks between parts and whole as an effective training loss for object-centric learning. Our method yields the best performance among all comparative methods. The result analyses hold true in subsequent tasks (Section 5.2 and 5.3).
We also visualize the attention masks predicted by our slot bases in Figure 3(a) (see Appendix A5.1 for more visualization results). We observed that the model trained with our cyclic walks outputs reasonable predictions. However, we also noticed several inconsistent predictions. For example, our predicted masks in the Flowers dataset (Row 3 and 4) group the pedals and the sepals of a flower altogether, because these two parts often co-occur in a scene. Moreover, we also spotted several wrongly annotated ground truth masks. For example, in Row 4, the foreground mask mistakenly incorporates the sky as part of the ground truth. It is also interesting to see that the foreground object belonging to the same object class is not always identified by a fixed slot basis (compare Row 1 and 2 in the Birds dataset). During the inference, we randomly sample parameters from learnable mean  $\mu$  and variance  $\sigma$  as the slot bases (Section 3.1), which could cause the reverse of foreground and background masks. Applying the same set of parameters for slot bases across the experiments avoids such issues.
5.2 Unsupervised Object Discovery
Task Setting, Metrics, Baselines and Datasets. The task aims to segment the image with a set of binary masks, each of which covers similar parts of the image, (aka "object masks"). However, different from image segmentation, each of the masks does not have to be assigned specific class labels. We follow recent works and evaluate all models [29, 38, 22, 37] and K-Means with ARI on foreground objects (ARI-FG). ARI reflects the proportion of sampled pixel pairs on an image, correctly classified into the same class or different classes. All baseline implementations are based on the open-source codebase [28]. Same as Section 5.1, we use the  $M_{x,\hat{s}}$  as the object masks.  $M_{x,\hat{s}}$  is the similarity matrix, indicating the transition probability between features and slots (see Equation 4 and Equation 5). We benchmark all methods on the common datasets Pascal VOC 2012 [15], COCO 2017 [27], Movi-C [17] and Movi-E[17]. We also evaluate all methods on a synthetic dataset ClevrTex for further comparisons (see Appendix A6).
Results and Analysis The results in ARI-FG on the four datasets are shown in Figure 4. The unsupervised object discovery task is harder than the unsupervised foreground extraction task (Section 5.1) due to the myriad scene complexity and object class diversity. Our method still consistently outperforms all SOTA methods and beats the second-best method DINOSAUR by a large margin of  $2 - 4\%$  over all four datasets. Different from all other methods attached with decoders for the image or feature reconstructions, our model trained with cyclic walks learns better object-centric representations. We visualized some example predicted object masks in Figure 3(b) (see Appendix A5.2 for more positive samples). Our method can clearly distinguish semantic areas in complex
Figure 4: Performance of Object Discovery. From left to right, the used datasets are Pascal VOC 2012 [15], COCO 2017 [27], Movi-C [17] and Movi-E [17]. We report the performance in foreground adjusted rand index (ARI-FG). The higher the ARI-FG, the better. We compare our method (cyclic walks) with K-means, Slot-Attention [29], SLATE [38], DINOSAUR [37], and BO-QSA [22].
Model	Pascal VOC 2012	COCO Stuff-27	COCO Stuff-3
ACSeg [26]	47.1	16.4	-
MaskDistill [16]	42.0	-	-
PiCIE + H [12]	-	14.4	-
STEGO [18]	-	26.8	-
SAN [47]	-	-	80.3
InfoSeg [19]	-	-	73.8
SlotCon [44]	-	18.3	-
COMUS [46]	50.0	-	-
Cyclic walks (ours)	43.3 ± 1.6	22.5 ± 1.1	82.4 ± 0.7
Table 2: Results of Unsupervised Semantic Segmentation in IoU on Pascal VOC 2012, COCO-Stuff-27 [7] and COCO-Stuff-3 [7]. (±) standard deviations of IoU over 5 runs are also reported. The ’-’ indicates that corresponding results are not provided in the original papers and the source codes are unavailable. Best is in bold.
scenes. For example, trains, trees, lands, and sky are segmented in Row 1 in an unsupervised manner. Note that the foreground train masks are provided as the ground truth annotations in PASCAL VOC only for qualitative comparisons. They have never been used for training. Additional visualization results on MOVi can be found in Appendix A5.3. In the experiments, we also noticed several failure cases. When the number of semantic classes in the image is less than the number of slots, our method segments the edges of semantic classes (see Appendix A5.2 for more analysis).
5.3 Unsupervised Semantic Segmentation
Task Setting, Metrics, Baselines and Datasets. In the task, each pixel of an image has to be classified into one of the pre-defined object categories. See Appendix A3.2 for implementations of obtaining the category labels for each predicted mask during inference. We report the intersection over union (IoU) between the predicted masks and the ground truth over all categories. We include ACSeg [26], MaskDistill [16], PiCIE [12], STEGO [18], SAN [47], InfoSeg [19], SlotCon [44], and COMUS [46] for comparison. All the results are directly obtained from their original papers. We evaluate all competitive methods on Pascal VOC 2012, COCO Stuff-27 [7], and COCO Stuff-3 [7]. COCO Stuff-27 and COCO Stuff-3 contain 27 and 3 supercategories respectively.
Results and Analysis We report the results in terms of IoU in Table 2. Our method has achieved the best performance on COCO-Stuff-3 and the second best on Pascal VOC 2012 and COCO-Stuff-27. This highlights that our cyclic walks in the part-whole hierarchy on the same images act as effective supervision signals and distill image pixels into diverse object-centric representations rich in semantic information. It is worth pointing out that our model is not specially designed for semantic segmentation. Yet, it is remarkable that our method still performs competitively well as the majority of the existing semantic segmentation models.
On Pascal VOC 20212, our method slightly underperforms COMUS [46] in mIoU metrics (43.3% versus 50.0%). We argue that, in comparison to our method, there are two additional components in COMUS possibly attributing to the performance differences and unfair comparisons. First, COMUS employs two pre-trained saliency detection architectures DeepUSPS [31] and BasNet [35] in addition to DINO. The saliency detection model requires an additional MSRA image dataset [34] during pretraining. Thus, compared to our cyclic walks, COMUS indirectly relies on extra useful information. Second, COMUS uses Deeplab-v3 as its backbone for predicting semantic segmentation masks. For better segmentation performances, Deeplab-v3 extracts large-resolution feature maps from images
Dataset	Full Model	Random Walk Direction	Temperature of Random Walks	Number of slots	Similarity Threshold	Feature Extractor
P-W-P	W-P-W	0.07	0.3	1	5	6	7	-inf	0.3	1	Moco-v3 [10]	MAE [20]	MSN [3]
Pascal VOC 2012	29.6	27.1	28.4	23.5	27.3	21.9	27.7	26.2	24.3	22.5	26.5	26.2	29.3	26.8	29.0
P-W-P	W-P-W	0.07	0.3	1	10	12	13	-inf	0.3	1	Moco-v3 [10]	MAE [20]	MSN [3]
COCO 2017	39.7	34.8	36.4	35.5	36.7	29.6	37.9	35.7	35.3	29.8	32.3	31.8	38.4	34.5	38.2
Table 3: Ablation Studies and Method Analysis on Pascal VOC 2012 and COCO 2017 in terms of ARI-FG. Full model refers to our default method introduced in Section 4. See Appendix A2 for the empirically determined hyper-parameter details of our full model. We vary one factor at a time and study its effect on ARI-FG performances in the Pascal VOC 2012 (Row 3) and COCO 2017 (Row 5). The best is our default model highlighted in bold. See Section 5.4 for details.
and applies Atrous Spatial Pyramid Pooling for aggregating these feature maps over multiple scales. In contrast, our method extracts low-resolution feature maps with DINO and is not specifically designed for unsupervised semantic segmentation.
Despite a slightly inferior performance to COMUS, our method is capable of parsing the entire scene on the image into semantically meaningful regions, which COMUS fails to do. These include distinct backgrounds, such as sky, lake, trees, and grass. For example, in Fig 3(b), our method successfully segments backgrounds, such as the lake, the trees, and the sky. However, COMUS fails to segment background elements. For example, in Column 4 of Fig 1 in the paper of COMUS [46], COMUS only segments the foreground ships and fails to segment the lake and the sky. Segmenting both salient objects and other semantic regions is essential for many computer vision applications. This emphasizes the importance and unique advantages of our method. We also attempted to apply object-centric methods in instance segmentation tasks. In the Appendix A5.4, we provided visualization results and discussions.
5.4 Ablation Study and Method Analysis
We examine the effect of hyperparameters, the components of training losses, and the variations of feature extractors on object-centric learning performances and report ARI-FG scores for all these experiments on Pascal VOC 2012 [15] and COCO 2017 [27] (Table 3).
Ablation on Random Walk Directions. To explore the effects of random walk directions, we use either P-W-P or W-P-W walks to train the network separately and observe the changes (Table 3, Col.3-4). We found that either direction of random walks can make the training loss of the network converge. The W-P-W walks perform slightly better than the P-W-P walks, but neither of these walks surpasses random walks in both directions. This aligns with the design motivation of our method (Section 4).
Analysis on Temperature of Random Walks. We titrate the temperate in Equation 4 from 0.07 to 1 and observe a non-monotonic performance trend versus temperature choices (Col.5-7). On one hand, the lower the temperature, the more concentrated the attention distribution over all the slots and the faster the network can converge due to the steeper gradients. On the other hand, the attention distribution becomes too sharp with much lower temperatures, resulting in optimization instability.
Analysis on Number of Slots. The number of slots imposes constraints on the diversity of objects captured in the scene. We trained models with various numbers of slots (Col.8-10). More slots do not always lead to better performances. With more slots, it is possible that all slots compete with one another for each image patch and the extra slots only capture redundant information, hurting the object discovery performance.
Analysis on Similarity Threshold in P-W-P Walks. During P-W-P cyclic walks, we introduce similarity threshold  $t$  (Equation 6). From Col.11-13, we can see that with the increase of threshold  $\gamma$  from  $-inf$  to 0.7, the P-W-P random walks become more selective, enforcing the slot bases to learn more discriminative features; hence, better performance in object discovery tasks. However, when the threshold approaches 1, the ability to walk to neighboring image patches with high semantic similarity to the starting patch is impaired, leading to the overfitting of slot bases.
Analysis on Different Feature Extractors. We replace pre-trained DINO transformer (Section 4.1) with MOCO-V3 [10], MAE [20], and MSN [3]. Together with DINO, these feature extractors are state-of-the-art unsupervised representation learning frameworks. From Col.14-16, we observe that various backbones with our method consistently achieve high ARI-FG scores over both datasets. This suggests that our method is agnostic to backbone variations and it can be readily adapted to any general SOTA unsupervised learning frameworks.
Analysis on Reconstruction Loss. We assess the effect of pixel-level reconstructions by adding a transformer-based decoder and introducing an extra reconstruction loss. The upgraded model achieved slightly better performance than our default model (from  $29.6\%$  to  $29.9\%$  in ARI-FG) on Pascal VOC 2012. This indicates that the reconstruction loss provides auxiliary information for object-centric learning. In future work, we will explore the possibilities of predicting object properties from the slot features.
5.5 Efficiency Analysis in Model Sizes, Training Speed, GPU Usages, and Inference Speed
Method	Parameters (103)	Training speed (image / sec)	GPU usage (M)	Inference speed (image / sec)
Slot-Attention	3144	114	4371	126
SLATE	14035	27	16327	32
DINOSAUR	11678	124	3443	130
BO-QSA	14223	25	16949	32
Cyclic walks (ours)	1927	208	2331	285
Table 4: Method Comparison in the number of parameters, training speed, GPU memory, and inference speed. From top to bottom, we include Slot-Attention, SLATE, DINOSUAR and BO-QSA. The inference speed was reported based on Object Discovery. The best is in bold.
We report the number of network parameters, training speed, GPU usage, and inference speed in Table 4. All these experiments are run with the same hardware specifications and method configurations: (1) one single RTX-A5000 GPU; and (2) input image size of  $224 \times 224$ , batch size of 8, and 4 slots. In comparison with all transformer decoder-based methods (SLATE, DINOSAUR and BO-QSA), our method uses the fewest parameters and yet, achieves the best performance in all the tasks (Sections 5.1, 5.2, 5.3). Compared to other decoder-based methods, Slot-Attention requires fewer parameters by sampling from a mixture of slot bases for image reconstruction. However, our method requires even fewer parameters than Slot-Attention (Col. 1). Moreover, although the transformer-based networks are slower in training compared with the CNN-based networks, DINOSAUR, and our method are much faster than SLATE and BO-QSA due to freezing the feature extractor (Col. 2). Besides, the losses of our method converge the fastest. On Pascal VOC 2012 and COCO 2017, our model can achieve the best performance within 10k training steps, while other models need at least 100k training steps. By additionally benefiting from cyclic walks and bypassing decoders for feature reconstruction, our method runs twice as much faster as DINOSAUR. With the same reasoning, the networks trained with our cyclic walks only require half of the GPU memory usage of DINOSAUR and achieve the fastest inference speed without sacrificing the performance in all three tasks (Col. 3 and Col. 4).
We propose cyclic walks in the part-whole hierarchy for unsupervised object-centric representation learning. In the slot-attention module, a finite set of slot bases compete to bind with certain regions of the image and distill into object-centric representations. Both P-W-P and W-P-W cyclic walks serve as implicit supervision signals for training the networks to learn compositional entities of the scene. Subsequently, these entities can be useful for many practical applications, such as scene understanding, reasoning, and explainable AIs. Our experiments demonstrate that our cyclic walks outperform all competitive baselines over seven datasets in three unsupervised tasks while being memory-efficient and computation-efficient during training. So far, our method has been developed on a frozen unsupervised feature extractor. In the future, hierarchical contrastive walks can be explored in any feed-forward architectures, where the models can simultaneously learn both pixel-level and object-centric representations incrementally over multiple layers of the network architecture. We provide in-depth discussion of limitations and future works in Appendix A7.
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-RP-2021-025), and its NRFF award NRF-NRFF15-2023-0001. Mike Zheng Shou is supported by the National Research Foundation, Singapore under its NRFF Award NRF-NRFF13-2021-0008. We also acknowledge Mengmi Zhang's Startup Grant from Agency for Science, Technology, and Research (A*STAR), and Early Career Investigatorship from Center for Frontier AI Research (CFAR), A*STAR.
[44] X. Wen, B. Zhao, A. Zheng, X. Zhang, and X. Qi. Self-supervised visual representation learning with semantic grouping. arXiv preprint arXiv:2205.15288, 2022.
[45] J. Yang, J. Mao, J. Wu, D. Parikh, D. D. Cox, J. B. Tenenbaum, and C. Gan. Object-centric diagnosis of visual reasoning. CoRR, abs/2012.11587, 2020.
[46] A. Zadaianchuk, M. Kleindessner, Y. Zhu, F. Locatello, and T. Brox. Unsupervised semantic segmentation with self-supervised object-centric representations. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[47] D. Zhang, C. Li, H. Li, W. Huang, L. Huang, and J. Zhang. Rethinking alignment and uniformity in unsupervised image semantic segmentation. CoRR, abs/2211.14513, 2022.
A1 Discussion of Using Pre-trained Features
In this section, we explain why we use the frozen feature extractor and highlight the advantages of using it. In addition, we fine-tune the feature extractor and the inferior results prove the importance of using frozen feature extractors.
A1.1 Using Pre-trained Features Is NOT a Limitation of Our Method.
We follow the same practice as previous works [37, 16] by freezing feature extractors. It is for a fair comparison with the baselines in the literature.
We argue that freezing feature extractors is not a limitation of our method because of the two reasons below. First, our method can still be easily adapted to any new datasets with domain shifts. To achieve this, we introduce the 2-stage training pipeline. At stage 1, the feature extractor of a model can be trained on the new dataset using self-supervised learning. At stage 2, the feature extractor is frozen and the slot-based attention module can be fine-tuned using our method on the new dataset. Note that both stages only require self-supervised learning without any human supervision. This enables our method to be easily applied in new datasets and new domains with 2-stage training.
Second, freezing feature extractors before learning object-centric representations is loosely connected with the neuroscience findings. These findings suggest that neural plasticity is increased from low-level visual processes (frozen feature extractor) to higher levels of cortical processing responsible for handling symbolic representations (slot-based attention module).
A1.2 Fine-tuning the Feature Extractor Hurts the Performances.
Method	FG-ARI
1-Fine-tune	12.2
2-Learning-rate	22.4
3-EMA	21.3
Frozen	29.6
Table A1: Results of different methods for finetuning the feature extractor. We use the experiment settings in the object discovery task on Pascal VOC 2012 and report the FG-ARI.
We did the following three experiments to investigate how fine-tuning the feature extractor contributes to the overall performance of object-centric learning in ARI-FG. The results are shown in Table A1. First, we fine-tune both the feature extractor and the slot-based attention (1-Finetune). The performance in ARI-FG is  $12.2\%$ . Second, we assign a small learning rate of 0.0001 for the feature extractor and a large learning rate of 0.0004 for the slot-based attention (2-Learning-rate). We observed a great performance improvement from  $12.2\%$  in 1-Fine-tune to  $22.4\%$  in 2-Learning-rate. Third, we apply EMA (Exponential Moving Averaging) on the entire model (3-
EMA). The performance of  $21.3\%$  in 3-EMA is slightly worse than the 2-Learning-rate.
From all these results, aligning with the neuroscience inspirations above in the previous subsection, we found that the slow update of the feature extractor stabilizes the learning of high-level object-centric representations. However, the performance is still inferior to our default model in the paper (29.6% in ARI-FG). This emphasizes the importance of freezing feature extractors for our method.
A2 Training Details and Model Configurations
We set the patch size to be 8. Our model is optimized by AdamW optimizer [30] with a learning rate of 0.0004, 250k training steps, linearly warm-up of 5000 steps, and an exponentially weight-decaying schedule. The gradient norm is clipped at 1. We use Pytorch automatic mixed-precision and data paralleling for training acceleration. All models are trained on 4 Nvidia RTX A5000 GPUs with a total batch size of 128. The temperature of cyclic walks is set to 0.1. We use similarity threshold 0.7 and ViT-S8 of DINO [9] for all experiments. We report the mean  $\pm$  standard deviation of 5 runs with 5 random seeds for all our experiments. We trained all models (Slot-Attention, SLATE, BO-QSA, DINOSAUR, and our Cyclic walks) with 250k training steps and selected their best models by keeping track of their best accuracies on the validation sets.
We list the number of slots and image size used for each dataset in Table A1. For Birds, Cars, Dogs, and Flowers datasets, we report the performance of Slot-Attention, SLATE, and BO-QSA from the work BO-QSA [22]. For other implementation details, we follow all method configurations in the work DINOSAUR [37].
Birds	Cars	Dogs	Flowers	Pascal VOC 2012	COCO 2017	COCO-Stuff	Movi-C	Movi-E
Slot-Attention	-	-	-	-	6	7	-	11	24
SLATE	-	-	-	-	6	7	-	11	24
DINOSAUR	2	2	2	2	4	7	-	11	24
BO-QSA	-	-	-	-	6	7	-	11	24
Cyclic walks (ours)	2	2	2	2	4	7	11	11	24
Image Size	128	128	128	128	224	224	224	224	224
Figure A1: The choice of the number of slots and image size for all the methods in each dataset. The  $-$  indicates that the performance results are directly taken from other papers and thus the configuration is not provided here.
A3 Inference Steps of Our Method
A3.1 Unsupervised Foreground Extraction and Unsupervised Object Discovery
During the inference, all the models are asked to predict foreground masks in the unsupervised foreground extraction task and object masks in the unsupervised object discovery task (Section 5.2). We use  $M_{x,s}$  (Equation 5) as the segmentation masks, where each feature vector at any spatial location of  $x$  is softly assigned to a cluster center. The mask with a maximum intersection with the ground truth masks is viewed as the corresponding predicted mask.
A3.2 Unsupervised Semantic Segmentation