SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results As depicted in Fig. 4, Rotating Features improve in performance on the 4Shapes dataset as we increase the size of the rotation dimension  $n$ . For the CAE and Rotating Features with  $n = 2$ , where the orientation information is one-dimensional, the models struggle to distinguish all four objects. However, as the rotation dimension grows, performance enhances, and with  $n = 6$ , the model perfectly separates all four objects. Additionally, we observe that the reconstruction performance improves accordingly, and significantly surpasses the same autoencoding architecture using standard features, despite having a comparable number of learnable parameters.
Moreover, Rotating Features are able to differentiate 10 objects simultaneously, as evidenced by their ARI-BG score of  $0.959 \pm 0.022$  on the 10Shapes dataset (Fig. 5). In Appendix D.5, we provide a detailed comparison of Rotating Features with  $n = 2$  and  $n = 10$  applied to images with increasing numbers of objects. Additionally, to determine if Rotating Features can distinguish even more objects, we conduct a theoretical investigation in Appendix D.1. In this analysis, we reveal that as we continue to add points to an  $n$ -dimensional hypersphere, the minimum cosine similarity achievable between these points remains relatively stable after the addition of the first ten points,
provided that  $n$  is sufficiently large. This implies that once an algorithm can separate ten points (or objects, in our case) on a hypersphere, it should be scalable to accommodate more points. Consequently, the number of objects to separate is not a critical hyperparameter of our model, in stark contrast to slot-based architectures.
Figure 5: Rotating Features learn to separate all ten objects in the 10Shapes dataset.
4.2 Rotating Features Are Applicable to Multi-Channel Images
Setup We explore the applicability of Rotating Features and our proposed evaluation method to multi-channel inputs by creating an RGB(-D) version of the 4Shapes dataset. This variation contains the same four shapes per image as the original dataset, but randomly assigns each object a color. We vary the number of potential colors, and create two versions of this dataset: an RGB version and an RGB-D version, in which each object is assigned a fixed depth value.
Results As illustrated in Fig. 6, Rotating Features struggle to distinguish objects in the RGB version of the 4Shapes dataset when the number of potential colors increases. This appears to stem from their tendency to group different shapes together based on shared color. In Appendix D.6, we observe the same problem when applying Rotating Features to commonly used object discovery datasets (multidSprites and CLEVR). When depth information is added, this issue is resolved, and Rotating Features successfully separate all objects. This demonstrates that our proposed evaluation procedure makes Rotating Features applicable to multi-channel inputs, but that they require higher-level information in their inputs to reliably separate objects.
4.3 Rotating Features Are Applicable to Real-World Images
Inspired by the previous experiment, which demonstrates that Rotating Features and our proposed evaluation procedure can be applied to multi-channel inputs but require higher-level features for optimal object discovery performance, we apply them to the features of a pretrained vision transformer.
Figure 6: Rotating Features on the 4Shapes RGB(-D) dataset. When applied to RGB images with more colors, object discovery performance in ARI-BG degrades (right; mean ± sem across 4 seeds) as the Rotating Features tend to group objects of the same color together (left). By adding depth information to the input (RGB-D), this problem can be resolved, and all objects are separated as intended.
Setup Following Seitzer et al. [64], we utilize a pretrained DINO model [5] to generate higher-level input features. Then, we apply an autoencoding architecture with Rotating Features to these features. We test our approach on two datasets: the Pascal VOC dataset [19] and FoodSeg103 [85], a benchmark for food image segmentation. On the Pascal VOC dataset, we do not compare ARI-BG scores, as we have found that a simple baseline significantly surpasses previously reported state-of-the-art results on this dataset (see Appendix D.4). On FoodSeg103, we limit the evaluation of our model to the  $\mathrm{MBO}_c$  score, as it only contains class-level segmentation masks.
Results The qualitative results in Fig. 2 highlight the effectiveness of Rotating Features in segmenting objects across both real-world datasets. Table 1 details the object discovery performance of Rotating Features compared to various models on the Pascal VOC dataset. The strongest performance on this dataset is achieved by the DINOSAUR Transformer model [64], a Slot Attention-based model with an autoregressive transformer decoder that is applied to DINO pretrained features. Closer to our setting (i.e., without autoregressive models), the DINOSAUR MLP model combines Slot Attention with a spatial broadcast decoder (MLP decoder) and achieves the second-highest performance reported in literature in terms of  $\mathrm{MBO}_c$  and  $\mathrm{MBO}_i$  scores. The performance of Rotating Features, embedded in a comparatively simple convolu
Table 1: Object discovery performance on the Pascal VOC dataset (mean ± sem across 5 seeds). Rotating Features' scores indicate a good separation of objects on this real-world dataset. Baseline results adapted from Seitzer et al. [64].
Model	MBOi↑	MBOc↑
Block Masks	0.247	0.259
Slot Attention	0.222 ± 0.008	0.237 ± 0.008
SLATE	0.310 ± 0.004	0.324 ± 0.004
Rotating Features -DINO	0.282 ± 0.006	0.320 ± 0.006
DINO k-means	0.363	0.405
DINO CAE	0.329 ± 0.009	0.374 ± 0.010
DINOSAUR Transformer	0.440 ± 0.008	0.512 ± 0.008
DINOSAUR MLP	0.395 ± 0.000	0.409 ± 0.000
Rotating Features	0.407 ± 0.001	0.460 ± 0.001
tional autoencoding architecture, lies in between these two Slot Attention-based models. Rotating Features also outperform a baseline that applies  $k$ -means directly to the DINO features (DINO  $k$ -means), showing that they learn a more meaningful separation of objects. Finally, the performance gain of Rotating Features over the original CAE model applied to the DINO features (DINO CAE) highlights their improved object modelling capabilities. For reference, object discovery approaches applied directly to the original input images (Slot Attention [46], SLATE [68], Rotating Features -DINO) struggle to segment the objects in this dataset and achieve performances close to a Block Masks baseline that divides images into regular rectangles. To verify that Rotating Features work on more settings, we also apply them to the FoodSeg103 dataset. Here, they achieve an  $\mathrm{MBO}_c$  of  $0.484 \pm 0.002$ , significantly outperforming the block masks baseline  $(0.296 \pm 0.000)$ . Overall, this highlights that Rotating Features learn to separate objects in real-world images through unsupervised training; thus scaling continuous and distributed object representations from simple toy to real-world data for the first time.
Figure 7: Clustering methods on Pascal (mean  $\pm$  sem across 4 seeds). After training, we can efficiently compare various clustering methods and hyperparameters such as  $k$ -means (left) with different values of  $k$  and agglomerative clustering (right) with different distance thresholds.
Figure 8: Uncertainty maps of Rotating Features. We plot the distance of each normalized output feature to its closest k-means cluster center, using brighter pixels for larger distances. This highlights uncertainty in areas of object overlap, incorrect predictions, and object boundaries.
4.4 Benefits of Rotating Features
Rotating Features present several advantageous properties: they learn to distinguish the correct number of objects automatically, they generalize well to different numbers of objects than observed during training, they express uncertainty in their object separations, and they are very efficient to run. In this section, we delve into these properties in detail.
Clustering Methods We can evaluate various clustering methods and hyperparameters very efficiently on a Rotating Features model, as they can be freely interchanged after training (Fig. 7). Through their continuous object assignments, Rotating Features learn to separate the correct number of objects automatically. By evaluating with agglomerative clustering, the requirement to specify the number of objects to be separated can be eliminated completely, albeit with slightly reduced results.
Generalization Performance Rotating Features can generalize beyond the number of objects observed during training, even when the number of objects in the test images is unknown. To highlight this, we train a Rotating Features model and a Slot Attention model [46] on a revised version of the 10Shapes dataset. This version includes the same ten unique shapes, but randomly selects six to be included in each image. After training, we test the models on a range of variants of this dataset, displaying between two and ten objects per image. As shown in Fig. 9, when the number of objects is known and the parameters of the models are set accordingly, both Rotating Features and SlotAttention maintain a relatively stable performance across
Figure 9: Testing the generalization performance of Rotating Features (RF) and Slot Attention (SA). All models are trained on images containing six objects. When the number of objects in the test images is unknown, only the Rotating Features model evaluated with agglomerative clustering maintains a relatively stable performance across varying numbers of objects.
various numbers of objects per scene; with SlotAttention consistently outperforming the Rotating Features model. However, when the number of objects in the test images is unknown, and we set the parameters of the model according to our best estimate given the training data, the performances of
the SlotAttention model and the Rotating Features model evaluated with  $k$ -means degrade considerably as the number of objects in the test images increases. This problem can be circumvented by evaluating Rotating Features with agglomerative clustering. After determining the distance threshold on the training dataset, this approach preserves a fairly consistent performance across varying numbers of objects in each scene, considerably outperforming the other approaches that require the number of objects to be set in advance. In summary, our results suggest that Rotating Features can generalize beyond the number of objects observed during training, even when the number of objects in the test images is unknown.
Uncertainty Maps Since Rotating Features learn continuous object-centric representations, they can express and process uncertainty in their object separations. We evaluate this by plotting the L2 distance of each normalized output feature to the closest  $k$ -means center in Fig. 8. This reveals higher uncertainty in areas of object overlap, object boundaries and incorrect predictions.
Training Time Rotating Features are very efficient to train. On a single Nvidia GTX 1080Ti, the presented Rotating Features model on the Pascal VOC dataset requires less than 3.5 hours of training. For comparison, the DINOSAUR models [64] were trained across eight Nvidia V100 GPUs.
Numerous approaches attempt to solve the binding problem in machine learning [28], with most focusing on slot-based object-centric representations. Here, the latent representation within a single layer of the architecture is divided into "slots", creating a discrete separation of object representations. The symmetries between slots can be broken in various ways: by enforcing an order on the slots [4, 16, 18], by assigning slots to spatial coordinates [1, 3, 6, 7, 9, 13, 45, 63], by learning specialized slots for different object types [32, 33, 80, 82], by using an iterative procedure [12, 24, 25, 26, 27, 44, 74, 90] or by a combination of these methods [17, 70, 77, 86]. Slot Attention [46], which uses an iterative attention mechanism, has received increasing interest lately with many proposed advancements [37, 38, 47, 62, 68, 71, 75, 81] and extensions to the video domain [14, 22, 41, 69, 87]. Recently, Seitzer et al. [64] proposed to apply Slot Attention to the features of a pretrained DINO model, successfully scaling it to real-world datasets.
In comparison, there has been little work on continuous and distributed approaches for object discovery. A line of work has made use of complex-valued activations with a similar interpretation of magnitude and orientation to our proposed Rotating Features, starting from supervised [51, 89] and weakly supervised methods [58, 59, 60], to unsupervised approaches [48, 61]. However, they are only applicable to very simple, grayscale data. Concurrent work [73] proposes a contrastive learning method for the Complex AutoEncoder, scaling it to simulated RGB datasets of simple, uniformly colored objects. With the proposed Rotating Features, new evaluation procedure and higher-level input features, we create distributed and continuous object-centric representations that are applicable to real-world data.
Summary We propose several key improvements for continuous and distributed object-centric representations, scaling them from toy to real-world data. Our introduction of Rotating Features, a new evaluation procedure, and a novel architecture that leverages pre-trained features has allowed for a more scalable and expressive approach to address the binding problem in machine learning.
Limitations and Future Work Rotating Features create the capacity to represent more objects at once. However, this capacity can currently only be fully leveraged in the output space. As we show in Appendix D.2, within the bottleneck of the autoencoder, the separation of objects is not pronounced enough, yet, and we can only extract up to two objects at a time. To overcome this limitation, further research into novel inference methods using Rotating Features will be necessary. In addition, when it comes to object discovery results, Rotating Features produce slightly inferior outcomes than the state-of-the-art DINOSAUR autoregressive Transformer model, while still surpassing standard MLP decoders. Nonetheless, we hope that by exploring a new paradigm for object-centric representation learning, we can spark further innovation and inspire researchers to develop novel methods that better reflect the complexity and richness of human cognition.
We thank Jascha Sohl-Dickstein and Sjoerd van Steenkiste for insightful discussions. Francesco Locatello worked on this paper outside of Amazon. Sindy Löwe was supported by the Google PhD Fellowship.
[78] Stefan Van der Walt, Johannes L Schonberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. scikit-image: Image processing in python. PeerJ, 2:e453, 2014.
[79] Christoph Von Der Malsburg and Werner Schneider. A neural cocktail-party processor. Biological Cybernetics, 54(1):29-40, 1986.
[80] Julius von Kugelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. *ICLR* 2020 Workshop: Causal Learning for Decision Making, 2020.
[81] Ziyu Wang, Mike Zheng Shou, and Mengmi Zhang. Object-centric learning with cyclic walks between parts and whole. arXiv preprint arXiv:2302.08023, 2023.
[82] Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[83] Max Wertheimer. Untersuchungen zur Lehre von der Gestalt. Psychologische Forschung, 1(1): 47-58, 1922.
[84] Ross Wightman. Pytorch image models. https://github.com/rwrightman/pytorch-image-models, 2019.
[85] Xiongwei Wu, Xin Fu, Ying Liu, Ee-Peng Lim, Steven CH Hoi, and Qianru Sun. A large-scale benchmark for food image segmentation. In Proceedings of ACM international conference on Multimedia, 2021.
[86] Yizhe Wu, Oiwi Parker Jones, and Ingmar Posner. Obpose: Leveraging canonical pose for object-centric scene inference in 3d. arXiv preprint arXiv:2206.03591, 2022.
[87] Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. International Conference on Learning Representations (ICLR), 2023.
[88] Andrii Zadaianchuk, Matthaeus Kleindessner, Yi Zhu, Francesco Locatello, and Thomas Brox. Unsupervised semantic segmentation with self-supervised object-centric representations. International Conference on Learning Representations (ICLR), 2023.
[89] Richard S Zemel, Christopher KI Williams, and Michael C Mozer. Lending direction to neural networks. Neural Networks, 8(4):503-512, 1995.
[90] Ruixiang Zhang, Tong Che, Boris Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, and Liam Paull. Robust and controllable object-centric learning through energy-based models. International Conference on Learning Representations (ICLR), 2023.
Appendix Rotating Features for Object Discovery
This research explores advancements in continuous and distributed object-centric representations, scaling them from simple toy to real-world data. The broader impact includes:
1. Advancing Machine Learning: The proposed techniques expand the methodological landscape for object discovery, which may benefit many areas of machine learning, such as computer vision, robotics, and natural language processing.
2. Biological Insight: By exploring biologically inspired approaches to resolve the binding problem in machine learning, we may advance our understanding of object representations in the brain.
3. Improved Interpretability: The research promotes more structured representations, which may improve interpretability in machine learning models.
4. Scalability and Real-World Applicability: The proposed approaches can be applied to complex, multi-dimensional data, such as RGB images, making continuous object-centric representations applicable in real-world scenarios.
This research inspires further innovation and interdisciplinary collaboration. While we currently do not anticipate any negative societal impacts, we recognize that the wide array of potential applications for object discovery methods could lead to unforeseen consequences. As such, we acknowledge that there may be potential impacts that we are unable to predict at this time.
In order to ensure the reproducibility of our experiments, we have provided a comprehensive overview of the employed model architectures, hyperparameters, evaluation procedures, datasets and baselines in Appendix C. Additionally, we published the code used to produce the main experimental results at github.com/loeweX/RotatingFeatures.
To achieve stable and reproducible results, we conducted all experiments using 4-5 different seeds. Training and evaluating the roughly 180 Rotating Features models highlighted across all our experiments took approximately 450 hours on a single Nvidia GTX 1080Ti. Training and evaluating four Slot Attention models for the baseline in Section 4.4 took another 200 hours on the same device. This estimate does not take into account the time spent on hyperparameter search and various trials conducted throughout the research process.
C.1 Architecture and Hyperparameters
We implement Rotating Features within an autoencoding model. Here, we will describe the precise architecture and hyperparameters used to train this model.
Architecture We provide a detailed description of the autoencoding architecture used for all of our experiments in Table 2. For the activation function (Eq. (4)), we employ Batch Normalization [35] in all convolutional layers, and Layer Normalization [2] in all fully connected layers.
Parameter Count As only the biases depend on the number of rotation dimensions  $n$ , the number of learnable parameters increases only minimally for larger values of  $n$ . For instance, in the models utilized for the 4Shapes dataset (Table 2 with  $d = 32$ ), the total number of parameters amounts to 343,626 for  $n = 2$  and 352,848 for  $n = 8$ , reflecting a mere  $2.7\%$  increase.
Table 2: Autoencoding architecture with input dimensions  $h, w, c$  and feature dimension  $d$  (values for  $d$  are specified in Tables 3, 4 and 7). All fractions are rounded up. Layers denoted by * are only present in the models that are applied to pretrained DINO features. Otherwise, the feature dimension of the final TransConv layer is  $h \times w \times c$ .
Layer	Feature Dimension (H × W × C)	Kernel	Stride	Padding Input / Output
Input	h × w × c
Encoder	Conv	h/2 × w/2 × d	3	2	1/0
Conv	h/2 × w/2 × d	3	1	1/0
Conv	h/4 × w/4 × 2d	3	2	1/0
Conv	h/4 × w/4 × 2d	3	1	1/0
Conv	h/8 × w/8 × 2d	3	2	1/0
Conv*	h/8 × w/8 × 2d	3	1	1/0
Reshape	1 × 1 × (h/8 * w/8 * 2d)	-	-	-
Linear	1 × 1 × 2d	-	-	-
Decoder	Linear	1 × 1 × (h/8 * w/8 * 2d)	-	-	-
Reshape	h/8 × w/8 × 2d	-	-	-
TransConv	h/4 × w/4 × 2d	3	2	1/1
Conv	h/4 × w/4 × 2d	3	1	1/0
TransConv	h/2 × w/2 × 2d	3	2	1/1
Conv	h/2 × w/2 × d	3	1	1/0
TransConv	h × w × d	3	2	1/1
Conv*	h × w × c	3	1	1/0
Hyperparameters All hyperparameters utilized for models trained directly on raw input images are detailed in Table 3, while those for models trained on DINO features can be found in Table 4. We initialize the weights  $\mathbf{w}_{\mathrm{out}} \in \mathbb{R}^c$  of  $f_{\mathrm{out}}$  by setting them to zero, and set biases  $\mathbf{b}_{\mathrm{out}} \in \mathbb{R}^c$  to one. The remaining parameters are initialized using PyTorch's default initialization methods.
Table 3: Hyperparameters for models with Rotating Features that are trained directly on the raw input images.
Dataset	4Shapes	4Shapes RGB(-D)	10Shapes
Training Steps	100k	100k	100k
Batch Size	64	64	64
LR Warmup Steps	500	500	500
Peak LR	0.001	0.001	0.001
Gradient Norm Clipping	0.1	0.1	0.1
Feature dim d	32	64	32
Rotating dim n	8	8	10
Image/Crop Size	32	32	48
Cropping Strategy	Full	Full	Full
Augmentations	-	-	-
k	5	5	11
ViT Encoder To generate higher-level input features of real-world images, we use a pretrained vision transformer model [11] to preprocess the input images. Specifically, we employ a pretrained DINO model [5] from the timm library [84], denoted as ViT-B/16 or "vit_base_batch16_224". This model features a token dimensionality of 768, 12 heads, a patch size of 16, and consists of 12 Transformer blocks. We use the final Transformer block's output as input for the Rotating Features model, excluding the last layer norm and the CLS token entry. All input images for the ViT have a resolution of  $224 \times 224$  pixels and are normalized based on the statistics of the ImageNet dataset
Table 4: Hyperparameters for models with Rotating Features that are trained on DINO preprocessed features.
Dataset	Pascal	FoodSeg103
Training Steps	30k	30k
Batch Size	64	64
LR Warmup Steps	5000	5000
Peak LR	0.001	0.001
Gradient Norm Clipping	0.1	0.1
Feature dim d	128	128
Rotating dim n	10	10
Image/Crop Size	224	224
Cropping Strategy	Random	Random
Augmentations	Random Horizontal Flip	Random Horizontal Flip and ±90° rotation
k	4	5
[10], with mean per channel (0.485, 0.456, 0.406) and standard deviation per channel (0.229, 0.224, 0.225). The output of the DINO model has a dimension of  $14 \times 14 \times 768$  and we train our model to reconstruct this output. Before using it as input, we apply Batch Normalization and ReLU to ensure all inputs to the Rotating Features are positive. Overall, this setup closely follows that of Seitzer et al. [64] to ensure comparability of our results.
C.2 Evaluation Details