SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To evaluate the learned object separation of Rotating Features, we cluster their output orientations. For this, we normalize all output features onto the unit-hypersphere and subsequently apply a weighted average that masks features with small magnitudes (Section 3.4). For this masking, we use a threshold of  $t = 0.1$  across all experiments. Subsequently, we apply a clustering method to the weighted average. For  $k$ -means, we utilize the scikit-learn implementation ([55], sklearn.cluster.KMeans) with  $k$  set to the value as specified in Tables 3, 4 and 7, and all other parameters set to their default values. For agglomerative clustering, we also use the scikit-learn implementation (sklearn.cluster.AgglomerativeClustering) with n_clusters set to None. The distance thresholds set to 5 for the results in Fig. 9, or a range of values as visualized in Figs. 7 and 22.
When utilizing DINO preprocessed features, our predicted object masks have a size of  $14 \times 14$ , and we resize them to match the size of the ground-truth labels using the "nearest" interpolation mode. Subsequently, we smooth the mask by applying a modal filter with a disk-shaped footprint of radius  $\lfloor 320 / 14 \rfloor = 22$  using the scikit-image functions skimage.filters.rank.modal and skimage)morphology.disk [78].
For calculating the final object discovery metrics, we treat unlabeled pixels and pixels from overlapping instances as background (i.e. they are not evaluated).
C.3 Datasets
4Shapes The 4Shapes dataset comprises grayscale images of dimensions  $32 \times 32$ , each containing four distinct white shapes  $(\square, \triangle, \nabla, \bigcirc)$  on a black background. The square has an outer side-length of 13 pixels, the circle has an outer radius of 11 pixels and both isosceles triangles have a base-length of 17 pixels and a height of 9 pixels. All shapes have an outline width of 3 pixels. The dataset consists of 50,000 images in the train set, and 10,000 images for the validation and test sets, respectively. All pixel values fall within the range [0,1].
4Shapes RGB(-D) The RGB(-D) variants of the 4Shapes dataset follow the same general setup, but randomly sample the color of each shape. To achieve this, we create sets of potential colors with varying sizes. Each set is generated by uniformly sampling an offset value within the range [0,1], and subsequently producing different colors by evenly dividing the hue space, starting from this offset value. The saturation and value are set to one for all colors, and the resulting HSV color representations are converted to RGB. To create the RGB-D variant, we incorporate a depth channel
to each image and assign a unique depth value within the range [0,1] to every object, maintaining equal distances between them.
10Shapes The 10Shapes dataset consists of RGB-D images with  $48 \times 48$  pixels. Each image features ten unique shapes ( $\square, \triangle, \bigcirc$  in varying orientations and sizes) on a black background. This dataset includes the same square, circle, and triangle as the 4Shapes dataset, with additional variations: the triangle is rotated by  $\pm 90^{\circ}$  degrees to form two more shapes, and the square is rotated by  $45^{\circ}$  to create a diamond shape. Another circle with an outer radius of 19 is also included. All these objects have an outline width of 3 pixels, with their centers remaining unfilled. The dataset also contains two smaller solid shapes: a downward-pointing triangle with a base-length of 11 and a height of 11, and a square with a side-length of 7. Unique colors are assigned to each object by evenly dividing the hue space, starting from an offset value uniformly sampled from the range [0,1]. The saturation and value are set to one for all colors, and the resulting HSV color representations are converted to RGB. Furthermore, each object is given a distinct depth value within the range [0,1], ensuring equal distances between them. The dataset consists of 200,000 images in the train set, and 10,000 images for the validation and test sets, respectively, and all pixel values fall within the range [0,1].
Pascal VOC The Pascal VOC dataset [19] is a widely-used dataset for object detection and semantic segmentation. Following the approach of Seitzer et al. [64], we train our model on the "trainaug" variant, an unofficial split consisting of 10,582 images, which includes 1,464 images from the original segmentation train set and 9,118 images from the SBD dataset [30]. During training, we preprocess images by first resizing them such that their shorter side measures 224 pixels, followed by taking a random crop of size  $224 \times 224$ . Additionally, we apply random horizontal flipping. We evaluate the performance of Rotating Features on the official segmentation validation set, containing 1,449 images. Consistent with previous work [29, 36, 64], we use a resolution of  $320 \times 320$  pixels for evaluation. To achieve this, we resize the ground-truth segmentation mask such that the shorter side measures 320 pixels and then take a center crop of dimensions  $320 \times 320$ . During evaluation, unlabeled pixels are ignored. To make our results comparable to Seitzer et al. [64], we evaluate the performance of our model across 5 seeds on this dataset.
FoodSeg103 The FoodSeg103 dataset [85] serves as a benchmark for food image segmentation, comprising 4,983 training images and 2,135 test images. We apply the same preprocessing steps as described for the Pascal VOC dataset, with the addition of a random augmentation. Besides randomly horizontally flipping the training images, we also rotate them randomly by  $90^{\circ}$  in either direction.
C.4Baselines
Complex AutoEncoder (CAE) The CAE [48] is a continuous and distributed object-centric representation learning approach that has been shown to work well on simple toy datasets. To reproduce this model, we make use of its open-source implementation at https://github.com/loeweX/ComplexAutoEncoder.
AutoEncoder To establish a standard autoencoder baseline, we employ the same architecture as described in Table 2, but replace the Rotating Features and layer structure of  $f_{\mathrm{rot}}$  with standard features and layers. Similarly to the Rotating Features model, we apply Batch Normalization / Layer Normalization before the ReLU non-linearity and maintain an input and output range of [0,1].
Block Masks To create the block masks baseline, we follow a similar approach as described by Seitzer et al. [64]. First, we divide the image into a specific number of columns: for fewer than 9 blocks, we create 2 columns; for 9 to 15 blocks, we create 3 columns, and for more than 15 blocks, we create 4 columns. Next, we distribute the blocks evenly across the columns. If equal division is not possible, we first allocate blocks to all but the last column and then display the remaining blocks in the final column. See Fig. 10 for examples with 5 and 10 blocks.
Figure 10: Block masks with 5 and 10 blocks.
Other Baselines on the Pascal VOC dataset The results of the Slot Attention, SLATE, DINO  $k$ -means, DINOSAUR Transformer, DINOSAUR MLP models on the Pascal VOC dataset are adapted from Seitzer et al. [64]. To ensure the comparability of our findings, we adhere as closely as possible to their setup, utilizing the same dataset structure, preprocessing, and DINO pretrained model.
Figure 11: Theoretical investigation into the capacity of Rotating Features to represent many objects simultaneously. We plot the minimal cosine similarity (y-axis) achievable between any pair of points when placing a specific number of points (x-axis) onto an  $n$ -dimensional hypersphere (mean ± SEM across 10 seeds). We observe that starting from 10 points and with  $n$  large enough, the minimal cosine similarity barely increases. Together with our experiment showing that Rotating Features are able to separate 10 objects, we thus infer that they should be able to scale to larger numbers of objects effectively.
Slot Attention To implement Slot Attention [46], we follow its open-source implementation at https://github.com/google-research/google-research/tree/master/slot attention. We use a hidden dimension of 64 across the model and modulate the decoding structure as outlined in Table 5.
Table 5: Architecture used for the Spatial-Broadcast Decoder in the Slot Attention model.
Layer	Feature Dimension (H × W × C)	Kernel	Stride	Padding Input / Output	Activation Function
Spatial Broadcast	6 × 6 × 64	-	-	-	-
Position Embedding	6 × 6 × 64	-	-	-
TransConv	12 × 12 × 64	5	2	2/1	ReLU
TransConv	24 × 24 × 64	5	2	2/1	ReLU
TransConv	48 × 48 × 64	5	2	2/1	ReLU
TransConv	48 × 48 × 64	5	1	2/0	ReLU
TransConv	48 × 48 × 5	3	1	1/0	ReLU
In this section, we provide additional results exploring the theoretical capacity of Rotating Features to represent many objects simultaneously, introducing a strong baseline for object discovery on the Pascal VOC dataset, and showcasing object separation in the latent space of a model trained with Rotating Features. Furthermore, we will provide additional quantitative and qualitative results for the experiments presented in the main paper.
D.1 High-dimensional Hyperspheres
We explore the theoretical capacity of Rotating Features to represent many objects simultaneously. Rotating Features learn continuous object-centric representations by separating object representations in orientation space. Depending on their orientation, the binding mechanism influences which features are processed together and which ones are masked out. This ultimately affects the number of objects that can be represented separately. Since this mechanism depends on the cosine similarity between features, we can analyze the capacity of Rotating Features by investigating the minimum cosine similarity achievable for a set of points on a hypersphere.
We investigate the behavior of the minimal cosine similarity between points on a hypersphere by implementing an optimization procedure that closely follows Mettes et al. [49]. This is necessary due to the lack of an exact solution for optimally distributing a variable number of points on higher-dimensional hyperspheres [52, 76]. To begin, we randomly sample points on an  $n$ -dimensional hypersphere and then, for 10,000 steps, iteratively minimize the maximum cosine similarity between any pair of points using Stochastic Gradient Descent (SGD) with a learning rate of 0.1 and momentum of 0.9. In Fig. 11, we display the final results of this optimization process, which indicate the minimal achievable cosine similarity between any pair of points when a certain number of points are positioned on an  $n$ -dimensional hypersphere.
From this plot, we observe that regardless of  $n$ , we can always separate two points by a minimum cosine similarity of -1. When  $n = 2$ , which corresponds to the setting of the Complex AutoEncoder, the minimal distance between points increases rapidly as more points are added. In this setting, when placing 20 points onto the hypersphere, the minimal cosine similarity approaches 1, indicating that points are closely packed together and barely separable from one another. With Rotating Features, we introduce a method to learn features with  $n \geq 2$ . As evident from the figure, larger values of  $n$  enable us to add more points to the hypersphere while maintaining small minimal cosine similarities. Finally, we observe that starting from ten points, if  $n$  is sufficiently large, the minimal cosine similarity achievable between points barely increases as more points are added. Having demonstrated that Rotating Features can separately represent ten objects with our experiments on the 10Shapes dataset (Section 4.1), the minimum cosine similarity achievable between 10 points is sufficient for the model to be able to separate them. Thus, we conclude that the number of objects to separate is not a critical hyperparameter for Rotating Features, in contrast to slot-based approaches.
D.2 Weakly Supervised Semantic Segmentation
In our main experiments, we demonstrate that Rotating Features create the capacity to represent more objects simultaneously (Section 4.1). Here, we explore the object separation within the bottleneck of an autoencoder utilizing Rotating Features. Our aim is to highlight that Rotating Features introduce object-centricity throughout the entire model and to examine whether their increased capacity is utilized in the latent space. To accomplish this, we create a weakly supervised semantic segmentation setup. During the autoencoder training, we introduce a supervised task in which a binary classifier is trained to identify the classes of all objects present in an image. Subsequently, we align the orientations of the class predictions with those of the object reconstructions in order to generate semantic segmentation masks.
Method To train a network with Rotating Features on a classification task, we apply a linear layer  $f_{\mathrm{class}}$  following the structure of  $f_{\mathrm{rot}}$  to the latent representation  $\mathbf{z}_{\mathrm{encoder}} \in \mathbb{R}^{n \times d_{\mathrm{encoder}}}$  created by the encoder. This layer maps the latent dimension of the encoder  $d_{\mathrm{encoder}}$  to the number of possible object classes  $o$ :
$$
\mathbf {z} _ {\text {c l a s s}} = f _ {\text {c l a s s}} \left(\mathbf {z} _ {\text {e n c o d e r}}\right) \quad \in \mathbb {R} ^ {n \times o} \tag {10}
$$
Then, we extract the magnitude from this output  $\| \mathbf{z}_{\mathrm{class}}\| _2\in \mathbb{R}^o$  and rescale it using a linear layer  $f_{\mathrm{out}}$  with sigmoid activation function before applying a binary cross-entropy loss (BCE) comparing the predicted to the ground-truth object labels o:
$$
\mathcal {L} _ {\text {c l a s s}} = \operatorname {B C E} \left(f _ {\text {o u t}} \left(\left\| \mathbf {z} _ {\text {c l a s s}} \right\| _ {2}\right), \mathbf {0}\right) \quad \in \mathbb {R} \tag {11}
$$
We use the resulting loss value  $\mathcal{L}_{\mathrm{class}}$  alongside the reconstruction loss  $\mathcal{L}$  to train the network while weighting both loss terms equally.
After training, we leverage the object separation within the bottleneck of the autoencoder to assign each individual pixel a label representing its underlying object class. To do so, we first measure the cosine similarity between each pixel in the normalized rotating reconstruction  $\mathbf{z}_{\mathrm{eval}} \in \mathbb{R}^{n \times h \times w}$  (Eq. (8)) and each value in  $\mathbf{z}_{\mathrm{class}}$  for which  $\sigma(\mathbf{z}_{\mathrm{class}}^i) > 0.5$ . In other words, we calculate the cosine similarity for the values corresponding to object classes predicted to be present in the current image. Subsequently, we label each pixel based on the class label of the value in  $\mathbf{z}_{\mathrm{class}}$  that it exhibits the highest cosine similarity with. If the maximum cosine similarity to all values in  $\mathbf{z}_{\mathrm{class}}$  is lower than a specific threshold (zero in our case), or if  $\| \mathbf{z}_{\mathrm{eval}} \|_2 < 0.1$ , we assign the pixel to the background label. Note that this procedure does not require any clustering method to be employed, but directly utilizes the continuous nature of Rotating Features.
Table 6: Weakly supervised semantic segmentation (WSSS) results (mean ± SEM across 4 seeds). While the latent representations within the bottleneck of an autoencoder trained with Rotating Features can be used to create accurate semantic segmentation masks on the MNIST1Shape dataset in terms of IoU, the same approach fails when adding another object to the input images (MNIST2Shape) despite achieving strong object discovery (ARI-BG) and object classification results (Accuracy). This highlights that Rotating Features can be successfully employed in a weakly supervised semantic segmentation setting, but that the object separation within the bottleneck of the autoencoder is not pronounced enough, yet, to separate more than two objects.
Dataset	Method	IoU ↑	ARI-BG ↑	Accuracy ↑
MNIST1Shape	WSSS	0.785 ± 0.011	0.926 ± 0.007	0.996 ± 0.000
Baseline	0.393 ± 0.006
MNIST2Shape	WSSS	0.230 ± 0.043	0.871 ± 0.029	0.991 ± 0.000
Baseline	0.265 ± 0.006
Setup We construct an autoencoder adhering to the architecture outlined in Table 2, setting  $d = 64$ . The model is trained for 10,000 steps with a batch size of 64 using the Adam optimizer [40]. The learning rate is linearly warmed up for the initial 500 steps of training to reach its final value of 0.001. This setup is applied to two datasets: MNIST1Shape and MNIST2Shape.
MNIST1Shape contains  $32 \times 32$  images, each featuring an MNIST digit and one of three possible shapes  $(\square, \triangle, \vee)$  that share the same format with those in the 4Shapes dataset. Conversely, MNIST2Shape combines an MNIST digit with two of these three shapes per image. Both datasets contain 50,000 train images and 10,000 images for validation and testing.
The network is trained to accurately reconstruct the input image while correctly classifying the MNIST digit, as well as the other shapes present. The setup described above enables the network to produce semantic segmentation masks, despite only being trained on class labels. We assess the performance of this weakly supervised semantic segmentation approach in terms of the intersection-over-union (IoU) score. For comparison purposes, we establish a baseline that generates unlabeled object masks according to the evaluation procedure presented in Section 3.4 using the outputs of the models under evaluation, and randomly assigns each object mask one of the predicted labels. For all results, overlapping areas between objects are masked and ignored during evaluation.
Results The results presented in Table 6 and Fig. 12 indicate that on the MNIST1Shape dataset, our approach with Rotating Features achieves strong weakly supervised semantic segmentation performance. It can accurately match objects separated in the latent space to their representations in the output space. Thereby, it almost doubles the performance of the baseline that randomly assigns labels to discovered objects, without requiring any clustering procedure. However, on the MNIST2Shape dataset, the weakly supervised semantic segmentation method falls short (see Table 6 and Fig. 13), even with a relatively large rotation dimension of  $n = 6$  and despite achieving strong object discovery and object classification results (in terms of ARI-BG and accuracy scores). This demonstrates that while Rotating Features create the capacity to represent more objects simultaneously, this capacity is not fully utilized within the bottleneck of the autoencoder. Since the object separation is not pronounced enough, the current approach can only reliably separate representations of two objects in the latent space. However, the findings also highlight that Rotating Features create object-centric representations throughout the entire model, as well as a potential new application for Rotating Features in weakly supervised semantic segmentation – an application with intriguing implications for interpretability.
D.3 Ablation on Chi
We perform an ablation of the binding mechanism. In this analysis, we modify the Rotating Features model by substituting  $\mathbf{m}_{\mathrm{bind}}$  with  $\| \psi \| _2$  in Eq. (4). This effectively removes the binding mechanism. Then, we apply the adjusted model to the grayscale 4Shapes dataset. While the original model achieves an ARI-BG score of  $0.987\pm 0.003$  on this dataset, the ablated model fails to learn any object separation  $(0.059\pm 0.017)$ . This result highlights the crucial role the binding mechanism plays in enabling Rotating Features to learn object-centric representations.
Figure 12: Weakly supervised semantic segmentation results on the MNIST1Shape dataset. Since this is a semantic segmentation task, colors between ground-truth and predicted objects need to match for strong performance. While this is the case for the predictions created by our proposed approach leveraging Rotating Features, the baseline randomly assigns labels to discovered objects, leading to mismatches between predicted and ground-truth object labels.
Figure 13: Weakly supervised semantic segmentation results on the MNIST2Shape dataset. The baseline creates accurate object masks, albeit with wrongly predicted labels due to its random matching process. Our proposed approach leveraging Rotating Features, on the other hand, falls short in creating accurate semantic segmentations.
Figure 14: Rotating Features with  $n = [2,10]$  on images containing varying numbers of objects (mean ± SEM across 4 seeds). As the number of objects increases, the performance of Rotating Features deteriorates strongly when  $n = 2$ , but remains strong when  $n = 10$ .
D.4 ARI-BG on Pascal VOC
We create a baseline that significantly surpasses all previously documented results in the literature in terms of the ARI-BG score on the Pascal VOC dataset. To achieve this, we simply predict the same object label for all pixels in an image, resulting in an ARI-BG score of 0.499. For comparison, the highest reported ARI-BG in the literature for this dataset is 0.296 [81]. We attribute the remarkable performance of this baseline to the characteristics of the Pascal VOC dataset, which often comprises images featuring a large object dominating the scene. Simply predicting the same label across the entire image results in the same label being predicted across this large, central object, and since the ARI-BG score is not evaluated on background pixels, this leads to relatively high scores. However, for the MBO scores, this baseline yields results far below the state-of-the-art, with 0.205 in  $\mathrm{MBO}_i$  and 0.247 in  $\mathrm{MBO}_c$ . Consequently, we focus our evaluation on these scores for this dataset.
D.5 Rotation Dimension  $n$  vs. Number of Objects
We conduct an experiment to assess how the choice of  $n$  impacts the object discovery performance of a Rotating Features model for different numbers of objects per scene. We accomplish this by creating variations of the 10Shapes dataset that contain between two and ten objects. For each variant, the number of objects per image equals the total number of distinct objects throughout the respective dataset. As illustrated in Fig. 14, the object discovery performance significantly drops as the number of objects increases when  $n = 2$ . However, the performance remains consistently high when  $n = 10$ . These findings indicate that the choice of  $n$  becomes increasingly important as the number of objects rises.
D.6 Multi-dSprites and CLEVR
We evaluate the performance of the proposed Rotating Features model on the commonly used Multi-Sprites and CLEVR datasets [39], comparing it to its predecessor, the CAE model [48]. The hyperparameters are listed in Table 7 and the outcomes are presented in Table 8. Note that since the original CAE was only applicable to grayscale images, we combine it with our proposed evaluation procedure to make it applicable to multi-channel input images, which we denote as CAE*.
The results indicate that the Rotating Features model significantly surpasses the performance of the CAE*, demonstrating good object separation on both datasets. However, the results still lack behind the state-of-the-art. The qualitative examples in Figs. 15 and 16 show that Rotating Features encounter the same issue here, as we have seen with the colored 4Shapes dataset (Section 4.2): objects of the same color tend to be grouped together. As demonstrated with the RGB-D version of the colored 4Shapes dataset and with our results using pretrained DINO features on real-world images, this issue can be addressed by using higher-level input features.
Figure 15: Qualitative performance comparison on the 4Shapes dataset. Rotating Features (RF) create a substantially better object separation than the CAE*, but fail to separate objects of the same color (see right-most column).
Figure 16: Qualitative performance comparison on the 4Shapes dataset. In contrast to the CAE*, the Rotating Features (RF) model learns to separate objects well. However, it fails to separate objects of the same color (see last three columns).