Rotating Features for Object Discovery
Authors: Sindy Löwe
Phillip Lippe
Francesco Locatello
Max Welling
Venue: NeurIPS 2023 oral (2023)
Keywords: Object Discovery
Object-Centric Representations
Structured Representation Learning
TL;DR: This paper introduces several advancements for continuous and distributed object-centric representations, scaling them from simple toy to real-world data, and thereby paving the way for a new paradigm in objects discovery.

Abstract: The binding problem in human cognition, concerning how the brain represents and connects objects within a fixed network of neural connections, remains a subject of intense debate. Most machine learning efforts addressing this issue in an unsupervised setting have focused on slot-based methods, which may be limiting due to their discrete nature and difficulty to express uncertainty. Recently, the Complex AutoEncoder was proposed as an alternative that learns continuous and distributed object-centric representations. However, it is only applicable to simple toy data. In this paper, we present Rotating Features, a generalization of complex-valued features to higher dimensions, and a new evaluation procedure for extracting objects from distributed representations. Additionally, we show the applicability of our approach to pre-trained features. Together, these advancements enable us to scale distributed object-centric representations from simple toy to real-world data. We believe this work advances a new paradigm for addressing the binding problem in machine learning and has the potential to inspire further innovation in the field.

Rotating Features for Object Discovery
Sindy Löwe
Phillip Lippe
Francesco Locatello Institute of Science and Technology Austria (ISTA)
AMLab University of Amsterdam
QUVA Lab University of Amsterdam
Max Welling AMLab University of Amsterdam
Abstract
The binding problem in human cognition, concerning how the brain represents and connects objects within a fixed network of neural connections, remains a subject of intense debate. Most machine learning efforts addressing this issue in unsupervised setting have focused on slot-based methods, which may be limiting due to their discrete nature and difficulty express uncertainty. Recently, the Complex AutoEncoder was proposed as an alternative that learns continuous and distributed object-centric representations. However, it only applicable to simple toy data. In this paper, we present Rotating Features, a generalization of complex- valued features to higher dimensions, and new evaluation procedure for extracting objects from distributed representations. Additionally, we show the applicability of our approach pre-trate -rrinadnddd features. Together, these advancements enable us scale distributed object-centric representations from simple toy to real- world data. We believe this work advances a new paradigm for addressing the binding problem in machine learning and has the potential to inspire further innovation in the field.
Introduction
Discovering and reasoning about objects is essential for human perception and cognition 8 al- lowing us to interact with our environment, reason about it, and adapt to new situations 42
43
To represent and connect symbol-like -ink entities such objects, our brain flexibly and dynamically combines distributed information. However, the binding problem remains heavily debated, question- ing how the brain achieves this within relatively fixed network of neural connections
28
Most efforts to address the binding problem machine learning center on slot-based methods 4. 23 27 33 41 46 64 88). These approaches divide their latent representations into "slots". creating a discrete separation of object representations within a single layer the architecture at some arbitrary depth. While highly interpretable and easy to use for downstream applications, this simplicity may limit slot-based approaches from representing the full diversity of objects. Their discrete nature makes it challenging to represent uncertainty in the separation of objects; and it remains unclear how slots may be recombined to learn flexible whote hierarchies [3i]. Finally, unlikely that the brain assigns entirely distinct groups of neurons each perceived object.
Recently, Löwe al. 48 proposed the Complex AutoFncoder (CAF), which learns continuous and distribured object-centric representations. Taking inspiration from neuroscience, it uses complex- valued activations to learn to encode feature information their magnitudes and object affiliation in their phase values. This allows the network to express uncertainty in its object separations and embeds
Correspondence to loowe.
gmail.com cow.iiiiddyggmi.oommmmmm
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Rotating Feature
Standard Feature
frot
Figure 1: Rotating Features. We propose extend standard features by an extra dimension across the entire architecture, including the input (highlighted in blue, here n = 3). We then set up the layer structure within fro in such a way that the Rotating Features' magnitudes m learn represent the presence of features, while their orientations learn to represent object affiliation.
object-centric representations in every layer of the architecture. However, due to its two-dimensional (i.e. complex-valueds features, is severely limited in the number object representations it can separate in its one-dimensional phase space; and due the lack of suitable evaluation procedure, it only applicable to single-channel inputs. Taken together, this means that the CAE is not scalable, and is only applicable to grayscale toy data containing up three simple shapes.
In this paper, we present a series of advancements for continuous and distributed object-centric representations that ultimately allow us to scale them from simple toy to real-world data. Our contributions are as follows:
1. We introduce Rotating Features, generalization of complex-valued features to higher dimensions. This allows our model represent a greater number of objects simultancously, and requires the development of new rotation mechanism.
2. We propose evaluation procedure for continuous object-centric representations. This allows us to extract discrete object masks from continuous object representations of inputs with more than one channel, such RGB images.
We show the applicability of our approach to features created by a pretrained vision transformer [1]. This enables Rotating Features to extract object-centric representations from real-world images.
We show how each of these improvements allows us to scale distributed object-centric representations to increasingly complicated data, ultimately making them applicable to real-world data. Overall, we are pioneering a new paradigm for field that has been largely focused on slot-based approaches, and we hope that the resulting richer methodological landscape will spark further innovation.
2 Neuroscientifie Motivation
Rotating Features draw inspiration from theories in neuroscience that describe how biological neurons might utilize their temporal firing patterns flexibly and dynamically bind information into coherent percepts, such as objects [1] 20 21
65 79]. In particular, we take inspiration from the It posits that the oscillating firing patterns of neurons (i.e. brain waves) give rise to two message types: the discharge frequency encodes the presence of features, and the relative timing of spikes encode feature binding. When neurons fire synchrony, their respective features processed jointly and thus bound together dynamically and flexibly.
50
53
temporal correlation hypothesis 166

---

FoodSeg
Pascal
Input
GT
Predictions
Figure Rotating Features applied to real-world images. By implementing series of advancements, we scale continuous and distributed object representations from simple toy real-world datasets.
Previous work in machine learning 48 theories to implement the same two message types using complex- complex-valued features: their magnitudes encode feature presence, and their relative phase differences encode feature binding. However, the one-dimensional orientation information of complex-valuee features constrains their expressivity, and existing approaches lack evaluation procedures that scale beyond single-channel inputs. As result, they are limited to grayscale toy datasets. To overcome these limitations, we propose several advancements for distributed and continuous object-centric representations, which we describe in detail in the next section.
58
59
61 has taken inspiration from these neuroscientific
3 Rotating Features
We create Rotating Features by augmenting standard features with additional dimensions and by creating a layer structure that parses the magnitude and multi-dimensional orientation of the resulting vectors separately. In the following, we describe how we represent, process, train and evaluate Rotating Features such that their magnitudes learn to represent the presence of features and their orientations learn to represent object affiliation.
3.1 Representing Rotating Features
In standard artificial neural network, each layer produces d-dimensional feature vector Zatandard Rd, where might be further subdivided to represent a particular structure of the feature map (e.g. d d h w. where c, h, represent the channel, height and width dimensions, respectively). To create Rotating Features, we extend each scalar feature within this d-dimensional feature vector into an n-ddiennsional vector, thus creating feature matrix Zrotating gxd Given the network structure described below, magnitude vector [Zrotating l| Rd (where |1-1|, the L.2-norm over the rotation dimension n) behaves similarly to standard feature vector and learns to represent the presence of certain features the input. The remaining n- dimensions represent the orientations of the Rotating Features, which the network uses as mechanism to bind features: features with similar orientations are processed together, while connections between features with different orientations are suppressed. Note, that we represent Rotating Features in Cartesian coordinates rather than spherical coordinates, as the latter may contain singularities that hinder the model's ability to learn good representations!
In an nt-dimensional Euclidean space, the spherical coordinates of a vector consist of a radial coordinate and v angular coordinates (P), ..., PP-1 with .... PP--2 [0, æ] and px E [0, This representation example, when a vector's 's radial
can lead singularities due to dependencies between the coordinates. component (i.e. magnitude) is zero, the angular coordinates can take any value without changing the underlying vector. As our network applies ReLU activations on the magnitudes, this singularity may occur regularly. hindering the network from training effectively.

---

3.2 Processing Rotating Features
We create layer structure fro that processes Rotating Features in such way that their magnituees represent the presence of certain features, while their orientations implement a binding mechanism. To achieve this, we largely follow the layer formulation of the CAE (48, but generalize it from complex-valued feature vectors >corplex E Cd to feature matrices of the form Zrotating E IRa both in the implementation and in the equations below; and introduce a new mechanism to rotate features. This allows us to process Rotating Features with 2 2.
Weights and Biases Given the input Zin Ruxda to a layer with n rotation dimensions and dy input feature dimensions, we apply a neural network layer parameterized with weights w E Rdax and biases b E Ruxda, where dour is the output feature dimension, in the following way:
+ = (zm) +
E ER
(I)
The function fw may represent different layer types, such as fully connected or convolutional layer. For the latter, weights may be shared across dm and daut appropriately. Regardless of the layer type, the weights are shared across the n rotation dimensions, while the biases not. By having separate biases, the model learns a different orientation offset for each feature, providing with the necessary mechanism to learn to rotate features. Notably, in contrast to the CAE, the model is not equivariant with respect to global orientation with this new formulation of the rotation mechanism: when there is change in the input orientation, the output can change freely. While it is possible to create an equivariant rotation mechanism for n 2 2 by learning or predicting the parameters of rotation matrix, preliminary experiments have demonstrated that this results in inferior performance.
Binding Mechanism We implement binding mech- anism that jointly processes features with similar ori- entations, while weakening the connections between
1.0
Output Magnitude
features with dissimilar orientations. This incentivizes features representing the same object to align, and fea- tures representing different objects to take on different orientations.
0.8
0.6
Maind
To implement this binding mechanism, we utilize the same weights as in Eq. (T) and apply them the magnitudes of the input features || E Rda:
l|vila
-1.0
-0.5
0.0
0.5
1.0
Cosine Similarity
x fw (2)
E &R
(2)
We then integrate this with our previous result by calcu- lating the average between X and the magnitude of y/:
Figure 3: Effect of the binding mechanism.
We start by randomly sampling two column vectors a, b E IR" with ||a||] = |lbll = Assuming dm = 3, d 1 and fw is linear layer, we set Zin = [a, a, b), weights
Mind =
0.5 || + 0.5 X E IR Iud
(3)
This binding mechanism results in features with sim- ilar orientations being processed together. while con- nections between features with dissimilar orientations are suppressed. Given a group of features of the same orientation, features of opposite orientations are masked out, and misaligned features are gradually scaled down. This effect has been shown by Löwe et al. [48) Reichert & Serre [1], and we illustrate it in Fig. 3 In contrast, without the binding mech- anism, the orientation of features does not influence their processing, the model has incentive to lever- age the additional rotation dimensions, and the model fails to learn object-centric representations (see Ap- pendix D.3). Overall, the binding mechanism allows the network create streams of information that it can process separately, which naturally leads to the emergence of object-centric representations.
w 3. 4 and biases = [0, .... OJT Subsequently, we plot the cosine similarity between a and b on the x-axis, and Moind and | on the y-axis, representing the magnitudes of the layer's output before ap- plying the activation function with (blue) and without (orange) the binding mechanism. Without the binding mechanism, misaligned features are effectively subtracted from the aligned features, resulting in smaller output magnitudes for | 2|2- The binding mecha- nism reduces this effect, leading to consis- tently larger magnitudes in Moind- In the most extreme scenario, features with oppo- site orientations (i.e., with a cosine similarity of -1) are masked by the binding mecha- nism, as the output magnitude (2) would be the same if Zin [a, 0).
Activation Function the final step, we apply an activation function to the output of the binding mecha-

---

nism Mannd, ensuring it yields positive-valued result. By rescaling the magnitude of a/ to the result- ing value and leaving its orientation unchanged, we obtain the output the layer Zout E Ruxdu:
Moul Ree..U (amaach))r
R
(4)
2b
Zuul
Mout
E Ranxda
(5)
1
3.3 Training Rotating Features
We apply Rotating Features to vision datasets which inherently lack the additional n dimensions. This section outlines the pre-processing of input images to generate rotating inputs and the post-processing of the model's rotating output to produce standard predictions, which are used for training the model.
Given a positive, real-valued input image x' Rxxxxx with e channels, height ht, and width w, we create the rotating feature input achieve this by setting the first dimension of equal to x', and assigning zeros to the remaining dimensions. Subsequently, we apply a neural network model fmodel that adheres the layer structure of foo to this input, generating the output features z = fmodel (x) E I 1=xddaat
[_xxexxx by incorporating
empty dimensions. We
To train the network using a reconstruction task, we set dawstel = c X h x w and extract the magnitude ||l||, E Rexixw from the output. We then rescale it using a linear layer fou with sigmoid activation function. This layer has weights Waul dimensions, and applies them separately to each channel c:
R and biases buot E Re that are shared across the spatial
& 8= Jon (122112 E Bexxxx
(6)
Finally, we compute the reconstruction loss L MSE(X' X) E R by comparing the input image x' to the reconstruction & using the mean squared error (MSE).
3.4 Evaluating Object Separation in Rotating Features
While training the network using the Rotating Features' magnitudes, their orientations learn to represent "objeectess"" in an unsupervised manner. Features that represent the same objects align, while those representing different objects take on distinct orientations. In this section, we outline how to process the continuous orientation values of the output Rotating Features z E Bexeexxx in order to generate a discrete object segmentation mask. This approach follows similar steps those described for the CAE [48), but introduces an additional procedure that enables the evaluation of Rotating Features when applied to inputs with 2 1.
As the first step, we normalize the output features such that their magnitudes equal one, mapping them onto the unit (hyper-)sphers This ensures that the object separation of features is assessed based on their orientation, rather than their feature values (i.e. their magnitudes):
z
xxxxxxx
Zeporn
|
Subsequently, we mask features with small magnitudes, they tend to exhibit increasingly random orientations, in a manner that avoids trivial solutions that may emerge on feature maps with e >> 1. For instance, consider image containing a red and a blue object. The reconstruction would be biased towards assigning small magnitudes to the color channels inactive for the respective objects. If we simply masked out features with small magnitudes by setting them to zero, would separate the objects based on their underlying color values, rather than their assigned orientations.
To avoid such trivial solutions, we propose to take weighted average of the orientations across channels, using the thresholded magnitudes of as weights:
if || > otherwise
Aijid evail
w
(8)
0
1= oval 2
Zeval
E Ruxhxx
(9)
D
W +
where we create the weights Weval by thresholding the magnitudes ||2||, for each channel E [1, .... d and spatial location j [I...., hb), I [L,... w| using the threshold t and then compute the weighted
5

---

GT
AE
CAE
RF
AE CAE
Rotating Features
MSB
0.004
0.002
0.000
2
2
3
8
AE
CAE
Rotating Features
1.0
ARLa6
0.8
0.7
0.8
(b)
Figure 4: Qualitative and quantitative performance comparison on the 4Shapes dataset. (a) In comparison to standard autoencoder (AE) and the Complex AutoEncoder (CAE), Rotating Features (RF) create sharper reconstructions (1st and 3rd row). Additionally, Rotating Features separate all four shapes, while the CAE does not (2nd and 4th row). (b) Larger rotation dimensions lead to better reconstruction performance (top) and better object discovery performance (bottom), with n 2 resulting in perfect separation of all shapes (mean + performance across four seeds).
average across channels c. Here, denotes the Hadamard product, is small numerical constant to avoid division by zero, and Weval is repeated across the rotation dimensions.
Lastly, apply clustering procedure Zeval and interpret the resulting discrete cluster assignment as the predicted object assignment for each feature location. Our experiments demonstrate that both k-means and agglomerative clustering achieve strong results, thus eliminating the need specify the number of objects in advance.
4 Experiments
In this section, evaluate whether proposed improvements enable distributed object-centric representations to scale from simple toy to real-world data. We begin by outlining the general settings common to all experiments, and then proceed to apply each proposed improvement to distinct setting. This approach allows us to isolate the impact of each enhancement. Finally, we will highlight some advantageous properties of Rotating Features. Our code is publicly available github.com//oeee wwtp://coo/ooeeee//ttttnnnngniie /ittee//t/g/FFattrree
General Setup We implement Rotating Features within a convolutional autoencoding architecture. Each model trained with batch-size of 64 for 10,000 to 100,000 steps, depending on the dataset, using the Adam optimizer 44). Our experiments are implemented in PyToorch [54) and run on single Nvidia GTX 1080Ti. See Appendix Cffor for more details on our experimental setup.
Evaluation Metries We utilize a variety of metrics to gauge the performance of Rotating Features and draw comparisons with baseline methods. Reconstruction performance is quantified using mean squared error (MSE). To evaluate object discovery performance, we employ Adjusted Rand Index (ARI) 34 57] and mean best overlap (MBO) [56] 64). ARI measures clustering similarity, where score of O indicates chance level and 1 denotes perfect match. Following standard practice in object discovery literature, we exclude background labels when calculating ARI (ARI-BG) and evaluate it instance-level masks. MBO is computed by assigning the predicted mask with the highest overlap to each ground truth mask, and then averaging the intersection-overuune
(IoU)
values of the resulting mask pairs. Unlike ARL MBO takes into account background pixels, thus

---

measuring the degree to which masks conform to objects. We assess this metric using both instance- level (MBO;) and class-level (MBO,) masks.
4.1 Rotating Features Can Represent More Objects
Setup We examine the ability of Rotating Features represent more objects compared to the CAE [88] using two datasets. First, we employ the 4Shapes dataset, which the CAE was previously shown to fail on. This grayscale dataset contains the same four shapes in each image (L, 4, V, O), Second, we create the 10Shapes dataset, featuring ten shapes per image. This dataset is designed to ensure the simplest possible separation between objects, allowing us to test the capacity of Rotating Features to simultancously represent many objects. To achieve this, employ diverse set of object shapes
(0) A, O with varying orientations and sizes) and assign each object a unique color and depth value.
Results As depicted in Fig. 4 Rotating Features improve in perfor- mance on the 4Shapes dataset as we increase the size of the rotation dimension n. For the CAE and Rotating Features with = 2, where the orientation information one-dimensional, the models struggle to distinguish all four objects. However, as the rotation dimension grows, performance enhances, and with n 6, the model perfectly separates all four objects. Additionally, observe that the recon- struction performance improves accordingly, and significantly sur- passes the same autoencoding architecture using standard features, despite having a comparable number of learnable parameters.
Input
Predictions
Moreover, Rotating Features are able to differentiate 10 objects si- multaneously, as evidenced by their ARI-BG score of 0.959 + 0.022 on the 10Shapes dataset (Fig. 5). In Appendix D.5 we provide a de- tailed comparison of Rotating Features with n and n = 10 ap- plied to images with increasing numbers of objects. Additionally, to determine if Rotating Features can distinguish even more objects, we conduct theoretical investigation in Appendix D.I In this analysis, we reveal that as we continue to add points to an ri-dimensional hy- persphere, the minimum cosine similarity achievable between these points remains relatively stable after the addition of the first ten points, provided that is sufficiently large. This implies that once algorithm separate ten points (or objects, in our case) on hypersphere, it should be scalable to accommodate more points. Conse- quently, the number of objects to separate is a critical hyperparameter of our model, in stark contrast to slot-based architectures.
Figure 5: Rotating Features learn to separate all ten ob- jects in the 10Shapes dataset.
4.2 Rotating Features Are Applicable to Multi-Channel Images
Setup We explore the applicability of Rotating Features and our proposed evaluation method to multi-channel inputs by creating an RGB(-D) version of the 4Shapes dataset. This variation contains the same four shapes per image as the original dataset, but randomly assigns each object color. We vary the number of potential colors, and create two versions of this dataset: an RGB version and an RGB-D version, in which each object is assigned a fixed depth value.
Results As illustrated in Fig. 6. Rotating Features struggle distinguish objects in the RGB version of the 4Shapes dataset when the number of potential colors increases. This appears to stem from their tendency to group different shapes together based shared color. In Appendix [.6] we observe the same problem when applying Rotating Features to commonly used object discovery datasets (multi- dSprites and CLEVR). When depth information added, this issue is resolved, and Rotating Features successfully separate all objects. This demonstrates that our proposed evaluation procedure makes Rotating Features applicable multi-channel inputs, but that they require higher-level information in their inputs to reliably separate objects.
4.3 Rotating Features Are Applicable to Real-World Images
Inspired by the previous experiment, which demonstrates that Rotating Features and our proposed evaluation procedure can applied to multi-channel inputs but require higher-level features for opti- mal object discovery performance, apply them to the features of a pretrained vision transformer.
7

---

Input
Predictions
RGB
RGB-D
1.0
ARI-BO
0.8
Input
0.6
RGB
RGB-D
5
Number of colors
Figure 6: Rotating Features on the 4Shapes RGB(-D) dataset. When applied to RGB images with more colors, object discovery performance ARI-BG degrades (right; mean + sem across seeds) as the Rotating Features tend to group objects of the same color together (left). By adding depth information the input (RGB-D), this problem can resolved, and all objects are separated as intended.
Setup Following Seitzer et al. 664, we utilize pretrained DINO model to generate higher- level input features. Then, we apply an autoencoding architecture with Rotating Features to these features. We test our approach on two datasets: the Pascal VOC dataset [99 and FoodSeg 103 benchmark for food image segmentation. On the Pascal VOC dataset, we do not compare ARI-BG scores, as we have found that a simple baseline significantly surpasses previously reported state-of- the-art results on this dataset (see Appendix D.4). On FoodSeg 103, we limit the evaluation of our model the MBO, score, as only contains class-level segmentation masks.
[85],
The qualitative results in Fig. 2 higgh light the effectiveness of Rotating Features in segmenting objects across both real-worid datasets. Table details the object discovery performance of Rotating Features compared to various models on the Pascal VOC dataset. The strongest performance this dataset is achieved by the DINOSAUR Transformer model [64), Slot Attention-based model with autore- gressive transformer decoder that is applied to setting
Results
Table Object discovery performance the Pas- cal VOC dataset (mean + sem across 5 seeds). Ro- tating Features' scores indicate good separation of objects on this real-world dataset. Baseline re- sults adapted from Seitzer et al. (44
Model
MBO, f
MBO,
Block Masks
0.247
0,259
Slot Attention
0.222 oue3
0.237 0.06
DINO pretrained features. Closer (i.c., without autoregressive models), the D/- NOSAUR MLP model combines Slot Attention with spatial broadcast decoder (MLP decoder) and achieves the second- -iighst performance ported literature terms MBO, and MBO, scores. The performance of Rotating Features, embedded in a comparatively simple convolu- tional autoencoding architecture, lies in between these Slot Attention-based models. Rotating Fea- tures also outperform a baseline that applies k-means directly to the DINO features (DINO k-means). showing that they learn more meaningful separation of objects. Finally, the performance gain of Rotating Features over the original CAE model applied to the DINO features (DINO CAE) highlights their improved object modelling capabilities. For reference, object discovery approaches applied di- rectly to the original input images (Slot Artention [46). SLATE 668, Rotating Features -DINO) strug- gle segment the objects in this dataset and achieve performances close Block Masks baseline that divides images into regular rectangles. To verify that Rotating Features work on more settings, we also apply them the FoodSeg 103 dataset. Here, they achieve an MBO, 0.484 0.002, signifi- cantly outperforming the block masks baseline (0.296 + 0.000). Overall, this highlights that Rotating Features learn to separate objects real-world images through unsupervised training; thus scaling continuous and distributed object representations from simple toy worldworld data for the first time.
SLATE
0,310 0,034
0.324 0.004
Rotating Features -DINO
0,282 oi00e
0.320 1.006
DINO k-means
0.363
0.405
DINO CAE
0.329 0,039
0.374 0,000
DINOSAUR Transformer
0.440 1. o.00s
0.512 00006
DINOSAUR MLP
0.395 00000
0.409 1,000
Rotating Features
0.407 oveos
0.460 0.001

---

Input
Prediction
Uncertainty
0.45
MBO.
0.40
0.35
0.40
MBO,
0.35
0.30
3
4
7
9
2
3
4
k
distance threshold
Figure 7: Clustering methods Pascal (mean sem across seeds). After training, we can efficiently compare various clustering methods and hyperparameters such as k-means (left) with different values of k and agglomerative clustering (right) with different distance thresholds.
Figure 8: Uncertainty maps of Rotating Features. We plot the distance of each normalized output feature its closest k-means cluster center, ing brighter pixels for larger distances. This high- lights uncertainty in areas of object overlap, in- correct predictions, and object boundaries.
4.4 Benefits of Rotating Features
Rotating Features present several advantageous properties: they learn to distinguish the correct number of objects automatically, they generalize well different numbers of objects than observed during training, they express uncertainty in their object separations, and they are very efficient to run. In this section, we delve into these properties detail.
Clustering Methods We can evaluate various clustering methods and hyperparameters very ef- ficiently Rotating Features model, they can be freely interchanged after training (Fig. 77- Through their continuous object assignments, Rotating Features learn to separate the correct number of objects automatically. By evaluating with agglomerative clustering, the requirement specify the number of objects to be separated can be eliminated completely, albeit with slightly reduced results.
Generalization Performance Rotat- ing Features can generalize beyond the number of objects observed during train- ing, even when the number of objects in the test images is unknown. To high- light this, we train a Rotating Features model and Slot Attention model [46] on a revised version of the 10Shapes dataset. This version includes the same ten unique shapes, but randomly selects six be included in each image. After training, we test the models on a range of variants of this dataset, displaying be- tween two and ten objects per image, As shown Fig. D when the number of ob- jects is known and the parameters of the models are accordingly, both Rotat- ing Features and SlotAttentio maintain a relatively stable performance across various numbers of objects per scene; with SlotAttention consistently outperforming the Rotating Features model. However, when the number of objects in the test images is unknown, we set the parameters of the model according to our best estimate given the training data, the performances of
# #bjject known
SA slots=#objects-11
0.9
RF k= #=#oojects--1
ARLBG
0.8
#Objects unknown
SA slots=7
0.7
RF km7
RF agglomeative
0.6
Number of Objects
Figure Testing the generalization performance of Rotat- ing Features (RF) and Slot Attention (SA). All models trained on images containing six objects. When the number of objects in the test images is unknown, only the Rotating Features model evaluated with agglomerative clustering maintains relatively stable performance across varying numbers objects.

---

the SlotAttention model and the Rotating Features model evaluated with k-means degrade consid- erably as the number of objects in the test images increases. This problem can be circumvented by evaluating Rotating Features with agglomerative clustering. After determining the distance thresh- old on the training dataset, this approach preserves fairly consistent performance across varying numbers of objects in each scene, considerably outperforming the other approaches that require the number of objects to be set in advance. In summary, our results suggest that Rotating Features can generalize beyond the number of objects observed during training, even when the number of objects in the test images is unknown.
Uncertainty Maps Since Rotating Features learn continuous object- representations, they can express and process uncertainty in their object separations. We evaluate this by plotting the L2 distance of each normalized output feature to the closest k-means center in Fig. [] This reveals higher uncertainty in areas of object overlap, object boundaries and incorrect predictions.
Training Time Rotating Features are very efficient to train. On a single Nvidia GTX 1080Ti, the presented Rotating Features model on the Pascal VOC dataset requires less than 3.5 hours of training. For comparison, the DINOSAUR models [64] were trained across eight Nvidia V100 GPUs.
5 Related Work
Numerous approaches attempt to solve the binding problem in machine learning 28 with most focusing on slot-based object-centric representations. Here, the latent representation within a single layer of the architecture is divided into "slots", creating discrete separation of object representations. The symmetries between slots can be broken in various ways: by enforcing an order on the slots
4 16 [8] by assigning slots to spatial coordinates T 3
6
7
9
45 63 by learning specialized
slots for different object types [32] 33 80 82). by using an iterative procedure [12 24 25 26
27
74 90 or by a combination of these methods
70 77 86]. Slot Attention 46 which
uses an iterative attention mechanism, has received increasing interest lately with many proposed
advancements [37 47
62
68
81 and extensions to the video domain [4 22 69 87
Recently, Seitzer et al. 64
proposed to apply Slot Attention to the features of pretrained DINO
model, successfully scaling to real-world datasets.
In comparison, there has been little work on continuous and distributed approaches for object discovery. A line of work has made use of complex-valued activations with similar interpretation of magnitude and orientation to our proposed Rotating Features, starting from supervised [5]] 89
and weakly supervised methods [58] 59 60). to unsupervised approaches [48 61]. However, they are only applicable to very simple. grayscale data. Concurrent work (73) proposes a contrastive learning method for the Complex AutoEncoder, scaling to simulated RGB datasets of simple, uniformly colored objects. With the proposed Rotating Features, new evaluation procedure and higher-level input features, we create distributed and continuous object-centric representations that are applicable to real-world data.
6 Conclusion
Summary We propose several key improvements for continuous and distributed object-centric representations, scaling them from toy to real-world data. Our introduction of Rotating Features, new evaluation procedure, and novel architecture that leverages pre-trained features has allowed for more scalable and expressive approach to address the binding problem in machine learning.
Limitations and Future Work Rotating Features create the capacity to represent more objects at once. However, this capacity can currently only be fully leveraged in the output space. As we show in Appendix D.2] within the bottleneck of the autoencoder, the separation of objects not pronounced enough, yet, and we can only extract up to two objects at time. To overcome this limitation, further research into novel inference methods using Rotating Features will be necessary. In addition, when it comes to object discovery results, Rotating Features produce slightly inferior outcomes than the state-of-the-art DINOSAUR autoregressive Transformer model, while still surpassing standard MLP decoders. Nonetheless, we hope that by exploring new paradigm for object-centric representation learning, we can spark further innovation and inspire researchers to develop novel methods that better reflect the complexity and richness of human cognition.
10

---

Acknowledgments and Disclosure of Funding
We thank Jascha Sohl-Dickstein and Sjoerd van Steenkiste for insightful discussions. Francesco Locatello worked this paper outside of Amazon. Sindy Löwe was supported by the Google PhD Fellowship.
References
[1] Titas Anciukevicius, Christoph H Lampert, and Paul Henderson. Object-centric image genera- tion with factored depths, locations, and appearances. arXiv preprint arXiv:2004.006-22 2020.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
[3] Federico Baldassarre and Hossein Azizpour. Towards self-supervised learning of global and object-centric representations. ICLR 2022 Workshop on the Elements of Reasoning: Objects,
Structure, and Causality, 2022.
[4] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and represen- tation. arXiv preprini 19Xivv:9001.11 2019.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties celf-supervised vision transformers. In International Conference on Computer VIsion (ICCV), pp. 9650-9660, 2021.
[6] Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes. The Journal of Machine Learning Research, 2200):11770-11885, 2021.
[7] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with con- volutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3412-3420, 2019.
[8] Stanislas Dehaene, How we learn: Why brains learn better than any machine... for now. Penguin, 2021.
[9] Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In International Conference Learning Representations (ICLR), 2020.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 248-255 Icee, 2009.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. International Conference Learning Representations (ICLR), 2021.
[12] Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems (NeurIPS), 34: 15.11-1566666 2021.
[13] Yilun Du, Kevin Smith, Tomer Ulman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised dis- covery of 3d physical objects from video. International Conference on Learning Representa- tions (ICLR), 2021.
[14] Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. Advances in Neural Information Processing Systems (NeurIPS), 55-28940-28954. 2022.
[15] Andreas K Engel, Pieter R Roelfsema, Pascal Fries, Michael Brecht, and Wolf Singer. Role of the temporal domain for response selection and perceptual binding. Cerebral Cortex (New York, NY: 1991). 76))577--582 1997.
[16] Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative scene inference and sampling with object-centric latent representations. International Conference Learning Representations (ICLR), 2020.

---

[17] Martin Engeleke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. Advances in Neural Information Processing Systems (NeurIPS), 34:8085-8099 2021.
[18] SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems (NeurIPS), pp. 3225-3233, 2016.
[19] M. Everingham, Van Gool, C. K. I. Williams, J. Winn, and Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
[20] Pascal Fries. A mechanism for cognitive dynamics: neuronal communication through neuronal coherence. Trends in Cognitive Sciences, 9 (10))474-480, 2005.
[21] Pascal Fries. Rhythms for cognition: communication through coherence. Neuron, 88(1):220- 235, 2015.
[22] Anand Gopalakrishnan, Kazuki Irie, Jürgen Schmidhuber, and Sjoerd van Steenkiste. Unsuper- vised learning of temporal abstractions with slot-based transformers. Neural Computation, 35 (4):593-626, 2023.
[23] Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael C Mozer, and Yoshua Bengio. Neural production systems. Advances in Neural Information Processing Systems (NewrIPS), 34:25673-25687, 2021.
[24] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. International Conference on Learning Representations (ICLR). 2021.
[25] Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4484-4492. 2016.
[26] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6691-6701, 2017.
[27] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In International Conference on Machine Learning (ICML). pp. 2424-2433, 2019.
[28] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05206 2020.
[29] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. International Conference on Learning Representations (ICLR), 2022.
[30] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In International Conference on Computer Vision (ICCV), pp. 991-998, 2011.
[31] Geoffrey E Hinton. How to represent part-whole hierarchies in neural network. Neural Computation, pp. 1-40, 2022.
[32] Geoffrey Hinton, Alex Krizhevsky, and Sida Wang. Transforming auto-encoders. In International Conference on Artificial Neural Networks (ICANN), pp. 44-51. Springer, 2011.
[33] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing. In International Conference on Learming Representations (ICLR), 2018.
[34] Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 2(1): 193-218, 1985.
[35] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, In International Conference on Machine Learning (ICML), pp. 448-456, 2015.
[36] Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In International Conference on Computer VIsion (ICCV), pp. 9865-9874, 2019.
12

---

[37]
Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object- learning with query optimization. In International Conference on Learning Representations (ICLR), 2023.
[38] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. arXiv preprini arXiv:2303. 10834, 2023.
[39] Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-objeet datasets. http://gittub.oo//depmm////
object datasets/, 2019.
[40] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations (ICLR), 2015.
[41] Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. International Conference on Learning Representations (ICLR), 2022.
[42] Kurt Kofika. Principles of gestalt psychology. Philosophy and Scientific Method, 32(8), 1935.
[43] Wolfgang Köhler. Gestalt psychology. Psychologische Forschung, 31(1):VVII-XXX. 1967.
[44] Adam R Kosiorek, Hyunjik Kim, and Danilo J Rezende. Conditional set generation with transformers. ICML. 2020 Object-Oriented Learning Workshop, 2020.
[45] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. International Conference on Learning Representations (ICLR), 2020.
[46] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances Neural Information Processing Systems (NeurIPS), 2020.
[47] Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. NeurlPS 2020 Workshop on Object Representa- tions for Learning and Reasoning, 2020.
[48] Sindy Löwe, Phillip Lippe, Maja Rudolph, and Max Welling, Complex-valued autoencoders for object discovery. Transactions on Machine Learning Research (TMLR), 2022.
[49] Pascal Mettes, Elise Van der Pol, and Cees Snoek. Hyperspherical prototype networks. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[50] Peter M Milner. A model for visual shape recognition. Psychological Review, 81(6):521, 1974.
[51] Michael C Mozer, Richard S Zemel, Marlene Behrmann, and Christopher KI Williams. Learning to segment images using dynamic feature binding. Neural Computation, ((5):650-665, 1992.
[52] Oleg R Musin and Alexey S Tarasov. The tammes problem for n= 14. Experimental Mathematics, 24(4):460-468, 2015.
[53] Agostina Palmigiano, Theo Geisel, Fred Wolf, and Demian Battaglia. Flexible information routing by transient synchrony. Nature Neuroscience, 107)::014-1022) 2017.
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. Advances in Neural Information Processing Systems 32, pp. 8024-8035, 8035, 2019.
[55] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 12:2825-2830, 2011.
[56] Jordi Pont-Tuset, Pablo Arbelaez, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE transactions on pattern analysis and machine intelligence, 122-:128-140, 2016.
[57] William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(336) 850- 1971.
13

---

[58] A Ravishankar Rao and Guillermo A Cecchi. An objective function utilizing complex sparsity for efficient segmentation in multi-layer oscillatory networks. International Journal of Intelligent Computing and Cybernetics, 2010.
[59] A Ravishankar Rao and Guillermo A Cecchi. The effects of feedback and lateral connections on perceptual processing: A study using oscillatory networks. In International Joint Conference on Neural Networks, pp- 1777-1184, 2011.
[60] A Ravishankar Rao, Guillermo A Cecchi, Charles C Peck, and James R Kozloski. Unsupervised segmentation with dynamical units. IEEE Transactions on Neural Networks, 19(1):168-182. 2008.
[61] David P Reichert and Thomas Serre. Neuronal synchrony in complex-valued deep networks. International Conference on Learning Representations (ICLR), 2014.
[62] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filipp Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems (NeurIPS), 35: 9512-9524, 2022.
[63] Adam Santoro, David Raposo, David G Barrett, Mateusz. Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4967-4976, 2017.
[64] Maximilian Seitzer, Max Hor, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl- Johann GaboneGa Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. International Conference Learning Representations (ICIR), 2023.
[65] Wolf Singer. Neuronal synchrony: versatile code for the definition of relations? Neuron, 24 (1):49-65, 1999.
[66] Wolf Singer. Distributed processing and temporal codes in neuronal networks. Cognitive Neurodynamics, 3(3):189-199) 2009.
[67] Wolf Singer and Charles M Gray. Visual feature integration and the temporal correlation hypothesis. Anmual Review of Neuroscience, 18(1):555-586, 1995.
[68] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. International Conference on Learning Representations (ICLR), 2022.
[69] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[70] Gautam Singh, Ycongbin Kim, and Sungjin Ahn. Neural systematic binder. In International Conference on Learning Representations (ICLR), 2023.
[71] Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Fredo Durand, Joshua Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint arXiv:2205.00 03399333.. 2022.
[72]
Elizabeth S Spelke. Principles of object perception. Cognitive science, 14(1):29-56, 1990.
[73] Aleksandar Stanié, Anand Gopalakrishnan, Kazuki Irie, and Jürgen Schmidhuber. Contrastive training of complex-valuee autoencoders for object discovery. arXiv preprint arXiv:230s. 15001, 2023.
[74] Karl Stelzner, Robert Peharz, and Kristian Kersting. Faster frtend-infer-eeeeee with tractable probabilistic models. In International Conference on Machine Learning (ICML), pp. 5966- 5975, 2019.
[75] Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01144 2021.
[76] Pieter Merkus Lambertus Tammes. On the origin number and arrangement the places of exit on the surface of pollen-grains. Recueil des travaux botaniques néerlandais, 27(1):1-84, 1930.
[77] Manuel Traub, Sebastian Otte, Tobias Menge, Matthias Karlbauer, Jannik Thuemmel, and Martin V Butz. Learning what and where: Disentangling location and identity tracking without supervision. In International Conference Learning Representations (ICLR), 2023.
14

---

[78] Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. scikit-image: Image pro- cessing in python. PeerJ, 2:e453, 2014.
[79] Christoph Von Der Malsburg and Werner Schneider. A neural cocktail-party processor. Biologi- cal Cybernetics, 54(1):29 40, 1986.
[80] Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. ICLR 2020
Workshop: Causal Learning for Decision Making, 2020.
[81] Ziyu Wang, Mike Zheng Shou, and Mengmi Zhang. Object-centric learning with eyclic walks between parts and whole. arXiv preprint arXiv:2302.00800 030022 2023.
[82] Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022.
[83] Max Wertheimer. Untersuchungen zur Lehre von der Gestalt. Psychologische Forschung, Forschhh 1(1):
47-58, 1922.
[84] Ross Wightman. Pytorch image models. pytorch- modee 2019.
https:://gihuub..co//rriggg
[85] Xiongwei Wu, Xin Fu, Ying Liu, Ec-Peng Lim, Steven CH Hoi, and Qianru Sun. A large-scale benchmark for food image segmentation. In Proceedings of ACM international conference on Multimedia, 2021.
[86] Yizhe Wu, Oiwi Parker Jones, and Ingmar Posner. Obpose: Leveraging canonical pose for object-centric scene inference in 3d. arXiv preprint arXiv:2206.0359), 2022.
[87] Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsu- pervised visual dynamics simulation with object-centric models. International Conference on Learning Representations (ICLR), 2023.
[88] Andrii Zadaianchuk, Matthaeus Kleindessner, Yi Zhu, Francesco Locatello, and Thomas Brox. Unsupervised semantic segmentation with self-supervised object-centric representations. International Conference on Learning Representations (ICLR), 2023.
[89] Richard $ Zemel, Christopher KI Williams, and Michael Mozer. Lending direction to neural networks. Neural Networks, 8(4):503-512, 1995.
[90] Ruixiang Zhang, Tong Che, Boris Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, and Liam Paull. Robust and controllable object-centric learning through energy-based models. International Conference Learning Representations (ICLR), 2023.
15

---

Appendix Rotating Features for Object Discovery
A Broader Impact
This research explores advancements in continuous and distributed object-centric representations, scaling them from simple toy real-world data. The broader impact includes:
Advancing Machine Learning: The proposed techniques expand the methodological landscape for object discovery, which may benefit many areas of machine learning, such computer vision, robotics, and natural language processing.
Biological Insight: By exploring biologically inspired approaches to resolve the binding problem in machine learning, may advance our understanding of object representations in the brain.
3. Improved Interpretability: The research promotes more structured representations, which may improve interpretability in machine learning models.
Scalability and World Applicability: The proposed approaches can be applied to complex, multi-dimensional data, such as RGB images, making continuous object-centric representations applicable real-world scenarios.
This research inspires further innovation and interdisciplinary collaboration. While we currently do anticipate any negative societal impacts, we recognize that the wide array of potential applications for object discovery methods could lead to unforeseen consequences. As such, we acknowledge that there may potential impacts that we are unable to predict at this time.
B Reproducibility Statement
In order ensure the reproducibility of our experiments, we have provided a comprehensive overview of the emploved model architectures, hyperparameters, evaluation procedures, datasets and baselines in Appendix [ Additionally, we published the code used to produce the main experimental results at www./oo///oeweeeeeeeeeeeee
To achieve stable and reproducible results, conducted all experiments using 4-5 different seeds. Training and evaluating the roughly 180 Rotating Features models highlighted across all our experi- single Nvidia GTX 1080Tí. Training and evaluating four Slot Attention models for the baseline in Section 4.4 took another 200 hours on the same device. This estimate does not take into account the time spent on hyperparameter search and various trials conducted throughout the research process.
ments took approximately 450 hours
C Implementation Details
C.1 Architecture and Hyperparameters
We implement Rotating Features within an autoencoding model. Here, will describe the precise architecture and hyperparameters used to train this model.
Architecture We provide detailed description of the autoencoding architecture used for all of our experiments in Table For the activation function (Eq. (4).. we employ Batch Normalization 335 in all convolutional layers, and Layer Normalization 2 in all fully connected layers.
Parameter Count As only the biases depend on the number of rotation dimensions n, the number of learnable parameters increases only minimally for larger values of n. For instance, in the models utilized for the 4Shapes dataset (Table 2 with 32), the total number of parameters amounts to 343,626 for 2 and 352,845 848 for n reflecting mere 2.79 7.7% increase.
16

---

Table 2: Autoencoding architecture with input dimensions h, w. and feature dimension d (values for d are specified in Tables 3 4 and 7). All fractions are rounded up. Layers denoted by are only present in the models that are applied to pretrained DINO features. Otherwise, the feature dimension of the final TransConv layer is h x w X C.
Layer
Feature Dimension
Kernel
Stride
Padding
(H x W x C)
Input / Output
Input
h c
Conv
h/2 x w/2 x
2
1/0
Conv
h/2 x w/2 x d
1
1/0
Conv
h/4 w/4 2d
2
1/0
Conv
h/4 X w/4x 2d
1
1/0
Encoder
Conv
h/8 X w/8 x 2d
2
1/0
Cony
h/8 X w/8 x 2d
1
1/0
Reshape
1 1 (h/8 w/8 * 2d)
Linear
1x1x 2d
Lincar
X I X (h/8 w/8 * 2d)
Reshape
h/8 w/8 x 2d
TransConv
h/4 w/4 x 2d
2
1/1
Conv
h/4 X w/4 x 2d
1
1/0
Decoder
TransConv
h/2 X w/2 x 2d
2
1/1
Conv
h1/2 X w/2
1
1/0
TransConv
h d h
2
1/1
Conv
1
1/0
Hyperparameters All hyperparameters utilized for models trained directly on raw input images are detailed in T while those for models trained on DINO features can be found in Table We initialize the weights Wout IR of fout by setting them to zero, and set biases bout E IR to one. The remaining parameters are initialized using PyTorch's default initialization methods.
Table 3: Hyperparameters for models with Rotating Features that are trained directly on the raw input images.
Dataset
4Shapes
4Shapes RGB(-D)
10Shapes
Training Steps
100k
100k
100k
Batch Size
64
64
64
LR Warmup Steps
500
500
500
Peak LR
0.001
0.001
0.001
Gradient Norm Clipping
0.1
0.1
0.1
Feature dim d Rotating dim
32
64
32
8
8
10
Image/Crop Size
32
32
48
Cropping Strategy
Full
Full
Full
Augmentations
5
11
VIT Encoder To generate higher-level input features of real-world images, we use pretrained vision transformer model [1] to preprocess the input images. Specifically, we employ a pretrained
ViT-B/16 or "vit base aass__aacch6 "24". This model features a token dimensionality of 768, 12 heads, a patch size of 16, and consists of 12 Transformer blocks. We use the final Transformer block's output input for the Rotating Features model, excluding the last layer norm and the CLSS token entry. All input images for the ViT have resolution of 224 x 224 pixels and are normalized based the statistics of the ImageNet dataset
DINO model [5] from the timm library [84]. denoted
17

---

Table 4: Hyperparameters for models with Rotating Features that trained DINO preprocessed
features.
Dataset
Pascal
FoodSeg 103
Training Steps
30k
30k
Batch Size
64
64
LR Warmup Steps
5000
5000
Peak LR
0.001
0.001
Gradient Norm Clipping
0.1
0.1
Feature dim
128
128
Rotating dim n
10
10
Image/Crop Size
224
224
Cropping Strategy
Random
Random
Augmentations
Random Horizontal Flip
Random Horizontal Flip and :00 rotation
k
4
5
10 with mean per channel (0.485, 0.456, 0.406) and standard deviation per channel (0.229, 0.224, 0.225). The output of the DINO model has dimension of 14 x 14 x 768 and we train our model to reconstruct this output. Before using it as input, we apply Batch Normalization and ReLU to ensure all inputs to the Rotating Features are positive. Overall, this setup closely follows that of Seitzer et al.
64 to ensure comparability of our results.
C.2
Evaluation Details
To evaluate the learned object separation of Rotating Features, we cluster their output orienta- tions. For this, normalize all output features onto the unit-hypersphere and subsequently ap- ply a weighted average that masks features with small magnitudes (Section (344). For this mask- ing, we use threshold of t 0.1 across all experiments. Subsequently, we apply clustering method to the weighted average. For k-means, we utilize the scikit-learn implementation ([55), skl sklearn. cluster. KMeans) with & parameters set to their default values. For agglomerative clustering, we also use the scikit-learn imple- mentation (sklearn. cluster. Aggl ongrammeat veClustering) with n_clusters set None, The distance thresholds set to for the results in Fig. 9 or a range of values as visualized in Figs. 7 and 22
the value specified in Tables 3 and 7 and all other
When utilizing DINO preprocessed features, our predicted object masks have size of 14 14, and we resize them to match the size of the ground-truth labels using the "nearest" interpolation mode. Subsequently, we smooth the mask by applying modal filter with disk-shaped footprint of radius |320/ 14] = 22 using the scikit-image functions skimage f filters. rank.modal and skimage www.gge..rrpooooggg.ooo oggy disk (78).
For calculating the final object discovery metrics, we treat unlabeled pixels and pixels from overlap- ping instances as background (i.e. they not evaluated).
C.3 Datasets
4Shapes The 4Shapes dataset comprises grayscale images of dimensions 32 32, each containing four distinct white shapes (α) A, V. O) on black background. The square has outer side-length of 13 pixels, the circle has an outer radius of 11 pixels and both isosceles triangles have a base-Length of 17 pixels and height of 9 pixels. All shapes have outline width of pixels. The dataset consists of 50,000 images the train set, and 10,000 images for the validation and test sets, respectively. All pixel values fall within the range [0.1].
4Shapes RGB(-D) The RGB(-D) variants of the 4Shapes dataset follow the same general setup, but randomly sample the color of each shape. To achieve this, we create sets of potential colors with varying sizes. Each set generated by uniformly sampling an offset value within the range [0,1], and subsequently producing different colors by evenly dividing the hue space, starting from this offset value. The saturation and value are set to one for all colors, and the resulting HSV color representations converted to RGB. To create the RGB-D variant, incorporate depth channel
18

---

to each image and assign a unique depth value within the range [0.1] to every object, maintaining equal distances between them.
10Shapes The 10Shapes dataset consists of RGB-D images with 48 x 48 pixels. Each image features unique shapes (M, A, O in varying orientations and sizes) on black background. This dataset includes the same square, circle, and triangle as the 4Shapes dataset, with additional variations: the triangle rotated by 1 degrees to form two more shapes, and the square is rotated by 45 to create a diamond shape. Another circle with an outer radius of 19 is also included. All these objects have an outline width of 3 pixels, with their centers remaining unfilled. The dataset also contains two smaller solid shapes: and a square with a side-length of Unique colors are assigned to each object by evenly dividing the hue space, starting from an offset value uniformly sampled from the range [0.1]. The saturation and value are set to one for all colors, and the resulting HSV color representations are converted to RGB. Furthermore, each object is given a distinct depth value within the range [0,1], ensuring equal distances between them. The dataset consists of 200,000 images in the train set, and 10,000 images for the validation and test sets, respectively, and all pixel values fall within the range [0.1].
wwwnward-poiningg triangle with a base-length of 11 and a height of 11,
Pascal VOC The Pascal VOC dataset 19 is widely-used dataset for object detection and semantic segmentation. Following the approach of Seitzer et al. [64, we train our model on the "trainaug" variant, an unofficial split consisting of 10,582 images, which includes 1,464 images from the original segmentation train set and 9, 18 images from the SBD dataset [30]. During training, we preprocess images by first resizing them such that their shorter side measures 224 pixels, followed by taking a random crop of size 224 x 224. Additionally, we apply random horizontal flipping. We evaluate the performance of Rotating Features on the official segmentation validation set, containing 1,449 images. Consistent with previous work [29 [36] 64], we use resolution of 320 x 320 pixels for evaluation. To achieve this, we resize the ground-truth segmentation mask such that the shorter side measures 320 pixels and then take center crop of dimensions 320 x 320. During evaluation, unlabeled pixels are ignored. To make our results comparable to Seitzer et al. 664, we evaluate the performance of our model across 5 seeds on this dataset.
FoodSeg103 The FoodSeg103 dataset 88 serves a benchmark for food image segmentation. comprising 4,983 training images and 2.135 test images. We apply the same preprocessing steps as described for the Pascal VOC dataset, with the addition of a random augmentation. Besides randomly horizontally flipping the training images, we also rotate them randomly by 90 in either direction.
C.4 Baselines
Complex AutoEncoder (CAE) The CAE 48
continuous and distributed object-centric repre- sentation learning approach that has been shown to work well on simple toy datasets. To reproduce this model, make of its open-source implementation
http:://ggttuu..cooooo
ComplexAutoonnoder
To establish standard autoencoder bascline, we employ the same architecture as described in Table 2 but replace the Rotating Features and layer structure of fru with standard features and layers. Similarly to the Rotating Features model, apply Batch Normalization / Layer Normalization before the ReLU non-linearity and maintain an input and output range of [0,1].
AutoEncoder
Block Masks To create the block masks baseline, we follow a similar approach as described by Seitzer et al. [64). First, we divide the image into a specific number of columns: for fewer than 9 blocks, we create 2 columns; for 9 to 15 blocks, we create 3 columns, and for create 4 columns. Next, we distribute the
more than 15 blocks, blocks evenly across the columns. If equal division is not possible, we first allocate blocks to all but the last column and then display the remaining blocks in the final column. See Fig. 10 for examples with 5 and 10 blocks.
Figure 10: Block masks with 5 and 10 blocks.
Other Baselines on the Pascal VOC dataset The results of the Slot Attention, SLATE, DINO
means, DINOSAUR Transformer, DINOSAUR MLP models on the Pascal VOC dataset are adapted from Seitzer et al. (4 To ensure the comparability of our findings, we adhere as closely as possible to their setup, utilizing the same dataset structure, preprocessing, and DINO pretrained model.
19

---

Minimum Cosine Similarity
1.0
n
0.5
2
4
0.0
6
8
-0.5
10
100
-1.0
5
10 15
20
Number of Points
Figure 11: Theoretical investigation into the capacity of Rotating Features to represent many objects simultaneously. We plot the minimal cosine similarity (y-axis) achievable between any pair of points when placing a specific number of points (X-axis) onto an n-dimensional hypersphere (mean + SEM across 10 seeds). We observe that starting from 10 points and with large enough, the minimal cosine similarity barely increases. Together with our experiment showing that Rotating Features are able to separate 10 objects, we thus infer that they should be able to scale to larger numbers of objects effectively.
Slot Attention To implement Slot Attention 46
we follow its open-source implemen-
tation h/ttps://gite thtbbt........ com/ coneech coceeaarccgggggleeeee rww.archeeeeeeeeeeeeeeeeeeeeeere wwweereeeereeeeeeeeeeereeeeeeeee slot_ attention We use a hidden dimension of 64 across the model and modulate the decoding structure as outlined Table[5]
research
Table 5: Architecture used for the Pratial-Broadcar Decoder the Slot Attention model.
Layer
Feature Dimension
Kernel
Stride
Padding Input / Output
Activation Function
(H x C) 6
Spatial Broadcast
64 x
Position Embedding
TransConv
12 x 12 x 64
5
2/1
ReLU
TransConv
24 x 24 x 64
5
2
2/1
ReLU
TransConv
48 48 64
5
2/1
ReLU
TransConv
x x 64
2/0
ReLU
TransConv
48 x 48 x
3
1/0
ReLU
D Additional Results
In this section, we provide additional results exploring the theoretical capacity Rotating Features to represent many objects simultaneously, introducing a strong baseline for object discovery on the Pascal VOC dataset, and showcasing object separation latent space of a model trained with Rotating Features. Furthermore, will provide additional quantitative and qualitative results for the experiments presented in the main paper.
D.1 High-dimensional Hyperspheres
We explore the theoretical capacity of Rotating Features to represent many objects simultaneously. Rotating Features learn continuous object-centric representations by separating object representations in orientation space. Depending on their orientation, the binding mechanism influences which features are processed together and which ones are masked out. This ultimately affects the number of objects that can be represented separately. Since this mechanism depends on the cosine similarity between features, we can analyze the capacity Rotating Features by investigating the minimum cosine similarity achievable for set of points on a hypersphere.
20

---

We investigate the behavior of the minimal cosine similarity between points on hypersphere by implementing an optimization procedure that closely follows Mettes al. 49 This is necessary due to the lack of an exact solution for optimally distributing variable number of points on higher- dimensional hyperspheres [52 [6]. To begin, we randomly sample points on an n-ddimensional hypersphere and then, for 10,000 steps, iteratively minimize the maximum cosine similarity between any pair of points using Stochastic Gradient Descent (SGD) with a learning rate of 0.1 and momentum of 0.9. In Fig. I1 we display the final results of this optimization process, which indicate the minimal achievable cosine similarity between any pair of points when a certain number of points are positioned on an n-dimensional hypersphere.
From this plot, we observe that regardless of n, we can always separate two points by a minimum cosine similarity -1. When n = 2, which corresponds to the setting of the Complex AutoEncoder, the minimal distance between points increases rapidly as more points are added. In this setting, when placing 20 points onto the hypersphere, the minimal cosine similarity approaches 1, indicating that points are closely packed together and barely separable from one another. With Rotating Features, we introduce a method to learn features with n 2 2. As evident from the figure, larger values of enable us to add more points to the hypersphere while maintaining small minimal cosine similarities. Finally, we observe that starting from ten points, if n. is sufficiently large, the minimal cosine similarity achievable between points barely increases as more points are added. Having demonstrated that the 10Shapes dataset
Rotating Features can separately represent ten objects with experiments (Section 4. the minimum cosine similarity achievable between 10 points is sufficient for the model to be able to separate them. Thus, we conclude that the number of objects to separate is not a critical hyperparameter for Rotating Features, in contrast to slot-based approaches.
D.2 Weakly Supervised Semantic Segmentation
In our main experiments, we demonstrate that Rotating Features create the capacity to represent more objects simultaneously (Section 4. Here, we explore the object separation within the bottleneck of an autoencoder utilizing Rotating Features. Our aim is to highlight that Rotating Features introduce object-centricity throughout the entire model and to examine whether their increased capacity is utilized in the latent space. To accomplish this, create a weakly supervised semantic segmentation setup. During the autoencoder training, we introduce a supervised task in which a binary classifier is trained to identify the classes of all objects present in an image. Subsequently, we align the orientations of the class predictions with those of the object reconstructions in order to generate semantic segmentation masks.
Method To train a network with Rotating Features on a classification task, we apply a linear layer fehnss following the structure of foa to the latent representation Zencnder E Raadte created by the encoder. This layer maps the latent dimension of the encoder dencoder to the number of possible object classes o:
Zelass = Selass (Zencoder)
) E R
(10)
Then, extract the magnitude from this output ||Zetass|/z 2 E R and rescale it using linear layer fout with sigmoid activation function before applying binary cross-entropy loss (BCH) comparing the predicted to the ground-truth object labels o:
Lelass BCE(fouu (l 12),
E I
(11)
We use the resulting loss value Letnss alongside the reconstruction loss L to train the network while weighting both loss terms equally.
After training, we leverage the object separation within the bottleneck of the autoencoder to assign each individual pixel a label representing its underlying object class. To do so, we first measure the cosine similarity between each pixel in the normalized rotating reconstruction Zeval E x (Eq. (8) and each value in Zelass for which a (z(lae) > 0.5. In other words, we calculate the cosine similarit for the values corresponding to object classes predicted be present in the current image. Subsequently, we label each pixel based on the class label of the value in Zelass that it exhibits the highest cosine similarity with. If the maximum cosine similarity to all values in Zelass is lower than a specific threshold (zero in our case), or if || 2 < 0.1, we assign the pixel to the background label. Note that this procedure does not require any clustering method to be employed, but directly utilizes the continuous nature of Rotating Features.
21

---

Table 6: Weakly supervised semantic segmentation (WSSS) results (mean + SEM across 4 seeds). While the latent representations within the bottleneck of an autoencoder trained with Rotating Features dataset in terms
can be used to create accurate semantic segmentation masks the MNISTI of IoU, the same approach fails when adding another object to the input images (MNIST2Shape) despite achieving strong object discovery (ARI-BG) and object classification results (Accuracy). This highlights that Rotating Features can be successfully employed in a weakly supervised semantic segmentation setting. but that the object separation within the bottleneck of the autoencoder is not pronounced enough, yet, to separate more than two objects.
Dataset
Method
IoU t
ARI-BG t
Accuracy 1
WSSS
0.785 = 0.011
0.926 = 0.077
0.996 0.000
ShapTe
Baseline
0.393 = 0.006
WSSS
0.230 +: 0,043
0.871 # 0.029
0.991 0,000
MNIST2Shape
Baseline
0.265 #: 0,006
We construct an autoencoder adhering to the architecture outlined Table 2 setting d = 64. The model is trained for 10,000 steps with batch size of 64 using the Adam optimizer [40]. The learning rate is linearly warmed up for the initial 500 steps of training to reach its final value of 0.001. This setup is applied to two datasets: MNISTIShapp and MNIST2Shape.
Setup
MNIST Shape contains 32 x 32 images, each featuring an MNIST digit and one of three possi- ble shapes (0, A, V) that share the same format with those in the 4Shapes dataset. Conversely, MNIST2Shape combines an MNIST digit with two of these three shapes per image. Both datasets contain 50,000 train images and 10,000 images for validation and testing.
The network trained to accurately reconstruct the input image while correctly classifying the MNIST digit, as well as the other shapes present. The setup described above enables the network to produce semantic segmentation masks, despite only being trained on class labels. We assess the performance of this weakly supervised semantic segmentation approach in terms of the intersection- over-union (IoU) score. For comparison purposes, we establish a baseline that generates unlabeled object masks according to the evaluation procedure presented in Section 3.4 using the outputs of the models under evaluation, and randomly assigns each object mask one of the predicted labels. For all results, overlapping areas between objects are masked and ignored during evaluation.
The results presented in Table 6 and Fig. 12 indicate that on the MNISTI Shape dataset, our approach with Rotating Features achieves strong weakly supervised semantic segmentation performance. It can accurately match objects separated in the latent space their representations in the output space. Thereby, it almost doubles the performance of the baseline that randomly assigns labels discovered objects. without requiring any clustering procedure. However, on the MNIST2Shape dataset, the weakly supervised semantic segmentation method falls short (see Table 6 and Fig. 13 even with a relatively large rotation dimension of = and despite achieving strong object discovery and object classification results (in terms of ARI-BG and accuracy scores). This demonstrates that while Rotating Features create the capacity represent more objects simultaneously, this capacity is not fully utilized within the bottleneck of the autoencoder. Since the object separation not pronounced enough, the current approach can only reliably separate representations of two objects in the latent space. However, the findings also highlight that Rotating Features create object- centric representations throughout the entire model, as well as potential new application for Rotating Features in weakly supervised semantic segmentation application with intriguing implications for interpretability.
Results
D.3 Ablation on Chi
We perform an ablation of the binding mechanism. In this analysis, we modify the Rotating Features model by substituting Moind with || 2 in Eq. (4). This effectively removes the binding mechanism. Then, we apply the adjusted model to the grayscale 4Shapes dataset. While the original model achieves an ARI-BG score of 0.987 + 0.003 on this dataset, the ablated model fails to learn any object separation (0.059 + 0.017). This result highlights the crucial role the binding mechanism plays in enabling Rotating Features to learn object-centric representations.
22

---

Input
GT
4
8
Predictions
4
8
Baseline
Figure 12: Weakly supervised semantic segmentation results on the MNISTI dataset. Since this is semantic segmentation task, colors between ground- truth and predicted objects need to match for strong performance. While this is the case for the predictions created by our proposed approach leveraging Rotating Features, the baseline randomly assigns labels to discovered objects, leading to mismatches between predicted and ground-truth object labels.
Input
P
GT
Predictions
Baseline
Weakly supervised semantic segmentation results the MNIST2Shape dataset. The baseline creates accurate object masks, albeit with wrongly predicted labels due to its random matching process. Our proposed approach leveraging Rotating Features, on the other hand, falls short in creating accurate semantic segmentations.
Figure 13:
23

---

1.0
0.015
2
ARI-BG
0.8
0.010
10
MSE
0.6
0.005
0.4
0.000
2
8 Number Objects
6
10
4
6
10
Number of Objects
Figure 14:
Rotating Features with n = [2. 10] on images containing varying numbers of objects
(mean = SEM across seeds). As the number of objects increases, the performance of Rotating Features deteriorates strongly when n = 2, but remains strong when n = 10.
D.4 ARI-BG Pascal VOC
We create a baseline that significantly surpasses all previously documented results in the literature in terms of the ARI-BG score the Pascal VOC dataset. To achieve this, we simply predict the same object label for all pixels in an image, resulting in an ARI-BG score of 0.499. For comparison, the highest reported ARI-BG in the literature for this dataset 0.296 (8I]. We attribute the remarkable performance of this baseline to the characteristics of the Pascal VOC dataset, which often comprises images featuring a large object dominating the scene. Simply predicting the same label across the entire image results in the same label being predicted across this large, central object, and since the ARI-BG score is not evaluated on background pixels, this leads to relatively high scores. However, for the MBO scores, this baseline yields results far below the state-of of-e-oo--in of-the-art, with 0.205 in MBO; and 0.247 in MBO,,. Consequently, we focus our evaluation on these scores this dataset.
D.5 Rotation Dimension vs. Number of Objects
We conduct experiment to assess how the choice of impacts the object discovery performance of a Rotating Features model for different numbers objects per scene. We accomplish this by creating variations of the 10Shapes dataset that contain between and ten objects. For each variant, the number of objects per image equals the total number of distinct objects throughout the respective dataset. As illustrated in Fig. [4, the object discovery performance significantly drops as the number of objects increases when n = 2. However, the performance remains consistently high when = 10. These findings indicate that the choice of becomes increasingly important the number of objects rises.
D.6 Multi-dSprites and CLEVR
We evaluate the performance the proposed Rotating Features model on the commonly used Multi- dSprites and CLEVR datasets [39], comparing it to its predecessor, the CAE model [48] The hyperparameters are listed in Table 7 and the outcomes are presented in Table g Note that since the original CAE was only applicable to grayscale images, combine it with proposed evaluation procedure to make it applicable to multi-channel input images, which we denote as CAE*.
The results indicate that the Rotating Features model significantly surpasses the performance of the CAE*, demonstrating good object separation on both datasets. However, the results still lack behind the state-of offee--ofttt of-the-art. The qualitative examples in Figs. 13 16 show that Rotating Features encounter same issue here, as we have seen with the colored 4Shapes dataset (Section 4.2): objects of same color tend to be grouped together. As demonstrated with the RGB-D version of the colored 4Shapes dataset and with our results using pretrained DINO features on real-wor sal-world images, this issue can be addressed by using higher-level input features.

---

Input
GT
Predictions
RF
Predictions CAE*
Figure 15: Qualitative performance comparison the 4Shapes dataset. Rotating Features (RF) create substantially better object separation than the CAE*, but fail separate objects of the same color (see right-most column).
Input
GT
Predictions
RF
Predictions C
Figure 16: Qualitative performance comparison the 4Shapes dataset. In contrast to the CAB*, the Rotating Features (RF) model learns to separate objects well. However, fails to separate objects of the same color (see last three columns).
25

---

Table 7: Hyperparameters for the Rotating Features models that trained the Multi-dSprites and CLEVR datasets.
Dataset
Multi-dSprites
CLEVR
Training Steps
100k
30k
Batch Size
16
64
LR Warmup Steps
5000
5000
Peak LR
0.001
0.001
Gradient Norm Clipping
0.1
0.1
Feature dim d
128
128
Rotating dim
10
10
Image/Crop Size
64
128
Cropping Strategy
Random
Random
Augmentations
k
6
5
Table 8: Performance of Rotating Features and CAE* on the Multi-dSprites and CLEVR datasets.
While the CAE* largely fails to separate objects these datasets, Rotating Features achieve strong results in terms of ARI-BG and MBO,,
Dataset
Model
MSE
ARI-BG 1
MBO;
Multi-dSprites
CAE*
3.540e-03
1.990-444
0.371
0.056
0.426
Rotating Features
7.082e-04
= 1.173--04
0.888 + n.015
0.863
0.011
CLEVR
CAE*
1.191c-03 1: 2.99% 0s
0,289 .. 0.042
0.234 0,074
Rotating Features
6.643e-04 0.564e-05
0.664
0.013
0.608
D.7
Additional Results
We provide supplementary qualitative and quantitative results for the experiments discussed in the main part of this paper. For the 4Shapes, 4Shapes RGB(-D), and 10Shapes datasets, additional qualitative results can be found Figs. to 19 with corresponding quantitative results detailed in Tables [o Additional qualitative results for the Pascal VOC dataset are presented in Fig. 20 and extra results for the FoodSeg103 dataset are provided Fig. 21 Lastly, we compare the performance of Rotating Features on the FoodSeg 103 dataset using k-means and agglomerative clustering with various hyperparameters in Fig. 22
26

---

Table 9: Detailed quantitative performance comparison on the 4Shapes (mean + SEM across seeds). As the number of rotation dimensions in the Rotating Features increases, both reconstruction (MSE) and object discovery (ARI-BG and MBO;) performance improve. This allows Rotating Features to significantly outperform the standard autoencoder (AE) and Complex AutoEncoder (CAE).
Model
MSE
ARI-BG 1
MBO; t
AE
5,492e-03 9.3036-444
CAE
3.435e-03
2.879
0.694 0:941
0.628 0,089
2
2.198c-03
1.110+
0,666 0,046
0.589 0.026
1.112c-03
2,698e
0,937 =: 0.018
0,861 0.020
Rotating
5,893e-04
4.350=05
0,944 0,016
0,908 0.012
5,439c-04 65984--5
0.975
0,934 0.006
Features
2.526e-04
1416e-11
0,991 DAKE
0.970 0,003
1.642e-04
0.992
0.974
1.360e-04
04
7.644-116
0.987
D:K0
0.968 0,008
Table 10: Detailed results the 4Shapes RGB(-D) datasets (mean + SEM across 4 seeds). As we increase the number of potential colors that objects may take on, the performance of Rotating Features worsens the RGB dataset. However, this effect can be mitigated when providing depth information for each object (RGB-D).
Dataset
Number of colors
MSE +
ARI-BG ?
MBO, 1
RGB
1
1.017e-05 +: 2.54e-06
0,982 : 0,009
0.874 0.008
RGB
2
1.597e-04
2.856-05
0.726 0.035
0.595 0.155
RGB
3
1.044e004 i: 6.547-066
0.850 : 0,008
0.767 0.016
RGB
4
1.166e-(4 5.85c-06
0.682 0.041
0.626 0.050
RGB
5
1.653e-04 + 3.65e-05
0.494 : 0.056
0.474 0:031
RGB-D
4.920e-06 + 5.421e007
1,000 = 0,000
0.875 0.015
RGB-D
2
4.693e-05- 16666-05
0.996 0.034
0.798 0.007
RGB-D 3
2.436e-05 + 3930c-06
0,936 = 0.029
0.785 0.17
RGB-D 4
4.562e-05 4.5676-06
0.909 0.039
0.759 0013
RGB-D 5
8.850e-05 + 3.780--15
0.922 + 0.040
0.727 0.014
Table 11: Detailed quantitative results on the 10Shapes dataset (mean + SEM across seeds). When is sufficiently large, Rotating Features learn separate all ten objects, highlighting their capacity to separate many objects simultaneously.
MSE
ARI-BG
MBO;
2
1.672e-02
1,893e
0.341
0,000
0.169 0013
10
1.918e-03 1.315e 04
0.959 0,022
0.589 0.099
27

---

GT
AE
CAE
Rotating Features
n= 2
4
Figure 17: Qualitative performance comparison on the 4Shapes dataset. Rotating Features with sufficiently large rotation dimension n create sharper reconstructions and significantly better object separations, despite similar numbers learnable parameters.
28

---

0
Input
0
GT
0
Predictions RGB
Predictions RGB-D
Figure 18: Rotating Features applied the 4Shapes RGB(-D) datasets. While the model fails to create accurate object separations when applied to the RGB version of this dataset, it can accurately distinguish objects when given additional depth information for each object (RGB-D). Nonetheless, it tends to group one of the four objects with the background.
Input
GT
Predictions
Figure 19: Rotating Features applied to the 10Shapes dataset. With a sufficiently large rotation dimension n, Rotating Features capable to separate all ten objects in this dataset, highlighting their increased capacity to represent many objects simultancously.
29

---

Input
GT
Predictions
Figure 20: Rotating Features applied to the Pascal VOC dataset. By leveraging higher-level input features created by a pretrained vision transformer, Rotating Features can learn to separate objects in real-world images.
Input
GT
Predictions
Figure 21: Rotating Features applied to the FoodSeg 103 dataset. By utilizing input features generated by a pretrained vision transformer, Rotating Features can learn to distinguish objects in real-world images.
0.45
MBO.
0.40
0.35
0.30
2
3
5
6
7
8
9
0.2
0.6
1.0
distance threshold
Figure 22: Clustering methods applied to Rotating Features after training on the FoodSeg100 dataset (mean + SEM across seeds). Since Rotating Features automatically learn separate the appropriate amount of objects, we can test various clustering methods and hyperparameters efficiently after training. While agglomerative clustering (right) achieves slightly worse results than k-means (left), eliminates the requirement to set the number of objects to be separated.
30


Reviews:
[{"review_id": "7Q2hhzEc9q", "reviewer": "Reviewer_e8db", "summary": "The paper presents a new approach for extracting objects from distributed representations, based on a binding mechanism called 'rotating features' that extends previous phase-based binding notions to a much higher dimension binding space, and avoids the use of separate slots for individual objects, showing promising results and scaling behavior with both toy and natural-image data sets.", "questions": "I found it a bit difficult to be sure I was seeing fair comparisons in table 1.  If I understand correctly, DINOSAUR MLP is much simpler than the Rotating Features CNN.  Is it slot based in some way?  \n\nI have found it difficult to understand how the binding mechanism described on p 4 works and I did not feel I got much out of figure 3.  perhaps the problem lies in my lack of understanding of the meaning of the superscripts on z_in.  Another source of confusion is the statement that the extra input dimensions of x (in line 134) are all set to 0, such that they don't seem to be capable of having any effect.  Why are these dimensions then needed?  I see that learned bias weights apply to the output of f_w and that there are R^(n x d_out) of these.  The references to Lowe et al and Reichert & Serre should not be my only source of an understanding of how the mechanism works.", "limitations": "The limitations as stated seem valid, and I look forward to seeing where attempts to address these limitations will lead.", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The exploration of alternatives to slot-based schemes, so prevalent in transformers and related contemporary architectures, for addressing how neural networks can successfully extract distinct, coherent representations of objects and other entities is an important contribution in itself.  For brain science, it is important because the brain is unlikely to have explicit slots of the kind that can be built into artificial networks.  For AI, this step may be equally important, as it allows for the possibility of more graded approaches to objecthood that could be important for capturing the kind of graded objecthood of many aspects of the natural world, and of avoiding some of the potential brittleness and arbitrariness of imposing slot-based approaches to structured objects made up of sub-objects.\n\nThe model seems to produce fairly impressive results compared to an alternative much more complex transformer models and beats other comparison models as well.  The excellent training time, and the possibility of avoiding explicit k-means clustering for segmentation, and the availability of uncertainly maps all seem like desirable properties of the model.", "weaknesses": "I have chosen to ask questions rather than express statements about weaknesses because I found that there were important features of the model and of the comparisons that I simply could not fully understand without going to source papers on DINOSAUR and the binding mechanism.  I do consider it a weakness of the paper that I was not able to understand these things better, and my rating and confidence would be increased further if these questions were addressed in the rebuttal and revision of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback from your thoughtful review. We would like to take this opportunity to address the two questions you posed:\n\n**I found it a bit difficult to be sure I was seeing fair comparisons in table 1. If I understand correctly, DINOSAUR MLP is much simpler than the Rotating Features CNN. Is it slot based in some way?**\n\nThe DINOSAUR MLP model is a slot-based model, which combines Slot Attention with a spatial broadcast decoder. In essence, it encodes the DINO preprocessed features into slots, and subsequently decodes each slot individually via an MLP decoder. The predicted slot mask from this decoder is then used to recombine the individual reconstructions, and to evaluate the object discovery performance of this model. We will clarify our description in the paper to make the comparison between the models more comprehensible.\n\n**I have found it difficult to understand how the binding mechanism described on p.4 works, and I did not feel I got much out of figure 3. perhaps the problem lies in my lack of understanding of the meaning of the superscripts on $z_{in}$. Another source of confusion is the statement that the extra input dimensions of $\\mathbf{x}$ (in line 134) are all set to 0, such that they don't seem to be capable of having any effect. Why are these dimensions then needed? I see that learned bias weights apply to the output of $f_{\\mathbf{w}}$ and that there are $\\mathbb{R}^{n \\times d_{\\text{out}}}$ of these.**\n\nThe rotating feature vector $z_{in} \\in \\mathbb{R}^{n \\times d_{\\text{in}}}$ has $n$ rotating dimensions and $d_{\\text{in}}$ feature dimensions, i.e., channels. Our data does not contain any rotating information, so we introduce the additional dimensions by padding the input with zeros. Your observation regarding the biases is correct - they serve as the model's mechanism that allows it to rotate the initial features, and thus to make use of the additional dimensions and to push them away from zero. Essentially, every feature dimension has the capacity to learn a distinct orientation offset through this bias.\n\nWithout the binding mechanism, the model would fail to learn to leverage these additional dimensions, as they inherently do not have a strong effect on the computations. The binding mechanism ensures that features with similar orientations are processed together, while features with dissimilar orientations are essentially masked out. This allows the network to create separate streams of information that it can process separately - which naturally leads to the emergence of object-centric representations.\n\nRegarding Figure 3, we agree that the superscripts on $z_{in}$ are ambiguous. We will amend the figure caption, improving notation and description, to the following:\n\nEffect of the binding mechanism. We start by randomly sampling two column vectors $\\mathbf{a}, \\mathbf{b} \\in \\mathbb{R}^n$ with $\\left\\lVert{\\mathbf{a}}\\right\\rVert_2 = \\left\\lVert{\\mathbf{b}}\\right\\rVert_2 = 1$. Assuming $d_{\\text{in}} = 3, d_{\\text{out}} = 1$ and $f_{\\mathbf{w}}$ is a linear layer, we set $z_{in} = \\left[ \\mathbf{a}, \\mathbf{a}, \\mathbf{b} \\right]$, weights $\\mathbf{w} = \\left[ \\frac{1}{3}, \\frac{1}{3}, \\frac{1}{3} \\right]^T $and biases $b = \\left[0, ..., 0\\right]^T$. Then, we plot $m_{bind}$ and $\\left\\lVert{\\psi}\\right\\rVert_2$ on the y-axis. These denote the magnitudes of the layer's output before the application of the activation function, with (blue) and without (orange) the binding mechanism. Without the binding mechanism, misaligned features are effectively subtracted from the aligned features, resulting in smaller output magnitudes. The binding mechanism masks out features with dissimilar orientations, reducing this effect and leading to consistently larger magnitudes in $m_{bind}$. In the most extreme scenario, features with opposite orientations (i.e., with a cosine similarity of -1) are cancelled out by the binding mechanism, as the output magnitude ($\\frac{2}{3}$) would remain the same if $z_{in} = \\left[ \\mathbf{a}, \\mathbf{a}, \\mathbf{0} \\right]$."}, {"review_id": "LJzWuvf2YA", "reviewer": "Reviewer_r4Rq", "summary": "This work addresses the problem of unsupervised object discovery. It seeks to remedy some limitations (primarily object storage capacity) of the recently introduced synchrony-based approach, CAE [1], and scales it to more visually complex scenes compared to CAE [1] which was only applied to simple grayscale (Shapes and MNIST) datasets. The key idea behind the proposed method is the extension of the number of feature dimensions used to manipulate “phase” (rotation) values associated with image features from ‘2’ in CAE to ‘n’ dimensions in RF. To control these ‘n’ dimensional rotation values use ‘n’ separate bias terms in every layer. Further, they apply their method on pre-trained DINO features to scale the grouping results to real-world datasets.\n", "questions": "1) Why do spherical coordinates lead to instabilities? Could not follow the explanation in footnote 1, specifically the point about ordering of the angles and under certain circumstances subsequent angles having no influence on the orientation. Could the authors clarify this point? A visualization would be greatly beneficial in this regard.\n\n2) Lines (242-243) : “On the Pascal VOC dataset, we do not compare ARI-BG scores, as we have found that a simple baseline significantly surpasses previously reported .….” . I didn’t understand the reasoning behind this choice.\n\n3) (related to weakness #2) Why can’t the proposed method segregate 2 objects of the same color but spatially well separated? If the network has learned a simple rule using just a single feature like color or shape to perform binding, that would be extremely limiting in the general case.\n\n4) (related to weakness #2) Could the authors show the grouping performance (FG-ARI, FULL-ARI) for the colorized ‘Shapes’ dataset without (simply using RGB images as inputs) the use of depth channel information? Have the authors performed an ablation to quantify how much the depth channel information assists the network to predict the object identities?  \n\n5) How does the proposed method work on colored multi-object datasets like Tetrominoes, Multi-dSprites or CLEVR that are the first benchmark suite typically used for evaluation in the object-centric literature?\n\n6) Table 1 (Pascal VOC dataset results), compared to the baseline models which use pre-trained DINO features in the encoder module (i.e. DINOSAUR Transformer/MLP) how much instance-level grouping has been achieved through the use of RF? It’s known that the DINO features already possess a high-level of specialization to semantic classes and therefore can perform semantic grouping (by inspecting the attention masks after conditioning on the CLS token). In this regard, SlotAttention and SLATE baselines cannot be considered like-for-like since they still use the pixel reconstruction objective and are trained from scratch. Short point being that how much instance-level grouping is being learned through the use of RF on top of the semantic-level grouping that is  already captured by the pretrained DINO features?", "limitations": "Authors discuss limitations of their method, in particular that the capacity to represent multiple objects is currently limited to the output space of the autoencoder and that the methods still lag behind slot-based methods such as DINOSAUR.\n\n\nReferences: \n\n[1] “Complex-Valued Autoencoders for Object Discovery”, Lowe et. al, TMLR 2022.\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The problem studied in this paper is well-motivated and important. \n\n2) The method presented is novel and as the model class it uses to perform binding (synchrony-based) has received significantly less attention in the literature compared to slot-based approaches.\n\n3) The paper is very well written.\n\n4) The experiments are well performed and the presented method compares favorably to the considered baselines.\n", "weaknesses": "1) The authors note that one of their three contributions is a new evaluation method. The phases are weighted but with binary values (0 or 1) which reduces to a difference in the averaging constant (N vs N-k) in the denominator of the weighted averaging operation. The threshold used to compute the weights is just a fixed value (i.e. 0.1 same as in CAE [1]) as opposed to being some learned value based on the magnitudes. So, in practice, how different/novel is this evaluation procedure compared to the one used by CAE to the extent that it can be deemed as a core technical contribution of this work?\n \n2) Depth masks provide a strong supervision signal for discovery of object grouping information. The problem of 2 (or more) objects having the same color being grouped together has been resolved through the use of such depth masks i.e. strong supervision (Figure 6 caption). \n\n3) Lines (244-245): “On FoodSeg103, we limit our evaluation of our model to the MBO_c score, …. ”. MBO_c score only measures semantic grouping not instance-level grouping. The pretrained DINO features already show a high level of specialization to semantic classes (attention maps when conditioning on CLS token). Therefore, this experiment does not really meaningfully test the instance-level grouping ability of the proposed method. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review. We would like to take the opportunity to respond to the questions and concerns that you have posed.\n\n### Weaknesses\n\n**1) How novel is this evaluation procedure compared to the one used by CAE?**\n\nWhile our proposed evaluation method closely resembles the CAE's evaluation method mathematically - merely requiring the addition of a weighted sum with binary weights - its conceptual significance cannot be understated: it avoids the trivial solutions described in lines 157-160 of the paper that would otherwise make a fair assessment of our approach on multi-channel images impossible.\n\n**2) Depth masks provide a strong supervision signal**\n\nProviding depth information is one possible way to prevent the Rotating Features from grouping together objects of the same color. Alternatively, we show that using features from a pretrained vision transformer works equally well. This approach, utilized by prior work for object discovery [1], is arguably more applicable in practice.\n\n**3) The experiment on FoodSeg103 does not meaningfully test the instance-level grouping ability.**\n\nUnfortunately, the FoodSeg103 dataset, like most semantic segmentation datasets, does not provide instance-level ground truth segmentation masks. This makes it challenging to meaningfully examine the instance-level grouping ability. Nonetheless, we believe that the class-level grouping performance of the Rotating Features on this dataset is noteworthy. For one, the DINO features exhibit a specialization towards the semantic class of a single object per scene when conditioned on the CLS token. In contrast, our experiments assess the performance in extracting multiple objects per scene, without feeding the CLS token into our network. Additionally, the DINO features are only used to set the input magnitudes, with input orientations being set to a fixed value. We evaluate the grouping learned by the output orientations, making our assessment independent of any grouping that may be extractable directly from the DINO features.\n\n### Questions\n\n**1) Why do spherical coordinates lead to instabilities?**\n\nWe will replace the example given in the paper with the following one: When a vector's magnitude is zero, the angular coordinates can take any value without changing the underlying vector. As our network applies ReLU activations on the magnitudes, this singularity may occur regularly, hindering the network from training effectively.\n\n**2) “On the Pascal VOC dataset, we do not compare ARI-BG scores”. I didn’t understand the reasoning behind this choice.**\n\nIn Appendix D.2, we provide a more in-depth discussion on this. \n\n**3) Why can’t the proposed method segregate 2 objects of the same color but spatially well separated?**\n\nIn its current form, Rotating Features do not include any inductive bias to enforce a spatial separation between object representations. We believe that this would be an interesting future direction to explore, as it could provide an alternative to our proposed solution that utilizes higher-level input features to perform binding on complex, real-world inputs.\n\n**4) Could the authors show the grouping performance for the colorized ‘Shapes’ dataset without depth information?**\n\nThese results can be found in Figure 6, and Appendix Table 7 and Figure 14. They indicate that the network benefits strongly from the additional depth information.\n\n**5) How does the proposed method work on colored multi-object datasets typically used in the object-centric literature?**\n\nIn response to the reviewer's suggestion, we evaluate the performance of the proposed Rotating Features model on the Multi-dSprites and CLEVR datasets, comparing it to its predecessor — the CAE model [2]. The outcomes are presented in the table below. Note that since the original CAE was only applicable to grayscale images, we combine it with our proposed evaluation procedure to make it applicable to multi-channel input images, which we denote as CAE*.\n\n|Dataset|Model|ARI-BG|\n|--|--|--|\n|Multi-dSprites|CAE*|0.371$\\pm$0.056|\n||Rotating Features|0.888$\\pm$0.015|\n|CLEVR|CAE*|0.289$\\pm$0.042|\n||Rotating Features|0.664$\\pm$0.013|\n\nThe results indicate that the Rotating Features model significantly surpasses the performance of the CAE*, demonstrating good object separation on these two datasets. However, the results still lack behind the state-of-the-art. The qualitative examples in Figure 3 of the PDF uploaded alongside the general response show that Rotating Features encounter the same issue here, as we have seen with the colored 4Shapes dataset: objects of the same color tend to be grouped together. As demonstrated with the RGB-D version of the colored 4Shapes dataset and with our results using pretrained DINO features on real-world images, this issue can be addressed by using higher-level input features.\n\n**6) How much instance-level grouping is being learned on top of the semantic-level grouping that is already captured by the pretrained DINO features?**\n\nAs described above, Rotating Features do not directly make use of the semantic specialization of the DINO features that may be found when conditioning on the CLS token. Nonetheless, we will add another baseline from the DINOSAUR paper [1] that allows for a direct comparison between the grouping that may be inherent to the DINO features and the grouping achieved by the Rotating Features model. This baseline applies k-means directly to the DINO preprocessed features and achieves an MBO$_i$ score of 0.363 and MBO$_c = 0.405$. For comparison, Rotating Features achieve scores of MBO$_i = 0.407$ and MBO$_c = 0.460$. We therefore conclude that the Rotating Features improve over the instance-level and semantic-level grouping that may be inherently present within the pretrained DINO features.\n\n---\n[1] Maximilian Seitzer, et al. Bridging the gap to real-world object-centric learning. ICLR, 2023.\n\n[2] Sindy Löwe, et al. Complex-valued autoencoders for object discovery. TMLR, 2022."}, {"review_id": "GDFf7zvVpF", "reviewer": "Reviewer_cH5o", "summary": "This work proposes a novel approach to unsupervised object discovery that does not depend on slots. Instead, the model uses an extra set of dimensions to code object assignment based on rotation, potentially allowing for a more flexible distributed form of object discovery than in standard slot-based approaches. The model is shown to perform well on toy tasks, with promising results on real-world images.", "questions": "I have listed some questions in the previous section.", "limitations": "There are no discernible negative societal impacts related to this work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This work presents an interesting new direction, substantially reconsidering the problem of object discovery relative to the now ubiquitous slot-based methods.\n- The method performs well on toy tasks, scaling to a relatively large number of objects, and also shows promising results on real-world images.\n- The method is significantly more efficient than popular slot-based methods.\n- The paper stimulates many interesting directions for future work.", "weaknesses": "I have a number of suggestions and questions that may help to further improve the paper:\n\n- A major potential advantage of the proposed approach is that it is more flexible than slot-based methods, specifically regarding the number of objects that can be segmented. The results in supplementary figure 10 suggest that, with a sufficient number of dimensions (e.g. ~10) a relatively large number of objects can be represented, and that even more objects can be represented without needing to add many more dimensions. Can this advantage over slot attention be empirically demonstrated? For instance, are there any instances in pascal VOC or FoodSeg that involve more objects than the number of slots in DINOSAUR? If so, it would be interesting to see whether rotating features outperforms DINOSAUR on those problems. Alternatively, controlled experiments with CLEVR, or even the Nshapes dataset, could be performed to demonstrate the superior representational capacity of rotating features over slot attention (especially when there are more objects than slots).\n- A desirable feature of slot attention is that it is permutation invariant, which allows for a dynamic binding of features to objects (i.e. by randomly initializing the slots and iteratively refining them through competition). Here, by contrast, particular features have learned biases toward particular orientations, does this interfere with the ability of the model to perform variable-binding in a dynamic manner? How would the model perform when tested on a greater number of objects than it was trained on (given that it has a sufficient number of dimensions to represent those objects)? In other words, is the method more efficient than slot attention because it is also less flexible?\n- Have the authors considered sharing the orientation across features at each location in a convolutional feature map? Intuitively, it seems that all features at a particular location should only be assigned to a single object, rather than having a unique assignment of each feature.\n- Have the authors investigated how the assignment of objects evolves across layers? I am wondering whether the competition that occurs over time in methods like slot attention is somehow distributed across layers in this model.\n- It would be informative to include an ablation of the binding mechanism. I also found the description of this mechanism somewhat counterintuitive. It is described as 'weakening the connections between features with dissimilar orientations', but it almost seems as if it does the exact opposite of this. The magnitude of the features in $\\chi$ is completely unrelated to the orientations of the inputs. Therefore, it seems that mixing $\\chi$ with $\\psi$ is only weakening the influence of orientation, allowing incoming features with dissimilar orientations (but high synaptic weight values) to exercise a greater influence on the magnitude of the feature representation. Can the authors add some additional explanation of this mechanism?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and constructive feedback. We welcome the opportunity to address the questions and concerns that have been brought up.\n\n**1 / 2) How would the model perform when tested on a greater number of objects than it was trained on? / Controlled experiments could be performed to demonstrate the superior representational capacity of rotating features over slot attention (especially when there are more objects than slots).**\n\nSee the general response above.\n\n**3) Have the authors considered sharing the orientation across features at each location in a convolutional feature map?**\n\nWe have experimented with this, but in preliminary results it generally performed worse. Intuitively, we believe it is helpful to have the possibility for each individual feature to have its own orientation (and therefore object binding), as this allows overlapping objects to be represented simultaneously within the same location. This becomes increasingly important as we reduce the feature map size in the architecture, and essential as we move through the fully-connected layers. \n\n**4) Have the authors investigated how the assignment of objects evolves across layers? I am wondering whether the competition that occurs over time in methods like slot attention is somehow distributed across layers in this model.**\n\nThis hypothesis could indeed be correct. In Appendix D.3, we have included an experiment that examines the object separation within the architecture's bottleneck (i.e., after the encoder). While we find a meaningful separation here, it does not seem to be prominent enough, yet, to be able to separate more than two objects at a time. Exploring whether this is indeed a result of some form of competition distributed across layers, and how to modify the network to achieve stronger object separation throughout the entire architecture, would be an intriguing direction for future research.\n\n**5) It would be informative to include an ablation of the binding mechanism. I also found the description of this mechanism somewhat counterintuitive.**\n\nFollowing the reviewer's suggestion, we performed an ablation of the binding mechanism. In this analysis, we modify the Rotating Features model by substituting $\\mathbf{m}_{\\text{bind}}$ with $\\left\\lVert{\\psi}\\right\\rVert_2$ in Equation 4. This effectively removes the binding mechanism ($\\chi$). Then, we apply the adjusted model to the grayscale 4Shapes dataset. While the original model achieves an ARI-BG score of $0.987 \\pm 0.003$ on this dataset, the ablated model fails to learn any object separation ($0.059 \\pm 0.017$). This result highlights the critical role the binding mechanism plays in enabling the Rotating Features to learn object-centric representations. We will include this experiment in the revised paper.\n\nThe model without binding mechanism fails to learn object-centric representations, as it cannot leverage the additional rotation dimensions. Without the binding mechanism, these dimensions inherently do not have a strong effect on the computations. The binding mechanism ensures that features with similar orientations are processed together, while features with dissimilar orientations are essentially masked out. This allows the network to create separate streams of information that it can process separately, which naturally leads to the emergence of object-centric representations.\n\nTo expand on this intuition, imagine the most extreme scenario where a feature is of the opposite orientation to a group of aligned features, as shown on the left-hand side of Figure 3 (cosine similarity = -1). Without the binding mechanism, the misaligned feature would effectively be subtracted from the aligned features, resulting in a smaller output magnitude (as shown for $\\left\\lVert{\\psi}\\right\\rVert_2$). The binding mechanism reduces this effect and results in a larger output magnitude (as shown for $\\mathbf{m}_{\\text{bind}}$). Effectively, the binding mechanism masks out the misaligned feature, as the output magnitude would be the same if the misaligned feature was replaced by a zero vector. We will amend our description in the paper to include this intuition."}, {"review_id": "C3Mbc3tORC", "reviewer": "Reviewer_VTkG", "summary": "CAEs promise to resolve some of the concerns of slot-based representation. They bring the promise of flexible object granularity, the promise of extracting part-whole hierarchies as per need, and faster training speeds. However, the original CAE was tested with grayscale images and on a rather small number of objects (2-3). This paper asks the question, *how can we scale CAEs* beyond this.\n\n**Approach.** Compared to classical neural networks, in the proposed model, all activations in the network are “tagged” with the binding information. For this, every scalar value of conventional vector representation is replaced with a $N$ length vector: the magnitude of this vector plays the role of classical activation while its orientation plays the role of capturing the binding information. Compared to original CAEs, the proposed model provides a generalization from having just 2 components (real and imaginary) to $N$ components, where $N$ can be arbitrarily large.\n\nIn **experiments**,\n1. The paper tests whether this change can help improve the capacity of CAEs in terms of the number of objects or not.\n2. The paper is also one of the first works to scale CAEs to real scenes.\n3. The paper also proposes a new procedure for extracting object segments from distributed representations.", "questions": "1. Is the layer choice important for segmentation? Do these patterns emerge also for the intermediate layers?\n2. I am curious what is the incentive for the model to assign different “phases” to different objects? In slot-based models, this role was played by the inherent capacity bottleneck of slots. I am curious about some insights about this.\n3. I could not entirely follow the motivation of Appendix D.3 “Object separation within the bottleneck of autoencoder utilizing Rotating Features…”", "limitations": "Yes, the paper discusses the limitations, but one should also consider the potential benefits of the CAE framework.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. (One of the) First successful attempts to make a CAE-like model scale to more objects and beyond simplistic grayscale images.\n2. The way weights and biases were applied seems to have been simplified (a welcome change) compared to the original CAEs. An ablation experiment highlighting this specific change for the case of $N=2$ could be useful though.\n3. Seems applicable to any data modality— demonstrated to some extent via the applicability of the model on RGB-D data in addition to RGB data.", "weaknesses": "1. One of the novelty seems to be the use of DINO pre-trained features for the first time in the context of CAEs. Therefore, a comparison with and without DINO pre-training would be useful to show in the main paper. \n2. Can original CAE be a baseline in the real-data experiment?\n3. The paper makes the jump from 4 object scenes directly to 10 object scenes. It may be useful to test perhaps a more gradual increase e.g., by also testing 7 object scenes and 13 object scenes, whether it places gradually more demand on the choice of $N$.\n4. It would be interesting to see the performance on a standard dataset like Tetris or CLEVR that was not deliberately designed for this paper. That being said, it need not outperform the previous slot-based methods here considering various other potential benefits of CAEs. \n5. At test time, can it handle a larger number of objects than shown during training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. We are taking this opportunity to address the concerns and inquiries raised.\n\n### Strengths\n**2) The way weights and biases were applied seems to have been simplified (a welcome change). An ablation experiment highlighting this specific change for the case of n=2 could be useful.**\n\nFigure 4 in the paper shows that when the CAE model is compared to Rotating Features with $n=2$, they achieve equivalent object discovery performances. This suggests that the modified rotation mechanism does not impact the performance significantly.\n\n### Weaknesses\n\n**1/2) A comparison with and without DINO pre-training would be useful to show in the main paper. / Can original CAE be a baseline in the real-data experiment?**\n\nIn line with the reviewer's suggestion, we ran two additional baselines on the Pascal VOC dataset. We firstly applied the Rotating Features model directly to the raw input images, referred to as RF-DINO. Secondly, we applied a CAE model to the DINO preprocessed features, while incorporating our novel evaluation procedure (CAE* +DINO). \n\n|Model|MBO$_i$|MBO$_c$|\n|--|--|--|\n|RF-DINO|0.282$\\pm$0.006|0.320$\\pm$0.006|\n|CAE* +DINO|0.329$\\pm$0.009|0.374$\\pm$0.010|\n|RF+DINO |0.407$\\pm$0.001|0.460$\\pm$0.001|\n\nThe results reveal that our proposed approach of applying Rotating Features to DINO features (RF+DINO) significantly outperforms both baselines. This highlights the importance of our contributions: generalizing the complex-valued features to higher dimensions and using DINO features as the input to our model are both essential to achieving competitive object discovery performance on real-world images. We will include these baselines in the revised paper.\n\n**3) The paper makes the jump from 4 object scenes directly to 10 object scenes. It may be useful to test a more gradual increase.**\n\nIn response to the reviewer's suggestion, we conduct a new experiment to assess how the choice of $n$ impacts the object discovery performance of a Rotating Features model depending on the number of objects in a scene. We accomplish this by creating variations of the 10Shapes dataset that contain between two and ten objects. For each variant, the number of objects per image equals the total number of distinct objects throughout the respective dataset.\n\nAs illustrated in Figure 2 in the PDF uploaded alongside the general response, the object discovery performance significantly drops as the number of objects increase when $n=2$. However, the performance remains consistently high when $n=10$. These findings indicate that the choice of $n$ becomes increasingly critical as the number of objects rises. We will include this experiment in the revised paper.\n\n**4) It would be interesting to see the performance on a standard dataset.**\n\nIn response to the reviewer's suggestion, we evaluate the performance of the proposed Rotating Features model on the Multi-dSprites and CLEVR datasets, comparing it to its predecessor — the CAE model [2]. The outcomes are presented in the table below. Note that since the original CAE was only applicable to grayscale images, we combine it with our proposed evaluation procedure to make it applicable to multi-channel input images, which we denote as CAE*.\n\n|Dataset|Model|ARI-BG|\n|--|--|--|\n|Multi-dSprites|CAE*|0.371$\\pm$0.056|\n||Rotating Features|0.888$\\pm$0.015|\n|CLEVR|CAE*|0.289$\\pm$0.042|\n||Rotating Features|0.664$\\pm$0.013|\n\nThe results indicate that the Rotating Features model significantly surpasses the performance of the CAE*, demonstrating good object separation on these two datasets. However, the results still lack behind the state-of-the-art. The qualitative examples in Figure 3 of the PDF uploaded alongside the general response show that Rotating Features encounter the same issue here, as we have seen with the colored 4Shapes dataset: objects of the same color tend to be grouped together. As demonstrated with the RGB-D version of the colored 4Shapes dataset and our results using pretrained DINO features on real-world images, this issue can be addressed by using higher-level input features.\n\n**5) At test time, can it handle a larger number of objects than shown during training?**\n\nSee the general response above.\n\n### Questions\n\n**1) Is the layer choice important for segmentation? Do these patterns emerge also for the intermediate layers?**\n\nFor object segmentation using Rotating Features, the current architectural layout suggests the output layer's representation as the most logical choice, as it has the highest spatial resolution. \n\nNevertheless, it is interesting to note that object-centric features also emerge in the intermediate layers of the architecture. In Appendix D.3, we examine the object separation within the autoencoder's bottleneck. This experiment reveals that it is possible to distinguish the representations of two separate objects within this layer's representation.\n\n**2) What is the incentive for the model to assign different phases to different objects?**\n\nThe binding mechanism enables the model to process information relatively independently of one another. If features of one object have orientations pointing in a different direction than features of another object, their features can be processed with little interference. This mechanism encourages the model to assign different orientations to features that it aims to process separately, which naturally leads to object-centric representations.\n\n**3) I could not entirely follow the motivation of Appendix D.3.**\n\nThis experiment is related to question 1 above, and investigates whether object separation also emerges in intermediate layers. Since it is not practical to evaluate the object segmentation performance here, we develop an alternative way to investigate object-centricity. Our approach in this case is a weakly-supervised semantic segmentation setup, which tests the alignment between the intermediate object representations and their counterparts at the output layer."}]

Author rebuttal:
We thank the reviewers for their constructive feedback. We are delighted to see the reviewers recognize that the exploration of alternatives to slot-based schemes as studied in our paper is important (e8db, r4Rq) and interesting (cH5o), and that our work may stimulate many interesting directions for future work (cH5o). Further, the reviewers have noted that the experiments are well performed (r4Rq), showing that the proposed Rotating Features achieve promising results (e8db, cH5o, VTkG) while being very efficient to train (e8db, CH5o). Additionally, reviewer r4Rq stated that our paper is very well written.

We want to use this general response to highlight an additional experiment that we have conducted following suggestions of reviewers cH4o and VTkG. This experiment shows that Rotating Features can generalize beyond the number of objects observed during training, even when the number of objects in the test images is unknown.

### Generalization to more objects

We conduct an experiment to assess the flexibility of Rotating Features in segmenting varying numbers of objects, thereby testing its ability to generalize beyond the number of objects observed during training. Additionally, this experiment evaluates the adaptability of the model when the number of objects in a scene is unknown.

To train the Rotating Features model, we make use of a modified version of the 10Shapes dataset. This version includes the same ten unique shapes as the original. However, only six of these shapes are randomly selected to appear in each image.

Post-training, we test the trained model with a range of variants of this dataset, each displaying between four and ten objects per image. We present the results in Figure 1 in the PDF uploaded alongside this general response. First, we observe that when the number of objects is known and $k$ for $k$-means is set accordingly, the performance of the Rotating Features model is best when fewer objects are present in each image, and decreases as more objects are added. However, considering the increase in difficulty when incorporating more objects in a scene of a fixed size, the model maintains a relatively stable performance across various numbers of objects per scene. Second, when the number of objects in a scene is not known, and we apply $k$-means with a fixed value of $k=7$ (corresponding to the number of objects observed during training, plus one for the background), performance degrades considerably the more the true number of objects deviates from the fixed value of $k$. However, this problem can be circumvented by using agglomerative clustering, and by setting the distance threshold on the training dataset to reflect the best performance when there are six objects in a scene. Using the same threshold across all test settings maintains consistent performance for varying numbers of objects in each scene, albeit slightly inferior to the $k$-means baseline, with $k$ representing the true number of objects.

In summary, our results suggest that Rotating Features can generalize beyond the number of objects observed during training, even when the number of objects in the test images is not known. We will include this experiment in the revised paper.

Meta-review: The reviewers were unanimous that this paper should be accepted – with strong support from two of them. There was unanimous agreement that the work is significant and that the method presented is both novel and interesting as an alternative to the popular slot-based approaches. There was also agreement that all experiments are well conducted and the presented method compares favorably to the considered baselines. The AC agrees and recommends the paper to be accepted.
Decision: Accept (oral)