Title: Rotating Features for Object Discovery
Venue: NeurIPS 2023
Accepted: yes
Keywords: Object Discovery

Object-Centric Representations

Structured Representation Learning
Abstract: The binding problem in human cognition, concerning how the brain represents and connects objects within a fixed network of neural connections, remains a subject of intense debate. Most machine learning efforts addressing this issue in an unsupervised setting have focused on slot-based methods, which may be limiting due to their discrete nature and difficulty to express uncertainty. Recently, the Complex AutoEncoder was proposed as an alternative that learns continuous and distributed object-centric representations. However, it is only applicable to simple toy data. In this paper, we present Rotating Features, a generalization of complex-valued features to higher dimensions, and a new evaluation procedure for extracting objects from distributed representations. Additionally, we show the applicability of our approach to pre-trained features. Together, these advancements enable us to scale distributed object-centric representations from simple toy to real-world data. We believe this work advances a new paradigm for addressing the binding problem in machine learning and has the potential to inspire further innovation in the field.

Rotating Features for Object Discovery

Abstract

The binding problem in human cognition, concerning how the brain represents and connects objects within a fixed network of neural connections, remains a subject of intense debate. Most machine learning efforts addressing this issue in an unsupervised setting have focused on slot-based methods, which may be limiting due to their discrete nature and difficulty to express uncertainty. Recently, the Complex AutoEncoder was proposed as an alternative that learns continuous and distributed object-centric representations. However, it is only applicable to simple toy data. In this paper, we present Rotating Features, a generalization of complexvalued features to higher dimensions, and a new evaluation procedure for extracting objects from distributed representations. Additionally, we show the applicability of our approach to pre-trained features. Together, these advancements enable us to scale distributed object-centric representations from simple toy to real-world data. We believe this work advances a new paradigm for addressing the binding problem in machine learning and has the potential to inspire further innovation in the field.

1 Introduction

Discovering and reasoning about objects is essential for human perception and cognition [8, 72], allowing us to interact with our environment, reason about it, and adapt to new situations [42, 43, 83]. To represent and connect symbol-like entities such as objects, our brain flexibly and dynamically combines distributed information. However, the binding problem remains heavily debated, questioning how the brain achieves this within a relatively fixed network of neural connections [28].

Most efforts to address the binding problem in machine learning center on slot-based methods [4, 23, 27, 33, 41, 46, 64, 88]. These approaches divide their latent representations into 'slots', creating a discrete separation of object representations within a single layer of the architecture at some arbitrary depth. While highly interpretable and easy to use for downstream applications, this simplicity may limit slot-based approaches from representing the full diversity of objects. Their discrete nature makes it challenging to represent uncertainty in the separation of objects; and it remains unclear how slots may be recombined to learn flexible part-whole hierarchies [31]. Finally, it seems unlikely that the brain assigns entirely distinct groups of neurons to each perceived object.

Recently, L¨ owe et al. [48] proposed the Complex AutoEncoder (CAE), which learns continuous and distributed object-centric representations. Taking inspiration from neuroscience, it uses complexvalued activations to learn to encode feature information in their magnitudes and object affiliation in their phase values. This allows the network to express uncertainty in its object separations and embeds

m

Rotating

Feature

Standard

Feature

m

m

...

n=3

f rot

object-centric representations in every layer of the architecture. However, due to its two-dimensional (i.e. complex-valued) features, it is severely limited in the number of object representations it can separate in its one-dimensional phase space; and due to the lack of a suitable evaluation procedure, it is only applicable to single-channel inputs. Taken together, this means that the CAE is not scalable, and is only applicable to grayscale toy data containing up to three simple shapes.

In this paper, we present a series of advancements for continuous and distributed object-centric representations that ultimately allow us to scale them from simple toy to real-world data. Our contributions are as follows:

We introduce Rotating Features , a generalization of complex-valued features to higher dimensions. This allows our model to represent a greater number of objects simultaneously, and requires the development of a new rotation mechanism.

We propose a new evaluation procedure for continuous object-centric representations. This allows us to extract discrete object masks from continuous object representations of inputs with more than one channel, such as RGB images.

We show the applicability of our approach to features created by a pretrained vision transformer [11]. This enables Rotating Features to extract object-centric representations from real-world images.

We show how each of these improvements allows us to scale distributed object-centric representations to increasingly complicated data, ultimately making them applicable to real-world data. Overall, we are pioneering a new paradigm for a field that has been largely focused on slot-based approaches, and we hope that the resulting richer methodological landscape will spark further innovation.

2 Neuroscientific Motivation

Rotating Features draw inspiration from theories in neuroscience that describe how biological neurons might utilize their temporal firing patterns to flexibly and dynamically bind information into coherent percepts, such as objects [15, 20, 21, 50, 53, 65, 79]. In particular, we take inspiration from the temporal correlation hypothesis [66, 67]. It posits that the oscillating firing patterns of neurons (i.e. brain waves) give rise to two message types: the discharge frequency encodes the presence of features, and the relative timing of spikes encode feature binding. When neurons fire in synchrony, their respective features are processed jointly and thus bound together dynamically and flexibly.

-FoodSeg -

-- Pascal --

Input

GT

Predictions

Previous work in machine learning [48, 58, 59, 60, 61] has taken inspiration from these neuroscientific theories to implement the same two message types using complex-valued features: their magnitudes encode feature presence, and their relative phase differences encode feature binding. However, the one-dimensional orientation information of complex-valued features constrains their expressivity, and existing approaches lack evaluation procedures that scale beyond single-channel inputs. As a result, they are limited to grayscale toy datasets. To overcome these limitations, we propose several advancements for distributed and continuous object-centric representations, which we describe in detail in the next section.

3 Rotating Features

We create Rotating Features by augmenting standard features with additional dimensions and by creating a layer structure that parses the magnitude and multi-dimensional orientation of the resulting vectors separately. In the following, we describe how we represent, process, train and evaluate Rotating Features such that their magnitudes learn to represent the presence of features and their orientations learn to represent object affiliation.

3.1 Representing Rotating Features

In a standard artificial neural network, each layer produces a d -dimensional feature vector z standard ∈ R d , where d might be further subdivided to represent a particular structure of the feature map (e.g. d = c × h × w , where c, h, w represent the channel, height and width dimensions, respectively). To create Rotating Features, we extend each scalar feature within this d -dimensional feature vector into an n -dimensional vector, thus creating a feature matrix z rotating ∈ R n × d . Given the network structure described below, its magnitude vector ∥ ∥ z rotating ∥ ∥ 2 ∈ R d (where ∥·∥ 2 is the L2-norm over the rotation dimension n ) behaves similarly to a standard feature vector and learns to represent the presence of certain features in the input. The remaining n -1 dimensions represent the orientations of the Rotating Features, which the network uses as a mechanism to bind features: features with similar orientations are processed together, while connections between features with different orientations are suppressed. Note, that we represent Rotating Features in Cartesian coordinates rather than spherical coordinates, as the latter may contain singularities that hinder the model's ability to learn good representations 1 .

3.2 Processing Rotating Features

We create a layer structure f rot that processes Rotating Features in such a way that their magnitudes represent the presence of certain features, while their orientations implement a binding mechanism. To achieve this, we largely follow the layer formulation of the CAE [48], but generalize it from complex-valued feature vectors z complex ∈ C d to feature matrices of the form z rotating ∈ R n × d both in the implementation and in the equations below; and introduce a new mechanism to rotate features. This allows us to process Rotating Features with n ≥ 2 .

Weights and Biases Given the input z in ∈ R n × d in to a layer with n rotation dimensions and d in input feature dimensions, we apply a neural network layer parameterized with weights w ∈ R d in × d out and biases b ∈ R n × d out , where d out is the output feature dimension, in the following way:

The function f w may represent different layer types, such as a fully connected or convolutional layer. For the latter, weights may be shared across d in and d out appropriately. Regardless of the layer type, the weights are shared across the n rotation dimensions, while the biases are not. By having separate biases, the model learns a different orientation offset for each feature, providing it with the necessary mechanism to learn to rotate features. Notably, in contrast to the CAE, the model is not equivariant with respect to global orientation with this new formulation of the rotation mechanism: when there is a change in the input orientation, the output can change freely. While it is possible to create an equivariant rotation mechanism for n ≥ 2 by learning or predicting the parameters of a rotation matrix, preliminary experiments have demonstrated that this results in inferior performance.

Binding Mechanism We implement a binding mechanism that jointly processes features with similar orientations, while weakening the connections between features with dissimilar orientations. This incentivizes features representing the same object to align, and features representing different objects to take on different orientations.

To implement this binding mechanism, we utilize the same weights as in Eq. (1) and apply them to the magnitudes of the input features ∥ z in ∥ 2 ∈ R d in :

Wethen integrate this with our previous result by calculating the average between χ and the magnitude of ψ :

This binding mechanism results in features with similar orientations being processed together, while connections between features with dissimilar orientations are suppressed. Given a group of features of the same orientation, features of opposite orientations are masked out, and misaligned features are gradually scaled down. This effect has been shown by L¨ owe et al. [48], Reichert & Serre [61], and we illustrate it in Fig. 3. In contrast, without the binding mechanism, the orientation of features does not influence their processing, the model has no incentive to leverage the additional rotation dimensions, and the model fails to learn object-centric representations (see Appendix D.3). Overall, the binding mechanism allows the network to create streams of information that it can process separately, which naturally leads to the emergence of object-centric representations.

Activation Function In the final step, we apply an activation function to the output of the binding mecha-

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

Cosine Similarity

0

.

4

0

.

6

0

.

8

1

.

0

Output Magnitude

m

bind

||

ψ

||

2

nism m bind, ensuring it yields a positive-valued result. By rescaling the magnitude of ψ to the resulting value and leaving its orientation unchanged, we obtain the output of the layer z out ∈ R n × d out :

3.3 Training Rotating Features

We apply Rotating Features to vision datasets which inherently lack the additional n dimensions. This section outlines the pre-processing of input images to generate rotating inputs and the post-processing of the model's rotating output to produce standard predictions, which are used for training the model.

Given a positive, real-valued input image x ′ ∈ R c × h × w with c channels, height h , and width w , we create the rotating feature input x ∈ R n × c × h × w by incorporating n -1 empty dimensions. We achieve this by setting the first dimension of x equal to x ′ , and assigning zeros to the remaining dimensions. Subsequently, we apply a neural network model f model that adheres to the layer structure of f rot to this input, generating the output features z = f model ( x ) ∈ R n × d model .

To train the network using a reconstruction task, we set d model = c × h × w and extract the magnitude ∥ z ∥ 2 ∈ R c × h × w from the output. We then rescale it using a linear layer f out with sigmoid activation function. This layer has weights w out ∈ R c and biases b out ∈ R c that are shared across the spatial dimensions, and applies them separately to each channel c :

Finally, we compute the reconstruction loss L = MSE ( x ′ , ˆ x ) ∈ R by comparing the input image x ′ to the reconstruction ˆ x using the mean squared error (MSE).

3.4 Evaluating Object Separation in Rotating Features

While training the network using the Rotating Features' magnitudes, their orientations learn to represent 'objectness' in an unsupervised manner. Features that represent the same objects align, while those representing different objects take on distinct orientations. In this section, we outline how to process the continuous orientation values of the output Rotating Features z ∈ R n × c × h × w , in order to generate a discrete object segmentation mask. This approach follows similar steps to those described for the CAE [48], but introduces an additional procedure that enables the evaluation of Rotating Features when applied to inputs with c ≥ 1 .

As the first step, we normalize the output features such that their magnitudes equal one, mapping them onto the unit (hyper-)sphere. This ensures that the object separation of features is assessed based on their orientation, rather than their feature values (i.e. their magnitudes):

Subsequently, we mask features with small magnitudes, as they tend to exhibit increasingly random orientations, in a manner that avoids trivial solutions that may emerge on feature maps with c > 1 . For instance, consider an image containing a red and a blue object. The reconstruction z would be biased towards assigning small magnitudes to the color channels inactive for the respective objects. If we simply masked out features with small magnitudes by setting them to zero, we would separate the objects based on their underlying color values, rather than their assigned orientations.

To avoid such trivial solutions, we propose to take a weighted average of the orientations across channels, using the thresholded magnitudes of z as weights:

where we create the weights w eval by thresholding the magnitudes ∥ z ∥ 2 for each channel i ∈ [1 , ..., c ] and spatial location j ∈ [1 , ..., h ] , l ∈ [1 , ..., w ] using the threshold t and then compute the weighted

GT

AE

CAE

RF

(a)

1

0.000

0.002

0.004

0.006

MSE

AE

2

CAE

2

3

4

5

6

7

8

n

Rotating Features

1

0.6

0.7

0.8

0.9

1.0

ARI-BG

AE

2

CAE

2

3

4

5

6

7

8

n

Rotating Features

(b)

average across channels c . Here, ◦ denotes the Hadamard product, ε is a small numerical constant to avoid division by zero, and w eval is repeated across the n rotation dimensions.

Lastly, we apply a clustering procedure to z eval and interpret the resulting discrete cluster assignment as the predicted object assignment for each feature location. Our experiments demonstrate that both k -means and agglomerative clustering achieve strong results, thus eliminating the need to specify the number of objects in advance.

4 Experiments

In this section, we evaluate whether our proposed improvements enable distributed object-centric representations to scale from simple toy to real-world data. We begin by outlining the general settings common to all experiments, and then proceed to apply each proposed improvement to a distinct setting. This approach allows us to isolate the impact of each enhancement. Finally, we will highlight some advantageous properties of Rotating Features. Our code is publicly available at github.com/loeweX/RotatingFeatures.

General Setup We implement Rotating Features within a convolutional autoencoding architecture. Each model is trained with a batch-size of 64 for 10,000 to 100,000 steps, depending on the dataset, using the Adam optimizer [40]. Our experiments are implemented in PyTorch [54] and run on a single Nvidia GTX 1080Ti. See Appendix C for more details on our experimental setup.

Evaluation Metrics We utilize a variety of metrics to gauge the performance of Rotating Features and draw comparisons with baseline methods. Reconstruction performance is quantified using mean squared error (MSE). To evaluate object discovery performance, we employ Adjusted Rand Index (ARI) [34, 57] and mean best overlap (MBO) [56, 64]. ARI measures clustering similarity, where a score of 0 indicates chance level and 1 denotes a perfect match. Following standard practice in the object discovery literature, we exclude background labels when calculating ARI (ARI-BG) and evaluate it on instance-level masks. MBO is computed by assigning the predicted mask with the highest overlap to each ground truth mask, and then averaging the intersection-over-union (IoU) values of the resulting mask pairs. Unlike ARI, MBO takes into account background pixels, thus

measuring the degree to which masks conform to objects. We assess this metric using both instancelevel (MBO i ) and class-level (MBO c ) masks.

4.1 Rotating Features Can Represent More Objects

Setup We examine the ability of Rotating Features to represent more objects compared to the CAE [48] using two datasets. First, we employ the 4Shapes dataset, which the CAE was previously shown to fail on. This grayscale dataset contains the same four shapes in each image ( □ , △ , ▽ , ⃝ ). Second, we create the 10Shapes dataset, featuring ten shapes per image. This dataset is designed to ensure the simplest possible separation between objects, allowing us to test the capacity of Rotating Features to simultaneously represent many objects. To achieve this, we employ a diverse set of object shapes ( □ , △ , ⃝ with varying orientations and sizes) and assign each object a unique color and depth value.

Results As depicted in Fig. 4, Rotating Features improve in performance on the 4Shapes dataset as we increase the size of the rotation dimension n . For the CAE and Rotating Features with n = 2 , where the orientation information is one-dimensional, the models struggle to distinguish all four objects. However, as the rotation dimension grows, performance enhances, and with n = 6 , the model perfectly separates all four objects. Additionally, we observe that the reconstruction performance improves accordingly, and significantly surpasses the same autoencoding architecture using standard features, despite having a comparable number of learnable parameters.

Moreover, Rotating Features are able to differentiate 10 objects simultaneously, as evidenced by their ARI-BG score of 0.959 ± 0.022 on the 10Shapes dataset (Fig. 5). In Appendix D.5, we provide a detailed comparison of Rotating Features with n = 2 and n = 10 applied to images with increasing numbers of objects. Additionally, to determine if Rotating Features can distinguish even more objects, we conduct a theoretical investigation in Appendix D.1. In this analysis, we reveal that as we continue to add points to an n -dimensional hypersphere, the minimum cosine similarity achievable between these points remains relatively stable after the addition of the first ten points,

Input

Predictions

provided that n is sufficiently large. This implies that once an algorithm can separate ten points (or objects, in our case) on a hypersphere, it should be scalable to accommodate more points. Consequently, the number of objects to separate is not a critical hyperparameter of our model, in stark contrast to slot-based architectures.

4.2 Rotating Features Are Applicable to Multi-Channel Images

Setup We explore the applicability of Rotating Features and our proposed evaluation method to multi-channel inputs by creating an RGB(-D) version of the 4Shapes dataset. This variation contains the same four shapes per image as the original dataset, but randomly assigns each object a color. We vary the number of potential colors, and create two versions of this dataset: an RGB version and an RGB-D version, in which each object is assigned a fixed depth value.

Results As illustrated in Fig. 6, Rotating Features struggle to distinguish objects in the RGB version of the 4Shapes dataset when the number of potential colors increases. This appears to stem from their tendency to group different shapes together based on shared color. In Appendix D.6, we observe the same problem when applying Rotating Features to commonly used object discovery datasets (multidSprites and CLEVR). When depth information is added, this issue is resolved, and Rotating Features successfully separate all objects. This demonstrates that our proposed evaluation procedure makes Rotating Features applicable to multi-channel inputs, but that they require higher-level information in their inputs to reliably separate objects.

4.3 Rotating Features Are Applicable to Real-World Images

Inspired by the previous experiment, which demonstrates that Rotating Features and our proposed evaluation procedure can be applied to multi-channel inputs but require higher-level features for optimal object discovery performance, we apply them to the features of a pretrained vision transformer.

Input

Predictions

RGB

RGB-D

1

2

3

4

5

Number of colors

0.6

0.8

1.0

ARI-BG

Input

RGB

RGB-D

Setup Following Seitzer et al. [64], we utilize a pretrained DINO model [5] to generate higherlevel input features. Then, we apply an autoencoding architecture with Rotating Features to these features. We test our approach on two datasets: the Pascal VOC dataset [19] and FoodSeg103 [85], a benchmark for food image segmentation. On the Pascal VOC dataset, we do not compare ARI-BG scores, as we have found that a simple baseline significantly surpasses previously reported state-ofthe-art results on this dataset (see Appendix D.4). On FoodSeg103, we limit the evaluation of our model to the MBO c score, as it only contains class-level segmentation masks.

Results The qualitative results in Fig. 2 highlight the effectiveness of Rotating Features in segmenting objects across both real-world datasets. Table 1 details the object discovery performance of Rotating Features compared to various models on the Pascal VOC dataset. The strongest performance on this dataset is achieved by the DINOSAUR Transformer model [64], a Slot Attention-based model with an autoregressive transformer decoder that is applied to DINO pretrained features. Closer to our setting (i.e., without autoregressive models), the DINOSAUR MLP model combines Slot Attention with a spatial broadcast decoder (MLP decoder) and achieves the second-highest performance reported in literature in terms of MBO c and MBO i scores. The performance of Rotating Features , embedded in a comparatively simple convolu-

tional autoencoding architecture, lies in between these two Slot Attention-based models. Rotating Features also outperform a baseline that applies k -means directly to the DINO features ( DINO k -means ), showing that they learn a more meaningful separation of objects. Finally, the performance gain of Rotating Features over the original CAE model applied to the DINO features ( DINO CAE ) highlights their improved object modelling capabilities. For reference, object discovery approaches applied directly to the original input images ( Slot Attention [46], SLATE [68], Rotating Features -DINO ) struggle to segment the objects in this dataset and achieve performances close to a Block Masks baseline that divides images into regular rectangles. To verify that Rotating Features work on more settings, we also apply them to the FoodSeg103 dataset. Here, they achieve an MBO c of 0 . 484 ± 0 . 002 , significantly outperforming the block masks baseline ( 0 . 296 ± 0 . 000 ). Overall, this highlights that Rotating Features learn to separate objects in real-world images through unsupervised training; thus scaling continuous and distributed object representations from simple toy to real-world data for the first time.

0

.

35

0

.

40

0

.

45

MBO

c

2

3

4

5

6

7

8

9

k

0

.

30

0

.

35

0

.

40

MBO

i

1

2

3

4

distance threshold

4.4 Benefits of Rotating Features

Rotating Features present several advantageous properties: they learn to distinguish the correct number of objects automatically, they generalize well to different numbers of objects than observed during training, they express uncertainty in their object separations, and they are very efficient to run. In this section, we delve into these properties in detail.

Clustering Methods We can evaluate various clustering methods and hyperparameters very efficiently on a Rotating Features model, as they can be freely interchanged after training (Fig. 7). Through their continuous object assignments, Rotating Features learn to separate the correct number of objects automatically. By evaluating with agglomerative clustering, the requirement to specify the number of objects to be separated can be eliminated completely, albeit with slightly reduced results.

Generalization Performance Rotating Features can generalize beyond the number of objects observed during training, even when the number of objects in the test images is unknown. To highlight this, we train a Rotating Features model and a Slot Attention model [46] on a revised version of the 10Shapes dataset. This version includes the same ten unique shapes, but randomly selects six to be included in each image. After training, we test the models on a range of variants of this dataset, displaying between two and ten objects per image. As shown in Fig. 9, when the number of objects is known and the parameters of the models are set accordingly, both Rotating Features and SlotAttention maintain a relatively stable performance across

2

4

6

8

10

Number of Objects

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

ARI-BG

#Objects known

SA slots=#objects+1

#Objects unknown

SA slots=7

RF agglomerative

#Objects known

SA slots=#objects+1

#Objects unknown

SA slots=7

RF agglomerative

various numbers of objects per scene; with SlotAttention consistently outperforming the Rotating Features model. However, when the number of objects in the test images is unknown, and we set the parameters of the model according to our best estimate given the training data, the performances of

Input

Prediction

Uncertainty

the SlotAttention model and the Rotating Features model evaluated with k -means degrade considerably as the number of objects in the test images increases. This problem can be circumvented by evaluating Rotating Features with agglomerative clustering. After determining the distance threshold on the training dataset, this approach preserves a fairly consistent performance across varying numbers of objects in each scene, considerably outperforming the other approaches that require the number of objects to be set in advance. In summary, our results suggest that Rotating Features can generalize beyond the number of objects observed during training, even when the number of objects in the test images is unknown.

Uncertainty Maps Since Rotating Features learn continuous object-centric representations, they can express and process uncertainty in their object separations. We evaluate this by plotting the L2 distance of each normalized output feature to the closest k -means center in Fig. 8. This reveals higher uncertainty in areas of object overlap, object boundaries and incorrect predictions.

Training Time Rotating Features are very efficient to train. On a single Nvidia GTX 1080Ti, the presented Rotating Features model on the Pascal VOC dataset requires less than 3.5 hours of training. For comparison, the DINOSAUR models [64] were trained across eight Nvidia V100 GPUs.

5 Related Work

Numerous approaches attempt to solve the binding problem in machine learning [28], with most focusing on slot-based object-centric representations. Here, the latent representation within a single layer of the architecture is divided into 'slots', creating a discrete separation of object representations. The symmetries between slots can be broken in various ways: by enforcing an order on the slots [4, 16, 18], by assigning slots to spatial coordinates [1, 3, 6, 7, 9, 13, 45, 63], by learning specialized slots for different object types [32, 33, 80, 82], by using an iterative procedure [12, 24, 25, 26, 27, 44, 74, 90] or by a combination of these methods [17, 70, 77, 86]. Slot Attention [46], which uses an iterative attention mechanism, has received increasing interest lately with many proposed advancements [37, 38, 47, 62, 68, 71, 75, 81] and extensions to the video domain [14, 22, 41, 69, 87]. Recently, Seitzer et al. [64] proposed to apply Slot Attention to the features of a pretrained DINO model, successfully scaling it to real-world datasets.

In comparison, there has been little work on continuous and distributed approaches for object discovery. A line of work has made use of complex-valued activations with a similar interpretation of magnitude and orientation to our proposed Rotating Features, starting from supervised [51, 89] and weakly supervised methods [58, 59, 60], to unsupervised approaches [48, 61]. However, they are only applicable to very simple, grayscale data. Concurrent work [73] proposes a contrastive learning method for the Complex AutoEncoder, scaling it to simulated RGB datasets of simple, uniformly colored objects. With the proposed Rotating Features, new evaluation procedure and higher-level input features, we create distributed and continuous object-centric representations that are applicable to real-world data.

6 Conclusion

Summary We propose several key improvements for continuous and distributed object-centric representations, scaling them from toy to real-world data. Our introduction of Rotating Features, a new evaluation procedure, and a novel architecture that leverages pre-trained features has allowed for a more scalable and expressive approach to address the binding problem in machine learning.

Limitations and Future Work Rotating Features create the capacity to represent more objects at once. However, this capacity can currently only be fully leveraged in the output space. As we show in Appendix D.2, within the bottleneck of the autoencoder, the separation of objects is not pronounced enough, yet, and we can only extract up to two objects at a time. To overcome this limitation, further research into novel inference methods using Rotating Features will be necessary. In addition, when it comes to object discovery results, Rotating Features produce slightly inferior outcomes than the state-of-the-art DINOSAUR autoregressive Transformer model, while still surpassing standard MLP decoders. Nonetheless, we hope that by exploring a new paradigm for object-centric representation learning, we can spark further innovation and inspire researchers to develop novel methods that better reflect the complexity and richness of human cognition.

Acknowledgments and Disclosure of Funding

We thank Jascha Sohl-Dickstein and Sjoerd van Steenkiste for insightful discussions. Francesco Locatello worked on this paper outside of Amazon. Sindy L¨ owe was supported by the Google PhD Fellowship.

References

Titas Anciukevicius, Christoph H Lampert, and Paul Henderson. Object-centric image generation with factored depths, locations, and appearances. arXiv preprint arXiv:2004.00642 , 2020.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Federico Baldassarre and Hossein Azizpour. Towards self-supervised learning of global and object-centric representations. ICLR 2022 Workshop on the Elements of Reasoning: Objects, Structure, and Causality , 2022.

Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390 , 2019.

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e J´ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In International Conference on Computer VIsion (ICCV) , pp. 9650-9660, 2021.

Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes. The Journal of Machine Learning Research , 22(1):11770-11805, 2021.

Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pp. 3412-3420, 2019.

Stanislas Dehaene. How we learn: Why brains learn better than any machine... for now . Penguin, 2021.

Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In International Conference on Learning Representations (ICLR) , 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 248-255. Ieee, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR) , 2021.

Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems (NeurIPS) , 34:15608-15620, 2021.

Yilun Du, Kevin Smith, Tomer Ulman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3d physical objects from video. International Conference on Learning Representations (ICLR) , 2021.

Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. Advances in Neural Information Processing Systems (NeurIPS) , 35:28940-28954, 2022.

Andreas K Engel, Pieter R Roelfsema, Pascal Fries, Michael Brecht, and Wolf Singer. Role of the temporal domain for response selection and perceptual binding. Cerebral Cortex (New York, NY: 1991) , 7(6):571-582, 1997.

Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative scene inference and sampling with object-centric latent representations. International Conference on Learning Representations (ICLR) , 2020.

Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. Advances in Neural Information Processing Systems (NeurIPS) , 34:8085-8094, 2021.

SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems (NeurIPS) , pp. 3225-3233, 2016.

M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision , 88(2):303-338, June 2010.

Pascal Fries. A mechanism for cognitive dynamics: neuronal communication through neuronal coherence. Trends in Cognitive Sciences , 9(10):474-480, 2005.

Pascal Fries. Rhythms for cognition: communication through coherence. Neuron , 88(1):220235, 2015.

Anand Gopalakrishnan, Kazuki Irie, J¨ urgen Schmidhuber, and Sjoerd van Steenkiste. Unsupervised learning of temporal abstractions with slot-based transformers. Neural Computation , 35 (4):593-626, 2023.

Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael C Mozer, and Yoshua Bengio. Neural production systems. Advances in Neural Information Processing Systems (NeurIPS) , 34:25673-25687, 2021.

Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Sch¨ olkopf. Recurrent independent mechanisms. International Conference on Learning Representations (ICLR) , 2021.

Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and J¨ urgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems (NeurIPS) , pp. 4484-4492, 2016.

Klaus Greff, Sjoerd Van Steenkiste, and J¨ urgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems (NeurIPS) , pp. 6691-6701, 2017.

Klaus Greff, Rapha¨ el Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning (ICML) , pp. 2424-2433, 2019.

Klaus Greff, Sjoerd van Steenkiste, and J¨ urgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208 , 2020.

Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. International Conference on Learning Representations (ICLR) , 2022.

Bharath Hariharan, Pablo Arbel´ aez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In International Conference on Computer VIsion (ICCV) , pp. 991-998, 2011.

Geoffrey E Hinton. How to represent part-whole hierarchies in a neural network. Neural Computation , pp. 1-40, 2022.

Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In International Conference on Artificial Neural Networks (ICANN) , pp. 44-51. Springer, 2011.

Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing. In International Conference on Learning Representations (ICLR) , 2018.

Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification , 2(1): 193-218, 1985.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML) , pp. 448-456, 2015.

Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In International Conference on Computer VIsion (ICCV) , pp. 9865-9874, 2019.

Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimization. In International Conference on Learning Representations (ICLR) , 2023.

Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. arXiv preprint arXiv:2303.10834 , 2023.

Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. https://github.com/deepmind/multi object datasets/, 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR) , 2015.

Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. International Conference on Learning Representations (ICLR) , 2022.

Kurt Koffka. Principles of gestalt psychology. Philosophy and Scientific Method , 32(8), 1935.

Wolfgang K¨ ohler. Gestalt psychology. Psychologische Forschung , 31(1):XVIII-XXX, 1967.

Adam R Kosiorek, Hyunjik Kim, and Danilo J Rezende. Conditional set generation with transformers. ICML 2020 Object-Oriented Learning Workshop , 2020.

Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. International Conference on Learning Representations (ICLR) , 2020.

Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Sindy L¨ owe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. NeurIPS 2020 Workshop on Object Representations for Learning and Reasoning , 2020.

Sindy L¨ owe, Phillip Lippe, Maja Rudolph, and Max Welling. Complex-valued autoencoders for object discovery. Transactions on Machine Learning Research (TMLR) , 2022.

Pascal Mettes, Elise Van der Pol, and Cees Snoek. Hyperspherical prototype networks. Advances in Neural Information Processing Systems (NeurIPS) , 32, 2019.

Peter M Milner. A model for visual shape recognition. Psychological Review , 81(6):521, 1974.

Michael C Mozer, Richard S Zemel, Marlene Behrmann, and Christopher KI Williams. Learning to segment images using dynamic feature binding. Neural Computation , 4(5):650-665, 1992.

Oleg R Musin and Alexey S Tarasov. The tammes problem for n= 14. Experimental Mathematics , 24(4):460-468, 2015.

Agostina Palmigiano, Theo Geisel, Fred Wolf, and Demian Battaglia. Flexible information routing by transient synchrony. Nature Neuroscience , 20(7):1014-1022, 2017.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. Advances in Neural Information Processing Systems 32 , pp. 8024-8035, 2019.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Jordi Pont-Tuset, Pablo Arbelaez, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE transactions on pattern analysis and machine intelligence , 39(1):128-140, 2016.

William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association , 66(336):846-850, 1971.

A Ravishankar Rao and Guillermo A Cecchi. An objective function utilizing complex sparsity for efficient segmentation in multi-layer oscillatory networks. International Journal of Intelligent Computing and Cybernetics , 2010.

A Ravishankar Rao and Guillermo A Cecchi. The effects of feedback and lateral connections on perceptual processing: A study using oscillatory networks. In International Joint Conference on Neural Networks , pp. 1177-1184, 2011.

ARavishankar Rao, Guillermo A Cecchi, Charles C Peck, and James R Kozloski. Unsupervised segmentation with dynamical units. IEEE Transactions on Neural Networks , 19(1):168-182, 2008.

David P Reichert and Thomas Serre. Neuronal synchrony in complex-valued deep networks. International Conference on Learning Representations (ICLR) , 2014.

Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems (NeurIPS) , 35: 9512-9524, 2022.

Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems (NeurIPS) , pp. 4967-4976, 2017.

Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, CarlJohann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Sch¨ olkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. International Conference on Learning Representations (ICLR) , 2023.

Wolf Singer. Neuronal synchrony: a versatile code for the definition of relations? Neuron , 24 (1):49-65, 1999.

Wolf Singer. Distributed processing and temporal codes in neuronal networks. Cognitive Neurodynamics , 3(3):189-196, 2009.

Wolf Singer and Charles M Gray. Visual feature integration and the temporal correlation hypothesis. Annual Review of Neuroscience , 18(1):555-586, 1995.

Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. International Conference on Learning Representations (ICLR) , 2022.

Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. In International Conference on Learning Representations (ICLR) , 2023.

Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Fredo Durand, Joshua B Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint arXiv:2205.03923 , 2022.

Elizabeth S Spelke. Principles of object perception. Cognitive science , 14(1):29-56, 1990.

Aleksandar Stani´ c, Anand Gopalakrishnan, Kazuki Irie, and J¨ urgen Schmidhuber. Contrastive training of complex-valued autoencoders for object discovery. arXiv preprint arXiv:2305.15001 , 2023.

Karl Stelzner, Robert Peharz, and Kristian Kersting. Faster attend-infer-repeat with tractable probabilistic models. In International Conference on Machine Learning (ICML) , pp. 59665975, 2019.

Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01148 , 2021.

Pieter Merkus Lambertus Tammes. On the origin of number and arrangement of the places of exit on the surface of pollen-grains. Recueil des travaux botaniques n´ eerlandais , 27(1):1-84, 1930.

Manuel Traub, Sebastian Otte, Tobias Menge, Matthias Karlbauer, Jannik Thuemmel, and Martin V Butz. Learning what and where: Disentangling location and identity tracking without supervision. In International Conference on Learning Representations (ICLR) , 2023.

Stefan Van der Walt, Johannes L Sch¨ onberger, Juan Nunez-Iglesias, Franc ¸ois Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. scikit-image: Image processing in python. PeerJ , 2:e453, 2014.

Christoph Von Der Malsburg and Werner Schneider. A neural cocktail-party processor. Biological Cybernetics , 54(1):29-40, 1986.

Julius von K¨ ugelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Sch¨ olkopf. Towards causal generative scene models via competition of experts. ICLR 2020 Workshop: Causal Learning for Decision Making , 2020.

Ziyu Wang, Mike Zheng Shou, and Mengmi Zhang. Object-centric learning with cyclic walks between parts and whole. arXiv preprint arXiv:2302.08023 , 2023.

Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Max Wertheimer. Untersuchungen zur Lehre von der Gestalt. Psychologische Forschung , 1(1): 47-58, 1922.

Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models , 2019.

Xiongwei Wu, Xin Fu, Ying Liu, Ee-Peng Lim, Steven CH Hoi, and Qianru Sun. A large-scale benchmark for food image segmentation. In Proceedings of ACM international conference on Multimedia , 2021.

Yizhe Wu, Oiwi Parker Jones, and Ingmar Posner. Obpose: Leveraging canonical pose for object-centric scene inference in 3d. arXiv preprint arXiv:2206.03591 , 2022.

Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. International Conference on Learning Representations (ICLR) , 2023.

Andrii Zadaianchuk, Matthaeus Kleindessner, Yi Zhu, Francesco Locatello, and Thomas Brox. Unsupervised semantic segmentation with self-supervised object-centric representations. International Conference on Learning Representations (ICLR) , 2023.

Richard S Zemel, Christopher KI Williams, and Michael C Mozer. Lending direction to neural networks. Neural Networks , 8(4):503-512, 1995.

Ruixiang Zhang, Tong Che, Boris Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, and Liam Paull. Robust and controllable object-centric learning through energy-based models. International Conference on Learning Representations (ICLR) , 2023.

Appendix Rotating Features for Object Discovery

A Broader Impact

This research explores advancements in continuous and distributed object-centric representations, scaling them from simple toy to real-world data. The broader impact includes:

Advancing Machine Learning: The proposed techniques expand the methodological landscape for object discovery, which may benefit many areas of machine learning, such as computer vision, robotics, and natural language processing.

Biological Insight: By exploring biologically inspired approaches to resolve the binding problem in machine learning, we may advance our understanding of object representations in the brain.

Improved Interpretability: The research promotes more structured representations, which may improve interpretability in machine learning models.

Scalability and Real-World Applicability: The proposed approaches can be applied to complex, multi-dimensional data, such as RGB images, making continuous object-centric representations applicable in real-world scenarios.

This research inspires further innovation and interdisciplinary collaboration. While we currently do not anticipate any negative societal impacts, we recognize that the wide array of potential applications for object discovery methods could lead to unforeseen consequences. As such, we acknowledge that there may be potential impacts that we are unable to predict at this time.

B Reproducibility Statement

In order to ensure the reproducibility of our experiments, we have provided a comprehensive overview of the employed model architectures, hyperparameters, evaluation procedures, datasets and baselines in Appendix C. Additionally, we published the code used to produce the main experimental results at github.com/loeweX/RotatingFeatures.

To achieve stable and reproducible results, we conducted all experiments using 4-5 different seeds. Training and evaluating the roughly 180 Rotating Features models highlighted across all our experiments took approximately 450 hours on a single Nvidia GTX 1080Ti. Training and evaluating four Slot Attention models for the baseline in Section 4.4 took another 200 hours on the same device. This estimate does not take into account the time spent on hyperparameter search and various trials conducted throughout the research process.

C Implementation Details

C.1 Architecture and Hyperparameters

We implement Rotating Features within an autoencoding model. Here, we will describe the precise architecture and hyperparameters used to train this model.

Architecture We provide a detailed description of the autoencoding architecture used for all of our experiments in Table 2. For the activation function (Eq. (4)), we employ Batch Normalization [35] in all convolutional layers, and Layer Normalization [2] in all fully connected layers.

Parameter Count As only the biases depend on the number of rotation dimensions n , the number of learnable parameters increases only minimally for larger values of n . For instance, in the models utilized for the 4Shapes dataset (Table 2 with d = 32 ), the total number of parameters amounts to 343,626 for n = 2 and 352,848 for n = 8 , reflecting a mere 2 . 7% increase.

×

×

Hyperparameters All hyperparameters utilized for models trained directly on raw input images are detailed in Table 3, while those for models trained on DINO features can be found in Table 4. We initialize the weights w out ∈ R c of f out by setting them to zero, and set biases b out ∈ R c to one. The remaining parameters are initialized using PyTorch's default initialization methods.

ViT Encoder To generate higher-level input features of real-world images, we use a pretrained vision transformer model [11] to preprocess the input images. Specifically, we employ a pretrained DINO model [5] from the timm library [84], denoted as ViT-B/16 or ' vit base patch16 224 '. This model features a token dimensionality of 768, 12 heads, a patch size of 16, and consists of 12 Transformer blocks. We use the final Transformer block's output as input for the Rotating Features model, excluding the last layer norm and the CLS token entry. All input images for the ViT have a resolution of 224 × 224 pixels and are normalized based on the statistics of the ImageNet dataset

[10], with mean per channel (0.485, 0.456, 0.406) and standard deviation per channel (0.229, 0.224, 0.225). The output of the DINO model has a dimension of 14 × 14 × 768 and we train our model to reconstruct this output. Before using it as input, we apply Batch Normalization and ReLU to ensure all inputs to the Rotating Features are positive. Overall, this setup closely follows that of Seitzer et al. [64] to ensure comparability of our results.

C.2 Evaluation Details

To evaluate the learned object separation of Rotating Features, we cluster their output orientations. For this, we normalize all output features onto the unit-hypersphere and subsequently apply a weighted average that masks features with small magnitudes (Section 3.4). For this masking, we use a threshold of t = 0 . 1 across all experiments. Subsequently, we apply a clustering method to the weighted average. For k -means, we utilize the scikit-learn implementation ([55], sklearn.cluster.KMeans ) with k set to the value as specified in Tables 3, 4 and 7, and all other parameters set to their default values. For agglomerative clustering, we also use the scikit-learn implementation ( sklearn.cluster.AgglomerativeClustering ) with n clusters set to None. The distance thresholds set to 5 for the results in Fig. 9, or a range of values as visualized in Figs. 7 and 22.

When utilizing DINO preprocessed features, our predicted object masks have a size of 14 × 14 , and we resize them to match the size of the ground-truth labels using the 'nearest' interpolation mode. Subsequently, we smooth the mask by applying a modal filter with a disk-shaped footprint of radius ⌊ 320 / 14 ⌋ = 22 using the scikit-image functions skimage.filters.rank.modal and skimage.morphology.disk [78].

For calculating the final object discovery metrics, we treat unlabeled pixels and pixels from overlapping instances as background (i.e. they are not evaluated).

C.3 Datasets

4Shapes The 4Shapes dataset comprises grayscale images of dimensions 32 × 32 , each containing four distinct white shapes ( □ , △ , ▽ , ⃝ ) on a black background. The square has an outer side-length of 13 pixels, the circle has an outer radius of 11 pixels and both isosceles triangles have a base-length of 17 pixels and a height of 9 pixels. All shapes have an outline width of 3 pixels. The dataset consists of 50,000 images in the train set, and 10,000 images for the validation and test sets, respectively. All pixel values fall within the range [0,1].

4Shapes RGB(-D) The RGB(-D) variants of the 4Shapes dataset follow the same general setup, but randomly sample the color of each shape. To achieve this, we create sets of potential colors with varying sizes. Each set is generated by uniformly sampling an offset value within the range [0,1], and subsequently producing different colors by evenly dividing the hue space, starting from this offset value. The saturation and value are set to one for all colors, and the resulting HSV color representations are converted to RGB. To create the RGB-D variant, we incorporate a depth channel

to each image and assign a unique depth value within the range [0,1] to every object, maintaining equal distances between them.

10Shapes The 10Shapes dataset consists of RGB-D images with 48 × 48 pixels. Each image features ten unique shapes ( □ , △ , ⃝ in varying orientations and sizes) on a black background. This dataset includes the same square, circle, and triangle as the 4Shapes dataset, with additional variations: the triangle is rotated by ± 90 ◦ degrees to form two more shapes, and the square is rotated by 45 ◦ to create a diamond shape. Another circle with an outer radius of 19 is also included. All these objects have an outline width of 3 pixels, with their centers remaining unfilled. The dataset also contains two smaller solid shapes: a downward-pointing triangle with a base-length of 11 and a height of 11, and a square with a side-length of 7. Unique colors are assigned to each object by evenly dividing the hue space, starting from an offset value uniformly sampled from the range [0,1]. The saturation and value are set to one for all colors, and the resulting HSV color representations are converted to RGB. Furthermore, each object is given a distinct depth value within the range [0,1], ensuring equal distances between them. The dataset consists of 200,000 images in the train set, and 10,000 images for the validation and test sets, respectively, and all pixel values fall within the range [0,1].

Pascal VOC The Pascal VOC dataset [19] is a widely-used dataset for object detection and semantic segmentation. Following the approach of Seitzer et al. [64], we train our model on the 'trainaug' variant, an unofficial split consisting of 10,582 images, which includes 1,464 images from the original segmentation train set and 9,118 images from the SBD dataset [30]. During training, we preprocess images by first resizing them such that their shorter side measures 224 pixels, followed by taking a random crop of size 224 × 224. Additionally, we apply random horizontal flipping. We evaluate the performance of Rotating Features on the official segmentation validation set, containing 1,449 images. Consistent with previous work [29, 36, 64], we use a resolution of 320 × 320 pixels for evaluation. To achieve this, we resize the ground-truth segmentation mask such that the shorter side measures 320 pixels and then take a center crop of dimensions 320 × 320. During evaluation, unlabeled pixels are ignored. To make our results comparable to Seitzer et al. [64], we evaluate the performance of our model across 5 seeds on this dataset.

FoodSeg103 The FoodSeg103 dataset [85] serves as a benchmark for food image segmentation, comprising 4,983 training images and 2,135 test images. We apply the same preprocessing steps as described for the Pascal VOC dataset, with the addition of a random augmentation. Besides randomly horizontally flipping the training images, we also rotate them randomly by 90 ◦ in either direction.

C.4 Baselines

Complex AutoEncoder (CAE) The CAE [48] is a continuous and distributed object-centric representation learning approach that has been shown to work well on simple toy datasets. To reproduce this model, we make use of its open-source implementation at https://github.com/loeweX/ ComplexAutoEncoder .

AutoEncoder To establish a standard autoencoder baseline, we employ the same architecture as described in Table 2, but replace the Rotating Features and layer structure of f rot with standard features and layers. Similarly to the Rotating Features model, we apply Batch Normalization / Layer Normalization before the ReLU non-linearity and maintain an input and output range of [0,1].

Block Masks To create the block masks baseline, we follow a similar approach as described by Seitzer et al. [64]. First, we divide the image into a specific number of columns: for fewer than 9 blocks, we create 2 columns; for 9 to 15 blocks, we create 3 columns, and for more than 15 blocks, we create 4 columns. Next, we distribute the blocks evenly across the columns. If equal division is not possible, we first allocate blocks to all but the last column and then display the remaining blocks in the final column. See Fig. 10 for examples with 5 and 10 blocks.

Other Baselines on the Pascal VOC dataset The results of the Slot Attention, SLATE, DINO k -means, DINOSAUR Transformer, DINOSAUR MLP models on the Pascal VOC dataset are adapted from Seitzer et al. [64]. To ensure the comparability of our findings, we adhere as closely as possible to their setup, utilizing the same dataset structure, preprocessing, and DINO pretrained model.

5

10

15

20

Number of Points

1.0

0.5

0.0

0.5

1.0

Minimum Cosine Similarity

n

2

4

6

8

10

100

Slot Attention To implement Slot Attention [46], we follow its open-source implementation at https://github.com/google-research/google-research/tree/master/slot_ attention . We use a hidden dimension of 64 across the model and modulate the decoding structure as outlined in Table 5.

×

D Additional Results

In this section, we provide additional results exploring the theoretical capacity of Rotating Features to represent many objects simultaneously, introducing a strong baseline for object discovery on the Pascal VOC dataset, and showcasing object separation in the latent space of a model trained with Rotating Features. Furthermore, we will provide additional quantitative and qualitative results for the experiments presented in the main paper.

D.1 High-dimensional Hyperspheres

We explore the theoretical capacity of Rotating Features to represent many objects simultaneously. Rotating Features learn continuous object-centric representations by separating object representations in orientation space. Depending on their orientation, the binding mechanism influences which features are processed together and which ones are masked out. This ultimately affects the number of objects that can be represented separately. Since this mechanism depends on the cosine similarity between features, we can analyze the capacity of Rotating Features by investigating the minimum cosine similarity achievable for a set of points on a hypersphere.

×

We investigate the behavior of the minimal cosine similarity between points on a hypersphere by implementing an optimization procedure that closely follows Mettes et al. [49]. This is necessary due to the lack of an exact solution for optimally distributing a variable number of points on higherdimensional hyperspheres [52, 76]. To begin, we randomly sample points on an n -dimensional hypersphere and then, for 10,000 steps, iteratively minimize the maximum cosine similarity between any pair of points using Stochastic Gradient Descent (SGD) with a learning rate of 0.1 and momentum of 0.9. In Fig. 11, we display the final results of this optimization process, which indicate the minimal achievable cosine similarity between any pair of points when a certain number of points are positioned on an n -dimensional hypersphere.

From this plot, we observe that regardless of n , we can always separate two points by a minimum cosine similarity of -1. When n = 2 , which corresponds to the setting of the Complex AutoEncoder, the minimal distance between points increases rapidly as more points are added. In this setting, when placing 20 points onto the hypersphere, the minimal cosine similarity approaches 1, indicating that points are closely packed together and barely separable from one another. With Rotating Features, we introduce a method to learn features with n ≥ 2 . As evident from the figure, larger values of n enable us to add more points to the hypersphere while maintaining small minimal cosine similarities. Finally, we observe that starting from ten points, if n is sufficiently large, the minimal cosine similarity achievable between points barely increases as more points are added. Having demonstrated that Rotating Features can separately represent ten objects with our experiments on the 10Shapes dataset (Section 4.1), the minimum cosine similarity achievable between 10 points is sufficient for the model to be able to separate them. Thus, we conclude that the number of objects to separate is not a critical hyperparameter for Rotating Features, in contrast to slot-based approaches.

D.2 Weakly Supervised Semantic Segmentation

In our main experiments, we demonstrate that Rotating Features create the capacity to represent more objects simultaneously (Section 4.1). Here, we explore the object separation within the bottleneck of an autoencoder utilizing Rotating Features. Our aim is to highlight that Rotating Features introduce object-centricity throughout the entire model and to examine whether their increased capacity is utilized in the latent space. To accomplish this, we create a weakly supervised semantic segmentation setup. During the autoencoder training, we introduce a supervised task in which a binary classifier is trained to identify the classes of all objects present in an image. Subsequently, we align the orientations of the class predictions with those of the object reconstructions in order to generate semantic segmentation masks.

Method To train a network with Rotating Features on a classification task, we apply a linear layer f class following the structure of f rot to the latent representation z encoder ∈ R n × d encoder created by the encoder. This layer maps the latent dimension of the encoder d encoder to the number of possible object classes o :

Then, we extract the magnitude from this output ∥ z class ∥ 2 ∈ R o and rescale it using a linear layer f out with sigmoid activation function before applying a binary cross-entropy loss (BCE) comparing the predicted to the ground-truth object labels o :

We use the resulting loss value L class alongside the reconstruction loss L to train the network while weighting both loss terms equally.

After training, we leverage the object separation within the bottleneck of the autoencoder to assign each individual pixel a label representing its underlying object class. To do so, we first measure the cosine similarity between each pixel in the normalized rotating reconstruction z eval ∈ R n × h × w (Eq. (8)) and each value in z class for which σ ( z i class ) > 0 . 5 . In other words, we calculate the cosine similarity for the values corresponding to object classes predicted to be present in the current image. Subsequently, we label each pixel based on the class label of the value in z class that it exhibits the highest cosine similarity with. If the maximum cosine similarity to all values in z class is lower than a specific threshold (zero in our case), or if ∥ z eval ∥ 2 < 0 . 1 , we assign the pixel to the background label. Note that this procedure does not require any clustering method to be employed, but directly utilizes the continuous nature of Rotating Features.

Table 6: Weakly supervised semantic segmentation (WSSS) results (mean ± SEM across 4 seeds). While the latent representations within the bottleneck of an autoencoder trained with Rotating Features can be used to create accurate semantic segmentation masks on the MNIST1Shape dataset in terms of IoU, the same approach fails when adding another object to the input images (MNIST2Shape) despite achieving strong object discovery (ARI-BG) and object classification results (Accuracy). This highlights that Rotating Features can be successfully employed in a weakly supervised semantic segmentation setting, but that the object separation within the bottleneck of the autoencoder is not pronounced enough, yet, to separate more than two objects.

Setup We construct an autoencoder adhering to the architecture outlined in Table 2, setting d = 64 . The model is trained for 10,000 steps with a batch size of 64 using the Adam optimizer [40]. The learning rate is linearly warmed up for the initial 500 steps of training to reach its final value of 0.001. This setup is applied to two datasets: MNIST1Shape and MNIST2Shape.

MNIST1Shape contains 32 × 32 images, each featuring an MNIST digit and one of three possible shapes ( □ , △ , ▽ ) that share the same format with those in the 4Shapes dataset. Conversely, MNIST2Shape combines an MNIST digit with two of these three shapes per image. Both datasets contain 50,000 train images and 10,000 images for validation and testing.

The network is trained to accurately reconstruct the input image while correctly classifying the MNIST digit, as well as the other shapes present. The setup described above enables the network to produce semantic segmentation masks, despite only being trained on class labels. We assess the performance of this weakly supervised semantic segmentation approach in terms of the intersectionover-union (IoU) score. For comparison purposes, we establish a baseline that generates unlabeled object masks according to the evaluation procedure presented in Section 3.4 using the outputs of the models under evaluation, and randomly assigns each object mask one of the predicted labels. For all results, overlapping areas between objects are masked and ignored during evaluation.

Results The results presented in Table 6 and Fig. 12 indicate that on the MNIST1Shape dataset, our approach with Rotating Features achieves strong weakly supervised semantic segmentation performance. It can accurately match objects separated in the latent space to their representations in the output space. Thereby, it almost doubles the performance of the baseline that randomly assigns labels to discovered objects, without requiring any clustering procedure. However, on the MNIST2Shape dataset, the weakly supervised semantic segmentation method falls short (see Table 6 and Fig. 13), even with a relatively large rotation dimension of n = 6 and despite achieving strong object discovery and object classification results (in terms of ARI-BG and accuracy scores). This demonstrates that while Rotating Features create the capacity to represent more objects simultaneously, this capacity is not fully utilized within the bottleneck of the autoencoder. Since the object separation is not pronounced enough, the current approach can only reliably separate representations of two objects in the latent space. However, the findings also highlight that Rotating Features create objectcentric representations throughout the entire model, as well as a potential new application for Rotating Features in weakly supervised semantic segmentation - an application with intriguing implications for interpretability.

D.3 Ablation on Chi

We perform an ablation of the binding mechanism. In this analysis, we modify the Rotating Features model by substituting m bind with ∥ ψ ∥ 2 in Eq. (4). This effectively removes the binding mechanism. Then, we apply the adjusted model to the grayscale 4Shapes dataset. While the original model achieves an ARI-BG score of 0 . 987 ± 0 . 003 on this dataset, the ablated model fails to learn any object separation ( 0 . 059 ± 0 . 017 ). This result highlights the crucial role the binding mechanism plays in enabling Rotating Features to learn object-centric representations.

Input

GT

Predictions

Baseline

口

3

中

口

3

46

口

Input

GT

Predictions

Baseline

5

口

口

2

4

6

8

10

Number of Objects

0.4

0.6

0.8

1.0

ARI-BG

2

4

6

8

10

Number of Objects

0.000

0.005

0.010

0.015

MSE

n

2

10

D.4 ARI-BG on Pascal VOC

We create a baseline that significantly surpasses all previously documented results in the literature in terms of the ARI-BG score on the Pascal VOC dataset. To achieve this, we simply predict the same object label for all pixels in an image, resulting in an ARI-BG score of 0.499. For comparison, the highest reported ARI-BG in the literature for this dataset is 0.296 [81]. We attribute the remarkable performance of this baseline to the characteristics of the Pascal VOC dataset, which often comprises images featuring a large object dominating the scene. Simply predicting the same label across the entire image results in the same label being predicted across this large, central object, and since the ARI-BG score is not evaluated on background pixels, this leads to relatively high scores. However, for the MBO scores, this baseline yields results far below the state-of-the-art, with 0.205 in MBO i and 0.247 in MBO c . Consequently, we focus our evaluation on these scores for this dataset.

D.5 Rotation Dimension n vs. Number of Objects

We conduct an experiment to assess how the choice of n impacts the object discovery performance of a Rotating Features model for different numbers of objects per scene. We accomplish this by creating variations of the 10Shapes dataset that contain between two and ten objects. For each variant, the number of objects per image equals the total number of distinct objects throughout the respective dataset. As illustrated in Fig. 14, the object discovery performance significantly drops as the number of objects increases when n = 2 . However, the performance remains consistently high when n = 10 . These findings indicate that the choice of n becomes increasingly important as the number of objects rises.

D.6 Multi-dSprites and CLEVR

We evaluate the performance of the proposed Rotating Features model on the commonly used MultidSprites and CLEVR datasets [39], comparing it to its predecessor, the CAE model [48]. The hyperparameters are listed in Table 7 and the outcomes are presented in Table 8. Note that since the original CAE was only applicable to grayscale images, we combine it with our proposed evaluation procedure to make it applicable to multi-channel input images, which we denote as CAE*.

The results indicate that the Rotating Features model significantly surpasses the performance of the CAE*, demonstrating good object separation on both datasets. However, the results still lack behind the state-of-the-art. The qualitative examples in Figs. 15 and 16 show that Rotating Features encounter the same issue here, as we have seen with the colored 4Shapes dataset (Section 4.2): objects of the same color tend to be grouped together. As demonstrated with the RGB-D version of the colored 4Shapes dataset and with our results using pretrained DINO features on real-world images, this issue can be addressed by using higher-level input features.

Input

GT

Predictions

RF

Predictions

CAE*

Input

GT

Predictions

RF

Predictions

CAE*

D.7 Additional Results

We provide supplementary qualitative and quantitative results for the experiments discussed in the main part of this paper. For the 4Shapes, 4Shapes RGB(-D), and 10Shapes datasets, additional qualitative results can be found in Figs. 17 to 19, with corresponding quantitative results detailed in Tables 9 to 11. Additional qualitative results for the Pascal VOC dataset are presented in Fig. 20, and extra results for the FoodSeg103 dataset are provided in Fig. 21. Lastly, we compare the performance of Rotating Features on the FoodSeg103 dataset using k -means and agglomerative clustering with various hyperparameters in Fig. 22.

GT

AE

CAE

Rotating Features

n

= 2

n

= 8

88888

哈哈电

连

Input

GT

Predictions

RGB

Predictions

RGB-D

口

Input

GT

Predictions

品

Input

GT

Predictions

Input

GT

Predictions

2

3

4

5

6

7

8

9

k

0

.

30

0

.

35

0

.

40

0

.

45

MBO

c

0

.

2

0

.

6

1

.

0

distance threshold

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCAEs promise to resolve some of the concerns of slot-based representation. They bring the promise of flexible object granularity, the promise of extracting part-whole hierarchies as per need, and faster training speeds. However, the original CAE was tested with grayscale images and on a rather small number of objects (2-3). This paper asks the question, *how can we scale CAEs* beyond this.\n\n**Approach.** Compared to classical neural networks, in the proposed model, all activations in the network are “tagged” with the binding information. For this, every scalar value of conventional vector representation is replaced with a $N$ length vector: the magnitude of this vector plays the role of classical activation while its orientation plays the role of capturing the binding information. Compared to original CAEs, the proposed model provides a generalization from having just 2 components (real and imaginary) to $N$ components, where $N$ can be arbitrarily large.\n\nIn **experiments**,\n1. The paper tests whether this change can help improve the capacity of CAEs in terms of the number of objects or not.\n2. The paper is also one of the first works to scale CAEs to real scenes.\n3. The paper also proposes a new procedure for extracting object segments from distributed representations.\n\nSTRENGTHS:\n1. (One of the) First successful attempts to make a CAE-like model scale to more objects and beyond simplistic grayscale images.\n2. The way weights and biases were applied seems to have been simplified (a welcome change) compared to the original CAEs. An ablation experiment highlighting this specific change for the case of $N=2$ could be useful though.\n3. Seems applicable to any data modality— demonstrated to some extent via the applicability of the model on RGB-D data in addition to RGB data.\n\nWEAKNESSES:\n1. One of the novelty seems to be the use of DINO pre-trained features for the first time in the context of CAEs. Therefore, a comparison with and without DINO pre-training would be useful to show in the main paper. \n2. Can original CAE be a baseline in the real-data experiment?\n3. The paper makes the jump from 4 object scenes directly to 10 object scenes. It may be useful to test perhaps a more gradual increase e.g., by also testing 7 object scenes and 13 object scenes, whether it places gradually more demand on the choice of $N$.\n4. It would be interesting to see the performance on a standard dataset like Tetris or CLEVR that was not deliberately designed for this paper. That being said, it need not outperform the previous slot-based methods here considering various other potential benefits of CAEs. \n5. At test time, can it handle a larger number of objects than shown during training?\n\nQUESTIONS:\n1. Is the layer choice important for segmentation? Do these patterns emerge also for the intermediate layers?\n2. I am curious what is the incentive for the model to assign different “phases” to different objects? In slot-based models, this role was played by the inherent capacity bottleneck of slots. I am curious about some insights about this.\n3. I could not entirely follow the motivation of Appendix D.3 “Object separation within the bottleneck of autoencoder utilizing Rotating Features…”\n\nLIMITATIONS:\nYes, the paper discusses the limitations, but one should also consider the potential benefits of the CAE framework.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a novel approach to unsupervised object discovery that does not depend on slots. Instead, the model uses an extra set of dimensions to code object assignment based on rotation, potentially allowing for a more flexible distributed form of object discovery than in standard slot-based approaches. The model is shown to perform well on toy tasks, with promising results on real-world images.\n\nSTRENGTHS:\n- This work presents an interesting new direction, substantially reconsidering the problem of object discovery relative to the now ubiquitous slot-based methods.\n- The method performs well on toy tasks, scaling to a relatively large number of objects, and also shows promising results on real-world images.\n- The method is significantly more efficient than popular slot-based methods.\n- The paper stimulates many interesting directions for future work.\n\nWEAKNESSES:\nI have a number of suggestions and questions that may help to further improve the paper:\n\n- A major potential advantage of the proposed approach is that it is more flexible than slot-based methods, specifically regarding the number of objects that can be segmented. The results in supplementary figure 10 suggest that, with a sufficient number of dimensions (e.g. ~10) a relatively large number of objects can be represented, and that even more objects can be represented without needing to add many more dimensions. Can this advantage over slot attention be empirically demonstrated? For instance, are there any instances in pascal VOC or FoodSeg that involve more objects than the number of slots in DINOSAUR? If so, it would be interesting to see whether rotating features outperforms DINOSAUR on those problems. Alternatively, controlled experiments with CLEVR, or even the Nshapes dataset, could be performed to demonstrate the superior representational capacity of rotating features over slot attention (especially when there are more objects than slots).\n- A desirable feature of slot attention is that it is permutation invariant, which allows for a dynamic binding of features to objects (i.e. by randomly initializing the slots and iteratively refining them through competition). Here, by contrast, particular features have learned biases toward particular orientations, does this interfere with the ability of the model to perform variable-binding in a dynamic manner? How would the model perform when tested on a greater number of objects than it was trained on (given that it has a sufficient number of dimensions to represent those objects)? In other words, is the method more efficient than slot attention because it is also less flexible?\n- Have the authors considered sharing the orientation across features at each location in a convolutional feature map? Intuitively, it seems that all features at a particular location should only be assigned to a single object, rather than having a unique assignment of each feature.\n- Have the authors investigated how the assignment of objects evolves across layers? I am wondering whether the competition that occurs over time in methods like slot attention is somehow distributed across layers in this model.\n- It would be informative to include an ablation of the binding mechanism. I also found the description of this mechanism somewhat counterintuitive. It is described as 'weakening the connections between features with dissimilar orientations', but it almost seems as if it does the exact opposite of this. The magnitude of the features in $\\chi$ is completely unrelated to the orientations of the inputs. Therefore, it seems that mixing $\\chi$ with $\\psi$ is only weakening the influence of orientation, allowing incoming features with dissimilar orientations (but high synaptic weight values) to exercise a greater influence on the magnitude of the feature representation. Can the authors add some additional explanation of this mechanism?\n\nQUESTIONS:\nI have listed some questions in the previous section.\n\nLIMITATIONS:\nThere are no discernible negative societal impacts related to this work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the problem of unsupervised object discovery. It seeks to remedy some limitations (primarily object storage capacity) of the recently introduced synchrony-based approach, CAE [1], and scales it to more visually complex scenes compared to CAE [1] which was only applied to simple grayscale (Shapes and MNIST) datasets. The key idea behind the proposed method is the extension of the number of feature dimensions used to manipulate “phase” (rotation) values associated with image features from ‘2’ in CAE to ‘n’ dimensions in RF. To control these ‘n’ dimensional rotation values use ‘n’ separate bias terms in every layer. Further, they apply their method on pre-trained DINO features to scale the grouping results to real-world datasets.\n\nSTRENGTHS:\n1) The problem studied in this paper is well-motivated and important. \n\n2) The method presented is novel and as the model class it uses to perform binding (synchrony-based) has received significantly less attention in the literature compared to slot-based approaches.\n\n3) The paper is very well written.\n\n4) The experiments are well performed and the presented method compares favorably to the considered baselines.\n\nWEAKNESSES:\n1) The authors note that one of their three contributions is a new evaluation method. The phases are weighted but with binary values (0 or 1) which reduces to a difference in the averaging constant (N vs N-k) in the denominator of the weighted averaging operation. The threshold used to compute the weights is just a fixed value (i.e. 0.1 same as in CAE [1]) as opposed to being some learned value based on the magnitudes. So, in practice, how different/novel is this evaluation procedure compared to the one used by CAE to the extent that it can be deemed as a core technical contribution of this work?\n \n2) Depth masks provide a strong supervision signal for discovery of object grouping information. The problem of 2 (or more) objects having the same color being grouped together has been resolved through the use of such depth masks i.e. strong supervision (Figure 6 caption). \n\n3) Lines (244-245): “On FoodSeg103, we limit our evaluation of our model to the MBO_c score, …. ”. MBO_c score only measures semantic grouping not instance-level grouping. The pretrained DINO features already show a high level of specialization to semantic classes (attention maps when conditioning on CLS token). Therefore, this experiment does not really meaningfully test the instance-level grouping ability of the proposed method.\n\nQUESTIONS:\n1) Why do spherical coordinates lead to instabilities? Could not follow the explanation in footnote 1, specifically the point about ordering of the angles and under certain circumstances subsequent angles having no influence on the orientation. Could the authors clarify this point? A visualization would be greatly beneficial in this regard.\n\n2) Lines (242-243) : “On the Pascal VOC dataset, we do not compare ARI-BG scores, as we have found that a simple baseline significantly surpasses previously reported .….” . I didn’t understand the reasoning behind this choice.\n\n3) (related to weakness #2) Why can’t the proposed method segregate 2 objects of the same color but spatially well separated? If the network has learned a simple rule using just a single feature like color or shape to perform binding, that would be extremely limiting in the general case.\n\n4) (related to weakness #2) Could the authors show the grouping performance (FG-ARI, FULL-ARI) for the colorized ‘Shapes’ dataset without (simply using RGB images as inputs) the use of depth channel information? Have the authors performed an ablation to quantify how much the depth channel information assists the network to predict the object identities?  \n\n5) How does the proposed method work on colored multi-object datasets like Tetrominoes, Multi-dSprites or CLEVR that are the first benchmark suite typically used for evaluation in the object-centric literature?\n\n6) Table 1 (Pascal VOC dataset results), compared to the baseline models which use pre-trained DINO features in the encoder module (i.e. DINOSAUR Transformer/MLP) how much instance-level grouping has been achieved through the use of RF? It’s known that the DINO features already possess a high-level of specialization to semantic classes and therefore can perform semantic grouping (by inspecting the attention masks after conditioning on the CLS token). In this regard, SlotAttention and SLATE baselines cannot be considered like-for-like since they still use the pixel reconstruction objective and are trained from scratch. Short point being that how much instance-level grouping is being learned through the use of RF on top of the semantic-level grouping that is  already captured by the pretrained DINO features?\n\nLIMITATIONS:\nAuthors discuss limitations of their method, in particular that the capacity to represent multiple objects is currently limited to the output space of the autoencoder and that the methods still lag behind slot-based methods such as DINOSAUR.\n\n\nReferences: \n\n[1] “Complex-Valued Autoencoders for Object Discovery”, Lowe et. al, TMLR 2022.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new approach for extracting objects from distributed representations, based on a binding mechanism called 'rotating features' that extends previous phase-based binding notions to a much higher dimension binding space, and avoids the use of separate slots for individual objects, showing promising results and scaling behavior with both toy and natural-image data sets.\n\nSTRENGTHS:\nThe exploration of alternatives to slot-based schemes, so prevalent in transformers and related contemporary architectures, for addressing how neural networks can successfully extract distinct, coherent representations of objects and other entities is an important contribution in itself.  For brain science, it is important because the brain is unlikely to have explicit slots of the kind that can be built into artificial networks.  For AI, this step may be equally important, as it allows for the possibility of more graded approaches to objecthood that could be important for capturing the kind of graded objecthood of many aspects of the natural world, and of avoiding some of the potential brittleness and arbitrariness of imposing slot-based approaches to structured objects made up of sub-objects.\n\nThe model seems to produce fairly impressive results compared to an alternative much more complex transformer models and beats other comparison models as well.  The excellent training time, and the possibility of avoiding explicit k-means clustering for segmentation, and the availability of uncertainly maps all seem like desirable properties of the model.\n\nWEAKNESSES:\nI have chosen to ask questions rather than express statements about weaknesses because I found that there were important features of the model and of the comparisons that I simply could not fully understand without going to source papers on DINOSAUR and the binding mechanism.  I do consider it a weakness of the paper that I was not able to understand these things better, and my rating and confidence would be increased further if these questions were addressed in the rebuttal and revision of the paper.\n\nQUESTIONS:\nI found it a bit difficult to be sure I was seeing fair comparisons in table 1.  If I understand correctly, DINOSAUR MLP is much simpler than the Rotating Features CNN.  Is it slot based in some way?  \n\nI have found it difficult to understand how the binding mechanism described on p 4 works and I did not feel I got much out of figure 3.  perhaps the problem lies in my lack of understanding of the meaning of the superscripts on z_in.  Another source of confusion is the statement that the extra input dimensions of x (in line 134) are all set to 0, such that they don't seem to be capable of having any effect.  Why are these dimensions then needed?  I see that learned bias weights apply to the output of f_w and that there are R^(n x d_out) of these.  The references to Lowe et al and Reichert & Serre should not be my only source of an understanding of how the mechanism works.\n\nLIMITATIONS:\nThe limitations as stated seem valid, and I look forward to seeing where attempts to address these limitations will lead.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}