SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sindy Löwe
AMLab
QUVA Lab
Institute of Science and
Technology Austria (ISTA)
The binding problem in human cognition, concerning how the brain represents and connects objects within a fixed network of neural connections, remains a subject of intense debate. Most machine learning efforts addressing this issue in an unsupervised setting have focused on slot-based methods, which may be limiting due to their discrete nature and difficulty to express uncertainty. Recently, the Complex AutoEncoder was proposed as an alternative that learns continuous and distributed object-centric representations. However, it is only applicable to simple toy data. In this paper, we present Rotating Features, a generalization of complex-valued features to higher dimensions, and a new evaluation procedure for extracting objects from distributed representations. Additionally, we show the applicability of our approach to pre-trained features. Together, these advancements enable us to scale distributed object-centric representations from simple toy to real-world data. We believe this work advances a new paradigm for addressing the binding problem in machine learning and has the potential to inspire further innovation in the field.
Discovering and reasoning about objects is essential for human perception and cognition [8, 72], allowing us to interact with our environment, reason about it, and adapt to new situations [42, 43, 83]. To represent and connect symbol-like entities such as objects, our brain flexibly and dynamically combines distributed information. However, the binding problem remains heavily debated, questioning how the brain achieves this within a relatively fixed network of neural connections [28].
Most efforts to address the binding problem in machine learning center on slot-based methods [4, 23, 27, 33, 41, 46, 64, 88]. These approaches divide their latent representations into "slots", creating a discrete separation of object representations within a single layer of the architecture at some arbitrary depth. While highly interpretable and easy to use for downstream applications, this simplicity may limit slot-based approaches from representing the full diversity of objects. Their discrete nature makes it challenging to represent uncertainty in the separation of objects; and it remains unclear how slots may be recombined to learn flexible part-whole hierarchies [31]. Finally, it seems unlikely that the brain assigns entirely distinct groups of neurons to each perceived object.
Recently, Löwe et al. [48] proposed the Complex AutoEncoder (CAE), which learns continuous and distributed object-centric representations. Taking inspiration from neuroscience, it uses complex-valued activations to learn to encode feature information in their magnitudes and object affiliation in their phase values. This allows the network to express uncertainty in its object separations and embeds
Correspondence to loewe.sindy@gmail.com
Figure 1: Rotating Features. We propose to extend standard features by an extra dimension  $n$  across the entire architecture, including the input (highlighted in blue, here  $n = 3$ ). We then set up the layer structure within  $f_{\mathrm{rot}}$  in such a way that the Rotating Features' magnitudes  $m$  learn to represent the presence of features, while their orientations learn to represent object affiliation.
object-centric representations in every layer of the architecture. However, due to its two-dimensional (i.e. complex-valued) features, it is severely limited in the number of object representations it can separate in its one-dimensional phase space; and due to the lack of a suitable evaluation procedure, it is only applicable to single-channel inputs. Taken together, this means that the CAE is not scalable, and is only applicable to grayscale toy data containing up to three simple shapes.
In this paper, we present a series of advancements for continuous and distributed object-centric representations that ultimately allow us to scale them from simple toy to real-world data. Our contributions are as follows:
1. We introduce Rotating Features, a generalization of complex-valued features to higher dimensions. This allows our model to represent a greater number of objects simultaneously, and requires the development of a new rotation mechanism.
2. We propose a new evaluation procedure for continuous object-centric representations. This allows us to extract discrete object masks from continuous object representations of inputs with more than one channel, such as RGB images.
3. We show the applicability of our approach to features created by a pretrained vision transformer [11]. This enables Rotating Features to extract object-centric representations from real-world images.
We show how each of these improvements allows us to scale distributed object-centric representations to increasingly complicated data, ultimately making them applicable to real-world data. Overall, we are pioneering a new paradigm for a field that has been largely focused on slot-based approaches, and we hope that the resulting richer methodological landscape will spark further innovation.
2 Neuroscientific Motivation
Rotating Features draw inspiration from theories in neuroscience that describe how biological neurons might utilize their temporal firing patterns to flexibly and dynamically bind information into coherent percepts, such as objects [15, 20, 21, 50, 53, 65, 79]. In particular, we take inspiration from the temporal correlation hypothesis [66, 67]. It posits that the oscillating firing patterns of neurons (i.e. brain waves) give rise to two message types: the discharge frequency encodes the presence of features, and the relative timing of spikes encode feature binding. When neurons fire in synchrony, their respective features are processed jointly and thus bound together dynamically and flexibly.
Figure 2: Rotating Features applied to real-world images. By implementing a series of advancements, we scale continuous and distributed object representations from simple toy to real-world datasets.
Previous work in machine learning [48, 58, 59, 60, 61] has taken inspiration from these neuroscientific theories to implement the same two message types using complex-valued features: their magnitudes encode feature presence, and their relative phase differences encode feature binding. However, the one-dimensional orientation information of complex-valued features constrains their expressivity, and existing approaches lack evaluation procedures that scale beyond single-channel inputs. As a result, they are limited to grayscale toy datasets. To overcome these limitations, we propose several advancements for distributed and continuous object-centric representations, which we describe in detail in the next section.
3 Rotating Features
We create Rotating Features by augmenting standard features with additional dimensions and by creating a layer structure that parses the magnitude and multi-dimensional orientation of the resulting vectors separately. In the following, we describe how we represent, process, train and evaluate Rotating Features such that their magnitudes learn to represent the presence of features and their orientations learn to represent object affiliation.
3.1 Representing Rotating Features
In a standard artificial neural network, each layer produces a  $d$ -dimensional feature vector  $\mathbf{z}_{\mathrm{standard}} \in \mathbb{R}^d$ , where  $d$  might be further subdivided to represent a particular structure of the feature map (e.g.  $d = c \times h \times w$ , where  $c, h, w$  represent the channel, height and width dimensions, respectively). To create Rotating Features, we extend each scalar feature within this  $d$ -dimensional feature vector into an  $n$ -dimensional vector, thus creating a feature matrix  $\mathbf{z}_{\mathrm{rotating}} \in \mathbb{R}^{n \times d}$ . Given the network structure described below, its magnitude vector  $\left\| \mathbf{z}_{\mathrm{rotating}} \right\|_2 \in \mathbb{R}^d$  (where  $\| \cdot \|_2$  is the L2-norm over the rotation dimension  $n$ ) behaves similarly to a standard feature vector and learns to represent the presence of certain features in the input. The remaining  $n-1$  dimensions represent the orientations of the Rotating Features, which the network uses as a mechanism to bind features: features with similar orientations are processed together, while connections between features with different orientations are suppressed. Note, that we represent Rotating Features in Cartesian coordinates rather than spherical coordinates, as the latter may contain singularities that hinder the model's ability to learn good representations<sup>1</sup>.
<sup>1</sup>In an  $n$ -dimensional Euclidean space, the spherical coordinates of a vector  $\mathbf{x}$  consist of a radial coordinate  $r$  and  $n - 1$  angular coordinates  $\varphi_{1},\ldots,\varphi_{n - 1}$  with  $\varphi_{1},\ldots,\varphi_{n - 2} \in [0,\pi]$  and  $\varphi_{n - 1} \in [0,2\pi)$ . This representation can lead to singularities due to dependencies between the coordinates. For example, when a vector's radial component (i.e. magnitude) is zero, the angular coordinates can take any value without changing the underlying vector. As our network applies ReLU activations on the magnitudes, this singularity may occur regularly, hindering the network from training effectively.
3.2 Processing Rotating Features
We create a layer structure  $f_{\mathrm{rot}}$  that processes Rotating Features in such a way that their magnitudes represent the presence of certain features, while their orientations implement a binding mechanism. To achieve this, we largely follow the layer formulation of the CAE [48], but generalize it from complex-valued feature vectors  $\mathbf{z}_{\mathrm{complex}} \in \mathbb{C}^d$  to feature matrices of the form  $\mathbf{z}_{\mathrm{rotating}} \in \mathbb{R}^{n \times d}$  both in the implementation and in the equations below; and introduce a new mechanism to rotate features. This allows us to process Rotating Features with  $n \geq 2$ .
Weights and Biases Given the input  $\mathbf{z}_{\mathrm{in}} \in \mathbb{R}^{n \times d_{\mathrm{in}}}$  to a layer with  $n$  rotation dimensions and  $d_{\mathrm{in}}$  input feature dimensions, we apply a neural network layer parameterized with weights  $\mathbf{w} \in \mathbb{R}^{d_{\mathrm{in}} \times d_{\mathrm{out}}}$  and biases  $\mathbf{b} \in \mathbb{R}^{n \times d_{\mathrm{out}}}$ , where  $d_{\mathrm{out}}$  is the output feature dimension, in the following way:
$$
\psi = f _ {\mathbf {w}} \left(\mathbf {z} _ {\text {i n}}\right) + \mathbf {b} \quad \in \mathbb {R} ^ {n \times d _ {\text {o u t}}} \tag {1}
$$
The function  $f_{\mathrm{w}}$  may represent different layer types, such as a fully connected or convolutional layer. For the latter, weights may be shared across  $d_{\mathrm{in}}$  and  $d_{\mathrm{out}}$  appropriately. Regardless of the layer type, the weights are shared across the  $n$  rotation dimensions, while the biases are not. By having separate biases, the model learns a different orientation offset for each feature, providing it with the necessary mechanism to learn to rotate features. Notably, in contrast to the CAE, the model is not equivariant with respect to global orientation with this new formulation of the rotation mechanism: when there is a change in the input orientation, the output can change freely. While it is possible to create an equivariant rotation mechanism for  $n \geq 2$  by learning or predicting the parameters of a rotation matrix, preliminary experiments have demonstrated that this results in inferior performance.
Binding Mechanism We implement a binding mechanism that jointly processes features with similar orientations, while weakening the connections between features with dissimilar orientations. This incentivizes features representing the same object to align, and features representing different objects to take on different orientations.
To implement this binding mechanism, we utilize the same weights as in Eq. (1) and apply them to the magnitudes of the input features  $\| \mathbf{z}_{\mathrm{in}}\| _2\in \mathbb{R}^{d_{\mathrm{in}}}$ :
$$
\chi = f _ {\mathbf {w}} \left(\left\| \mathbf {z} _ {\text {i n}} \right\| _ {2}\right) \quad \in \mathbb {R} ^ {d _ {\text {o u t}}} \tag {2}
$$
We then integrate this with our previous result by calculating the average between  $\chi$  and the magnitude of  $\psi$ :
$$
\mathbf {m} _ {\text {b i n d}} = 0. 5 \cdot \| \psi \| _ {2} + 0. 5 \cdot \chi \in \mathbb {R} ^ {d _ {\text {o u t}}} \tag {3}
$$
This binding mechanism results in features with similar orientations being processed together, while connections between features with dissimilar orientations are suppressed. Given a group of features of the same orientation, features of opposite orientations are masked out, and misaligned features are gradually scaled down. This effect has been shown by Löwe et al. [48], Reichert & Serre [61], and we illustrate it in Fig. 3. In contrast, without the binding mechanism, the orientation of features does not influence their processing, the model has no incentive to leverage the additional rotation dimensions, and the model fails to learn object-centric representations (see Appendix D.3). Overall, the binding mechanism allows the network to create streams of information that it can process separately, which naturally leads to the emergence of object-centric representations.
Activation Function In the final step, we apply an activation function to the output of the binding mecha
Figure 3: Effect of the binding mechanism. We start by randomly sampling two column vectors  $\mathbf{a}$ ,  $\mathbf{b} \in \mathbb{R}^n$  with  $\| \mathbf{a} \|_2 = \| \mathbf{b} \|_2 = 1$ . Assuming  $d_{\mathrm{in}} = 3$ ,  $d_{\mathrm{out}} = 1$  and  $f_{\mathbf{w}}$  is a linear layer, we set  $\mathbf{z}_{\mathrm{in}} = [\mathbf{a}, \mathbf{a}, \mathbf{b}]$ , weights  $\mathbf{w} = \left[\frac{1}{3}, \frac{1}{3}, \frac{1}{3}\right]^T$  and biases  $\mathbf{b} = [0, \dots, 0]^T$ . Subsequently, we plot the cosine similarity between  $\mathbf{a}$  and  $\mathbf{b}$  on the x-axis, and  $\mathbf{m}_{\mathrm{bind}}$  and  $\| \psi \|_2$  on the y-axis, representing the magnitudes of the layer's output before applying the activation function with (blue) and without (orange) the binding mechanism. Without the binding mechanism, misaligned features are effectively subtracted from the aligned features, resulting in smaller output magnitudes for  $\| \psi \|_2$ . The binding mechanism reduces this effect, leading to consistently larger magnitudes in  $\mathbf{m}_{\mathrm{bind}}$ . In the most extreme scenario, features with opposite orientations (i.e., with a cosine similarity of -1) are masked out by the binding mechanism, as the output magnitude  $(\frac{2}{3})$  would be the same if  $\mathbf{z}_{\mathrm{in}} = [\mathbf{a}, \mathbf{a}, \mathbf{0}]$ .
nism  $\mathbf{m}_{\mathrm{bind}}$ , ensuring it yields a positive-valued result. By rescaling the magnitude of  $\psi$  to the resulting value and leaving its orientation unchanged, we obtain the output of the layer  $\mathbf{z}_{\mathrm{out}} \in \mathbb{R}^{n \times d_{\mathrm{out}}}$ :
$$
\mathbf {m} _ {\text {o u t}} = \operatorname {R e L U} \left(\operatorname {B a t c h N o r m} \left(\mathbf {m} _ {\text {b i n d}}\right)\right) \quad \in \mathbb {R} ^ {d _ {\text {o u t}}} \tag {4}
$$
$$
\mathbf {z} _ {\text {o u t}} = \frac {\boldsymbol {\psi}}{\| \boldsymbol {\psi} \| _ {2}} \cdot \mathbf {m} _ {\text {o u t}} \in \mathbb {R} ^ {n \times d _ {\text {o u t}}} \tag {5}
$$
3.3 Training Rotating Features
We apply Rotating Features to vision datasets which inherently lack the additional  $n$  dimensions. This section outlines the pre-processing of input images to generate rotating inputs and the post-processing of the model's rotating output to produce standard predictions, which are used for training the model.
Given a positive, real-valued input image  $\mathbf{x}' \in \mathbb{R}^{c \times h \times w}$  with  $c$  channels, height  $h$ , and width  $w$ , we create the rotating feature input  $\mathbf{x} \in \mathbb{R}^{n \times c \times h \times w}$  by incorporating  $n - 1$  empty dimensions. We achieve this by setting the first dimension of  $\mathbf{x}$  equal to  $\mathbf{x}'$ , and assigning zeros to the remaining dimensions. Subsequently, we apply a neural network model  $f_{\mathrm{model}}$  that adheres to the layer structure of  $f_{\mathrm{rot}}$  to this input, generating the output features  $\mathbf{z} = f_{\mathrm{model}}(\mathbf{x}) \in \mathbb{R}^{n \times d_{\mathrm{model}}}$ .
To train the network using a reconstruction task, we set  $d_{\mathrm{model}} = c \times h \times w$  and extract the magnitude  $\| \mathbf{z} \|_2 \in \mathbb{R}^{c \times h \times w}$  from the output. We then rescale it using a linear layer  $f_{\mathrm{out}}$  with sigmoid activation function. This layer has weights  $\mathbf{w}_{\mathrm{out}} \in \mathbb{R}^c$  and biases  $\mathbf{b}_{\mathrm{out}} \in \mathbb{R}^c$  that are shared across the spatial dimensions, and applies them separately to each channel  $c$ :
$$
\hat {\mathbf {x}} = f _ {\text {o u t}} \left(\| \mathbf {z} \| _ {2}\right) \quad \in \mathbb {R} ^ {c \times h \times w} \tag {6}
$$
Finally, we compute the reconstruction loss  $\mathcal{L} = \mathrm{MSE}(\mathbf{x}',\hat{\mathbf{x}})\in \mathbb{R}$  by comparing the input image  $\mathbf{x}'$  to the reconstruction  $\hat{\mathbf{x}}$  using the mean squared error (MSE).
3.4 Evaluating Object Separation in Rotating Features
While training the network using the Rotating Features' magnitudes, their orientations learn to represent "objectness" in an unsupervised manner. Features that represent the same objects align, while those representing different objects take on distinct orientations. In this section, we outline how to process the continuous orientation values of the output Rotating Features  $\mathbf{z} \in \mathbb{R}^{n \times c \times h \times w}$ , in order to generate a discrete object segmentation mask. This approach follows similar steps to those described for the CAE [48], but introduces an additional procedure that enables the evaluation of Rotating Features when applied to inputs with  $c \geq 1$ .
As the first step, we normalize the output features such that their magnitudes equal one, mapping them onto the unit (hyper-)sphere. This ensures that the object separation of features is assessed based on their orientation, rather than their feature values (i.e. their magnitudes):
$$
\mathbf {z} _ {\text {n o r m}} = \frac {\mathbf {z}}{\| \mathbf {z} \| _ {2}} \quad \in \mathbb {R} ^ {n \times c \times h \times w} \tag {7}
$$
Subsequently, we mask features with small magnitudes, as they tend to exhibit increasingly random orientations, in a manner that avoids trivial solutions that may emerge on feature maps with  $c > 1$ . For instance, consider an image containing a red and a blue object. The reconstruction  $\mathbf{z}$  would be biased towards assigning small magnitudes to the color channels inactive for the respective objects. If we simply masked out features with small magnitudes by setting them to zero, we would separate the objects based on their underlying color values, rather than their assigned orientations.
To avoid such trivial solutions, we propose to take a weighted average of the orientations across channels, using the thresholded magnitudes of  $\mathbf{z}$  as weights:
$$
\mathbf {w} _ {\text {e v a l}} ^ {i, j, l} = \left\{ \begin{array}{l l} 1 & \text {i f} \| \mathbf {z} \| _ {2} ^ {i, j, l} > t \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {8}
$$
$$
\mathbf {z} _ {\text {e v a l}} = \frac {\sum_ {i = 1} ^ {c} \mathbf {w} _ {\text {e v a l}} ^ {i} \circ \mathbf {z} _ {\text {n o r m}} ^ {i}}{\sum_ {i = 1} ^ {c} \mathbf {w} _ {\text {e v a l}} ^ {i} + \varepsilon} \in \mathbb {R} ^ {n \times h \times w} \tag {9}
$$
where we create the weights  $\mathbf{w}_{\mathrm{eval}}$  by thresholding the magnitudes  $\| \mathbf{z}\| _2$  for each channel  $i\in [1,\dots,c]$  and spatial location  $j\in [1,\dots,h],l\in [1,\dots,w]$  using the threshold  $t$  and then compute the weighted
(a)
Figure 4: Qualitative and quantitative performance comparison on the 4Shapes dataset. (a) In comparison to a standard autoencoder (AE) and the Complex AutoEncoder (CAE), Rotating Features (RF) create sharper reconstructions (1st and 3rd row). Additionally, Rotating Features separate all four shapes, while the CAE does not (2nd and 4th row). (b) Larger rotation dimensions  $n$  lead to better reconstruction performance (top) and better object discovery performance (bottom), with  $n \geq 6$  resulting in a perfect separation of all shapes (mean ± sem performance across four seeds).
average across channels  $c$ . Here,  $\circ$  denotes the Hadamard product,  $\varepsilon$  is a small numerical constant to avoid division by zero, and  $\mathbf{w}_{\mathrm{eval}}$  is repeated across the  $n$  rotation dimensions.
Lastly, we apply a clustering procedure to  $\mathbf{z}_{\mathrm{eval}}$  and interpret the resulting discrete cluster assignment as the predicted object assignment for each feature location. Our experiments demonstrate that both  $k$ -means and agglomerative clustering achieve strong results, thus eliminating the need to specify the number of objects in advance.
In this section, we evaluate whether our proposed improvements enable distributed object-centric representations to scale from simple toy to real-world data. We begin by outlining the general settings common to all experiments, and then proceed to apply each proposed improvement to a distinct setting. This approach allows us to isolate the impact of each enhancement. Finally, we will highlight some advantageous properties of Rotating Features. Our code is publicly available at github.com/loeweX/RotatingFeatures.
General Setup We implement Rotating Features within a convolutional autoencoding architecture. Each model is trained with a batch-size of 64 for 10,000 to 100,000 steps, depending on the dataset, using the Adam optimizer [40]. Our experiments are implemented in PyTorch [54] and run on a single Nvidia GTX 1080Ti. See Appendix C for more details on our experimental setup.
Evaluation Metrics We utilize a variety of metrics to gauge the performance of Rotating Features and draw comparisons with baseline methods. Reconstruction performance is quantified using mean squared error (MSE). To evaluate object discovery performance, we employ Adjusted Rand Index (ARI) [34, 57] and mean best overlap (MBO) [56, 64]. ARI measures clustering similarity, where a score of 0 indicates chance level and 1 denotes a perfect match. Following standard practice in the object discovery literature, we exclude background labels when calculating ARI (ARI-BG) and evaluate it on instance-level masks. MBO is computed by assigning the predicted mask with the highest overlap to each ground truth mask, and then averaging the intersection-over-union (IoU) values of the resulting mask pairs. Unlike ARI, MBO takes into account background pixels, thus
measuring the degree to which masks conform to objects. We assess this metric using both instance-level  $(\mathrm{MBO}_i)$  and class-level  $(\mathrm{MBO}_c)$  masks.
4.1 Rotating Features Can Represent More Objects
Setup We examine the ability of Rotating Features to represent more objects compared to the CAE [48] using two datasets. First, we employ the 4Shapes dataset, which the CAE was previously shown to fail on. This grayscale dataset contains the same four shapes in each image  $(\square, \triangle, \triangledown, \bigcirc)$ . Second, we create the 10Shapes dataset, featuring ten shapes per image. This dataset is designed to ensure the simplest possible separation between objects, allowing us to test the capacity of Rotating Features to simultaneously represent many objects. To achieve this, we employ a diverse set of object shapes  $(\square, \triangle, \bigcirc$  with varying orientations and sizes) and assign each object a unique color and depth value.