Complex-Valued Autoencoders for Object Discovery
Authors: Sindy Löwe
Phillip Lippe
Maja Rudolph
Max Welling
Venue: Accepted by TMLR ()

Abstract: Object-centric representations form the basis of human perception, and enable us to reason about the world and to systematically generalize to new settings. Currently, most works on unsupervised object discovery focus on slot-based approaches, which explicitly separate the latent representations of individual objects. While the result is easily interpretable, it usually requires the design of involved architectures. In contrast to this, we propose a comparatively simple approach – the Complex AutoEncoder (CAE) – that creates distributed object-centric representations. Following a coding scheme theorized to underlie object representations in biological neurons, its complex-valued activations represent two messages: their magnitudes express the presence of a feature, while the relative phase differences between neurons express which features should be bound together to create joint object representations. In contrast to previous approaches using complex-valued activations for object discovery, we present a fully unsupervised approach that is trained end-to-end – resulting in significant improvements in performance and efficiency. Further, we show that the CAE achieves competitive or better unsupervised object discovery performance on simple multi-object datasets compared to a state-of-the-art slot-based approach while being up to 100 times faster to train.

Published in Transactions on Machine Learning Research (11/2022)
Complex-Valued Autoencoders for Object Discovery
Sindy Löwe
lewue..indy@gmai.l com
UvA-Bosch Delta Lab, University of Amsterdam
Phillip Lippe
p.lippe@uvo.tt
QUVA Lab, University of Amsterdam
Maja Rudolph
maja.rudolphe bosch. com
Bosch Center for Al
Max Welling
m. www.ling@uue.... ml
UvA-Bosck Delta Lab, University of Amsterdam
Reviewed on OpenReview: https: // openreview. net/ pet/formm? id= 1d=
Abstract
Object-centric representations form the basis of human perception, and enable us to reason about the world and to systematically generalize to new settings. Currently, most works on unsupervised object discovery focus on slot-based approaches, which explicitly separate the latent representations individual objects. While the result easily interpretable, it usually requires the design of involved architectures. contrast to this, we propose comparatively simple approach the Complex AutoEncoder (CAE) that creates distributed object-centric representations. Following coding scheme theorized underlie object representations in biological neurons, its complex-valued activations represent two messages: their magnitudes express the presence of a feature, while the relative phase differences between neurons express which features should be bound together to create joint object representations. In contrast to previous approaches using complex-valuee activations for object discovery, we present fully unsupervised approach that trained end-to-end resulting in significant improvements in performance and efficiency. Further, show that the CAE achieves competitive better unsupervised object discovery performance simple multi-object datasets compared to state-of-the-arl slot-based approach while being up to 100 times faster to train.
Introduction
Object discovery plays a crucial role in human perception and cognition (Wertheimer 1922 Koffka, 1935
Köhler 1967). It allows us to interact seamlessly with our environment, to reason about it, and to generalize systematically to new settings. To achieve this, our brain flexibly and dynamically combines information that distrrbuued across the network to represent and relate symbol-like entities, such as objects. The open question of how the brain implements these capabilitics network of relatively fixed connections is known as the binding problem (Grell et al. 2020).
Currently, most work dedicated to solving the binding problem in machine learning focuses slot-based approaches (Hinton et al. 2018 Burgess et al. 2019 Greff et al. 2019 Locatello et al. 2020 Kipf et al. 2022). Here, the latent representations are explicitly separated into "slots" which learn lo represent different objects. These slots are highly interpretable; however, the introduction of a separate object-centine representation module model that otherwise does not exhibit object-centric features causes number of problems. For one, it usually requires the design of involved architectures with iterative procedures, elaborate structural biases, and intricate training schemes to achieve good separation of object features into slots. Moreover,

---

Published in Transactions on Machine Learning Research (11/2022)
fdec
enc
Figure 1: We propose the Complex AutoEncoder (CAE) simple and efficient object discovery approach leveraging complex-vaued activations in autoencoding architecture. Given complex-valued input whose magnitude represents the input image (top left) and whose phase is set to fixed value (bottom left), the model is trained to reconstruct the input image (top right), and learns to represent the disentangled object identitics in its phase values without supervision (bottom right).
this separation often achieved by limiting the information flow and expressiveness of the model, leading to failure cases for complex objects, e.g. with textured surfaces (Karazija et al.) 2021). Finally, since all slots are created at the same level of representation, this approach cannot inherently represent part-whole hierarchies.
To overcome these issues of slot-based approaches, take inspiration from the temporal correlation hypothesis from neuroscience (Singer & Gray 1995 Singer 2009) and design a model that learns representations of objects that are distributed across and embedded the entire architecture. The temporal correlation hypothesis describes a coding scheme that biological neurons are theorized to use to overcome the binding problem. Essentially, posits that each neuron sends two lypes of messages: (1) whether certain feature is present or not, encoded by the discharge frequency or rate code, and (2) which other neurons to bind information to, encoded by the synchronicity of firing patterns.
Following Reichert & Serre 2014)) we abstract away these two messages requiring binary spikes and lemporal dynamics by making use of complex-valued activations in artificial neural networks. This allows each neuron to represent the presence of a feature through the complex numbor's magnitude, and to bind this feature to other neurons' features by aligning its phase value with theirs. After training a Deep Boltzmann Machine (DBM) with real-valued activations, Reichert & Serre (2014) apply this coding scheme at test-time to create object-centric representations. In our experiments, we show that this approach is slow train due to its greedy layerwise training with Contrastive Divergence (Hinton) 2012), that it is slow to evaluate as it requires 100s-1000s of iterations to settle to output configuration, and that it leads to unreliable results: even after training the DBM succossfully as indicated by its reconstruction performance, its created phase values may not be representative of object identity.
To overcome these limitations of existing complex-valued approaches for object discovery, we propose a convolutional autocncoding architecture the Complex AutoEncoder (CAE, Fig. [ that trained end-to- end with colplex-valued activations. First, we introduce several new, but simple, operations that enable each layer to learn to explicitly control the phase shift that it applies to its features. Then, we train the model in fully unsupervised way by using standard mean squared error between its real-valued input and the magnitudes of its complex-valued reconstructions. Interestingly, this simple setup suffices to reliably create phase values representative of object identity in the CAF. Our contributiinns are as follows:
We propose the Complex AutoEncoder (CAE), a convolutional architecture that takes inspiration from the neuroscientifie temporal correlation hypothesis create distributed object-centric features.
We show that the CAE achieves competitive or better object discovery performance on simple, grayscale multi-object datasets compared to SlotAttention (Locatello al. 2020), -tate-of-the-art slot-based approach, while being 10-100 times faster to train.
We show that the CAE achieves significant improvements in performance and efficiency over the DBM approach proposed by Reichert & Serre (2014).
2

---

Published in Transactions on Machine Learning Research (11/2022)
Im
p
Re
The temporal correlation hypothesis. Left: Input image with different objects. Middle: Imple- mentation of the temporal correlation hypothesis with spiking neurons. Their spiking rate represents the presence of feature, while their synchronization represents which features should be bound together to jointly represent object. Right: Implementation of the temporal correlation hypothesis with complex-valued activations. Each complex number e E C is defined by its magnitude m and phase so. This allows for an equivalent representation of feature presence and synchronization through the magnitude and phase values, respectively.
Figure 2:
2 The Temporal Correlation Hypothesis
The Complex AutoEncoder takes inspiration from neuroscience, where the temporal correlation hypothesis describes a possible mechanism underlying object-centric representations in the brain. In this section, we will outline this hypothesis, and draw connection the complex-valie valued activations implemented in our proposed model.
In neuroscience, the binding problem describes the open question of how the brain binds information flexibly and dynamically within network of fixed connectivity to give rise to coherent percepts, c.g. for different objects. Only by overcoming the binding problem, the brain is capable to represent all manner of objects, to attain a compositional understanding of the world, and to generalize robustly to new environments. While there is an ongoing debate as to their functional importance (Shadlen & Movshon 1999 Ray & Maunsell
2010), various works posit that the brain uses temporal dynamics to overcome the binding problem (Milner
1974
Von Der Malsburg & Schneider 1986) Siinger & Gray 1995 Engel et al. 1997 [Singer 1999; Fries
2005 Singer 2009 Fries 2015 Palmigiano et al. 2017). Essentially, these thcorics postulate that the brain binds information from different neurons by synchronizing their firing patterns, while dehynnhrd firing represents information that ought to be processed separately. There are various manifestations of this theory;
in this work, we will focus on the temporal correlation hypothosis (Singer & Gray 1995
Singer 2009).
The temporal correlation hypothesis describes how the oscillating behavior of biological neurons (a.k.a. brain waves) could be leveraged overcome the binding problem. It posits that each neuron sends two messages through its spiking pattern (Fig. - Middle): (1) The discharge frequency
rate code of neuron encodes whether the feature that it is tuned to is present not. The real-valued activation of neurons in artificial neural networks can be interpreted as the technical implementation of this message.
(2) The relative timing between two neurons' spikes encodes whether the represented features of these neurons should be bound together or not. When firing in synchrony, the features they represent will be evaluated jointly by the target neuron and are thus combined in flexible and dynamic way. Currently, very few works explore the use of this second message type artificial ncural networks.
In this paper, take inspiration from the temporal correlation hypothesis to develop machine learning approach capable of overcoming the binding problem. Inspired by previous work (Reichert & Serre 2014), we abstract away from the spiking nature of biological neural networks and instead represent the two message types described above with the help of complex numbers (Fig, 2 Right). As a result, we create an artificial neural network with complex-valted activations 2 = m e. e in which the magnitude m can be interpreted as the rate code emitted by a spiking neuron (message (1) above) and the phase yp can be used as the mathematical mechanism to capture the temporal alignment of the firing patterns (message (2) above). In the next section, we describe how we implement this coding scheme.
3

---

Published in Transactions on Machine Learning Research (11/2022)
3 Complex AutoEncoder
We propose the Complex AutoEncoder (CAE) an object discovery model that leverages mechanisms inspired by the temporal correlation hypothesis to create distributed cejert-centric representations. We start by injecting complex-valued activations into standard autoencoding architecture (Section 3.1p. Ultimately, we want these complex-valued activations to convey two messages: their magnitudes should represent whether a Teature is present, and their phase values should represent which features ought. to be bound together. In Sections 3.2 and 3.3. we describe the setup of the model that gives rise to this coding scheme. As an added benefit, the resulting architecture is equivariant w.r.t. global rotations an aspect of the model which will motivate and outline in Section 3.4 Finally, after unsupervised training on multi-object dataset, the CAE's phase values represent different objects in a scene. In Section 3.5 describe how we discretize these phase values to produce object-wise representations, as well as pixel-accurate segmentation masks for evaluation.
3.1 Complex- Valued Activations in Autoencoders
To enable an autoencoder to develop object-centric representations, we equip it with complex-valued activations. In this section, we will describe how we translate between the real-valued inputs and outputs used for training the model and the complex-valued activations used for representing object-centric features.
takes a positive, real-valued input image x' E Bgxx with height h and The Complex AutoEncoder (Fig- 1
width w and associates each pixel with an initial phase yp = 0 E gax to create the complex-valee valued input x to the model:
x= e
E Chx
(1)
where o denotes the Hadamard product. The CAE applies a convolutional encoder fenc and decoder fdee with real-valued parameters to this complex-valued input to create a complex-valued reconstruction 2 = faee E Chxw. To make use of cxisting deep learning frameworks, in our implementation, we do not apply layers to their complex-valued inputs directly. Instead, each layer extracts real-valued components (the real and imaginary part, or the magnitude and phase) from its input and processes them separately, before combining the results into a complex-valued output. We will describe this process in more detail in the following section.
We create the real-valued reconstruction & by applying fout, a 1 x 1 convolutional layer with a sigmoid activation function, on the magnitudes of the complex-valued output 2 of the decoder: & = fout ([2)) E hxx This allows the model to learn an appropriate scaling and shift of the magnitudes to better match the input values. The model is trained by comparing this reconstruction to the original input image using a mean squared error loss L MSE(x', x) E iR+ and by using the resulting gradients to update the model parameters.
Finally, we interpret the phase values & arg(z) [0, 2oo of the dimeenimnl,,,,,,, complex-valued activations z E Cdaae as object assignments - either to extract object-wise representations from the latent space or to obtain a pixcl-accurate segmentation mask in the output space. Here, arg(z) describes the angles between the positive real axis and the lines joining the origin and each element in z.
3.2 Phase Alignment of Complex Numbers
For the CAE to accomplish good object discovery performance, the phases of activations representing the same object should be synchronized, while activations induced by different objects should be desynchronized. To achieve this, we need to enable and encourage the network to assign the same phases to some activations and different phases to others, and to precisely control phase shifts throughout the network. We accomplish this by implementing the following three design principles in each network layer fe E {fenes faee) parameterized by 0 E R (where l represents the number of parameters in that layer) and applied to the dimmdimnnionn input to that layer z e Cdi:
Synchronization First, we need to encourage the network to synchronize the phase values of features that should be bound together. This principle is fulfilled naturally when using additive operations wetween
4

---

Published in Transactions on Machine Learning Research (11/2022)
complex numbers: when adding two complex numbers of opposing phases, they suppress one another even cancel one another out (a.k.a. destructive interference). Thus, to preserve features, the network needs to align their phase values (a.k.a. constructive interference).
Desynchronization Next, we need a mechanism that can desynchronize the phase values. Again, this is achieved naturally when using additive operations between complex numbers: when adding two complex numbers with phase difference of 90º, for example, the result will lie in between these two numbers and thus be shifted, ie. desynchronized by 45". On top of this inherent mechanism, we add second mechanism that lends the network more control over the precise phase shifts. Specifically, we first apply the weights w E 0 of each layer separately to the real and imaginary components of its input:
a = fw (z) fw (Re(z)) + fw (Im(x)) - E Cdom
(2)
where dout represents the output dimensionality of the layer and denotes scalar multiplication. Then, we add separate biases bm, by E 0 to the magnitudes and phases of the resulting complex-valued representations ab to create the intermediate magnitude my and phase Pp:
m
| + bm
SΦ arg((b) + b,p E &oot
(3)
This formulation allows the model to influence the phase value of each activation directly through the bias byp and thus to learn explicit phase shifts throughout the network. Additionally, the bias b, allows the model to break the symmetry created by the equal phase initialization (Eq.
Gating Finally, we add a gating mechanism that selectively weakens out-of-phase inputs. We implement this by applying each layer to the magnitude of its input and combine the result with my to create the intermediate values mz:
X fw ([z)) + bm
E R
(4)
my = 0.5 my + 0.5 X E RR
As shown by Reichert & Serre 2014). this implementation effectively masks out inputs that have an opposite phase compared to the overall output. Formally speaking, given a set of input features Yn with the same phase value and an out-of-phase input feature 22 with a maximally distant phase] [ if [W1, 2 >> |w>za|, (w1.Izz) (zxl) 0 and w2 > 0, the net contribution of 22 to the output is zerd Besides this hard gating for maximally out-of-phase imputs, this mechanism further leads to progressivcly softer gating for smaller phase differences. Overall, this allows for the flexible binding of features by determining the effective connectivity between neurons based on the phase values of their activations. Since features with different phase values are processed separately, this ultimately encourages the network to assign different phase values to the features of different objects.
3.3 Complex- Valued Activation Function
We propose a new activation function for complex-valted activations to further ensure maximal control of the network over all phase shifts. To create a layer's final output x', we apply a non-linearity on the intermediate values mz, but keep the phases 4 unchanged:
z' = =BeLU/BatchNoom(m))) o e Cdoat
(5)
There are several things to note about this setup. First, my is positive valued, unless the magnitude bias bm pushes it into the negative domain (Eq. (3 Eq. ( Second, by applying BatchNormalization (Loffe) & Szegedy 2015), we ensure that at least imitially mm bocomes zero-centerod and therefore makes use of the non-linear part of the ReLU activation function (Krizhevsky et al. 2012). At the same time, BatealNormaiiiation provides the flexibility to learn to shift and scale these values if appropriate. Finally, the ReLU non-linearity ensures that the magnitude of z' is positive and thus prevents any phase flips.
We say that n complex numbers have "maximally distant phases" if they maximize their pairwise cosine distance to one another.
'Denoting the inner product as ( and leaving out the bias bn for clarity, the proof for this is: m, = 21)+ (01))++050www [z, wa (al)) = 0.5- 0.5-(1(w1, - 22)) 0.5-((w1, (2,1)) + (a/g/2/))) |=al) = 0.5-(1(w1, (5)((w),,)))+0033 +.55) [z)
5

---

Published in Transactions on Machine Learning Research (11/2022)
2
2
Input
Reconstruction
Reconstruction
Phase Values
Prediction
Prediction
Prediction
AutoEncoder
Complex AutoEncoder
DBM
SlotAttention
Figure 3: Visual comparison the performance of the Complex AutoEncoder, a corresponding real-valued autoencoder, DBM model and SlotAttention model on random test-samples of the 2Shapes (Top), 3Shapes (Middle) and &NIST&Shape (Bottom) datasets. Areas in which objects overlap are removed before evaluation, resulting in ycllow arcas in the predictions of the CAE and the DBM model. The Complex AutoEncoder produces accurate reconstructions and object separations.
3.4 Global Rotation Equivariance
Given the setup as described above, the CAE is equivariant with regard to global rotations, i.e. to global phase shifts of the input.
Proposition 3.1. Let CAE(x) = feeo(feno(x)) E Chxw be the output of the autoencoding model given the input x E Chxw_ Let a E [0,2n) 2 an arbitrary phase offset. Then, the following holds:
CAE(x (cos(a) + sin(a)))) CAE(x) (cos(a) + sin(a)i)
(6)
where multiplication with cos(a) + sin(a)) corresponds to counter-clockuise rotation by a.
The proof and empirical validation for this proposition be found in Appendix D.1 This means that when we shift the phase value associated with the pixels of an input image (e.g. by choosing different value for so in Eq. (T)). the output phases will be shifted correspondingly, but the output magnitudes will remain unchanged. This property is important when considering the temporal analogy underlying the CAE: the point in time at which stimulus is perceived (here: input phase) should influence when the stimulus is processed (here: output phase), but it should not influence how it is processed (here: output magnitudes). Additionally, without this equivaríance property, the model could use the absolute phase values to learn to separate objects instead of binding features "by synchrony", i.e. based their relative phase differences to one another.
3.5 Creating Discrete Object Assignments from Continuous Phase Values
To extract object-wise representations from the latent space, well as pixel-accurate segmentation masks from the output space, we create discrcte object assignments for cach feature by applying a clustering procedure lo the phase values. We start this clustering procedure with two pre-processing steps. First, we account for the circular nature of the phase values by mapping them onto unit circle. This prevents values close and 2x from being assigned to different clusters despite representing similar angles. Then, account for the fact that the phase values complex numbers with small magnitudes become increasingly random, scale features by factor of 10 if their corresponding magnitude m 0.1. As result of this

---

Published in Transactions on Machine Learning Research (11/2022)
Table 1: MSE and ARI scores (mean + standard error across seeds) for the Complex AutoEneoder, its real-valued counterpart (AutoEncoder), a DBM model and SlotAttention model in three simple multi-object dalasets. The proposed Complex AutoEncoder achieves belter reconstruction performance than its real-valued counterpart on all three datasets. Additionally, it significantly outperforms the DBM model across all three datascts, and its object discovery performance is competitive to SlotAttention on the 2Shapes and 3Shapes datasets. Finally, the CAE manages to disentangle the objects in the MNIST&&Shape dataset, where both
DBM and StotAttention fail.
Dataset
Model
MSE +
ARI+BG 1
ARI-BG 1
Complex AutoEncoder
3.322c-04
0.999
1.000
2Shapes
AutoEncoder
5.565e-04
2.900e-04
DBM
3.308e-03
1.024c-04
0.920 + 0.002
0.744 + 0.010
Attentton
1.419c-04
1.410e-04"
0.812
+ 0.081
1.000 + 0.000
Complex AutoEncoder
1.313c-04
0.976
0.002
1.000
3Shapes
AutoFncoder
8.568e-04
9.8786-05
DBM
1.045e-02 1.494c-04
0.856 + 0.006
0.419 0.028
Stotntttnttoon
1.827c-04
3.125e-05*
0.107
:: 0.008
0.997 0.001
Complex AutoEncoder
3.185e-03
1.514e-04
0.783
0.004
0.971 0.011
&hipT&&Shape
AutoFncoder
5.792e-03
5.553e-04
DBM
1.560e-02
8.000c.05*
0.718 + 0.002
0.175 0.006
SlotAttention
5.438e-03
1.807e-04'
0.047 + 0.013
0.089 + 0.028
"The grayed-out performances are not comparable due the use different autoencoding setupe.
scaling, these features will fall within the unit circle, close to the origin. In our experiments, we find that they tend to be assigned their own cluster and usually represent the background. As outlined in Appendix C. this re-scaling of features is crucial to ensure little noise in the evaluation procedure, but currently prevents the CAE from being applied to RGB data. Finally, we apply k-means, with & corresponding to the number of objects in the input plus one for the background, and interpret the resulting cluster assignment for cach phase value as the predicted object assignment of the corresponding feature.
Note that we could replace k-means with any other clustering algorithm to relax the requirement of knowing the number of objects in advance. Additionally, this discretization of object assignments only required for the evaluation of the CAE. During training, the CAE learns continuous object assignments through its phase values and thus creates the appropriate amount of clusters automatically.
4 Results
In this soction, we evaluate whether the Complex AutoEncoder can learn create mcaningful phase separations representing different objects in an unsupervised way. We will first describe the general setuu of our experiments, before investigating the results across various settings. Our code is publicly available at httte///gtthaa..oooorrrerr/////d
4.1 Setup
Datasets We evaluate the Complex AutoEncoder on three grayscale datasets: 2Shapes, 3Shapes, and MNIST&&Shape. For each of these datasets, we generate 50,000 training images and 10,000 images for validation and testing, respectively. All images contain 32 x 32 pixels. The 2Shapes dataset represents the casiest setting, with two randomly placed objects (O, A) in each image. The 3Shapes datasct contains third randomly placed object (v) per image. This creates a slightly more complex setting due to the higher object count, the two similar shapes (4, v), and stronger overlap between objects. Finally, the MNIST&SShape datasct combines an MNIST digit (LeCun et al.) 2010) and a randomly placed shape (0 or v) in cach image. This creates challenging setting with more diverse objects. Finally, for evaluation, we generate pixel-accurate segmentation masks for all images. More details in Appendix C.4
7

---

Published in Transactions on Machine Learning Research (11/2022)
Model & Training We make use of fairly standard convolutional autoencoder architecture, presented in Lippe] (2027) (details in Appendix C.1). We train the model using Adam (Kingma & Ba 2015) and a batch-size of 64 for 10,000 100,000 steps depending on the dataset. Within the first 500 steps of training, we linearly warm up the learning rate (Goyal et al.) 2017) to its final value of 1e-3. All experiments are implemented in PyTorch 1.10 (Paszke et al. 2019) and were run on a single Nvidia GTX 1080Ti. To ensure the comparability of runtimes between models, all experiments were run on the same machine and with the same underlying implementations for data-loading, training and evaluation wherever possible.
Basclines We compare the CAE to three baseline models. First, compare it against a Deep Boltzmann Machine (DBM; Salakhuudinov & Hinton (2009) that closely follows Reichert & Serre (2014). We Lest two architectures for this model, one resembling the setup of the CAE (6-layer DBM) and one following the architecture used by Reichert & Serre (2014) (3-layer DBM), and report the results of the best performing model for each dataset. For a full breakdown of all results, see Appendix D.2 Additionally, we provide an in-depth discussion on the limitations of our re-implementation in Appendix C.2 The second baseline we consider is a state-of-the-art slot-based approach: SlotAttention {Locatello et al.) 2020). SlotAttention is iterative attention mechanism that produces i slots which learn to represent individual objects in the input. It has achieved remarkable unsupervised object discovery results on synthetic multi-object datasets, while being more memory efficient and faster to train than other slot-based approaches. For more details, see Appendix C.3 Finally, we compare the CAE against corresponding real-valued autoencoder. This model uses the same general architecture and training procedure as the CAE, but does not employ complex-valuee activations or any of the mechanisms described in Section 3.2 It does, however, apply BatchNormaalization before cach ReLU as we have found that this improves performance in the real domain as well.
Metrics We use three metrics to evaluate the performance of the CAE and to compare it with the baselines. We measure the reconstruction performance in terms of mean squared error (MSE). To assess the object discovery performance, we compute Adjusted Rand Index (ART) scores (Rand 1971 Hubert & Arabie 1985). ARI mcasures clustering similarity, where score of O indicates chance level and a score of 1 indicates a perfect match. We utilize this metric in two ways. First, following previous work |Grelf et al. 2019 Locatello el al.) 2020), we evaluate "ARI-BG" where we exclude the background labels from the evaluation. Additionally, we assoss "ARI+BG" which evaluates the performance on all pixels. One thing to note here is that a model can achieve relatively high ARI+BG scores by assigning all pixels to the same label thus, this score is only meaningful in conjunction with a high ARI-BG score to ensure good object separation. For both ART scores, remove areas in which objects overlap from the evaluation, as they are ambignous in the grayscale setting.
4.2 Evaluation
First, compare the quantitative performance of the CAE against the three baselines in Table L
Object Discovery Performance The Complez AutoFncoder achieves considerably better object discovery performance than the DBM model across all three tested datasets. In fact, we find that our re-implementation of the DBM model only achieves consistent object separation in the simplest 2Shapes dataset, and largely fails on the other two - despite achieving a reasonable reconstruction performance on all datasets (see Fig. in the Appendix). When comparing the object discovery performance of the CAE against SlotAttention, we make three observations: (1) Both models achieve (near) perfect ARI-BG scores on the 2Shapes and 3Shapes datasets. (2) On all datasets, the CAE achieves considerably better ARI+BG scores indicating more accurate separation of foreground and background.
(3) On the MNIST&Shapee dataset, the CAE achieves an almost perfect ARI-BG score, while Altentitns performance is close to chance level (we provide a detailed analysis of this failure mode of SlotAttention in Appendix C.3 Overall, this shows that the Complex AutoEncoder achieves strong object discovery performance on the three datasets considered. On top of this, despite its simple and efficient design, it can overcome the challenges set by the MNIST&Shan dataset (high diversity in object shapes and relatively large object sizes), while neither the DBM model nor SlotAttention can.
8

---

Published in Transactions on Machine Learning Research (11/2022)
q
9
2
2
9
O
c
150
180 180
9
270
Figure Phase separation in the CAE. Top: Out- put phase images. Bottom: Plotting every output value in the complex plane and applying the same color coding as above. The cluster centroids of the phase values belonging to the individual objects have almost maximally distant phases. Interestingly, arcas in which the objects overlap get assigned intermediate phase values.
Figure 5: Investigating object-centricity of the latent features the CAF. Columns 4: input images. Columns 2-3 & 5-6: object-wise reconstructions. By clustering features created by the encoder according to their phase values, we can extract representations of the individual objects and reconstruct them sepa- rately.
Reconstruction Performance The Complez AutoEncoder creates more accurate reconstructions than its real-valued counterpart (AutoEncoder) on all three multi-object datascts. This illustrates that the CAE uses complex-valued activations to creale better disentanglement of object features.
Object- Centric Representations To evaluate whether the CAE creates object-centric representations throughout the model, cluster the latent. features created by the encoder (Section 3.5), and fine-tune the decoder to reconstruct separate objects from the individual clusters. As shown in Fig, 5 this allows to create accurate reconstructions of the individual objects. We conclude that the latent representations of our model are object-centric: the representations of the individual objects can be clearly separated from one another based on their phase values, and they contain all the necessary information for the individual objects to reconstructed accurately.
Qualitative Evaluation In Fig. 3 we show exemplary outputs of the four compared models for cach dataset (for more results, see Appendix D.6h. When looking more closely at the phase separation created by the Complex AutoEncoder shown in Fig. we find that it assigns almost maximally distant phase values to the different objects. Interestingly, the phase values of overlapping areas tend to fall in between the phase values of the individual objects. Since it is ambignous in the 3Shapes dataset, which object is in the foreground and which one in the background, this shows that the model accurately expresses the uncertainty that it encounters these overlapping areas.
Dataset Sensitivity Analysis We explore the capabilities and limitations of the CAE in different datasot. settings. To achieve this, we generate several variations of the grayscale datasets considered before. For a detailed description of the datasets considered, as well full breakdown of the results, see Appendix D.4
Variable backgrounds. We implement a version of the 2Shapes dataset in which we randomly sample the background color of each image. Since the CAE achieves an ARI-BG score of 0.995 + 0.001, we conclude that it can handle images with variable backgrounds.
Duplicate objects. We assess the CAE on a datasct in which each image contains two triangles of the same orientation. The CAE achieves an ARI-BG score of 0.948 + 0.045 here, indicating that it can handle multiple instances of the same object within image.
9

---

Published in Transactions on Machine Learning Research (11/2022)
Object count. To investigate whether the CAE can represent a larger number of objects, we construct dataset with four objects per image. This builds on the 3Shapes dataset, but adds an additional circle to each image. On this dataset, the performance of the CAE drops substantially to an ARI-BG score of 0.669 + 0.029. To distinguish whether this lowered performance is due to the larger object count per image or due to the larger variability of objects, we implemented a dataset with five different shapes (Tour of which are the same as the previous dataset, plus an additional larger circle) out of which two are randomly sampled for each image. Here, the CAE' achieves an ARI-BG score of 0.969 + 0.007 highlighting that it is mostly restricted in the number of objects it can represent at once, but that it is able to represent larger range of object types across images.
Generolization. We test whether the CAE generalizes to different object counts. For this, we train the model on the 2Shapes dataset and test it on a dataset in which each image contains 3 shapes (one square and two triangles of the same orientation). We find that the performance drops substantially this setting, with the model achieving an ARI-BG score of 0.758 + 0.007. Thus, it scems that the CAE does not generalize to more objects than observed during training. It. doess however, generalize to settings in which fewer objects are present: after training the CAE on the 3Shapes dataset, its performance remains high at 0.956 + 0.019 ARI-BG when tested on a variation of this datasct where images may contain either onc, two or three objects.
Model Sensitivity Analysis We evaluate the influence of certain design decisions on the performance of the Complex AutoEncoder in Table
Table 2: Model sensitivity analysis on the 2Shapes dataset (mean + standard error across 8 seeds). We find that there are several crucial components contributing to the CAE's ability to achieve meaningful phase separation without su- pervision.
2 We find that without the phase- bbas by (Eq. (3) ) the model is unable to push its activa- tions off the real-axis and therefore achieves no meaningful phase separation between objects; without the X term (Eq. (4). activations are not gated according to their phases and the model has no incentive to separate the object representations; and without BatchNormalliz-
Name
MSE
ARI-BG
Complex AutoEncoder
3.322e-04 e 18886-00
1.000 0.000
b4
5.127e-04 e u. 0.7035-05
0.100 = 0.055
tion (Eq. (5), the network cannot make appro- priate use of the non-lincar part of the ReLU fout
3. 3.227e-03
5.013t-04
0.074
0.08s
BatchNorm
6. 6.165e-02 :: 2.2286-02
0.373
0.337
2.462e-03
1.4086-0s
0.939 0.099
leading to inferior performance. In all three
cases, this results in the CAE failing to achieve any meaningful object separation. While the functionality that each of these components provides is crucial for the CAE's performance, alternative approaches implementing similar functionalities may be possible. Our results further indicate that the final 1 convolutional layer fout that creates the real-valued reconstructions from the magnitudes of the complex-valuee outputs improves the model's performance. Finally, we observe that the bottleneck size has relatively little influence the performance of the Complex AutoEncoder (Fig. 9 in the Appendix). This indicates that the CAE does not need to restrict the expressivity of the model to create disentangled object representations.
Global Rotation Equivariance We visually highlight the global rotation equivariance property of the CAE in Fig. 6 As expected, a shift in the input phase value leads to a corresponding shift in the output phases, but leaves the output magnitudes unchanged.
Runtime Due to its non-iterative design and its end-to-end optimization with comparatively few training steps, the CAE is between 1.1 and 21.0 times faster to train than the DBM model, and between 10.0 and 99.8 times faster than SlotAttention. The precise values depend on the dataset and DBM architecture used. On the 2Shapes dataset, for example, the CAE is 2.1 times faster to train than the DBM model and 99.8 times faster than Attentitn Besides this, we find that the discretization of phase values with k-means leads to significantly slower evaluation times for both the CAE and DBM. As a result, the CAE is approximately 10 times slower evaluate than Attenninn. The DBM is slowed down even further by its iterative scttling procedure, leading to 2.0 16.6 times slower evaluation times compared to the CAE See Appendix D.3 for a detailed breakdown of all runtimes.
10

---

Published in Transactions on Machine Learning Research (11/2022)
3
Input image and plases
3
3
3
Reconstructions
3
Outpat platses
Output magnittdes and platees
Figure 6: Visual evaluation of the global rotation equivariance of the CAE. Given an input with different phase values (1st row), the output magnitudes (and therefore the reconstructions) remain unchanged (2nd/4th row), but the output phase values shift correspondingly (3rd/ 44th row).
5 Related Work
Object Discovery There is broad range of approaches attempting to solve the binding problem in artificial neural networks (see Greff et al. (2020) for great overview). However, most works focus on one particular aepreenttaiinal format for objects: slots. These slots create an explicit separation of the latent representations for different objects. Additionally, they create an explicit separation between a specialized object-centric representation part of the model and non-object-ceutric parts that merely support the former, for example, through encoding and decoding functionality. In contrast to this, the Complex AutoEncoder embeds object-centricity into the entire architecture and creates distributed object-centric representation ations.
To create slot-based object-centric representations, different mechanisms have been proposed to overcome the binding problem, i.c. to break the symmetry between the representations of different objects in a network of fixed weights. One way to break this symmetry is to enforce an order along certain axis. This can be achicved by imposing an order on the slots (Eslami et al. 2016 Burgess ct al.
2019: Engeleke et al. 2020
by assigning slots to particular spatial coordinates (Santoro el al. [017 Craw & Pineau 2019 Lin el al. 2020), or by learning specialized slots for different object types (Hinton et al.
2011 2018). Approaches that
create the most general slot reprosentations do not enforce any such order, but
require iterative procedures to
break the symmetries instead (Greff et al. 2016 2017 2019 Goyal et al.
2021 Kosiorek et al. 2020 Stelzner
et al. 2021 Du et al. 2021). Atoentiontion (Locatello et al. 2020), which falls into this final category, breaks
the symmetry between slots through an iterative attention mechanism. Many recent works make use of
Slotntttntion: Kipf et al. ): Singh et al. (2022): Elsayed et al. (2022) extend it to vidco data, Singh
et al. (2021) propose an image generation model utilizing Aloentinnion;; and Löwe et al. (2020) propose a
self-supervised training approach for SlotAttention.
Object Discovery with Complex- Valued Networks A variety of research has explored different activation functions, training regimes, and applications for complex-valued neural networks (see Bassey et al. (2021) for a review). Despite this, there has been little research on the use of complex-valued networks for object discovery. The earliest works in this direction are by Mozer et al. (1992); Zemel et al. (1995).
Their architectures learn to assign different phase values to different objects through a supervised training procedure. Rao et al. (2008); Rao & Cocchi (2010 2011) enable their complex- valued neural networks to separate overlapping objects on test. images by training them on images of individual objects of the same lype. Finally, the only existing complex-valiee valued network for object discovery that is fully unsupervised was developed by Reichert Serre (2014). They train a real-valued Deep Boltzmann Machine on datascts similar to the ones presented here. At test time, they inject complex-valued activations create phases representative of object identity.
11

---

Published in Transactions on Machine Learning Research (11/2022)
All these methods initialize the complex-valued inputs to their networks with the magnitudes of the imput images and with random phase values. As result, they require iterative procedures with 10s-1000s of iterations to settle to an output configuration. In contrast to that, the proposed Complex AutoEncoder initializes all phase values with a fixed value and after fully unsupervised, end-to-end training with complex- valued activations only requires a single forward-pass through the model. Further, the CAE the first. complex-valuct network for object discovery that has been shown to be equivariant to global rotations. All together, this greatly improves the efficiency and performance of complex-valued neural networks for object discovery.
6 Conclusion
Summary We present the Complex AutoEncoder an object discovery approach that takes inspiration from ncuroscience to implement distributed object-centric representations. After introducing complex-value aclivations into convolut ional autoencoder, it learns to encode feature information in the activations' magnitudes and object affiliation in their phase values. We show that this simple and fully unsupervised setup suffices to achieve strong object discovery results on simple multi-object datasets while being fast to train.
Limitations and Future Work The proposed Complex AutoEncoder constitutes a first step towards efficient distributed object-centric representation learning, but some limitations remain. Most importantly, Attention and other slot-based approaches generally applicable to more challenging datasets: they can handle RGB inputs, for example, and images with larger number of objects. Nonetheless, the Complex AutoEncoder provides important step forward by proposing simple and elficient non-iterative design that considerably outperforms previous complex-valued approaches and, for the first time, was shown to achieve competitive results to a slot-based approach in simple multi-object datasets. It remains an intriguing direction for future research to overcome its current limitations and to uncover the full potential of distributed object-centric representation learning approaches.
Acknowledgements
We thank David P. Reichert and Thomas Serre for their helpful guidance for re-implementing their proposed DBM model. Additionally, we thank Emiel Hoogoboom, T. Anderson Keller, Joop Pascha and Jascha Sohl-Dickstein for their valuable feedback on the manuscript.
References
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. arXiu:1607 2016.
Layer normalization.
arXiv preprint
Joshua Bassey, Lijun Qian, and Xianfang Li. A survey of complex-valued neural networks. arXie preprint arXiv:2101.12248 2021.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representation. arXiv preprinl arXiv:1901.1113 2019.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3412 -4200, 2019.
Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Adoonces in Neural Information Processing Systems, 34:15608 15620, 2021.
Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. In Alice H.
12

---

Published in Transactions on Machine Learning Research (11/2022)
Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advonces in Newral Information Processing Systems, 2022.
Andreas K Engel, Pieter R Roelfsema, Pascal Fries, Michael Brecht, and Wolf Singer. Role of the temporal domain for response selection and perceptual binding. Cerebral Corter (New York, NY: 1991), 7(6):571-582, 1997.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmarr Posner. GENESIS: Generative scene inference and sampling with object-centric latent representations. International Conference on Learning Representations (ICLR), 2020.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neurol Information Processing Systems, pp. 3225-3233, 2016.
Pascal Fries. A mechanism for cognitive dynamics: neuronal communication through neuronal coherence.
Trends in Cognitive Sciences, 9(10):4744880, 2005.
Pascal Fries. Rhythms for cognition: communication through coherence. Neuron, 88(1):220 235, 2015.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. International Conference on Learning Representations (ICLR), 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jía, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arx; 1706.006000 026777 2017.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger:
Deep unsupervised perceptual grouping. Advances in Newral Information Processing Systems, 29, 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning (ICML), pp. 2424-2433, 2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. Xrxiv preprint Xrx:2001 05208, 2020.
Geoffrey E Hinton. A practical guide to training restricted boltzmann machines. In Neural Networks: Tricks of the Trade, pp. 599 619. Springer, 2012.
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In International Conference on Artificial Neural Networks, pp. 44-51. Springer, 2011.
GeofTrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In International Conference on Learning Representations (ICLR), 2018.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 2(1):193 218, 1985.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pp. 448 456, 2015.
Laurynas Karazija, Iro Laina, and Christian Rupprocht. Clevrtex: A texturo-rich benchmark for unsupervisod multi-object segmentation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Represenlations (ICLR), 2015.
Thomas Kipl, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. International Conference on Learning Representations (ICLR), 2022.
Kurt Koffka. Principles gestalt psychology. Philosophy and Scientific Method, 32(8). 1985.
Wolfgang Köhler. Gestalt psychology. Psychologische Forschung, 31(1:XVVII-XXX 1967.
Adam R Kosiorek, Hyunjik Kim, and Danilo J Rezende. Conditional set generation with transformers. ICML 2020 Object- Oriented Learning Workshop, 2020.
Alex Krizhevsky, Πya Sutskever, and Geoffrey E Hinton. Imagenct classification with doep convolutional neural networks. Advances in Newral Information Processing Systems, 25, 2012.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online].
Available: http://yann.lecne lectnnneeccucccm cowee/ccmeeeeee /erdb /mnist, 2010.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. International Conference on Learning Representations (ICLR), 2020.
Phillip Lippe. UvA Deep Learning Tutorials, 2021. URL http://ooottbooooeeeeettteeeeeee
io
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 2020.
Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. arXiv preprint vrXic:2011.1020 2020.
Peter M Milner. A model for visual shape recognition. Psychological Review, (6)):52,, 1974.
Michael C Mozer, Richard Zemel, Marlene Behrmann, and Christopher KI Williams. Learning to segment. images using dynamic feature binding. Neural Computation, 4(5):650 665, 1992.
Agostina Palmigiano, Theo Geisel, Fred Wolf, and Demian Battaglia. Flexible information routing by transient synchrony. Nature Neuroscience, 20(7):1014 1022, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit, Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- perf-errffrmeece deep learning library. Advances in Neural Information Processing Systems 32, pp. 8024 8035, 2019.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the Americon Statistical Association, 66(336):846-550, 1971.
A Ravishankar Rao and Guillermo A Cecchi. An objective function utilizing complex sparsity for efficient segmentation in multi-layer oscillatory networks. International Journal of Intelligent Compuling and Cybernetics, 2010.
A Ravishankar Rao and Guillermo A Cecchi. The effects of feedback and lateral connections perceptual processing: A study using oscillatory networks. In The 2011 International Joint Conference on Newral Networks, pp. 1177-1184. IEEE, 2011.
A Ravishankar Rao, Guillermo Cecchi, Charles C Pock, and James Kozloski. Unsupervised segmentation with dynamical units. IEEE Transactions on Neural Networks, 19(1):168-18) 182, 2008.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Supratim Ray and John HR Maunsell. Differences in gamma frequencies across visual cortex restrict their possible use in computation. Newron, (7()))885-888) 2010.
David Reichert and Thomas Serre. Neuronal synchrony in complex-valle valued deep networks. International Conference Learning Representations (TCIR); 2014.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5, pp. 448 455. PMLR, 2009.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems, pp. 4967-4977 2017.
Michael N Shadlen and J Anthony Movshon. Synchrony unbound: review critical evaluation of the temporal binding hypothesis. Newron, 24:67-77 77, 1999.
Wolf Singer. Neuronal synchrony: a versatile code for the definition of relations? Nearon, 24(1):49 65, 1999.
Wolf Singer. Distributed processing and temporal codes in neuronal networks. Cognitive Neurodymamics, 3 (3):189-199) 196, 2009.
Wolf Singer and Charles M Gray. Visual feature integration and the temporal correlation hypothesis. Annual Review of Newroscience, 18(1):555 586, 1995.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns compose. International Conference on Learning Representations, 2021.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic vidcos. In Alice H. Oh, Alckh Agarwal, Daniclle Belgrave, and Kyunghyun Cho (ods.), Advances in Neural Information Processing Systems, 2022.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01148, 2021.
Tijmen Tieleman. Training restricted boltzmann machines using approximations the likelihood gradient. In International Conference on Machine Learning (ICML), pp. 1064-1071, 2008.
Christoph Von Der Malsburg and Werner Schneider. A neural cocktail-party processor. Biologieal Cybernelics, 54(1):29-40, 1986.
Max Wertheimer. Untersuchungen zur Lehre von der Gestalt. Psychologiache Forschung, 1(1):47-55 58, 1922. Richard S Zemel, Christopher KI Williams, and Michael C Mozer. Lending direction to neural networks. Newral Networks, 8(4):503-52) -12, 1995.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
A Broader Impact
The Complex AutoEncoder is a comparatively simple object discovery approach that learns to accurately and reliably separate the phase values for different objects without supervision. On the simple multi-object dalasets that we consider, achieves highly promising results. As such, we believe opens up a new research direction for object-centric representation learning, which investigates distributed instead of slot-based representations. We further believe that the CAE outlines general approach for object discovery that can be extended to wide range of domains and applications, and expect that the object-centric representations that it creates from perceptual input may prove useful to create more transparent and interpretable predictions. We do not foresee any potential negative societal impacts, but acknowledge that given the broad range of possible applications of object discovery methods, there might be potential impacts that we cannot foresee at. the current time.
B Reproducibility Statement
To ensure the reproducibility of our experiments, we provide a detailed overview of the model architectures, hyperparameters and additional implementation details in Appendix C Besides this, we published the code used produce the main experimental results at |nttps://gi
We repeated all experiments with eight different seeds to obtaín stable and reproducible results. The overall compulation time for all experiments, including the baselines, corresponds to approxinately 40 GPU days Nvidia GTX 1080Ti (not including hyperparameter search and trials throughout the research process).
C Experimental Details
C.1 (Complex) AutoEncoder
Table 3 shows the architecture of the Complex AutoEncoder, as well as its real-valued counterpart. We used the default parameter initialization of PyTorch for all layers, except fout for which we set the initial weight w 1 and the initial bias = Additionally, we initialize all phase-biases b, with zero. We introduced the magnitude bias to create a more consistent overall formulation, which applics a bias on both the magnitude and the phase. However, this term is not strictly needed, as is effectively canceled out by the BatchNornalization that we apply in Eq. ())- As result, we experience that the magnitude bias does not have noticeable impact in experiments. After the linear layers, apply Layer Normalization (Ba ct al. 2016) instead of Batch Normalization.
We optimized all hyperparameters, except the number of training steps, on the validation set of the 3Shapes dataset and subsequently applied them for the training on all datasets. Across the board, we found that each hyperparameter setting that improved the performance of the Complex AutoEncoder also improved the performance of the real-valued autoencoder and vice versa. As result, we the same hyperparameters to train both models.
To create the object-wise reconstructions in Fig. 5 we first cluster the features created by the encoder fene by following the procedure described in Section 3.5 Then, we mask out all values that are not part of a particular cluster with zeros. Finally, we finc-tune the decoder to reconstruct individual objects given these masked-out feature vectors for 10,000 steps using Adam with a learning rate of 5e-5. Since our network generally assigns the same phase values to the same object types, we manually match the reconstructions created by the separate feature vectors the respective objects once and use the same assignment for the rest of the dataset.
16

---

Published in Transactions on Machine Learning Research (11/2022)
Table 3: Autoencoding architecture used for the Complex AutoEncoder, well as the real-valued autoencoder
bascline. Additionally, the l-yayer DBM uses setup that closoly resembles the encoder fene as outlined in this table.
Layer
Feature Dimension
Kernel
Stride
Padding
Activation Function
(H
W X C)
Input / Output
Conv
16 x 16 x 32
3
0
Complex- ReLU
Conv
16 x 16 x 32
3
/ 0
(Complex- ReLU
Conv
8 8 x
3
/ 0
(Complex- ReLU
fenc
Conv
8 8 8 x 64
3
1/0 0
(Complex- ReLU
Cony
x 4 x 64
3
1 1/0
(Complex- ReLU
Reshape
1 1 x 1024
Linear
X 1 x 64
Remppeex-RReLUU (Complex- ReLU
Linear
X 1
1024
(Complex ReLU
Reshape
4 4
64
TransConv
8 8
64
3
2
1 1/1
(Complex- ReLU
Reel
fee
Conv
8
64
3
1/0
(Complex- ReLU
TransConv
16
16
32
3
1/1
(Complex- )ReLU
ReeL
Conv
16
16
32
3
1 1/0
(Complex-)Roe
TransConv
32
1
3
1/1
Commlee-)Rella
RGB images In its current form, the CAE cannot be applied to RGB data. This restriction is largely due to its evaluation method rather than the model itself. In fact, it is possible to simply scale the input/output dimensions to three channels and train the CAE on RGB data - the problem is that it unclear how to evaluate the resulting phase values in a convincing way. This problem arises from the re-scaling of features with small magnitudes.
During evaluation, we re-scale features with small magnitudes to avoid unnecessary noise (Section 3.5 The closer feature is to the origin, the less its phase will influence how it will be processed by the network, and result, the network has no incentive to assign anything but random phase values to these features. To evaluate how well the phases separate the objects in the scene, we need to evaluate them as independently of the magnitudes as possible. To achieve this, we normalize most features such that they will fall on the unit circle. we did not treat features with small magnitudes separately, however, these would be projected out onto the unit circle, essentially amplifying their noisiness. Instead, we re-scale them such that they remain relatively close the origin ensuring little noise the evaluation, while relying minimally on the magnitudes.
This re-scaling of features with small magnitudes may lead to a trivial separation of objects in RGB images. When following a naive approach to apply the CAE to three-channel images, such as RGB, we would output three single-channel complex numbers and interpret their respective magnitudes to represent the RGB reconstructions and the phases to represent the object identity. The problem with this approach is that for objects of specific colors, the separation becomes trivial. For example, when we have red and a blue object, the respective reconstructed values might look something like this: For the red object, the magnitudes would take on the values [1, O, and the phases could take on arbitrary values [ "). For the blue object, the magnitudes would take the values [0, 0, and the phases again could take arbitrary values i _). Since we need to mask out the phases that belong to value with magnitude m < 0.1 to avoid evaluating increasingly random values, this allows for trivial separation of the blue and red object irrespective of the phases assigned to them. As a result, applying complex- valued methods to object discovery in RGB images is not straightforward. Note, however, that the CAE already makes step forward compared the DBM model by Reichert & Serre (2014) by being able process grayscale images with continuous pixel values instead of only binarized ones.
17

---

Published in Transactions on Machine Learning Research (11/2022)
C.2 DBM
In this section, we will first discuss the limitations of our re-implementation, then outline our final experimental setting, and finally list all other setups that we considered.
Limitations of our Re-lmplementttion To implement the Deep Boltzmann Machine (DBM) model, we closely followed the descriptions in Reichert & Serre (2014). However, due to the lack of a publicly available codebase, some aspects of the implementation remained unclear, and we used private communication with the authors, as well as exlensive experimentation to fill in the gaps. Despite the considerable effort that we put into reproducing their results involving more fine-tuning than we performed for the proposed CAF model our implementation seems to perform worse than the original implementation, visual comparison in Fig. 7 We note two things about this comparison: (1) Since Reichert & Serre (2014) report stability issues for their implementation and state that they only show results of the "well performing networks", we accordingly chose the seed with the best performance in our implementation to create the samples for the qualitative comparison. (2) We can not quantify the observed performance gap, as Reicherl & Serre (014) do not provide quantitative results in their paper.
indicated by the
Independent of the implementation, the DBM model is less efficient than the CAE by design, due to its greedy layerwise training with Contrastive Divergence that requires a comparatively large number of training steps, and the iterative settling procedure used for evaluation.
Final Setup We implement all layers as fully-connected layers with a potentially restricted receplive field as proposed by Reichert & Serre (2014). To achieve this, we mask out all weights outside the receptive field with zero. This results in a sctup that resembles convolutional layer without shared weights. Additionally, we apply a separate bias to each spatial location. We implement these layers in two different architectures. First, we closely followed the model description provided by Reichert & Serre (2014) and use a three-layer architecture (3-layer DBM). The layers in this architecture have a receptive field size in [height, width] of [7, 7]. [7.7] 7| and [20, 20] (i.c. global), and hidden dimensions in [height, width, channel] or [channel] of [26, 26,3], [20, 20, 5], and [676], respectively. As a second architecture, we test a DBM whose layers resemble the setup of the encoder fene of the (Complex) AutoEncoder (6-layer DBM, Table 3). Instead of convolutional layers, we use the fully-connected layers with restricted receptive field sizes as described above, where the kernel size corresponds to the receptive field size. Note that even though this setup resembles that of the CAE, its parameter count differs considerably as weights are not shared across locations, but they are shared for the forward and backward pass (i.e. encoding and decoding). For all layers in both models, we use sigmoid non-linearity as activation function.
We train each layer greedily as a separate Restricted Boltzmann Machine (RBM) using 1-step Contrastive Divergence (Hinonn 2012) with a learning rate of 1e-1, momentum cocfficient of 0.5 and weight-docay factor of 1e-4, for 50,000 and 100,000 steps. After training, to create object-centric representations, we inject, complex-valued activations into the model. To do so, we use the input images as magnitudes, randomly sample phase values for cach pixel from 2T)) and initialize the hidden state of each layer with an initial forward pass. Then, clamping the magnitudes of the visible units to the input images, we ilerate the complex-valued activations 500 times through the model. To evaluate the resulting phase values, we use the same discretization procedure as for the CAE' (see Section 3.5p. This procedure includes a step in which small magnitudes are used to mask out their corresponding phase values. Since we use the final output of the DBM to extract both the phases and magnitudes, and not some intermediate values as in the CAF, this model creates good foreground backgroud separation by relying explicitly on a black background resulting in small magnitudes for the reconstruction and thus masked out phases. Thus, the ARI+BG performance of the DBM models is negligible, as it depends directly on its reconstruction performance rather than its object discovery performance.
Finally, since the DBM is only well-defined on binary inputs, we apply this model to a binarized version of the &NIST&Shape dataset. To create this version of the dataset, we use the same threshold as for the crcation of the wixel-wise labels (-0.8 after normalization to the [-1,1] range, see Appendix C.4 to decide whether a pixel is assigned "0" or "I" value.
18

---

Published in Transactions on Machine Learning Research (11/2022)
A
Input
Reconstruction
Phases
Prediction
Masked Phases
Masked Phases
Our Implementation DBM
DBM
RRichert & Serre 2014
Figure 7: Qualitative comparison between random samples from the best performing seed of our re- implementation of the DBM model in Reichert & Serre (2014) (Ieft) and original samples taken from their paper on slightly different datascts (right). To create the "Masked Phases" images, we follow Reichert & Serre (2014) and use the input values (and thus the fact that objects are white, and the background is black) to mask out the phase values of the background pixels. Despite achieving a reasonable reconstruction performance on all three tested datasets, the object discovery performance of our re-implementation seems to lag behind the original implementation.
19

---

Published in Transactions on Machine Learning Research (11/2022)
Tested Setups Besides this final setup which achieved the best validation performance, we have tried wide range of other experimental settings:
We trained the 3-layer DBM model for up to 1,500,000 steps. Interestingly, even though the reconstruction performance keeps on improving, object discovery performance worsens (similarly to the effects observed in Table
We tested different learning rates [1, le, 1e-2, 1,-3, 1e-4] and different momentum terms [0.5, 1].
We implemented and tested 5-step persistent contrastive divergence (Tieleman, 2008 an alternative training algorithm for RBMs which was shown to achieve better results, with different learning rates, momentum terms and learning rate schedules (fixed and exponential decay).
We tried to share the weights and biases across spatial locations to create layers that resemble a convolutional layer more closcly.
We tested a different initialization of the hidden states during evaluation, in which we randomly sampled magnitudes from a Bernoulli distribution with 0.5 and phase values from ((0, 2n).
During evaluation, number that gave the best trade-off between performance and speed.
the model for different numbers of iterations (100, 500, 1000), and chose the
C.3
SlotAttention
To implement Stoentttnnion, followed the description and hyperparameters provided by Locatello et al. (2020) well their open-source implementation We used a hidden dimension of 64 throughout the model and adjusted the decoding architecture as described in Table 4 as this improved Altenttonoon''s 's performance on our datasets. Besides this final setup that we found to perform best in termss of object discovery performance, we tested the following setups on the validation set of the 3Shapes dataset: the decoder setup as used by Locatello al. (2020) for the Tetrominoes and Multi-dSprites datasets (i.e. spatially broadcast to resolution of 32 32 and apply four caanspssed-convluution fourth waannspssdd-oovolutttee
layers). a setup in which the layer in Table 4 is removed from the decoder, as well as setup in which the number of channels is halved across all layers. None of these sctups learned to disentangle objects the MNIST&Shape dalaset. Note that, since select the SlotAttention architecture based on its object discovery performance, its reconstruction performance is not comparable to the CAE due to a different number of lcarnable parameters and a different bottlencck sizc.
Table 4: Architocture used for the Spatial-Broadeast Broadcast Docoder in the SlotAttention model.
Layer
Feature Dimension Kernel
Stride
Padding
Activation Function
(H x W x C)
Input / Output
Spatial Broadcast
X
Position Embedding
4
TransConv
7 x
64
2 /
ReLU
TransConv
15
x
15 x 64
2
2 2
ReLU
TransConv
32
x 32 x 64
2
2
2/1
ReLU
TransConv
32 x
32 x 64
1 1/0
ReLU
TransConv
32
x
82
x 2
1 1/0
ReLU
Investigating the failure on the &NNST&Shappe dataset To investigate the different factors that lead to SlotAttention's failure on the MNIST&Shape dataset, we ran four additional experiments (Table 5). First, we trained SlotAltention on a version of the MNIST&Shape dataset which the MNIST digits are downsized to 16 16 pixels (small MNISTeShape). On this dataset, SlotAttention fails separate the two objects. Sccond, trained Atteniion version of the MNIST&Shape datasct in which the MNIST
20

---

Published in Transactions on Machine Learning Research (11/2022)
Table 5: Investigating the failure of SlotAttention on the ShisT& dataset. The results on the 2Shapes-randBG dataset and across variations of the MNIST&Shape datasct (mean + sem across 8 sceds) indicate that SlotAttention strugyles with large objects and grayscale images.
Dataset
MSE 4
ARI+BG
ARI-BG 1
small MhiST&Shape
4.495c-03
0.203
0.365
MNIST MNiSS&Shape binarized
3.071e-02
1:126e-03
0.081 4 0.018
0.153 4 0.038
small Shie binarized
1.025e-02
7.853c-04
0.380 + 0.108
0,851 + 0.022
2Shapes randBG
1.190e-05 2.235c-06
0,538 0.174
0,604 0.166
4 working seeds: 2Shapes randBG
1.526c-05
0.997 0.001
1.000 0.000
4 failing seeds: 2Shapes randBG
8.539e-06 0655-006
0.078 0.032
0.208 1 0.154
digits are of the original size, but binarized (MNIST&Shape binarized). Again, SlotAttention fails on this dataset. If we combine the two modifications and create a dataset with smaller, binarized MNIST digits, SlotAttention finally starts separate the objects (small MNIST&Shape binarized). However, it still does not perform perfectly in some cases, it splits the digits into two objects, e-g. by splitting eight into two separate circles. Last but not least, applied SlotAttention on the 2Shapes-randBG dataset. Here, SlotAttention performs perfectly for half of the seeds; and largely fails for the other half. Interestingly, the seeds that fail in terms of object discovery performance achieve a better reconstruction performance.
Based on these observations, we believe that SlotAttention fails on the MNIST&Shhpe datasct due to two lactors: (1) the MNIST digits are loo large to be covered by SlotAttention's receptive field. Only when the MNIST digits are downsized, can the SlotAttention model learn to separate the objects; (2) the SlotAttention model performs poorly on grayscale values, as highlighted by its results the non-binarized MNIST& datasets and the 2Shape dataset. We hypothesize that this is due to the alpha mask that is used in the spatial-broadcas. decoder: the StotAtttntion model might learn to use this mask to reconstruct the precise grayscale value rather than to correctly merge the reconstructions from the individual slots. This problem would be naturally circumvented by RGB data.
C.4 Datasets
For our experiments, we generate three grayscale datasets: 2Shapes, 3Shapes, and MNIST&&Shape. All images within these datasets feature a black background and white objects of differing shapes. In the 2Shapes and 3Shapes datasets, the foreground objects and the background are plain white and plain black, respectively, without noise. In the &ShsT& dataset, the digits exhibit differing grayscale values. All objects are placed in random locations while ensuring that no part of the object is cut-off at the image boundary.
We use four different object types (O, V. and MNIST digits). The square has outer side-length of 13 pixels. Both triangles are isosceles triangles, have base-length of 17 pixels, and are 9 pixels high. Both the square's and the triangles' outlines have a width of 3 pixels.
For the Shape& dataset, we resize each MNIST digit to match the input image size our dataset (i.e. 32 x 32) before applying it to an image. Then, we label pixels as "digit" when their value is > -0.8 after normalization to the [-1, range. This threshold ensures that most of the digit pixels are labeled as such, while minimizing the influence of potentially noisy background pixels. We follow the original datasct split to create the test images and divide the original training set to get 50,000 MNIST digits for our training set and 10,000 MNIST digits for our validation set.
We scale and shift all inputs to the range [0, for the autoencoding models, and we use an input range of [-1, 1] for the SlotAttention model.
21

---

Published in Transactions on Machine Learning Research (11/2022)
D Additional Results
D.1 Global Rotation Equivariance
In this section, we will first proof that the CAE model equivariant global rotations, before verifying this property empirically.
D.1.1 Proof of Global Rotation Equivariance
Before giving the proof for Proposition 3.1 we formally define global rotation equivariance.
Definition D.1. (Global Rotation Equivariance) A function f C C is equivariant w.r.t. global rotations if for any arbitrary phase offset E (T) it holds that:
f(x (cos(a) + sin(a)))) f(x) (cos(a) + sin(a))))
(7)
Proof. To proof that the CAE autocncoder oquivariant w.r.t. global rotations, we consider cach operation performed in its layers separately. If all of these operations are equivariant, their composite function (i.e. the overall model) is equivariant well.
Equation (2) describes the way that the weights applied to the input. We consider the simple setting of fully connected layer, which extends trivially to other layer types such as convolutions and transposed convolutions:
fw (z) (cos(a) + sa))))) = (w,z) (cos(a) + sin(a)))
(8)
Due to the complex multiplication being commutative, we get:
(w, z (cos(a) + (in((:))))
(9)
= fw (z (cos(a) (a)))
(10)
Equation
3 applics the bias terms to th:
my | + by
ddut
9 arg((b) + b E Rodt
(11)
The addition of the magnitude bias 6m is equivariant w.r.t. global rotations, since the rotation by any angle leaves the magnitude unchanged. The addition of the phase bias by is rotation equivariant, as it applies a fixed shift to the phases independent of their values.
Equation describes the application of the network to the magnitudes of the input features:
X = fw ([zl) + bm
E Rdout
(12)
my 0.5 m. 0.5
E ddout
(13)
Similarly to above, this computation is equivariant w.r.t. global rotations, since the rotation by any angle leaves the magnitudes unchanged.
Equation applies non-linearity on the activations:
z' = ReLUU[BatchNorm((m) "PP
E Cdoou
(14)
Since this non-linearity only affects the magnitudes, but leaves the phase values unchanged, this function is equivariant to global rotations (i.e. phase shifts) as well.
Thus, all operations within the CAE are equivariant w.r.t. global rotations, and as result, the CAE model is equivariant w.r.t. global rotations.
22

---

Published in Transactions on Machine Learning Research (11/2022)
le-5
1.0
mwwm Vno Mam Mappemannn
Equivariance Error
0.8
0.6
Magnitude
Phase
0.4
0.2
0.0
0
2
4
Global Input Phase Shift
Figure 8: Equivariance error of the CAE. We measure the average L,a-distance (i.e. equivariance error, y-axis) between the output magnitudes and phases created with default input phase value of zero and outputs created with input phase shifts (x-axis) between 0 and 2T. We observe that the equivariance error is minimal for both the magnitude and phase.
D.1.2 Empirical Validation of Global Rotation Equivariance
We verify empirically that the CAE is equivariant w.r.t. global rotations. For this, we take randomly initialized CAE model and input batches of random noise images with various phase values. Then, we measure the average magnitude and phase difference between the default input phase of zero and input phase interpolations between zero and 2n. More precisely, we measure the Lス-distance between the individual pixel magnitudes and phases, and aggregate the results into an average error per global phase shift. For the equivariance error of the phase, we further subtract the expected shift from the output phases. As can be seen from Fig. S the resulting equivariance error for both the magnitude and phase are minimal.
D.2 DBM
We lest. two architectures for the DBM model, 3-layer DBM and 6-layer DBM, trained for 50,000 and 100,000 steps per layer (Table G). On the 2Shapes dataset, the 3-layer DBM trained for 100,000 steps per layer (i.e. 300,000 steps in total) achieves the best reconstruction (MSE) and object discovery (ARI-BG) performance. On the 3Shapes and &NIST&:hhape datasets, the 6-layer DBM achieves the best object discovery results in terms of ARI-BG, despite creating less accurate reconstructions. fact, for the 6-layer DBM model trained on the 3Shapes dataset, the object discovery performance terms ARI-BG worsens when the model is trained for longer, despite an improvement reconstruction performance. In the main results in Table L we state the performance of the best model in terms of ARI-BG performance per dataset.
23

---

Published in Transactions on Machine Learning Research (11/2022)
Table 6: MSE and ARI scores (mean standard error across 8 seeds) of the two DBM models (3-layer DBM and 6-layer DBM) across the three tosted multi-object datascts. On the 2Shapos datasct, the 3-layer DBM achieves the best reconstruction and object discovery performance in terms of ARI-BG, whereas the G-layer DBM achieves better object discovery performance on the 3Shapes and &NIST&&Shape dataset. Intorestingly, training the 6-layer DBM model for longer on the 3Shapes dataset results in a worse object discovery performance terms of ARI-BG despite improvements in the reconstruction performance.
Dataset
Model
Steps
MSE
ART+BG
ARI-BG f
Complex AutoEncoder
10000
3.322e-04
0.999
1,000
3-layer DBM
150000
4.358e-03
0.893
0.694
2Shapes
3-layer DBM
300000
3.308e-03
0.920
0.744
G-layer DBM
300000
8,699e-03
1.412e-04
0,893 0,001
0.689 0.015
G-layer DBM
600000
4.892e-03
5.3044-03
0.934
0.696
Complex AutoEncoder
100000
1.313e-04
2:0201-055
0.976 0.002
1.000 0,000
3Shapes
3-layer DBM
150000
6.6444--33
1.6288-004
0,866 0,001
0.333 0.015
3-layer DBM
300000
6.111e-03
0.888 0.09
0.377
6-layer DBM
300000
1.045e-02
1.49do.01
0.856
0.419
6-layer DBM
600000
6.206e-03
0.913
0.397
Complex AutoEncoder
10000
3.185e-03
0.783 = 0.004
0.971 0.011
3-layer DBM
150000
1.381e-02
0.762 = 0.w0
0.047 0.005
MNIST&&Shape
3-layer DBM
300000
1.321e-02
5.2406-05
0.755
0,001
0.063 0.005
6-layer DBM
300000
1.839e-02
2.171e-04
0.697
0,000
0.153 0.0LD
6-layer DBM
600000
1.560e-02
0000-00
0.718
0.009
0.175 0.005
D.3 Model Sensitivity Analysis
Fig. highlights the influence of the feature dimension that is output by the CAE's encoder feue on the model's s performance. We find that the model achieves strong performance for broad range of feature dimensions indicating that the CAE does not require restricted bottleneck size to create disentangled object representations.
1.00
0.004
0.98
ARI-BG
0.003
0.96
MSE
0.002
0.94
0.001
0.92
23 24 25
26 27 2
29 10
2 2 2
26
2 29 20
Latent Dim
Latent Dim
Figure 9: Influence the latent dimension on CAE's performance on the 2Shapes dataset (mean + standard error across seeds). We find that the model does not require a restricted bottleneck to create disentangled object representations.
D.4 Dataset Sensitivity Analysis
To explore the capabilities and limitations of the Complex AutoEncoder under various conditions, apply it to set of new datascts. In Table 7 we provide detailed breakdown of all results and in Figs. 10 and 11 we show qualitalive results for each dataset.
Below, we provide additional details for each dataset:
For the 2Shapes-randBG dataset, we vary the 2Shapes dataset by applying a different background color to each image. This background color is randomly sampled from the [0, 0.75) range given that the pixel
24

---

Published in Transactions on Machine Learning Research (11/2022)
Table 7: Results on the exploratory datasets investigating capabilities and limitations of the CAE (mean + standard error across 8 sceds). We find that the CAE can handle wide variety of dataset settings, such variable backgrounds, repeated objects and decreasing objects counts. However, it struggles to represent a larger number of objects at a time and to generalize to more objects than observed during training.
Dalaset
MSE 4
ARI+BG f
ARI-BG 1
2Shapes-randBG
4.805e-04 6,380-005
0.971
0.995
2TTriagless
4.441e-06 4.046c-06
0.879 0.019
0.948 0.045
4Shapes
3.242c-03 i 1.321e-04
0.860 + 0.014
0.669 + 0.029
2/5Shapes
2.463e-04 è 4.707e-05
0.939 + 0.014
0.969 1 0.007
>2hhapes
3.979e-02 + 0.438c-04
0.844 + 0.008
0.758 0.007
<3Shapes Test
4.702c-03 :: 8.8866-00
0.923 + 0.013
0.956 : 0.019
values of the images range from 0 to 1 and that objects have color value of this ensures that background colors vary maximally while the objects stay clearly discernible from the background. After training on this dataset for 10,000 steps, the CAE creates accurate reconstructions and clearly separates the objects through its assigned phase values, highlighting that it can handle variable backgrounds. Interostingly, for brighter background colors, the model also learns to assign a separale phase to the background enabling the model to differentiate foreground from background even this more challenging setting.
For the 2Triangles dataset, we place two fowiward-facing triangles in cach image and train the model for 100,000 steps. Given the CAE's high performance on this dataset, conclude that it can separately represent multiple instances of the same object co-occuring in the same image.
The 4Shapes dataset builds on the 3Shapes dataset and adds an additional circle with an outer radius of 6 pixels to each image. Similarly to the 3Shapes dataset, we train the model for 100,000 steps. On this dataset, the performance of the CAE drops significantly indicating that it cannot represent this amount of objects at time.
For the 2/ 2/5Shapes dataset, we randomly sample two shapes for each image out of set of five possible shapes, the same four shapes as in the 4Shapes dataset plus an additional larger circle with an outer radius of 10 pixels. After 100,000 training steps, the CAE achieves good performance on this dataset indicating that it can learn to represent a larger set of objects as long as the number of objects per image is limited.
To test the generalization capabilities of the CAE, we create the >2Shapes and <3Shapes datasets. For the >2Shapes dataset, we train the CAE on the 2Shapes dataset and test on a dataset in which the downward-facing triangle that appears in the 2Shapes dataset appears twice in each image (and thus we have three objets per image). The relatively low performace of the CAE this dataset indicates that it cannot generalize well to more objects that observed during trainig.
For the <3Shapes dataset, we use the same set of objects as in the 3Shapes dataset, but randomly decide whether to place one, two or three of these objects into each image. We evaluate the CAE's generalization performance on this dataset by training it on the 3Shapes datasel for 100,000 steps, and subsequently testing it on the <3Shapes dataset. Based on the strong performance of the CAE, we conclude that can handle variable numbers of objects in images and that it can generalize scamlessly to sctting in which fewer objects are present Ihan observed during training.
25

---

Published in Transactions on Machine Learning Research (11/2022)
a S randBG
2
2
KShapes
4
4
2/5Shapes
-
Input
Reconstruction PrconstrutiionPhaaee Phase Values
Prediction
Imput
Reconstruction Peconstruction Phase Values
Prediction
Comples
Complex Autonncoodeooooo EntooEnodee
Complee
Complex AutoEncoder soooBBnoder--a. A
analysia Qualitative results for the dataset sensitivity analysis the Complex AutoEncoder. The depicted images are test-samples which were here hand-selecten tand-selected to best highlight the variability of samples within each dataset.
Figure 10:
26

---

Published in Transactions on Machine Learning Research (11/2022)
>o 2Shapes
A
A
SShhapes
Input
Reconstruction Phase Values
Prediction
Input
Reronstruction
Phase Values
Prediction
Complex AutoEncoder
Complex AutoEncoder
Figure 11: Qualitative results for the datasel sensitivity analysis of the Complex AutoEncoder. The depicted images are test-samples which were hand-selected to best highlight the variability of samples within each dataset. While the CAE does not generalize to a dataset depicting more objects (> (>2Shapes), it does generalize smaller object count than observed during training ($3Shapes).
D.5 Runtimes
To compare the training times between models, we first investigate the number of training steps that they require to achieve their best performance. For this, we plot the training curves of the Complex AutoFncoder and SlotAttention model the 2Shapes and 3Shapes datasets in Fig. 12 In both datasets, the SlotAttention model keeps improving in performance throughout its 500,000 training steps. The Complex AutoEncoder, the other hand, converges much faster, within 10,000 100,000 steps. For the DBM model, we cannot provide training curves due to its greedy layerwise training. Instead, we evaluate its performance after 50,000 and 100,000 training steps per layer Table s and find that longer training times generally result improved reconstruction performance. For the final run-time comparison, we choose the DBM model with the best. ARI-BG performance per dataset.
We provide comparison of the training times of the Complex AutoEncoder, SlotAttention and DBM model in Table 8 We find that the DBM models have the fastest training time per 10,000 steps, but are overall slower to train than the CAE due to the high number of training steps that they require. Aloentttnnonn's 's training steps are two times slower than those the CAE, which, together with its high number of training steps, leads significantly slower training times.
In Table we compare the evaluation times of each model on the 2Shapes dataset. Since all datasets in experiments use the same dimensionality (32 x 32 pixels) and contain 10,000 images, expect these numbers to generalize well across the tested datasets. Since SlotAttention creates explicit separation of object features into slots, its predicted object assignments are readily available and single evaluation of the lest. set. takes approximately 25.4 seconds. Both the CAE and DBM model, on the other hand, need to extract discrete object assignments from the continuous phase values using k-means, resulting in considerably longer evaluation times. While the CAE takes approximately 263.9 seconds to evaluate, due itcrative settling procodure, the DBM takes between 520.9 and 4377.6 seconds depending on whether we use the 3-yayer 6-layer variant.
27

---

Published in Transactions on Machine Learning Research (11/2022)
1.00
1.00
N
0.99
0.99
0.98
0.98
1.0
1.0
0.8
0.8
ARI-BG
ARI-BG
0.6
0.6
0.4
Model
0.4
Model
CAE
CAE
0.2
0.2
SlotAttention
SlotAttention
1000
5000
10000
50000
100000
500000
1000
5000
10000
50000
400000
500000
Step
Step
Figure 12: Training curves of the Complex AutoEncoder (CAE) and SlotAttention model (mean + standard error across sceds) on the 2Shapes (Left) and 3Shapes (Right) datasct. We plot the ARI-BG scores on the validation set throughout training. The CAE achieves comparable or belter performance within 5-50 times fewer training steps compared the SlotAttention model.
Table 8: Training times (mean + standard error across seeds) in terms of wall clock time for the Complex AutoEncoder, SlotAttention and DBM model across the three tested multi-object datasets. We measured the training limes per 10,000 sleps the 2Shapes datasel and extrapolated the results to create a detailed and realistic break-down of training times for each dataset. This is possible, as all datasets are of the same dimensionality. For the DBM, we report the average training time per step across layers. We find that the CAE is considerably faster to train on all datasets compared to the Altentttnooon and DBM model.
Dataset
Model
Steps
Training Time per
Total Training Time (min)
Relative Training Sperd compared to CAE
10,000 Steps (sec)
Complex AutoEncoder
10000
456.6
7.6
1.0
2Shapes
3-layer DBM
300000
31.5
15.7
2.1 2.0
SlotAtteution
500000
911.1
759.3 3.7
99.8
Complex AutoEneoder
100000
456.6
76.1 6.4
6.0
3Shapes
6-layer DBM
300000
160.0 a
80.0 0.
1.1 a.0
SlotAAtcction
500000
911.1 455
759 3.7.7
10.0 a.0.
Complex AutoEneoder
10000
456.6
7.6
1.0
MNIST&SSaape
6-layer DBM
600000
160.0
160.0
21.0
SlotAAtieution
500000
911.1
759.3
99.8
D.6 Additional Qualitative Results
We highlight the phase separations created by the Complex AutoEncoder in Fig. 13 object-wise reconstructions
in Fig. 14 and compare all models on the 2Shapes, 3Shapes and ShiST&Shape datasets in Figures 15
and 17 respectively.
28

---

Published in Transactions on Machine Learning Research (11/2022)
Table Evaluation times (mean + standard error across seeds) in terms of wall clock time for the Complex AutoFncoderr Sttennton and DBM model the 2Shapes dataset. Since all datasets are of the dimensionality, expect the evaluation times on the 3Shapes and MNIST&:Shape datasets to be equivalent. The SlotAttention model is the fastest to evaluate, as it immediately creates discrete object assignments. The CAE and DBM, on the other hand, require a discretization procedure with k-means leading to slower evaluation times. The DBM is further slowed down by its iterative settling procedure.
Model
Total Testing Time
Relative Testing Speod compared to CAE
(sec)
Complex AutoEncoder
263.9
1.0
3-layer DBM
520.9
G-layer DBM
4377.6
16.6
Attenttoo
25.4
6.0
29

---

Published in Transactions on Machine Learning Research (11/2022)
180
180
180
g"
180
180
270
270"
270
270
270
90"
180
0º
100
0
180
0"
180
0r
180
270*
90"
90º
90"
90*
180
180
180
0"
180
180
Figure 13: Phase separation in the Complex AutoEncoder on random lest-samples from the 2Shapes (Top), 3Shapes (Middle) and MNIST&Shhape datasets (Bottom). For each sample, we show the output phase images on top, and the corresponding output values in the complex planc in the bottom.
30

---

Published in Transactions on Machine Learning Research (11/2022)
9
7
4
7
Figure 14: Investigating object-centricity of the latent features in the CAE Top: 2Shapes, bottom: &NIST&Shape dataset. Columns & 4: input images. Columns 2-3 5-6: object-wise reconstructions. By clustering features created by the encoder according their phase values, can extract representations of the individual objects and reconstruct them separately.
31

---

Published in Transactions on Machine Learning Research (11/2022)
Apput
Reconstruction R
Reconstruction
Phane
Phase & Values
Prediction
Prediction
Prediction
Comples
SlotAtterno ?
AautoEncode :
-Compllpee ComplexXAite AutoEncoder
Cotoonccder--ar -
DBM
DBM
Figure 15: Visual comparison of the performance of the Complex AutoFncoder, its real-valued counterpart. (AutoEncoder), the DBM model and andta the Altenntooooo model on random test-samples from the 2Shapes dataset.
32

---

Canaarcttooo in Fubblstoo
Researto Seamringg 22/0222
Machhaa on
Recousttiinn Autootioon AatooEcooee
Reconstruction - A
Phase Values
Prediction
Prediction
SlottAttene 
Predeetii
Input
Reconstruction
Prediction DBN
Complex Complex_Ani AutoEncoder .
Aeenccodde Visual comparison of the performance of the Complex AutoFncoder, its real-vllun comnterr (Auto antooenco,)), the DBM model and the Alo model on random fest-ssammng from from the 5Shaapes datases
Figure
TigureTTe 16: Visual
dataset
39

---

Published in Transactions on Machine Learning Research (11/2022)
9
6
Input
Reconstruction
Reconstruction
Phase Values
Prediction
Prediction
Prediction
AutoEncoder
Complex AutoEncoder
DBM
SlotAttention
Figure 17: Visual comparison of the performance of the Complex AutoFncoder, its real-valued counterpart (AudoEncoder), the DBM model and the SlotAttention model on random test-samples from the ANIIT&Shapp dataset.
34


Reviews:
[{"review_id": "6n6JpnHJIi", "reviewer": "Reviewer_NrE9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n    - The paper proposes the first end-to-end trainable complex-valued neural network for unsupervised object discovery. The idea is quite interesting as it is inspired by how biological neural networks solve the binding problem.\n    - The qualitative results convincingly show that the model can indeed output similar phase values for the same object.\n    - The MNIST&Shape dataset seems particularly challenging for slot-based methods, as it contains objects of different sizes. It is quite impressive that the propose model achieves almost perfect foreground segmentation while Slot Attention completely fails.\n    - There is ablation study that validates the design choices.\n    - The paper is generally well written. The idea of using complex values and the network design are well motivated.\n    - The paper makes modest claims, and clearly states its limitations.\n    - Related work seems sufficiently discussed.\n    - Implementation and dataset details are provided in Appendix.\n    - The experiments are run with multiple seeds.\n- Weaknesses\n    - I didn't find any major weakness. However, some clarifications (listed in Requested Changes) could be helpful to better understand the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the Complex AutoEncoder (CAE) for unsupervised learning of distributed object-centric representations. In contrast to state-of-the-art methods that explicitly represent each object as a separate slot in the model architecture, the proposed method explores using synchronization among distributed neural activations for object discovery. This is motivated by the temporal correlation hypothesis in neuroscience, that posits that the firing rate of a biological neuron encodes the presence of some feature, and the synchronous firing patterns among multiple neurons indicate the binding of features. To that end, the proposed CAE uses complex-valued activations, where the phase of the complex value can indicate synchronization. The paper makes some design choices to encourage the emergence of synchronization. The resulting model has no iterative procedure and is simpler compared to slot-based methods. After training, a clustering procedure is applied to the phase values to read out the synchronized groups, which are shown to correspond to objects. Experiments on a few simple datasets show that the proposed method outperforms a previous work that uses complex values at test time by a large margin in terms of reconstruction quality and object discovery. When compared to Slot Attention, a state-of-the-art slot-based method, the proposed CAE succeeds in decomposing images containing a small shape and a large MNIST digit where Slot Attention completely fails. CAE also does not over-segment the background, and is much faster to train than both baselines. The paper also theoretically proves and visually demonstrates that CAE is equivariant to phase shifts in the input.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Below are some minor comments that I hope the authors could address.\n- It looks like the dot ($\\cdot$) stands for element-wise multiplication. Can you state that explicitly? Also in Eq 8-9, does $w^\\top \\cdot z$ mean inner product, element-wise product, or matrix multiplication?\n- I didn't find the explanation or ablation for adding a bias term on the magnitude ($b_m$ in Eq 3). I understand that this may be just a natural choice to make the network more flexible. Can you clarify?\n- Is the $f_w$ in Eq 2 and Eq 4 shared (similar with $b_m$)? If so, can you elaborate on why they are shared?\n- The term \"maximally distant phase\" is used a few times. It would be better to include a precise definition for that.\n- I didn't fully understand how to \"fine-tune the decoder to reconstruct separate objects from the individual clusters\". How do you know which object to reconstruct?\n- It is mentioned that the background cluster tends to have low magnitude. Is this general or just because the background is black? What happens when you use random sampled background (2Shapes-randBG dataset)?", "broader_impact_concerns": "sufficiently addressed", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2qXAjlrON7", "reviewer": "Reviewer_V5RU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The paper does a good job of presenting the approach and motivation, and it explains its choices very well.\n   1. However I would say that by Section 2 I had heard the motivation enough times and I would recommend trimming it down. It is quite clear where the idea comes from, so the paper would benefit from getting to the point faster. There is quite a lot of redundancy from the abstract to the start of Section 2.\n   2. Sections 3.1 - 3.3 cover the model choices well, although once again I probably would have compressed them a bit as all of them appear really sensible and “simple” (not a bad thing!)\n   3. Section 3.4 is nice to have (albeit slightly trivial), but you do not emphasize why/when having such an equivariance would help / hurt?\n      1. However, Figure 6 and the associated section in results isn’t particularly useful, and I would cut it entirely or move it to the Appendix. Again, I am not sure this is a particularly “useful” example to demonstrate.\n2. The fact that the model is constrained to grayscale isn’t made clearly and early enough.\n   1. This is the main limitation of the method, and having this discussed only briefly in the last section and Appendix C.1 is not sufficient. It only comes in Section 4.1, as up till then I assumed you would cover extra color datasets.\n   2. It also feels to me like your answer to why RGB is difficult isn’t that convincing.\n      1. Shouldn’t the model be able to output RGB yet be forced to use phase as implicit segmentation still?\n      2. Your example is very particular, instead how would you want to represent a yellow object?\n      3. Why can’t you use a single phase channel and 3 real activation channels? I understand that this breaks the mental model of “complex numbers everywhere”, but your model already handles the real and complex part nearly independently, so that connection is strenuous at best? I would have expected a more innovative discussion of how to make this work.\n   3. Given this constrained your choice of datasets, you should discuss it earlier, and perhaps indicate that the datasets considered might actually be more challenging to extract objects out of given their lack of color separation.\n3. The model’s behavior on the dataset assessed is interesting and convincing enough of the usefulness of the method and would allow people to build upon them. \n   1. Figure 4 and 5 are the ones I found most informative, especially with the phase diagram and the object-centric reconstructions. \n      1. Could we have more of these?\n      2. However, where is the background encoded in the phase diagram?\n   2. In Figure 3, I did not understand the yellow area or why you’d remove them before evaluation? What is the “correct” output?\n4. However, the model’s ability to be a “good” object-centric method is not thoroughly investigated in this current draft. \n   1. For example, one would need to show that training on a varying number of objects allows the model to handle both more and fewer objects. You try to do that with <3Shapes, but this isn’t hard enough, e.g. what if you train on 1, 3, 5, can you generalise to 2 and 6?\n   2. Can you leverage the “phase separation” at the representation level? If this is a representation learning method, what matters is how useful the latent code is, not its reconstruction ability / quirknesses.\n      1. Can you decode the features of individual objects well?\n      2. Are features specializing into subsets with phase as well? Or is that only “clean” at the output level?\n   3. How does this change when varying the number of objects in the input? Is it stable or does it decay catastrophically past a certain “input load”?\n      1. Looking at the phase plots in Figure 4, it feels like there would be a natural “maximum number of objects” that one can attend to. \n      2. As an aside, this isn’t intrinsically bad, as this is sometimes used in human visual short term memory to explain the limited memory capacity (e.g. “Magical number 4 in short-term memory” by Cowan, although don’t take this number literally). In the brain the phasic load is probably within one gamma cycle, but the principle stays the same.\n      3. If there was a way to have control over which ones are attended to, or randomizing them, this could be worked around.\n   4. Having to do a post-hoc clustering isn’t particularly elegant, especially since areas that overlap appear like another object (my reading of the phases in Figure 4).\n      1. How many “objects clusters” did you choose in Figure 4 right-most? The phase diagram might indicate you would need 4, how would you select this automatically?\n      2. Could you define a task which would leverage the fact that the “segmentation” information is present in a continuous fashion? Having a continuous indicator is actually quite nice if that allows later stages of processing (maybe using “normal” neural networks) to do a better job than if that wasn’t encoded/available at all?\n   5. I also did not really understand why you had to do the scaling operation for small magnitudes. \n      1. Wouldn’t this have allowed you to detect “unused” dimensions and prune them away?\n      2. Where would they have fallen in the phase diagram of Figure 4 before / after this modification?\n5. The performance of SlotAttention on your datasets is surprisingly bad, and I would not have expected such a gap.\n   1. This would need to be discussed more deeply. \n      1. Is it due to the grayscale nature of the data? \n      2. Is it that the objects are slightly too large? (but in that case you could have made them smaller, if SlotAttention has an intrinsic prior over object size due to the number of effective slots after its convolutional stack)\n   2. In effect, the SlotAttention predictions in Figure 3 just make me feel like something went wrong, because there is no reason to expect this kind of failure mode (given such harsh binary inputs, another failure would have been to mix the 2 shapes into one slot).\n   3. What is its behavior on the 2Shapes-randBG dataset?\n6. I understand that the DBM baseline was the closest to this current work, but I am not sure of how much value it brings. Nobody uses them anymore for many reasons, which you also point out, so improving on that is not a surprising fact?\n   1. It is really apparent in the Appendix that a lot of work has been put in reproducing them, so this is quite unfortunate, but I am not sure this adds much value to the paper.\n7. Nits / typos\n   1. In section 3.1, when mentioning f_out, add a mention that you found this helpful and already ablate it in Table 2. It feels slightly overkill when one gets to this point of the manuscript and I was about to ask for an ablation.\n   2. In section 3.2, last paragraph, you use z_2 and w_2 but these should be bold symbols as they are vectors.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel avenue to learn object-centric representations, by leveraging complex valued neural networks. More precisely, they set up an Autoencoder with complex activations throughout, and use the phase at the output of the decoder as an indicator for feature binding (i.e. an implicit segmentation per object as a function of phase).\n\nThey present results on toy grayscale datasets (a clear limitation of the approach, as I will cover below), comparing against DBM (the only previously similar instantiation of such an idea) and SlotAttention (current baseline in slotted object representation learning, near SOTA enough).\n\nOverall, the results are interesting and convincing of the approach’s potential, but in its current state this is a very preliminary line of research. This is both an issue and an opportunity, as there are clearly defined improvements that need to be made to this line of research, which this paper can help publicize and create a community for. \n\nGiven the “TMLR two questions” for submissions, I would answer yes to both and lean towards acceptance, but some of the work I would like to see might be too large for a minor revision to accept the paper at this time.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Comment on the grayscale limitation and on the difficulty of scaling to RGB earlier in the paper. I would also expect a deeper discussion of why an “hybrid” solution like I presented above would not be satisfactory.\n2. Comment / demonstrate the model’s ability to learn useful object-centric “representations”, i.e. have something that uses the latent codes for an object-driven task. This might be necessary to bring this paper to an effective form, which might be too much work for a minor revision.\n3. Comment on the failure of SlotAttention, or modify the presentation to more clearly indicate what it does well or not. \n   1. I would have expected it to work, so it is important to explain why this failed. Right now I am not reassured that the model was applied “well-enough”.\n   2. For example, you report ARI+BG first, even though the field uses ARI no BG as a metric (this is usually because backgrounds do not have a single “correct” segmentation, so failing to find the one that exists in the segmentation labels isn’t that fair)\n4. I would recommend moving Figure 6 to Appendix to make space for some extra datasets to go into the main text (e.g. take Figure 12 in the Appendix, we need more phase diagrams!)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b0mG9B0f58", "reviewer": "Reviewer_tMFe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-\tThe fact that the correlations between the phases can directly discover different objects is quite inspiring and interesting. \n-\tThe results shown on the datasets tested by the authors also support their claim that the reconstruction of the input images are successful and the discovery of the objects also works well.\n-\tThe paper is well written and clear in its logics.\n\nWeaknesses:\n-\tThe biggest issue, and in fact a critical one, is that the datasets evaluated in this dataset are too simple and easy. This is especially concerning as the SlotAttention method, which is (or was) the state-of-the-art method in this object discovery problem, was proposed and tested on datasets that are far more complicated, realistic, interesting, and useful, like CLEVER6 and Multi-dSprites. Not mentioning that SlotAttention was published already 2 years ago and there should be even better methods (like MONET) than it. Although the authors mentioned this as the limitation of this work in the end of this paper, it still makes the value of this work significantly less. So in my opinion, it would be critical to show that this method can at least get reasonable results on the datasets SlotAttention was tested on in its paper.\n-\tThe use of datasets that are simpler than what were used in SlotAttention’s paper creates another issue for a fair comparison to that method, as it was never tested on these datasets and hyperparameters may not be fine-tuned for them.\n-\tThe right part of Figure 2 makes me wonder whether there will be an upper limit in the number of objects that can be discovered in the image. In fact, from this visualization, I would think that having more objects in the images will make the performance on the object discovery worse, is this true? How many objects can this method handle without the loss of performance?\n-\tThis is a smaller issue, can this method discover and separately identify multiple instances of the same object shape? The current used datasets do not seem to have this situation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes using complex-valued auto-encoders to automatically discover objects in the images through finding the aligned phases in the output vectors. The authors claim that this method achieves comparable performance to the earlier method SlotAttention, but uses much less computation time during training. After showing results on multiple manually constructed datasets, which contain several shapes or one shape plus one digit, the paper shows that this method reaches cleaner object separation and low reconstruction loss.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See the points in the weaknesses. Addressing the first point would be required. Other points can strengthen the work.", "broader_impact_concerns": "The authors have already addressed this in the paper and there is no clear negative societal impacts from this method.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers provided a number of constructive comments on the paper and the authors proposed some changes to address these points. The reviewers were satisfied with the proposed changes, and a unanimous 'accept' decision was arrived at. Thus, once the authors have incorporated that changes they promised to the reviewers the paper is ready for publication.
Decision: Accept (poster)