SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. We evaluate the models on three datasets from the Multi-Object datasets suite [41] namely Tetrominoes, dSprites and CLEVR (Figure 2) used by prior work in object-centric learning [14, 15, 45]. For CLEVR, we use the filtered version [45] which consists of images containing less than seven objects. For the main evaluation, we use the same image resolution as Emami et al. [45], i.e.,  $32 \times 32$  for Tetrominoes,  $64 \times 64$  for dSprites and  $96 \times 96$  for CLEVR (a center crop of  $192 \times 192$  that is then resized to  $96 \times 96$ ). For computational reasons, we perform all ablations and analysis on  $32 \times 32$  resolution. Performance of all models are ordered in the same way on  $32 \times 32$  resolution as the original resolution (see Table 1), but with significant training and evaluation speed up. In Tetrominoes and dSprites the number of training images is  $60 \mathrm{~K}$  whereas in CLEVR it is  $50 \mathrm{~K}$ . All three datasets have 320 test images on which we report all the evaluation metrics. For more details about the datasets and preprocessing, please refer to Appendix A.
Models & Training Details. We compare our CtCAE model to the state-of-the-art synchrony-based method for unsupervised object discovery (CAE [36]) as well as to our own improved version thereof  $(\mathrm{CAE} + +)$  introduced in Section 3. For more details about the encoder and decoder architecture of all models see Appendix A. We use the same architecture as CAE [36], except with increased number of convolution channels (same across all models). We train models for 50K steps on Tetrominoes, and 100K steps on dSprites and CLEVR with Adam optimizer [46] with a constant learning rate of 4e-4, i.e., no warmup schedules or annealing (all hyperparameter details are given in Appendix A).
Evaluation Metrics. We use the same evaluation protocol as prior work [14, 15, 45, 36] which compares the grouping performance of models using the Adjusted Rand Index (ARI) [47, 48]. We report two variants of the ARI score, i.e., ARI-FG and ARI-FULL consistent with Löwe et al. [36]. ARI-FG measures the ARI score only for the foreground and ARI-FULL takes into account all pixels.
Unsupervised Object Discovery. Table 1 shows the performance of our CAE++, CtCAE, and the baseline CAE [36] on Tetrominoes, dSprites and CLEVR. We first observe that the CAE baseline almost completely fails on all datasets as shown by its very low ARI-FG and ARI-FULL scores. The MSE values in Table 1 indicate that CAE even struggles to reconstruct these color
Figure 3: CAE, CAE++ and CtCAE comparison on Tetrominoes (columns 1-3), dSprites (columns 4-6) and CLEVR (columns 7-9). First row: ground truth masks and input images.
Table 2: Quantifying the Separability through inter- and intra-cluster metrics of the phase space. For the inter-cluster metric, we report both the minimum and mean across clusters.
Dataset	Model	Inter-cluster (min) ↑	Inter-cluster (mean) ↑	Intra-cluster ↓
Tetrominoes	CAE++	0.14 ± 0.00	0.30 ± 0.02	0.022 ± 0.010
CtCAE	0.15 ± 0.01	0.31 ± 0.03	0.020 ± 0.010
dSprites	CAE++	0.13 ± 0.05	0.51 ± 0.05	0.034 ± 0.007
CtCAE	0.13 ± 0.03	0.39 ± 0.10	0.027 ± 0.009
CLEVR	CAE++	0.10 ± 0.06	0.53 ± 0.15	0.033 ± 0.013
CtCAE	0.12 ± 0.05	0.50 ± 0.12	0.024 ± 0.005
images. In contrast, CAE++ achieves significantly higher ARI scores, consistently across all three datasets; this demonstrates the impact of the architectural modifications we propose. However, on the most challenging CLEVR dataset, CAE++ still achieves relatively low ARI scores. Its contrastive learning-augmented counterpart, CtCAE consistently outperforms CAE++ both in terms of ARI-FG and ARI-FULL metrics. Notably, CtCAE achieves more than double the ARI scores of CAE++ on the most challenging CLEVR dataset which highlights the benefits of our contrastive method. All these results demonstrate that CtCAE is capable of object discovery (still far from perfect) on all datasets which include color images and more than three objects per scene, unlike the existing state-of-the-art synchrony-based model, CAE.
Quantitative Evaluation of Separability. To gain further insights into why our contrastive method is beneficial, we quantitatively analyse the phase maps using two distance metrics: inter-cluster and intra-cluster distances. In fact, in all CAE-family of models, final object grouping is obtained through K-means clustering based on the phases at the output of the decoder; each pixel is assigned to a cluster with the corresponding centroid. Inter-cluster distance measures the Euclidean distance between centroids of each pair of clusters averaged over all such pairs. Larger inter-cluster distance allows for easier discriminability during clustering to obtain object assignments from phase maps. On the other hand, intra-cluster distance quantifies the "concentration" of points within a cluster, and is computed as the average Euclidean distance between each point in the cluster and the cluster centroid. Smaller intra-cluster distance results in an easier clustering task as the clusters are then more condensed. We compute these distance metrics on a per-image basis before averaging over all samples in the dataset. The results in Table 2 show that, the mean intra-cluster distance (last column) is smaller for CtCAE than  $\mathrm{CAE}++$  on two (dSprites and CLEVR) of the three datasets. Also, even though the average inter-cluster distance (fourth column) is sometimes higher for  $\mathrm{CAE}++$ , the minimum inter-cluster distance (third column)—which is a more relevant metric for separability—is
Figure 4: CtCAE on CLEVR is able to infer more than four objects, although it sometimes makes mistakes, such as specular effects or grouping together based on color (two yellow objects).
Table 3: Object storage capacity: training on the full dSprites dataset and evaluating separately on subsets containing images with 2, 3, 4 or 5 objects. Analogously, training on the full CLEVR dataset and evaluating separately on subsets containing images with 3, 4, 5 or 6 objects.
ARI-FG	ARI-FULL	ARI-FG	ARI-FULL	ARI-FG	ARI-FULL	ARI-FG	ARI-FULL
dSprites	2 objects	3 objects	4 objects	5 objects
CAE++	0.29 ± 0.07	0.58 ± 0.15	0.39 ± 0.06	0.53 ± 0.15	0.43 ± 0.04	0.45 ± 0.13	0.45 ± 0.05	0.44 ± 0.11
CtCAE	0.45 ± 0.11	0.76 ± 0.07	0.48 ± 0.06	0.72 ± 0.11	0.48 ± 0.04	0.65 ± 0.13	0.48 ± 0.04	0.61 ± 0.13
CLEVR	3 objects	4 objects	5 objects	6 objects
CAE++	0.32 ± 0.06	0.35 ± 0.29	0.33 ± 0.05	0.32 ± 0.26	0.31 ± 0.04	0.26 ± 0.20	0.32 ± 0.04	0.27 ± 0.18
CtCAE	0.53 ± 0.07	0.71 ± 0.20	0.52 ± 0.06	0.70 ± 0.21	0.47 ± 0.07	0.67 ± 0.21	0.45 ± 0.06	0.68 ± 0.20
larger for CtCAE. This confirms that compared to  $\mathrm{CAE}++$ , CtCAE tends to have better phase map properties for object grouping, as is originally motivated by our contrastive method.
Object Storage Capacity. Löwe et al. [36] note that the performance of CAE sharply decreases for images with more than 3 objects. We report the performance of CtCAE and CAE++ on subsets of the test set split by the number of objects, to measure how their grouping performance changes w.r.t. the number of objects. In dSprites the images contain 2, 3, 4 or 5 objects, in CLEVR 3, 4, 5 or 6 objects. Note that the models are trained on the entire training split of the respective datasets. Table 3 shows that both methods perform well on images containing more than 3 objects (their performance does not drop much on images with 4 or more objects). We also observe that the CtCAE consistently maintains a significant lead over CAE in terms of ARI scores across different numbers of objects. In another set of experiments (see Appendix B), we also show that CtCAE generalizes well to more objects (e.g. 5 or 6) when trained only on a subset of images containing less than this number of objects.
Ablation on Architectural Modifications.
Table 4 shows an ablation study on the proposed architectural modifications on Tetrominoes (for similar findings on other datasets, see Appendix B.2). We observe that the sigmoid activation on the output layer of the decoder significantly impedes learning on color datasets. A significant performance jump is also observed when replacing transposed convolution layers [36] with
convolution and upsample layers. By applying all these modifications, we obtain our CAE++ model that results in significantly better ARI scores on all datasets, therefore supporting our design choices.
Table 4: Architectural ablations on Tetrominoes.
Model	ARI-FG ↑	ARI-FULL ↑
CAE	0.00 ± 0.00	0.12 ± 0.02
CAE-(fout 1x1 conv)	0.00 ± 0.00	0.00 ± 0.00
CAE-(fout sigmoid)	0.12 ± 0.12	0.35 ± 0.36
CAE-transp.+upsamp.	0.10 ± 0.21	0.10 ± 0.22
CAE++	0.78 ± 0.07	0.84 ± 0.01
CtCAE	0.84 ± 0.09	0.85 ± 0.01
Ablation on Feature Layers to Contrast. In CtCAE we contrast feature vectors both at the output of the encoder and the output of the decoder. Table 5 justifies this choice; this default setting (Enc+Dec) outperforms both other options where we apply the constrastive loss either only in the
encoder or the decoder output. We hypothesize that this is because these two contrastive strategies are complementary: one uses low-level cues (dec) and the other high-level abstract features (enc).
Qualitative Evaluation of Grouping. Finally, we conduct some qualitative analyses of both successful and failed grouping modes shown by  $\mathrm{CAE}++$  and CtCAE models through visual inspection of representative samples. In Figure 3, Tetrominoes (columns 1-2), we observe that CtCAE (row 4) exhibits better grouping on scenes with multiple objects of the same color than  $\mathrm{CAE}++$  (row 3). This is reflected in the radial phase
Table 5:Contrastive loss ablation for CtCAE on CLEVR.
Model	ARI-FG ↑	ARI-FULL ↑
Enc-only	0.21 ± 0.11	0.29 ± 0.15
Dec-only	0.38 ± 0.17	0.69 ± 0.18
Enc+Dec	0.50 ± 0.05	0.69 ± 0.25
plots (column 2) which show better separability for CtCAE than  $\mathrm{CAE} + +$ . Further, on dSprites (rows 3-4, columns 4-5) and CLEVR (rows 3-4, columns 7-8), CtCAE handles the increased number of objects more gracefully while  $\mathrm{CAE} + +$  struggles and groups several of them together. For the failure cases, Figure 4 shows an example where CtCAE still has some difficulties in segregating objects of the same color (row 2, yellow cube and ball) (also observed sometimes on dSprites, see Figure 14). Further, we observe how the specular highlights on metallic objects (purple ball in row 1 and yellow ball in row 2) form a separate sub-part from the object (additional grouping examples in Appendix C).
Discussion on the Evaluation Protocol. The reviewers raised some concerns about the validity of our evaluation protocol<sup>3</sup>. The main point of contention was that the thresholding applied before clustering the phase values may lead to a trivial separation of objects in RGB images based on their color. While it is true that in our case certain color information may contribute to the separation process, a trivial separation of objects based on solely their color cannot happen since CtCAE largely outperforms  $\mathrm{CAE}++$  (see Table 1) despite both having near perfect reconstruction losses and with the same evaluation protocol. This indicates that this potential issue only has a marginal effect in practice in our case and separation in phase space learned by the model is still crucial. In fact, "pure" RGB color tones (which allow such trivial separation) rarely occur in the datasets (dSprites and CLEVR) used here. The percentage of pixels with a magnitude less than 0.1 are  $0.44\%$ ,  $0.88\%$  and  $9.25\%$  in CLEVR, dSprites and Tetrominoes respectively. While  $9.25\%$  in Tetrominoes is not marginal, we observe that this does not pose an issue in practice, as many examples in Tetrominoes where two or three blocks with the same color are separated by  $\mathrm{CAE}++/\mathrm{CtCAE}$  (e.g. three yellow blocks in Figure 3 or two magenta blocks in Figure 8).
To support our argument that using a threshold of 0.1 has no effect on the findings and conclusions, we conduct additional evaluations with a threshold of 0 and no threshold at all. From Table 9 (see Appendix B) we can see that using a threshold of 0 hardly changes the ARI scores at all or in some cases it even improves the scores slightly. Using no threshold results in marginally improved ARI scores on CLEVR, near-identical scores on dSprites but worse scores on Tetrominoes (CtCAE still widely outperforms  $\mathrm{CAE}++$ ). However, this is not due to a drawback in the evaluation or our model. Instead, it stems from an inherent difficulty with estimating the phase of a zero-magnitude complex number. The evaluation clusters pixels based on their phase values, and if a complex number has a magnitude of exactly zero (or very small, e.g., order of  $1e^{-5}$ ), the phase estimation is ill-defined and will inherently be random as was also previously noted by Löwe et al. [36]. In fact, filtering those pixels in the evaluation is crucial in their work [36] as they largely used datasets with pure black background (exact zero pixel value). Evaluation of discrete object assignments from continuous phase outputs in a model-agnostic way remains an open research question.
Slot-based binding. A wide range of unsupervised models have been introduced to perform perceptual grouping summarized well by Greff et al. [30]. They categorize models based on the segregation (routing) mechanism used to break the symmetry in representations and infer latent representations (i.e. slots). Models that use "instance slots" cast the routing problem as inference in a mixture model whose solution is given by amortized variational inference [8, 14], Expectation-Maximization [10] or other approximations (Soft K-means) thereof [15]. While others [9, 12, 49] that use "sequential slots" solve the routing problem by imposing an ordering across time. These
Discussion thread: https://openreview.net/forum?id=nF6X3u0FaA&noteId=5BeEERfCvI
models use recurrent neural networks and an attention mechanism to route information about a different object into the same slot at every timestep. Some models [13, 17] combine the above strategies and use recurrent attention only for routing but not for inferring slots. Other models break the representational symmetry based on spatial coordinates [50, 16] ("spatial slots") or based on specific object types [51] ("category slots"). All these models still maintain the "separation" of representations only at one latent layer (slot-level) but continue to "entangle" them at other layers.
Synchrony-based binding. Synchrony-based models use complex-valued activations to implement binding by relying on their constructive or destructive phase interference phenomena. This class of models have been sparsely explored with only few prior works that implement this conceptual design for object binding [38-40, 36]. These methods differ based on whether they employ both complex-valued weights and activations [38, 39] or complex-valued activations with real-valued weights and a gating mechanism [40, 36]. They also differ in their reliance on explicit supervision for grouping [38] or not [39, 40, 36]. Synchrony-based models in contrast to slot-based ones maintain the "separation" of representations throughout the network in the phase components of their complex-valued activations. However, none of these prior methods can group objects in color images with up to 6 objects or visual realism of multi-object benchmarks in a fully unsupervised manner unlike ours. Concurrent work [52] extends CAE by introducing new feature dimensions ("rotating features"; RF) to the complex-valued activations. However, RF cannot be directly compared to CtCAE as it relies on either depth masks to get instance grouping on simple colored Shapes or features from powerful vision backbones (DINO [53]) to get largely semantic grouping on real-world images such as multiple instances of cows/trains or bicycles in their Figures 2 and 20 respectively.
Binding in the brain. The temporal correlation hypothesis posits that the mammalian brain binds together information emitted from groups of neurons that fire synchronously. According to this theory [54, 55], biological neurons transmit information in two ways through their spikes. The spike amplitude indicates the strength of presence of a feature while relative time between spikes indicates which neuronal responses need to bound together during further processing. It also suggests candidate rhythmic cycles in the brain such as Gamma that could play this role in binding [56, 57]. Synchrony-based models functionally implement the same coding scheme [58-60] using complex-valued activations where the relative phase plays the role of relative time between spikes. This abstracts away all aspects of the spiking neuron model to allow easier reproduction on digital hardware.
Contrastive learning with object-centric models. Contrastive learning for object-centric representations has not been extensively explored, with a few notable exceptions. The first method [61] works only on toy images of up to 3 objects on a black background while the second [62] shows results on more complex data, but requires complex hand-crafted data augmentation techniques to contrast samples across a batch. Our method samples positive and negative pairs within a single image and does not require any data augmentation. Most importantly, unlike ours, these models still use slots and attention-based routing, and thereby inherit all of its conceptual limitations. Lastly, ODIN [63] alternately refines the segmentation masks and representation of objects using two networks that are jointly optimized by a contrastive objective that maximizes the similarity between different views of the same object, while minimizing the similarity between different objects. To obtain the segmentation masks for objects, they simply spatially group the features using K-means. Their method relies on careful data augmentation techniques such as local or global crops, viewpoint changes, color perturbations etc. which are applied to one object.
We propose several architectural improvements and a novel contrastive learning method to address limitations of the current state-of-the-art synchrony-based model for object binding, the complex-valued autoencoder (CAE [36]). Our improved architecture,  $\mathrm{CAE}++$ , is the first synchrony-based model capable of dealing with color images (e.g., Tetrominoes). Our contrastive learning method further boosts  $\mathrm{CAE}++$  by improving its phase separation process. The resulting model, CtCAE, largely outperforms  $\mathrm{CAE}++$  on the rather challenging CLEVR and dSprites datasets. Admittedly, our synchrony-based models still lag behind the state-of-the-art slot-based models [15, 18], but this is to be expected, as research on modern synchrony-based models is still in its infancy. We hope our work will inspire the community to invest greater effort into such promising models.
Acknowledgments. We thank Sindy Löwe and Michael C. Mozer for insightful discussions and valuable feedback. This research was funded by Swiss National Science Foundation grant: 200021_192356, project NEUSYM and the ERC Advanced grant no: 742870, AlgoRNN. This work was also supported by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID s1205 and d123. We also thank NVIDIA Corporation for donating DGX machines as part of the Pioneers of AI Research Award.
[1] Anne Treisman. The binding problem. Current opinion in neurobiology, 6(2):171-178, 1996.
[2] Adina L Roskies. The binding problem. Neuron, 24(1):7-9, 1999.
[3] Elizabeth S. Spelke. Principles of object perception. Cognitive Science, 14(1):29-56, 1990.
[4] Max Wertheimer. Untersuchungen zur lehre von der gestalt. ii. Psychologische Forschung, 4(1): 301-350, 1923.
[5] Kurt Koffka. Principles of gestalt psychology. Philosophy and Scientific Method, 32(8), 1935.
[6] Wolfgang Kohler. Gestalt psychology. Psychologische Forschung, 31(1), 1967.
[7] Daniel Kahneman, Anne Treisman, and Brian J Gibbs. The reviewing of object files: Object-specific integration of information. Cognitive psychology, 24(2):175-219, 1992.
[8] Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems, volume 29, 2016.
[9] SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pages 3225-3233, 2016.
[10] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 6691-6701, Long Beach, CA, USA, December 2017.
[11] Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In International Conference on Learning Representations, 2018.
[12] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, pages 8606-8616, 2018.
[13] Christopher P Burgess, Loic Matthew, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390, 2019.
[14] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Proc. Int. Conf. on Machine Learning (ICML), pages 2424-2433, Long Beach, CA, USA, June 2019.
[15] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2020.
[16] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020.