Title: Contrastive Training of Complex-Valued Autoencoders for Object Discovery
Venue: NeurIPS 2023
Accepted: yes
Keywords: object-centric learning

complex-valued networks

unsupervised learning

temporal correlation hypothesis
Abstract: Current state-of-the-art object-centric models use slots and attention-based routing for binding. However, this class of models has several conceptual limitations: the number of slots is hardwired; all slots have equal capacity; training has high computational cost; there are no object-level relational factors within slots. Synchrony-based models in principle can address these limitations by using complex-valued activations which store binding information in their phase components. However, working examples of such synchrony-based models have been developed only very recently, and are still limited to toy grayscale datasets and simultaneous storage of less than three objects in practice. Here we introduce architectural modifications and a novel contrastive learning method that greatly improve the state-of-the-art synchrony-based model. For the first time, we obtain a class of synchrony-based models capable of discovering objects in an unsupervised manner in multi-object color datasets and simultaneously representing more than three objects.

Contrastive Training of Complex-Valued Autoencoders for Object Discovery

Abstract

Current state-of-the-art object-centric models use slots and attention-based routing for binding. However, this class of models has several conceptual limitations: the number of slots is hardwired; all slots have equal capacity; training has high computational cost; there are no object-level relational factors within slots. Synchronybased models in principle can address these limitations by using complex-valued activations which store binding information in their phase components. However, working examples of such synchrony-based models have been developed only very recently, and are still limited to toy grayscale datasets and simultaneous storage of less than three objects in practice. Here we introduce architectural modifications and a novel contrastive learning method that greatly improve the state-of-the-art synchrony-based model. For the first time, we obtain a class of synchrony-based models capable of discovering objects in an unsupervised manner in multi-object color datasets and simultaneously representing more than three objects. 1

1 Introduction

The visual binding problem [1, 2] is of great importance to human perception [3] and cognition [4-6]. It is fundamental to our visual capabilities to integrate several features together such as color, shape, texture, etc., into a unified whole [7]. In recent years, there has been a growing interest in deep learning models [8-19] capable of grouping visual inputs into a set of 'meaningful' entities in a fully unsupervised fashion (often called object-centric learning). Such compositional objectcentric representations facilitate relational reasoning and generalization, thereby leading to better performance on downstream tasks such as visual question-answering [20, 21], video game playing [22-26], and robotics [27-29], compared to other monolithic representations of the visual input.

The current mainstream approach to implement such object binding mechanisms in artificial neural networks is to maintain a set of separate activation vectors (so-called slots ) [15]. Various segregation mechanisms [30] are then used to route requisite information from the inputs to infer each slot in a iterative fashion. Such slot-based approaches have several conceptual limitations. First, the binding information (i.e., addresses) about object instances are maintained only by the constant number of slots-a hard-wired component which cannot be adapted through learning. This restricts the ability of slot-based models to flexibly represent varying number of objects with variable precision without tuning the slot size, number of slots, number of iterations, etc. Second, the inductive bias used by the grouping module strongly enforces independence among all pairs of slots. This restricts individual

slots to store relational features at the object-level, and requires additional processing of slots using a relational module, e.g., Graph Neural Networks [31, 32] or Transformer models [33-35]. Third, binding based on iterative attention is in general computationally very demanding to train [36]. Additionally, the spatial broadcast decoder [37] (a necessary component in these models) requires multiple forward/backward passes to render the slot-wise reconstruction and alpha masks, resulting in a large memory overhead as well.

Recently, Löwe et al. [36] revived another class of neural object binding models [38-40] ( synchronybased models) which are based on complex-valued neural networks. Synchrony-based models are conceptually very promising. In principle they address most of the conceptual challenges faced by slot-based models. The binding mechanism is implemented via constructive or destructive phase interference caused by addition of complex-valued activations. They store and process information about object instances in the phases of complex activations which are more amenable to adaptation through gradient-based learning. Further, they can in principle store a variable number of objects with variable precision by partitioning the phase components of complex activations at varying levels of granularity. Additionally, synchrony-based models can represent relational information directly in their distributed representation, i.e., distance in phase space yields an implicit relational metric between object instances (e.g., inferring part-whole hierarchy from distance in 'tag' space [39]). Lastly, the training of synchrony-based models is computationally more efficient by two orders of magnitude [36].

However, the true potential of synchrony-based models for object binding is yet to be explored; the current state-of-the-art synchrony-based model, the Complex-valued AutoEncoder (CAE) [36], still has several limitations. First, it is yet to be benchmarked on any multi-object datasets [41] with color images (even simplisitic ones like Tetrominoes ) due to limitations in the evaluation method to extract discrete object identities from continuous phase maps [36]. Second, we empirically observe that it shows low separability (Table 2) in the phase space, thereby leading to very poor (near chance-level) grouping performance on dSprites and CLEVR . Lastly, CAE can simultaenously represent at most 3 objects [36], making it infeasible for harder benchmark datasets [41, 42].

Our goal is to improve the state-of-art synchrony models by addressing these limitations of CAE [36]. First, we propose a few simple architectural changes to the CAE: i) remove the 1x1 convolution kernel as well as the sigmoid activation in the output layer of decoder, and ii) use convolution and upsample layers instead of transposed convolution in the decoder. These changes enable our improved CAE, which we call CAE++, to achieve good grouping performance on the Tetrominoes dataset-a task on which the original CAE completely fails. Further, we introduce a novel contrastive learning method to increase separability in phase values of pixels (regions) belonging to two different objects. The resulting model, which we call Contrastively Trained Complex-valued AutoEncoders (CtCAE), is the first kind of synchrony-based object binding models to achieve good grouping performance on multi-object color datasets with more than three objects (Figure 2). Our contrastive learning method yields significant gains in grouping performance over CAE++, consistently across three multi-object color datasets ( Tetrominoes , dSprites and CLEVR ). Finally, we qualitatively and quantitatively evaluate the separability in phase space and generalization of CtCAE w.r.t. number of objects seen at train/test time.

2 Background

We briefly overview the CAE architecture [36] which forms the basis of our proposed models. CAE performs binding through complex-valued activations which transmit two types of messages: magnitudes of complex activations to represent the strength of a feature and phases to represent which features must be processed together. The constructive or destructive interference through addition of complex activations in every layer pressurizes the network to use similar phase values for all patches belonging to the same object while separating those associated with different objects. Patches of the same object contain a high amount of pointwise mutual information so their destructive interference would degrade its reconstruction.

The CAE is an autoencoder with real-valued weights that manipulate complex-valued activations. Let h and w denote positive integers. The input is a positive real-valued image x ′ ∈ R h × w × 3 (height h and width w , with 3 channels for color images). An artificial initial phase of zero is added to each pixel of x ′ (i.e., ϕ = 0 ∈ R h × w × 3 ) to obtain a complex-valued input x ∈ C h × w × 3 :

Let d in , d out and p denote positive integers. Every layer in the CAE transforms complex-valued input x ∈ C d in to complex-valued output z ∈ C d out (where we simply denote input/output sizes as d in and d out which typically have multiple dimensions, e.g., h × w × 3 for the input layer), using a function f w : R d in → R d out with real-valued trainable parameters w ∈ R p . f w is typically a convolutional or linear layer. First, f w is applied separately to the real and imaginary components of the input:

Note that both Re ( x ) , Im ( x ) ∈ R d in . Second, separate trainable bias vectors b m , b ϕ ∈ R d out are applied to the magnitude and phase components of ψ ∈ C d out :

Third, the CAE uses an additional gating function proposed by Reichert and Serre [40] to further transform this 'intermediate' magnitude m ψ ∈ R d out . This gating function dampens the response of an output unit as a function of the phase difference between two inputs. It is designed such that the corresponding response curve approximates experimental recordings of the analogous curve from a Hodgkin-Huxley model of a biological neuron [40]. Concretely, an intermediate activation vector χ ∈ R d out (called classic term [40]) is computed by applying f w to the magnitude of the input x ∈ C d in , and a convex combination of this classic term and the magnitude m ψ (called synchrony term [40]) from Eq. 3 is computed to yield 'gated magnitudes' m z ∈ R d out as follows:

Finally, the output of the layer z ∈ C d out is obtained by applying non-linearities to this magnitude m z (Eq. 4) while leaving the phase values ϕ ψ (Eq. 3) untouched:

The ReLU activation ensures that the magnitude of z is positive, and any phase flips are prevented by its application solely to the magnitude component m z . For more details of the CAE [36] and gating function [40] we refer the readers to the respective papers. The final object grouping in CAE is obtained through K-means clustering based on the phases at the output of the decoder; each pixel is assigned to a cluster corresponding to an object [36].

3 Method

We describe the architectural and contrastive training details used by our proposed CAE++ and CtCAE models respectively below.

CAE++. We first propose some simple but crucial architectural modifications that enable the vanilla CAE [36] to achieve good grouping performance on multi-object datasets such as Tetrominoes with color images. These architectural modifications include - i) Remove the 1x1 convolution kernel and associated sigmoid activation in the output layer (' f out' in Löwe et al. [36]) of the decoder, ii) Use convolution and upsample layers in place of transposed convolution layers in the decoder (cf. ' f dec' architecture in Table 3 from Löwe et al. [36]). We term this improved CAE variant that adopts these architectural modifications as CAE++. As we will show below in Table 1, these modifications allow our CAE++ to consistently outperform the CAE across all 3 multi-object datasets with color images.

Contrastive Training of CAEs. Despite the improved grouping of CAE++ compared to CAE, we still empirically observe that CAE++ shows poor separability 2 (we also illustrate this in Section 4). This motivates us to introduce an auxiliary training objective that explicitly encourages higher separability in phase space. For that, we propose a contrastive learning method [43, 44] that modulates the distance between pairs of distributed representations based on some notion of (dis)similarity between them (which we define below). This design reflects the desired behavior to drive the phase separation process between two different objects thereby facilitating better grouping performance.

Before describing our contrastive learning method mathematically below, here we explain its essential ingredients (illustrated in Figure 1). For setting up the contrastive objective, we first (randomly)

Encoder

Decoder

Magnitudes

Phases

Addresses

Features

Addresses

Features

Anchor

Address

Anchor

Feature

Magnitudes    Addresses

Phases    Features

Sample Negative Pairs Features

Sample Positive Pair Features

Anchor Features

+

_

_

_

Encoder

Phases

Encoder

Magnitudes

Decoder

Decoder

X

X

Algorithm 1 Mining positive and negative pairs for a single anchor for the contrastive objective. Scalar parameters are k top -the number of candidates from which to sample one positive pair and m bottom -the number of candidates from which to sample ( M -1) negative pairs.

Inputs: anchor_address ∈ R d f , addresses ∈ R h ′ × w ′ × d f , features ∈ R h ′ × w ′ × d f . Params: Scalars k top , m bottom , M .

1: addresses ← Flatten(addresses, dims=(0,1))

2: features ← Flatten(features, dims=(0,1))

3: distances ← CosineDistance(addresses, anchor_address)

4: top_k_features ← features[argsort(distances, dim=0)[: k top ]]

5: bottom_m_features ← features[argsort(distances, dim=0)[ -m bottom :]]

6: pos_pair_idx ∼ Uniform [0, k top ]

▷ sample 1 positive pair

7: neg_pair_idxs ∼ Uniform [0, m bottom ] × ( M -1) ▷ M -1 samples without replacement

8: positive_pair ← top_k_features[pos_pair_idx]

9: negative_pairs ← bottom_m_features[neg_pair_idxs]

10: return positive_pair, negative_pairs

sample 'anchors' from a set of 'datapoints'. The main idea of contrastive learning is to 'contrast' these anchors to their respective 'positive' and 'negative' examples in a certain representation space. This requires us to define two representation spaces: one associated with the similarity measure to define positive/negative examples given an anchor, and another one on which we effectively apply the contrastive objective, itself defined as a certain distance function to be minimized. We use the term addresses to refer to the representations used to measure similarity, and consequently extract positive and negative pairs w.r.t. to the anchor. We use the term features to refer to the representations that are contrasted. As outlined earlier, the goal of the contrastive objective is to facilitate separability of phase values. It is then a natural choice to use the phase components of complex-valued outputs as features and the magnitude components as addresses in the contrastive loss. This results in angular distance between phases ( features ) being modulated by the contrastive objective based on how (dis)similar their corresponding magnitude components ( addresses ) are. Since the magnitude components of complexvalued activations are used to reconstruct the image (Equation (7)), they capture requisite visual properties of objects. In short, the contrastive objective increases or decreases the angular distance of phase components of points (pixels/image regions) in relation to how (dis)similar their visual properties are.

Our contrastive learning method works as follows. Let h ′ , w ′ , d f , N A , and M denote positive integers. Here we generically denote the dimension of the output of any CAE layer as h ′ × w ′ × d f . This results in a set of h ′ × w ′ 'datapoints' of dimension d f for our contrastive learning. From this set of datapoints, we randomly sample N A anchors. We denote this set of anchors as a matrix A ∈ R N A × d f ; each anchor is thus denoted as A k ∈ R d f for all k ∈ { 1 , ..., N A } . Now by using

Input

Recon.

GT Mask

Mask

Phase

Magnitude

Algorithm 1, we extract 1 positive and M -1 negative examples for each anchor. We denote these examples by a matrix P k ∈ R M × d f for each anchor k ∈ { 1 , ..., N A } arranged such that P k 1 ∈ R d f is the positive example and all other rows P k j ∈ R d f for j ∈ { 2 , ..., M } are negative ones. Finally, our contrastive loss is an adaptation of the standard InfoNCE loss [44] which is defined as follows:

where N A is the number of anchors sampled for each input, d ( x k ; x l ) refers to the cosine distance between a pair of vectors x k , x l ∈ R d f and τ ∈ R > 0 is the softmax temperature.

We empirically observe that applying the contrastive loss on outputs of both encoder and decoder is better than applying it on only either one (Table 5). We hypothesize that this is the case because it utilizes both high-level, abstract and global features (on the encoder-side) as well as low-level and local visual cues (on the decoder-side) that better capture visual (dis)similarity between positive and negative pairs. We also observe that using magnitude components of complex outputs of both the encoder and decoder as addresses for mining positive and negative pairs while using the phase components of complex-valued outputs as the features for the contrastive loss performs the best among all the other possible alternatives (Table 13). These ablations also support our initial intuitions (described above) while designing the contrastive objective for improving separability in phase space.

Finally, the complete training objective function of CtCAE is:

where L defines the loss for a single input image x ′ ∈ R h × w × 3 , and L mse is the standard reconstruction loss used by the CAE [36]. The reconstructed image ˆ x ∈ R h × w × 3 is generated from the complex-valued outputs of the decoder y ∈ C h × w × 3 by using its magnitude component. In practice, we train all models by minimizing the training loss L over a batch of images. The CAE baseline model and our proposed CAE++ variant are trained using only the reconstruction objective (i.e. β = 0 ) whereas our proposed CtCAE model is trained using the complete training objective.

4 Results

Here we provide our experimental results. We first describe details of the datasets, baseline models, training procedure and evaluation metrics. We then show results (always across 5 seeds) on grouping of our CtCAE model compared to the baselines (CAE and our variant CAE++), separability in phase space, generalization capabilities w.r.t to number of objects seen at train/test time and ablation studies for each of our design choices. Finally, we comment on the limitations of our proposed method.

Datasets. We evaluate the models on three datasets from the Multi-Object datasets suite [41] namely Tetrominoes , dSprites and CLEVR (Figure 2) used by prior work in object-centric learning [14, 15, 45]. For CLEVR , we use the filtered version [45] which consists of images containing less than seven objects. For the main evaluation, we use the same image resolution as Emami et al. [45], i.e., 32x32 for Tetrominoes , 64x64 for dSprites and 96x96 for CLEVR (a center crop of 192x192 that is then resized to 96x96). For computational reasons, we perform all ablations and analysis on 32x32 resolution. Performance of all models are ordered in the same way on 32x32 resolution as the original resolution (see Table 1), but with significant training and evaluation speed up. In Tetrominoes and dSprites the number of training images is 60K whereas in CLEVR it is 50K. All three datasets have 320 test images on which we report all the evaluation metrics. For more details about the datasets and preprocessing, please refer to Appendix A.

Models & Training Details. We compare our CtCAE model to the state-of-the-art synchrony-based method for unsupervised object discovery (CAE [36]) as well as to our own improved version thereof (CAE++) introduced in Section 3. For more details about the encoder and decoder architecture of all models see Appendix A. We use the same architecture as CAE [36], except with increased number of convolution channels (same across all models). We train models for 50K steps on Tetrominoes , and 100K steps on dSprites and CLEVR with Adam optimizer [46] with a constant learning rate of 4e-4, i.e., no warmup schedules or annealing (all hyperparameter details are given in Appendix A).

Evaluation Metrics. We use the same evaluation protocol as prior work [14, 15, 45, 36] which compares the grouping performance of models using the Adjusted Rand Index (ARI) [47, 48]. We report two variants of the ARI score, i.e., ARI-FG and ARI-FULL consistent with Löwe et al. [36]. ARI-FG measures the ARI score only for the foreground and ARI-FULL takes into account all pixels.

Unsupervised Object Discovery. Table 1 shows the performance of our CAE++, CtCAE, and the baseline CAE [36] on Tetrominoes , dSprites and CLEVR . We first observe that the CAE baseline almost completely fails on all datasets as shown by its very low ARI-FG and ARI-FULL scores. The MSE values in Table 1 indicate that CAE even struggles to reconstruct these color

Mask

Mask

Mask

GT Mask

Input

CAE

CAE++

CtCAE

GT Mask

GT Mask

Input

Input

Tetrominoes

dSprites

CLEVR

H

images. In contrast, CAE++ achieves significantly higher ARI scores, consistently across all three datasets; this demonstrates the impact of the architectural modifications we propose. However, on the most challenging CLEVR dataset, CAE++ still achieves relatively low ARI scores. Its contrastive learning-augmented counterpart, CtCAE consistently outperforms CAE++ both in terms of ARI-FG and ARI-FULL metrics. Notably, CtCAE achieves more than double the ARI scores of CAE++ on the most challenging CLEVR dataset which highlights the benefits of our contrastive method. All these results demonstrate that CtCAE is capable of object discovery (still far from perfect) on all datasets which include color images and more than three objects per scene, unlike the exisiting state-of-the-art synchrony-based model, CAE.

Quantitative Evaluation of Separability . To gain further insights into why our contrastive method is beneficial, we quantitatively analyse the phase maps using two distance metrics: inter-cluster and intra-cluster distances. In fact, in all CAE-family of models, final object grouping is obtained through K-means clustering based on the phases at the output of the decoder; each pixel is assigned to a cluster with the corresponding centroid. Inter -cluster distance measures the Euclidean distance between centroids of each pair of clusters averaged over all such pairs. Larger inter-cluster distance allows for easier discriminability during clustering to obtain object assignments from phase maps. On the other hand, intra -cluster distance quantifies the 'concentration' of points within a cluster, and is computed as the average Euclidean distance between each point in the cluster and the cluster centroid. Smaller intra-cluster distance results in an easier clustering task as the clusters are then more condensed. We compute these distance metrics on a per-image basis before averaging over all samples in the dataset. The results in Table 2 show that, the mean intra-cluster distance (last column) is smaller for CtCAE than CAE++ on two (dSprites and CLEVR) of the three datasets. Also, even though the average inter-cluster distance (fourth column) is sometimes higher for CAE++, the minimum inter-cluster distance (third column)-which is a more relevant metric for separability-is

Input

Recon.

GT Mask

Mask

Phase

Magnitude

11

口

larger for CtCAE. This confirms that compared to CAE++, CtCAE tends to have better phase map properties for object grouping, as is originally motivated by our contrastive method.

Object Storage Capacity. Löwe et al. [36] note that the performance of CAE sharply decreases for images with more than 3 objects. We report the performance of CtCAE and CAE++ on subsets of the test set split by the number of objects, to measure how their grouping performance changes w.r.t. the number of objects. In dSprites the images contain 2 , 3 , 4 or 5 objects, in CLEVR 3 , 4 , 5 or 6 objects. Note that the models are trained on the entire training split of the respective datasets. Table 3 shows that both methods perform well on images containing more than 3 objects (their performance does not drop much on images with 4 or more objects). We also observe that the CtCAE consistently maintains a significant lead over CAE in terms of ARI scores across different numbers of objects. In another set of experiments (see Appendix B), we also show that CtCAE generalizes well to more objects (e.g. 5 or 6 ) when trained only on a subset of images containing less than this number of objects.

Ablation on Architectural Modifications.

Table 4 shows an ablation study on the proposed architectural modifications on Tetrominoes (for similar findings on other datasets, see Appendix B.2). We observe that the sigmoid activation on the output layer of the decoder significantly impedes learning on color datasets. A significant performance jump is also observed when replacing transposed convolution layers [36] with

convolution and upsample layers. By applying all these modifications, we obtain our CAE++ model that results in significantly better ARI scores on all datasets, therefore supporting our design choices.

Ablation on Feature Layers to Contrast. In CtCAE we contrast feature vectors both at the output of the encoder and the output of the decoder. Table 5 justifies this choice; this default setting (Enc+Dec) outperforms both other options where we apply the constrastive loss either only in the

encoder or the decoder output. We hypothesize that this is because these two contrastive strategies are complementary: one uses low-level cues (dec) and the other high-level abstract features (enc).

Qualitative Evaluation of Grouping. Finally, we conduct some qualitative analyses of both successful and failed grouping modes shown by CAE++ and CtCAE models through visual inspection of representative samples. In Figure 3, Tetrominoes (columns 1-2), we observe that CtCAE (row 4) exhibits better grouping on scenes with multiple objects of the same color than CAE++ (row 3). This is reflected in the radial phase

plots (column 2) which show better separability for CtCAE than CAE++. Further, on dSprites (rows 3-4, columns 4-5) and CLEVR (rows 3-4, columns 7-8), CtCAE handles the increased number of objects more gracefully while CAE++ struggles and groups several of them together. For the failure cases, Figure 4 shows an example where CtCAE still has some difficulties in segregating objects of the same color (row 2, yellow cube and ball) (also observed sometimes on dSprites , see Figure 14). Further, we observe how the specular highlights on metallic objects (purple ball in row 1 and yellow ball in row 2) form a separate sub-part from the object (additional grouping examples in Appendix C).

Discussion on the Evaluation Protocol. The reviewers raised some concerns about the validity of our evaluation protocol 3 . The main point of contention was that the thresholding applied before clustering the phase values may lead to a trivial separation of objects in RGB images based on their color. While it is true that in our case certain color information may contribute to the separation process, a trivial separation of objects based on solely their color cannot happen since CtCAE largely outperforms CAE++ (see Table 1) despite both having near perfect reconstruction losses and with the same evaluation protocol. This indicates that this potential issue only has a marginal effect in practice in our case and separation in phase space learned by the model is still crucial. In fact, 'pure' RGB color tones (which allow such trivial separation) rarely occur in the datasets ( dSprites and CLEVR ) used here. The percentage of pixels with a magnitude less than 0.1 are 0 . 44% , 0 . 88% and 9 . 25% in CLEVR , dSprites and Tetrominoes respectively. While 9 . 25% in Tetrominoes is not marginal, we observe that this does not pose an issue in practice, as many examples in Tetrominoes where two or three blocks with the same color are separated by CAE++/CtCAE (e.g. three yellow blocks in Figure 3 or two magenta blocks in Figure 8).

To support our argument that using a threshold of 0.1 has no effect on the findings and conclusions, we conduct additional evaluations with a threshold of 0 and no threshold at all. From Table 9 (see Appendix B) we can see that using a threshold of 0 hardly change the ARI scores at all or in some cases it even improves the scores slightly. Using no threshold results in marginally improved ARI scores on CLEVR , near-identical scores on dSprites but worse scores on Tetrominoes (CtCAE still widely outperforms CAE++). However, this is not due to a drawback in the evaluation or our model. Instead, it stems from an inherent difficulty with estimating the phase of a zero-magnitude complex number. The evaluation clusters pixels based on their phase values, and if a complex number has a magnitude of exactly zero (or very small, e.g., order of 1 e -5 ), the phase estimation is ill-defined and will inherently be random as was also previously noted by Löwe et al. [36]. In fact, filtering those pixels in the evaluation is crucial in their work [36] as they largely used datasets with pure black background (exact zero pixel value). Evaluation of discrete object assignments from continuous phase outputs in a model-agnostic way remains an open research question.

5 Related Work

Slot-based binding. A wide range of unsupervised models have been introduced to perform perceptual grouping summarized well by Greff et al. [30]. They categorize models based on the segregation (routing) mechanism used to break the symmetry in representations and infer latent representations (i.e. slots). Models that use 'instance slots' cast the routing problem as inference in a mixture model whose solution is given by amortized variational inference [8, 14], ExpectationMaximization [10] or other approximations (Soft K-means) thereof [15]. While others [9, 12, 49] that use 'sequential slots' solve the routing problem by imposing an ordering across time. These

models use recurrent neural networks and an attention mechanism to route information about a different object into the same slot at every timestep. Some models [13, 17] combine the above strategies and use recurrent attention only for routing but not for inferring slots. Other models break the representational symmetry based on spatial coordinates [50, 16] ('spatial slots') or based on specific object types [51] ('category slots'). All these models still maintain the 'separation' of representations only at one latent layer (slot-level) but continue to 'entangle' them at other layers.

Synchrony-based binding. Synchrony-based models use complex-valued activations to implement binding by relying on their constructive or destructive phase interference phenomena. This class of models have been sparsely explored with only few prior works that implement this conceptual design for object binding [38-40, 36]. These methods differ based on whether they employ both complex-valued weights and activations [38, 39] or complex-valued activations with real-valued weights and a gating mechanism [40, 36]. They also differ in their reliance on explicit supervision for grouping [38] or not [39, 40, 36]. Synchrony-based models in contrast to slot-based ones maintain the 'separation' of representations throughout the network in the phase components of their complexvalued activations. However, none of these prior methods can group objects in color images with up to 6 objects or visual realism of multi-object benchmarks in a fully unsupervised manner unlike ours. Concurrent work [52] extends CAE by introducing new feature dimensions ('rotating features'; RF) to the complex-valued activations. However, RF cannot be directly compared to CtCAE as it relies on either depth masks to get instance grouping on simple colored Shapes or features from powerful vision backbones (DINO [53]) to get largely semantic grouping on real-world images such as multiple instances of cows/trains or bicycles in their Figures 2 and 20 respectively.

Binding in the brain. The temporal correlation hypothesis posits that the mammalian brain binds together information emitted from groups of neurons that fire synchronously. According to this theory [54, 55], biological neurons transmit information in two ways through their spikes. The spike amplitude indicates the strength of presence of a feature while relative time between spikes indicates which neuronal responses need to bound together during further processing. It also suggests candidate rhythmic cycles in the brain such as Gamma that could play this role in binding [56, 57]. Synchrony-based models functionally implement the same coding scheme [58-60] using complex-valued activations where the relative phase plays the role of relative time between spikes. This abstracts away all aspects of the spiking neuron model to allow easier reproduction on digital hardware.

Contrastive learning with object-centric models. Contrastive learning for object-centric representations has not been extensively explored, with a few notable exceptions. The first method [61] works only on toy images of up to 3 objects on a black background while the second [62] shows results on more complex data, but requires complex hand-crafted data augmentation techniques to contrast samples across a batch. Our method samples positive and negative pairs within a single image and does not require any data augmentation. Most importantly, unlike ours, these models still use slots and attention-based routing, and thereby inherit all of its conceptual limitations. Lastly, ODIN [63] alternately refines the segmentation masks and representation of objects using two networks that are jointly optimized by a contrastive objective that maximizes the similarity between different views of the same object, while minimizing the similarity between different objects. To obtain the segmentation masks for objects, they simply spatially group the features using K-means. Their method relies on careful data augmentation techniques such as local or global crops, viewpoint changes, color perturbations etc. which are applied to one object.

6 Conclusion

We propose several architectural improvements and a novel contrastive learning method to address limitations of the current state-of-the-art synchrony-based model for object binding, the complexvalued autoencoder (CAE [36]). Our improved architecture, CAE++, is the first synchrony-based model capable of dealing with color images (e.g., Tetrominoes ). Our contrastive learning method further boosts CAE++ by improving its phase separation process. The resulting model, CtCAE, largely outperforms CAE++ on the rather challenging CLEVR and dSprites datasets. Admittedly, our synchrony-based models still lag behind the state-of-the-art slot -based models [15, 18], but this is to be expected, as research on modern synchrony-based models is still in its infancy. We hope our work will inspire the community to invest greater effort into such promising models.

Acknowledgments. We thank Sindy Löwe and Michael C. Mozer for insightful discussions and valuable feedback. This research was funded by Swiss National Science Foundation grant: 200021_192356, project NEUSYM and the ERC Advanced grant no: 742870, AlgoRNN. This work was also supported by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID s1205 and d123. We also thank NVIDIA Corporation for donating DGX machines as part of the Pioneers of AI Research Award.

References

Anne Treisman. The binding problem. Current opinion in neurobiology , 6(2):171-178, 1996.

Adina L Roskies. The binding problem. Neuron , 24(1):7-9, 1999.

Elizabeth S. Spelke. Principles of object perception. Cognitive Science , 14(1):29-56, 1990.

Max Wertheimer. Untersuchungen zur lehre von der gestalt. ii. Psychologische Forschung , 4(1): 301-350, 1923.

Kurt Koffka. Principles of gestalt psychology. Philosophy and Scientific Method , 32(8), 1935.

Wolfgang Köhler. Gestalt psychology. Psychologische Forschung , 31(1), 1967.

Daniel Kahneman, Anne Treisman, and Brian J Gibbs. The reviewing of object files: Objectspecific integration of information. Cognitive psychology , 24(2):175-219, 1992.

Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems , volume 29, 2016.

SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems , pages 3225-3233, 2016.

Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Proc. Advances in Neural Information Processing Systems (NIPS) , pages 6691-6701, Long Beach, CA, USA, December 2017.

Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In International Conference on Learning Representations , 2018.

Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems , pages 8606-8616, 2018.

Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390 , 2019.

Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Proc. Int. Conf. on Machine Learning (ICML) , pages 2424-2433, Long Beach, CA, USA, June 2019.

Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , Virtual only, December 2020.

Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations , 2020.

Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations , 2020.

Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALLE learns to compose. In Int. Conf. on Learning Representations (ICLR) , Virtual only, April 2022.

Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. In Advances in Neural Information Processing Systems , 2022.

David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matthew Botvinick. Attention over learned object embeddings enables complex visual reasoning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. In The Eleventh International Conference on Learning Representations , 2023.

Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations , 2019.

Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems , 32, 2019.

Anand Gopalakrishnan, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Unsupervised object keypoint learning using local spatial predictability. In International Conference on Learning Representations , 2021.

Aleksandar Stani´ c, Yujin Tang, David Ha, and Jürgen Schmidhuber. Learning to generalize with object-centric agents in the open world survival game crafter. ArXiv , abs/2208.03374, 2022.

Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object-centric representations for reinforcement learning. arXiv preprint arXiv:2302.04419 , 2023.

Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In 2021 IEEE international conference on robotics and automation (ICRA) , pages 6169-6176. IEEE, 2021.

Yizhe Wu, Oiwi Parker Jones, Martin Engelcke, and Ingmar Posner. Apex: Unsupervised, object-centric scene segmentation and tracking for robot manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3375-3382, 2021.

Mohit Sharma and Oliver Kroemer. Generalizing object-centric task-axes controllers using keypoints. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 7548-7554, 2021.

Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. Preprint arXiv:2012.05208 , 2020.

Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems , 29, 2016.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS) , pages 5998-6008, December 2017.

Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation , 4(1):131-139, 1992.

Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Proc. Int. Conf. on Machine Learning (ICML) , 2021.

Sindy Löwe, Phillip Lippe, Maja Rudolph, and Max Welling. Complex-valued autoencoders for object discovery. Transactions on Machine Learning Research , 2022. ISSN 2835-8856.

Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in V AEs. In Learning from Limited Labeled Data (LLD) Workshop, ICLR , New Orleans, LA, USA, May 2019.

Michael C Mozer, Richard Zemel, and Marlene Behrmann. Learning to segment images using dynamic feature binding. Advances in Neural Information Processing Systems , 4, 1991.

Michael C Mozer. A principle for unsupervised hierarchical decomposition of visual scenes. In Advances in Neural Information Processing Systems , volume 11, 1998.

David P Reichert and Thomas Serre. Neuronal synchrony in complex-valued deep networks. In International Conference on Learning Representations , 2014.

Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. https://github.com/deepmind/multiobject-datasets/, 2019.

Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: a scalable dataset generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 297-304, 2010.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized inference for learning symmetric and disentangled multi-object representations. In International Conference on Machine Learning , pages 2970-2981. PMLR, 2021.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations , 2015.

William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association , 66(336):846-850, 1971.

Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification , 2:193-218, 1985.

Aleksandar Stani´ c and Jürgen Schmidhuber. R-sqair: Relational sequential attend, infer, repeat. ArXiv , abs/1910.05231, 2019.

Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , 2019.

Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing. In International Conference on Learning Representations , 2018.

Sindy Löwe, Phillip Lippe, Francesco Locatello, and Max Welling. Rotating features for object discovery. arXiv preprint arXiv:2306.00600 , 2023.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

Christoph von der Malsburg. Binding in models of perception and brain function. Current opinion in neurobiology , 5(4):520-526, 1995.

Wolf Singer and Charles M Gray. Visual feature integration and the temporal correlation hypothesis. Annual review of neuroscience , 18(1):555-586, 1995.

John G.R Jefferys, Roger D Traub, and Miles A Whittington. Neuronal networks for induced '40 hz' rhythms. Trends in Neurosciences , 19(5):202-208, 1996.

Pascal Fries, Danko Nikoli´ c, and Wolf Singer. The gamma cycle. Trends in neurosciences , 30 (7):309-316, 2007.

Christoph von der Malsburg and Werner Schneider. A neural cocktail-party processor. Biological cybernetics , 54(1):29-40, 1986.

Christoph von der Malsburg and Joachim Buhmann. Sensory segmentation with coupled neural oscillators. Biological cybernetics , 67(3):233-242, 1992.

Andreas K Engel, Peter König, Andreas K Kreiter, Thomas B Schillen, and Wolf Singer. Temporal coding in the visual cortex: new vistas on integration in the nervous system. Trends in neurosciences , 15(6):218-226, 1992.

Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. arXiv preprint arXiv:2011.10287 , 2020.

Federico Baldassarre and Hossein Azizpour. Towards self-supervised learning of global and object-centric representations. arXiv preprint arXiv:2203.05997 , 2022.

Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelovi´ c. Object discovery and representation networks. In European Conference on Computer Vision , pages 123-143, 2022.

A Experimental Details

Datasets. We evaluate all models (i.e., CAE, CAE++ and CtCAE) on a subset of Multi-Object dataset [41] suite. We use three datasets: Tetrominoes which consists of colored tetris blocks on a black background, dSprites with colored sprites of various shapes like heart, square, oval, etc., on a grayscale background, and lastly, CLEVR , a dataset from a synthetic 3D environment. For CLEVR , we use the filtered version [45] which consists of images containing less than seven objects sometimes referred to as CLEVR6 as in Locatello et al. [15]. We normalize all input RGB images to have pixel values in the range [0 , 1] consistent with prior work [36].

Models. Table 6 shows the architecture specifications such as number of layers, kernel sizes, stride lengths, number of filter channels, normalization layers, activations, etc., for the convolutional neural networks used by the Encoder and Decoder modules in CAE, CAE++ and CtCAE.

Encoder

3

×

3 conv, 128 channels, stride 2, ReLU, BatchNorm

3

3

3

×

×

×

3 conv, 128 channels, stride 1, ReLU, BatchNorm

3 conv, 256 channels, stride 2, ReLU, BatchNorm

3 conv, 256 channels, stride 1, ReLU, BatchNorm

3

×

3 conv, 256 channels, stride 2, ReLU, BatchNorm

----------------

For 64

96 inputs, 2 additional encoder layers:

3

×

×

64 and 96

×

3 conv, 256 channels, stride 1, ReLU, BatchNorm

3

×

3 conv, 256 channels, stride 2, ReLU, BatchNorm

Linear Layer

Linear, 256 ReLU units, LayerNorm

Decoder

For 64

×

64 and 96

×

96 inputs, 2 additional decoder layers:

Bilinear upsample x2

3

×

3 conv, 256 channels, stride 1, ReLU, BatchNorm

3

×

3 conv, 256 channels, stride 1, ReLU, BatchNorm

----------------

Bilinear upsample x2

3

×

3 conv, 256 channels, stride 1, ReLU, BatchNorm

3

×

3 conv, 256 channels, stride 1, ReLU, BatchNorm

Bilinear upsample x2

3

×

3 conv, 256 channels, stride 1, ReLU, BatchNorm

3

×

3 conv, 128 channels, stride 1, ReLU, BatchNorm

Bilinear upsample x2

3

×

3 conv, 128 channels, stride 1, ReLU, BatchNorm

Output Layer (CAE only)

1

×

1 conv, 3 channels, stride 1, sigmoid

Training Details. Table 7 and Table 8 show the hyperparameter configurations used to train CAE/CAE++ and to contrastively train the CtCAE model(s) respectively.

Computational Efficiency. We report the training and inference time (wall-clock) for our models across the various image resolutions of the 3 multi-object datasets. First, we find that inference time(s) is similar for all models, and it only depends on the image resolution and number of ground truth clusters in the input image. Inference for the test set containing 320 images of 32x32 resolution takes 25 seconds, for 64x64 images it takes 65 seconds and for the 96x96 images it takes 105 seconds. Training time(s) on the other hand differs both across models and image resolutions. We report all

training time(s) using a single Nvidia V100 GPU. CAE and CAE++ have similar training times: for 32x32 images training for 100k steps takes 3.2 hours, for 64x64 images it takes 4.8 hours, and for 96x96 images it takes 7 hours. CtCAE on the other hand takes 5 hours, 9.2 hours and 10.2 hours to train on dataset of 32x32, 64x64 and 96x96 images respectively. To reproduce all the results/tables (mean and std-dev across 5 seeds) reported in this work we estimate the compute requirement to be 840 GPU hours in total. Further, we estimate that the total compute used in this project is approximately 5-10x more, including experiments with preliminary prototypes.

Object Assignments from Phase Maps. We describe the process of extracting discrete-valued object assignments from continuous-valued phase components of decoder outputs y ∈ C h × w × 3 . First, the phase components of decoder outputs are mapped onto a unit circle and those phase values are masked out whose magnitudes are below the threshold value of 0 . 1 . After this normalization and filtering step, the resultant phase values are converted from polar to Cartesian form on a per-channel basis. Finally, we apply K-means clustering where the number of clusters K parameter is retrieved from the ground-truth segmentation mask for each image. This extraction methodology of object assignments from output phase maps is consistent with Löwe et al. [36].

B Additional results

We show some additional experimental results: generalization capabilities of CtCAE w.r.t. the number of objects, and ablation studies on various design choices made in CAE++ and CtCAE models.

B.1 Generalization to Higher Number of Objects

Here we evaluate generalization capabilities of CtCAE by training only on a subset of images containing less than a certain number of objects, on CLEVR . We have two cases-training on subsets with either up to 4 or up to 5 objects while the original trainig split contain between 3 and 6 objects. The results in Table 10 show that CtCAE generalizes well. The performance drops only marginally when trained on up to 4 objects and tested on 5 and 6 objects, or when trained on up to 5 objects and tested on 6 objects. We also observe that training on 5 or less objects also consistently improves ARI scores for every subset of less than 5 objects. We suspect that the reason for this, apart from having more training data, is that the network has more pressure to separate objects in the phase space when observing more objects, which in turn also helps for images with a smaller number of objects.

B.2 Architecture Modifications

Table 11 shows the results from the ablation study that measures the effect of applying each of our proposed architectural modifications cumulatively to finally result in the CAE++ model. Across all datasets, we consistently observe that, starting from the vanilla CAE baseline which completely fails, grouping performance gradually improves as more components of the CAE++ model is added. Lastly, our proposed contrastive objective used to train CtCAE further improves CAE++ across all 3 datasets.

B.3 Contrastive Learning Ablations

Table 12 shows an ablation study for the choice of layer(s) to which we apply our contrastive learning method. Please note that all variants here always use magnitude components of complex-valued activations as addresses and phase components as features to contrast. We observe that the variant

which applies the contrastive objective to both the outputs of the encoder and decoder ('enc+dec') outperforms the others which apply it only to either one ('enc-only' or 'dec-only'). This behavior is consistent across Tetrominoes , dSprites and CLEVR . We hypothesize that this is because these two contrastive strategies are complementary: one is on low-level cues (decoder) and the other one based on high-level abstract features (encoder).

Table 13 shows an ablation study for the choice of using magnitude or phase as addresses (and the other one as features ) in our contrastive learning method. As we apply contrastive learning to both the decoder and encoder outputs, we can make this decision independently for each output (resulting in four possible combinations). We observe that the variant which uses magnitude for both the encoder and decoder ('mg+mg') outperforms other variants across all 3 multi-object datasets. These ablations support our initial intuitions when designing our contrastive objective.

C Additional Visualizations

We show additional visualization samples of groupings from CAE, CAE++ and CtCAE on Tetrominoes , dSprites and CLEVR , and qualitatively highlight both grouping failures and successes of CAE, CAE++ and CtCAE.

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

T

L

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

Ei.t.

：

..

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

CAE

CAE++

CtCAE

Input

Recon.

GT Mask

Mask

Phase

Magnitude

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper improves the state-of-the-art in synchrony based object centric learning methods. The current SoTA is the Complex-valued AutoEncoder (CAE). But CAE is unable to handle even the Tetrominoes dataset. The paper presents CAE++ and CtCAE.\n\nCAE++ = CAE + minor architectural improvements\n\nCtCAE = CAE++ + Contrastive loss term.\n\nExperiments show that CAE++ and CtCAE are the first synchrony based methods that can handle colour images, and CtCAE can handle up to 6 objects per image.\n\nThe contrastive loss term in CtCAE is intended to make the object clusters well separated in phase space. This is then evaluated in Table 2 by measuring inter-cluster and intra-cluster distances. The rest of the paper measures ARI and FG-ARI on Tetrominoes, dSprites, and CLEVR datasets. Ablation experiments are used to justify the design choices.\n\nSTRENGTHS:\nAlthough this line of work is niche, it is promising and engages scientific curiosity among many readers. The paper advances this line of work closer to the overall SoTA among object centric learning methods; although there is still a large gap here.\n\nThe method is easy to understand and the paper is well written.\n\nThe experiments cover most design choices and compare against CAE, the obvious baseline. The experiments also address important questions such as the robustness across varying number of objects and generalization to more objects. Error bars are included in all tables.\n\nWEAKNESSES:\nPlease incorporate the following minor improvements\n- In line 175, it is claimed that performance is closely matched at 32x32. This is not true for dSprites ARI-FULL is 0.90 vs 0.68. I believe it may be sufficient to say that the methods are ordered the same way across resolutions.\n- In Table 2, please indicate in row 1 whether higher is better (up arrow) or down is better (lower arrow) to aid interpretation. In some cases numbers have been bolded despite overlapping error bars. Are these statistically significant?\n- In lines 299-305, consider citing Odin (https://arxiv.org/abs/2203.08777) which does pixel level BYOL followed by kmeans to get objects. It is kind of a contrastive object centric method (although mean-teacher distillation is probably the right word for BYOL).\n- Some background explanation about the encoder and decoder and where the kmeans happens to get masks would make it even easier to understand the method, for readers not familiar with CAE.\n\nThe main drawback is that slot based methods (DINOSAUR, https://openreview.net/forum?id=b9tUk-f_aG) are now working on real world data whereas Synchrony based methods are yet to make the break into textured images. But I think this paper is still relevant and that future developments in this line of work may catch-up to Slot based methods.\n\nQUESTIONS:\nI've asked my questions in the weaknesses section.\n\nAdditionally a sweep over the \\beta parameter would be nice.\n\nLIMITATIONS:\nThe authors have adequately addressed the weakness although the empirical gap between slot based and synchrony based methods should be made clearer in my opinion.\n\nI do not see any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCAEs hold potential to address several issues associated with slot-based representation. They may provide flexible object granularity, enable the extraction of part-whole hierarchies as needed, and provide faster training speeds. However, testing of the original CAE was confined to grayscale images and a small number of objects (2-3). This paper explores the question: how can we scale up CAEs beyond this initial scope?\n\nFor this, the paper takes the following approach:\n1. Discovers certain architectural modifications that make original CAE perform much better on more complex datasets, in this case, Tetris, CLEVR, and dSprites.\n2. However, noting that this is not enough, the paper proposes a novel contrastive loss, which improves the segmentation performance even further when added to the original reconstruction loss of CAE.\n\nSTRENGTHS:\n1. Useful ablations show that architectural optimizations can be important to achieve the full potential of CAE. A good thing about some of these optimizations is that they actually relax and don’t complicate the original architecture e.g., by removing sigmoid activation.\n2. Empirically show higher storage capacity in terms of the number of objects than CAE.\n3. Paper is largely well-written.\n\nWEAKNESSES:\n1. While it is interesting that contrastive loss helps the segmentation performance, it somehow does not provide me new insights about the CAE framework itself. For instance, it doesn’t illuminate why separability was poor in the original CAE in the first place. As such, I am a bit worried that the proposed solution addresses a symptom rather than the root cause.\n2. Line 215: *“The results in Table 2 indicate that indeed, intra-cluster distance (the 216 last column) is smaller for CtCAE than CAE++ for every dataset, especially for the most difficult ones”* It is actually difficult to conclude this from Table 2 considering the large standard deviations of these measurements. My suggestion would be to tone down this claim and/or confine this result to the appendix. Regarding the separability results more broadly, it is not clear whether separability in the representation space (as defined in this paper) is actually important with respect to the things we may eventually care about e.g., downstream performance or other tasks.\n3. (Optional) It would be good to also report performances of some slot-based baselines, not for strict comparison, but to help the reader position the current performance in a more complete perspective. I see a line mentioned about it in the conclusion but without a pointer to a supporting result.\n\nQUESTIONS:\n1. Why is CAE performance not reported in Table 3?\n2. Why not apply the contrastive loss on all layers rather than just the output of the encoder and the decoder? It appears a bit arbitrary to choose two specific layers to apply this loss.\n\nLIMITATIONS:\nYes, the paper discusses the limitations relative to current slot-based methods, but one should also consider the potential benefits of the CAE family of models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a collection of small modifications of the Complex-valued AutoEncoder architecture leading to improved reconstruction/segmentation performance on simple RGB datasets; a contrastive learning objective is introduced to further improve the performance.\n\nSTRENGTHS:\n- The paper's story is easy to follow, and the motivation is clearly stated.\n- Architectural choices are justified through ablation studies.\n- Experimental results are averaged over multiple random seeds.\n\nWEAKNESSES:\n- The captions of figures/tables could be improved to make it easier for readers to understand what is displayed. For example, Figure 1 is hard to understand without reading the relevant section.\n- Table 1: While I see that this is a comparatively new line of research on new approaches for object-centric learning, I'd still appreciate it if the authors compared to state-of-the-art models so that readers get a feeling for how far the gap still is/how small it has become.\n\nQUESTIONS:\n- Eq. 2: A short note on why this separation is used and not one that uses the magnitude and phase could be helpful.\n- L134: Moving the reference to Figure 1 to the previous paragraph might make it easier to follow the general idea of this loss.\n- Eq. 6: Is this the loss for a single input image, meaning that the loss is then averaged over a batch? Or are the negative/positive samples chosen across a batch? If it's not across the entire batch, what speaks against doing this?\n- Figure 2: An explanation of what the last three columns show/what one can learn from looking at them can be useful for readers.\n- L218: Please elaborate on why the minimum distance is more relevant than the average.\n\nLIMITATIONS:\nWhile the authors mention the strongest limitation, a more extensive discussion of this approach's limitations is missing.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes several improvements over the Complex AutoEncoder (CAE) to scale it to more complex multi-object datasets. The CAE is a synchrony-based model that learns object-centric representations by injecting complex-valued activations into every layer of its architecture, but it is only applicable to grayscale images with up to 3 objects. In this paper, two modifications to the original architecture and a novel auxiliary loss function are proposed that make the CAE applicable to RGB datasets with more than 3 objects.\n\nSTRENGTHS:\nOverall, the paper is very well written and easy to follow. The introduction provides a good overview over the relevant research and a convincing motivation for the proposed approach. The methods section describes the proposed improvements and novel contrastive loss clearly. The results section is well-structured, and the related works section embeds the proposed research well into the existing research landscape.\n\nThe proposed contrastive loss seems like an elegant solution towards improving the \"separability\" of objects in the phase space. The complex-valued features contain two separate types of information (feature presence in the magnitudes and feature binding in the phases), and the proposed contrastive loss smartly utilizes this separation.\n\nThe provided ablation studies provide interesting insights into the proposed model improvements and design choices.\n\nWEAKNESSES:\n1) The main problem with the paper is that it seems to evaluate the models incorrectly. As described in section C.1 - \"RGB Images\" of the paper proposing the CAE [1], it cannot simply be applied to RGB images due to its evaluation method. During evaluation of the CAE, as well as in the proposed approach, features with small magnitudes are masked out, as they may exhibit increasingly random orientation values. However, this rescaling may lead to a trivial separation of objects in RGB images. For an image containing a red and a blue object, for example, the reconstruction will assign small magnitudes to the color channels that are inactive for the respective objects. By masking out values with small magnitudes, objects will subsequently be separated based on their underlying color values, rather than their separation as learned by the model. As far as I can tell, the paper does not describe any mechanism that would prevent the evaluation from doing this. I believe this effect is even visible in the qualitative examples. In Figure 3, the phase image of the CtCAE model on the dSprites example does not seem to clearly separate the purple and orange objects. The mask, however, separates them perfectly. In my eyes, this is only possible if the evaluation procedure makes use of additional (color) information besides the pure phase output. Overall, this makes it unclear whether the presented results are valid.\n\n2) Eq. 6: When measuring the cosine distance between the phase feature vectors, extra care would be necessary to take into account their circular nature. Otherwise, two phase vectors $a=(0,0,0)$ and $b=(2 \\pi, 2 \\pi, 2 \\pi)$ would have a large cosine distance to one another even though they represent the same angles. Has this been taken into account?\n\n3) It would be interesting to see an evaluation of how the proposed improvements influence the separation of objects in the latent space of the model. Ultimately, the goal of object discovery is to create object-centric representations within the model, but the paper currently only evaluates the separation at the output.\n\nMinor points:\n- line 25: not all slot-based method work with iterative attention procedures\n- Personally, I would move the CAE overview into a separate Backgrounds section\n- line 90: $m_\\psi$ can technically not be called a magnitude, as it may contain negative values.\n- Eq. (6): Is there no exp() applied to the cosine distances, such that the equation would correspond to a Softmax?\n- Using a dimensionality of 32x32 for Tetrominoes changes the shapes of the objects. While at the original dimension, all objects are made up from perfect squares, these squares get squeezed randomly when resizing the images. Thus, it would be better to make use of the original input dimension.\n\n\n\n---\n\n[1] Löwe, S., Lippe, P., Rudolph, M., & Welling, M. (2022). Complex-valued autoencoders for object discovery. arXiv preprint arXiv:2204.02075.\n\nQUESTIONS:\n1) As described above, I believe the main weakness of this paper is its evaluation procedure. If the authors could come up with a solution to the described problem or convince me that this problem does not apply to their setup, I'd be happy to increase my score. \n\n\nOther questions:\n\n2) Why do you not apply the contrastive loss to every (convolutional) layer of the architecture?\n\n3) Do you have an explanation as to why the standard deviation is so large for some of the results of the CtCAE (Table 1)?\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-05"}