Contrastive Training of Complex-Valued Autoencoders for Object Discovery
Authors: Aleksandar Stanić
Anand Gopalakrishnan
Kazuki Irie
Jürgen Schmidhuber
Venue: NeurIPS 2023 poster (2023)
Keywords: object-centric learning
complex-valued networks
unsupervised learning
temporal correlation hypothesis
TL;DR: Improvements to the architecture and training of current state-of-the-art synchrony-based model that facilitates it to group multi-object datasets with colour images and simultaenous represent more than three objects.

Abstract: Current state-of-the-art object-centric models use slots and attention-based routing for binding. However, this class of models has several conceptual limitations: the number of slots is hardwired; all slots have equal capacity; training has high computational cost; there are no object-level relational factors within slots. Synchrony-based models in principle can address these limitations by using complex-valued activations which store binding information in their phase components. However, working examples of such synchrony-based models have been developed only very recently, and are still limited to toy grayscale datasets and simultaneous storage of less than three objects in practice. Here we introduce architectural modifications and a novel contrastive learning method that greatly improve the state-of-the-art synchrony-based model. For the first time, we obtain a class of synchrony-based models capable of discovering objects in an unsupervised manner in multi-object color datasets and simultaneously representing more than three objects.

Contrastive Training of Complex- Complex-Valued Autoencoders for Object Discovery
Aleksandar Stanić!*
Anand Gopalakrishnan' Kazuki Irie Jürgen Schmidhuber 'The Swiss AI Lab, IDSIA, USI & SUPSI, Lugano, Switzerland Center for Brain Science, Harvard University, Cambridge, USA Al Initiative, KAUST, Thuwal, Saudi Arabia faleksandar, anand, juergen) Gidsia.. kirießfas harvard.edi
Abstract
Current state-of-the-an object-centric models slots and attention- based routing for binding. However, this class of models has several conceptual limitations: the number of slots is hardwired; all slots have equal capacity; training has high com- putational cost; there are no object-level relational factors within slots. Synchrony- based models in principle activations which store binding information their phase components. However, working examples of such synchrony- based models have been developed only very recently, and are still limited to toy grayscale datasets and simultancous storage of less than three objects in practice. Here we introduce architectural modifications and a novel contrastive learning method that greatly improve the state-of-the synchrony- -based model. For the first time, we obtain class of synchrony- based models capable of discovering objects in an unsupervised manner in multi-object color datasets and simultaneously representing more than three objects
address these limitations by using complex-valued
Introduction
The visual binding problem 1 is of great importance human perception S and cognition [ It is fundamental to our visual capabilities to integrate several features together such color, shape, texture, etc., into unified whole [7]. recent years, there has been a growing interest in deep learning models [9] capable of grouping visual inputs into a set of 'meaningful' entities in fully unsupervised fashion (often called object-centric learning). Such compositional object- centric representations facilitate relational reasoning and generalization, thereby leading to better performance on downstream tasks such as visual guestion-answering 20 21). video game playing (2 and robotics [77 27.29.. compared to other monolithic representations of the visual input.
The current mainstream approach to implement such object binding mechanisms in artificial neural networks is to maintain a set of separate activation vectors (so-called slots) [1]. Various segregation mechanisms 30 are then used route requisite information from the inputs to infer each slot in a iterative fashion. Such blsed approaches have several conceptual limitations, First, the binding information (i.e., addresses) about object instances are maintained only by the constant number of slots hard-wired component which cannot be adapted through learning. This restricts the ability of slot-based models to flexibly represent varying number of objects with variable precision without tuning the slot size, number of slots, number of iterations, etc. Second, the inductive bias used by the grouping module strongly enforces independence among all pairs of slots. This restricts individual
"Equal contribution.
i Work done at IDSIA. Official code repository:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

slots to store relational features at the object-level, and requires additional processing of slots using a relational module, e.g., Graph Neural Networks [3] 32 or Transformer models [5]. Third, binding based on iterative attention is in general computationally very demanding to train [36]. Additionally; the spatial broadcast decoder [37 (a necessary component in these models) requires
multiple forward/backward passes to render the slot-wise reconstruction and alpha masks, resulting
in large memory overhead as well.
(synchrony- based models) which are based on complex-valuee neural networks. Synchrony- based models are conceptually very promising. In principle they address most of the conceptual challenges faced by slot-based models. The binding mechanism is implemented via constructive or destructive phase inter- ference caused by addition of complex-valued valued activations. They store and process information about object instances the phases of complex activations which are more amenable to adaptation through gradient-bar based learning. Further, they can in principle store variable number of objects with variable precision by partitioning the phase components of complex activations at varying levels of granularity. Additionally, synchrony-based models can represent relational information directly in their distributed representation, i.e., distance in phase space yields an implicit relational metric between object in- stances (e.g., inferring part-whole hierarchy from distance in "tag" space (39]. ). Lastly, the training of synchrony- models is computationally more efficient by two orders of magnitude [36).
Recently, Löwe et al. [36] revived another class of neural object binding models [38-40
However, the true potential of synchrony-basee models for object binding is yet to be explored; the current state-of-the-ar synchrony-ber baased model, the Complex-valued AutoEncoder (CAE) (36 still has several limitations. First, it is yet to be benchmarked on any multi-object datasets 44 with color images (even simplisitic ones like Tetrominoes) due to limitations the evaluation method to extract discrete object identities from continuous phase maps [36]. Second, we empirically observe that it shows low separability (Table 2) in the phase space, thereby leading to very poor (near chance-level) grouping performance on dSprites and CLEVR. Lastly, CAE can simultacnously represent at most 3 objects (36,, making it infeasible for harder benchmark datasets 41
Our goal is to improve the state-of-ar art synchrony models by addressing these limitations of CAE 33) First, we propose few simple architectural changes to the CAE: i) remove the 1x1 convolution kernel as well as the sigmoid activation in the output layer of decoder, and ii) use convolution and upsample layers instead of transposed convolution in the decoder. These changes enable our improved CAE, which we call CAE++, to achieve good grouping performance on the Tetrominos noes dataset task on which the original CAE completely fails. Further, we introduce a novel contrastive learning method to increase separability in phase values of pixels (regions) belonging two different objects. The result- ing model, which we call Contrastively Trained Complex-valle camplex-valued AutoEncoders (CICAE), is the first kind of synchrony- bbsed object binding models achieve good grouping performance on multi-object color datasets with more than three objects (Figure 2p. Our contrastive learning method yields signifi- cant gains in grouping performance over CAE++, consistently across three multi-object color datasets (Tetrominoes, dSprites and CLEVR). Finally, we qualitatively and quantitatively evaluate the sep- arability in phase space and generalization of C(CAE w.r.t. number of objects seen at train/test time.
Background
We briefly overview the CAE architecture (36] which forms the basis of our proposed models. CAE performs binding through complex-valued activations which transmit two types of messages: magnitudes of complex activations to represent the strength of feature and phases represent which features must be processed together. The constructive or destructive interference through addition of complex activations in every layer pressurizes the network to use similar phase values for all patches belonging to the same object while separating those associated with different objects. Patches of the same object contain high amount of pointwise mutual information their destructive interference would degrade its reconstruction.
The CAE is an autoencoder with real-valued weights that manipulate complex-valued activations. Let and w denote positive integers. The input is a positive real-valued image x' E k4x=x3 (height h and width w, with 3 channels for color images). An artificial initial phase of zero is added to each pixel of x' (i.e., Φ = 0 E IRA hxxxx3 to obtain a complex-vai -valued input x E Chxux3,
x = x' 3 @e. where @ denotes Hadamard product
(I)

---

Let dins dout and denote positive integers. Every layer in the CAE transforms x E Cdd complex-valuee output z E C.ue (where we simply denote imput/output sizes as dn and doa which typically have multiple dimensions, e.g., h x x 3 for the input layer), using function Sw IRd -> dade with real-valued trainable parameters w E RR, fw is typically a convolutional or linear layer. First, fw is applied separately to the real and imaginary components of the input:
input
vb fw (Re(x)) + fw (Im(x)) E C
Note that both Re(x), Im(x) applied to the magnitude and phase components of 2/6 E Odwn;
R Second, separate trainable bias vectors Dm by E R are
ma = |bb| + bm E Rh
Φv = arg( by
R
Third, the CAE uses an additional gating function proposed by Reichert and Serre 40 to further transform this "intermediate" magnitude my E R This gating function dampens the response of an output unit as function of the phase difference between two inputs. It is designed such that the corresponding response curve approximates experimental recordings of the analogous curve from Hodgkin-Huxley model of a biological neuron [40]. Concretely, an intermediate activation vector X E dR (called classic term [40) is computed by applying fw to the magnitude of the input E Cdd,, and convex combination of this classic term and the magnitude my (called synchrony
term 40 from Eq. is computed to yield "gated magnitudes" my E Rh as follows:
1 m
x fw (1x() + bm
E IR
dur
m
+
2 E IR.
(4)
2
Finally, the output of the layer z E Chuu is obtained by applying non-linearities to this magnitude m, (Eq. while leaving the phase values Db (Eq. untouched:
(eLUU(BatchNNom((m.) + e
Cd
The ReLU activation ensures that the magnitude of is positive, and any phase flips are prevented by its application solely to the magnitude component my- For more details of the CAE [36] and gating function [40] we refer the readers to the respective papers. The final object grouping in CAE is obtained through K- K-means clustering based on the phases the output of the decoder; each pixel is assigned to a cluster corresponding to an object [36].
3 Method
We describe the architectural and contrastive training details used by our proposed CAE++ and CíCAE models respectively below.
We first propose some simple but crucial architectural modifications that enable the vanilla CAE (36] to achieve good grouping performance on multi-object datasets such as Tetrominoes with color images. These architectural modifications include - i) Remove the 1x1 convolution kernel and associated sigmoid activation in the output layer (*fom" " in Löwe et al. (36) ) of the decoder, ii) Use convolution and upsample layers in place of transposed convolution layers in the decoder (cf. " "faes" architecture Table 3 from Löwe et (36)). We term this improved CAE variant that adopts these architectural modifications as CAE++. As we will show below in Table I these modifications allow our CAE++ to consistently outperform the CAE across all multi-object datasets with color images.
CAE++.
Contrastive Training of CAEs. Despite the improved grouping of CAE++ compared to CAE, we still empirically observe that CAE++ shows poor sepurability 2 (we also illustrate this in Section 4). This motivates us to introduce an auxiliary training objective that explicitly encourages higher separability in phase space. For that, we propose contrastive learning method 43 44
that modulates
the distance between pairs of distributed representations based on some notion of (dis)similarity between them (which we define below). This design reflects the desired behavior to drive the phase separation process between two different objects thereby facilitating better grouping performance.
Before describing our contrastive learning method mathematically below, here we explain its essential ingredients (illustrated in Figure I). For setting up the contrastive objective, we first (randomly)
We define separability as the minimum angular distance of phase values between a pair of prototypical points (centroids) that belong different objects
3

---

Decoolo Magnitudes
Deccoder
Addresses
Vaghhutet Addteet
Anchor Features
Features
Decoder
Anchor
X
Encoder
Address
Sample Positive Pair Features
Hagnitudes
Anchor Feature
Encoder
Features
Sample Negative Pairs Features
Sampling process of positive (green) and negative (red) pairs for anchor (purple) in the CıCAE model. The sampling process here is visualized only for the decoder output. Note that we contrast the encoder output X) corresponds to the patch of magnitude values and the feature corresponds to the phase values. See Contrastive Training of CAEs Section for more details.
Figure 1:
identical manner. Anchor address (the purple box marked with
Algorithm 1 Mining positive and negative pairs for single anchor for the contrastive objective. Scalar paramcters are kop - the number of candidates from which to sample one positive pair and
Mhottmm the number of candidates from which to sample (M 1) negative pairs.
Inputs: anchor
address @iccor_address
R24, addresses E RR N'l'xw'xe
features E R xw' xdd,
Params: Scalars kepp Wibottomo M.
1: addresses Flatten (addresses, dims=(0,1))
2: features <- Flatten (features, sims=(0,1))
3: distances CosineDistance
(addresses, (adaaiissttacceddadeessssss
4: fop_k_fetture +- features [argsort (distances, dim=0) [ kip ]]
5: bottom_m_ bottom___features +- features [argsort (distances, dim=0) [- Mbottm :]]
6: pos_pair_idx ~ Uniform [0, kip
sample 1 positive pair
neg_pair_idxs Uniform [o, mhbooom (
M samples without replacement
8: positive_pair + top_k_ fop_k_eeauures [pouress[pps___ parss_air_idee
9: negative_pairz bottom_ ___ w_w.tturreeeeeemeemmee (eatures(neg_ppaa [neg_pairrrrrreeeeeeeeeeeeeeeeee idxs]
10: return positive_pair, negative_pairs
sample "anchors" from of "datapoints" The main idea of contrastive learning "contrast" these anchors to their respective "positive" and "negative" examples in certain representation space. This requires define two representation spaces: one associated with the similarity measure to define positive/negative examples given an anchor, and another one on which we effectively apply the contrastive objective, itself defined certain distance function to minimized. We use the term addresses to refer to the representations used to measure similarity, and consequently extract positive and negative pairs w.r.t. the anchor. We use the term features refer the representations that are contrasted. As outlined earlier, the goal of the contrastive objective is to facilitate separability of phase values. is then a natural choice to use the phase components complex-ve complex-valued outputs as features and the magnitude components addresses in the contrastive loss. This results in angular distance be- tween phases (features) being modulated by the contrastive objective based on how (dis)similar their corresponding magnitude components (addresses) are. Since the magnitude components of complex- valued activations are used to reconstruct the image (Equation (7)), they capture requisite visual prop- erties of objects. In short, the contrastive objective increases decreases the angular distance of phase components of points (pixels/image regions) in relation to how (dis)similar their visual properties are.
Our contrastive learning method works follows. Let h', w', dr. NA. and M denote positive integers. Here we generically denote the dimension of the output of any CAE layer l/' w/ dr. This results in a set of l w/ "datapoints" of dimension de for our contrastive learning. From this set of datapoints, we randomly sample NA anchors. We denote this set of anchors matrix A IR Naxd, each anchor is thus denoted A E R& for all k E NA). Now by using

---

Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 2: Unsupervised object discovery on Tetrominoes, dSprites and CLEVR with CICAE. "Phase Rad." (col. 5) is the radial plot with the phase values from to radians. "Phase" (col. 6), are phase values (in radians) averaged over the output channels as a heatmap (colors correspond to those from "Phase Rad.") and "Magnitude" (col. 7) is the magnitude component of the outputs.
Algorithm extract positive and M negative examples for each anchor. We denote these examples by matrix P E R R2Mxd for each anchor & E {1,..., Na} arranged such that Pf E R4 the positive example and other rows P E JR for j E {2, .... M) are negative ones. Finally, contrastive loss is adaptation of the standard InfoNCE loss 44 which is defined as follows:
exp (d (Ax;
La
NA Σ log
E exp (d (As; (A); /T)
where NA is the number of anchors sampled for each input, d(xxi x;) refers the cosine distance between pair of vectors X;; X₁ Rdd and R5o is the softmax temperature.
We empirically observe that applying the contrastive loss on outputs of both encoder and decoder is better than applying it on only either one TTble We hypothesize that this is the case because utilizes both high-level, abstract and global features (on the encoder-side) as well as low-level and local visual cues (on the decoder-side) that better capture visual (dis)similarity between positive and negative pairs. We also observe that using magnitude components complex outputs both the encoder and decoder addresses for mining positive and negative pairs while using the phase components of complex-valle valued outputs as the features for the contrastive loss performs the best among all the other possible alternatives TTable 13). These ablations also support initial intuitions (described above) while designing the contrastive objective for improving separability in phase space.
Finally, the complete training objective function of CICAE is:
L Lmse + + La Cusee = ||x x
X
ly
where defines the loss for single input image x' E Rhxwx3 and Lisee is the standard recon- struction loss used by the CAE [36]. The reconstructed image complex-valued outputs of the decoder E Chxux3 by using its magnitude component. In practice, train all models by minimizing the training loss L over batch of images. The CAE baseline model and our proposed CAE+ variant are trained using only the reconstruction objective (i.e. 0) whereas our proposed CtCAE model trained using the complete training objective.
Rhxux3 is generated from the
4
Results
Here we provide our experimental results. We first describe details of the datasets, baseline models, training procedure and evaluation metrics. We then show results (always across seeds) on grouping of our CtCAE model compared to the baselines (CAE and our variant CAE++), separability in phase space, generalization capabilities w.r.t to number of objects seen at train/test time and ablation studies for each of our design choices. Finally, comment the limitations of our proposed method.

---

Table 1: MSE and ARI scores (mean standard deviation across seeds) for CAE, CAE++ and CíCAE models for Tetrominoes, dSprites and CLEVR on their respective full resolutions. For all datasets, CıCAE vastly outperforms CAE++ which in turn outperforms the CAE baseline. Results for 32x32 dSpriter bes and CLEVR are also provided, these follow closely the scores on the full resolutions. SlotAttention results are from Emami et [45]
Dataset
Model
MSE 4
ARI-FG 1
ARI-FULL
Tetrominoes
CAE
4.57c-2 1.08-3
0.00 0.00
0.12 0.02
(32x32)
CAE++
5,07c-5 + 2.80-5
0.78 = 0.07
0.84 = 0.01
CICAE
9.73e-5 4.64e-5
0.84 0.09
0.85 0.01
SlotAttention
0,99 = 0.00
dSprites (64x64)
CAE
8. 8.16e-3 + 2.54p-5
0.05 0.02
0.10 = 0.02
CAE++
1.60e-3 1.33e-3
0.51 = 0.08
0.54 = 0.14
CICAE
1.56e-3 1.58c-4
0.56 0.11
0.90 0.03
SlotAttention
0.91 = 0.01
CLEVR (96x96)
CAE
1.50c-3 + 4.53e-4
0.04 = 0.03
0.18 = 0.06
CAE++
2.41e-4 + 3.45e-
0.27 0.13
0.31 0.07
CICAE
3.39c-4 + 3.656-
0.54 = 0.02
0.68 = 0.08
SlotAttention
0.99
dSprites (32x32)
CAE
7.24e-3 1 8.45e-5
0.01
1 0.00
0.05 : 0.00
CAE+
8.67e-4 1.92
0.38
0.05
0.49
: 0.15
CICAE
1.10e-3 2.59e4
0.48
1 0.03
0.68
0.13
CLEVR
CAE
1.84e-3 1: 5.68
0.11 = 0.07
0.12 = 0.11
(32x32)
CAE++
4.04e-4 4.04e-4
0.22
0.10
0.30 0.18
CICAE
9.88e-4 1.42e-3
0.50 = 0.05
0.69 = 0.25
Datasets. We evaluate the models on three datasets from the Multi-Object datasets suite namely Tetrominoes, dSprites and CLEVR (Figure 2) used by prior work in object-centric learning [4 [5 45). For CLEVR, we use the filtered version 45] which consists images containing less than seven objects. For the main evaluation, we use the same image resolution as Emami et al. [45] i.e. 32x32 for Tetrominoes, 64x64 for dSpritee and 96x96 for CLEVR (a center crop of 192x192 that is then resized to 96x96). For computational reasons, we perform all ablations and analysis on 32x32 resolution. Performance of all models are ordered in the same way on 32x32 resolution as the original resolution (see Table 1), but with significant training and evaluation speed up. In Tetrominoes and dSprites the number of training images is 60K whereas in CLEVR is 50K. All three datasets have 320 test images which report all the evaluation metrics. For more details about the datasets and preprocessing, please refer to Appendix A
Models & Training Details. We compare our CtCAE model to the state-of-the-art synchrony-based method for unsupervised object discovery (CAE (36)) as well as our own improved version thereof (CAE++) introduced in [Section 3 For more details about the encoder and decoder architecture of all models see Appendix A We use the same architecture as CAE [36), except with increased number of convolution channels (same across all models). We train models for 50K steps on Tetrominoes and 100K steps on dSprites and CLEVR with Adam optimizer [46) with a constant learning rate of 4e-4, i.e., warmup schedules annealing (all hyperparameter details are given in Appendix A).
Evaluation Metrics. We use the same evaluation protocol as prior work 14 15
45
36 which
compares the grouping performance of models using the Adjusted Rand Index (ARI) report two variants of the ARI score, i.c., ARI-FG and ARI-FULL consistent with Löwe et al. [36]. ARI-FG measures the ARI score only for the foreground and ARI-FULL takes into account all pixels.
[47
48 We
Unsupervised Object Discovery. Table shows the performance of our CAE++, CICAE, and the baseline CAE (36] on Tetrominoes. dSprites and CLEVR. We first observe that the CAE baseline almost completely fails on all datasets as shown by very low ARI-FG and ARI-FULL scores. The MSE values in Table indicate that CAE even struggles to reconstruct these color

---

Tetrominoes
dSprites
CLEVR
GT Mask
GT Mask
GT Mask
Inpur
Input
Input
CAE
CAE++
CICAE
Mask
Phase Rad. Phase Img.
Mask
Phase Rad. Phase Img.
Mask
Phase Rad. Phase Img.
Figure 3: CAE, CAE++ and CtCAE comparison Tetrominoes (columns 1-3), dSprites (columns 4-6) and CLEVR (columns 7-9). First row: ground truth masks and input images.
Table 2: Quantifying the Separability through inter- and intra-cluster metrics of the phase space. For the inter-cluster metric, report both the minimum and mean across clusters.
Dataset
Model
Inter-cluster (min)
Inter-cluster (mean)
Intra-cluster
Tetrominoes
CAE++
0.14 i: 0.00
0.30 + 0.02
0.022 1: 0.010
CíCAE
0.15 0.01
0.31
0.03
0.020
dSprites
CAE++
0.13 0.05
0.51 0.05
0.034 0.007
CíCAE
0.13 0.03
0.39 0.10
0.027 0.009
CLEVR
CAE++
0.10 0.06
0.53
0.15
0.033
CíCAE
0.12 + 0.05
0,50 = 0.12
0.024 + 0.005
images. In contrast, CAE++ achieves significantly higher ARI scores, consistently across all three datasets; this demonstrates the impact of the architectural modifications we propose. However, on the most challenging CLEVR dataset, CAE++ still achieves relatively low ARI scores. Its contrastive learning-augmenteee augmented counterpart, CiCAE consistently outperforms CAE++ both in terms of ARI-FG and ARI-FULL metrics. Notably, CıCAR achieves the most challenging CLEVR dataset which highlights the benefits our contrastive method. All these results demonstrate that CtCAE is capable of object discovery (still far from perfect) all datasets which include color images and more than three objects per scene, unlike the exisiting state-of-the-art synchrony-band model, CAE.
than double the ARI scores of CAE++
Quantitative Evaluation of Separability. To gain further insights into why our contrastive method beneficial, quantitatively analyse the phase maps using two distance metrics: inter-cluster and intra-cluster distances. In fact, in all CAE-family of models, final object grouping is obtained through K-means clustering based on the phases the output of the decoder; each pixel assigned to cluster with the corresponding centroid. Inter-cluster distance measures the Fuclidean distance between centroids of each pair of clusters averaged over all such pairs. Larger inter-cluster distance allows for easier discriminability during clustering to obtain object assignments from phase maps. On the other hand, intra-cluster distance quantifies the "concentration" of points within cluster, and is computed as the average Euclidean distance between each point in the cluster and the cluster centroid. Smaller intra-cluster distance results an easier clustering task as the clusters are then more condensed. We compute these distance metrics on per-image basis before averaging over all samples the dataset. The results in Table 2 show that, the mean intra-cluster distance (last column) is smaller for CıCAE than CAE++ on two (dSprites and CLEVR) of the three datasets. Also, even though the average inter-cluster distance (fourth column) sometimes higher for CAE++, the minimum inter-cluster distance (third (alumnn-whie is more relevant metric for separability-

---

Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 4: CtCAE on CLEVR is able to infer more than four objects, although it sometimes makes
mistakes, such as specular effects or grouping together based on color (two yellow objects).
Table Object storage capacity: training on the full dSprites dataset and evaluating separately on subsets containing images with 2, 3, 4 or 5 objects. Analogously, training on the full CLEVR dataset and evaluating separately on subsets containing images with 3, 4, 5 or 6 objects.
ARI-FG
ARI-FULL
ARI-FG
ARI-FULL
ARI-FG
ARI-FULL
ARI-FG
ARI-FULL
dSprites
2 objects
objects
objects
5 objects
CAE++
0.29 0.07
0.58 + 2015
0.39 0.6
0.53 0.15
0.43 + on4
0.45 +
0.45 + ons 0.48 : 0.04
0.44 +a
CiCAE
0.45 + 0.11
0.76 +: a.m
0.66
0.72 = 0.11
0.48 : 0,44
0.65 0.13
0.61 + a.a
CLEVR
3 objects
objects
5 objects
6 objects
CAE++
0.32
0.06
0.35 1 0.29
0.05
0.32
0.26
0.31 0.04
0.26
0.32 1. 0.04
0.27 0.18
CICAE
0.53 0.07
0.71 0.20
0.06
0.70 0.21
0.47 0.07
0.21
0.45
0.06
0.68 0.20
larger for CICAE. This confirms that compared to CAE++, CíCAE tends to have better phase map properties for object grouping, as is originally motivated by our contrastive method.
Object Storage Capacity. Löwe et al. [36] note that the performance of CAE sharply decreases for images with more than 3 objects. We report the performance of CICAE and CAE++ subsets of the test set split by the number of objects, to measure how their grouping performance changes w.r.t. the number of objects. In dSpri tes the images contain 3, 4 or 5 objects, CLEVR 3, 4. 5 objects. Note that the models are trained on the entire training split of the respective datasets. TTall 3 shows that both methods perform well images containing drop much on images with 4 or more objects). We also observe that the CíCAE consistently maintains significant lead over CAE in terms of ARI scores across different numbers of objects. In another set of experiments (see Appendix B) we also show that CıCAF generalizes well to more objects (e.g. 5 6) when trained only subset of images containing less than this number of objects.
than 3 objects (their performance does not
Ablation Architectural Modifications. Table 4 shows an ablation study on the proposed architectural modifications on Tetrominoes (for similar findings on other datasets, see Appendix B.2). We observe that the sigmoid activation on the output layer the decoder significantly impedes learning on color datasets. A significant per- formance jump is also observed when replac- ing transposed convolution layers (36] with
Table 4: Architectural ablations on Tetrominoes.
Model
ARI-FG
ARI-FULL
CAE
0.00 + 0.00
0.12 = 0.02
CAE- 1x1 conv)
0.00 + 0.00
0.00 = 0.00
CAE- (four sigmoid)
0.12 a:
0.35
CAB-Iransp.tupsamy
0.10 + 0.21
0.10 022
CAE+ CíCAE
0.78 a: 0.07
0.84 0.01
0.84 1: 0.85
convolution and upsample layers. By applying all these modifications, we obtain our CAE++ model
that results in significantly better ARI scores on all datasets, therefore supporting our design choices.
Ablation on Feature Layers to Contrast. In CíCAE we contrast feature vectors both at the output of the encoder and the output of the decoder. Tabll 5ljustifies this choice; this default setting (Enc+Dec) outperforms both other options where we apply the constrastive loss either only in the

---

encoder or the decoder output. We hypothesize that this is because these two contrastive strategies are complementary: one uses low-level cues (dec) and the other high-level abstract features (enc).
Qualitative Evaluation of Grouping. Finally, we conduct some qualitative analyses of both successful and failed grouping modes shown by CAE++ and CíCAE models through visual inspection of representa- tive samples. In In/Figure 3] Tetrominoes (columns 1-2), we observe that CICAE (row 4) exhibits better grouping on scenes with multiple objects of the same color than CAE++ (row 3). This is reflected in the radial phase plots (column 2) which show better separability for CICAE than CAE++. Further, on dSpri tes (rows 3-4, columns 4-5) and CLEVR (rows 3-4, columns 7-8), CICAE handles the increased number of objects more gracefully while CAE++ struggles and groups several of them together. For the failure cases.[Figure shows an example where CICAE still has some difficulties in segregating objects of the same color (row 2, yellow cube and ball) (also observed sometimes on dSprites, see Figure 14). Further, we observe how the specular highlights metallic objects (purple ball in row 1 and yellow ball in row 2) form a separate uub-part from the object (additional grouping examples in in|Appendix C).
Table 5: Contrastive loss ablation for CtCAE on CLEVR.
Model
ARI-FG f
ARI-FULL 1
Enc-only
0.21 0.11
0.29 0.15
Dec-only
0.38 0.17
0.69 0.18
Enc+Dec
0,50 = 0.05
0.69 = 0.25
Discussion on the Evaluation Protocol. The reviewers raised some concerns about the validity of our evaluation protoco The main point of contention was that the thresholding applied before clustering the phase values may lead to a trivial separation of objects in RGB images based on their color. While it is true that in our case certain color information may contribute to the separation process, a trivial separation of objects based on solely their color cannot happen since CiCAE largely outperforms CAE++ (see Table 1) despite both having near perfect reconstruction losses and with the same evaluation protocol. This indicates that this potential issue only has a marginal effect in practice in our case and separation in phase space learned by the model is still crucial. In fact, "pure" RGB color tones (which allow such trivial separation) rarely occur in the datasets aSprites and CLEVR) used here. The percentage of pixels with magnitude less than 0.1 are 0.44%, 0.88% and 9 9.25% in CLEVR, dSprites and Tetrominoes respectively. While 9.25% in Tetrominoes is not marginal, we observe that this does not pose issue in practice, as many examples in Tetrominoes where two or three blocks with the same color are separated by CAE++CICAE (e.g. three yellow blocks in Figure 3 or two magenta blocks in Figure 8).
To support our argument that using a threshold of 0.1 has no effect on the findings and conclusions, conduct additional evaluations with a threshold of 0 and no threshold at all. From Table 9 (see Appendix B) we can see that using a threshold of 0 hardly change the ARI scores at all or in some cases it even improves the scores slightly. Using threshold results in marginally improved ART scores on CLEVR, near-identical scores on dSprites but worse scores on Tetrominoes (CICAE still widely outperforms CAE++). However, this is not due to a drawback in the evaluation or model. Instead, it stems from an inherent difficulty with estimating the phase of a zero-magnitude complex number. The evaluation clusters pixels based on their phase values, and if complex number has a magnitude of exactly zero (or very small, e.g., order of le 1)= the phase estimation ill-defined and will inherently be random as was also previously noted by Löwe et al. [36]. In fact. filtering those pixels in the evaluation is crucial in their work (36) as they largely used datasets with pure black background (exact zero pixel value). Evaluation of discrete object assignments from continuous phase outputs in a model-agnostic way remains an open research question.
5 Related Work
Slot-based binding. A wide range of unsupervised models have been introduced to perform perceptual grouping summarized well by Greff et al. [30]. They categorize models based on the segregation (routing) mechanism used to break the symmetry in representations and infer latent representations (i.e. slots). Models that use "instance slots" cast the routing problem as inference in mixture model whose solution is given by amortized variational inference [8] [4], Expectation- Maximization [10] or other approx (Soft K-means) thereof [15). While others [9. 12 49
that use "sequential slots" solve the routing problem by imposing an ordering across time. These
Discussion thread: http:///w//pp/.eee.////////////a

---

models use recurrent neural networks and an attention mechanism to route information about a different object into the same slot at every timestep. Some models [ 17 combine the above strategies and use recurrent attention only for routing but not for inferring slots. Other models break the representational symmetry based on spatial coordinates [50] [6] ("spatial slots") or based on specific object types [5T] ("category slots"). All these models still maintain the "separation" of representations only at one latent layer (slot-level) but continue to "entangle" them at other layers.
Synchrony- bbaed binding. Synchrony- baased models use complex-valued activations to implement binding by relying on their constructive or destructive phase interference phenomena. This class of models have been sparsely explored with only few prior works that implement this conceptual design for object binding [38/40] 36). These methods differ based on whether they employ both activations with real-valued weights and a gating mechanism 40 36]. They also differ in their reliance on explicit supervision for
complex-valued weights and activations [38] 39 or complex-valus
grouping [8 or not [39 36 Synchrony- based models contrast to slot-based ones maintain
the "separation" of representations throughout the network in the phase components of their complex- valued activations. However, none of these prior methods can group objects in color images with up to 6 objects or visual realism of multi-object benchmarks in fully unsupervised manner unlike ours. Concurrent work [52] extends CAE by introducing new feature dimensions ("rotating features"; RF) to the complex-valued activations. However, RF cannot be directly compared to CiCAE as it relies on either depth masks to get instance grouping on simple colored Shapes or features from powerful vision backbones (DINO [53) to get largely semantic grouping on real- world images such as multiple instances of cows/trains or bicycles in their Figures 2 and 20 respectively.
Binding in the brain. The temporal correlation hypothesis posits that the mammalian brain binds together information emitted from groups of neurons that fire synchronously. According to this theory (5), biological neurons transmit information in two ways through their spikes. The spike ampli- tude indicates the strength of presence of feature while relative time between spikes indicates which neuronal responses need to bound together during further processing. It also suggests candidate rhyth- mic cycles in the brain such as Gamma that could play this role in binding [50] 571. Synchrony- based models functionally implement the same coding scheme [58 60 using complex- -aauudd activations where the relative phase plays the role of relative time between spikes. This abstracts away all aspects of the spiking neuron model to allow easier reproduettii reproduction on digital hardware.
Contrastive learning with object-centric models. Contrastive learning for object-centric representations has not been extensively explored, with a few notable exceptions. The first method [61] works only on toy images of up to objects on black background while the second [62] shows results on more complex data, but requires complex hand-crafted data augmentation techniques to contrast samples across batch. Our method samples positive and negative pairs within a single image and does not require any data augmentation. Most importantly, unlike ours, these models still use slots and iitentiionbbsed routing, and thereby inherit all of its conceptual limitations. Lastly, ODIN (63) alternately refines the segmentation masks and representation of objects using two networks that are jointly optimized by a contrastive objective that maximizes the similarity between different views of the same object, while minimizing the similarity between different objects. To obtain the segmentation masks for objects, they simply spatially group the features using K-means. Their method relies on careful data augmentation techniques such as local or global crops, viewpoint changes, color perturbations etc. which are applied to one object.
6 Conclusion
We propose several architectural improvements and a novel contrastive learning method to address limitations of the current state-of-the-at synchrony-based model for object binding, the complex- valued autoencoder (CAE (36)). Our improved architecture, CAE++, is the first synchrony- model capable of dealing with color images (e.g., Tetrominoes). Our contrastive learning method further boosts CAE++ by improving its phase separation process. The resulting model, CICAE, largely outperforms CAE++ on the rather challenging CLEVR and dSprites datasets. Admittedly, our synchrony-based models still lag behind the state-of-the-ard slot-based models [55 18] but this is
to be expected, as research on modern synchrony-bane models still in its infancy. We hope our work will inspire the community invest greater effort into such promising models.
10

---

Acknowledgments. We thank Sindy Löwe and Michael C. Mozer for insightful discussions and valuable feedback. This research was funded by Swiss National Science Foundation grant: 200021_ 110021_192356, project NEUSYM and the ERC Advanced grant no: 742870, AlgoRNN. This work was also supported by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID s1205 and d123. We also thank NVIDIA Corporation for donating DGX machines as part of the Pioneers of Al Research Award.
References
[1] Anne Treisman. The binding problem. Current opinion in neurobiology, 6(2):171-178, 1996.
[2] Adina L Roskies. The binding problem. Neuron, 24(1):7-9, 1999.
[3] Elizabeth Spelke. Principles of object perception. Cognitive Science, 14(1):29-56, 1990.
[4] Max Wertheimer. Untersuchungen zur lehre von der gestalt. ii, Psychologische Forschung, 4(1): 301-350, 1923.
[5] Kurt Koffka. Principles of gestalt psychology. Philosophy and Scientific Method, 32(8), 1935.
[6] Wolfgang Köhler. Gestalt psychology. Psychologische Forschung, 31(1), 1967.
[7] Daniel Kahneman, Anne Treisman, and Brian Gibbs. The reviewing of object files: Object- specific integration of information. Cognitive psychology, 24(2):155-219) 1992.
[8] Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems, volume 29, 2016.
[9] SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pages 3225-32 3233,, 2016.
[10] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 6691-6701, Long Beach, CA, USA, December 2017.
[11] Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In International Conference on Learning Representations, 2018.
[12] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, pages 8606-8616, 2018.
[13] Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint 1rXiv:1901.1 2019.
[14] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In Proc. Int. Conf. Machine Learning (ICML), pages 2424-2433, Long Beach, CA, USA, June 2019.
[15] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2020.
[16] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng. Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised orject-oriented scene representation via spatial attention and decomposition. In International Conference Learning Representations, 2020.

---

[17] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Genera- tive scene inference and sampling with object-centric latent representations. In International Conference Learning Representations, 2020.
[18] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALLE learns to compose. In Int. Conf. on Learning Representations (ICLR), Virtual only, April 2022.
[19] Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. In Advances in Neural Information Processing Systems, 2022.
[20] David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matthew Botvinick. Atten- tion over learned object embeddings enables complex visual reasoning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[21] Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsuper- vised visual dynamics simulation with object-centric models. In The Eleventh International Conference Leurning Representations, 2023.
[22] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2019.
[23] Tejas Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems, 32, 2019.
[24] Anand Gopalakrishnan, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Unsupervised object keypoint learning using local spatial predictability. In International Conference on Learning Representations, 2021.
[25] Aleksandar Stanié, Yujin Tang, David Ha, and Jürgen Schmidhuber. Learning to generalize with object-centric agents in the open world survival game crafter. ArXiv, abs/2208.0333 00374, 2022.
[26] Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object-centric representations for reinforcement learning. arXiv preprint (rXiv:2302.04419 2023.
[27] Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In 2021 IEEE international conference on robotics and automation (ICRA), pages
IEEE, 2021.
[28] Yizhe Wu, Oiwi Parker Jones, Martin Engelcke, and Ingmar Posner. Apex: Unsupervised, object-centric scene segmentation and tracking for robot manipulation. In 2021 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages
2021.
[29] Mohit Sharma and Oliver Kroemer. Generalizing object-centric task-axes controllers using keypoints. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 7548-7554, 2021.
[30] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. Preprint arX5v:20/2.05208, 2020.
[31] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez. Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[32] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International conference on machine learning, pages 1263-1272, 2017. 2017.
12

---

[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 3998-6008, December 2017.
[34] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4(1 4(1):131-19, 1992.
[35] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Proc. Int. Conf. on Machine Learning (ICML), 2021.
[36] Sindy Löwe, Phillip Lippe, Maja Rudolph, and Max Welling. Complex-value autoencoders for object discovery. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
[37] Nicholas Watters, Loic Matthey, Christopher Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in VAËs. In Learning from Limited Labeled Data (LLD) Workshop, ICLR, New Orlcans, LA, USA, May 2019.
[38] Michael C Mozer, Richard Zemel, and Marlene Behrmann. Learning to segment images using dynamic feature binding. Advances in Neural Information Processing Systems, 4, 1991.
[39] Michael C Mozer. A principle for unsupervised hierarchical decomposition of visual scenes. In Advances in Neural Information Processing Systems, volume 11, 1998.
[40] David P Reichert and Thomas Serre. Neuronal synchrony in complex-valee valued deep networks. In International Conference Learning Representations, 2014.
[41] Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. https:/giimbb.ccmmdeemm////aaa/ object-datasets/, 2019.
[42] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, K Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: a scalable dataset generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[43] Michael Gutmann and Aapo Hyvärinen. Noise ciise-oontrattvve estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statisties, pages 297-304, 2010.
[44] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 00000007.77...... 2018
[45] Patrick Emami. Pan He, Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized inference for learning symmetric and disentangled multi-object representations. In International Conference Machine Learning, pages 2970-2981 PMLR, 2021.
[46] Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Interna- tional Conference for Learning Representations, 2015.
[47] William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846-85)) 1971.
[48] Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2:193-218. 1985.
[49] Aleksandar Stanié and Jürgen Schmidhuber. R-sqair: Relational sequential attend, infer, repeat. ArXiv, abs/1910.05231, 2019.
13

---

[50] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with con- volutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence. 2019.
[51] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing. In International Conference on Learning Representations, 2018.
[52] Sindy Löwe, Phillip Lippe, Francesco Locatello, and Max Welling. Rotating features for object discovery. arXiv preprint arXiv:2306.0000 00600, 2023.
[53] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[54] Christoph von der Malsburg. Binding in models of perception and brain function. Current opinion in neurobiology, 5(4):520-526, 1995.
[55] Wolf Singer and Charles M Gray. Visual feature integration and the temporal correlation hypothesis. Anmual review of neuroscience, 18(1 (88(1)555--55)) 5866, 1995.
[56] John G.R Jefferys, Roger D Traub, and Miles A Whittington. Neuronal networks for induced '40 hz' rhythms. Trendy in Neurosciences, 1955)200-2008 1996.
[57] Pascal Fries, Danko Nikolić, and Wolf Singer. The gamma cycle. Trends in neurosciences, 30 (7):309-316, 2007.
[58) Christoph von der Malsburg and Werner Schneider. A neural cocktail-party processor. Biological cybernetics, 54(1):29-4)) 1986.
[59] Christoph von der Malsburg and Joachim Buhmann. Sensory segmentation with coupled neural oscillators. Biological cybernetics, 67(3):233-22 242, 1992.
[60] Andreas K Engel, Peter König, Andreas K Kreiter, Thomas B Schillen, and Wolf Singer. Temporal coding in the visual cortex: new vistas on integration in the nervous system. Trends in neurosciences, 15(6):218-226, 1992.
[61] Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. arXiv preprint arXiv:2011.10287, 2020.
[62] Federico Baldassarre and Hossein Azizpour. Towards sepf-supervised learning of global and object-centric representations. arXiv preprint arXiv:2203.059997, 2022.
[63] Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. In European Conference on Computer Vision, pages 123-143, 2022.
14

---

A Experimental Details
Datasets. We evaluate all models (i.e., CAE, CAE++ and CICAE) on subset of Multi-Object dataset [41] suite. We use three datasets: Tetrominoes which consists of colored tetris blocks on black background, dSpri iee with colored sprites of various shapes like heart, square, oval, etc., on grayscale background, and lastly, CLEVR, a dataset from a synthetic 3D environment, For CLEVR, we use the filtered version
[45] which consists of images containing less than seven objects sometimes referred to as CLEVR6 as in Locatello et al. [[S]. We normalize all input RGB images to have pixel values the range [0, consistent with prior work (36].
Models. Table shows the architecture specifications such as number of layers, kernel sizes, stride lengths, number of filter channels, normalization layers, activations, etc., for the convolutional neural networks used by the Encoder and Decoder modules in CAE, CAE+ and CICAE.
Table 6: Encoder and Decoder architecture specifications for CAE, CAE++ and CtCAE models.
Encoder
x 3 conv, 128 channels, stride 2, ReLU, BatchNorm
3 conv, 128 channels, stride 1, ReLU, BatchNorm
3 X 3 conv, 256 channels, stride 2, ReLU, BatchNorm
3 conv, 256 channels, stride 1, ReLU, BatchNorm
3 x 3 conv, 256 channels, stride 2, ReLU, BatchNorm
For 64x64 and 96x inputs, additional encoder layers: x conv, 256 channels, stride I, ReLU, BatchNorm
3 x 3 conv, 256 channels, stride 2, ReLU, BatchNorm
Linear Layer
Flatten
Linear, 256 ReI.U units, LayerNorm
Decoder
For 64 and 96x 96 inputs, additional decoder layers: Bilinear upsample x2 3 conv, 256 channels, stride 1, ReLU, BatchNorm
3 x 3 conv, 256 channels, stride 1, ReLU, BatchNorm
Bilinear upsample x2
conv, 256 channels, stride 1, ReLU, BatchNorm 3 3 conv, 256 channels, stride 1, ReLU, BatchNorm Bilinear upsample x2
3 conv, 256 channels, stride 1, ReLU, BatchNorm 3 conv, 128 channels, stride 1, ReLU, BatchNorm
Bilinear upsample x2
3 conv, 128 channels, stride 1, ReLU, BatchNorm
Output Layer (CAE only)
conv, 3 channels, stride 1, sigmoid
Training Details. Table 7 and Table 8 show the hyperparameter configurations used to train CAE/CAE+ and to contrastively train the CtCAE model(s) respectively.
Computational Efficiency. We report the training and inference time (wall-clock) for our models across the various image resolutions of the multi-object datasets. First, we find that inference time(s) similar for all models, and it only depends on the image resolution and number of ground truth clusters in the input image. Inference for the test set containing 320 images of 32x32 resolution takes 25 seconds, for 64x64 images takes 65 seconds and for the 96x96 images it takes 105 seconds. Training time(s) on the other hand differs both across models and image resolutions. We report all
15

---

Table 7: General training hyperparameters.
Hyperparameter
Tetrominoes
dSprites
CLEVR
Training Steps
50'000
100'000
100'000
Batch size
64
64
64
Learning rate
4e-4
4e-4
4e-4
Table 8: Contrastive learning hyperparameters for the CtCAE model.
Common Hyperparameters
Loss coefficient (in total loss sum)
le-4
Temperature
0.05
Contrastive Learning Addresses
Magnitude
Contrastive Learning Features
Phase
Encoder Hyperparameters
32x32
64x64
96x96
Number of anchors
4
4
Number of positive pairs
Top-K select positive pairs from
1
Number of negative pairs
2
2
Bottom-M to select negative pairs from
18
Decoder Hyperparameters
32x32
64x64
96x96
Patch Size
Number of anchors
100
100
100
Number of positive pairs
Top-K select positive pairs from
5
Number of negative pairs
100
100
100
Bottom-M to select negative pairs from
500
500
500
training time(s) using single Nvidia V100 GPU. CAE and CAE++ have similar training times: 32x32 images training for 100k steps takes 3.2 hours, for 64x64 images takes 4.8 hours, and for 96x96 images it takes 7 hours. CtCAE the other hand takes 5 hours, 9.2 hours and 10.2 hours to train on dataset of 32x32, 64x64 and 96x96 images respectively. To reproduce all the results/tables (mean and std-dev across seeds) reported in this work we estimate the compute requirement to be 840 GPU hours in total. Further, we estimate that the total compute used this project is approximately 5-10x more, including experiments with preliminary prototypes.
Object Assignments from Phase Maps. We describe the process of extracting discrete-valued object assignments from continuous-valued phase components of decoder outputs y Chxux3 First, the phase components of decoder outputs are mapped onto a unit circle and those phase values masked out whose magnitudes below the threshold value of 0.1. After this normalization and filtering step. the resultant phase values are converted from polar to Cartesian form on a per-channel basis. Finally, apply K-means clustering where the number of clusters K parameter is retrieved from the ground-truth segmentation mask for each image. This extraction methodology of object assignments from output phase maps is consistent with Löwe al. (36]
B Additional results
We show some additional experimental results: generalization capabilities CtCAE w.r.t the number of objects, and ablation studies various design choices made in CAE++ and CiCAE models.
16

---

Table Grouping results for CAE++ and CíCAE models with different threshold values applied to post process the continuous output phase maps.
Dataset
Model
Threshold=O.1
Threshold=0.0
No threshold
ARI-FG f
ARI-FULL
ARI-PG 1
ARI-FULL f
ARI-PG f
ARI-FULL
Tetrominoes
CAE++
0.78 +: 00?
0.84 i 0.01
0.77 0.07
0.79 0.00
0.54 a: 0.09
0.21 +: 0.05
CICAE
0.84 + 0.99
0.85 001
0.86 0.5
0.82 0000
0.67 2
0.26 0.09
dSprites
CAE++
0.38 +
0.49 +
0.38
0.49
0.37
0.49
CICAE
0.48 1 0.03
0.68 0,13
0,46 0.07
0.69
0.47 0.06
0.69 1 0.10
CLEVR
CAE++
0.22 1 0.10
0.30 0.18
0.33 0.04
0.32 0.25
0.34 0.04
0.32 0.25
CICAE
0.50 +: 0.0
0.69 0.25
0.52 0.05
0.69 0.20
0.52 0.06
0.72 0.21
Table 10: Generalization evaluation the CLEVR dataset. Training only a subset, but evaluating on all possible subsets containing images with 3, 4, 5 or 6 objects.
Evaluation ARI-FG ARI-FULL
Training Subset: or less objects
3 objects
0.39 = 0.20
0.53 0.30
4 objects
0.41 0.18
0.55 0.29
5 objects
0.38 0.15
0.54 0.28
objects
0.38 = 0.13
0.55 0.24
All images
0.39 = 0.04
0.54 0.28
Training Subset: or less objects
3 objects
0.49
0.69
4 objects
0.50 0.03
0.66 0.24
5 objects
0.46 0.02
0.65 0.24
6 objects
0.45 0.03
0,64 0.23
All images
0.48 0.02
0.66 0.24
B.1 Generalization to Higher Number of Objects
Here we evaluate generalization capabilities of CíCAE by training only on a subset of images containing less than certain number of objects, CLEVR. We have cases-traie cases-training with either up to 4 or up to objects while the original trainig split contain between and objects. The results in Table show that CíCAE generalizes well. The performance drops only marginally when trained on up to objects and tested on and 6 objects, or when trained on up to objects and tested objects. We also observe that training on 5 or less objects also consistently improves ART scores for every subset of less than 5 objects. We suspect that the reason for this, apart from having more training data, is that the network has more pressure separate objects in the phase space when observing more objects, which in turn also helps for images with smaller number of objects.
subsets
B.2 Architecture Modifications
Table I1 shows the results from the ablation study that measures the effect of applying each of our proposed architectural modifications cumulatively to finally result in the CAE++ model. Across all datasets, we consistently observe that, starting from the vanilla CAE baseline which completely fails, grouping performance gradually improves more components of the CAE++ model is added. Lastly, our proposed contrastive objective used to train CiCAE further improves CAE++ across all 3 datasets.
B.3 Contrastive Learning Ablations
Table 12 shows an ablation study for the choice of layer(s) to which we apply our contrastive learning method. Please note that all variants here always use magnitude components of complex-valued activations addresses and phase components as features to contrast. We observe that the variant
17

---

Table 11: Grouping metrics achieved by various model variants from our proposed architectural modifications and resulting finally in the CAE++ model. Extends the results of Table 4 to all multi-object datasets.
Dataset
Model
ARI-FG 1
ARI-FULL 1
Tetrominoes
CAE
0.00 0.00
0.12 0.02
CAE-(fout 1x1 conv)
0.00 = 0.00
0.00 = 0.00
CAE-(fout sigmoid)
0.12 0.12
0.35
CAE-transp.+uusanme
0.10 = 0.21
0.10 = 0.22
CAE++ (above combined)
0.78 0.07
0.84 0.01
CICAE
0.84 = 0.09
0.85 0.01
dSprites
CAE
0,02 = 0.00
0.07 0.01
CAE-(fout 1x1 conv)
0,06 = 0.01
0.07
0.07
CAE-(four sigmoid)
0.02 + 0.01
0.07 0.01
SAA--tranpp+upsaamee
0.19 0.02
0.10 0.04
CAE++ (above combined)
0.38 = 0.05
0.49 0.15
CICAE
0.48 = 0.03
0.68 0.13
CLEVR
CAE
0.09 = 0.05
0.08 0.06
CAB-(fuut 1x1 conv)
0.05 0.01
0.06 0.011
CAE-(fau sigmoid)
0.06 1: 0.02
0.01 0.02
CAE-transp.+upsame
0.19 0.07
0.10 0.11
CAE++ (above combined)
0.22 0.10
0.30 0.18
CICAE
0.50
0.69
which applies the contrastive objective to both the outputs the encoder and decoder ('enc+dec') outperforms the others which apply it only to either one ('enc-only' or 'dec-only'). This behavior consistent across Tetrominoee noes, dSprites and CLEVR. We hypothesize that this is because these two contrastive strategies are complementary: one is on low-level cues (decoder) and the other one based on high- abstract features (encoder).
shows an ablation study for the choice using magnitude or phase as addresses (and the other one features) in contrastive learning method. As we apply contrastive learning both the decoder and encoder outputs, we can make this decision independently for each output (resulting in four possible combinations). We observe that the variant which uses magnitude for both the encoder and decoder ('mg+mg') outperforms other variants across all 3 multi-object datasets. These ablations support our initial intuitions when designing our contrastive objective.
Table 13
Table 12:
Grouping metrics achieved by CICAE model variants that apply the contrastive loss on output features from only the encoder (enc-only) or only the decoder (dec-only) or both (enc+dec) for all 3 multi-object datasets. Extends the results of Table to all 3 multi-object datasets.
Dataset
Model
ARI-FG 1
ARI-FULL
Tetrominoes
CíCAE (enc-only)
0.81
0.06
0.85 0.01
CíCAE (dec-only)
0.74
0.04
0.86 0.00
CICAE (enc+dec)
0.84 0.09
0.85 0.01
dSprites
CíCAE (enc-only)
0.40 0.06
0.58 0.05
CICAE (dec-only)
0,48 0.05
0.72 0,07
C1CAE (enc+dec)
0.48 0.03
0.68 0.13
CLEVR
CICAE (enc-only)
0.21 + 0.11
0.29 + 0.15
CíCAE (dec-only)
0.38 0.17
0.69 0.18
CíCAF (enc+dee)
0.50 0.05
0.69 0.25
18

---

Table 13: Grouping metrics on multi-object datasets achieved by CiCAE model variants that use either magnitude or phase components of the encoder/decoder outputs the addresses for contrastive learning. For example, 'mg+ph' means that magnitude components used as addresses of the encoder outputs and phase components used as addresses of the decoder outputs (conversely, the phase components of the encoder outputs are used as features and the magnitude components of the decoder outputs are used features).
Dataset
Model
ARI-FG
ARI-FULL
Tetrominoes
CiCAE w/ mg+mg
0.84 + 0.09
0.85 + 0.01
CICAE w/ ph+ph
0.85 0.05
0.84 + 0.01
C(CAE w/ mg+ph
0.86 + 0.03
0.85 + 0.02
CíCAE w/ ph+mg
0.77
0.85 1 0.001
dSprites
CíCAE w/ mg+mg
0.48 +
0.68 + 0.13
CiCAE w/ ph+ph
0.38 + 0.08
0.42 +: 0.16
CíCAE w/ mg+ph
0.36 + 0.07
0.57 + 0.13
CiCAE w/ ph+mg
0.45 + 0.05
0.67 + 0.18
CLEVR
CíCAE w/ mg+mg
0.50
0.69 +:
CíCAE w/ ph+ph
0.22 + 0.06
0.22 + 0.09
CıCAE w/ mg+ph
0.18 1:
0.28 0.155
CiCAE w/ ph+mg
0.40 1 0.15
0.65 0.199
C Additional Visualizations
We show additional visualization samples of groupings from CAE, CAE++ and CíCAE on Tetrominoes, dSprites and CLEVR, and qualitatively highlight both grouping failures and suc- cesses of CAE, CAE++ and CICAE.
CAE
CAE++
CtcAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 5: Visualizations for CAE (row 1), CAE++ (row 2) and CtCAE (row 3) on the Tetrominoes
dataset example from Figure 3 We see that CiCAE achieves near perfect grouping (column 4) and better separability phase space (column 5) compared our improved CAE++ variant which shows significant grouping interference (parts of all three objects modelled by same group, e.g., orange cluster). We also observe that the baseline CAE completely fails reconstruct or group the image.
19

---

CAE
CAE++
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 6: Visualizations for CAE (row 1), CAE++ (row 2) and CtCAE (row 3) the dSprit ies dataset example from Fogure 3 see that CiCAB successfully separates the scene objects into clusters while CAE++ mistakenly groups scene objects into cluster, i.e., the blue cluster (column 4). Again, the baseline CAE fails to reconstruct (no color) or group the image.
CAE
CAE++
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 7: Visualizations for CAE (row 1), CAE++ (row 2) and CiCAE (row 3) on the CLEVR dataset example from Fiiuure 3 We see that the CtCAE in this example is able separate the 4 scene objects into 4 distinct clusters while our improved CAE++ fails (column 4). This good separation in phase space is reflected in the continuous phase maps (column 6) where the pixel colors in the phase maps are representative their values/differences when comparing CíCAE and CAE++. Finally, we see that although the CAE is able to reasonably reconstruct the image shows very poor grouping the scene objects.
20

---

CAE
CAE++
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
CtcAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure Visualization on grouping performance for CAE, CAE++ and CICAE on images containing same colored objects on Tetrominoes. CAE struggles to even reconstruct the images, whereas CAE++ often groups objects with the same colour together. CtCAE on the other hand has no problem separating objects of the same colour on Tetrominoes.
21

---

CAE
CAE++
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
011
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CtcAE CAE++
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 9: Visualization on grouping performance for CAE, CAE++ and CICAE on Tetrominoes.
CiCAE shows a larger spread in phase values for different clusters (column 5) compared to CAE. We hypothesize that this facilitates CıCAE to perform better grouping the scenario of multiple object instances with the same color.
22

---

CAE
CAE++
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
CtcAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 10: Visualization grouping performance for CAE, CAE++ and C(CAE images containing similar colored objects on dSprited tes. CAE has very poor grouping capability, whereas CAE++ often groups objects with the same colour together For example, purple ellipse and square in the first example (top panel) or red and brown hearts grouped together in the last example (bottom panel) are grouped together. CtCAE the other hand has significantly better performance, grouping correctly objects with the same (or similar) colors, except the first example (top panel) where it groups the pink square and ellipse together.
23

---

CAE
CAE++
CiCAE
Input
Recon,
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
ctcAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 11: Performance for CAE, CAE++ and CíCAE on images containing many objects on dSprites. CAE shows poor grouping performance, as noted previously. CiCAE has no issues grouping up to objects the same image, whereas CAE++ struggles, often grouping multiple objects (typically of same or similar colors) together into the same cluster.
24

---

CAE
CAE++
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
CtcAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
CtcAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 12: Visualization on grouping performance for CAE, CAE++ and CíCAE on CLEVR dataset. We are particularly interested in the phase space spread (column 5) here. CAE has very large spread phase values (column 5), but almost random object discovery performance, often grouping all objects together. But if we see the actual phase values assigned to objects (column 6), we observe that CAE tends assign very similar phase values (e.g. all object phases' are in shades of magenta seen in column 6, rows 1, 4, and 7) to regions belonging to different objects whereas the corresponding maps show better separation in CAE++ and much superior CtCAE, This explains better grouping performance (column 4) shown by CAE++ and CICAE over the vanilla CAE.
25

---

CAE
CAE++
CtCAE
G
Input
Recon,
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
CtcAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
CtcAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 13: Visualization on grouping performance for CAE, CAE++ and CiCAE on samples with
4. and 6 objects from the CLEVR dataset. CAE consistently shows poor object discovery, often grouping all objects together. CAE++ improves grouping substantially over the CAE baseline, but also struggles in certain scenarios. For example, in the scene with objects it groups two different objects together, whereas for scenes with and objects on two samples here we see that it wrongly groups two objects together. CıCAE improves on these grouping failures of CAE++. CíCAE correctly discovers all objects the scene with 4 and 5 objects, even though it makes error grouping two objects together in the hardest example scene containing 6 objects.
26

---

CAE
CAE++
CtCAE
Input
Recon,
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CAE++
CtCAE
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
CAE
CtcAE CAE++
Input
Recon.
GT Mask
Mask
Phase Rad.
Phase
Magnitude
Figure 14: One of the limitations (failure modes) we observe that in some images containing many objects, with some of them having the same (or similar) colors, our CICAE model groups two such objects into the same cluster. Even in this example, CtCAE still clearly outperforms CAE++, which in turn outperforms the CAE baseline.
27


Reviews:
[{"review_id": "5BeEERfCvI", "reviewer": "Reviewer_QpC7", "summary": "This paper proposes several improvements over the Complex AutoEncoder (CAE) to scale it to more complex multi-object datasets. The CAE is a synchrony-based model that learns object-centric representations by injecting complex-valued activations into every layer of its architecture, but it is only applicable to grayscale images with up to 3 objects. In this paper, two modifications to the original architecture and a novel auxiliary loss function are proposed that make the CAE applicable to RGB datasets with more than 3 objects.", "questions": "1) As described above, I believe the main weakness of this paper is its evaluation procedure. If the authors could come up with a solution to the described problem or convince me that this problem does not apply to their setup, I'd be happy to increase my score. \n\n\nOther questions:\n\n2) Why do you not apply the contrastive loss to every (convolutional) layer of the architecture?\n\n3) Do you have an explanation as to why the standard deviation is so large for some of the results of the CtCAE (Table 1)?", "limitations": "Limitations are adequately addressed.", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is very well written and easy to follow. The introduction provides a good overview over the relevant research and a convincing motivation for the proposed approach. The methods section describes the proposed improvements and novel contrastive loss clearly. The results section is well-structured, and the related works section embeds the proposed research well into the existing research landscape.\n\nThe proposed contrastive loss seems like an elegant solution towards improving the \"separability\" of objects in the phase space. The complex-valued features contain two separate types of information (feature presence in the magnitudes and feature binding in the phases), and the proposed contrastive loss smartly utilizes this separation.\n\nThe provided ablation studies provide interesting insights into the proposed model improvements and design choices.", "weaknesses": "1) The main problem with the paper is that it seems to evaluate the models incorrectly. As described in section C.1 - \"RGB Images\" of the paper proposing the CAE [1], it cannot simply be applied to RGB images due to its evaluation method. During evaluation of the CAE, as well as in the proposed approach, features with small magnitudes are masked out, as they may exhibit increasingly random orientation values. However, this rescaling may lead to a trivial separation of objects in RGB images. For an image containing a red and a blue object, for example, the reconstruction will assign small magnitudes to the color channels that are inactive for the respective objects. By masking out values with small magnitudes, objects will subsequently be separated based on their underlying color values, rather than their separation as learned by the model. As far as I can tell, the paper does not describe any mechanism that would prevent the evaluation from doing this. I believe this effect is even visible in the qualitative examples. In Figure 3, the phase image of the CtCAE model on the dSprites example does not seem to clearly separate the purple and orange objects. The mask, however, separates them perfectly. In my eyes, this is only possible if the evaluation procedure makes use of additional (color) information besides the pure phase output. Overall, this makes it unclear whether the presented results are valid.\n\n2) Eq. 6: When measuring the cosine distance between the phase feature vectors, extra care would be necessary to take into account their circular nature. Otherwise, two phase vectors $a=(0,0,0)$ and $b=(2 \\pi, 2 \\pi, 2 \\pi)$ would have a large cosine distance to one another even though they represent the same angles. Has this been taken into account?\n\n3) It would be interesting to see an evaluation of how the proposed improvements influence the separation of objects in the latent space of the model. Ultimately, the goal of object discovery is to create object-centric representations within the model, but the paper currently only evaluates the separation at the output.\n\nMinor points:\n- line 25: not all slot-based method work with iterative attention procedures\n- Personally, I would move the CAE overview into a separate Backgrounds section\n- line 90: $m_\\psi$ can technically not be called a magnitude, as it may contain negative values.\n- Eq. (6): Is there no exp() applied to the cosine distances, such that the equation would correspond to a Softmax?\n- Using a dimensionality of 32x32 for Tetrominoes changes the shapes of the objects. While at the original dimension, all objects are made up from perfect squares, these squares get squeezed randomly when resizing the images. Thus, it would be better to make use of the original input dimension.\n\n\n\n---\n\n[1] Löwe, S., Lippe, P., Rudolph, M., & Welling, M. (2022). Complex-valued autoencoders for object discovery. arXiv preprint arXiv:2204.02075.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing valuable feedback on our work. We’re glad that the reviewer found our paper well-written, the proposed contrastive loss well-motivated and an elegant solution to the problem and the experiments/ablation studies insightful.  \n\nBelow are our responses:\n\n*”main problem with the paper is that it seems to evaluate the models incorrectly.”*\n\nIt is true that in our setting, certain color information may contribute to the separation process, and the example by the reviewer provides an illustration. However, the reviewer's claim \"By masking out values with small magnitudes, objects will subsequently be separated based on their underlying color values, rather than their separation as learned by the model.\" cannot be true, since CtCAE largely outperforms CAE++ (see Table 1) while both of them have a similar (almost perfect) reconstruction loss and (of course) are evaluated in the same way. This clearly indicates that the potential issue pointed out by the reviewer only has a marginal effect in practice in our setting, and that separation learned by the model is crucial to achieve good performance. In fact, “pure” RGB colors (as in the reviewer's example “containing a red and a blue object”) very rarely occur in images from the multi-object datasets (dSprites, CLEVR) used in this work. The percentages of pixels with a magnitude below 0.1 are 0.44%, 0.88%, and 9.25% for CLEVR, dSprites, and Tetrominoes respectively. While 9% on Tetrominoes is not marginal, we empirically observe that this is not an issue in practice, since we observe many examples in Tetrominoes where two or three blocks with the same color get grouped into different clusters by either CAE++ or CtCAE (see three yellow blocks in Figure 3, two magenta blocks in Figure 8, two magenta blocks in Figure 9), demonstrating that the separation is clearly not simply based on color values of pixels. Therefore, while we agree that our object extraction process should ideally be fixed for these edge cases, we leave that as an investigation for future work.\n\n*”In Figure 3, the phase image of the CtCAE model on the dSprites example does not seem to clearly separate the purple and orange objects ....... only possible if the evaluation procedure makes use of additional (color) information besides the pure phase output.”*\n\nWe would like to emphasize that the ‘Phase Img.’ (shown in columns 3/6/9 of Figure 3) is the average of the 3 output phase values plotted as a heatmap. Due to this averaging it cannot distinguish between different phase values eg. [0.1*$\\pi$, 0.5*$\\pi$, 0.2*$\\pi$], [0.2*$\\pi$, 0.1*$\\pi$, 0.5*$\\pi$], [0.5*$\\pi$, 0.2*$\\pi$, 0.1*$\\pi$]. Therefore the visual discrepancies between a ‘Mask’ and corresponding ‘Phase Img.’ are due to the use of the “average” phase plot. (Note that for the evaluation the KMeans clustering is applied on all 3 phase values (of a pixel) at the decoder output and is therefore able to discriminate between these different phase values leading to a correct prediction of the ‘Mask’.) We understand this aspect of the visualization is not explained and will add a note in the updated manuscript. Thank you for drawing our attention to this.\n\n*”Eq. 6: When measuring the cosine distance between the phase feature vectors, extra care would be necessary to take into account their circular nature.”*\n\nWe investigated a variant which accounted for the circularity of phase space but it did not improve the results. We observed in practice that phases are initialized from zero and rarely go beyond the range [-0.75*$\\pi$, 0.75*$\\pi$] (as observed in Figure 2 and 3).\n\n*”.. would be interesting to see an evaluation of how the proposed improvements influence the separation of objects in the latent space of the model ... currently only evaluates the separation at the output.”*\n\nEvaluating the separation of objects only at the output level follows the method used by the original CAE model. It allows a fair comparison of grouping performance learned by the baseline models and our proposed variants. We agree with the reviewer that this idea of evaluating object separation at several layers within the synchrony-based model is interesting and valuable but it is outside the scope of our current work.\n\n*”Why do you not apply the contrastive loss to every (convolutional) layer?”*\n\nWe experimented with various configurations applying the contrastive loss to different layers of the encoder and decoder, including applying the loss to every layer. We observed that adding more such loss terms did not help improve grouping performance and so we utilized the minimal (but still the best performing) choice of contrasting only the encoder and decoder outputs.\n\n*Do you have an explanation as to why the standard deviation is so large for some of the results of the CtCAE (Table 1)?*\n\nRegarding the std-dev for CtCAE in Table 1, please note that the comment about the larger std-dev for CtCAE compared to other models is only true for the ARI-FULL score on CLEVR dataset. Please note that on some dataset variants like dSprites (32x32) or CLEVR (96x96) the std-dev for CtCAE is lower and in other cases it is comparable to the std-dev of CAE++.\n\n*”On minor points.”*\n\nWe thank the reviewer for the suggestion to move the CAE description to a background section and agree to make this change. Thank you for pointing out that $m_{psi}$ cannot technically be called a magnitude due to potential negative values, we will correct this. We will correct the statement (line 25) to include a broader reference (not just iterative attention) to segregation mechanisms in slot-based models. Finally, thank you for spotting the error in Eq. 6. Indeed, we exponentiate the cosine distances resulting in a Softmax term. We apologize for this typo and will correct it in the updated manuscript.\n\nWe hope our responses resolve the major comments/questions posed by the reviewer. Please let us know if any other questions are yet to be addressed.\n"}, {"review_id": "glBXM5zMXC", "reviewer": "Reviewer_BSdH", "summary": "This paper presents a collection of small modifications of the Complex-valued AutoEncoder architecture leading to improved reconstruction/segmentation performance on simple RGB datasets; a contrastive learning objective is introduced to further improve the performance.", "questions": "- Eq. 2: A short note on why this separation is used and not one that uses the magnitude and phase could be helpful.\n- L134: Moving the reference to Figure 1 to the previous paragraph might make it easier to follow the general idea of this loss.\n- Eq. 6: Is this the loss for a single input image, meaning that the loss is then averaged over a batch? Or are the negative/positive samples chosen across a batch? If it's not across the entire batch, what speaks against doing this?\n- Figure 2: An explanation of what the last three columns show/what one can learn from looking at them can be useful for readers.\n- L218: Please elaborate on why the minimum distance is more relevant than the average.", "limitations": "While the authors mention the strongest limitation, a more extensive discussion of this approach's limitations is missing. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper's story is easy to follow, and the motivation is clearly stated.\n- Architectural choices are justified through ablation studies.\n- Experimental results are averaged over multiple random seeds.", "weaknesses": "- The captions of figures/tables could be improved to make it easier for readers to understand what is displayed. For example, Figure 1 is hard to understand without reading the relevant section.\n- Table 1: While I see that this is a comparatively new line of research on new approaches for object-centric learning, I'd still appreciate it if the authors compared to state-of-the-art models so that readers get a feeling for how far the gap still is/how small it has become.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and suggestions to improve our work. We are happy that the reviewer found the manuscript easy to follow and the main ideas presented to be clearly motivated. Below are our responses to the comments/questions from the reviewer: \n\n*“ … captions of figures/tables could be improved to make it easier for readers to understand what is displayed.”*\n\nIndeed, due to the space limitation, we had to avoid redundancy between the caption and section text. To improve this, we will add a link in the caption that refers to the corresponding text in the section (which provides the necessary contextual information), wherever appropriate. We thank the reviewer for pointing this out.\n\n*“ I'd still appreciate it if the authors compared to state-of-the-art models so that readers get a feeling for how far the gap still is/how small it has become.”*\n\nWe thank the reviewer for this suggestion (as was also requested by reviewer bBXA). We agree to this recommendation and will update the manuscript adding the grouping results for a baseline SlotAttention model to Table 1. For reference, SlotAttention achieves ARI-FG of $98.8\\pm0.3$ for CLEVR, $91.3\\pm0.3$ for dSprites and $99.5\\pm0.2$ for Tetrominoes (results from Emami et al. [3]).\n\n*“ Eq. 2: A short note on why this separation is used and not one that uses the magnitude and phase could be helpful.”*\n\nThis use of the cartesian form (real and imaginary components of a complex-valued activation) in a layer’s forward pass was the formulation introduced by Reichert and Serre [1] which has been maintained in Löwe et. al [2] as well.\n\n*“ L134: Moving the reference to Figure 1 to the previous paragraph might make it easier to follow the general idea of this loss.”*\n\nWe thank the reviewer for this suggestion and agree to move the reference to Figure 1 to the paragraph above to improve the readability. \n\n*“ Eq. 6: Is this the loss for a single input image, meaning that the loss is then averaged over a batch? Or are the negative/positive samples chosen across a batch? If it's not across the entire batch, what speaks against doing this?”*\n\nThe contrastive loss is computed on a per-image basis and averaged across the samples in a batch. The positive and negative pairs are extracted from within one image. This is because we would like to enforce the phases of pixels within a single image to be far apart or close depending on the objects they belong to. The phase values of objects across images are not relevant to achieve this instance-level grouping of objects within a scene. For example, given 2 images: img_1: contains a red ball and green cube || img_2: red ball, green cube and yellow triangle. Ideally we would want phases of the ball and cube in img_1 to be separated by 180 degrees, whereas in img_2 their difference is ideally 120 degrees. The red ball and green cube cannot have the same absolute phase values across both these images. Rather, what matters more is the relative phase between objects within a single image (scene). Therefore, it would be incorrect to contrast object instances across image samples in a batch to achieve instance-level grouping. \n\n*“ Figure 2: An explanation of what the last three columns show/what one can learn from looking at them can be useful for readers.”*\n\nWe thank the reviewer for this suggestion. We will include descriptive text for e.g., “column 5 shows the radial plot with the phase values from $-\\pi$ to $\\pi$ radians (colors correspond to phase angles), column 6 shows the raw phase values (in radians) averaged over the 3 output channels as a heatmap (where colors correspond to the colors from Phase Rad. column 5) and column 7 shows magnitude component of the complex-valued outputs” in the updated manuscript to assist the reader’s interpretation of the last three column's contents in Figure 2. \n\n*“ L218: Please elaborate on why the minimum distance is more relevant than the average.”*\n\nSeparability indicates how difficult it is for the phase pattern of one object to interfere with another. During the evaluation procedure to extract discrete object assignments from the continuous phase maps, a clustering procedure is applied on the phase values. Our clustering procedure (KMeans) uses the centroid that is closest (minimum) distance to assign a phase value as being part of that cluster. Therefore, minimum distance is more relevant to the accuracy of the extracted object assignments through clustering than average distance.\n\n*“ … a more extensive discussion of this approach's limitations is missing.”*\n\nWe would like to highlight the qualitative failure modes of CAE++/CtCAE discussed in lines: 259-263. We visualize and qualitatively examine these failure modes in Figure 4 in the main text as well as provide more such qualitative samples shown in Figure 14 in Appendix C.  Please let us know if you have some specific recommendations in-mind for the discussion of limitations that we have not included in the paper. \n\nPlease let us know if there are any concerns remaining that the reviewer raised that our responses have not sufficiently addressed. \n\nReferences:\n \n[1] Reichert et. al, “Neuronal Synchrony in Complex-Valued Deep Networks”, ICLR 2014.  \n\n[2] Löwe et. al, “Complex-Valued Autoencoders for Object Discovery”, TMLR 2022. \n\n[3] Emami et. al, “Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representations”, ICML 2021. \n"}, {"review_id": "wda7f0UT6e", "reviewer": "Reviewer_bBXA", "summary": "CAEs hold potential to address several issues associated with slot-based representation. They may provide flexible object granularity, enable the extraction of part-whole hierarchies as needed, and provide faster training speeds. However, testing of the original CAE was confined to grayscale images and a small number of objects (2-3). This paper explores the question: how can we scale up CAEs beyond this initial scope?\n\nFor this, the paper takes the following approach:\n1. Discovers certain architectural modifications that make original CAE perform much better on more complex datasets, in this case, Tetris, CLEVR, and dSprites.\n2. However, noting that this is not enough, the paper proposes a novel contrastive loss, which improves the segmentation performance even further when added to the original reconstruction loss of CAE.", "questions": "1. Why is CAE performance not reported in Table 3?\n2. Why not apply the contrastive loss on all layers rather than just the output of the encoder and the decoder? It appears a bit arbitrary to choose two specific layers to apply this loss.", "limitations": "Yes, the paper discusses the limitations relative to current slot-based methods, but one should also consider the potential benefits of the CAE family of models.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. Useful ablations show that architectural optimizations can be important to achieve the full potential of CAE. A good thing about some of these optimizations is that they actually relax and don’t complicate the original architecture e.g., by removing sigmoid activation.\n2. Empirically show higher storage capacity in terms of the number of objects than CAE.\n3. Paper is largely well-written.", "weaknesses": "1. While it is interesting that contrastive loss helps the segmentation performance, it somehow does not provide me new insights about the CAE framework itself. For instance, it doesn’t illuminate why separability was poor in the original CAE in the first place. As such, I am a bit worried that the proposed solution addresses a symptom rather than the root cause.\n2. Line 215: *“The results in Table 2 indicate that indeed, intra-cluster distance (the 216 last column) is smaller for CtCAE than CAE++ for every dataset, especially for the most difficult ones”* It is actually difficult to conclude this from Table 2 considering the large standard deviations of these measurements. My suggestion would be to tone down this claim and/or confine this result to the appendix. Regarding the separability results more broadly, it is not clear whether separability in the representation space (as defined in this paper) is actually important with respect to the things we may eventually care about e.g., downstream performance or other tasks.\n3. (Optional) It would be good to also report performances of some slot-based baselines, not for strict comparison, but to help the reader position the current performance in a more complete perspective. I see a line mentioned about it in the conclusion but without a pointer to a supporting result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for giving us valuable feedback and critiques on our manuscript. We’re happy to hear that the reviewer found our paper to be well-written and ablation studies on architectural design to be useful and simplifying. Below are our responses to your questions/comments: \n\n*“... does not provide me new insights about the CAE framework itself. For instance, it doesn’t illuminate why separability was poor in the original CAE in the first place.”*\n\nOur intuition for designing the contrastive loss stems from training the CAE model with solely the pixel-level reconstruction objective provided insufficient optimization pressure to guide the learning of bias terms ($b_{\\psi}$) to discover the various independent factors (objects) in the scene. Beyond that, we do not claim to provide further insights into why just the phase interference mechanisms in CAE activations are insufficient to allow it to learn meaningful groups on more complex datasets (like the ones tested here). We believe this could be an interesting analysis for future work.  \n\n*“... difficult to conclude this from Table 2 considering the large standard deviations of these measurements.”*\n\nWe thank the reviewer for highlighting this aspect of the results in Table 2. We agree with the reviewer, and will reduce the strong claim in the updated manuscript as follows: The results in Table 2 show that, the mean intra-cluster distance (the last column) is smaller for CtCAE than CAE++ on two (dSprites and CLEVR) of the three datasets. However, we believe it is important to retain Table 2 in the main text due to its relevance to the proposed contrastive loss. \n\n*“... it is not clear whether separability in the representation space (as defined in this paper) is actually important with respect to the things we may eventually care about e.g., downstream performance or other tasks.”*\n\nSeparability improves the ability of the model to group/store independently the representations of various objects in a scene. This separation of information internally allows the model to generalize better in compositional scenarios such as varying the number of objects, novel combinations of known objects etc by avoiding the “superposition catastrophe” problem [1]. The ability to separately maintain the information of different object instances also allows us to compute relational factors between them useful to solve tasks that involve object-level reasoning. This is the same motivation behind the task of perceptual grouping with slot-based models [1]. We do not make any specific claims about how “separability” is positively correlated to downstream task performance in our work.   \n\n*“(Optional) It would be good to also report performances of some slot-based baselines, not for strict comparison, but to help the reader position the current performance in a more complete perspective. I see a line mentioned about it in the conclusion but without a pointer to a supporting result.”*\n\nWe thank the reviewer for this suggestion (as was also noted by Reviewer BSdH). We agree to this recommendation and will update the manuscript adding the grouping results for a baseline SlotAttention model to Table 1.  For reference, SlotAttention achieves ARI-FG of $98.8\\pm0.3$ for CLEVR, $91.3\\pm0.3$ for dSprites and $99.5\\pm0.2$ for Tetrominoes (results from Emami et al. [2]).\n\n*“Why is CAE performance not reported in Table 3?”*\n\nThis was because the CAE baseline does not learn any meaningful grouping at all on dSprites and CLEVR as can be seen in Table 1. It achieves very low ARI-FG and ARI-FULL scores (near zero in several cases). Therefore, we decided to remove it from the results in Table 3 as at this very low level of grouping performance we cannot draw any meaningful conclusions about its ability to generalize to different numbers of objects in the scene.     \n\n*\"Why not apply the contrastive loss on all layers rather than just the output of the encoder and the decoder? It appears a bit arbitrary to choose two specific layers to apply this loss.\"*\n\nThis is a useful suggestion (as was noted by Reviewer QpC7 as well). We did experiment with various configurations applying the contrastive loss to different layers of both the encoder and decoder modules, including the reviewer’s recommendation of applying the contrastive loss to every convolutional layer. However, we observed that adding more such contrastive loss terms did not help improve the grouping performance and so we utilized the minimal (but still the best performing) choice of contrasting only the encoder and decoder outputs. \n\nPlease let us know if there are any concerns that the reviewer raised that our responses have not addressed satisfactorily. If the reviewer found our response useful, could the reviewer please consider increasing the score. Thank you very much!\n\nReferences: \n\n[1] Greff et. al, “On the Binding Problem in Artificial Neural Networks”, 2020. \n\n[2] Emami et. al, “Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representations”, ICML 2021. \n"}, {"review_id": "VvWAmYOsUZ", "reviewer": "Reviewer_pFdh", "summary": "This paper improves the state-of-the-art in synchrony based object centric learning methods. The current SoTA is the Complex-valued AutoEncoder (CAE). But CAE is unable to handle even the Tetrominoes dataset. The paper presents CAE++ and CtCAE.\n\nCAE++ = CAE + minor architectural improvements\n\nCtCAE = CAE++ + Contrastive loss term.\n\nExperiments show that CAE++ and CtCAE are the first synchrony based methods that can handle colour images, and CtCAE can handle up to 6 objects per image.\n\nThe contrastive loss term in CtCAE is intended to make the object clusters well separated in phase space. This is then evaluated in Table 2 by measuring inter-cluster and intra-cluster distances. The rest of the paper measures ARI and FG-ARI on Tetrominoes, dSprites, and CLEVR datasets. Ablation experiments are used to justify the design choices.", "questions": "I've asked my questions in the weaknesses section.\n\nAdditionally a sweep over the \\beta parameter would be nice. ", "limitations": "The authors have adequately addressed the weakness although the empirical gap between slot based and synchrony based methods should be made clearer in my opinion.\n\nI do not see any potential negative societal impact of this work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Although this line of work is niche, it is promising and engages scientific curiosity among many readers. The paper advances this line of work closer to the overall SoTA among object centric learning methods; although there is still a large gap here.\n\nThe method is easy to understand and the paper is well written.\n\nThe experiments cover most design choices and compare against CAE, the obvious baseline. The experiments also address important questions such as the robustness across varying number of objects and generalization to more objects. Error bars are included in all tables.", "weaknesses": "Please incorporate the following minor improvements\n- In line 175, it is claimed that performance is closely matched at 32x32. This is not true for dSprites ARI-FULL is 0.90 vs 0.68. I believe it may be sufficient to say that the methods are ordered the same way across resolutions.\n- In Table 2, please indicate in row 1 whether higher is better (up arrow) or down is better (lower arrow) to aid interpretation. In some cases numbers have been bolded despite overlapping error bars. Are these statistically significant?\n- In lines 299-305, consider citing Odin (https://arxiv.org/abs/2203.08777) which does pixel level BYOL followed by kmeans to get objects. It is kind of a contrastive object centric method (although mean-teacher distillation is probably the right word for BYOL).\n- Some background explanation about the encoder and decoder and where the kmeans happens to get masks would make it even easier to understand the method, for readers not familiar with CAE.\n\nThe main drawback is that slot based methods (DINOSAUR, https://openreview.net/forum?id=b9tUk-f_aG) are now working on real world data whereas Synchrony based methods are yet to make the break into textured images. But I think this paper is still relevant and that future developments in this line of work may catch-up to Slot based methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing valuable feedback on our work and for the highly positive comments on the strengths of our contributions. We are very happy to hear that the reviewer found the method easy to understand, the paper to be well written and the experiments thorough.\n\nBelow are our responses to your questions/comments: \n\n*“ In line 175, it is claimed that performance is closely matched at 32x32. This is not true for dSprites ARI-FULL is 0.90 vs 0.68. I believe it may be sufficient to say that the methods are ordered the same way across resolutions.”*\n\nWe thank the reviewer for highlighting this error. We will make the requested change by indicating the broader point that the various models continued to be ordered in the saw way across resolutions.\n\n*“In Table 2, please indicate in row 1 whether higher is better (up arrow) or down is better (lower arrow) to aid interpretation.”*\n\nThank you for this suggestion, we will make the requested change to Table 2 in the updated manuscript.\n\n*“In lines 299-305, consider citing Odin (https://arxiv.org/abs/2203.08777) which does pixel level BYOL followed by kmeans to get objects.\"*\n\nWe thank the reviewer for pointing out this relevant work. We will cite and discuss this work in the related work section under “Contrastive Learning with object-centric models”.  \n\n*“ The main drawback is that slot based methods (DINOSAUR, https://openreview.net/forum?id=b9tUk-f_aG) are now working on real world data whereas Synchrony based methods are yet to make the break into textured images.”*\n\nWe share the view of the reviewer (noted in lines: 312-313) that the current synchrony-based family of models lag behind state-of-the-art slot-based approaches to real-world visual complexities. They have received far less attention by the research community who have directed significant algorithmic/engineering efforts over the last several years to scale them from simple multi-object datasets (MNIST/Shapes/Tetrominoes) to the current visual complexities (MoVi-C/E). We hope the research community invests a similar amount of effort towards scaling synchrony-based models as well.\n\nRegarding where the ‘masks’ used by K-means for clustering are obtained, we take the phases at the decoder output layer as inputs for the clustering algorithm to compute the object assignments (please refer to L101-103).\n\n*“Additionally a sweep over the $\\beta$ parameter would be nice.”*\n\nWe initially evaluated several values for the $\\beta$ parameter such as [0.01, 0.001, 0.0001, 0.00001, 0.000001] and quickly settled to the value of 0.0001 (kept constant across all experiments in this work) as it led to better training and grouping performance across different datasets. As per your request, we additionally ran a sweep for $\\beta \\in [0.01, 0.001, 0.0001, 0.00001, 0.000001]$ on the 32x32 dSprites dataset.\n\nHere are the ARI scores we obtained for various values of $\\beta$ (shown on the first column below):\n\n0.01         - ARI-FG: 0.20$\\pm$0.06; ARI-FULL: 0.14$\\pm$0.04\n\n0.001       - ARI-FG: 0.43$\\pm$0.05; ARI-FULL: 0.66$\\pm$0.11\n\n0.0001     - ARI-FG: 0.48$\\pm$0.03; ARI-FULL: 0.68$\\pm$0.13\n\n0.00001   - ARI-FG: 0.48$\\pm$0.07; ARI-FULL: 0.53$\\pm$0.05\n\n0.000001 - ARI-FG: 0.46$\\pm$0.04; ARI-FULL: 0.49$\\pm$0.11\n\t\nThis extra sweep confirms that $\\beta=0.0001$ (used throughout the work) is a good choice.\n\t\nPlease let us know if there are any concerns you have raised that our responses have not yet addressed satisfactorily. \n"}]

Author rebuttal:
### General Response to all Reviewers

We have provided individual responses to each reviewer's questions/comments. We have not made any changes to the PDF of the paper to ensure the ease of referring to the line numbers, Figure/Table numbers as in the original version. We’ve provided additional results (Tables) as part of the individual rebuttal responses. We’re happy to continue engaging with all reviewers to resolve any remaining concerns. 


Meta-review: This paper presents a collection of architectural improvements over Complex-Valued Autoencoders (CAEs) and introduces a novel contrastive objective to improve scene decomposition performance on a range of synthetic multi-object datasets. The architectural improvements and contrastive loss, while simple, provide clear benefits and allow this class of model to scale to more complex datasets (although still simplistic compared to real-world data). 

All reviewers agree that this paper is very well-written and provides a clear advancement to an interesting architecture class that holds promise for learning structured scene representations. Even though this architecture class is still far from the performance of slot-based methods, the advancements are pointing in the right direction, and the paper should be of interest to the NeurIPS community.

There have been concerns around the evaluation protocol which the paper adopted, related to a thresholding heuristic that the authors chose to include (which was only described in the appendix) that could render the experimental comparison difficult to interpret. After an engaged discussion, the authors have sufficiently convinced the reviewers and myself that this heuristic does not have a large effect on the results. Nonetheless, it would be valuable to clearly highlight this aspect in the revised version of the paper.

Overall, this paper meets the bar for acceptance in my view and I am looking forward to seeing how future work will scale this class of models further to (hopefully) someday model real-world data to a convincing degree.
Decision: Accept (poster)