Learning to Compose: Improving Object Centric Learning by Injecting Compositionality
Authors: Whie Jung
Jaehoon Yoo
Sungjin Ahn
Seunghoon Hong
Venue: ICLR 2024 poster (2024)
Keywords: Object-Centric learning
Compositionality
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a novel objective that explicitly encourages compositionality of the representations.

Abstract: Learning compositional representation is a key aspect of object-centric learning as it enables flexible systematic generalization and supports complex visual reasoning. However, most of the existing approaches rely on auto-encoding objective, while the compositionality is implicitly imposed by the architectural or algorithmic bias in the encoder. This misalignment between auto-encoding objective and learning compositionality often results in failure of capturing meaningful object representations. In this study, we propose a novel objective that explicitly encourages compositionality of the representations. Built upon the existing object-centric learning framework (e.g., slot attention), our method incorporates additional constraints that an arbitrary mixture of object representations from two images should be valid by maximizing the likelihood of the composite data. We demonstrate that incorporating our objective to the existing framework consistently improves the objective-centric learning and enhances the robustness to the architectural choices.

Published as a conference paper at ICLR 2024
LEARNING TO COMPOSE: IMPROVING OBJECT CEN- TRIC LEARNING BY INJECTING COMPOSITIONALITY
Whie Jung, Jaehoon Yoo, Sungjin Ahn, Seunghoon Hong
School of Computing, KAIST [ (whieya, wogns98, sungjin. ahn, seunghoon craiaioooooonngnooo @kaist...........kk ac kr
ABSTRACT
Learning compositional representation is key aspect of object-centric learning as it enables flexible systematic generalization and supports complex visual reasoning. However, most of the existing approaches rely on auto-encoding objective, while the compositionality is implicitly imposed by the architectural or algorithmic bias in the encoder. This misalignment between auto-encoding objective and learning compositionality often results in failure of capturing meaningful object represen- tations. In this study, we propose a novel objective that explicitly encourages compositionality of the representations. Built upon the existing object-centric learning framework (e.g., slot attention), our method incorporates additional con- straints that an arbitrary mixture of object representations from images should be valid by maximizing the likelihood of the composite data. We demonstrate that incorporating our objective to the existing framework consistently improves the objective-cente centrrice learning and enhances the robustness to the architectural choices.
INTRODUCTION
As the world is highly compositional nature, relatively few composable units, such objects or words, can describe infinitely many observations. Consequently, human intelligence has evolved to recognize the environment as combination of composable units, (e.g,, objects) which enables rapid
adaptation to unseen situations by recomposing the already learned concepts (Spelke) 1990 Lake
al. 2017). Mimicking human intelligence, perceiving environment with composable abstractions
have shown consistent improvement in tasks related to systematic generalization (Kuo al. 2021
Bogin et al. 2021] Rahaman et al. 202I), and visual reasoning tasks (D' Amario et al. 2021 Assoucl
al. 2022 compared distributed counterparts.
Inheriting this spirit, object-centric learning (Burgess et al. 2019 Greff et al. 2019 Engelcke et al.
2020: Locatello al., (220)) aims to discover a composable abstraction purely from data without external supervision. Instead of depicting a scene with a distributed representation, it decomposes of latent representations, where each latent is expected to capture distinet object. To discover such representation in an unsupervised manner, most existing works employed an auto-encoding framework, where the model is trained to encode the scene into of representations and decode them back to the original image.
the scene into
However, the auto-encoding objective is inherently insufficient to learn compositional representation, since maximizing the reconstruction quality does not necessarily requires the object-level disentangle- ment. To reduce this gap. the existing works incorporate strong inductive biases to further regularize the encoder, such as architectural bias (Locatello et al., 2020) or algorithmic bias (Burgess et al. 2019 Lin et al., 2020; Jiang al., (020). However, has been widely observed that these methods are highly sensitive to the choice of syper--arameters, such as encoder and decoder architectures, and number of slots, often resulting in suboptimal decompositions by position or partial attributes (Singh et al. 2022a Saajadi al.) 2022 Jiang al. (023) instead of objects. Finding the optimal model configuration is also not straightforward in practice due to the missing object labels.
In this work, present novel objective that directly optimizes the compositionality of repre- sentations. Based upon the auto-encoding framework, our method extracts object representations independently from two distinct images and simulates their composition by the random mixture.

---

Published as a conference paper at ICLR 2024
The composite representation is rendered an image by the decoder, whose likelihood is evaluated by the generative prior. The encoder is then jointly optimized to minimize the reconstruction error of the individual images to encode relevant information of the scene (auto-encoding path) while maximizing the likelihood of the composite image to ensure the compositionality of the representation (composition path). Overall, approach with an additional regularization on compositionality. We show that directly injecting compositionality this way significantly boosts the overall quality of object-centric representations and robustness in training.
method can be viewed as extending the conventional auto-encoding
Our contributions are as follows. (1) We introduce a novel objective that explicitly encourages compositionality of representations. To this end, we investigate strategies to simulate the composi- tional construction of an image and propose learning objective for maximizing the likelihood of the composite images. (2) We evaluate our framework on four datasets and verify that our model consistently surpasses auto-encoding based baselines by substantial margin. (3) We show that our objective enhances the robustness of object-centric learning on three major factors, such number of latents, encoder and decoder architectures.
PRELIMINARY
Problem setup Object-centric learning aims to discover set of composable representations from an unlabeled image. Formally, given image I "X represented by either RGB pixels or feature from the pre-trained encoder, the objective is to extract the set S = (si...... s where each element 8, E RRP corresponds to the representation of composable concept (e.g., an object). Since object concepts should emerge from the data without supervision, a typical approach is use auto-encoding framework to formulate the learning process. Formally, the object-centric encoder Eg IR Hx xC + IR ND is trained jointly with a decoder Do R N - IR Hxw X by
= Ex),..
minimizing the reconstruction loss.
CAE(O, Φ) Ex [d(x, (E
(I)
where d is a distance metric (e.g., MSE).
Slot Attention Encoder Fg Since the auto-encoding objective is insufficient to learn highly structured representation, the existing approaches incorporate a strong architectural bias in the encoder Eg to guide the object-level disentanglement in Among many variants, we consider Slot Attention encoder[Locatell attention mechanism between a query (slot) and a key (input), where normalization applied over the slots by:
al. (2020) due to its popularity and generality. It employs dot-product
((Z)) ((S)T)
A S) sofumax
MxN
(2)
VD
where z = fo(x) Mx is flattened input feature encoded by CNN encoder fe. and k. represents linear projection matrices. Note that softmax operation is normalized the query (slots) direction, inducing competition among slots. Based on Equation the slots are iteratively refined by:
( (R") (orxalize(AA(xSStta S(m)y" v(2))), ((O) ~ NN,, diag(o)).
Here, ((n) denotes the slot representation after n iterations, μt, a are learnable parameters characteriz- ing the distribution of the initial slots, v is linear projection matrix, and Normalize(-) is weighted mean operation introduced by Locatello et al. (2020) to improve stability of the attention.
Slot Decoder D& While the architectural choice for Do is not constrained to a specific form in principle, subsequent works (Singh et al.) (222aa] Jiang et al. 2023) have empirically found that the choice of the decoder crucially impacts the quality of the object-centric representation. Locatello al. (2020) proposed pixel-mixture decoder that renders each slot independently into pixels and combines them with aipha-blending. Although slot-wise decoding provides a strong incentive for the encoder capture distinct objects in each slot, its limited expressiveness hinders its application to complex scenes. To address this issue, Singh et al. (2022a) employed Transformer decoder that takes the entire slots S an input and produces image in autoregressive manner. By modeling
2

---

Published as a conference paper at ICLR 2024
the complex interactions among the slots, it has shown great improvements in slot representation learning even in complex scenes.
Recently, Jiang al. (2023) employed diffusion model for the slot decoder. Instead of directly recon- structing an input image x, optimizes the auto-encoding of Equation via denoising objective (Ho et al. 2020 by:
Lo Φ) = -..-N((..) [wi(t) (1 t, S Ex - -
where X = V + vI a2 is an corrupted image of an imput x by the forward diffusion process at step I, a = [1 - (+) is schedule function, and w(tt is the weighting parameter. In practice, the diffusion decoder is implemented based UNet architecture (Rombach al. 20022), where each layer consists of CNN-layer followed by slot-conditioned Transformer. Once trained, the decoder generates an image x
(2(x5 (S) using iterative denoising. starting from the random Gaussian noise (Ho et al. 2020 Rombach et al. 2022). Employing diffusion decoder significantly enhances object-centric representation and generation quality compared previous arts especially complex scenes Jiang et al. )2023).
2.1 LIMITATIONS
While the slot attention with auto-encoding objectives has shown promise in object-centric learning, its success highly depends on the model architectures, such as number of slots and architectures of the encoder and decoder, where suboptimal configuration often leads to dividing the scenes into tessellations (Singh et al. 2022a Sajjadi et al. 2022) and objects into the parts (Jiang et al. (023). However, the optimal model configuration varies depending on the datasets, and discovering them through cross- validation is practically infeasible due to the missing object labels in an unsupervised setting. We argue that such instability is primarily because the auto-encoding objective inherently misaligned with the one for object-centric learning, since the former guides the encoder only to minimize the information loss on the input, while the latter demands the object-level disentanglement in the representation, potentially sacrificing the reconstruction quality. This motivates us seek an alternative approach that directly encourages object-level disentanglement in the objective function instead of designing architectural biases.
LEARNING TO COMPOSE
Our goal is to improve object-centric learning by modifying its objective function to be more directly aligned with learning compositional slot representation than the auto-encoding loss. Our main intuition is that arbitrary compositions of object representation are likely to yield another valid representation. To realize this intuition, our framework designed to generate composite images by mixing slot representations from two images and maximize their validity measured by the data prior.
Figure illustrates the overall framework of our method. Our framework is built upon the conven- tional object-centric learning that learns both the slot encoder and decoder by the auto-encoding path on individual images (Section 2) To impose compositionality on slot representation, we incorporate an additional composition path that constructs a composite slot representation from two images by the mixing strategy (Section (.TI and assesses the quality of the image generated from the mixed slots by the generative prior (Section 3.2p. This way, the auto-encoding path ensures that each slot contains the relevant information of an input image, while such slots are constrained to capture composable components of the scenes (e.g., objects) by regularizing the encoder through the composition path.
3.1 MIXING STRATEGY FOR COMPOSING SLOT REPRESENTATION
Given S1, S E NDD extracted from two distinct images x!, x2, we construct their composite slot representation S E NxD by
(5)
where (..-) denotes composition function of two sets. The primary role of the composition function is to simulate potential combinations of slot compositions. Since our goal to maximize the compositionality of unseen slot combinations, the composition function should be capable of exploring a broad range of compositional possibilities. Below, we introduce simple instantiations of such function.
3

---

Published as a conference paper at ICLR 2024
Auto-enn Path
Image
Slot
Reconstruction 1
Encoder
Decoder
Composition Path
Mixing Slots
(one-shot) Decoder
Generative Prior
Validity
Composite Image
Encoder
Decoder
Image
Slot
Reconstruction 2
Overview of method. Our framework consists of two paths: auto-encoding path and a composition path. The auto-encoding path ensures slot representations encode relevant information about an image. contrast, the composition path encourages the compositionality of the representations by constructing the composite representation through the mixture of slots from two separate images (Section 3.1]).nn..1..... and assessing the quality of the composite image by the generative prior (Section 3.2). The encoder is jointly optimized by both paths.
Figure 1:
Random Sampling In this approach, we randomly sample N slots among 2.V slots i.e., S N (S' U U As it explores over all of the possible combinations, this composition function encourages the slot representation itself to be highly composable to generate valid images for any combinations. the other hand, it may produce invalid combinations of slots on rare occasions, e.g., omitting the background slots sampling two objects placed in the same location.
Sharing Slot initialization One way to mitigate such suspicious compositions is to constrain S to valid composition of the scene. However, strictly ensuring this constraint is non- trivialv due to the stochastic nature of slot attention i.e., each slot is sampled stochastically from its underlying distribution and the association between the slots and scenes varies depending on the initialization. Instead, we adopt a rather simple approach that employs the identical slot initialization ( in Equation for two images, and sample the exclusive set of slots. Formally, let / and /2 be random partition of slot indices i.e., I 12 = {1, .... NN. 41 n Ig = 0. Then we construct the composite slot by S = 8), U 18, where S and S are slots extracted by Equation from x' and x², respectively, which are initialized with the same ((0), The underlying intuition that the slot initialization reasonably correlated with the objects it captures (Figure(6). hence sampling from exclusive slots is likely to be valid scenes than the random sampling.
3.2 MAXIMIZING THOOD OF THE COMPOSITE IMAGE
Given the composite slot S obtained by the previous section, our next step is quantifying its validity i.e., measuring how valid the composition of two image slots is. To this end, we decode back an image by x" D& (S*) and measure the likelihood of the image using the generative prior p(x*).
To model the generative prior p(x"), we opt for a diffusion model (Ho et al. 2020) due excellence in generation quality and mode coverage (Xiao et al.] 2022). The latter is especially important in our framework since the model evaluates the prior over potentially out-of-distribution samples generated by the composition (Section 3.1). Instead of introducing an additional pre-trained diffusion model, employ the diffusion- decoder in the auto-encoding path (Section 2. and reuse as a generative prior. This way, our decoder D is trained by minimizing the reconstruction loss by denoising objective in Equation 4 while serving generative prior in the composition path. It greatly improves the parameter-efficee efficiency and memory, and the need for pre-trained generative prior per dataset.
Generative Prior
4

---

Published as a conference paper at ICLR 2024
Maximizing p(x") Given the generative prior, we maximize the likelihood p(x*) with respect to x" in the composition path. Since Lui in Equation Fiatiionnisss minimizing the upper bound of negative log likelihood of zf (Ho et al. 2020), minimizing LDiT with respect to x" leads to the maximization of the likelihood ). However, computing the gradient of L requires expensive computation of Jacobian maxtrix of the decoder and it often degrades the overall training stability. Following (Poole)
ct al. 2022), the gradient of LDiff with respect to O can be approximated as:
0x" a0
V VeCrrior (0) (E,,,lu(t)DDo((
(6)
where E IO, is noise, t ~ [t fmax] is a timestep, respectively, w(t) is weighting function dependent f, and x V + orE is corrupted image of x" from forward diffusion process. By updating the encoder parameters O with VoLrrior x is guided toward high probability density region following the diffusion prior. Note that optimization of the Equation 6 is with only respect to the encoder parameter while fixing the decoder. It prevents suspicious collaboration between the encoder and decoders in generating composite images from suboptimal slots.
Surrogate One-Shot Decoder As discussed earlier, our framework exploits the diffusion model D& as decoder and generative prior in the auto-encoding and composition paths, respectively. One drawback is that the diffusion decoder requires an iterative denoising process to generate the composite image x which takes significant time and makes the backpropagation through the decoder non-trivial. To address this problem, we employ a one-shot decoder Dy as a surrogate for D& to support fast and differentiable decoding of x
We employ a bidirectional Transformer (Devlin et al. 2019) that takes the composite slot S and the learnable mask tokens m E ## as input, and produces the composite image by a single forward process by Dw m, S*). The decoder is trained along with the auto-encoding path by:
CRecOn (0. (m, (x)) x112.
Note that the generation quality of the one-shot decoder Da: is behind the powerful diffusion decoder D&, and serves only to compute the x" in Equation We observe that such weak decoder is sufficient to compute the meaningful gradient through the Equation 6 presumably because the gradients are accumulated over various noise levels t.
3.3 LEARNING OBJECTIVE
In this section, we summarize the overall framework and objective function. Our framework consists of two paths; auto-encoding path and composition path. In auto-encoding path, encoder Eg and two different decoders D., Dy are trained to minimize auto-encoding objective in Equation 4 and Equation| 7 In composition path, we first extract S with Equation 5 and generate x" with the deterministic decoder D., and update the encoder maximize the Equation 6 while fixing decoders D& and De.- We find that incorporating an additional regularization term on the slot attention mask is helpful in enhancing object-centric representations:
LRee(()) = A sg([|x² x913) + A2 sg([ |x2 x91));
(8)
where A' = A S1 (M))
are attention masks from the last iteration of slot attention for x2, x" (Equation 2p. respectively, and sg(-) denotes stop-gradient operator. It encourages the source and the composite images to be consistent over the object area captured by the slots, enhancing the content-preserving composition. The overall objective is then formulated as follow:
A A ( )
Cioual (8, Φ, 1b) Aprice Crior (0) + CuurCour(O, Φ) + ARRcon (0, v) + CReg (0)
where Apriors ADiff AReson: AReg are hyperparameters for controlling the importance of each term. We empirically find that Aptior ADaa ARecon 1.0, Auag 0.25 generally works well and use it throughout the experiments.
We also consider one-step denoising result of the diffusion decoder using Tweedie's 's formula Stein 1981 Robbins 1992) but observe severe degradation in performance due to its inferior quality.
5

---

Published as a conference paper at ICLR 2024
4 RELATED WORK
Object-centric learning
The most dominant paradigm of object-centric learning is employing the auto-encoding objective (Burgess al. 2019 Greff et al. 2019 Engelcke et al. 2020 [021] Lin
et al. 2020 Jiang et al. [020; Eslami et al. 2016: Crawford & P 2019). To guide the model to learn structured representation under reconstruction loss, Locatello et al. (202)) introduces Slot Attention, where each slot is iteratively refined with dot-product attention mechanism normalized
in slot direction, inducing competition between the slots. uollow-up studies (Singh et al., 2022a Seitzer et al., 2022; Saijadi et al. 2022) demonstrate that Slot Attention with an auto-encoding objective has the potential to attain object-wise disentanglement even in complex scenes. Nonetheless, auto-encoding alone often involves training instability, which leads to attention-leaking problem (Kim)
eet al. 2023), or dividing the scene into Voronoi tessellations (Sajjadi et al. 2022 Jiang et al. (023)
To overcome such challenges, there have been few attempts on revising the learning objective such as replacing image reconstruction loss with denoising objectíve (Jiang et al. 2023 Wu et al. 2024)
or contrastive loss (Hénaff et al. 2022; Wen al. 2022). Nevertheless, these approaches still do not impose direct learning of object-centric representations.
Generative Prior There are increasing interests in exploiting the knowledge pre-trained from gen- erative prior to various applications such as solving inverse problems (Chung et al. 2023). guidance in conditional generation (Graikos et al. 2022; Liu et al. 2023), and image manipulations (Ruiz et al..
2023a: Zhang et al. 2023 Ruiz et al.] (023b). One prominent approach in this direction is text-to-3D
Generation, where a large-scale pre- trained 2D diffusion model (Rombach et al. 2022 Saharia et al.
2022 is leveraged to generate realistic 3D data without ground-truth (Wang et al. [023a]] Lin et al.
2023 Metzer et al. 2023 Wang et al., 2023b). The seminal work by (Poole et al. 2022) formulates a loss based on probability density distillation to distill pre-trained 21) image prior to 3D model. Back-propagating the loss through a randomly initialized 3D model, e.g., NeRF (Mildenhall et al. 2020), the model gradually updates to generate high-fidelity 3D renderings. Inspired by this line of work, we employ a generative model in our approach to maximize the validity of the given images.
EXPERIMENT
Implementation Details We base our implementation on existing frameworks (Singh al. 2022a Jiang et al., 2023). We employ the features from the pre-trained auto-encode to represent an image. For the slot encoder, we employ the CNN based on UNet architecture (Singh al. 2022b Diang et al. 2023) to produce high-resolution attention map. Also, we employ an implicit Slot Attention (Chang al. 2022) stabilize the iterative refinement process in slot attention. For the slot mixing strategy, we opt for a sampling with sharing slot initializations for all the experiments unless specified, since it shows slightly better performance than the random sampling strategy. When we compute Lrnor (Equation 6). we use min 0.02, mmax 0.5 following recent report in (Wang al. employing too high noise level impairs the optimization.
2023b) that
Datasets We validate method on four datasets. CLEVRTex (Karazija al. 202 consists of various rigid objects with homogencous textures. MultiShapeNet (Stelzner et al. (222 includes more complex and realistic furniture objects. PTR (Hong al. 2021) and Super-CLEEV (Li et al. 2023) contain objects composed of and resized to 128x 128 resolution images...
parts and textures. All of the datasets are center-cropped
We compare our method against two strong baselines in the literature, SLATE (Singh al. (202aa) and LSD (Jiang et al. 2023), which employ autoregressive Transformer and diffusion- based decoders, respectively. Note that our method without composition path reduces to LSD. For a fair comparison, we employ the same encoder architecture based on slot attention (Locatello et al.. 2020) in all compared methods including ours. For LSD and our method, we employ the same pre-trained auto-encoder (Rombach et al. 2022) to represent an input image. Since SLATE runs on discrete features, we employ the features from the pre-trained VQGAN model (Esser et al. 2021 and denote it as SLATE+. All baselines including ours are trained for 200K iterations.
Baselines
Evaluation Metries
Following the previous works (Jiang et al. 2023 Singh et al.) 2022a b Chang et al. 2022), we report the unsupervised segmentation performance with three measures: Adjusted

---

Published as a conference paper at ICLR 2024
Table Comparison results on unsupervised object segmentation. We evaluate the how well the slot attention masks coincide with the ground-truie objects using FG-ARI, mloU, and mBO (The higher better). All results are evaluated on held-out validation set.
(a) CLEVRTex
(b) MultiShapeNet
Model
FG-ARI
mioU
mBO
Model
FG-ARI
mloU
mBO
SLATE+
71.29
52.04
52.17
SLATE+
70.44
15.55
15.64
LSD
76.44
72.32
72.44
LSD
67.72
15.39
15.46
Ours
93.06
74.82
75.36
Ours
59.21
59.4
(c) PTR
(d) Super-CLEVR
Model
FG-ARI
mloU
mBO
Model
FG-ARI
mloU
mBO
SLATE+
91.25
14.1
14.22
SLATE+
43.73
29.12
29.49
LSD
61.1
10.18
10.33
LSD
54.79
14.12
14.43
Ours
90.65
40.89
41.45
Ours
63.08
47.17
48.03
GT
Ours
SLATE+ LSD
ClevrTex
MultiShapeNet
PTR
Super-CLEVR
Figure 2: Qualitative results unsupervised object segmentation. The baselines tend to split an object into different slots (CleverTex) and/or combine different objects and background into a single (MultiShapeNet, PGR. Super-CLEVES On the other hand, our method produces consistently better object masks, showing improved disentanglement of objects and background in all datasets. More results presented in the Figure 7 Zoom in for better view.
rand index for foreground objects (FG-ARI), mean intersection over union (mloU), and mean best overlap (mBO). These metries measure the overlap between the slot attention masks and ground-truth object masks, where FG-ARI focuses more on the coverage of theobject area.
5.1 UNSUPERVISED OBJECT SEGMENTATION
We first present the comparison results of our method with baselines unsupervised object segmen- tation. Table summarizze the quantitative results. Our method significantly improves the FG-ARI scores over the baselines in all datasets (8 to 29% improvement) except PTR, indicating that captures an object holistically into an individual slot while the baselines tend to split the object into multiple parts and distribute across multiple slots. In terms of mloU and mBO, our method improves the baselines over all datasets, especially when the background is monolithic (MultiShapeNet, PTR, and Super-CLEVR, indicates that the baselines struggle to separate the objects from the background when there exists a strong correlation between them, while our method can still robustly identify the objects. Overall, the results indicate that our method consistently outperforms the baselines by significant margin. Notably, the consistent and significant improvement over LSD indicates that regularization on the compositionality is effective in learning object-centric representation.
We also present the qualitative results in Figure It shows that SLATE frequently splits the foreground object masks into multiple segments in CLEVRTex and Super- CLEVR datasets, and fails to capture object entities in PTR and MultiShapeNet. Similarly, LSD fails to segment the object in all datasets except CLEVRTex dataset, and tends to rely on positional bias PTR and Super-CLEV In contrast, our method consistently captures objects with tight boundaries.
7

---

Published as a conference paper at ICLR 2024
90
80
90
FG-ARI
80
FG-ARI
FG-ARI
80
60
X
50
40
100%
133%
15
UNet
60
166%
17
20
CNN
200%
Ours 10
SLATE
LSD
SLATE+
LSD
Ours
SLATE+
LSD
Ours
(a) Number of slots
(b) Encoder architecture
(c) Decoder capacity
Figure 3: Robustness against various architectural biases. We evaluate the robustness of our model various different number of slots, encoder architectures, and decoder capacities. Results based on mloU and mBO are presented in Figure 5
5.2 ROBUSTNESS OF COMPOSITIONAL OBJECTIVE
Compared to approaches based on auto-encoding, our method directly incorporates the objective to learn compositional representation, thus is more robust to the choice of architectural biases and hyperparameters. To demonstrate this, we evaluate our method while varying three major factors that are known to be highly sensitive in the previous approaches, such as number of slots, encoder architecture, and decoder capacity. Figure 3 summarizes the result on CLEVRTex dataset based on FG-ARI. All methods are trained up to 100K iterations for fair comparison.
Number of slots Since object-centric learning assumes no prior knowledge on data, the mismatch between the number of objects and slots is inevitable in practice. To evaluate such robustness, vary the number of slots from 11 to 17. Figure Sal presents the result. It shows that the performance of the baselines is highly sensitive to the number of slots. Specifically, SLATE tends deteriorate more as the number of slots increases. Compared to the baseline, our method achieves more robust performance by encoding object into slot while leaving excess slots empty.
Encoder architecture To identify the effect of slot encoder, we consider two popular architectures in the literature; multi-layer CNN encoder (Singh al. 2022b) and UNet-based encoder (Ron- neberger et al., (215). Figure 3b summarizes the result. It shows that employing the weaker encoder generally deteriorates the performance of the baselines significantly, indicating that architectural bias in the encoder is critical in the auto-encoding objective. Interestingly, the performance of our method hardly affected by such drastic modifications, showing great robustness.
Decoder capacity is widely observed that the choice of decoder is also crucial in object-centric learning, since the highly expressive decoder can often bypass the object representation to minimize the reconstruction loss (Singh et al. 2022a). To examine such effect, we gradually increase the feature dimensions of the decoder to 133%. 166%, and 200%. Figure 3c summmariess the result. It shows that increasing the decoder capacity hampers the performance SLATE. LSD exhibits the opposite trends showing large improvement in FG-ARI, although its performance drops significantly in mloU (Figure 5). Compared to the baselines, our method is much less sensitive to the decoder capacity, while the performance tends to improve slightly with increased capacity in all measures.
Overall, the results indicate that the quality of object-centric representation is significantly influenced by various factors in the auto-encoding- based methods. Conversely, our model consistently delivers outstanding performance across all configurations, even with major alterations to the encoder archi- tecture. It demonstrates that our regularization through the composite path can directly encourage the model to learn compositional representation, greatly enhancing robustness to architectural biases.
5.3 INTERNAL ANALYSIS
Component-wir Contributions To identify the contributions of each component our framework, we conduct an ablation study and present the result in Table 2 The first row corresponds to our model with only the auto-encoding path, while the last row is the complete version of our model. Comparing the first row with the others shows that incorporating the composition path significantly improves overall quality. Adding Lpiar we observe a substantial improvement in all three metrics. Considering that FG-ARI measures the correct cluster membership of pixels within the objects,
8

---

Published as a conference paper at ICLR 2024
Table 2: Ablation study on CLEVRTex dataset. All models are trained up to 100K iterations.
Cprior
LReg
Share ())
FG-ARI
mloU
mBO
x
x
X
42.48
52.26
52.41
X
65.76
67.72
67.62
X
70.29
69.08
69.28
x
V
65.26
58.81
58.99
/
V
88.15
75.30
75.64
Sourse
Nouroe
GT
Output
Output
Output
Output
Ous
SLATE+ LSD
CLEVRTTx
MultShapeNNet
Super-CLEVE
Figure 4: Investigating object representation through compositional generation. We investigate the compositionality of learned representations by removing (red arrow) and adding (blue arrow) object slots between two images and generating the composite image. More results are in Figure X
increased FG-ARI indicates that the generative prior encourages the encoder to capture more holistic object representations. This is because the generative prior penalizes the encoder for fragmenting the objects, thereby discouraging the generation of unrealistic partial objects in the composite image. Comparing the second and the third rows, we observe that sharing the slot initialization ( slightly enhances mloU and mBO scores. This improvement likely attributed the increased training stability by avoiding invalid slot combinations shown in Figure 6) Incorporating regularization Cigg alone in the composition path does not improve the performance (fourth row), while combined with generative prior, it leads to significant improvement.
Compositional Generation We present the compositional generation results to further investigate the impact of our composition path. Figure X presents the results. Given images, we construct the composite representation by replacing one object slot from the first image (red arrow) to another from the second image (blue arrow), and producing the image by the decoder. Based on visualization of the learned slots, we observe that the baselines often fail to learn compositional slot representation, by separating objects into multiple slots or encoding background with an object. It leads to failures in object-level manipulation, such retaining object after the removal (LSD in MultiShapeNet and PTR), altering the content of the added object (SLATE in MultiShapeNet), or transforming back- ground with the object (SLATE in PTR and LSD in Super-CLEVR) In contrast, method produces both semantically meaningful and realistic images from composite slot representations, supporting claim that we regularize object-centric learning through the proposed compositional path.
CONCLUSION
In this paper, we introduced method to address the misalignment between object-centric learning and the auto-encoding objective. Our method based on auto-encoding framework, and incorporates an additional branch to directly assess the compositionality of the representation. This involves constructing composite representations from two separate images and optimizing the encoder jointly with the auto-encoding path to maximize the likelihood of the composite image. Despite the simplicity, our extensive experiments demonstrate that our framework consistently improves the object-centric learning over the auto-encoding frameworks. It also shows that method greatly enhances the robustness to the choice of architectural biases and hyperparameters, which typically pose sensitivity challenges in auto-encoding-centri approaches.

---

Published as a conference paper at ICLR 2024
Acknowledgemmntts This work supported in part by Institute of Information & communications Technology Planning & Evaluation (ITPP) grant (No.2022-0-00922 2022-0-00959, 2021-0-02068, and 2019 2019-0-00075) and National Research Foundation of Korea(NRF) grant (202/RICIC101240 and 2022R (00200411))) funded by the Korea goVernmen((MS11).
REFERENCES
Rim Assouel, Pau Rodriguez, Perouz Taslakian, David Vazquez, and Yoshua Bengio. Object-centric compositional imagination for visual abstract reasoning. In ICLR2022 Workshop the Elements of Reasoning: Objects, Structure and Causality, 2022.
Ben Bogin, Sanjay Subramanian, Matt Gardner, and Jonathan Berant. Latent compositional represen- tations improve systematic generalization in grounded question answering. Transactions of the Association for Computational Linguistics, 9: 9:195-210 2021.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 2019.
Michael Chang, Tom Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. In NeurIPS, volume 35, pp. 32694- 32708, 2022.
Hyungjin Chung, Dohoon Ryu, Michael T McCann, Mare I. Klasky, and Jong Chul Ye. Solving 3d inverse problems using pre-trained 2d diffusion models. In CVPR, 2023.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. AAAI, 2019.
Vanessa D' Amario, Tomotake Sasaki, and Xavier Boix. How modular should neural module networks be for systematic generalization? In NeurlPS, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4188 Minneapolis, Minnesota. June 2019, Association for Computational Linguistics. doi: 10.18653/v/NN1- 1423. URL http://aacaanttoooo.oooo//o///// 1423
Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. In ICML. 2022.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICLR, 2020.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. In NeurlPS, 2021.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari. Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In NeurIPS, 2016.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. In NewrIPS, 2022.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In ICML., 2019.
Olivier Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. ECCV, 2022.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Yining Hong, Li Yi, Josh Tenenbaum, Antonio Torralba, and Chuang Gan. Ptr: A benchmark for part-based conceptual, relational, and physical reasoning. In NeurlPS, 2021.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In ICLR, 2020.
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. In NeurIPS, 2023.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. ClevrTex: Texture- Rich Benchmark for Unsupervised Multi- Segmentation. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Jinwoo Kim, Janghyuk Choi, Ho-Jin Choi, and Seon Joo Kim. Shepherding slots to objects: Towards stable and robust object-centric learning. CVPR, 2023.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In CVPR, 2023.
Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Compositional networks enable systematic generaliza- tion for grounded language understanding. In EMNLP, 2021.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Zhuowan Li, Xingrui Wang, Elias Stengel-Fskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In CVPR, pp. 14963-14973, 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. ICLR, 2020.
Xihui Liu, Dong Huk Park, Samanch Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In WACV, 2023.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurIPS, 2020.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. In CVPR, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernande7, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arX1v:2304.07193, 2023.
Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Drcamfusion: Text-to-3d using 2d diffusion. In ICLR, 2022.
Nasim Rahaman, Muhammad Waleed Gondal, Shruti Joshi, Peter Gehler, Yoshua Bengio, Francesco Locatello, and Bernhard Schölkopf. Dynamic inference with neural interpreters. In NeurlPS, 2021.

---

Published as a conference paper at ICLR 2024
Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pp. 388-394. Springer, 1992.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10000 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro Frangi (eds.), Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015, pp. 234 241, Cham, 2015. Springer International Publishing. ISBN 978-3-319-24574-44
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023a.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. arXiv preprint wwXiv:2307.06949 2023b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. In NeurlPS, 2022.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabries Tong He, Zheng Zhang. Bernhard Schölkopf, Thomas Brox, al. Bridging the gap to real-world object-centric learning. In ICLR, 2022.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In ICLR, 2022a.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. In NeurlPS, 2022b.
Elizabeth S Spelke. Principles of object perception. Cognitive science, 14(1):29-56 1990.
Charles M Stein, Estimation of the mean of multivariate normal distribution. The annals of Statistics, pp. 1135-1151, 1981.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.011 2021.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023a.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS, 2023b.
Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. In NeurIPS, 2022.
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object-centric generative modeling with diffusion models. In NewrlPS, 2024.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with
denoising diffusion gans. In ICLR, 2022.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.
Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma. Weiming Dong, and Chang- sheng Xu. Inversion-basee style transfer with diffusion models. In CVPR, 2023.
12

---

Published as a conference paper at ICLR 2024
A ADDITIONAL IMPLEMENTATION DETAILS
Table 3 provides details of hyperparameters used in experiments. For the Slot Attention encoder Eg and diffusion decoder Dø, base our implementation Jiang et al. 2023 Specifically, in the Slot Attention encoder, we employ a CNN-based UNet image encoder. Prior to the UNet encoder, we 64 x 64 image. Implementing the diffusion decoder Dø. we follow the design of the LSD decoder. The overall structure of D is based the U-Net architecture, where each layer is composed of CNN layers and a transformer layer. The surrogate decoder D& is implemented with the Transformer Architecture in Singh et al. (2022a). It takes slots as input through cross-attention layers. the experimental setting, we augment the Super- Coper-CLEVR dataset by randomly altering the background color to another color.
incorporate single layer CNN to downsample the original 128 128 image
General
Batch Size
Training Steps
200K
Learning Rate
0.0001
CNN Backbone
Input Resolution
128
Output Resolution
64
Self Attention
Middle Layer
Base Channels
128
Channel Multipliers
[1,1,2,4]
Heads
# Res Blocks / Layer
2
Slot Size
Slot Attention
Input Resolution
64
Iterations
7
Slot Size
192
Auto-Encoder
Model
KL-8
Input Resolution
Output Resolution
16
Output Channels
Diffusion Decoder
Input Resolution
16
Input Channels
scheduler
Linear
Mid Layer Attention
# Res Blocks / Layer
2
Heads
Base Channels
192
Attention Resolution
[1,2,4.4]
Channel Multipliers
[1,2.4.4]
Surrogate Decoder
Layers
Heads
8
Hidden Dim
384
Table 3: Hyperparameters used in our experiments.
B ADDITIONAL RESULTS
B.1 ADDITIONAL RESULTS ON ROBUSTNESS TESTS
We include results of the robustness test on mloU, mBO metrics in Figure Similar to the results on FG-ARI (Figure 3 model is surprisingly robust to wide range of hyperparameters. It suggests that directly optimizing the compositionality the representation significantly reduce a dependency choice of hyperparameters.
13

---

Published as a conference paper at ICLR 2024
80
80
75
70
75
70
70
60
mloU
mloU
mloU
65
X
50
60
100%
40
133%
50
200%
SLATE+
LSD
Ours
SLATE+ LSD
Ours
SLATE+
LSD
Ours
80
80
75
70
75
X
70
mB0
mB0
65
X
50
X
100%
55
40
55
UNet
166%
50
30
CNN
50
200%
SLATE+
LSD
Ours
SLATE+
Ours
SLATE+
LSD
Ours
(a) Number of slots
(b) Encoder architecture
(c) Decoder capacity
Figure 5: Robustness across Various Hyperparameters. We evaluate the robustness of our model across different number of slots, encoder type, and decoder capacity. Among various hyperparameters, our model steadily shows powerful performance against baselines.
B.2 UNSUPERVISED OBJECT SEGMENTATION
We present additional qualitative results for unsupervised segmentation results in Figure 7 Our method successfully segmented the object regions across all four datasets. In contrast, baselines easily divide each object into multiple segments capture wide around the objects.
B.3 EFFECT OF MIXING SLOT STRATEGY
As discussed in Section 3.1 and Section 5.3 sharing (()) slightly enhances the performance by roughly avoiding suspicious compositions during training. To investigate how sharing slot initialization affects the composition, we obtained the slot representations from multiple scenes with the same slot initialization and grouped those representations by their order, i.e., belongs i-th group. we observe that the captured objects from the same initialization are correlated to some degree. The slots in the first row mostly capture the backgrounds of the scenes, while other slots tend to capture foreground objects. Moreover, we observe that the slots in the fourth row tend to capture the objects located in the lower part the scene. Based on these observations, we conjecture that sharing slot initialization stabilizes our framework by alleviating some suspicious compositions, such as the occlusion of foreground objects or composing multiple backgrounds.
Figure [6]
B.4 INVESTIGATION ON COMPOSITIONALITT OF SLOTS
In this section, we provide more visual samples of composite images to investigate the composition- ality slot representations in our method. Figure g illustrates the results of generating composite images by mixing slots from two images, which supplements the Figure in the main paper. It shows that the baselines often fail to capture compositional objects into independent slots, while our method successfully learns object-level slots through the composition path. As a result, the composite images generated by the baselines often fail to adhere the object-level manipulation, such as retaining the removed objects transforming the object identity and background pattern while adding object. In contrast, our method preserves these semantics more precisely based on accurate object slots.
14

---

Published as a conference paper at ICLR 2024
$
2
$
Figure 6: Grouping of slots by sharing the slot initialization. We obtain slot representation from various images while sharing the initial values of ( and cluster the representation based on their initial values. Slots initialized as 81 consistently capture backgrounds.
B.5 ADDITIONAL QUALITATIVE RESULTS ON COMPOSITIONAL GENERATION
To help a comprehensive understanding of the baselines, we provide more qualitative samples on compositional generation in Figure While Figure 4and Figure illustrates the common failure cases the baselines, we additionally present compositional generation results where the baselines also reasonably capture object into slot. Despite the reasonable slot attention masks, the composite image produced by the baseline model often distorts the original appearance of the object or creates unrealistic partial objects. In contrast, our model consistently produces faithful composite images, which highlights the importance of the compositional objective.
15

---

Published as a conference paper at ICLR 2024
CLEVRTex
GT
Ours
SLATE- LSD
MultiShapeNNet
GT
Ours
SLATE+ LSD
PTR
GT
Ours
SLATE+ LSD
Super-CLEVR
GT
Ours
LSD
SLATE+
Figure 7: More Qualitative Results on Unsupervised Object Segmentation.
16

---

Published as a conference paper at ICLR 2024
SOHISS
Souns
SouDe
Output
Output
Output
Output
LSD
(a) CLEVRTex
Sompe
Samse
Output
Output
Oulput
Outpu
SLATE: LSD
(b) MultiShapeNet
SOuss
Seurss
GY
Output
Output
Output
Outpue
LSD
(c) PTR
BESTO
Output
Output
Output
Output
LSD
(d) Super-CLEVT
Figure 8: More Qualitative Results Compositional Generation between two images.
17

---

Published as a conference paper at ICLR 2024
SSHOS
SOUDS
SOHDE
Souiee
Output
Output
Output
Output
LSD
SLATE
(a) CLEVRTex
Soure
Souse
Oulput
Output
Oulput
Outpu
LSD
SLATET
(b) MultiShapeNet
Scurse
GF
Output
Output
Output
Output
LSD
(c) PTR
BESNOS
Output
Output
Output
Output
LSD
SLATE
(d) SLEER
Figure 9: More Qualitative Results Compositional Generation between two images.
18

---

Published as a conference paper at ICLR 2024
Table 4: Results object property prediction. We evaluate the quality of the learned representation through object property prediction. Our model consistently performs better than the baselines across different properties and datasets.
Dataset
CLEVRTex
PTR
Super-CT Seper-CCEERR
Position
Shape (t)
Material (1)
Position
Shape (t)
Position
Shape (t)
Material (1)
Property
(4)
(U)
(4)
SLATE+
0.1757
78.72
67.99
0.2218
88.21
0.5397
76.28
68.43
LSD
0,1563
85.07
82.33
0.5999
75.80
0.4372
76.5
69.24
Ours
0.1044
88.86
84.29
0.1424
90.00
0,4262
80.67
71.31
B.6 ADDITIONAL. EVALUATION ON OBIECT PROPERTY PREDICTION
To assess the quality of acquired object representations, we employ object property prediction using the learned representation, following the methodology outlined Jiang et al. (2023): Dittadi et al. (2022). During this process, we train a network to predict the property based on fixed slot representation. The true label for the slot representation is established through Hungarian matching. comparing the mask of slots with the foreground objects. The remaining slots after matching are considered as backgrounds. For predicting properties, we employ a 4-layer MLPs with a hidden dimension of 196. Accuracy is reported for categorical properties, while mean squared error is reported for continuous properties. We assess the models on datasets that include object properties.
The results for object property prediction presented in Table 4 Our model consistently performs better than the baselines across different properties and datasets. Notably, it excels in predicting shape and position, observed in the high segmentation performance depicted in Figure 2 and Table [ Furthermore, our model demonstrates improved performance predicting materials indicating its ability capture local and high-frequency information.
On the Super-CLEEV dataset, despite our model's higher segmentation performance, the mean square error of position remains competitive with other baselines. We attribute this to the challenging nature of the dataset, where scenes often include many small and occluded objects. As a result, both our model and the baselines face increased difficulty in predicting position, leading higher error rate compared to other datasets.
19

---

Published as a conference paper at ICLR 2024
B.7 ADDITIONAL RESULTS ON REAL-WORLD DATASET
To explore the scalability of our novel objective in a complex real-world dataset, we examine our framework in BDD100K dataset Yu al. (020)) which consists of diverse driving scenes. Since the images captured on night or rainy days often produce blurry and dark images, we filter the data to collect only sunny and daytime images using metadata, which leaves about 12k, 1.7k images in the training/validatiin validation set, respectively. Since it has been widely observed that learning the object-centric representation directly on real-world dataset is challenging, we bootstrap our auto-encoding path with off-the-shelf models following Diang et al. (2023). Specifically, we employ pretrained DINOv2 Oquab jct al. (2023) and Stable Diffusion Rombach et al. (2022) for the image encoder and slot decoder in our auto-encoding path, respectively. Instead of using frozen Stable-Diffusion we update key and value mapping layers in pross-attention layers to enhance the overall auco-enc performance following Kumari et al. (2023). For efficient training, we first warm up the auto cento-encoding path for 200k iterations and then train only the surrogate decoder for 140k iterations on top of frozen slot representations, which significantly boosts up the convergence of the surrogate decoder. Finally, optimize our compositional path for 100k iterations. For the baseline, we compare our model trained with only auto-encoding objective for 300k iterations, which converges closely to the Stable-LSD Jiang et al. (2033).
Figure T0 illustrates qualitative results on unsupervised object segmentation. The slot attention masks of our model successfully capture composable instances such as cars, buildings, trees, font hoods, etc. In contrast, the diffusion model trained without compositional objective often divides the objects into multiple slots or encodes multiple objects into slot. For example, the car or truck is frequently divided into multiple masks, and multiple cars are often encoded into a single slot.
To further examine the compositionality of the learned slot representations, we qualitatively analyze the visual samples of composite images in Figure [T similar to Section [.4] We observe that our method successfully generates realistic scenes, modeling complex correlations among objects and environments. appropriately adapts the appearance of newly added/removed objects, their shadow, reflections in the front glass and hood, and sometimes removing the sun. In contrast, the auto-encoding model often fails to achieve faithful composition. For example, in Row 1 of TT the car still appears in the composite image after the removal of the corresponding slot. Also, we observe that removing slots containing partial information of the object often leads to undesirable artifacts in composite images such creating new car in the first example of Row 2, or leaving unrealistic artifacts in the third example of Row 2. In contrast, our model produces natural object-wise manipulation. Moreover, the baseline model often fails to faithfully generate the inserted object as shown in Row 3, while our model tends to maintain the target object. In Row 4, we identify that our model successfully models complex interaction between slots such as removing sunlight changing the reflection of the bonnet in the first image, or changing blurry car into sharp car corresponding to bright weather. In summary, we identify that our novel objective compositionality can help to learn object-wise disentanglement even in complex scenes and helps to model complex interactions among objects.
global illumination change caused by
20

---

Published as a conference paper at ICLR 2024
GT
Ours
Stable- LSD
GT
Ours
Stable- LSD LSD
Figure 10: Qualitative results on unsupervised segmentation in BDD100K.
21

---

Published as a conference paper at ICLR 2024
Source
Source
Source
GT
Output
Output
Output
Ours
Stable LSD
Source
Source
Source
GT
Output
Output
Output
Stable-LSD Ours
Source
Source
GT
Output
Oulput
Output
Ours
Stable- LSD
Source
Source
Source
GT
Output
Output
Output
Ours
Sable-LSD
Figure 11: Qualitative results compositional generation in BDD100k.
22


Reviews:
[{"review_id": "gOoTO8qUec", "reviewer": "Reviewer_Cp4U", "summary": "The paper proposes a new training objective that encourages learning object-centric representation by considering mixtures over objects across images, and empirically explores it in conjunction with a slot attention architecture over CLEVR-like data.\n\n**Update:** Dear authors, thanks for your response, it basically addressed all of my concerns, and I think in particular the additional results on BDD100k strengthen the paper. I would recommend including at least some of them, e.g. some of the qualitative results (a smaller version with less examples of figure 11), in the main paper. Comparison to slot attention is also useful and shows large performance improvement. The comment in the response regarding slot initialization makes it clearer for me now, and I think adding more of it into the paper would be good to make sure it's clear too! Overall, as the response addressed my concerns and presented additional important results, I'm happy to raise my score.", "questions": "- **Baselines**: It would be good to also compare to a vanilla slot attention as a baseline.\n- **Slot Initialization* I didn’t fully understand the intuition why a shared slot initialization should help avoiding bad mixes of objects. It seems to me that learning good combinations of objects could be achieved instead by not sharing the initialization scheme between them and letting them interact across the two images to coordinate valid object compositions. If possible please explain the initialization point further.\n- **Diffusion**: It would be helpful to make it clearer how the denoising diffusion and auto-encoding fit together in the proposed framework. \n- **Super-CLEVR**: The ARI performance drop on super-clevr is particularly large, even though multiShapenet and PTR also include objects with subparts, and the latter even includes more complicated backgrounds. Do you have an intuition of why is that the case? \n- “it decompose” -> “it decomposes”\n- “employed auto-encoding framework” -> “employ an auto-encoding framework”\n- “as architectural/algorithmic bias” -> “as an architectural/algorithmic bias”\n- “atttention” -> “attention”\n\nOverall, while the paper is well-made, I'm in between 5 and 6 due to mentioned weaknesses above. But I'll be glad to raise my score if results will be presented over realistic data.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Evaluation**: Multiple types of experiments are conducted, including quantitative/qualitative unsupervised segmentation, ablation studies, and analysis of robustness. The approach is shown to be robust to factors such as number of slots, as well as settings of the encoder and decoder, aspects which slot attention methods are usually sensitive to.\n- **Research Context**: The authors do a good job in providing the relevant research context as well as model’s preliminaries, pointing to the limitations of some of the prior works.\n- **Presentation**: The writing quality is good and the paper is clear, well-organized and easy to follow. The first parts of the model section are particularly clear, and both the diagrams and visualizations help understanding the approach and its behavior. The supplementary is also good, providing implementation details and additional quantitative and qualitative results.", "weaknesses": "- **Synthetic Data & Scalability**: The experiments are performed over synthetic data only. I recommend exploring scalability to real-world data too. This is especially important since it is unclear to me whether such an approach would scale well to more diverse real-world data, where there are correlations between object occurrence as well as appearance. I don’t know for sure but it might be the case that for realistic data the loss could damage the model’s learning compared to standard auto-encoding, by encouraging it to unlearn important object correlations. For instance, even for the images at figure 1, the lighting of the mixed objects does not fit the context, potentially making the model less aware of the impact of lighting conditions on the object’s appearance. For this reason I believe it is really critical to study this objective in the context of realistic data too.\n- **Compositional Synthesis**: The results about compositional synthesis aren't the most compelling. I would like to see object mixing in cases that involve more interesting interactions among the objects (like reflection, more substantial occlusions, and matching of the appearance between the object and its environment). It does perform better than the baselines so that’s still an improvment.\n- **Novelty & Related Works**: The specific proposed idea is novel, but object mixing as a form of regularization isn’t. For instance,  it would be good to cite and discuss the “Object Discovery with a Copy-Pasting GAN” paper as a related prior work that also uses mixes over image to encourage object discovery.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ThN0XYiQqO", "reviewer": "Reviewer_6966", "summary": "The authors propose a scheme for encouraging compositionality in slot-based Object-Centric Learning (OCL) models by introducing a compositional objective alongside the standard auto-encoding reconstruction loss. In particular, they enforce that mixing subsets of slots from two separate scenes should result in a \"valid\" image, as determined by a generative prior which is trained through the auto-encoding process - this is accomplished by using a diffusion model as the decoder. To enable fast-decoding and cheaper evaluation of gradients for the encoder through the compositional path, they also train a transformer-decoder on the autoencoding tasks.\n\nThey perform extensive experiments and ablations, demonstrating the efficacy of their method against other state-of-the art methods, both in terms of performance and robustness to hyperparameters such as slot-count and decoder size (which are sensitive in many OCL methods). To the best of my knowledge, the idea of encouraging compositionality of slots through a valid-mixing objective is novel and forms a valuable contribution to the OCL community.", "questions": "Overall the work is very interesting and comprehensively explained, and I was left only with minor questions:\n1) *Why background slots do not conflict*: Could the authors explain whether / why, if not, background slots conflict when sampling slots with the random mixing strategy? In this case 1) is there any reason to expect canonical slot ordering, such that two background slots are unlikely to be sampled or 2) any reason why if two background slots are sampled, this does not lead to invalid scenes?\n2) *On the relative contributions of Regularization+Shared Slot Initializations (Row 2) and the compositional prior (Row 1) in Table 2*: It is Interesting that Row 2, is roughly as effective as Row 1. Was there a significant difference in the qualitative behaviour of the models in these two settings; i.e. did Row 1 yield \"better\" slots, but lead to worse metrics for some other reason - or can we conclude that the bias towards compositional slot representations in both of these settings was comparably strong? (which would be somewhat surprising!). Perhaps the decoder receives a stronger learning signal in Row 2, and in Row 1 the generative prior acting by itself is constrained because the decoders don't learn as well, and so provide a weaker compositional signal?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper's methodology is well described with helpful figures, and the experiments are comprehensive. Notably, the experiments on parameter robustness in sec 5.2, as well as the qualitative results shown alongside experiments with slot mixing strategy in the appendices, provide a compelling case for the benefits of this method against other SoTA methods, beyond the mere improvement of performance.", "weaknesses": "There are no major weaknesses with the methodology or evaluations. The presentation is clear in most places, but there are many grammatical mistakes. It is recommended that these be fixed through the use of automated grammar checkers or proof-reading by a native speaker.\n\nSome minor notes on clarification:\n* On the first reading, it was not entirely clear how the one-shot decoder and diffusion decoders were being trained (i.e. whether one, or both, were being used in the auto-encoding path) - this is made clear at the start of sec 3.3 \"in auto-encoding path [...] two different decoders [...] are trained\", but it would be good to make it more explicit in prior paragraphs when the relevant decoders are introduced and their training is discussed.\n* Related to the above, it would be good to make it clear in the figure that *both* decoders are trained/used on the auto-encoding branches, for example by putting \"(both) Decoder\" on the branches", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aB1fyYZ9bM", "reviewer": "Reviewer_f9fX", "summary": "The paper introduces a generative objective to regularize the learned representation of a Slot Attention model. In particular, the generative objective ensures that decoding a composition of slots from different images results in a realistic image. The additional regularizer trades off reconstruction quality for better compositionality of the learned representation.", "questions": "May I recommend a bar plot for Figure 3? The current plot based on colored Xs is quite hard to read. Moreover, a bar plot would allow to show the standard deviation of the results over multiple independent runs, which is important to strengthen the claims made in the text.\n\nCan you provide non-segmentation based evaluation metrics? They can be the same as in Jiang et al. (2023) for example.\n\nIn Jiang et al. (2023) the compositional generation appear to be much stronger that what is shown in Figure 4. It seems to me that for the proposed method the authors chose a source image where all objects are well separated and therefore the composition looks good. Whereas the segmentation in LSD and SLATE+ on the same image is poor and therefore the composed image looks bad. Can you clarify in the caption if the examples were cherry-picked? Can you find another image where all 3 methods segment correctly the objects that you want to compose? This way the comparison would be fairer.\n\nTypo: \"is not contrained to\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The authors identify a weakness in the standard reconstruction objective of Slot Attention and similar papers: the reconstruction objective does not explicitly encourage compositionality of the learned representation. This observation calls for a generative objective that ensures that composition of slots can be decoded into realistic images. Instead of implementing this with a GAN discriminator, which would require an additional component, the authors follow the approach of Poole et al. (2022) where the diffusion model is reused as a generative prior.\n\nThe idea of using a cheap surrogate one-shot decoder to approximate an expensive denoising process is quite clever and surely helps speeding up training.\n\nThe qualitative results on compositional generation are convincing. I would like to see if the method would work on real-world images like the ones in COCO.", "weaknesses": "In the proposed method, the number of losses, regularizers, and training tricks to balance is high. I would appreciate a more thorough ablation study where each component is isolated and tested. The combinations in table 2 are valid but do not cover all possibilities.\n\nI would appreciate additional evaluation metrics that are not based on segmentation. Learning a good object-centric representation means much more than achieving good segmentation. It is important to show that the learned slots are useful for downstream tasks such as counting, tracking, property prediction, etc. This way the claim made in the introduction that \"[the method] significantly boosts the overall quality of object-centric representations\" can be supported. My rating on \"soundness\" would be higher if the authors had included such experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eMmXvOukWV", "reviewer": "Reviewer_H2Bo", "summary": "This paper focuses on the compositional ability in object-centric learning. Existing works mostly rely on auto-encoding training paradigm and may sacrifice the object disentanglement. While this work explicitly introduces an object composition path in addition to the original reconstruction objective, and employs generative prior to validate the rendered object compositions. The experiments show that the proposed method achieves better resutls in object disentanglement and enhances the robustness to hyper-parameters.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well motivated. Compositionality is a vital property in object-centric perception and generation. The authors conclude the potential weakness in existing auto-encoding based object-centric learning methods and introduce an object composition path for explicit optimization on composition.\n2. The method is simple and intuitive. It uses generative prior to help validate the rendered object composition from mixed slot representations and guides the object disentanglement and composition learning.\n3. The experiments show the improvement on object-centric benchmarks. And the proposed method is robust to the number of slots, which is a significant hyper-parameter in object-centric learning and difficult to determine.", "weaknesses": "1. Most experiments are conducted on the synthetic objects, more experiments on realistic complex scenes, e.g., COCO, are desired.\n2. More compared methods should be included, e.g., DINOSAURE [1].\n3. There are some recent works focusing on the binding ability of the slots to specific object types or properties, e.g., [2]. Is it possible to employ the binding ability to enhance the interpretablity of slot mixing and further enahnce the composition ability?\n\n[1] Seitzer et al. Bridging the gap to real-world object-centric learning. ICLR 2023.\n[2] Jia et al. Improving Object-centric Learning with Query Optimization. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)