Object-Centric Slot Diffusion
Authors: Jindong Jiang
Fei Deng
Gautam Singh
Sungjin Ahn
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Object-Centric Representation Learning
Diffusion Models
Unsupervised Representation Learning
TL;DR: We propose the Latent Slot Diffusion which combines an object-centric encoder with diffusion decoders. It achieves unsupervised learning of object segmentation, compositional generation, and image editing, surpassing the state-of-the-art models.

Abstract: The recent success of transformer-based image generative models in object-centric learning highlights the importance of powerful image generators for handling complex scenes. However, despite the high expressiveness of diffusion models in image generation, their integration into object-centric learning remains largely unexplored in this domain. In this paper, we explore the feasibility and potential of integrating diffusion models into object-centric learning and investigate the pros and cons of this approach. We introduce Latent Slot Diffusion (LSD), a novel model that serves dual purposes: it is the first object-centric learning model to replace conventional slot decoders with a latent diffusion model conditioned on object slots, and it is also the first unsupervised compositional conditional diffusion model that operates without the need for supervised annotations like text. Through experiments on various object-centric tasks, including the first application of the FFHQ dataset in this field, we demonstrate that LSD significantly outperforms state-of-the-art transformer-based decoders, particularly in more complex scenes, and exhibits superior unsupervised compositional generation quality. In addition, we conduct a preliminary investigation into the integration of pre-trained diffusion models in LSD and demonstrate its effectiveness in real-world image segmentation and generation. Project page is available at https://latentslotdiffusion.github.io

Object-Centric Slot Diffusion
Jindong Jiang
Fei Deng
Rutgers University
Rutgers University
j iindngg- jwingaintterss.eee. edu
fei den.onutgers. edu
Gautam Singh
Sungjin Ahn"
Rutgers University
KAIST sungjin awn@kaist.ccmcmcm .c.t..cc.. ac. kr
singh. gautam@uutgers.ee. edu
Abstract
The recent success of transformer-based image generative models in object-centric learning highlights the importance of powerful image generators for handling complex scenes. However, despite the high expressiveness of diffusion models in image generation, their integration into object-centric learning remains largely unexplored in this domain. In this paper, we explore the feasibility and potential of integrating diffusion models into object-centric learning and investigate the pros and cons of this approach. We introduce Latent Slot Diffusion (LSD), novel model that serves dual purposes: it is the first object-centric learning model to replace conventional slot decoders with latent diffusion model conditioned on object slots, and is also the first unsupervised compositional conditional diffusion model that operates without the need for supervised annotations like text. Through experiments on various object-centric tasks, including the first application of the FFHQ dataset in this field, we demonstrate that LSD significantly outperforms state-of-the-an transformer-baa decoders, particularly in more complex scenes, and exhibits superior unsupervised compositional generation quality. In addition, we conduct a preliminary investigation into the integration of pre- pre-trained diffusion models in LSD and demonstrate its effectiveness in Project page is available at fttp:::://aattttsooofffffioeeeee
image segmentation and generation. github..ii 10
Introduction
The fundamental structure of the physical world is compositional and modular. While this structure naturally revealed in some data modalities like language the form tokens or words, in other modalities such as images, it is elusive how one may discover this structure. However, this representational compositionality and modularity is essential for various applications that require the systematic manipulation of knowledge pieces to achieve high-level cognitive abilities. This includes reasoning [46]] 5) causal inference [68], and nat-of-distribution generalization
genirraitioonnnnn 22
Object-centric learning 27 aims to discover the latent compositional structure from unstructured observation by learning to bind relevant features, thereby forming useful tokens in an unsupervised way. For images, one of the most popular approaches is to auto-encode the image via the Slot Attention ( encoder. Slot Attention applies competitive spatial attention to partition the image into separate local areas and then obtain a representation, called slot, from each area. Then, decoder generates the image from the slots with the aim of minimizing the reconstruction error. Due to limited capacity and competition between slots, each slot is encouraged to capture reusable and compositional entity, such as an object.
"Correspondence sungj ahnokaist.co
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The primary challenge that remains in the framework of unsupervised object-centric learning is making work for complex naturalistic scene images. Until recently, most object-centric models have adopted a special type of decoder known as the mixture decoder. While weak slot-wise decoder [8]] predominantly employed in this mixture decoder due to its efficacy in promoting the emergence of object-centrie representation in relatively simple scene images, further studies [70] 73] have shown that this strong prior can make handling complex naturalistic scene images more challenging. Contrary to conventional belief, Singh et al. [70] recently proposed departing from this low-capacity mixture-decoder approach and suggested to use an expressive transformer-basee autoregressive image generator in object-centric learning 770 733. It shown that increasing the decoder capacity is crucial for handling complex and naturalistic scenes in this framework [8
84 71].
The success of transformer-bast based image generative modeling in object-centric learning naturally leads to the question: can diffusion models, another pillar of modern deep generative modeling known for their highly expressive generation capabilities, also benefit object-centric learning? Diffusion models performance a variety of image generation tasks [6] 58 65
75 31 are based on stochastic denoising process and have demonstrated impressive
[14
63 66 56] sometimes surpassing
transformer- based autoregressive models. Moreover, diffusion models possess unique modeling autoregressive models cannot provide [75]. However, despite their
capabilities that transformer-bai potential, the applicability of diffusion models to unsupervised object-centric learning remains largely unexplored. Consequently, it is crucial to examine the feasibility of this approach and to identify the associated benefits and limitations.
In this paper, we address this question by introducing a novel model called Latent Slot Diffusion (LSD). The LSD model can be interpreted from two perspectives. From the perspective of object- centric learning, LSD can be viewed as the first model substituting conventional slot decoders with a conditional latent diffusion model, in which the conditioning is object-centric slots provided by Slot Attention. From the diffusion model perspective, our approach is the first unsupervised compositional conditional diffusion model. While traditional conditional diffusion models [6 63 65 [2 require supervised annotations, such as text description of an image for compositional generation, LSD is diffusion model that enables the construction of such compositional descriptions in terms of visual concepts extracted from images through unsupervised object-centric learning.
In our experiments, we evaluate the proposed model across various object-centric tasks, including unsupervised object segmentation, downstream property prediction, compositional generation, and image editing. We show that the LSD model delivers significantly superior performance compared to the state-of-the-art model, namely, the transformer-bel based autoregressive generative model. A notable attribute of the proposed model that LSD's performance advantage over autoregressive transformers increases the scene complexity increases. In particular, LSD enables exploring, for the first time, the applicability of object-centric model to the FFH dataset [4I, a collection of high-resolution and high-qualit face images that surpasses the generative capabilities of existing object-centric models. Additionally, we discuss the overfitting issue faced by LSD on very simple scene images, such as those in the CLEVR dataset (37), and offer suggestions for addressing this problem.
2 Latent Slot Diffusion
In this section, we outline the auto-encoding framework of our proposed model, Latent Slot Diffusion beginning with our object-centric encoder and subsequently describing our proposed decoder.
LSD
2.1 Object-Centric Encoder
Given an input image E X our object-centric encoder seeks to decompose and represent it as a collection of N vectors or slots S E NDD where each slot (denoted as s, E R) is encouraged to represent compositional entity in the image. For this, we adopt Slot Attention, architecture that is also used in the current of object-centric learning approaches [53 70 43 now describe how Slot Attention works in model.
We
In Slot Attention, we first encode the input image x as a set of M input features E E R Mx x Daja via a backbone network fo
phackbone i.e., E- f
phackbone (x). The network f
phackbone is implemented as CNN
"Code will be made available at http://///tttttth/ooo////////a/a

---

Input image
Generated Image
Compose dloss
Slot Attention
Visual Concept Library
Image Encoder
Image Decoder
Z+
Denoising é
c,
Denoising Network
Network
E1
Minimize Squared Error
N(D, !)
Now Sample
Add Noise
Training
Compositional Image Generation
Figure I: Method. Left: In training, we encode the given image as image latent and as slots. We then add noise to the image latent and we train denoising network to predict the noise given the noisy latent and the slots. Right: Given the trained model, we can generate novel images by composing slot-based concept prompt and decoding it using the trained latent slot diffusion decoder.
(detailed in Appendix (.2) whose final output feature map flattened to form set. Next, the features E are grouped into spatial groupings and the information each grouping aggregated to produce slot. The grouping is achieved via an iterative slot refinement procedure. At the start the refinement procedure, the slots S are filled with random Gaussian noise. Then, they are refined via competitive attention over the input features, where the N slots as the queries and the M input features act as the keys and values. The queries and keys undergo dot-product to produce N x M attention proportions. Next, on these attention proportions, the softmax function applied along the axis N to produce attention weights A that capture the soft assignment of each input feature to slot. Then, for each n, all input features are sum-pooled weighted by their attention weights An.l...., An,M to produce attention readout E DR. These steps can be formally described as follows:
((()) K(E)T VD
An.m Σ
M
A = sof tmax
An.m
u =
Σ m=1
D Em )mm)n.m
Anms
1m=1 Am.m
where, g, k are linear projections that map the slots and input features to common dimension D. Using the bottom-up information captured by the readout u, the slots are updated by an RNN S., f JRNN (sn, u,). In practice, competitive attention and RNN update performed iteratively several times and slots from the last iteration are considered the final slot representation S.
2.2 Latent Slot Diffusion Decoder
In this section, we describe our proposed decoding approach called Latent Slot Diffusion Decoder or LSD decoder for reconstructing the image given the slot representation S. The design of the LSD decoder takes advantage of the recent advances in generative modeling based on diffusion 66 31). An overview of our decoding approach provided in Figure
2.2.1 Pre-Trained Image Auto- Auto-Encoder
One of the key design components of the LSD decoder is pre- mre-trained auto auto-encoder (AE) 21 63 which provides a way to map an image x to a diver-dimensional latent representation Zu via an encoder fAE This enables LSD to reduce the computational burden of reconstructing high-resolution images by using the lower-dimensional latent Zn as an intermediate reconstruction target. It also allows to later obtain the original resolution image without compromising image content and fidelity by decoding the latent Zn using the AE decoder gø These can be summarized as:
Zo = AA (x),
= go (zo).
where zo E HAs HA WAR DAE and x E R XC
In our model, we employ an auto-encoder pre-trained on OpenImages
We use the pre-trained weights from http://oonnee-laabccf/ieeeeeeee -ttp:///aaatttttiiiiuiioooiaa/a

---

2.2.2 Slot-Conditioned Diffusion
In LSD, we leverage diffusion modeling to reconstruct the image latent conditioned on the slots S. This modeling approach has been primarily explored in supervised contexts, egg. for text-to-image generation in Latent Diffusion Models (LDM) [63]. However, unlike LDM, in this work, instead of conditioning the decoder on embeddings of supervised labels, we condition it on slots where the process of obtaining the slots themselves, i.e. Slot Attention, is jointly trained with the decoder without supervision. Following LDM, our decoder works by training decoding distribution py (2 (S) to maximize the log-likelihood log pe (z)|S) of the image latent Zo given the slots S. This decoding distribution pe (zo|S) is modeled as step denoising process:
po (S) = p(ZT)
pa t,,t dZ I=I,....
II
where p(zr) NO, pa (2--1/zz; L S)
3) is one-step denoising distribution, and zz, -1. ...... Zo is a sequence of progressively denoised latents. The one-step denoising distribution po (e [zz, S) parametrized via neural network g/5D in the following manner:
po Z4, S) =N (
(v/ (vac
ZE
V1
Be
Ctt
:
3.1
where é = = L.DD (zt: 1 S)
,1,..., Br is a linearly increasing variance schedule, 0₁ = 1 Be. and o₁ II B:).
Sampling Procedure. To sample ~ (o0(2))S) we adopt an iterative denoising procedure as in [1]. The sampling process starts with latent representation zt ~ (O, I) filled with random Gaussian noise. Next, conditioned the slots, we denoise it ' times by sampling sequentially from the one-step denoising distribution 2;-1 pe (2:z111zt: f, S) for f = T, .,...,, 1. This produces a sequence of latents 2r, 21-1..... Zg that become progressively cleaner. Finally, Zu can be considered as the reconstructed latent representation.
Training Procedure. Following LDM [63], the training of pe (zo can be cast to simple procedure for training gb as follows. Given an image x, its slot representation S, and its image latent zo. we first randomly choose noise level E (1...., T) from uniform distribution. Given the I, we corrupt the clean latent Zg and obtain a noised latent 2 as follows:
24 = 20, + V1 CigE2
where & ~ N(O.), ā = III - B.)-
The noised latent 2, is then given input LD along with the slots S and denoising time-step to predict the noise c, The network ..S. is trained by minimizing the mean squared error between the predicted noise é; and the true noise e::
E 9g
LSD (zz, S)
L(),0) = =Ilē: Ex11?
2.2.3 Denoising Network
We implement the denoising network g/-SD as a variant of the conventional UNet architecture adapted incorporate slot-conditioning. Our denoising network consists of stack of L layers where each layer is a UNet-style CNN layer followed by slot-conditioned transformer:
h CNN', (h)-1, basp(()) !),
h Transformer (h, Pt, cond=S),
where ho 2, is the input, h h.,..., hr are the hidden states and é; hr, is the output.
CNN Layers. Following UNet [64], the convolutional layers CNN). CNN downsample the feature map via the first & layers and then upsample it back to the original resolution via the remaining layers. Following standard UNet, these CNN layers also receive inputs via skip connection from an earlier layer denoted by skip((). This network design has also been explored in LDM 663
Alot-Conditioned Transformer: The role of the transformer to incorporate the information from the slots into the UNet- based denoising process. For this, in each layer the intermediate feature map h produced by the CNN layer is flattened into a set of features. To this, positional embeddings pr are added and the resulting features are provided as input to the transformer. Within the transformer, these features interact with each other and with the slots S, thus incorporating the information from the slots into the denoising process. The transformer output is then reshaped back to feature map h

---

3 Compositional Image Synthesis
In this section, we describe how a trained LSD model can be used compose and synthesize novel images. The conventional approach to composing novel images is commonly supervised and rely on text prompts. This involves using words from vocabulary to create sentence, which is then given to text-to-image model to synthesize the desired image [62] 61 65, 63 [58]. However, in fully unsupervised setting like ours, we first need to build a library of visual concepts by simply observing large set of unlabelled images. Then, similarly to composing sentence prompt using words, we compose concept prompi by selecting concepts from the visual concept library. By providing this concept prompt to the LSD decoder, we can then synthesize a desired novel image. This approach of unsupervised compositional image synthesis was also explored in 70
Unsupervised Visual Concept Library. To build library of visual concepts from unlabelled images, we first take a large batch of B images XI.... XB. We then apply slot attention to obtain slots for these images .i.... Sg. Next, we gather all these slots into single set S and perform K K-means on it. The K-means procedure assigns each slot in S to one of the K clusters. We consider the set of slots assigned to a k-th cluster as visual concept library VE. With K as the number of clusters, this method results K visual concept libraries Vi,..., VK. Our experiments will demonstrate that this basic K-means technique can generate semantically meaningful concept libraries. For example, on a dataset of human face images like FFH [4]] the K libraries correspond to practical concept categories such as hair style, face, clothing, and background.
Novel Image Synthesis. Given libraries Vi,..., VK. we can compose a concept prompt Scoposse by picking K slots, each from the corresponding k-th library, and stacking them together:
compose Sk)
where Sş ~ Uniform(Va)
We then give the composed prompt Scompose to the LSD decoder to generate the image latent: Zoompose ~ po (2(o)SScomposa Applying the image decoder on Zoompos: generates the desired novel scene image Xcompose Do A (. For instance, in the FFHQ dataset, the concept prompt can be a collection of chosen hair style, face, clothing, and background. Decoding this prompt would generate a face image that conforms to this prompt.
4 Related Work
Unsupervised Object-Centrea Learning. Object-centric representation learning aims to decompose multi-object scenes into meaningful object entities. A common approach this is by auto uncoden....ng. In this line, the focus has been to design an appropriate decoder that supports good decomposition. The most widely used decoders include the mixture-decoder
(6]
25
53
26
19
78
17
18
39
89 80)
spatial transformer decoder (20 12 50 35
13
9
36
49
72
86
Neural Radiance Fields (NeRF)
23 energy- based models [66] and complex- valued functions [55]. Among these, the mixture decoder has been shown to struggle on complex scene images 70 [73 while the spatial transformer decoder requires careful tuning hyperparameters which limits their applicability in realistic scenes. Neural Radiance Fields require camera poses, and the learning setting involves 3D scenes unlike ours. Another line of work seeks object-centric scene decomposition without reconstruction. This includes works such as [69 82
777 83 74, transformer decoder 70 73
71
84
67
28 79
[4| 87).
However, unlike ours, these approaches lack the ability to generate images. Preceding object-centric learning, several works pursued disentanglement within single-vecto representation of the scene and use it for compositional image generation [T] 30 42
45 10) However, lacking spatial binding
mechanisms, these methods struggle in multi-object scenes
25 unlike ours.
Diffusion Models. Diffusion models (DMs) are a recent class of generative models that can produce high-quality images by reversing a stochastic process that gradually adds noise to an image [3]] 76).
DMs have been applied to various tasks in computer vision, such as class-conditioned generation, text- to-image generation, image editing, super-resolution, and inpainting [14 33 61
58 65
56
32 66]
Recently, (63] proposed Latent Diffusion Models (LDM). By virtue of operating on a low-dimensional latent space, LDM reduces the computational demands of DMs significantly. [51] introduced method for multi-object scene generation by combining signals from multiple text-conditioned denoising networks. However, to achieve controllable generation, many of these existing DMs require additional labels such as text descriptions to train and control the generation process. contrast, our model can generate images compositionally using object concepts directly extracted from images. Moreover,
5

---

SLATE LSD (Ours) Duiinets
SLATE
CLEVRTex
MOVi-E
FRHQ
Figure 2: Visualization of Unsupervised Object Segmentation. We show visualizations of predicted segments on CLEVR, CLEVRTex, MOVi-E, and FFHQ datasets.
DMs have also been used for representation learning. 4 demonstrated that the intermediate features of learned DM are useful representations for label-efficient semantic segmentation. [60] introduced an image encoder that compresses an image into a latent representation, jointly trained with the denoising objective. Nevertheless, these representations
unstructured and not modular like ours.
Concurrent Research. A concurrent study in 85 also explores similar idea of object-centric learning using DMs and its downstream applications such as the image editing tasks akin to our experiments. Just like LSD,, they employ the Latent Diffusion Model (LDM) as the slot decoder. However, unlike our method, [85) independently train a latent encoder for each dataset, whereas LSD employs a pre-trained latent encoder shared across all input data.
5 Experiments
extensively evaluate our proposed Latent Slot Diffusion (LSD) model on various object-centric tasks, including unsupervised object segmentation, downstream property prediction, compositional generation, and image editing. As will be shown, our model demonstrates substantial improvements the oftte-of-thh-art including FFHQ [4] which has been beyond the generative capability of object-centric models. In addition, also include preliminary exploration into the potential of LSD using pre-trained diffusion models, discussing its performance in both multi-object datasets and unconstrained real- world images in Section 5.4 and Appendix
multiple challenging datasets with complex texture and background,
Datasets. We evaluate our model on five datasets. Four of them are synthetic multi-object datasets- CLEVR 37), CLEVRTex 40). MOVi-C, MOVi-E (24). They present increasing levels of difficulty- CLEVRTex adds texture to objects and backgrounds, MOVi-C uses more complex objects and natural backgrounds, and MOVi-E contains large numbers of objects (up to 23) per scene. Furthermore, explore the applicability of object-centric models to FFHQ [41), a dataset of high-quality face images. Unlike previous works in this line that only investigate low-resolution images, e.g., up 128 x 128, we use a resolution of 256 x 256 for all datasets our experiments.
We compare our model against SLATE, the state-of ofeee--r-tee--tttt object-centric learning and unsupervised compositional image generation approach. We use its improved version [73] which is more robust in complex scenes. For fair comparison with LSD that leverages pre-trained auto-encoder, we also develop a variant of SLATE denoted as SLATE+ where its low-capacity dVAE (70 is replaced with pre-trained VQGAN model 21 For all models in this work, we use OpenImages-prottee pretrained auto-encoders [63]
Baselines.
5.1 Object-Centric Representation Learning
we use unsupervised object segmentation and downstream property prediction to evaluate the object-centric learning capability and representation quality. Within each dataset, all models we compare use the same number of slots.
In line with previous work 53

---

Table I: Segmentation Performance and Representation Quality. Left: We evaluate the segmenta- tion quality and report mBO, mloU, and FG-ARI scores across various datasets and baselines. Right: We measure the representation quality by learning a probe to predict object properties given frozen slots. For position and 3D bounding box, we report MSE. For shape, material, and category, we report accuracy. The results averaged over three random seeds, and the complete table, including standard deviations, can be found in Figure in appendix.
(a) CLEVRTex
Segmentation
SLATE
SLATE
LSD (Ours)
Representation
SLATE
SLATE*
LSD (Ours)
mBO (f)
50.88
54.90
63.93
Shape (t)
74.24
71.63
80.23
mloU (1)
49.54
52.96
62.52
Material (1)
69.73
63.61
75.56
FG-ARI (t)
44.19
70.71
64.41
Position (1)
1.28
1.26
1.13
(b) MOVi-C
Segmentation
SLATE
SLATE
LSD (Ours)
Representation
SLATE
SLATE
LSD (Ours)
mBO (t)
39.37
38.17
45,57
Position (L)
1.37
1.28
1.14
mIoU (1)
37.75
36.44
44.19
3D B-Box (L)
1.48
1,44
1.44
FG-ARI (1)
49.54
52.04
51.98
Category (1)
42.45
45.32
46.11
(c) MOVi-E
Segmentation
SLATE
SLATE
LSD (Ours)
Representation
SLATE
SLATE
LSD (Ours)
mBO (f)
30.17
22.17
38.96
Position (L)
2.09
2.15
1.85
mIoU (1)
28.59
20.63
37.64
3D B- Boxo (U)
3,36
3.37
2.94
FG-ARI (1)
46.06
45.25
52.17
Category (1)
38.93
38.00
42.96
Unsupervised Object Segmentation. Following 53 69 to measure segmentation quality, we
report the foreground adjusted rand index (FG-ARI), the mean intersection union (mloU), and the mean best overlap (mBO). These metrics are computed based on the attention masks generated by Slot Attention. Specifically, utilize the attention weights described in Section 2.1 where attention
((S))k(E))
weights A are computed A softmax
VD
Our results in Table that LSD is particularly strong in visually complex scenes. For example, on the most challenging MOVi-E dataset, LSD outperforms the strongest baseline by >8% in mBO, >9% in mloU, and >6% FG-ARI. On CLEVRTex and MOVi-C featuring complex textures, LSD also achieves > and >6% gain respectively, in both mBO and mloU. We visually show the superior segmentation quality of LSD in Figure 2
LSD obtains tighter object boundaries, less object most noticeable on CLEVRTex and MOVi-E, where the basclines over-segment the objects more frequently, or exhibit common failure mode that divides images into approximately uniformly distributed block masks.
splitting, and cleaner background segmentation. The advantages
We also note that the FG-ARI score sometimes not an ideal metric for evaluating segmentation quality, because it only considers the foreground pixels and disregards the correctness of the mask shape, as also highlighted by [19 [40] In the MOVi-E experiment, even though SLATE* partitions the images arbitrarily into uniform patches, shown in Figure 2 it still achieves FG-ARI score comparable to SLATE, which produces significantly more reasonable object masks. This highlights the need for additional metrics, such the mloU score that we also measure for evaluating the segmentation quality in this work.
53), we evaluate the quality of the learned object- centric representations through downstream property prediction. Specifically, for each property, we train network to predict the property given a frozen slot representation input. The correspondence between the slot and its true label is determined by Hungarian matching (44) using masks. We linear heads and 2-larer MLP as the prediction networks for discrete and continuous values, respectively. We report prediction accuracy for discrete properties (e.g., shape and material), and mean squared error for continuous properties (e.g., position).
Downstream Property Prediction. Following 15
As shown in Table L.SD is competitive with or better than the strongest baseline on CLEVRTex, MOVi-C, and MOVi-E, which is consistent with our finding in unsupervised object segmentation that
7

---

Table Compositional Image Generation. On various datasets, we generate images using object slots randomly sampled from the dataset. We compare the fidelity of the generated images via the FID score. The lower the FID score, the better the fidelity. We find that our model produces the best fidelity scores compared to the baselines.
FID
SLATE
SLATE
LSD (Ours)
CLEVRTex
105.83
69.23
29.53
MOVi-C
170.83
148.27
69.12
MOVi-E
169.32
126.51
64.76
FFH
112.38
98.76
27.83
SLATE LSD (Ouns)
SLATE"
CLEVRTex
MOVi-E
FFHQ
Figure 3: Compositional Generation Samples. We qualitatively compare the quality of image samples generated by our model with the baselines. We observe that LSD provides significantly higher fidelity and more clear details compared to the other methods.
LSD shines in visually complex scenes. For example, LSD obtains >4% gain in predicting object category on MOVi-E and object material on CLEVRTex.
5.2 Compositional Generation with Visual Concept Library
Like text-to-image generative models, LSD is able to take unseen slot-based prompts at test time and compose new images. Unlike cext-to-image models, however, LSD obtains the compositional generation ability solely from images without relying on additional supervision like text inputs, As described in Scction we first build a concept library. Then, we sample one slot representation from each visual concept library and concatenate them into sequence form slot-based prompt. We then feed the Alot-based prompts to the LSD decoder to generate the images. This produces scenes with novel object layouts and faces with unseen attribute combinations.
We report in Table 2 the FID score 29 as a measure of the compositional generation quality. Following standard practice [14], we compute the FID score using 2K generated images and the full training dataset. Across datasets, LSD achieves significantly better FID scores than SLATE and SLATE* We further demonstrate the superior compositional generation quality of LSD in Figure 3 We observe that on the more challenging MOVi-E and FFHQ datasets, LSD generates images with substantially more clear details and better coherence. In contrast, SLATE is limited by its decoder and produces images with missing details the objects or human faces. While the details can be improved by the pre- pre-trained auto-encoder in SLATE* some samples still exhibit severe distortions.
5.3 Slot-Based Image Editing
In addition generating new images from randomly sampled slot-based prompts, LSD also allows editing existing images by directly modifying their slot representations. Our experiments showcase LSS''s capability in slot-based image editing, including object removal, single-object segmentation, object insertion, background extraction, and background swapping. Remarkably, we demonstrate, for the first time in object-centric generative models, the ability to perform face editing real-world images, illustrated in Figure
8

---

Original
Removal
Extract
Insert
BG
BG Swap
Facc Replacement
Sautee
Source
Figure 4: Slot-Based Image Editing. Left: Our model demonstrates image editing capabilities, including object removal, extraction, insertion, background extraction, and swapping. Right: In the FFHQ dataset, we showcase face replacement by combining face slots from Source-B images with hairstyle, clothing, and background slots from Source-A images.
We perform slot manipulation the CLEVRTex dataset, including object removal, single-object extraction, object insertion, background extraction, and background swapping. To remove an object extract the background, discard the relevant slot all object slots, respectively. Our findings indicate that with near-perfect object decomposition, removing an object can be achieved by simply discarding its slot. Additionally, the background image be rendered from single background slot, even without single-slot conditioning during training. In single-object extraction, we render an object with the same background using the respective object and background slots. For object insertion and background swapping, we split the image into top and bottom pairs (Figure and introduce slot from another image or swap background slots. This demonstrates that an image's background can be entirely altered while preserving its original objects.
We further explore face replacement the FFHQ dataset. L.SD decomposes each image into four slots, corresponding to face, hairstyle, clothing, and background. Our results show that by replacing able coherently change the image while maintaining the
the face slots of the images, hairstyle, clothing, and background. The resulting images look realistic, suggesting that LSD can effectively blend various attributes even when given novel combinations.
PIP
Input
Slots
Sample 1
Sample Sample 3
Sample Sample 5
Figure 5: Unsupervised Object-Centrie Learning and Conditional Generation on Real-World Images with Pre-Trained Diffusion Models. Each image sample is conditioned on the same set of learned slots and with a different initial noise map in the denoising process. We cfg 1.3 for all samples. Additional details are available in Appendix C

---

5.4 Pre-Trained Diffusion Models and Real-Worke Object-Centric Learning.
In recent studies of DMs, much attention has been given controllable image generation using large- scale pre-trained diffusion models such as Stable Diffusion [47] 57 88 One intriguing question is whether such pre-trained diffusion models can facilitate the learning of object-centric representation. In this regard, we present a preliminary investigation in Appendix C where we test two LSD variants utilizing the pre-trained Stable Diffusion. Our findings indicate that (1) although the pre-trained Stable Diffusion model has not been trained on any multi-object datasets, it still provides a learning signal for reasonable object-centric learning, (2) with the strong generative capabilities pre-trained DMs, LSD can scale up object-centric learning to real-world images; however, we also note the part-whole ambiguity issue in slot-based object learning in real-world scenarios, and (3) remarkably, the propose LSD variant achieves conditional generation with unconstrained real-world objects by applying classifier-fres guidance [33] to the learned slot, which also provide a visualization of the semantic information captured in the learned representations. We show some image samples in Figure
5 please refer to Appendix for additional details. We believe these preliminary attempts to be
valuable explorations toward large-scale object-centric learning and hope our finding will inspire future research in this area.
6
Discussion
LSD on Simple Images. While LSD shows significant gains in complex naturalistic scenes, our preliminary investigation suggests somewhat diminished performance in terms of segmentation and representation when the dataset consists of only visually simple and monotonous scene images like CLEVR. We conjecture that LSD may suffer from overfitting problem when its high expressiveness encounters visually simple images. To address this issue, additional complexities can be introduced during training, such integrating images from other datasets. In Appendix B we offer a more detailed analysis of this challenge using the CLEVR dataset and introduce a training scheme that includes data samples from CLEVRTex to alleviate the problem. Our findings indicate that by integrating greater complexities into the training set, LSD can be effectively employed on simple datasets as well.
Computation Requirement. We compare the computation requirement of LSD and the baseline models in the CLEVRTex setting. LSD requires approximately 32 GB of training memory, compared to 31 GB and 36 GB for SLATE and SLATE respectively. Notably, LSD does not require much higher memory consumption than regular unconditional latent diffusion, which takes 28 GB under the same setting. We train LSD on 2 NVIDIA RTX 6000 GPUs for 4.5 days, while SLATE and SLATE+ trained in day and 2.7 days using the same GPU setup. For test-time generation, LSD takes 50.7 seconds to generate 50 images, while SLATE and SLATE+ takes 86. and 87.6 seconds, respectively.
7 Conclusion
The integration of diffusion models into unsupervised object-centric learning has been largely unexplored, but holds significant potential. To assess the feasibility of this approach and identify its strengths and weaknesses, we introduced the Latent Slot Diffusion model in this study, which serves two purposes: (1) the first model integrating diffusion models into unsupervised object-centric learning, and also (2) the first unsupervised compositional diffusion model which does not require supervised annotations like text. Our key findings indicate that the proposed model surpasses state-of- object-centric models in multiple tasks, with its performance advantage
art transformer-basee increasing as image complexity grows. Furthermore, the unsupervised compositional generation quality of our model exceeds that of transformer-bar based counterparts. Finally, we also introduce LSD variants that utilize pre-train Diffusion Models and demonstrate their ability real-world object-centric learning and conditional generation with real-world objects. We believe that this work represents significant advancement in object-centric learning towards dealing with complex naturalistic images.
10

---

Acknowledgements
This work is supported by Young Researcher Program (No. 2022RIC1C1009442 through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT. The GPU computing used for this work is partly supported by KI Cloud of Division of National Supercomput- ing Center, Korea Institute of Science and Technology Information (KISTI, We would like to thank Junmo Cho for managing specific experiments. Yi-Fu Wu for his contributions to the initial draft of this paper, and Ligong Han and Yuxiao Chen for their valuable feedback on the manuscript.
References
[1] Titas Anciukevicius, Christoph H Lampert, and Paul Henderson. Object-centric image gen- eration with factored depths, locations, and appearances. arXiv preprint arXiv:2004.0063 2020.
[2] Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville, Systematic generalization: what is required and can it be learned? arXiv preprint arXiv: 181 18889, 2018.
[3] Federico Baldassarre and Hossein Azizpour. Towards self self-supervised learning global and object-centric representations. In ICLR 2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
[4] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. International Conference on Learning Representations, 2022.
[5] Léon Bottou. From machine learning to machine reasoning. Machine learning, 94(2):133-149, 2014.
[6] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXh::1901.11.99 2019.
[7] Michael Chang, Alyssa L. Dayan, Franziska Meier, Thomas Griffiths, Sergey Levine, and Amy Zhang. Hierarchical abstraction for combinatorial generalization in object rearrange- ment. In NeurIPS 2022 Workshop All Things Attention: Bridging Different Perspectives on Atlention, 2022.
[8] Michael Chang, Thomas L Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. arXiv preprint arXiv:220..00787 2022.
[9] Chang Chen, Fei Deng, and Sungjin Ahn. ROOTS: Object-centric representation and rendering of 3D scenes. Journal of Machine Learning Research, 22(259):1-36 2021.
[10] Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances In Neural Information Processing Systems, 2018
[11] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- GAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pages 2172-2180, 2016.
[12] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convo- lutional neural networks. In Proceedings of AAAI, 2019.
[13] Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In International Conference Learning Representations, 2020.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.

---

[15] Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. arXiv preprini 1rXiv:2107.00637, 2021.
[16] Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 34, 2021.
[17] Yilun Du, Kevin Smith, Tomer Ulman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3d physical objects from video. arXiv preprint 7xXv:20077.12348 2020.
[18] Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS- V2: Inferring unordered object representations without iterative refinement. arXiv preprini urXiv:2104.09958 2021.
[19] Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. International Conference Learning Representations, 2020.
[20] SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, and Ge- offrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pages 3225-3233, 2016.
[21] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolutin image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12873-12883, 2021.
[22] Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2)3-71, 1988.
[23] Anand Gopalakrishnan, Kazuki Trie, Jürgen Schmidhuber, and Sjoerd van Steenkiste. Un- supervised learning of temporal abstractions with slot-based transformers. arXiv preprint arXiv:2203.13577 2022.
[24] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam H. Laradji, Hsuch-Ti Liu, Henning Meyer, Yishu Miao. Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator. arXiv preprint arXiv:2203.0 035500 2022.
[25] Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint 19X3v:9903.004300 2019.
[26] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pages 6691-6701, 2017.
[27] Klaus Greff, Sjoerd Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:20/2.05208; 2020.
[28] Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. In ECCV, pages 243..33.0 Springer, 2022.
[29] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[30] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICER 2017: International Conference on Learning Representations 2017, 2017.
12

---

[31] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances In Neural Information Processing Systems, 2020.
[32] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded Diffusion Models for High Fidelity Image Generation. Journal of Machine Learning Research (JMLR), 23:47:1-47:33: 2022.
[33] Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. DGMs and Applications @ NeurlPS 2021 Poster, 2021.
[34] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
[35] Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. In Advances in Neural Information Processing Systems, 2020.
[36] Jindong Jiang. Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In International Conference on Learning Representations, 2019.
[37] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2910, 2017.
[38] Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. object-datasets/, 2019.
[39] Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher Burgess. Simone: View-invariant, temporally- abstracted object representations via unsupervised video decomposition. arXiv preprint (0K3:22006008849, 2021.
[40] Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint arXiv:2111. 2021.
[41] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410 2019.
[42] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv:1802.0523 2018.
[43] Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv preprint arXiv:2111. 2021.
[44] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[45] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference Learning Representations, 2017.
[46] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[47] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint 200.00330, 2023.
[48] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. European Conference Computer Vision (ECCV), pages 740-755. Springer, 2014.
13

---

[49] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In International Conference on Machine Learning, pages 41 14-4124, 2020.
[50] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020.
[51] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua Tenenbaum. Compositional Visual Generation with Composable Diffusion Models. In European Conference on Computer Vision (ECCV), 2022.
[52] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pages
423 439. Springer, 2022.
[53] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.
[54] Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. arXiv preprint arX1v:2011.10287, 2020.
[55] Sindy Lowe, Phillip Lippe, Maja R. Rudolph, and Max Welling. Complex-valued autoencoders for object discovery. arXiv preprint arXiv:2204.000000 02075, 2022.
[56] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided Image Synthesis and Editing with Stochastic Differential Equations. International Conference on Learning Representations, 2022.
[57] Chong Mou, Xintao Wang. Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint XXi::2302.08453, 2023.
[58] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In International Conference Machine Learning (ICML), pages 16784-16804, 2022.
[59] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning 2023.
robust visual features without supervision. arXiv preprint
[60] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Diffusion Autoencoders: Toward a Meaningful and Decodable Representation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10619-10629, 2022.
[61] A. Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv preprint urXiv:2204.06125, 2022.
[62] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021.
[63] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 10684-10695, 2022.
[64] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assiste Intervention 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241 Springer, 2015.
14

---

[65]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Bureu Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic Text-to-lmage Diffusion Models with Deep Language Understanding. Advances In Neural Information Processing Systems, 2022.
[66]
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image Super-Resolution Via Iterative Refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP:1-14, 2022.
[67] Mchdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste. Filip Pavetić, Mario Lučié, Leonidas J. Guibas, Klaus Greff, and Thomas Kipf. Object Scene Representation Transformer. Advances In Neural Information Processing Systems, 2022.
[68] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning, Proceedings of the IEEE, 10965)661--634, 2021.
[69] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl- Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Scholkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. arXiv preprint (rXiv:2209.14860 2022.
[70] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In International Conference Learning Representations, 2022.
[71] Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural Systematic Binder. In International Conference on Learning Representations, 2023.
[72] Gautam Singh, Skand Peri, Junghyun Kim, Hyunseok Kim, and Sungjin Ahn. Structured world belief for reinforcement learning in pomdp. In International Conference on Machine Learning, pages 9744-9755. PMLR, 2021.
[73] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. arXiv preprint 2rXiv:2205.14000 2022.
[74] Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Frédo Durand, Joshua B. Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint vrXiv:2205.0392. 2022.
[75] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2206222 2265. PMLR, 2015.
[76] Yang Song, Jascha Sohl- Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. International Conference on Learning Representations (ICLR), 2021.
[77] Karl Stelzner, Kristian Kersting, and Adam Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.0114 2021.
[78] Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. arXiv preprint arXiv:2004.12900 12906, 2020.
[79] Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. arXiv preprint arXiv:2301. 2023.
[80] Yanbo Wang, Letao Liu, and Justin Dauwels. Slot-vae: Object-centric scene generation with slot attention. arXiv preprint arXiv:2306.06997, 2023.
[81] Nick Watters, Loic Matthey, Chris P. Burgess, and Alexander Lerchner. Spatial Broadcast Decoder: Simple Architecture for Disentangled Representations in VAEs. ICLR 2019 Workshop LLD, 2019.
15

---

[82] Xin Wen, Bingchen Zhao, Anlin Zheng, X. Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. arXiv preprint arXiv:2205.15288, 2022.
[83] Yizhe Wu, Oiwi Parker Jones, and Ingmar Posner. Obpose: Leveraging canonical pose for object-centric scene inference in 3d. arXiv preprint arXiv:2206.0359) 2022.
[84] Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsuper- vised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05861, 2022.
[85] Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object- centric generative modeling with diffusion models. arXiv preprint 1rXii:2305.1128/ 2023.
[86] Jaesik Yoon, Yi-Fu Wu, Heechul Bae. and Sungjin Ahn. An investigation into pre-training object-centric representations for reinforcement learning. arXiv preprint wrXiv:2302.04111 19, 2023.
[87] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real- world videos by predicting temporal feature similarities. Thirty- Conference on Neural Information Processing Systems (NeurIPS 2023), 2023.
[88] Lvmin Zhang and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. arXiv preprint arXiv:2302.05543 2023.
[89] Ruixiang Zhang, Tong Che, B. Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, and Liam Paull. Robust and controllable object-centric learning through energy-basee based models. arXiv preprint arXiv:2210.05519 2022.
16

---

A Limitations and Broader Impact
Limitations LSD faces several limitations common to unsupervised object-centric representation learning. First, it struggles with part-whole ambiguity, meaning it has trouble distinguishing between individual parts of an object and the whole object itself, especially when objects have complex textures in real-world images. Second, the quality and level of detail in LSD's segmentation results are sensitive to the number of slots it uses. If there are too few slots compared to the actual objects in an image, the model tends to produce generalized masks instead of distinct object segments. Conversely, when there are too many slots, the model tends to over-segment the image, splitting objects into multiple parts. This sensitivity don't 't know the exact number of objects in advance. Finally, even with a strong Diffusion decoder, LSD struggles when dealing with real-world images that have highly diverse object appearances. To address this final limitation, we conduct a preliminary exploration in Appendix C two LSD variants and demonstrate their effectiveness in handling real-world data.
slot numbers is a problem in scenarios where we
where we propose
Broader Impact The proposed Latent Slot Diffusion (LSD) model a generative model for object- centric representation learning. The main potential negative social impacts is associated with its application in image editing and novel image symthesis. By enabling the extraction of object-centric or component-bace based representations from images, LSD allows for the generation of entirely new scenes with previously unseen configurations. This capability raises concerns related to image manipulation and privacy. For example, the capability to synthesize new human faces introduces potential privacy issues like deepfake generation and identity theft. Moreover, this ability could potentially cause digital misinformation, as it can generate and manipulate images with high accuracy in object-based level, making it increasingly difficult to distinguish between real and fake or edited visual content. Additionally, the image composition task gives rise to concerns about ethnic bias, ageism, other forms of misrepresentation. This is due to the uniform sampling process of the learned slots within the same cluster and their independence across clusters. This can result in two significant problems: (1) the sampled distribution may reflect the biases present in the collected data, potentially leading to ethnic, age, or other population biases, and (2) when performing slot swapping for applications such as face replacement, mismatches in gender, age, and skin tone between existing parts and introduced parts may occur. Therefore, further application of this method should be proceeded under strict ethical guidelines and regulation.
B LSD on Simple Images
In the paper, we demonstrate that LSD achieves considerable performance improvements for complex images. However, our early experiments reveal that when applied to visually simplistic datasets like CLEVR, LSD exhibits reduced performance in terms of segmentation and representation quality, To gain further insights into these results, we provide visualizations of the segmentation outputs in Figure 6
as illustrated in Table 3
As illustrated by the LSD samples in Figure 6 certain parts of the background, such as the upper regions of the images, are split and assigned to multiple object slots. This behavior leads to informa- tion mixing between objects and the background, which substantially reduces the performance of slot segmentation and representation. We conjecture that this issue may stem from overfitting in the model. Due to the simplicity of the texture and the absence of objects in these areas, the powerful decoder of LSD can easily memorize the rendering process of those regions independently of the slot-conditioning. As a result, it does not provide a sufficient training signal for the encoder to group those background parts into meaningful slot.
In light of these findings, we tested a straightforward approach to address this problem. We pro- pose introducing additional complexity to the background by incorporating data samples from the CLEVRTex dataset into the training process. By integrating the CLEVRTex samples, we introduce 60 different background textures, which makes implicit memorization difficult. This effectively provides the learning signal to capture the background component into the condiiinii.. The results of this approach, labeled as LSD(Mix), can be seen in Figure 6 As clearly illustrated, including CLEVRTex data samples in the training results cleaner background segmentation and alleviates the information mixing issue. As a result, LSD(Mix) achieves approximately 22% gains of improvement in both mloU and mBO compared to the original model trained solely on CLEVR samples, as shown in Table 3 However, we also that the FID score indicated decrease in the generation quality of
17

---

Dataset
LSD
LSD(MiX)
SLATE
SLATE*
Figure 6: Visualization of Unsupervised Object Segmentation on CLEVR dataset. The LSD(Mix) variant corresponds the model trained both CLEVR and CLEVRTex training images.
LSD(Mix) compared to the original LSD, likely due the increased demand of model capacity to model both datasets. Overall, this experiment confirms that introducing complexity into the training dataset can mitigate the overfitting problem and allow LSD effectively perform representation learning on simple datasets.
C LSD with
Diffusion Models
One of the main focuses of recent research in diffusion models have been adapting pre-trained models for additional control signals [47 [7 88). Specifically, these studies propose the integration of learnable adapter network into large-scale pre-trained diffusion models, such Stable Diffusion, to enable controllable generation based on various types of conditioning such as sketches, semantic layouts, and depth maps. An inherent advantage of these approaches is their ability to bypass the resource-intensive processes of training the entire generative model, while still enabling the production of high-qualin controllable images through minimal training of the adapter network.
We note that such a model design can also utilized for slot-based conditioning. However, unlike existing works that primarily focus on learning the adapter to integrate the provided conditioning into the pre-trained diffusion model, our focus lies in training an adapter to achieve two objectives simultaneously: (1) aim to learn how extract object representations (slots) from input images in an unsupervised manner, and (2) we aim to learn how to incorporate these slot representations into the pre-trained diffusion model facilitate slot-based conditional generation. This leads to intriguing question: can learning through such model, which has not been specifically trained on multi-object
Table 3: Quantitative Evaluation on CLEVR dataset. We evalute the models by reporting mBO, mloU and FG-ARI scores for segmentation quality, property prediction scores for representation quality, and FID scores for compositional generation quality.
SLATE
SLATE
LSD
LSD (Mix)
mBO (t)
64.86
67.42
38.49
61.09
mloU (t)
63.96
66.62
37.49
59.76
FG-ARI (t)
69.20
88.56
76.32
76.30
Shape (t)
95.66
95.70
75.16
80.48
Material (1)
97.62
97.96
94.21
96.24
Position (L)
0.59
0.51
0.86
0.99
FID (L)
52.96
20.93
16.22
19.45
18

---

CLEVR
CLEVR(MIN)
CLEVRTex
MOVI-
COCO
Figure 7: Qualitative Results of Control-LSD on CLEVR, CLEVRTex, MOVi-E, and COCO. The CLEVR(Mix) corresponds the model trained on both CLEVR and CLEVRTex samples.
scenarios, provide the necessary learning signal acquire object-centric representations? In this section, we provide preliminary exploration on this question by introducing two LSDD) variants: Control-LSDD and Latent Slot Stable Diffusion (Stable-LSSD).
C.1 Control-LSD
Model Design Control-LSD utilizes a similar design of the adapter structure proposed in ControlNet [88 and consists of three main components: (1) a pre-trained Stable Diffusion model that is frozen throughout the training, (2) a "control block" that trainable copy of the unet encoder from Stable Diffusion, and (3) a object-centric encoder from the LSD model. Similar to LSD, the object- centric encoder takes an image as input and generates a set of slots, which are then conditioned on using Cot-Connditioee Transformer. However, instead of providing the slots to the diffusion decoder, Control-LSD feeds the slots into the trainable control block while solely employing null text conditioning in the Stable Diffusion. To integrate the slot control signal, the outputs of each intermediate control block are element- wise added to the intermediate blocks and skip-connections within the frozen Stable Diffusson model. Following LSD, and attention map resolution of 64 x 64 for all datasets. also use the same number slots as in LSD for multi-object dataset, and 10 slots for the COCO dataset.
a input resolution of 256 256
Results We evaluate Control-LSD on three multi-object datasets: CLEVR, CLEVRTex, and MOVi-E, as well as one real-world dataset, COCO [88]. Following the proposed LSD, we also include CLEVR(Mix) variant the CLEVR dataset which was trained on both CLEVR and CLEVRTex samples. The segmentation results are presented in Figure 7 We can see that Control-LSD struggles achieve satisfactory object segmentation on the CLEVR dataset. The model has difficulties in distinguishing between the background and the objects. We also note that the model demonstrate
19

---

more reasonable results on CLVERTex and introducing samples from CLEVRTex to the training process of CLEVR (i.e., CLEVR(Mix)) also led to an improvement on CLEVR, but the results are still inferior to those achieved by LSD.. Interestingly, we also observe that Control-I-DD achieve better segmentation quality when applied to more realistic scenes, such as those in the MOVi-E dataset. We conjecture that this is attributed to the fact that the complexity of MOVi-E aligns more closely with real-world images used for pre-training the Stable Diffusion model, as compared to the samples from CLEVR and CLEVRTex datasets. This indicates that the distribution gap between the data used for pre-training Stable Diffusion and the multi-object datasets used for learning object-centric representation might be potential contributing factor to the lower performance of Control-LSD on the objti- datasets.
48 Notably, Finally, we also evaluate Control-LSD on a real-world multi-object dataset COCO
Control-LSSD showcases its ability to capture semantically meaningful entities in the slots, as illus- trated in the segmentation results in Figure However, we find it challenging to obtain accurate object segmentations aligned with COCO annotations. In this regard, we would like to share two key observations when applying the model on real-world scenes: (I) Without human annotations, unsupervised models have difficulty distinguishing between discrete parts of an object and the entire object. This is known as the part-whole ambiguity. For instance, while COCO annotates entire human as one segment, Control-LSD might segment into two distinct parts: the upper body and the legs. It is important to emphasize that neither of the two segmentation results are incorrect, each could be beneficial for specific downstream tasks. (2) The segmentation granularity is sensitive to the number of slots. This sensitivity can be attributed to the model's tendency to always utilize the entire set of slots to segment the image. Consequently, if the slot count is less than the number of entities in an image, the model tends to produce semantic masks rather than object-level segments. Conversely, if the slots outnumber the entities, the model tends to over-segment the image, i.e., splitting some of the entity into multiple slots. This behavior limits the applicability of Control-LSD in real-world data, where the true number of objects is not known to the model. Therefore, we believe further investigation is needed for applying the model to real-world applications.
C.2 Stable-LSD
While Control-LSD bypasses the need to train Diffusion decoder, the computational cost of training the adapter network-which includes both the control block and the object encoder-is still substantial. As an illustration, the memory demands are significant that even with four 48GB GPUs, our experiments could only handle a training batch size of 32, which also leads to significant training latency. To address this problem, we introduce the the second LSD variant which further reduce trainable modules, we name it Latent Slot Stable Diffusion, or Stable-LSD for short. We show that despite its simple architecture, Stable-LSD demonstrate strong performance on real-world scenarios. We will start with describing the model design.
Model Design Firstly, following the DINOSAUR model (69, we employ a pre-trained image encoder, DINOv2 [59]. as the CNN backbone for the object encoder. This encoder remains frozen during training. It is important to emphasize that unlike DINOSAUR, Stable-LSD is generative model capable of generating images. Secondly, the learned slots are provided directly to the cross attention layers of pre-trained DMs as special text tokens. This means that the slot attention module acts dual purposes- - as an object encoder extracting object-centric representations and as feature adapter converting input tokens from the image the text domain. Notably, within the Stable-LSID framework, only the slot attention module requires training. The resulting design significantly reduces the computational demands observed with Control-LSD. In our experiment, we were able to train the model with batch size of 128 using two 48GB GPUs.
World Segmentation Results Despite its simplicity, Stable-LSD demonstrate its effectiveness in real-world object-centric learning. To evaluate its object-centric learning ability, we compare Stable-LSD with DINOSAUR instance-leve FG-ARI and mBO. The result can be find in Table The results show that Stable-LSD achieves comparative performance to DINOSAUR with relatively superior FG-ARI value and diminished mBO value. We show the mask visualization on Figure g
'The Sta implementation will be available in the official release: [ diffuuinsinn
https://github.c. com/
20

---

Table 4: Quantitative Evaluation of Stable-LSD on COCO dataset. We compare Stable-LSD with DINOSAUR [69] on instance-level FG-ARI and mBO. The results of DINOSAUR is copied from the original paper.
Stable-LSD (Ours)
DINOSAUR
mBO (1)
30.38
31.60
FG-ARI (1)
35.02
34.10
STOP
STOP
KIFER
CIUZ
Input
Slots
cfg=1.0
cfg
1.
1.
2f.-2.0
cfg=3.0
Figure 8: Unsupervised Image Segmentation and Generation Results of Stable-LSD on COCO dataset. For image generation, we apply different cfg values on the learned slots with cfg 1.0 equivalent to image reconstruction.
Real-Wort Image Generation Results We further evaluate the image generation quality of Stable- LSD. In Figure we show the image reconstruction results (denoted by cfg 1.0). The results demonstrate that the model output suffers from visual artifacts which constrained both the image reconstruction and generation capabilities. To further improve the generation results, explore the technique of classifier-free guidance which is widely employed in text-to-image diffusion models.
Classifier- free guidance 34 uses mixing weight (abbreviated as cfg) to control the weighted sum of noise predictions between a conditional and an unconditional diffusion model. With cfg = operates similarly to standard conditional generation. When efg 1. the model generates images that are more align to the text input, but comes with a cost of reduced image diversity. Practically speaking, optimal efg > 1 can yield images that not only better match the conditions but also exhibit enhanced quality. Since Stable-LSD provides object slots as text tokens to the pre-trained DMs, classifier-free guidance be applied directly to the input slot, akin to how it's applied to input text in standard DMs, without any model modifications. And our experiment shows that a of > for slots can also significantly improve the image generation quality.
In Figure g we tested various cfg values with Stable-I.SD to optimize image generation quality. As can the figure, from cfg 1.0 to 1.3, the images appear cleaner with less visual artifacts,
21

---

STLP
STOP
2089
Input
Slots
Sample
Sample 2
Sample Sample 4
Sample 5
Figure 9: Real-World Image Generations Stable-LSD. Each image is conditioned on the same of learned slots and with different initial noise map in the denoising process. We use cfg 1.3 for all samples.
delivering conceptually more consistent images with the input slots. Nevertheless, also observe that higher cfg values strip away intricate visual details, leading to overly simplified images. We settled on cfg value of 1.3 for following experiments.
In Figure g we show image samples generated based on the same set of learned slots. The variance in these samples comes from differing the initial noise maps in the denoising process. These results demonstrate that with a proper efg value, Stable-LSD achieves slot-based conditional image gener- ation with unconstrained real-world objects, which is for the first time in object-centric generative models. addition, beyond showing the diversity the generated images, these results also un- derscore a crucial point from the representation learning prospective: the learned slots contain rich semantic information rather than merely performing texture-based clustering. An evidence to this is the model's is ability to maintain overall semantic consistency while cxhibiting diverse presentations of the same semantic content across samples. For instance, even if the background of the stop sign images shows different tree shapes, they all clearly have trees in the background. Achieving this level of consistency would be challenging the slots lacked an abstract semantic understanding of the objects in the scene. We show additional real-world segmentation and generation results in Figures 18
We believe the prelimilary attempts detailed in this section to be valuable explorations toward large- scale object-centric learning, and we hope our insights pave the way for further research in this area.
22

---

Table 5: Full Table of Segmentation Performance and Representation Quality. Extended results from the main text, now including standard deviation values.
(a) CLEVRTex
Segmentation
SLATE
SLATE*
LSD (Ours)
Representation
SLATE
SLATE
LSD (Ours)
mBO (1)
50.88 + 0:70
54,90 + 1.15
63,93 + 199
Shape (1)
74.24 + 2.41
71.63 + 4.94
80.23 + L.59
mloU (1)
49.54 = 0:70
$2.96
Material (f)
2.40
75.56 +
FG-ART (1)
44,19 1.80
70.71 + 3,35
64.41 8.53
Position (U)
1,28 : 0.10
1.26
1.13 = 0.04
(b) MOVi-C
Segmentation
SLATE
SLATE
LSD (Ours)
Representation
SLATE
SLATE
LSD (Ours)
mBQ (?)
39.37 +: 0.80
38.17 +: 1.29
45.57 0.30
Position (L)
1.37 :: 0.06
1.28 0.07
1.14
mioU (T)
37.75 +: 0.71
36.44 à: 1:35
44.19 0.1
3D B-Box (1)
1,48 4: 0.03
1.44 + 0.11
1.44 0,02
FG-ARI (1)
49.54 = 1.44
52.04 + 0.45
51.98 3.53
Category (1)
42.45 0.13
45.32 + 1.13
46.11 + 0.59
(c) MOVi-E
Segmentation
SLATE
SLATE
LSD (Ours)
Representation
SLATE
SLATE
LSD (Ours)
mBO
30.17 1,70
22.17 0.38
38.96 0.47
Position (L)
2,09 1: 0.13
2.15
1,85
mloU (T)
28.59 + 1,66
20.63 + 0.35
37.64 + 0.45
3D B-Box (L)
3.36 + 0.12
3.37 + 0.27
2.94 + 0,003
FG-ARI (1)
46.06 + 45.25 0.77
52.17
Category (1)
38,93
38.00 +
42.96 + 0.21
D Additional Implementation Details
To ensure reproducibility, we are sharing additional implementation details of LSD in this sec- tion. Our implementation will be made available at httts://giithb: com//indong
latent- slant-ssot-onn sioent-sottddiffuiion diffusion
D.1 Experiment Setup
Datasets. In experiments, create three data subsets for each dataset: training, validation, and testing. For CLEVR, we utilize the official split for training and validation sets. However, for evaluation, do not the official test set as CLEVR does not provide ground-truth object segmentation masks. Instead, we use the "clevr with mask" subset from the Multi-Object Datasets 38 for evaluation. This subset is designed have the same object distribution providing ground-trun -tuth masks and attributes annotations for each object in the image, which allows us to evaluate the segmentation and representation quality of each model. We use the full subset the test set for evaluation. For CLEVRTex, there is no official split provided. so we allocate 80% of the data for training, 10% for validation, and 10% for testing. Regarding MOVi-C and MOVi-E, use 90% of the training set data for training and reserve 10% for validation. We use the official validation for testing. It is important to note that the test sets in MOVi-C and MOVi-E are designed for out-of oittof-diistibutiin (OOD) evaluation, so we do not use them for test time evaluation. For the FFHQ dataset, we use 86% of the dataset (~ 60K images) for training and 7% (~ 5K images) for validation. To ensure consistency across all models, we shuffled the dataset and utilized fixed random seed when splitting the data.
CLEVR while
To evaluate the unsupervised object segmentation, we use three metrics: the foreground adjusted rand index (FG-ARI), the mean intersection over union (mloU), and the mean best overlap (mBO). We predict the object mask by using the argmax along the slot dimension in the attention map. In computing the mloU, use Hungarian matching to obtain the ground-truth and slots assignment. Following (15), we use the negative cosine similarity as the matching loss. In mBO, we assign the ground-truth segment to the slot with the largest overlap. Note that the mBO metric is generally simpler than mloU since it does not require strict one-to-one mapping between the ground-truit and predicted masks.
Segmentation Metrics.
D.2 Implementation Details
We will provide an overview of the implementation details in this section. The hyperparameters used in our approach are listed in Table
Auto-Encoding CNN backbone. The object-centric encoder our model includes a CNN backbone, which transforms the raw image input into feature vectors that can be processed by the slot attention
23

---

Dataset
Module
Hyperparameter
CLEVR
CLEVRTex
MOVi-C/E
FFHQ
General
Batch Size
64
64
128
128
Training Steps
200K
200K
200K
200K
# K-means Clusters
5
5
18
18
CNN Backbone
Input Resolution
256
Output Resolution
64
Self Attention
Middle Layer
Base Channels
128
Channel Multipliers
[1,1,2,4]
Heads
# Res Blocks Layers
2
Output Channels
128
128
192
192
Learning Rate
0.0001
0.00003
0.0001
0.0001
Slot Attention
Input Resolution
64
# Iterations
3
Slot Size
128
128
192
192
# Slots Learning Rate
11
11
24
4
0.0001
0.00003
0.0001
0.0001
Auto-Encoder
Model
KL-8
Input Resolution
256
Output Resolution
32
Output Channels
4
LSD Decoder
Input Resolution
32
Input Channels
4
3 scheduler
Linear
Mid Layer Attention
Yes
# Res Blocks Layers
Image Latent Scaling
0.18215
Learning Rate
0.0001
# Heads
4
4
8
8
Base Channels
128
128
192
192
Attention Resolution
[1,2,4]
[1,2,4]
[1,2,4,4]
[1,2,4,8]
Channel Multipliers
[1,2,4]
[1,2,4]
[1,2,4.4]
[1,2,4.4]
Table 6: Hyperparameters of our model used in our experiments.
module. Previous studies have used multi-layer CNN networks as image encoders, as demonstrated However, in our experiments with Latent Diffusion Models, we have found that such encoders tend to produce low-level features that mainly represent pixel position or color information. This leads the slot attention module grouping pixels based on these low-level features, rather than on high-level object information. We hypothesize that this problem may arise from the early stages of training, where model lacks semantic understanding of the input image. This results LSD Decoder relying only on low-level and local information from slots to denoise the noisy input, without encouraging the object-centric encoder to learn semantic-level pixel grouping. As a result, without additional guidance, the model may become trapped in suboptimal solution. We have observed this issue only when Diffusion Decoder is used.
by [53 70 73..
To overcome this issue, we have incorporated a UNet architecture 64 as the image encoder in the object-centric encoder. The main advantage of this approach that the uncooen structure of UNet allows high-level global context information to be blended into the output features and slots, facilitating the learning process and helping the model escape from suboptimal pixel groupings. Additionally, the skip connections in UNet allow the output features to retain fine-grained low-level details from early CNN layers, ensuring that the resulting representations are rich in both high-level object information and low-level texture information. These improvements to the object-centric

---

encoder result in more accurate object-centric representations and enable better image generation results. We provide detailed design information Table O Prior to applying the UNet structure, we first employ a single CNN layer with kernel size of 4 and stride of 4 to downsample the image from 256 x 256 x 64 x 64 x S, where S is the base channel size UNet. Appendix F. we provide an additional ablation study for the UNet encoder.
Slot-Conditiones Cross Attention. We implement the conditioning mechanism between the object slots and decoder features through transformer network as in LDM [63 and SLATE [70]. This conditioning mechanism consists of two transformer layers, including self-attention layer that computes self-attention within the features generated by the UNet in the LSD Decoder, followed by cross-attention layer that computes cross-attention between the object slots and UNet features.
In the cross-attention layer, we use the object slots S as key-value pairs, and the queries are interme-
diate layers of the diffusion network augmented with positional embedding (h, + Pt). We then apply
multi-head cross attention for each head as Attention Q, K, V) = softmax QK?
V, where
vd
Q Q= (h, + P:). K W
S. V W S.
Here, i is the head index, by represents the intermediate layers of the diffusion network, Pi is the positional embedding, S is the set of object slots computed from the object-centric encoder, and W/ W W are learnable projection matrices.
Additional Details for Object-Centric Encoder. For all of slots to be the maximum number of objects per image plus one (intended for background). For the FFHQ dataset, we segment the image into 4 slots. We use the 256 x 256 resolution images for all experiments and, if necessary, we use bilinear interpolation for re-scaling and center cropping to get square images. We do not introduce additional data augmentations during pre-processing. When computing the object segmentation, we use the attention map from the slot attention as the mask prediction. The resolution of the mask is designed to be 64 x 64 for all models.
datasets, we set the number
Additional Details for LSD Decoder. We add learnable positional embeddings into the self and cross-attention layers of UNet, which are found to greatly improve the stability of the training process for the CLEVR and CLEVRTex datasets. However, we have observed that this modification has a minor impact on the MOVi-C/E and FFHQ datasets, but have applied it across all models to maintain consistency. For the image auto-encoder, we use the KL-8 version provided in the LDM repository. During the generation processes for all tasks, we employ the DDIM sampler with v 1 and run it for 200 steps.
Baseline Implementation. Our baseline implementations are closely based on the official releases. For the SLATE baseline, we adopt the improved version proposed 73 We empirically find this version to be more robust to complex scenes in terms of both object segmentation and generation quality. The key difference of this version from the original SLATE in the slot encoder. The original SLATE compute object slots from the latent space of the dVAE. Therefore, the attention resolution and information granularity of the slots are constrained by the dVVAE latents. The improved SLATE addresses this issue by using jointly trained CNN backbone to directly learn the slots from the original image. As shown in [73], this improved version of SLATE achieves improved performance on complex datasets such as MOVi-C and MOVi-F. For SLATE+ we replace the jointly trained dVVE with a VQGAN model pre-trained on OpenImages
E
Additional Experiments
E.1
Ablation on CNN backhone in Object-Centric Encoder
This section presents an ablation study on the CNN backbone used in the object-centric encoder. To investigate the impact of the CNN backbone choice, introduce variant of LSD named LSDD-CNN Specifically, LSD-CNN uses a multi-layer CNN network as the image encoder in the object-centric encoder, following the same CNN design in the improved SLATE model [73]. We compare the
fattpa://github. wwttshuu..om//annaeeeeee com/ conm//iingggantaa/stteee We use the pre-trained weights from
25

---

Table 7: Ablation Study CNN backhone. This table presents the results of an ablation study the CNN backbone the object-centric encoder. Specifically, we compare the performance of the LSD method that uses UNet backbone the performance of LSD-CNN, which utilizes multi-layer Convnet backbone.
(a) CLEVRTex
Segmentation
LSD-CNN
LSD
Representation
LSD-CNN
L.D
mBO (t)
55.61
66.56
Shape (t)
75.00
81.60
mloU (t)
53.82
65.02
Material (f)
84.43
77.77
FG-ARI (1)
43.88
61.74
Position (4)
1.50
1.10
(b) MOVi-C
Segmentation
LSDD-NN
L.SD)
Representation
LSD-CNN
L.SD
mBO (t)
27.43
46.29
Position (1)
1.25
1.14
mloU (T)
26.31
44.99
3D B-Box (1)
1.46
1.44
FG-ARI (1)
42.83
50.53
Category (1)
41.29
46.71
(c) MOVi-E
Segmentation
LSD-CNN
LSD
Representation
LSD-CNN
LSD
mBO (t)
22.00
39.63
Position (L)
2.01
1.92
mioU (t)
20.71
38,28
3D B-Box (L)
3.33
2.94
FG-ARI (f)
42.14
53.40
Category (1)
35.82
43.15
results of this variant with the proposed LSD, which utilizes the UNet structure as the image encoder in the object-centric encoder, on CLEVRTex, MOVi-C, and MOVi-E. The results presented in Table 7 Overall, we observe that LSD outperforms LSD-CNN across nearly all tasks and datasets, suggesting that the UNet CNN backbone critical for achieving the high performance of model.
To further clarify the architecture choice, we also explore a LSD variant named LSD-Latent that employs image latent encoder its CNN backbone. We evaluate LSD-Latent MOVi-E dataset and compare its object segmentation quality with LSD. The results can be seen Table g The sub-optimal performance of LSD-Latent underscores the importance of having separate CNN encoder the object encoder.
F
Additional Image Samples
We include additional generation samples in Figures 10 23 The additional samples demonstrate that LSD achieves more accurate object segmentation and generates more detailed and coherent scenes compared to the baselines. We provide additional observations and insights in the following sections.
F1 Visualization of Visual Concept Prompts
In this section, we present visualizations of the visual concept prompts employed for generating image samples in the CLEVRTex and FFHQ datasets, illustrated in Figure I0 To generate these visualizations, we apply the attention map corresponding to each specific slot as a mask over the original image, which highlights the regions captured by that slot. We observe that the LSD model
Table Ablation Study CNN backbone using Difflusion Latent Encoder. We compare LSD with LSD-Latent on segmentation quality in the MOVi-E dataset.
Segmentation
LSD-Latent
LSD
mBO (t)
5.52
39.63
mloU (t)
4.04
38,28
FG-ARI (t)
10.11
53.40
26

---

Concept Prompts
Compose
Concept Prompts
Compose
Figure 10: Compositional Image Generation with Concept Prompts. In this visualization, show concept prompt constructed by composing arbitrary slots from our visual concept library and the corresponding generated image by LSD..
can seamlessly combine components from distinct images to create a coherent image sample while preserving the attributes of each component in the final image. Note that each concept library is associated with one k-means cluster within the extracted slots, and each prompt displayed in Figure I0 sampled from one such library. The visualization of the prompts suggests that the concept libraries is able to capture practical concept categories, such objects and backgrounds in the CLEVRTex dataset and facial attributes, hairstyles, clothing, and backgrounds in the FFHQ dataset.
Impact of Pre-Traine Trained Auto-Encoder on Generation Quality
We investigate the effect of the pre-trained image auto-encoder in the comparison between SLATE and SLATE+ Figure 12 and 13 Our results show that the use of the pre-trained auto-encoder leads to less blurry images and significantly more details, such as the floor and object texture in CLEVRTex and face detail in FFHQ. However, we also observe that applying the pre-trained auto-encoder does not necessarily improve the coherence of the generated scenes. For instance, in some SLATE samples, the shape of the objects appear distorted in CLEVRTex and the background and hair style of FFHQ images are rendered as color patches. These findings indicate the limitation of the transformer decoders in SLATE and SLATE* models and highlights the crucial role of LSD's diffusion decoder in achieving optimal generation quality.
F33 Impact of Segmentation Quality on Slot-Based Editing
In Figure and T5] we investigate the effect of object segmentation on the slot-based editing task. Our result show that when the object segmentation is almost perfect, editing tasks such as object removing can be easily accomplished by discarding the corresponding slot during image decoding. important to note that the object segmentation masks are derived from the attention weights image features. Therefore, perfect segmentation mask indicates that the object information is completely assigned to single slot. However, also observe such as when one object is divided into multiple slots. such cases, removing an object would require removing all associated slots. These observations emphasize the importance of accurate object segmentation in achieving effective slot-based editing. Although LSD has demonstrated superior performance compared existing approaches, further exploration in object segmentation necessary to extend it to real-world applications.
incomplete, when the object assignments
27

---

CLEVRTex
LSD
SLATE
MOVi-E
Dataret
SLATE
FFHQ
SLATE
Figure 11: Visualization of Unsupervised Object Segmentation. LSD achieves accuracy object segmentation when comparing to the other methods.
28

---

CLEVR
SLATE
CLEVRTex
LSD
SLATE!
MOVi-E
SLATE!
FFHQ
LSD
SLATE!
Figure 12: Compositional Generation Samples using Visual Concept Library, LSD provides significantly higher fidelity and more clear details when comparing to the other methods.
29

---

LSD (Ours)
Ki
Sourco
Source
SLATE
Somne
Souree B
SLATE+
Figure 13: Slot-Based Image Editing: Face Replacement. We show face replacement the FFHQ dataset, where we compose new images by combining the face slots from Source-B B images with the hairstyle, clothing, and background slots from Source-A images.
30

---

Input
Progressive Object Removal
Masks
Slot Attention
Figure 14: Slot-Based Image Editing: Object Removal and Segmentation Mask. Sample 1. Top We show object removal by discarding the corresponding slots. Bottom: We show the segmentation masks of each slot.
31

---

Input
Progressive Object Removal
Masks
Slot Attention
Figure 15: Slot-Based Image Editing: Object Removal and Segmentation Mask. Sample 2.
32

---

Source B
Background Extraction
Source
Background Extraction
Source A
Source
Figure 16: Slot-Based Image Editing: Background Replacement. The background replacement
task is performed by replacing the background-associated slots.
Souree
Source
Single Object Exmacctio
Single Objjct Extratiion
Source A
Source A
Figure 17: Slot-Based Image Editing: Object Insertion. The background replacement task is performed by simply adding the new slot to the existing set of slots.
33

---

Input Figure 18:
Slots
cfg=1.0
cfg=1.1
cfg=1.3
cfg=1.5
cfg=2.0
cfg=3.0
Random Samples for Worl-World Object-Centric Learning and Conditional Genera-
tion 1.
34

---

STOP
SIOP
PMorean
PMorran
Input Figure 19:
Slots
cfg=1.0
cfg=1.1
cfg=1.3
cfg=1.5
cfg-2.0
cfg=3.0
Random Samples for Object-Centric Learning and Conditional Genera-
tion 2.
35

---

11
Input Figure 20:
Slots
cfg=1.0
cfg=1.1
c.g=1.3
cfg=1.5
ffg-2.0
cfg=3.0
3.0
Random Samples for Real-Worke Object-Centric Learning and Conditional Genera-
tion 3.
36

---

Input
Slots
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
Figure 21: Random Samples for Real-World Image Generation
37

---

STLP
STOP
2089
PMorran
PMorran
Input
Slots
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
Figure 22: Random Samples for Real-World Image Generation
38

---

Input
Slots
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
Figure 23: Random Samples for Real-World Image Generation
39


Reviews:
[{"review_id": "FPN4VYxSFc", "reviewer": "Reviewer_uRVS", "summary": "This work introduces a methodology for using diffusion-based models to obtain object-centric representations. \nThis is done using slot representations, obtained from the Slot Attention module applied to the original image, as a conditioning variable for a latent diffusion model. The diffusion model and the Slot Attention module are trained end-to-end.\n\nThe proposed model is called Latent Slot Diffusion (LSD) and is evaluated on unsupervised object segmentation, downstream property prediction, compositional image generation, and image editing. The datasets used for evaluation are ClevrTex, MOVi-E, MOVi-C and, for the first time in the field, FFHQ. The performance of LSD is evaluated quantitatively on the first 4 tasks and is compared against SLATE and SLATE+. The proposed model outperforms both baselines in the tasks according to most metrics.", "questions": "As object segmentation is performed using the masks from the slot attention module, it is clear that it is not possible to obtain masks with the full resolution of the images in many cases. Could this be an area of improvement for future work or is it possible to already try using some unsupervised super-resolution technique to get higher-quality masks? Could be worth considering object segmentation as a visualization of what the model is doing instead of a separate task?\n\nDo you have any insights into how the model would perform if slot attention is applied to the latent representation of the latent diffusion model instead of the original image? This would help clarify the choices made during the development of the method.", "limitations": "The authors have sufficiently addressed the limitations of the proposed model, as well as the broader impact of their approach. However, this can be improved further, by being more explicit about direct effect of ethnic bias, ageism or other forms of mis-representation that the model could lead to, which is standard practice for modern diffusion model, considering the extreme high-quality of the generated images. Direct manipulation of certain characteristics enables these effects in a much easier way.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**. \nThe paper proposes a novel way to combine two existing models: slot attention and latent diffusion. The presentation clearly shows what are the novel elements.\n\n**Quality**. \nThe method shows a successful way to leverage diffusion in object-centric learning. Especially the results on property prediction show that it has concrete benefits for the representation learning itself, which might be useful for several different tasks, not only the ones shown in the paper.\n\n**Clarity**. \nThe submission neatly shows all the experiments that were carried out and the description of the underlying method is clear.\n\n**Significance**. \nThe work is a necessary exploration of leveraging the generative power of the diffusion approach in the object-centric setting, highlighting the complexities related to having too-strong of a decoder and providing clear examples of its strengths, especially by using a very complex dataset (FFHQ).", "weaknesses": "**Quality**. \nThe lack of error bars (i.e. standard deviation) in the analysis makes the quantitative analysis weaker. Additionally, it would be interesting to further explore the problems with FG-ARI, as it is currently the standard metric used in the field, and although I agree with the statements, this is only based on experience and intuition, and not a proper scientific analysis, which has not been carried out yet. Lack of comparison with traditional models such as the improved Slot Attention architecture proposed in [1] makes the performance of the model harder to evaluate. Are the good results obtained primarily due to the improved architecture (e.g. larger encoders, better decoder), or is there something fundamentally good about using diffusion (e.g. the iterative improvement) that results in better object representations?\n\n**Clarity**\nThe description of the method used for unsupervised object segmentation is lacking. There is only a reference to the use of the attention masks from slot attention, it would be much better to refer directly to the nice mathematical notation used before in the text.\n\n*References*\n1) Biza, Ondrej, et al. \"Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames.\" arXiv preprint arXiv:2302.04973 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We sincerely thank you for your positive recommendation and thoughtful comments!\n> ... lack of error bars\n\nWe agree that incorporating error bars can better evaluate model’s robustness, and we intend to provide the results here. However, due to constraints in computing resources, we only managed to complete the MOVi-E experiment within the rebuttal window. The table below shows MOVi-E results with 3 seeds per model. The results demonstrate LSD’s consistent high performance, outperforming baseline models across all metrics. We hope the new analysis provides clear evidence of LSD’s strength and has properly addressed your concern. We will include the rest of the results in the revised version of the paper.\n\n|Segmentation|SLATE|SLATE$^+$|LSD (Ours)|\n|:-:|:-:|:-:|:-:|\n|mBO ($\\uparrow$)|30.17 $\\pm$ 2.09|22.17 $\\pm$ 0.47|**38.96 $\\pm$ 0.58**|\n|mIoU ($\\uparrow$)|28.59 $\\pm$ 2.03|20.63 $\\pm$ 0.43|**37.64 $\\pm$ 0.55**|\n|FG-ARI ($\\uparrow$)| 46.06 $\\pm$ 4.07|45.25 $\\pm$ 0.94|**52.17 $\\pm$ 1.09**|\n\n|Representation|SLATE|SLATE$^+$|LSD (Ours)|\n|:-:|:-:|:-:|:-:|\n|Position ($\\downarrow$)|2.09 $\\pm$ 0.15| 2.15 $\\pm$ 0.09|**1.85 $\\pm$ 0.06**|\n|3D B-Box ($\\downarrow$)|3.36 $\\pm$ 0.14| 3.37 $\\pm$ 0.33|**2.94 $\\pm$ 0.00**|\n|Category ($\\uparrow$)|38.93 $\\pm$ 0.20| 38.00 $\\pm$ 0.45|**42.96 $\\pm$ 0.26**|\n\n> ... would be interesting to further explore the problems with FG-ARI\n\nWe appreciate that you agree with our observations. Previous studies [1,2] have also noted similar limitations of FG-ARI and recommended using additional metrics, e.g., mIoU. Therefore, in our experiments, we report 3 metrics: mIoU, mBO, FG-ARI; and our model has shown significant benefits in all of them. Since these are standard metrics in the line of object-centric learning, exploring their limitations is perhaps beyond the scope of this work. That being said, it is an interesting and important topic of future research.\n\n> ... comparison with traditional models such as the Invariant Slot Attention (ISA)\n\nAlthough we acknowledge that including a comparison to ISA would provide additional insights to our results, we do not consider it critical. Firstly, our study and the ISA paper approach different aspects. Our work focuses on improving representation learning and image generation quality, while ISA aims to obtain object representations that are invariant to position and scale. An interesting direction would be to combine our work with ISA to introduce invariance in the object representation. However, we leave this exploration to future studies.\n\n> ... importance of improved architecture vs diffusion process\n\nThank you for your insightful question. We believe that the improvement in performance stems from a combination of both. The diffusion architecture (e.g., the UNet) utilized in LSD has been meticulously explored within the diffusion models literature. And the iterative denoising technique has also been adopted to achieve high image generation quality, as evident from the progression from DALL-E [3] to DALL-E 2 [4]. On the other hand, as highlighted in the SLATE paper and reaffirmed in our study, improved generation capacity can significantly improve the representation learning capability, particularly in scenarios involving complex scenes. Therefore, we hold the view that the combination of both architecture and the iterative process contributes to the observed improvement.\n\n> ... mathematical notation for unsupervised object segmentation\n\nWe will clarify this in the revised version (e.g., in Appendix) using the notations in Section 2.1.\n\n> ... improvement to get higher-quality masks?\n\nWe agree that it is a promising direction for future work. One reason for the reduced resolution is to reduce memory and computational costs. There are existing potential solutions to this challenge, such as using bilateral solvers or conditional random fields for post-processing. These methods incorporate low level features like RGB colors and pixel positions to refine the boundary of the scaled segmentation masks and have been effectively utilized in other works [5]. We believe they could potentially be integrated into LSD for even higher-quality masks. We will include this discussion in our revised manuscript.\n\n> ... considering object segmentation as a visualization of what the model is doing?\n\nWe appreciate this perspective. The segmentation are derived from the cross-attention between the slot representation and image patches. It does serve as a visualization of the model's spatial disentangling process during inference of slot representations.\n\n> ... how the model would perform if slot attention is applied to the latent representation of the latent diffusion model?\n\nYes, this particular configuration of LSD was explored in our preliminary stages. Initial experiments indicated that object segmentation was not optimal when sharing the VAE encoder's latents to learn the slots. \n\nAdditionally, we note that with the VAE encoder shared with the object encoder, the segmentation resolution will be constrained by the VAE latent's resolution, which in the case of LSD, would be ⅛ of the original image size. We acknowledge the importance of clarifying this development choice and will be including a discussion on this investigation in the revised version.\n\n> ... limitations and broader impact\n\nThank you for your valuable suggestion. We will be adding a dedicated limitation section in the revised version of our paper to provide a comprehensive explanation of these aspects.\n\n[1] Monnier, et al. \"Unsupervised layered image decomposition into object prototypes.\" ICCV. 2021.\n\n[2] Zimmermann et al. \"Sensitivity of Slot-Based Object-Centric Models to their Number of Slots.\" arXiv. 2023.\n\n[3] Ramesh, et al. \"Zero-shot text-to-image generation.\" ICML. 2021.\n\n[4] Ramesh, et al. \"Hierarchical text-conditional image generation with clip latents.\" arXiv. 2022.\n\n[5] Wang, et al. \"Cut and learn for unsupervised object detection and instance segmentation.\" CVPR. 2023."}, {"review_id": "kktwCmKQbE", "reviewer": "Reviewer_TTEN", "summary": "The paper studies the diffusion models into object-centric learning. The authors introduce the concept of latent slot diffusion (LSD) that can replace slot decoders conditioned on object slots. The model can work in an unsupervised compositional mode without requiring annotations such as text. Their experiments show that LSD performs better in complex scenes comparing with other methods that do unsupervised compositional generation. ", "questions": "What are potential ways to improve segmentation using LSD?", "limitations": "As a generative model, the method needs to consider the privacy and impacts of image manipulation. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method LSD can be viewed as a model substituting conventional slot-decoders with a conditional latent diffusion model, where the conditioning is done via the object slot attentions. It can also be viewed as a unsupervised conditional compositional diffusion based generation. The ablation experiments are helping in explaining the crux.", "weaknesses": "Object segmentation using LDS is yet to be perfected, leading to suboptimal downstream applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We would like to express our appreciation to your insightful feedback!\n> Object segmentation using LDS is yet to be perfected, leading to suboptimal downstream applications.\n\nThank you for your observation. While the object segmentation in LSD may have room for improvement, it's crucial to underscore that LSD is functioning under the **unsupervised learning setting**. The problem of unsupervised entity-level segmentation is a well-recognized challenge in both object-centric learning and computer vision, with no perfect solution yet. Nevertheless, our work shows significant progress compared to the previous state-of-the-art, which is demonstrated by the effectiveness of LSD in comparison with baseline models under this demanding condition.\n\n> What are potential ways to improve segmentation using LSD?\n\nThank you for your question. We would like to share some potential ways to improve the segmentation performance of LSD as followings:\n\n- **Using post processing techniques to improve the resolution and boundary prediction of the mask.**\n    - As also discussed in our response to reviewer uRVS, improving the quality of segmentation masks might be achieved with post-processing techniques like bilateral solvers and conditional random fields. These methods utilize low-level features, such as RGB colors and pixel positions, to fine-tune the boundaries of the segmentation masks. Integrating this refinement process into our model can potentially lead to improved segmentation outcomes.\n- **Applying LSD on pre-trained diffusion models.**\n    - As mentioned in our paper's appendix, LSD's segmentation can further benefit from pre-trained diffusion models. Such models may alleviate the slots' burden to capture the perceptual details of the images, allowing them to concentrate more on object discovery and produce more accurate segmentation masks. We will provide further investigation of this direction in our revised appendix.\n- **Adding supervised signals from large scale segmentation datasets.**\n    - While LSD operates within unsupervised learning contexts in this study, one might consider shifting to a semi-supervised learning approach when object segmentation is a primary concern for the downstream application. Integrating supervised segmentation signals into the object-centric encoder for specific data samples has the potential to greatly improve segmentation accuracy.\n\n> Ethics Concerns: As a generative model, the method needs to consider the privacy and impacts of image manipulation.\n\nWe acknowledge the importance of privacy issues and the impacts of the ability to perform image manipulation. We have discussed these issues and other potential social implications of LSD in the \"Broader Impact\" section in the appendix, and we will address additional ethical consideration in our revised version."}, {"review_id": "32A7YSI9EW", "reviewer": "Reviewer_ZjQw", "summary": "Previous work has shown that transformer-based image generative models can be trained for object-centric learning which can handle complex scenes. That is, given unstructured observation, transformer-based models learn to find latent compositional structures to bind relevant features.\n\nThis work makes an attempt to explore the feasibility and potential of integrating diffusion models into object-centric learning.\n\nA major contribution of this work is a novel Latent Slot Diffusion (LSD) model which combines conventional slot decoders with a conditional latent diffusion model, conditioned on object-centric slots provided by Slot Attention.\n\nThe authors have shown the effectiveness of the proposed model by evaluating several object-centric tasks including unsupervised object segmentation, downstream property prediction, compositional generation, and image editing.", "questions": "In addition to the above weaknesses, here are two more questions:\n\n- In line 231, given that the object segmentation masks are derived from the attention masks of Slot Attention, why using diffusion-based generative modeling can help unsupervised object segmentation?\n\n- The LSD model has been evaluated across multiple tasks and outperforms existing state-of-the-art methods. However, it raises the question: to what extent does the enhanced performance truly stem from the diffusion-based generative modeling rather than the improved image encoder? It's an intriguing point to ponder.\n\nOverall, this paper is a good effect. I will raise my rating if the authors can address my concerns.", "limitations": "The authors did not adequately address the limitations of this work.\n\nA significant constraint of this study is the requirement for two distinct visual encoders: a pre-trained image auto-encoder and an object-centric encoder. This dual requirement results in a relatively heavy model for inference. This leads to the question: Is there any potential to share certain components between these two encoders to enhance efficiency?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Scope and relevance**:  Considering the growing interest in diffusion models, this paper is exceptionally timely as it broadens their application towards object-centric learning.\n\n- **Significance of contributions**:  This paper presents a novel diffusion model for object-centric learning without the requirement of supervised annotations. It answers the raised question about how diffusion-based generative modeling can benefit object-centric learning properly. \n\n- **Experimental results**:  The experiments in this paper are enough to prove the effectiveness of the proposed model.\n\n- **Clarity**:  The main body of the paper is written very well.", "weaknesses": "- **Limited technical contribution**:  The proposed slot-contained diffusion is a trivial extension of existing text-to-image generation in Latent Diffusion Models (LDM) [59], which replaces text inputs with latent slots.\n\n- **Unclear model details**:  To my understanding,  the whole model is trained for image generation objectives. How to apply such a generative model to unsupervised object segmentation is not clear.\n\n- **Unclear training details**: if I am not mistaken, the object-centric encoder is trained independently rather than jointly with the latent slot diffusion decoder. I would suggest the authors clarify this. Thus, control experiments might be needed to study the effect of training strategies.\n\n- **Some implementation details are missing**:  Without providing the source code, some important hyperparameters are not presented to justify the reproducibility of this work, eg., the number of $K$ for k-means cluster,  noise schedule $\\alpha$, and steps $T$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We sincerely appreciate for your constructive recommendation and valuable insights!\n\n> If I am not mistaken, the object-centric encoder is trained independently rather than jointly with the latent slot diffusion decoder. I would suggest the authors clarify this. Thus, control experiments might be needed to study the effect of training strategies.\n\nThank you for your valuable feedback. We would like to clarify that the object-centric encoder is **trained jointly** with the diffusion decoder using the denoising loss. It is not trained independently. \n\n> The proposed slot-contained diffusion is a trivial extension of existing text-to-image generation in LDM, which replaces text inputs with latent slots.\n\nWe appreciate your feedback and the opportunity to clarify our contributions. While the proposed model might seem like a combination of known components, the impact of our work exceeds the sum of its parts. In our model, the object-centric encoder (that provides the slots) and the diffusion decoder are **trained jointly**. This joint training has two important consequences:\n\n1. **From an Object-Centric Learning Perspective**: Before our work, it was not known if using a diffusion decoder instead of transformer-based autoregressive decoders would lead to object discovery in the encoder. In this sense, it is remarkable that our model actually surpasses transformer-based autoregressive decoders—the current state-of-the-art in unsupervised scene decomposition. The importance of this finding is also resonated by reviewer QNuS.\n2. **From a Diffusion Model Perspective**: Prior to our work, compositionality in diffusion models was achieved via text annotations. We show for the first time that compositionality can emerge in diffusion models without requiring text—solely via unsupervised training—allowing us to compositionally generate and edit scenes without text. This also points to the potential of harnessing vast amounts of unlabelled image data as a future avenue.\n\n> To my understanding, the whole model is trained for image generation objectives. How to apply such a generative model to unsupervised object segmentation is not clear.\n\nThank you for this question. The model is trained with an image reconstruction objective, however, the learning signal is back-propagated through both the slot attention encoder and the decoder. The attention masks of slot attention emerging from this training process serve as unsupervised object segmentation.\n\nSpecifically, in the encoder, the slots attend to a grid of image features. The area each slot attends is considered an object segment. This method for obtaining $\\mathbf{A}$ is already detailed in Section 2.1. Please let us know what we might have overlooked; we're happy to provide additional details.\n\n> Some implementation details are missing: Without providing the source code, some important hyperparameters are not presented to justify the reproducibility of this work, eg., the number of for k-means cluster, noise schedule, and steps.\n\nWe appreciate your feedback and attention to detail. While we have included the implementation details in the appendix of the paper, we acknowledge that there was an oversight regarding the number of $k$-means clusters. We will thoroughly review our implementation and documentation to ensure that all information is appropriately included. Additionally, we are committed to releasing our complete implementation upon acceptance.\n\n> In line 231, given that the object segmentation masks are derived from the attention masks of Slot Attention, why using diffusion-based generative modeling can help unsupervised object segmentation?\n\nThank you for sharing the concern. As we clarified above, the attention masks are learned through a joint training of the object encoder module and the diffusion decoder using only the denoising loss. In our study, we have found that by combining these two components, the model naturally develops object segmentation capabilities without needing a supervision signal.\n\n> The LSD model has been evaluated across multiple tasks and outperforms existing state-of-the-art methods. However, it raises the question: to what extent does the enhanced performance truly stem from the diffusion-based generative modeling rather than the improved image encoder? It's an intriguing point to ponder.\n\nTo answer this question, we have provided the comparison between LSD and SLATE+ in our study. This comparison essentially contrasts the performance of a transformer decoder versus a diffusion decoder. We believe the notable performance gap between LSD and SLATE+ does suggest that the diffusion decoder plays a key role in achieving the superior results.\n\n> The authors did not adequately address the limitations of this work.\n\nThank you for your insightful suggestion. We will add a dedicated Limitation section in the revised manuscript to explain the limitations of our work.\n\n> A significant constraint of this study is the requirement for two distinct visual encoders: a pre-trained image auto-encoder and an object-centric encoder. This dual requirement results in a relatively hefty model for inference. This leads to the question: Is there potential to share certain components between these two encoders to enhance efficiency?\n\nThank you for the insightful comment! One potential way to enhance efficiency is to share the encoder of the VAE with the object-centric encoder. In fact, as also discussed in our response to reviewer uRVS, we have investigated this specific configuration of LSD during our early experiments. However, our initial tests indicated suboptimal object segmentation and, as a result, we opted to keep the two components separate. We will include a discussion on this question in the revised version of the paper.\n\nAdditionally, it is noteworthy that during inference, if the downstream tasks only require object segmentation or object representations, then only the object-centric encoder is required."}, {"review_id": "f36Bh4Swc0", "reviewer": "Reviewer_QNuS", "summary": "This paper proposes Latent Slot Diffusion (LSD), an object-centric learning framework combining the Slot Attention module and a Latent Diffusion Model (LDM) based slot decoder. The model is trained in an auto-encoding manner, where the loss is the slot-conditioned denoising loss in the LDM-based decoder. Extensive experiments demonstrate the effectiveness of LSD in 1) unsupervised scene decomposition  2) object property prediction  3) image generation/editing, surpassing previous SOTA SLATE which uses a auto-regressive Transformer-based slot decoder.", "questions": "1. Regarding the pre-trained VAE model:\n- I appreciate the authors' efforts in having SLATE+ with the same VAE for a fair comparison. Have you tried training a VAE from scratch, and comparing this LSD variant with SLATE (which also trains dVAE from scratch)?\n\n2. Regarding the LSD + Stable Diffusion (SD) experiments in the Appendix (**not affecting paper decision**):\n- This is an interesting experiment, especially the results on the real-world COCO dataset. Do you have any numerical results to compare with the SOTA object-centric model DINOSAUR [1]?\n- Can the authors show some generation or reconstruction results? I am curious because I believe even LSD cannot generate realistic unconstrained real-world images (FFHQ images are constrained as they only capture human faces). Will using a pre-trained SD decoder help here?\n\n[1] Seitzer, Maximilian, et al. \"Bridging the gap to real-world object-centric learning.\" ICLR. 2023.\n\n3. Minor question (**not affecting paper decision**): a [concurrent work](https://arxiv.org/abs/2305.11281) you might want to cite, but I understand that paper comes out after the submission deadline.", "limitations": "The authors discuss limitations in Section 6. However, there are more limitations of this work (and the entire object-centric learning field):\n- LSD is still unable to generate realistic unconstrained real-world images\n- LSD suffers from the part-whole ambiguity issue, as can be seen from the COCO examples\n\nI'd like to see the authors discuss these points in the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. The figures are of high quality\n- The use of LDM as the slot decoder is very intuitive and reasonable, considering the recent trends in object-centric models (CNN decoder --> dVAE+Transformer decoder --> KL-VAE+LDM decoder). The results verify this design choice\n- The experimental results cover lots of different tasks, and show very strong performance. The improvement in generation capacity is very impressive compared to baselines", "weaknesses": "I don't see any big issue with the paper. One might consider LSD as \"simply replacing the slot decoder with an LDM\", but I think this modification is reasonable, and works pretty well across datasets and tasks. SLATE/STEVE also just replace the CNN decoder in Slot Attention with Transformer decoders, and they have been widely used due to their strong unsupervised segmentation performance. I believe LSD's improvement in generation quality will also facilitate future research in object-centric generative models.\n\nThat being said, I think the paper should have more discussion of its limitations. See the `Questions` and `Limitations` sections below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We genuinely appreciate your positive recommendations and insightful feedback!\n\n> I appreciate the authors' efforts in having SLATE+ with the same VAE for a fair comparison. Have you tried training a VAE from scratch, and comparing this LSD variant with SLATE (which also trains dVAE from scratch)?\n\nThank you for this insightful suggestion! We agree that this experiment would provide valuable insights and we have plans to incorporate it into our final version. \n\nIn our study, we chose to use the pretrained VAE instead of training a VAE from scratch, based on the following considerations:\n\n- The use of pre-trained VAEs is common practice when training diffusion models, primarily due to its training stability and demonstrated effectiveness.\n- Our experiments indicate that with a pretrained VAE, SLATE+ exhibits superior performance when compared to SLATE, which uses a VAE trained from scratch.\n- Furthermore, a single pretrained VAE demonstrated satisfactory performance across all four datasets we utilized. This suggests that the use of pretrained VAE can contribute to more efficient training processes.\n\nGiven these reasons, we opt for using a pre-trained VAE in our approach.\n\n> Regarding the LSD + Stable Diffusion (SD) experiments in the Appendix (not affecting paper decision).\n\n> Do you have any numerical results to compare with the SOTA object-centric model DINOSAUR [1] (on COCO dataset)? Can the authors show some generation or reconstruction results? I am curious because I believe even LSD cannot generate realistic unconstrained real-world images (FFHQ images are constrained as they only capture human faces). Will using a pre-trained SD decoder help here?\n\nThank you for the insightful suggestions. We agree that the recommended experiments would be significant additions to our work. In our tests, we do observe visual artifacts in decoded images when dealing with real-world scenes under the current (LSD+SD) model configuration. Such artifacts have constrained both the image reconstruction and generation capabilities of the model on the COCO dataset. However, we believe this is a very interesting direction worth dedicated effort, and we are diligently investigating this model design. If possible, we will include our findings including the quantitative results in our final revision.\n\n> Minor question (not affecting paper decision): a concurrent work (slotdiffusion) you might want to cite, but I understand that paper comes out after the submission deadline.\n\nThank you for bringing this to our attention, we will include the discussion of the concurrent work in the revised paper.\n\n> The authors discuss limitations in Section 6. However, there are more limitations of this work (and the entire object-centric learning field):\n> \n> 1. LSD is still unable to generate realistic unconstrained real-world images\n> 2. LSD suffers from the part-whole ambiguity issue, as can be seen from the COCO examples\n\nThank you for your valuable suggestion. We will be adding a dedicated limitation section in the revised version of our paper to provide a comprehensive explanation of these aspects."}]

Author rebuttal:
## **General Response**

We thank all reviewers for their insightful and positive feedback! We are encouraged that they find our work **novel** (uRVS, ZjQw), **timely** (ZjQw), and **both intuitive and nontrivial** (QNuS). They also highlighted its **practical implications** (uRVS) and **potential to facilitate future research** (QNuS). We are pleased that they recognized our empirical evaluation as **thoroughly conducted** (uRVS, QNuS, ZjQw), **demonstrating impressive improvement** (QNuS), and our paper **well-written and easy to follow** (uRVS, ZjQw, QNuS).

We would like to extend our gratitude to the ethics reviewers for their insights on the ethical dimensions of our work. We acknowledge the concerns about the image manipulation ability and are committed to addressing them explicitly in our revised manuscript.

We will respond to each reviewer’s concerns and questions separately below.

Meta-review: Reviewers have come to a consensus on accepting this paper, the authors are expected to incorporate the suggestions from reviewers in the final camera ready version.
Decision: Accept (spotlight)