SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models
Authors: Ziyi Wu
Jingyu Hu
Wuyue Lu
Igor Gilitschenski
Animesh Garg
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Unsupervised object-centric learning
diffusion model
generative modeling
TL;DR: We incorporate diffusion models to the Slot Attention framework and achieve better segmentation and generation results

Abstract: Object-centric learning aims to represent visual data with a set of object entities (a.k.a. slots), providing structured representations that enable systematic generalization.
Leveraging advanced architectures like Transformers, recent approaches have made significant progress in unsupervised object discovery.
In addition, slot-based representations hold great potential for generative modeling, such as controllable image generation and object manipulation in image editing.
However, current slot-based methods often produce blurry images and distorted objects, exhibiting poor generative modeling capabilities.
In this paper, we focus on improving slot-to-image decoding, a crucial aspect for high-quality visual generation.
We introduce SlotDiffusion -- an object-centric Latent Diffusion Model (LDM) designed for both image and video data.
Thanks to the powerful modeling capacity of LDMs, SlotDiffusion surpasses previous slot models in unsupervised object segmentation and visual generation across six datasets.
Furthermore, our learned object features can be utilized by existing object-centric dynamics models, improving video prediction quality and downstream temporal reasoning tasks.
Finally, we demonstrate the scalability of SlotDiffusion to unconstrained real-world datasets such as PASCAL VOC and COCO, when integrated with self-supervised pre-trained image encoders.

SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models
Ziyi Wul Jingyu Hu!- Wuyue Lu Igor Gilitschenski Animesh Garg!.
University of Toronto Vector Institute
Abstract
Object-centric learning aims represent visual data with a set of object entities (a.k.a. slots), providing structured representations that enable systematic generaliza- tion. Leveraging advanced architectures like Transformers, recent approaches have made significant progress in unsupervised object discovery. In addition, slot-based representations hold great potential for generative modeling, such as controllable image generation and object manipulation in image editing. However, current slot-based methods often produce blurry images and distorted objects, cxhibiting poor generative modeling capabilities. In this paper, focus on improving slot-to- image decoding. a crucial aspect for high-quality visual generation. We introduce SlotDiffusion object-centric Latent Diffusion Model (LDM) designed for both image and video data. Thanks to the powerful modeling capacity of LDMs, SlotD- iffusion surpasses previous slot models in unsupervised object segmentation and visual generation across six datasets. Furthermore, our learned object features can be utilized by existing object-centric dynamics models, improving video prediction quality and downstream temporal reasoning tasks. Finally, we demonstrate the scalability of SlotDiffusion to unconstrained real-world datasets such as PASCAL VOC and COCO, when integrated with self-supervised pre-trained image encoders. Additional results and details are available our website.
Introduction
Humans perceive the world by identifying discrete concepts such as objects and events [88], which serve as intermediate representations to support high-level reasoning and systematic generalization of intelligence [24]. In contrast, modern deep learning models typically represent visual data with patch- based features [14, 28), disregarding the compositional structure of scenes. Inspired by the human perception system, object-centric learning aims to discover the modular and causal structure of visual inputs. This approach thus holds the potential to improve the generalizability and interpretability of AI algorithms [50, 80J. For example, explicitly decomposing scenes into conceptual entities facilitates visual reasoning [8, 12, 13] and causal inference [80, 109]. Also, capturing the compositional structure of the world proves beneficial for both image generation [83, 90] and video prediction [100, 103].
However, unsupervised object-centric learning from raw perceptual input is challenging, as requires capturing the appearance, position, and motion of each object. Earlier attempts at this problem usually bake strong scene or object priors in their frameworks [19, 40, 55, 56], limiting their applicability to synthetic datasets [5, 42]. Later works extend the Scaled Dot-Product Attention mechanism [94] and introduce a general Slot Attention operation [60], which eliminates domain-specific priors. With simple input reconstruction objective, these models learn segment objects from raw images [60] and videos [46]. To date, state-of-the- unsupervised slot models [83, 84] have shown promising scene decomposition results on naturalistic data under specialized scenarios such traffic videos.
Despite tremendous progress in object segmentation, the generative capability of slot-based methods has been underexplored. For example, while the state-of-the the-tth-rr art object-centric model STEVE [84]
* Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

is able to decompose complex data into slots, it reconstructs videos with severe object distortion and temporal inconsistency from slots [100]. Moreover, our own experiments reveal that all popular slot models [46, 60, 83] suffer from low slot-to-image decoding quality, preventing their adoption in visual generation such as image editing. Therefore, the goal of this work is to improve the generative capacity of object-centric models on complex data while preserving their segmentation performance.
We propose SlotDiffusion, an unsupervised object-centric model with a Latent Diffusion Model (LDM) [73] based slot decoder. We consider slots extracted from images as basic visual concepts (i.e., objects with different attributes), akin to text embeddings of phrases from sentences. The LDM decoder learns to denoise image feature maps guided by object slots, providing training signals for object discovery. Thanks to the expressive power of diffusion models, SlotDiffusion achieves a better trade-off between scene decomposition and visual generation. We evaluate our method on six synthetic and three real-world datasets consisting of complex images and videos. SlotDiffusion improves object segmentation results and achieves significant advancements in compositional generation. Crucially, we demonstrate that SlotDiffusion seamlessly integrates with recent development in object-centric learning, such as dynamics modeling [100] and the integration of pre-trained image encoders [81].
Our main contributions are as follows: (i) SlotDiffusion: a diffusion model based object-centric learning method. (ii) We apply SlotDiffusion to unsupervised object discovery and visual generation, where it achieves state-of-the-an results across both image and video datasets. (iii) We demonstrate that our learned slots can be directly utilized by the state-of-the oftte-ff-the-rt object-centric dynamics model, im- proving future prediction and temporal reasoning performance. (iv) We further scale up SlotDiffusion to real-world datasets by integrating it with self-supervised pre-trained image encoder.
2 Related Work
In this section, we provide a brief overview of related works on unsupervised object-centric learning and diffusion models, which is further expanded in Appendix B.
Unsupervised object-centric learning aims to discover the underlying structure of visual data and represent it with a set of feature vectors (a.k.a, slots). To achieve unsupervised scene decomposition, most methods use input reconstruction as the training signal. Earlier works often perform iterative inference to extract object slots, followed by CNN-based decoder for reconstruction [6, 17, 23, 40, 55, 56]. AIR [19] and SQAIR [48] use a patch-based decoder to decode each object a canonical pose, and then transform them back to the original position. Slot Attention [60] and SAVi [46] adopt the spatial broadcast decoder [97] to predict the RGB images and segmentation masks from each slot, which are combined via alpha masking. However, the per-slot CNN decoder limits the modeling capacity of these methods to synthetic datasets [5, 42]. Recently, SLATE [83] and STEVE [84] proposed a Transformer- brassforreer-beeedddddee slot decoder. They first pre-train a dVAE tokenize the input, and then train the alot-conditioned Transformer decoder to reconstruct patch tokens in an autoregressive manner. The powerful Transformer architecture and the feature-level reconstruction target enable them to segment naturalistic images and videos [79, 81, 85). However, as observed in [83, 100], the slot-to-image reconstruction quality of Transformer decoders is low on complex data, sometimes even underperform CNN decoders. In this work, we leverage diffusion models [34, 73] as the slot decoder, which runs iterative denoising conditioned on slots. Thanks their expressive power, our method not only improves scene decomposition, but also generates results with higher visual quality.
Diffusion models (DMs) [34, 87] have recently achieved tremendous progress in image genera- tion [11, 64, 71, 77], showing their great capability in sample quality and input conditioning. The generative process of DMs is formulated as an iterative denoising procedure from Gaussian noise to clean data, where the denoiser is typically implemented as a U-Net [74]. However, the mem- ory and computation requirements of DMs scale quadratically with the input resolution due the self-attention layers in the U-Net. To reduce the training cost, LDM [73] proposes to first down- sample images to feature maps with pre-trained VAE encoder, and then run the diffusion process in this low-resolution latent space. LDM also introduces cross-attention as a flexible mechanism for conditional generation. For example, text-guided LDMs [73] perform cross-attention between text embeddings and U-Net's feature maps at multiple resolutions to guide the denoising process. In this work, we adopt LDM as the slot decoder due to its strong generation capacity, where the conditioning is achieved by cross- atss-attention between the denoising feature maps and the object slots.
While conventional DMs excel in controllable generation such as image editing [31, 44] and composi- tional generation [57, 75], they often require specific supervision such as text to guide the generation

---

Pp
Sp
5
Slot
Predictor
Allention
cross attentton
Image Encoder
N(.
VQ-VAE
LDM
tokenize
Denoiser
Zo
(o(2,,t,)
Figure 1: SlotDiffusion architecture overview. Given an initial frame of a video In. we initialize slots from a set of learnable vectors So, and perform Slot Attention with image features update object slots So. During training, a U-Net denoiser predicts the noise e, added to the image tokens 2g conditioned on slots via cross-attention. The entire model is applied recurrently to all vidco frames with a Predictor initializing future slots from current step.
process. In contrast, SlotDiffusion discovers meaningful visual concepts from data in unsupervised manner, which can be used for controllable generation as will be shown in the experiments.
Concurrent work. Recently, Jiang et al. [41] also employ LDM in object-centric learning. However, they utilize an image tokenizer pre-trained on external datasets, while all components of our model trained purely target datasets. Moreover, their work only evaluates SlotDiffusion on both images and videos. addition, we combine our model with recent development in object-centric learning, showing potential in dynamics modeling and handling real-worle data.
images, while test
3 SlotDiffusion: Object-Oriented Diffusion Model
In this section, we describe SlotDiffusion by first reviewing previous unsupervised object-centric learning methods (Section 3.1). Then, we introduce our slot-conditioned diffusion model decoder (Section 3.2). Finally, we show how to perform compositional generation with the visual concepts learned by SlotDiffusion (Section 3.3). The overall model architecture is illustrated in Figure 1.
3.1 Background: Slot-based Object-Centric Learning
Unsupervised object-centric learning aims to represent a scene with a set of object slots without object- level supervision. Here, we review the SAVi family [46, 84] as they achieve wtateoff-the-a object discovery on videos. SAVi builds upon Slot Attention [60] and runs on videos in a recurrent encoder- decoder manner. Given T input frames {x, ) SAVi first leverages an encoder extract per-frame features, adds positional encodings, and flattens them into set of vectors h, [am) IR Das. Then, the model initializes N slots S, E R
x Dan from set of learnable vectors (t = 1), and updates them with Slot Attention as S; = (S,, hz). fSA performs soft feature clustering, where slots compete with each other to capture certain areas of the input via iterative attention [94]. To achieve temporally aligned slots, SAVi leverages Transformer-based predictor to initialize S, (t 2 2) as S, = Sprea (S,-1). Finally, the model utilizes a decoder fas to reconstruct input æ, from slots S, and uses the reconstruction loss training signal. Below detail the slot decoder faee studied in previous works as we focus on improving it in this paper, and refer the readers to [46] for more information about other model components. See Appendix A for figures of different slot decoders.
Mixture- decoder. Vanilla SAVi [46] adopts the Spatial Broadcast decoder [97] consisting of stack of up-sampling deconvolution layers. It first broadcasts each slot S to a 2D grid to form feature map. Each feature map is then decoded an RGB image y; and an alpha mask m which are combined into the final reconstructed image 2t. SAVi is trained with an MSE reconstruction loss:
m)) f L
m O vi Limage
Σ
I 12
Trransformer-basee decoder: The above mixture-based decoder has limited modeling capacity it decodes each slot separately without interactions. Also, pixel-level reconstruction biases the model to low-level color statistics, which only proves effective objects with uniform colors, and cannot

---

scale to complex data with textured objects. Current state-of of-thhe-rtt art model STEVE [84] thus proposes to reconstruct intermediate features produced by another network [83]. Given frame 2,, STEVE leverages pre-trained dVAF encoder to convert it into sequence of patch tokens which serve as the reconstruction targets for the autoregressive Transformer decoder:
{o:)
02 = f (1). 6, firans (St; of............ ........ of 1) Lioken >>
CrossEntropy(dd d1). (2)
Thanks to the cross-attention mechanism in the Transformer decoder and the feature-level reconstruc- tion objective, STEVE works on naturalistic videos with textured objects and backgrounds. However, the generation quality of the Transformer slot decoder is still low complex data [83, 100].
3.2 Slot-conditioned Diffusion Model
Object-centric generative models [90, 1 110] often decompose the generation process to first predicting object slots in the latent space, and then decoding slots back to the pixel space. Here, the image-to slot abstraction and the slot-to-image decoding are often performed by a pre-trained object-centric model. Therefore, the generation quality is greatly affected by the scene decomposition result and the slot decoder capacity. As observed in [83, 100], methods with mixture-based decoders usually generate images with blurry objects, they fail to capture objects from complex data. While models with Transformer- based decoders are better at scene decomposition, they still produce results of low visual quality. We attribute this to two issues: (i) treating images sequences of tokens ignores their inherent spatial structure; (ii) autoregressive token prediction causes severe error accumulation. In this work, we overcome these drawbacks by introducing diffusion models as the slot decoder, which preserves the spatial dimension of images, and iteratively refines the generation results. From now on, we omit as the video timestamp, and will only use t denote the timestep the diffusion process.
Diffusion model. Diffusion models (DMs) [34, 87] are probabilistic models that learn a data distribution po (X)) by gradually denoising standard Gaussian distribution, in the form of po (x)) Po (Xorr) d.x.rr, where XI:r are intermediate denoising results. The forward process of DMs is a Markov Chain that iteratively adds Gaussian noise to the clean data Xo. which is controlled by pre-defined variance schedule { Let Oz - 33 and ãe II. ass, we have:
((x,(X,-1) = NNX. Br B,T) + x(), = Nxx, Va, (1 T.) (3)
During training, denoising model eØ (1,{) trained to predict the noise applied to a noisy sample:
X, xo + v1 c, LDM llee es (x,, 1)?,, where 6, ~ ~V(0 I).
(4)
At inference time, we can start from a random Gaussian noise, and apply the trained denoiser to iteratively refine the sample. See Appendix C for a detailed formulation of diffusion models.
SlotDiffusion. Our model consists of the same model components as SAVi and STEVE, while only replacing the slot decoder with the DM-based one. Inspired by the success of feature-level reconstruc- tion in STEVE, we adopt the Latent Diffusion Model (LDM) [73] as the decoder, which denoises features in the latent space. This improves the segmentation results with a higher-level reconstruction target, and greatly reduces the training cost as it runs at a lower resolution. Specifically, we pre-train VQ-VAE [72] to extract feature maps RR kxxx Dvn from x before training SlotDiffusion:
(5) where LPIPS(:..) is the VGG perceptual loss [108]. In preliminary experiments, we also tried KAE [73] and VQ-GAN [20] as the image tokenizer, but did not observe clear improvements.
z = (x)) a (o Lvo = | 2||2 LPIPS(xx
To condition the LDM decoder on slots S, we notice that slots are a set of N feature vectors, which similar to text embeddings output by language models [69, 70]. Therefore, we follow recent text-guided LDMs [73] to guide the denoising process via cross-attention:
C CrossAttention(( (Q(eittiiiiiiiei)))e)eeeee (Q()), K(S), VS))..
(6)
Here, Q, K. V are learnable linear projections, č is an intermediate feature map from the denoising U-Net Eg. and is the feature map updated with slot information. An important property of cross- attention is that the result in Eq. (6) is order-invarian to the conditional input S, thus preserving the permutation- eewpriiiiiivvvvvineeeeee of slots, which is key property of object-centric models. In practice, we perform conditioning after several layers in the U-Net at multiple resolutions. Overall, our model is trained with conticoond denoising loss over VQ- VAE feature maps:
2: Vānz + (1 āe)Ec, Latot llee (z, t, S)12, where e ~ (0.
(7)

---

3.3 Compositional Generation with Visual Concepts
Previous conditional DMs [71, 73, 77] rely on labels such as text to control the generation process. In contrast, SlotDiffusion is conditioned on slots output by Slot Attention, which is trained jointly with the DM. We consider each object slot extracted from an image as an instance of a basic visual concept (e.g.. red cube). Therefore, we can discover visual concepts from unlabeled data, and build library of slots for each of them. Then, we can compose concepts via slots to generate novel samples.
Visual concept library. We follow SLATE [83] to build visual concept libraries by clustering slots. Take image datasets for example, we first collect slots extracted from all training images, and then apply K-Means clustering to discover K' clusters, each serves as a concept. All slots assigned to cluster form a library for that concept. On video datasets, we discover concepts by clustering slots extracted from the first frame. As shown in our experiments, SlotDiffusion discovers semantically meaningful concepts, such as objects with different shapes, and different components of human faces.
To generate new samples, we first select N concepts from K' candidates, pick one slot from each concept's 's library, and then decode them with our LDM-based slot decoder. To avoid severe occlusions, we reject slots with object segmentation masks overlapping greater than an mloU threshold.
4 Experiments
SlotDiffusion is a generic unsupervised object-centric learning framework that is applicable to both image and video data. We conduct extensive experiments to answer the following questions: (i) Can we learn object-oriented scene decomposition supervised by the denoising loss of DMs? (Section 4.2)
(ii) Will our L.DM-based decoder improve the visual generation quality of slot models? (Section 4.3)
(iii) Is the object-centric representation learned by SlotDiffusion useful for downstream dynamics
modeling tasks? (Section 4.4) (iv) Can we extend our method to handle real-world data? (Section 4.5)
(v) Can SlotDiffusion benefit from other recent improvements in object-centric learning? (Section 4.6)
(vi) What is the impact of each design choice on SlotDiffusion? (Section 4.7)
4.1 Experimental Setup
Datasets. We evaluate our method in unsupervised object discovery and blot-based visual generation on six datasets, namely, the two most complex image datasets CLEVRTex [43] and CelebA [59] from SLATE [83]. and four video datasets TOOV-DD///ollid/ee: [25] from STEVE [84]. Then, we show SlotDiffusion's capability for downstream video prediction and reasoning tasks on Physion [4]. Finally, we scale our method to unconstrained real-world images on PASCAL VOC 2012 [21] and MS COCO 2017 [54]. We briefly introduce each dataset below and provide more details in Appendix D.I.
CLEVRTex augments the CLEVR [42] dataset with more diverse object shapes, materials, and textures. The backgrounds also present complex textures compared to the plain gray one in CLEVR. We train our model on the training set with 40k images, and test on the test set with 5k samples.
CelebA contains over 200k real-world celebrity images. All images are mainly occupied by human faces, with roughly front-view head poses. This dataset is more challenging as real-world images typically have background clutter and complicated lighting conditions. We train our model on the training set with around 160k images, and test on the test split with around 20k images.
MOVi-D/E/Solid/Tex consist of videos generated using the Kubric simulator [25]. Their videos feature photo-realistic backgrounds and diverse real- world objects. where one or several objects are thrown to the ground to collide with other objects. We follow the official train-test split to evaluate our model.
Physion is VQA dataset containing realistic simulations of eight physical scenarios, such as dropping and sodt-body deformation. The goal of Physion is to predict whether two target objects will contact as the scene evolves. Following the official protocol, we first train models learn scene dynamics and evaluate the video prediction results, and then perform VQA on the model's future rollout.
PASCAL VOC 2012/MS COCO 2017 real-world datasets commonly used in object detection and segmentation. They are more challenging than CelebA as the images capture unconstrained natural scenes with multiple objects of varying sizes. We will denote them VOC and COCO for short.
Baselines. We compare SlotDiffusion with sfate-of-the the- fully unsupervised object-centric models. On image datasets, we adopt Slot Attention (SA for short) [60] and SLATE [83]. On video datasets, we adopt SAVi [46] and STEVE [84]. They are representative models which use the mixture-base based
5

---

Segmentation FG-ARI (T)
Segmentation mloU
Segmentation mBQ
70
Misture
Transfommer
biffusion
40
40
30
50
30
MDVi-SaldlMex--lx
MoVi-ollaMaviies
Figure 2: Unsupervised object segmentation measured by FG-ARI (left), mloU (middle), and mBO (right). Mixture, Transformer, and Diffusion stand for SA/SAVi, SLATE/STEVE, and SlotDiffusion, respectively.
10
13
t=1
10
13
GT
SAVi
STEVE
Ours
Figure Visualization of video segmentation results on MOVi-D (left) and MOVi-E (right).
decoder and the Transformer-based decoder. We will introduce other baselines in each task below. See Appendix D.3 for implementation details of all baselines adopted in this paper.
Our Implementation Details. We use the same image encoder, Slot Attention module, and transition predictor baselines, while only replacing the slot decoder with the conditional LDM. We first pre-train VQ- VAE on each dataset, and then freeze it and train the object-centric model with Eq. (7). Please refer to Appendix D.4 for complete training details, and Appendix E.1 for a comparison of computation requirements of SlotDiffusion and baseline methods.
4.2 Evaluation on Object Segmentation
Evaluation Metries. Following previous works, use foreground Adjusted Rand Index (FG-ARI) to measure how well the predicted object masks match the ground-truit segmentation. As pointed out by [17, 43], FG-ARI is sub-optimal since it only considers foreground pixels and ignores false segmentation of backgrounds. Therefore, we also adopt the mean Intersection over Union (mloU) and mean Best Overlap (mBO) computed over all pixels. See Appendix D.2 for metric implementations.
Results. Figure presents the results on unsupervised object segmentation. We do not evaluate on CelebA it does provide object mask annotations. SlotDiffusion outperforms baselines in three metrics across all datasets, proving that the denoising loss in LDMs is good training signal for scene decomposition. Compared to CLEVRTex, our method achieves larger improvements on video datasets since SlotDiffusion also improves the tracking consistency of objects. Figure 3 shows qualitative result of video segmentation. As observed in previous work [84], MOVi-E's moving camera provides motion cues compared to static cameras in MOVi-D, making object discovery easier. Indeed, SAVi degenerates to stripe patterns on MOVi-D, but is able to produce meaningful masks on MOVi-E. Compared to STEVE, our method has fewer object-sharing issues (one object being captured by multiple slots), especially on large objects. See Appendix E.2 for more qualitative results.
4.3 Evaluation on Generation Capacity
Evaluation Metrics. To generate high-quality results, object-centric generative models need to first represent objects with slots faithfully, and then compose them novel ways. Hence, we first evaluate slot-to-image reconstruction, which inspects the expressive power of object slots. We adopt MSE and IPPIS [108] following previous works. As discussed [100, 108], IPIPS better aligns with human perception, while MSE favors blurry images. We thus focus comparing LPIPS, and report MSE

---

Reconstruction MSE
Reconstruction LPIPS
Tansformer
600
Diffuiion (Ouns]
0.4
300
290
0.3
Figure 4: Slot-based image and video reconstruction measured by MSE (left) and LPIPS (right).
GT
SA
SLATE
Ours
SA
SLATE
Ours
GT
SA
SLATE
Ours
Figure 5: Reconstruction results CLEVRTex (left) and CelebA (right). Our method hetter preserves the textures on objects and backgrounds CLEVVTex, and the human hairs and facial features CelebA
Image
Slots as Visual Concepts
Generation FID (
Generation FVD (4)
200
2000
Mixture
175
3750
Transformer
Diffusion
100
1000
Novel Concept Composition
Result
75
750
50
500
MoU-Tox
(a) Visual concepts discovery and composition
(b) Quantitative results compositional generation
Figure 6: The mechanism of slot-based compositional generation (a) and the quantitative results (b). Our method first discovers visual concepts from images the form of slots, such as objects and hackground maps in CLEVRTex. Then, we can synthesize novel samples by composing concept slots from different images.
Dataset
SA
SLATE
Ours
Dataset
SLATE
Ours
Dataset
SAVi
STEVE
Ours
Figure 7: Compositional generation samples on CLEVRTex (left), CelebA (middle), and MOVi-E (right). We randomly select slots from the visual concept libraries built from training data, and decode them into images videos. SlotDiffusion synthesizes higher-fidelity results with more details compared basclines.
for completeness. In addition to reconstruction, we conduct compositional generation with the visual concepts learned in Section 3.3. FID [32] and FVD [92] are used to measure the generation quality.
Results on reconstruction. Figure presents the reconstruction results. SlotDiffusion outperforms both baselines with sizeable margin in LPIPS on all datasets, and achieves the second best MSE. As shown the qualitative results Figure 5, SA with a mixture-based decoder produces blurry images despite lower MSE. Also, the Transformer-be based decoder reconstructs distorted objects, and loses all the textures on object surfaces and backgrounds. In contrast, our LDM-based decoder is able to iteratively refine the results, leading accurate textures and details such as human hairs.
Results on compositional generation. As shown in Figure (a), we discover visual concepts by clustering slots extracted from images, and randomly composing slots from each concept's 's library to generate novel samples. Figure 6 (b) summarizes the quantitative results. SlotDiffusion significantly improves the FID and FVD scores, since it synthesizes images and videos with coherent contents and much better details compared to baselines. This verified by the qualitative results in Figure See Appendix E.3 for more visualizations of compositional generation results.

---

Original
Remove
Segment
Insert
BG
Swep 9G
Replace Face
(a) Scene editing CLEVRTex
(b) Face replacement on CelebA
Figure Slot-based image editing. Left: we can manipulate objects, retrieve and swap background maps in a scene. Right: we can replace the human faces in the first column with the faces from the first row.
Table Physion VQA accuracy. report observation (Obs.)
Table 1: Video prediction results on Physion.
observation plus rollout (Dyn.), and the improvement (f).
Method
MSE
LPIPS
FVD
Method
Obs. (%)
Dyn. (%)
f (%)
PredRNN
286.3
0,38
756.7
RPIN
62.8
63.8
+1.0
VQFormer
619.0
0.33
694.5
PDEIT-Istm
59.2
60.0
+0.8
STEVE + SlotFormer
832.0
0.43
930.6
STEVE + SlotFormer
65.2
67.1
+1.9
Ours SlotFormer
477.7
0.26
582.2
Ours + SlotFormer
67.5
69.7
+2.2
t=
10
15
20
25
30
35
40
45
STEVE + SlotF
Ours SlotFormer
Figure 9: Qualitative results of video prediction on Physion. The first row shows ground ground-truth frames, while the other rows show the model rollout results. See Appendix E.4 for visualizations of all baselines.
Slot-based image editing. In Figure 8, we apply SlotDiffusion to image editing by manipulating slots. On CLEVRTex, we can remove an object by segmenting it out, and insert it to another scene. We can also extract background maps from an image and swap with another image. On CelebA, SlotDiffusion decomposes images to faces, hairs, clothes, and backgrounds. By swapping the face slot, our method generates realistic images with new faces, while keeping other components unchanged.
4.4 Downstream Tasks: Video Prediction and VQA
In this subsection, we evaluate the object slots learned by SlotDiffusion in downstream tasks. We select Physion dataset as requires capturing the motion and physical properties of objects for accurate dynamics simulation. We train the state-of-the-art object-centric dynamics model SlotFormer [100] over slots extracted by our model, and unroll it to perform video prediction and question answering.
Video prediction baselines. We adopt three baselines that build upon different image feature representations: PredRNN [96] that uses global image feature maps, VQFormer which utilizes image patch tokens encoded by VQ- VAE [72], and STEVE + SlotFormer that runs the same SlotFormer dynamics model over object-centric slot representations extracted by STEVE.
Results on video prediction. Table 1 presents the results on the visual quality of the generated videos. SlotDiffusion with SlotFormer outperforms baselines clearly in LPIPS and FVD, and achieves a competitive result on MSE. As discussed in Section 4.3 and [100], MSE is a poor metric in generation. Figure visualizes the model rollouts. Compared to STEVE with FlotFormer, our method generates videos with higher fidelity such as the consistent boxes and the red object.

---

Table 3: Unsupervised object segmentation results real-world datasets: VOC (left) and COCO (right).
PASCAL VOC
FG-ARI
mBO'
mBO
MS COCO
FG-ARI
mBO'
mBO
SA + DINO VIT SLATE + DINO VIT DINOSAUR
12.3
24,6
24,9
SA + DINO VIT
21,4
17.2
19.2
15.6
35.9
41.5
SLATE + DINO VIT
32.5
29.1
33.6
23.2
43.6
50.8
DINOSAUR
34.3
32.3
38.8
Ours DINO VIT
17.8
50.4
55.3
Ours DINO VIT
37.2
31.0
35.0
GT
Ours
Figure 10: Segmentation results on real-worle images: COCO (left) and VOC (right). SlotDiffusion is able to
discover unannotated but meaningful regions highlighted by the red arrows, such as trees and keyboards.
Table 4: Unsupervised object segmentation results SlotDiffusion with other improvements on CLEVRTex.
Method
FG-ART
mloU
mBO
Method
FG-ARI
mloU
mBO
BO-QSA
80.47
64,26
66.27
SlotDiffusion
68.39
57.56
61.94
SlotDiffusion + Slot-TTA
69.66
58.67
62.82
SlotDiffusion 68.39
57.56
61.94
BO-SlotDiffusion 78.50
64.35
68.68
VQA baselines. We select two baselines following [100]. RPIN is an object-centric dynamics model with ground- truth object bounding box information. PDE/T-Istm builds LSTM over frame features extracted by an ImageNet [10] pre-trained DeiT [91]. We also compare with STEVE + SlotFormer. Results on VQA. Table 2 summarizes the VQA accuracy on observation only (Obs.) and observation plus rollout frames (Dyn.). SlotDiffusion with SlotFormer achieves the best result in both cases, with the highest rollout gain. This proves that our method learns informative slots which capture correct features for learning object dynamics, such as the motion of objects falling down in Figure 9.
4.5 Scaling to RerldWorld Data
To show the scalability our method to real-world data, we follow [81] to leverage pre-trained models. We replace the image encoder in SlotDiffusion with self-supervised DINO [7] pre-trained ViT encoder. The model is still trained with the LDM denoising loss over VQ- VAE image tokens.
Baselines. To directly compare the effect of the slot decoders, we adopt SA and SLATE with the same DINO pre-traine pre-trained ViT image encoder. We also compare with DINOSAUR [81] with the same image encoder, which reconstructs the ViT feature maps instead of input images as training signal.
Results. Table 3 presents the segmentation results VOC and COCO. We follow [81] report mBO' and mBO which are mBOs between predicted masks and ground-truth instance and semantic segmentation masks. SlotDiffusion consistently outperforms SA and SLATTE over all metrics, proving the effectiveness of our LDM-bbsed slot decoder. Our model is also competitive with the state-of-the- art method DINOSAUR. The performance variations may because COCO has more instances per image than VOC. As shown in the qualitative results in Figure 10, SlotDiffusion is able to segment the major objects in an image, and discover unannotated but semantically meaningful regions.
4.6 Combining SlotDiffusion with Recent Improvements in Slot-based Models
As an active research topic, there have been several recent improvements on slot-based models [39, 67, 85]. Since these ideas are orthogonal our modifications the slot decoder, we can incorporate them into our framework to further boost the performance.
Slot-TTA [67] proposes test-time adaptation method by optimizing the reconstruction loss of slot models on testing images. Similarly, we optimize the denoising loss of SlotDiffusion at test time. As shown in Table left, Slot-TTA consistently improves the segmentation results of method.
BO-QSA [39] applies bi-level optimization in the iterative slot update process. We directly combine it with SlotDiffusion (dubbed BO B0-SlotDiffusion) As shown in Table right, BO BO-DlotDiftusion greatly

---

700
MOVI-D MOVI-E
650
MSC
600
FG-ARI
mlou
30
28
500
1500 2000
5.3
1000
1500 2000
450
1000
2000
1000
2000
Diffusion Steps T
Diffusion Steps
Diffusion Steps
Diffusion Steps
Figure 11: Ablation study on the number of diffusion steps T of SlotDiffusio.. We show the reconstruction performance (MSE, LPIPS) and segmentation results (FG-ARI, mloU) MOVi-D and MOVi-E. We also plot the best baseline results each dataset dashed lines. We do not plot MSE favors blurry generations.
improves the segmentation results compared to vanilla SlotDiffusion, and achieves comparable performance with BO-QSA. Note that the generation quality of BO-QSA is close to vanilla Slot Attention, and thus both our methods are significantly better than it.
4.7 Ablation Study
We study the effect of the number of diffusion process steps T in our LDM-based decoder. We plot both the reconstruction and segmentation results on MOVi-D and MOVi-E in Figure 11. As expected, using more denoising steps leads to better generation quality. Interestingly, smaller 7 results in better segmentation performance. This indicates that there is trade-off between the reconstruction and segmentation quality of our model, and we select T' 1000 to strike a good balance. Literature on self-supervised representation learning [29, 38] suggests that (reasonably) more difficult pretext task usually leads to better representations. Analogously, we hypothesize that a smaller makes the denoising pretext task harder, resulting in better object-centrie representations.
Overall, SlotDiffusion outperforms baselines consistently across different T in LPIPS, mloU, and FG-ARI, proving that our method is robust against this hyper-parameter.
5
Conclusion
In this paper, we propose SlotDiffusio by introducing diffusion models to object-centric learning. Conditioned on object slots, our LDM-based decoder performs iterative denoising over latent image features, providing strong learning signals for unsupervised scene decomposition. Experimental results on both image and video datasets demonstrate our advanced segmentation and visual generation performance. Moreover, SlotDiffusion can be combined with recent progress in object-centric models to achieve state-of-the-ard video prediction and VQA results, and scale up to real-world data. We briefly discuss the limitations and future directions here, which are further extended in Appendix F.
Limitations. With the help of pre-trained encoders such as DINO ViT [7], SlotDiffusion is able to segment objects from naturalistic images [21, 54]. However, we are still unable to decode natural images faithfully from slots. In addition, objects in real-world data are not well-defined. For example, the phrt-whole hierarchy causes ambiguity in the segmentation of articulated objects, and the best way of decomposition depends on the downstream task. Finally, our learned slot representations are still entangled, while many tasks require disentanglement between object positions and semantics.
Future Works. From the view of generative modeling, we can explore better DM designs to improve the modeling capacity of local details. To generate high-quality natural images, one solution is to incorporate rre-trained generative models such as the text- -oiidgu Stable Diffusion [73]. From the view of object-centric learning, we want to apply SlotDiffusion to more downstream tasks such as reinforcement learning [95, 105] and 3D modeling [86, 104]. It is interesting to see if task-specific supervision signals (e.g., rewards in RL) can lead to better scene decomposition.
Broader Impacts
Unsupervised object-centric learning is important task in machine learning and computer vision. Equipping machines with structured latent space enables more interpretable and robust AI algo- rithms, and can benefit downstream tasks such as scene understanding and robotics. We believe this work benefits both the research community and the society.
Potential negative societal impacts. We do not see significant risks of human rights violations or security threats our work. However, since our method can be applied to generation tasks, might generate unpleasing content. Future research should also avoid malicious uses such as social surveillance, and be careful about the training cost of large diffusion models.
10

---

Acknowledgments
We would like thank Gautam Singh, Maximilian Seitzer for the help with data preparation and discussions about technical details of baselines, Xuanchi Ren, Tianyu Hua, Wei Yu, all colleagues from TISL for fruitful discussions about diffusion models, and Xuanchi Ren, Liquan Wang, Quiney Yu. Calvin Yu, Ritviks Singh for support in computing resources. We would also like to thank the anonymous reviewers for their constructive feedback. Animesh Garg is supported by CIFAR AI chair, NSERC Discovery Award, University of Toronto XSeed Grant, and NSERC Exploration grant. We would like to acknowledge Vector Institute for computation support.
References
[1] Federico Baldassarre and Hossein Azizpour. Towards self-supervised learning of global and object-centric representations. ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
[2] Fan Bao, Chongxuan Li, Jiacheng Sun, and Jun Zhu. Why are conditional generative models better than unconditional ones? In NeurIPS 2022 Workshop Score-Based Methods, 2022.
[3] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In ICLR, 2022.
[4] Daniel Bear, Elias Wang, Damian Mrowca, Felix Jedidja Binder, Hsiao-Yi Yu Tung, RT Pramod, Cameron Holdaway. Sirui Tao, Kevin A Smith, Yun Sun, et al. Physion: Evaluating physical prediction from vision in humans and machines. In NeurlPS Datasets and Benchmarks Track, 2021.
[5] Chris Burgess and Hyunjik Kim. 3d shapes dataset. httpss//gitibboomm///pppnnlillas dataset/, 2018.
[6] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv: 1390, 2019.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[8] Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yes Kenneth Wong, Joshua B Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. In ICLR, 2020.
[9] Antonia Creswell, Rishabh Kabra, Chris Burgess, and Murray Shanahan. Unsupervised object-based based transition models for 3d partially observable environments. NeurlPS, 2021.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. NeurIPS, 2021.
[12] David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning. NeurIPS, 2021.
[13] Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Josh Tenenbaum, and Chuang Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. NeurIPS, 2021.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. In ICLR, 2020.
11

---

[15] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high- quality dataset of 3d scanned household items. arXiv preprint arXiv:2204. /9110, 2022.
[16] Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. NeurIPS, 2022.
[17] Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gen- erative scene inference and sampling with object-centric latent representations. In ICLR, 2019.
[18] Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. NeurlPS, 2021.
[19] SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. NeurIPS, 2016.
[20] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[21] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisser- man. The pascal visual object classes (voc) challenge. IJCV, 2010.
[22] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. NeurlPS, 2017.
[23] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-object representation learning with iterative variational inference. In ICMI., 2019.
[24] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020.
[25] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubrie: A scalable dataset generator. In CVPR, 2022.
[26] William Harvey and Frank Wood. Visual thain-of-though diffusion models. arXiv preprint arXiv:2303. 16787, 2023.
[27] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. NewrIPS, 2022.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[29] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[30] Olivier Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. In ECCV, 2022.
[31] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint 22X60.200..01666, 2022.
[32] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. NeurIPS, 2017.
[33] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021.
12

---

[34] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
[35] Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22 2022.
[36] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In NeurIPS, 2022.
[37] Tao V Hu, David W Zhang, Yuki M Asano, Gertjan Burghouts, and Cees G M Snoek. Self-guided diffusion models. In CVPR, 2023.
[38] Tianyu Hua, Yonglong Tian, Sucheng Ren, Hang Zhao, and Leonid Sigal. Self-supervision through random segments with autoregressive coding (randsac). In ICIR,, 2023.
[39] Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimization. In ICLR, 2022.
[40] Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. ICLR, 2019.
[41] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. arXiv preprint arXiv:2303. 2023.
[42] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
[43] Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. In NeurlPS Datasets and Benchmarks Track, 2021.
[44] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276, 2022.
[45] Thomas Kipf, Elise van der Pol, and Max Welling, Contrastive learning of structured world models. In ICLR, 2019.
[46] Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object- centric learning from video. In ICLR, 2021.
[47] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint 2XXiv:3304.02443 2023.
[48] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. NeurlPS, 2018.
[49] Jannik Kossen, Karl Stelzner, Marcel Hussing, Claas Voelcker, and Kristian Kersting. Struc- tured object-aware physics prediction for video modeling and planning. In ICLR, 2019.
[50] Brenden M Lake, Tomer Ullman, Joshua B Tenenbaum, and Samuel J Gershman, Building machines that learn and think like people. Behavioral and brain sciences, 2017.
[51] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, 2020.
[52] Alexander Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. arXiv preprint 2rXiv:2303.16203 2023.
13

---

[53] Chen-Hsuai Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler. Ming- Yu Liu, and Tsung-Y Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
[54] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[55] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng. Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In ICLR, 2019.
[56] Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving generative imagination object-centrie world models. In ICMI., 2020.
[57] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. ECCV, 2022.
[58] Zhen Liu, Yao Feng, Michael Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. In ICLR, 2023.
[59] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
[60] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centrie learning with slot attention. NeurlPS, 2020.
[61] Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. arXiv preprint arXiv:2011. 10287, 2020.
[62] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022.
[63] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2021.
[64] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[65] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
[66] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
[67] Mihir Prabhudesai, Anirudh Goyal, Sujoy Paul, Sjoerd Van Steenkiste, Mehdi SM Sajjadi, Gau- rav Aggarwal, Thomas Kipf, Deepak Pathak, and Katerina Fragkiadaki. Test-time adaptation with slot-centric models. In /CML. 2023.
[68] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward a meaningful and decodable representation. In CVPR, 2022.
[69] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. et al. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[70] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits transfer learning with a unified text-to-text transformer. JMLR, 2020.
14

---

[71] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint (xXiv:204.001255 2022.
[72] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. NeurIPS, 2019.
[73] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[74] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[75] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12242, 2022.
[76] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: mmage-to-image diffusion models. In ACM SIGGRAPHI 2022 Conference Proceedings, 2022.
[77] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
[78] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super- repouutesoo via iterative refinement. TPAMI, 2022.
[79] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. NeurlPS, 2022.
[80] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 2021.
[81] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl- Johann
Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, al. Bridging the gap to real-world object-centric learning. In ICLR, 2023.
[82] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, al. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023.
[83] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In ICLR, 2021.
[84] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. NeurlPS, 2022.
[85] Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. In ICLR, 2023.
[86] Cameron Smith, Hong-Xina Yu. Sergey Zakharov, Fredo Durand, Joshua B Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint arXiv:2205.03923 2022.
[87] Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICML, 2015.
[88] Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 2007.
[89]
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.0/ 148, 2021.
[90] Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, Devon Hjelm, and Shikhar Sharma. Object-centric image generation from layouts. In AAAI, 2021.
15

---

[91] Hugo Touvron, Matthieu Cord. Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training cata-efficient image transformers distillation through attention. In ICMI,, 2021.
[92] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michal- ski, and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv preprint arXiv:/8/2.01177,, 2018.
[93] Sjoerd van Steenkiste, Michael Chang. Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions, In ICLR, 2018.
[94] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
[95] Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jia- jun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In CoRI., 2020.
[96] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predrın: Recurrent neural networks for predictive learning using spatiotemporal Istms. Neur/PS, 2017.
[97] Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vacs. arXiv preprint arXiv: 1901:0701110111 2019.
[98] Chen Wei, Karttikeya Mangalam, Po-Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models as masked autoencoders. arXiv preprint arXiv:2304.0328. 2023.
[99] Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping NeurIPS, 2022.
[100] Ziyi Wu, Nikita Dvomik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsu- pervised visual dynamics simulation with object-centric models. In ICL.R, 2023.
[101] Jiarui Xu, Sifci Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. vocabulcayylaryy panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
[102] Charig Yang. Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. ICCV, 2021.
[103] Yufei Ye, Maneesh Singh, Abhinav Gupta, and Shubham Tulsiani. Compositional video prediction. In ICCV, 2019.
[104] Hong-Xing Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In ICLR, 2022.
[105] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Self-supervised visual reinforce- ment learning with object-centric representations. In ICLR, 2020.
[106] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real-world videos by predicting temporal feature similarities. arXiv preprint 230.:2:30300022,, 2023.
[107] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXivv2302.05543, 2023.
[108] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
[109] Artem Zholus, Yaroslav Ivchenkov, and Aleksandr Panov. Factorized world models for learning causal relationships. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
[110] Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo J Rezende. Parts: Unsupervised segmentation with slots, attention and independence maximization. In ICCV, 2021.
16

---

A Model Architecture
Figure 12 shows the general pipeline of video Slot Attention models, and compare existing slot decoders with our proposed Latent Diffusion Model (LDM) [73] based decoder.
S
St
S
Slot
Predictor
Attention
Image Encoder
Slo: Decoder
Loss
(a) General pipeline of video-based Slot Attention models.
weighted
200
CNN Decoder
MSE Loss
&
X
(m,
(b) Mixture-based CNN decoder.
S
2
Trans. Decoder
dVAE
flatten
3
4
CE Loss
tokenize
(c) Transformer-hawee hased autoregressive decoder.
MSE Loss
VQ-VAE
LDM Denoiser
takenize
Zo
2
Cg t,
cross- attention
O
(d) Latent diffusion model based decoder.
Figure 12: Illustration (a) the training pipeline video Slot Attention models, (b) SAVi's mixture-based decoder [46], (c) STEVE's Transformer-bar based decoder [84], and (d) our LDM-based decoder.
17

---

B Additional Related Work
Unsupervised object-centric learning from images. Our work is directly related to research that aims at learning to represent visual data with a set of feature vectors without explicit supervision. Earlier attempts start from synthetic datasets with naive shape primitives, uniformly colored objects, and simple backgrounds [6. 17, 18, 19, 22, 23, 60]. They typically perform iterative inference to extract object-centric features from images, followed by mixture-based CNN decoder applied to each slot separately for reconstruction. For example, MONet [6] and Slot Attention [60] both adopt the spatial broadcast decoder [97] to predict an RGB image and an objectness mask from each slot, and combine them via alpha masking. Recently, SLATE [83] challenges the traditional design with a Transformer-basee decoder, which helps the model scale to more complex data such as textured objects and backgrounds. Such decoder has also been employed in later works [79, 85], proving its effectiveness on naturalistic inputs. Another line of works [1, 30, 81, 99] focuses on feature-level reconstruction or contrastive learning for object discovery without pixel-level decoding. However, they are not applicable for generation tasks as they cannot reconstruct images from slots. With the recent progress in 3D reconstruction, there are also methods leveraging Neural Radiance Fields (NeRF) [89, 104] and light fields [79, 86] as the slot decoder. However, they usually require multi- view images and camera poses as training data, which is incompatible with our setting. Compared to previous works, SlotDiffusion is the first attempt introducing diffusion models to unsupervised object-centric learning, achieving ofate-of-the thef--th-arrt art results in both object discovery and generation.
Unsupervised object-centric learning from videos. Compared to images, videos provide additional information such as motion cues. Our work also builds upon recent efforts in decomposing raw videos into temporally aligned object slots [40, 46, 48, 56, 93, 106]. These works usually integrate a dynamics module to model the interactions between objects compared to their image-based counter- parts. For example, STOVE [49] adopts a Graph Neural Network (GNN), while OAT [9]. SAVi [46], and PARTS [110] leverage the powerful Transformer architecture. However, the limited modeling capacity of their mixture- based decoders still prevents them from scaling to more realistic videos. Some works [16, 102] thus introduce additional supervisions such as optical flow, depth maps, and initial object positions. Recently, STEVE [84] pushes the limit of fully unsupervised object- learning videos with the Transformer slot decoder from SLATE and the Transformer dynamics module from SAVi. Nevertheless, the autoregressive generation mechanism of Transformer degrades its generation capacity, which we aim to address in this paper. There are also methods learning scene decomposition from videos by contrasting feature [45, 61], which cannot generate videos. Similar to SAVi and STEVE, SlotDiffusion can also be applied to video data with the Transformer-basee dynamics module. In addition, we generate videos with much better temporal consistency.
Diffusion model for generation. Recently, diffusson models [87] (DMs) have achieved tremendous progress in generation tasks, including 2D images [11, 34, 64, 65, 71, 77], 2D videos [27, 35, 36, 82], 3D shapes [53, 58, 66), and scene editing [31, 44, 63, 75], showing their great capacity in generation quality and input conditioning. The generative process of DMs is formulated as an iterative denoising procedure with a denoising network, usually implemented as U-Net [74]. However, the memory consumption of DMs scales quadratically with the input resolution due to the use of self-attention layers in the U-Net. To reduce the training cost, LDM [73] proposes to run the diffusion process in the latent space of a pre-trained image auto-encoder, e.g.. VQ-VAE [72]. LDM also introduces a flexible conditioning mechanism via cross-attention between the U-Net feature maps and the conditional inputs. For example, recent text-guided LDMs [73] first extract text embeddings using pre-trained language models [69, 70], and then perform cross-attention between the language features and the U-Net denoising feature maps at multiple resolutions. In this work, we adopt the LDM as our slot decoder due to its strong generation capacity and low computation cost. To condition the decoder on ID object slots, we draw an analogy with ID text embeddings, and also apply cross- -ross-attention between slots and feature maps in the U-Net denoiser.
Diffusion model for representation learning. There have been a few works studying the repre- sentation learned by DMs. Baranchuk et al. [3] and Xu et al. [101] utilize the denoising feature maps of DMs for data-efficient segmentation, as the feature maps already capture object boundary information. Preechakul et al. [68] jointly trains a CNN encoder and DM to perform auto-encoding for representation learning. DifIMAE [98] combines MAE [29] and DMs to learn better inpainting. Li et al. [52] leverages pre-trained text- -xxidg DMs for zero-shot image classification. In contrast, SlotDiffusion does not rely on any supervision or pre- pre-trained DMs. Instead, we incorporate it with the unsupervised Slot Attention framework to leam structured object-centric representations.
18

---

Conditional diffusion model without labels. Several works have explored learning conditional DMs in an unsupervised manner. Hu et al. [37] and Bao et al. [2] utilize self-supervised feature extractors to obtain image features, and then perform clustering to create pseudo labels such as cluster ids for training conditional DM. Harvey and Wood [26] show that conditional DMs usually outperform unconditional DMs in terms of FID. Inspired by this observation, they leverage CLIP embeddings [69] of unlabeled images to train conditional DMs without manual annotations. Similarly, SlotDiffusion also learns a conditional DM without labels. Instead of relying on pre-trained feature extractors, we build upon the unsupervised Slot Attention framework, and can learn both object- centric representations and conditional DMs from scratch.
C Details on Diffusion Models
Diffusion models are probabilistic models that learn data distribution po (X)) by gradually denoising a standard Gaussian distribution, the form of po (Xo) = f pe oo.r ixT::-- Here, X1.7 are intermediate denoising results with the same shape the clean data Xo (X), and @ are learnable parameters of the denoising U-Net model.
The joint distribution | is called the forward process or diffusion process, which is fixed Markov Chain that gradually adds Gaussian noise to Xo. The noise is controlled by a pre-defined variance schedule [B1]T=1
T
q( X X
II (X((xx)),, x(x, = XXX (I.IVVVVV
(8)
Thanks to the nice property of Gaussian distributions, X, can be sampled directly from X, in closed form without adding the noise times. Let 0r 3r and a. II-=1 (xs, we have:
((., = N 0r Xo, (1- x, Va, Xo + v1 - aett, where & NO
We can now train a model to reverse this process and thus generate target data from random noise XT ~ -/ I). The reverse process pe (X)o)) is also defined as a Markov Chain with a learned Gaussian transition:
T
Po (o.T) = ((T)) II Po Xt), Po (X X.) = NN (X,( (x,,))) ()) (10)
In practice, we do not learn the variance and usually set it to Σ, = 3,1 or 1- Bel since it leads to unstable training [34]. Also, instead of learning the mean lam directly, we learn to predict the noise & in Equation (8). See Ho al. [34] for how we can sample X,-1 given X, and the predicted e, at the inference stage.
The training process of diffusion models is thus straightforward given Equation (9). each training step, we sample a batch of clean data Xo from the training set, timesteps i uniformly from {1,..., 7). and random Gaussian noise & (O, I). We then create the noisy version of data X, by applying Equation (9). A denoising model eo is trained to predict the noise with an MSE loss:
CDm Ex llle- eo (X,,
(11)
D Detailed Experimental Setup
In this section, we detail the datasets, evaluation metrics, baseline methods used in the experiments, and the implementation details of our model.
D.1 Datasets
CLEVRTex [43]. This dataset augments the CLEVR [42] dataset with more diverse object shapes, materials, and textures. The backgrounds in CLEVRTex also present complex textures compared
to the plain gray ones in CLEVR. Therefore, this dataset is visually much more complex than CLEVR. As shown in their paper, only out 10 benchmarked unsupervised object-centric models can achieve mloU higher than 30%. We train our model on the training set consisting of 40k images, and test on the test set with 5k samples. We use the same data pre-processing steps, i.e. first center-crop to 192x 192, and then resize to 128x 128.
19

---

CelebA [59]. This dataset contains over 200k real-world celebrity images. All images are mainly occupied by human faces, covering a large variation of poses and backgrounds. This dataset is more challenging compared to previous synthetic datasets as real-world images typically have background clutter and complicated lighting conditions. We train our model on the training set with around 160k images, and test on the test split with around 20k images. For data pre- pre-processing, we simply resize all images to 128 128.
MOVi-D/E [25]. MOVi-D and MOVi-E are the 2 most challenging versions from the MOVi benchmark generated using the Kubric simulator. Their videos feature photo-realistic backgrounds and real-world objects from the Google Scanned Objects (GSO) repository [15], where one or several objects are thrown to the ground collide with other objects. Compared to MOVi-D, MOVi-E applies linear camera motion. We follow the official train-test split to evaluate our model. For data pre-processing, we simply resize all frames to 128 128.
MOVi-Solid/Tex [84]. MOVi-Solid and MOVi-Tex share similar backgrounds and object motions with MOVi-D and MOVi-E. Besides, MOVi-Solid introduces more complex shapes, and MOVi- Tex maps richer textures to the object surfaces, making them harder to be distinguished from the background. We follow the train-test split in [84], and resize all frames to x x128.
Physion [4]. This dataset contains eight categories of videos, each showing a common physical phenomenon, such as rigid- and soft-body collisions, falling, rolling, and sliding motions. The foreground objects used in the simulation vary in categories, textures, colors, and sizes. It also uses diverse background as the scene environment, and randomize the camera pose in rendering the videos. Physion splits the videos into three sets, namely, Training, Readout Fitting, and Texting. We truncate all videos by 150 frames as most of the interactions end before that, and sub-sample the videos by factor of 3 for training the dynamics model. Following the official evaluation protocol, the dynamics models are first trained on videos from the Training set under future prediction loss. Then, they observe the first 45 frames of videos in the Readout Fitting and Testing set, and perform rollout to generate future scene representations (e.g., image feature maps or object slots). A linear readout model is trained on observed and rollout scene representations from the Readout Fitting set to classify whether the two cued objects (one in red and one in yellow) contact. Finally, the classification accuracy of the trained readout model on the Testing set scene representations is reported. Please refer to their paper [4] for detailed descriptions of the VQA evaluation. For video prediction, we evaluate on the test set videos. All video frames are resized to 128x 128 as pre-processing.
PASCAL VOC 2012 [21]. PASCAL VOC is real-world image dataset containing natural scenes with diverse objects. We follow Seitzer et al. [81] to train on the "trainaug" variant with 10,582 images, and test on the official validation set with 1.449 images. For data pre-processing, we resize the smaller dimension of the image to 224, and perform random cropping to x24x followed by random horizontal flip during training. For testing, we evaluate the 224x224 center crop of images, and ignore pixels that are unlabeled.
Microsoft COCO 2017 [54]. Similar to VOC, MS COCO is world benchmark commonly used in object detection and segmentation. We follow Seitzer et al. [81] to train on the training set with 1 118,287 images, and test the validation set with 5,000 images. For data prepprcesssigg we resize the smaller dimension of the image to 224, and perform center crop to 224x224 for both training and testing. We apply random horizontal flip during training. During evaluation, we filter out crowd instance annotations, and ignore pixels where objects overlap.
D.2 Evaluation Metrics
To evaluate the generation quality of SlotDiffusion, we compute the mean squared error (MSE) of the images reconstructed from the object slots. Following prior works, we scale the images to [o, 11, and sum the errors over channel and spatial dimensions. As pointed out by Zhang et al. [108], MSE does not align well with human perception of visual quality as it favors over-smooth results. Therefore, we additionally compute the VGG perceptual distance (LPIPS) [108]. For video generation, we compute the fer-frame MSE and LPIPS, and then average them over the temporal dimension. Finally, to evaluate the compositional generation results, we adopt the Fréchet Inception Distance (FID)' [32] and the Frécher Video Distance (FVD)² [92]. We compute these metrics between 5k generated images or 500 generated videos and the entire training set of that dataset.
I IImplementation from http:///ghhaahccom/mmmrrrrrryyyt
2 Implementation from http://ggiaabbooommmnmieoeeeee/s
20

---

Table 5: Variations in model architectures and training settings different datasets.
Dataset
CLEVRTex
CelebA
MOVi-D/E
MOVi-Solid/Tex
Physion
VOC / COCO
Image Encoder fane
ResNet18
ResNer18
ResNet18
Plain CNN
ResNet18
VIT-S/8
Number of Slots N
11
4
15
12
8
6/7
Slot Size Dalot
192
192
192
192
192
192 / 256
Slot Attention Iteration
2
2
2
3
Max Learning Rate
2e.4
2e-4
le-4
le-4
le-4
le-4
Gradient Clipping
1.0
1.0
0.05
0,05
0.05
0.05
Batch Size
32
32
48
64
Training Epochs
100
50
30
30
10 500 100
To evaluate the scene decomposition results, we compute the FG-ARI, mloU, and mBO of the object masks. FG-ART is widely used in previous object-centric learning papers, which only consider foreground objects. As suggested in [17, 43], we should also compute the mloU which evaluates background segmentation. For mBO, first assigns each ground-truth object mask with the max- overlapping predicted mask, and then averages the IoU of all pairs of masks. It is worth noting that on video datasets, we flatten the temporal dimension into spatial dimensions when computing the metrics, thus taking the temporal consistency of object masks into consideration, i.e., slot ID swapping of the same object will degrade the performance. Being able consistently track all the objects is an important property of video slot models.
D.3 Baselines
Object-centric models. We adopt Slot Attention [60] and SAVi [46] as representative models which use a mixture-based CNN decoder, and SLATE [83] and STEVE [84] for models with a Transformer based decoder. We use the unconditional version of these models, i.e., they are not trained with the initial frame supervision proposed in [46]. Since we augment SlotDiffusion with stronger ResNet 8 encoder compared to the previous stacked CNN encoder, we also re-train baselines with the same ResNetl 8 encoder, which achieves better performance than reported in their papers. In addition, we tried scaling up the slot decoder of baselines by adding more layers and channels, but observed severe training instability beyond certain point. Instead, SlotDiffusion shows great scalability with model sizes. Its performance grows consistently with larger LDM U-Net. On real-world datasets PASCAL VOC and COCO, we directly copy the numbers of DINOSAUR [81] from its paper, as they did not release their code until 15 days before the paper submission deadline. For Slot Attention and SLATE here, replace the CNN encoder with the same DINO [7] pre-trained VIT encoder. We trained longer enough and achieved better results than reported in the DINOSAUR paper.
Video prediction and VQA. We adopt the same baselines as SlotFormer [100]. Please refer to their papers about the implementation details of these future prediction models. We train baselines on Physion with the default
For VQA, we directly copy the numbers from SlotFormer.
D.4 Our Implementation Details
VQ-VAE [72]. We use the same architecture for all datasets, which is adopted from LDM [73]. We use 3 encoder and decoder blocks, resulting in 4x down-sampling of the feature maps z compared to input images æ. We pre-train the VQ-VAE for 100 epochs on each dataset with a cosine learning rate schedule decaying from 1 x 10-3, and fix it during the object-centric model training.
SlotDiffusion. For the objeric model, we only replace the decoder with LDM [73] compared to Slot Attention on images and SAVi on videos. We use a modified ResNet18 encoder [46] to extract image features', except for real-world datasets PASCAL VOC and COCO, we use the DINO self- supervised pre-trained [7] encoder, where the ViT-S/8 variant is chosen following DINOSAUR [81]. On video datasets, we train on small clips of frames. For the LDM-based slot decoder, the training target is to predict the noise added to the features z produced by pre-trained VQ-VAE. Following prior works, the denoising network co (,,,t, attention and cross-attention layers in each block. We use the same noise schedule [/, and U-Net hyper-parameters as LDM [73]. The attention map from the last Slot Attention iteration is used the predicted segmentation. See Table 5 for detailed slot configurations and training settings.
f, S) is implemented as U-Net [74] with global self-
3 'Interestingly. we found that R encoder leads to training divergence on MOVi-Solid and MOVi-Tes datasets for both DiftDuiuusio and baselines. Therefore, we adopt the plain CNN encoder on these two datasets. Pre-trained weight obtained from the HuggingFace transformers library.
21

---

Table 6: Comparison of model complexity on the MOVi-D/E video datasets. We measure the training memory consumption, time per training step, and generation time of 100 vidcos at test stage. For training, report the default settings (batch size 32 of length-3 video clips, frame resolution 128 128) on NVIDIA A40 GPUs. For lesting, we report the inference time on NVIDIA T4 GPUs (each video contains 24 frames).
SAVi
STEVE
SlorDiffusion
Memory (GB)
32
51
24
Train
Time (s)
0.57
0.87
0.77
Test
Time (min)
0.7
226
7
Image
Per-Slot Masks
Image
Per-Slot Masks
Figure 13: Visualization of image segmentation results on CLEVRTex (left) and CelebA (right).
Slot-to-image decoding. Diffusion models are notoriously slow in doing generation due to the iterative denoising process, where the diffusion step 7 is often 1,000. Fortunately, researchers have developed several methods to accelerate this procedure. We employ the DPM-Solver [62] which reduces the sampling steps to 20. Therefore, our reconstruction speed is even faster than models with Transformer- decoders, as they need to forward their model number of tokens (typically 1,024) times for autoregressive token prediction, while we only need 20 times.
FlotFormer [100] Physion. We use its online official implementation". We first pre-train SlotDiffusion model on Physion, and use it to extract object slots from all videos and save them offline. Then, we train SlotFormer to predict slots future timesteps conditioned on slots from input timesteps. We follow the same model architecture and training settings as [100]. For video prediction, we use the preitrained L.DM-based slot decoder to decode the predicted slots back frames, so that we can evaluate the generation quality. For VQA, we train additional readout models over the predicted slots to answer questions. Please refer to [100] for more information.
E
Additional Experimental Results
E.1
Computational Requirements
We empirically show the speed and GPU memory requirement at training time, well the time required to reconstruct 100 vidcos at test time of SAVi, STEVE, and SlotDiffusion in Table Interestingly, model requires the least GPU memory during training despite achieving the best performance. This is because we run the diffusion process at latent space, whose spatial dimension is only 1/4 of the input resolution. In contrast, SAVi applies CNN to directly reconstruct images the original resolution, and STEVE uses an autoregressive Transformer to predict long sequence of 1,024 tokens. Both of these designs consume large GPU memory. In terms of training and generation speed, SlotDiffusion ranks second. SAVi runs extremely fast since it decodes images in sho-shot, while STEVE and model both need do iterative sampling. Thanks the efficient DPM-Solver sampler, we only require 20 times forward pass, while STEVE requires 1,024 times.
Total training time. SAVi is the fastest to train which takes 30 hours single NVIDIA A40 GPU, STEVE and SlotDiffusion both need 20 hours to train the VAE image tokenizer. STEVE's second stage training requires 50 hours 2 GPUs, while SlotDiffusion requires 40 hours GPU.
E.2 Object Segmentation
Image. We visualize our image segmentation results Figure 13. SlotDiffusion is able to discover objects semantically meaningful and consistent regions from images in an unsupervised way. For
22

---

7
10
13
t=1
4
7
10
13
GT
SAVi
STEVE
Ours
Figure 14: Visualization of video segmentation results on MOVi-D (left) and MOVi-E (right).
example, on CelebA, we decompose images into four parts: clothes, hairs, faces, and backgrounds. As shown in the section of compositional generation, we can generate novel samples by randomly composing these basic visual concepts. Notably, the object masks produced on CLEVRTex usually do not match the exact boundary of objects. This is because the segmentation masks are predicted in low resolution (4x nown-sampling in case) and up- up-sampled to the original resolution. How to refine the object masks from a coarse initialization is an interesting future direction.
Video. We show more video segmentation results in Figure 14. SAVi typically generates blob-shape masks that do not follow the exact boundary of objects due to the low modeling capacity of the mixture-basee decoder. Compared to STEVE, SlotDiffusion usually has more consistent tracking results and fewer object-sharing issues, especially on large objects. We also note that the shown examples of SlotDiffusion on MOVi-D have FG-ARI scores higher than 80%, although their visual quality is far from perfect. This indicates that the FG-ARI metric has been saturated, and we should focus more on better metrics such as mloU [43].
E.3 Compositional Generation
Reconstruction. As the main paper, we first show more reconstruction results Figure 15. Com- pared to baselines that produce blurry frames, SlotDiffusion is able to preserve the local appearances such as textures on the objects and backgrounds. However, there is still a large room for improvement in finer details. For example, in the top example, we cannot reconstruct the texts the object surfaces. Also, in the bottom example, we fail to retain the smooth surface of the brown shoes.
Generation. In Figure 16, we compare original dataset samples to novel results produced by baselines and SlotDiffusion. Due to the limited capacity, models with the mixture-basee decoder generate blurry images. Although the Transformer- maasedd decoder helps SLATE and STEVE decompose complex scenes, they typically produce objects with distorted attributes, such as deformed object shapes and unnatural facial features. Thanks to the iterative refinement process of DMs, SlotDiffusion able to generate high-fidelity images and videos with fine details, coherent with the real samples.
the generation diversity of SlotDiffusion's 's LDM decoder. One good property of DMs is that they can generate diverse results from single conditional input [11, 66, 73]. This is because the conditional inputs do not fully describe the output's 's content. For example, in text-guided generation, the input texts usually only describe the global style and layout of the images, while LDMs have the freedom to generate diverse local details. In contrast, SlotDiffusion conditions the LDM on slots, which are designed to capture all information (position, shape, texture, etc.) about objects. Therefore, we expect our LDM to faithfully reconstruct the original objects, instead of generating diverse results, which is also desired property our downstream applications. For example, when performing face replacement (Figure 8 (b) in the main paper), we want to maintain the facial attributes of the source face. Nevertheless, we note that SlotDiffusion is still able to generate novel content when necessary. For example, when removing object from the scene in image editing (Figure 8 (a) in the main paper), it inpaints the background with plausible textures.
Remark:
To study how much randomness our LDM decoder has, we run it to decode the same set of slots given six different random seeds. The results are shown in Figure 17 of the submitted PDF file, where our model generates images with minor differences (mainly local textures and surface reflections), preserving the identity of objects.
23

---

t=
8
12
16
20
GT
MSE
LPIPS
SAVI
478.2
0.566
STEVE
793.4
0.547
Ours
594.6
0.407
GT
MSEI
LPIPS
SAVi
335.3
0.534
STEVE
619.1
0.493
Ours
542.0
0.355
Figure 15: Qualitative results of slot-to-video reconstructions on MOVi-D (top) and MOVi-E (bottom).
Dataset
SA
SLATE
Ours
Dataset
SA
SLATE
Ours
Dataset
SAVi
STEVE
(a) CLEVRTex
(b) CelebA
(c) MOVi-E
Figure 16: Compositional generation results CLEVRTex (a), CelebA (b), and MOVi-E (e). Compared baselines, SlotDiffusion produces results with higher fidelity and more details, such as textures on objects and backgrounds, and hairs and facial features of humans, which are visually closer to the dataset samples.
E.4 Video Prediction and VQA
We show more qualitative results of Physion video prediction in Figure 18. PredRNN produces blurry frames with objects disappearing while still achieves low MSE. proving that MSE a poor metric in generation tasks. Although VQFormer generates sharp images, the generated frames are almost static, lacking scene dynamics. STEVE with SlotFormer able to predict the rough motion of objects, yet the results are of low quality. In contrast, SlotDiffusion learns highly informative slots and can decode high-fidelity images. Therefore, our method predicts videos with the correct object motion and consistent scenes, even for the challenging dynamics of deformable cloth.

---

Figure 17: Images sampled from SlotDiffusion's LDM decoder with different random seeds.
t 15
20
25
30
35
40
45
50
GT
PredRNN
VQFormer
STEVE SF
Ours SF
= 15
20
25
30
40
50
GT
PredRNN
VQFormer
STEVE SF
Ours + SF
Figure 18: Qualitative results of vidco prediction Physion. stands for SlotFormer.
25

---

E,5
Failed Attempts
Here, we record some failed model variants we tried.
Image-space difflusion model. At the early stage of this work, we adopt an image-space DM as the slot decoder without the VQ-VAE tokenizer. This works well with low resolution inputs (64x 64). However, when we increase the resolution to 128 x xx1x22 128, we observe color shifting artifacts which is common issue among high resolution DMs [76, 78]. Switching an LDM-based decoder solves this problem, as the VQ- (AE decoder always maps latent codes to images with natural color statistics. In addition, since LDM consumes less memory, we can use a larger U-Net as the denoiser, which further improves the performance.
Prediction target of the diffusion model. By default, our DM adopts the traditional noise prediction (e) target. Recent works in DM propose two other formulations, namely, data prediction (xo) [71] and velocity prediction (v) [35], and claim to have better generation quality. We tried both targets in SlotDiffusion, but both model variants degenerate to the stripe pattern solution (similar to the SAVi
result on MOVi-D in Figure 14) under multiple random seeds.
Classifier-free guidance [33]. We also tried classifier-fret guidance (CFG for short) to improve the generation quality of SlotDiffusion. In order apply CFG during inference, we need to randomly drop the conditional input (i.e., slots) during model training. However, with a high dropping rate, we observe severe training instability, where the model degenerates to the stripe pattern solution. When trained with low dropping rate, we do not observe clear improvement in the generation quality,
F Limitations and Future Works
Limitations. The ultimate goal of unsupervised object-centric models is to perform image-to- slot decomposition and slot-to-image generation real-world data without supervision. With the help pre-trained encoders such as DINO VIT [7], SlotDiffusion is able to segment objects from naturalistic images [21, 54J. However, the images reconstructed from slots are still of low quality, preventing unsupervised object-centric generative models from scaling to these data. In addition, objects in real-world data are not well-defined. For example, the part- part-whole hierarchy causes ambiguity in the segmentation of articulated objects, and the best way of decomposition depends the downstream task. Finally, our learned slot representations still entangled, while many tasks require disentanglement between object positions and semantics.
Future Works. From the view of generative modeling, we can explore better DM designs and techniques such as classifier-free guidance to improve the modeling capacity of fine local details. Moreover, our LDM decoder currently generates video frames individually without considering the temporal interactions. We can try the temporal attention layer in video DMs [27, 36] to improve the temporal consistency. To generate high-quality natural images, one solution is to incorporate pre-trained generative models such as the text-guided Stable Diffusion [73]. As discussed in the main paper, object slots in image similar to phrases in a sentence. We may try to map the slots to the text embedding space or via lightweight adapters [107].
From the view of object-centric learning, we want to apply SlotDiffusion to more downstream tasks such as reinforcement learning [95, 105] and 3D modeling [86, 104]. It is interesting see task-specific supervision signals (e.g., rewards in RL) can lead to better scene decomposition. To address the part-whole ambiguity objects, Segment Anything [47] proposes to output three levels of masks per pixel, which cover the whole, part, and subpart hierarchy. We can apply such design to slot-based models to support richer scene decomposition. Finally, there can new protocols for evaluating these object-centric models. For example, MAE [29] leverages supervised fine-tuning to verify the effectiveness of their learned representations. We can also consider SlotDiffusion with labels such as class labels or object bounding boxes [46].
26

---

G Full Experimental Results
To facilitate future research, we report the numeric results of all bar charts used in the main paper in Table 7, Table 8, and Table 9.
Table 7:
Full results CLEVRTex and CelebA datasets, FG-ARL, mloU, and mBO are given in %. we use the masks provided by the CelebA-Mas Mask dataset [51] to evaluate the segmentation results on CelebA. We select four classes the provided labels, original "hair", "cloth", and "background" classes, and new "face" class by merging masks all other facial attributes such as eye, nose, mouth. data split CelebA AMMs slightly different from the original CelebA dataset, so we do not report this result in the main paper.
CLEVVTex
CelehA
Method
FG-ARI
mloU
mB0
MSE L
LPIPS
FID
PG-ARI
mloU
mB0 1
MSB 4
LPIPS +
FID
SA
28.79
SLATE
67.50
56.82
59.35
313.3
0.39
88.73
40.23
35.72
37.44
744.8
0.32
78.95
Ours
68.39
57.56
61.94
237.5
0.13
32.07
44.01
38.46
41.35
439.0
0.21
27.72
Table 8: Full results MOVi-D and MOVi-E datasets, FG-ARI, mloU, and mBO given in %.
MOVi-D
MOVIE
Method
FG.ARI
mloU?
mBO
MSE.
I.PPS. 4
FVD
FG-ART
mloll
mBO
MSE
I.IPS 4.
FVD. 4
SAVi STEVE
39.63
17.76
18.49
237.2
0.53
1123.8
46.70
23.49
22.23
253.2
0.55
11102
49.43
29.51
29.16
673.2
0.51
857.0
56.04
27.86
27.97
Ours
52.09
31.65
31.24
632.7
0.40
493.2
59.99
30.16
30.22
586.5
0.39
353.3
Table Full results MOVi-Solid and MOVi-Tex datasets, FG-ARI. mloU, and mBO are given
MOVi-Solid mBO
MOVi-Tex
Method
FG-ARI
mioU*
MSB
LPIPS 4 FVD
FG-ARI f mloU
mB0 1
MSE 4
LPIPS
FVD 4
SAVI STEVE
44,95
30.74
29.88
158.1
0.44
1752.1
26.40
7.60
7.01
193.1
0.40
1139.2
60.16
40.72
42.67
525.7
0.45
1245.2
39.02
34.27
34,89
476.6
0.32
704.6
46.55
38.33
39.03
27


Reviews:
[{"review_id": "AaCq1u38nE", "reviewer": "Reviewer_QakT", "summary": "The paper proposes a diffusion-based decoder for slot-centric models. Previously slot-centric models are trained with spatial broadcast or transformer autoregressive decoders, however these decoders result in a comparitively poor image generation or segmentation. The paper shows that using diffusion based decoders can result in better generation, segmentation and video prediction. ", "questions": "i) In Table 1, how does the model compare against spatial broadcast decoder, instead of steve that uses transformer decoder?\n\nii) On real world benchmarks how would this method compare against CutLER (https://arxiv.org/abs/2301.11320), a method that doesn't use any slot-based training but does normalized cuts on top of Dino features and calculates segmentation from there.\n\niii) How does the method compare on original CLEVR, as i find all the fg-ari scores in Figure 2 to be below 0.7.\n\niv) What about classifier-free guidance? have the authors tried that with slots?\n\nv) I would be interested in seeing (maybe in future) how does this method do when coupled with supervised labels, like evaluations shown in classical SSL methods such as MAE (https://arxiv.org/abs/2111.06377) or recent Slot-based methods such as Slot-TTA (https://arxiv.org/abs/2203.11194)\n\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "i) The paper does a dense evaluation of it's method \nii) The paper gets better peformance on almost all of the settings\niii) The paper is well written and is easy to understand.\n ", "weaknesses": "I couldn't find any major weakness in the paper. \nAlthough I have a few questions that i have mentioned below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback and encouraging comments.\n\n\n**1. Performance of spatial broadcast decoder based models in Table 1 (video prediction on Physion).**\n\nA: We train SlotFormer over slots extracted by SAVi, and report its video prediction results below. SlotDiffusion + SlotFormer outperforms SAVi + SlotFormer by a sizable margin in all metrics.\n\n|Physion Video Prediction|MSE &darr;|LPIPS &darr;|FVD &darr;|\n|-|-|-|-|\n|SAVi + SlotFormer|746.3|0.62|1349.4|\n|STEVE + SlotFormer|832.0|0.43|930.6|\n|**SlotDiffusion** + SlotFormer|**477.7**|**0.26**|**582.2**|\n\n\n**2. Comparison with CutLER [1] on unsupervised object segmentation.**\n\nA: We ran additional experiments to compare with CutLER on *COCO val2017* and summarize the results below. *Prev. SOTA* is the previous state-of-the-art results reported in the CutLER paper. SlotDiffusion outperforms *Prev. SOTA*, but is worse than CutLER. This is because CutLER involves multi-round self-training of complicated object segmentation models (Cascade Mask R-CNN [2]), while our method leverages simple Slot Attention for object discovery. An interesting future direction is to study how to combine slot-based models with well-studied object segmentation approaches to further boost performance.\n\n|COCO val2017|AP$_{50}^{\\text{mask}}$ &uarr;|AP$_{75}^{\\text{mask}}$ &uarr;|AP$^{\\text{mask}}$ &uarr;|\n|-|-|-|-|\n|Prev. SOTA|9.4|3.3|4.3|\n|CutLER|**18.9**|**9.7**|**9.2**|\n|**SlotDiffusion** + DINO ViT|13.2|6.0|6.2|\n\n\n[1] Wang, Xudong, et al. \"Cut and learn for unsupervised object detection and instance segmentation.\" CVPR. 2023.\n\n[2] Cai, Zhaowei, and Nuno Vasconcelos. \"Cascade r-cnn: Delving into high quality object detection.\" CVPR. 2018.\n\n\n**3. Model performance on the original CLEVR dataset.**\n\nA: We did not experiment on CLEVR in our initial submission since its images are visually very simple. We trained SlotDiffusion on the CLEVR (with masks) dataset, and report the results below. SlotDiffusion outperforms SLATE, but is worse than Slot Attention. This is because Slot Attention decodes object masks with the same resolution as input images, while we use the cross-attention maps from the final slot attention iteration as the segmentation masks. Its 4x downsampled resolution limits the performance of our method. This result is also consistent with concurrent work [3] (see their Appendix B). Nevertheless, we achieve an FG-ARI of over 90%, which is still reasonably high.\n\nWe note that despite a better performance on simple datasets, directly decoding masks at the original resolution prevents Slot Attention and SAVi from scaling to complicated datasets, as shown in our experiments. This is because using raw pixels as reconstruction targets biases the model towards low-level appearance features such as color statistics, which works well on CLEVR with uniformly colored objects and plain gray backgrounds. However, to decompose scenes with textured objects and backgrounds, the model needs to learn semantic features, which requires high-level reconstruction targets such as image features tokenized by pre-trained VAEs.\n\n|CLEVR|FG-ARI &uarr;|mIoU &uarr;|mBO &uarr;|\n|-|-|-|-|\n|Slot Attention|**95.67**|**76.23**|**78.67**|\n|SLATE|88.56|66.62|67.42|\n|**SlotDiffusion**|90.32|71.63|72.05|\n\n[3] Jiang, Jindong, et al. \"Object-centric slot diffusion.\" arXiv preprint arXiv:2303.10834 (2023).\n\n\n**4. Classifier-free guidance (CFG for short) with SlotDiffusion.**\n\nA: Thanks for this great suggestion. We implemented SlotDiffusion with CFG in preparation of this rebuttal. In order to apply CFG at inference time, it is necessary to randomly drop the conditional input (i.e. slots) during training. We observed severe training instability when using a high dropping rate, where the segmentation degenerated to stripe patterns (as in the SAVi results in Figure 3 of the main paper). When training with a low dropping rate, we did not observe clear improvements in the testing generation results.\n\nWe will add CFG to Appendix E.5 as one of our failed attempts and Appendix F as an interesting future direction.\n\n\n**5. SlotDiffusion coupled with supervised labels or recent slot-based methods such as Slot-TTA [4].**\n\nA: We agree that combining SlotDiffusion with supervised labels is an interesting future work, and will add it to Appendix F.\n\nFor Slot-TTA, we conducted preliminary experiments by optimizing the denoising loss of SlotDiffusion on input images at test time. We use the default hyper-parameters listed in the Slot-TTA paper. Below we show the segmentation results on CLEVRTex. Slot-TTA consistently improves the performance of SlotDiffusion across all metrics, demonstrating the ability of our method to benefit from recent advances in slot-based models.\n\n|CLEVRTex|FG-ARI &uarr;|mIoU &uarr;|mBO &uarr;|\n|-|-|-|-|\n|SlotDiffusion|68.39|57.56|61.94|\n|SlotDiffusion + Slot-TTA|**69.66**|**58.67**|**62.82**|\n\n[4] Prabhudesai, Mihir, et al. \"Test-time adaptation with slot-centric models.\" ICML. 2023.\n\n\n\n"}, {"review_id": "VQAjEWnlGU", "reviewer": "Reviewer_GFM9", "summary": "A latent diffusion-based slot decoder is introduced to improve the segmentation and reconstruction quality for slot-based autoencoders. This decoder addresses issues with previous advanced Transformer-based decoders such as SAVi and STEVE which i) do not preserve the spatial dimension of images and ii) lack an iterative refinement component for capturing high-frequency image details. SlotDiffusion combines a pretrained VQ-VAE with a cross-attention denoising diffusion decoder. The key idea is to treat slots like text embeddings and therefore to use cross-attention in the decoder following text-guided latent diffusion models. Rigorous qualitative and quantitative results confirm the improvements in segmentation and reconstruction quality.", "questions": "- Table 2: What is the improvement with respect to? What is the Obs. % and Dyn. % for random guessing? \n\n---\n\nUpdate after rebuttal: I am maintaining my initially positive outlook on this paper.", "limitations": "Limitations and Broader Impacts are sufficiently discussed in Appendix F and G. I would encourage the authors to include this in the main text, space permitting.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "#### Originality\n\n- This work fills a gap in the literature by exploring how to combine ideas from latent diffusion modeling with slot-based object-centric learning.\n- The insight that slots can be used like text-embeddings in the cross-attention module of a denoising diffusion decoder is novel and potentially quite powerful.\n\n#### Quality\n\n- The technical description of the model is sound and the experiments are rigorous (2 complex image datasets, four video datasets, 1 video prediction and reasoning benchmark, and 2 real-world image datasets). \n\n#### Clarity\n\n- The paper is well-written and easy to follow.\n\n#### Significance\n\n- I believe this work is a valuable contribution to the object-centric learning community.\n- In particular, the breadth of experiments in this paper help demonstrate the value of this research direction and the generality of the models being explored.", "weaknesses": "I have very few concerns with this work.\n\n- While I applaud the authors for evaluating SlotDiffusion on video prediction, VQA, and real-world image datasets, the results on these benchmarks are not as impressive as on the synthetic datasets. \n- Moreover, scaling these models to real-world images appears to require a complex, multi-stage modeling effort with various important hyperparameters to be tuned: 1) pretraining the DINO ViT model, then 2) pretraining the VQ-VAE model, then 3) stitching all these together and training the denoising diffusion slot decoder.\n- These are not big issues, but it suggests that perhaps something fundamental is being missed by this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and valuable comments.\n\n\n**1. Scaling to real-world data requires complex efforts and designs.**\n\nA: We would like to note that the weights of DINO ViT and the architecture of VQ-VAE are directly adopted from prior works with minimal modifications. Also, SlotDiffusion turns out to be robust to hyper-parameter choices in these building blocks.\n\nThat being said, we completely agree with the reviewer that it is still challenging to scale up current unsupervised slot-based methods to real-world data, due to the variations of objects and backgrounds in real-world images. We have discussed the limitations of this work as well as recent object-centric models in Appendix F, which indeed poses many future directions to explore.\n\n\n**2. Improvement with respect to random guessing in Table 2 (VQA task on Physion).**\n\nA: We notice that there seems to be an error in the question “*What is the improvement with respect to?*”, and we are not entirely sure which baseline the reviewer is referring to. Based on the question that follows, we assume the reviewer is asking about the \"random guessing baseline”.\n\nThe questions in the Physion dataset are binary classification questions (whether the two objects contact or not). Therefore, the accuracy for random guessing is 50%. As a comparison, the Obs. and Dyn. accuracy of our method is 67.5% and 69.7%, respectively.\n\nPlease let us know if this is the correct interpretation or if there is any specific baseline you would like us to compare with.\n\n\n**3. Moving the Limitations and Broader Impacts sections to the main text.**\n\nA: Thanks for this great suggestion. We will include them in the main text in the camera-ready version of the paper, as we will be allowed one additional content page then.\n\n"}, {"review_id": "jS5Eewp5tr", "reviewer": "Reviewer_jjQP", "summary": "The paper presents, SlotDiffusion, an object-centric model that incorporates a latent diffusion model as an image decoder. Extracted slots are used as a conditioning signal for the cross-attention layers of the diffusion UNet. Other than that, the model follows the formulation of SA/STEVE/SAVi models. SlotDiffusion is tested on segmentation, reconstruction and compositional generation tasks on various simulated and real datasets, comparing favourably against Slot Attention, SLATE, STEVE, SAVi and DINOSAUR models in their respective settings. The paper additionally considers the tasks of video prediction and VQA by augmenting the model with SlotFormer, where similarly strong results are observed. ", "questions": "For evaluating the segmentation, what masks are used? Is this the attention of the final slot attention iteration? It would be helpful to clarify how the masks are predicted for evaluation.\n\nHow much do samples from the decoder vary? In other words, can the LDM model maintain the ability to generate diverse samples? Or does initial noise input have only limited to no effect? \n\nIn the checklist, it is mentioned that results are reported as an average of 3 seeds and the std. dev. are small. Would it be possible to report them, at least for SlotDiffusion models? It is particularly interesting given the added stochasticity of the LDM.\n\nIn Figure 2, are the 3 models with different decoders specific models, eg SA, SLATE and SlotDiffusion?\n", "limitations": "\nThe limitations seem to be largely addressed. Perhaps it would be worth mentioning that it appears that the learned slots encode the position of the objects they represent. This means that moving an object in a scene would require having a slot for the same object in a new position. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposal incorporates a latent diffusion model as a decoder, which is novel in the context of object-centric learning (up to concurrent work).\n\nThe model shows strong results on a variety of benchmarks.\n\nThe writing is clear and easy to follow, and the presentation is good with only a few missing details (see weaknesses/questions).", "weaknesses": "\nWhile the method shows improved reconstruction quality and segmentation results over methods that do not use LDM as a decoder (and are otherwise, similar or identical), it is not clear if the LDM Unet model is comparable to a small spatial broadcast decoder (as in SA) or transformer decoder of SLATE/STEVE. Are these model components comparable in terms of e.g. the number of parameters? As stated in Appendix D.3 (L180) \"We found that using a _larger decoder_ and training longer leads to better performance\". Could this be the reason for the improved results?\n\nAdditionally, there are some more recent works [Jie et al., Singh et al.] omitted, which might serve as better points of comparison than SA/SAVi/SLATE/STEVE.\n\nIt is not entirely clear what procedure is adapted for video prediction and VQA tasks, where the SlotDiffusion is augmented with SlotFormer. Would this not remove the LDM entirely from the model?\n\n\nSingh et al. \"Neural Systematic Binder\"\n\nJie et al. \"Improving Object-centric Learning with Query Optimization\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We are glad to see the positive assessment of our paper and appreciate the detailed feedback.\n\n**1. Parameter & size comparison of SlotDiffusion and baselines.**\n\nA: We list the parameter size of SlotDiffusion and baselines on the MOVi-E dataset below. Our model is indeed larger than SAVi and STEVE. We tried scaling up these baselines by adding more layers and channels, but observed severe training instability beyond a certain point. Current sizes yielded optimal results. Instead, SlotDiffusion shows great scalability with model sizes. Its performance grows consistently with a larger LDM U-Net. Such scalability is a good property observed in recent diffusion model works, e.g. Stable Diffusion.\n\nWe also note that despite a larger model size, the memory requirement of SlotDiffusion during training is lower than both baselines, and the training and inference speed of SlotDiffusion is faster than STEVE, as reported in Appendix E.1.\n\n|Model|SAVi|STEVE|SlotDiffusion|\n|-|-|-|-|\n|Parameter Size (MB)|28|63|110|\n\n**2. Comparison with more recent baselines.**\n\nA: Thanks for bringing up these related works. Below, we compare them to SlotDiffusion on CLEVRTex.\n\n**SysBinder** [1] focuses on disentangling visual concepts from images. Its segmentation and generation performance is similar to SLATE as shown in their paper. We trained SysBinder and report its results below. SysBinder achieves slightly better segmentation results than SLATE, while its generation quality is worse. In contrast, SlotDiffusion consistently outperforms SysBinder across all metrics.\n\n|CLEVRTex|FG-ARI &uarr;|mIoU &uarr;|mBO &uarr;|MSE &darr;|LPIPS &darr;|FID &darr;|\n|-|-|-|-|-|-|-|\n|SLATE|67.50|56.82|59.35|313.3|0.39|88.73|\n|SysBinder|67.80|57.01|60.55|322.3|0.39|90.73|\n|**SlotDiffusion**|**68.39**|**57.56**|**61.94**|**237.5**|**0.13**|**32.07**|\n\n**BO-QSA** [2] applies bi-level optimization in the iterative slot update process. This idea is orthogonal to our modifications to the slot decoder, and thus can also be used in SlotDiffusion (dubbed *BO-SlotDiffusion*). The comparison between BO-QSA and our methods is shown below. BO-SlotDiffusion achieves comparable segmentation results with BO-QSA. In addition, both SlotDiffusion and BO-SlotDiffusion outperform BO-QSA by sizable margins in generation quality. This proves that SlotDiffusion can benefit from recent advances in slot-based methods.\n\n|CLEVRTex|FG-ARI &uarr;|mIoU &uarr;|mBO &uarr;|MSE &darr;|LPIPS &darr;|FID &darr;|\n|-|-|-|-|-|-|-|\n|BO-QSA|**80.47**|64.26|66.27|268.2|0.38|98.93|\n|SlotDiffusion|68.39|57.56|61.94|237.5|0.13|32.07|\n|BO-SlotDiffusion|78.50|64.35|**68.68**|240.3|0.13|**30.06**|\n\n[1] Singh, Gautam, et al. \"Neural systematic binder.\" ICLR. 2023.\n\n[2] Jia, Baoxiong, et al. \"Improving object-centric learning with query optimization.\" ICLR. 2023.\n\n**3. Clarity on SlotDiffusion with SlotFormer for video prediction and VQA on Physion.**\n\nA: We will improve clarity by incorporating the following information: \"*We first pre-train SlotDiffusion on Physion videos, and use it to extract slots from video frames and save them offline. Then, we train SlotFormer by taking in slots from previous timesteps, and predicting slots at future timesteps.*\"\n\nIn VQA, we train additional readout models over the predicted future slots to answer questions. As you correctly assume, the LDM decoder is not involved here. We mainly want to test whether SlotDiffusion can learn informative slots that are useful for downstream tasks.\n\nIn video prediction, we leverage LDM to decode the predicted future slots back to video frames, so that we can evaluate the generation quality. The better video prediction results prove the effectiveness of our LDM-based slot decoder.\n\n**4. Segmentation mask output of SlotDiffusion.**\n\nA: Yes, the attention map of the final slot attention iteration is used as the predicted segmentation mask. We will clarify this in the revised manuscript.\n\n**5. The generation diversity of SlotDiffusion’s LDM decoder.**\n\nA: This is a good question. Previous LDMs show diversity in conditional generation because the conditional inputs underspecify the output’s content. For example, in text-guided generation, the input text usually only describes the global style and layout of the images. This leaves LDMs with the freedom to generate diverse local details.\n\nIn contrast, SlotDiffusion conditions the LDM on slots, which are designed to capture all information about objects (position, shape, texture, etc.). Therefore, we expect our LDM to faithfully reconstruct the original objects, instead of generating diverse results. This is also a desired property in our applications. For example, when performing face replacement (Figure 8 (b) of the main paper), we want to maintain the facial attributes of the source face.\n\nTo study how much the initial noise map affects the generation results of our LDM decoder, we run it to decode the same set of slots extracted from CLEVRTex images given different random seeds. The results are shown in **Figure 1 of the submitted PDF file**. Our LDM generates images with minor differences, preserving the identity of objects.\n\nNevertheless, we note that SlotDiffusion is still able to generate novel content when necessary. For example, when removing an object from the scene in image editing (Figure 8 (a)), it inpaints the background with plausible textures.\n\n**6. Std. dev. of SlotDiffusion results.**\n\nA: We have reported the experimental results of SlotDiffusion with mean and std. dev. in **Table 1 of the submitted PDF file**. Overall, the std. dev. is relatively small, usually less than 2\\% of the mean.\n\n**7. Specific models in Figure 2.**\n\nA: Yes. Mixture, Transformer, and Diffusion decoders stand for SA/SAVi, SLATE/STEVE, and SlotDiffusion models, respectively. We will clarify this in the caption.\n\n**8. Positional bias of the learned object slots.**\n\nA: Thanks. We will add this to the Limitation Section.\n"}, {"review_id": "gOUH8g7Nj5", "reviewer": "Reviewer_JU9y", "summary": "This paper presents SlotDiffusion, an object-centric Latent Diffusion Model (LDM) designed for image and video data. The authors focus on improving slot-to-image decoding, a critical aspect of visual generation. SlotDiffusion outperforms previous slot models in unsupervised object segmentation and visual generation across multiple datasets. The model's learned object features also enhance video prediction quality and temporal reasoning tasks. The scalability of SlotDiffusion is demonstrated on real-world datasets like PASCAL VOC and COCO, showcasing its potential when combined with self-supervised pre-trained image encoders.", "questions": "As above.\n\nI am inclined to raise my score if my concerns are fully resolved.", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This direction of slot-based generation is promising.\n- The experimental section provides fruitful results that effectively validate the proposed method from various perspectives.", "weaknesses": "- In the experimental section, the authors primarily compare their method with mixture and transformer-based baselines. As a result, the main contribution of the observed gains may primarily stem from the diffusion models themselves, potentially undermining the novelty of the proposed methods.\n- It would be beneficial to evaluate the proposed method on the CelebaMask dataset, as mentioned in Line 235.\n- In Line 96, the authors mention the differences between their work and Jiang et al. It would be useful if they also included a basic comparison in the experimental results.\n- Regarding Figure 11, it would be helpful to understand the performance gap between your method and the baseline at different diffusion steps.\n- The paper lacks additional discussion. Since slot-attention is a paradigm for identifying meaningful visual concepts, there are other self-supervised approaches, such as [1], [2], and [3], that achieve similar goals. Including a discussion relating to these approaches would be appreciated. It seems that [1] follows a similar process of obtaining semantic meaning in a post-hoc manner (e.g., sampling, observation, assigning semantic meaning).\n\n[1]. Self-Guided Diffusion Models\n[2].Why Are Conditional Generative Models Better Than Unconditional Ones?\n[3].Visual Chain-of-Thought Diffusion Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and insightful comments.\n\n\n**1. Are performance gains mainly due to the use of diffusion models?**\n\nA: Yes, the performance gains indeed mainly come from the use of an LDM-based slot decoder, as diffusion models have both strong modeling capacity and generation quality. We actually view the introduction of diffusion models to object-centric learning as one of our major contributions driving the performance gains. It is true that LDMs and Slot Attention were known individually. But the fact that incorporating LDMs with slot conditioning can improve both unsupervised segmentation and visual generation was neither known nor obvious before our work. We believe that the breadth of our experiments and the strong results justify this design choice. We would also like to note that both reviewer GFM9 and reviewer jjQP recognize our diffusion-based slot decoder as novel and sound, which is a contribution to object-centric learning.\n\nBesides the general idea, we also made several technical contributions:\n- We study how to condition the LDM on slots to guide the denoising process. Drawing inspiration from text-guided generation, we propose to fuse U-Net feature maps and slots via cross-attention. As shown in our experiments, this conditioning mechanism facilitates both object segmentation and visual generation;\n- SlotDiffusion is also a novel conditional DM trained in a fully unsupervised manner (see extended discussions in the response to `5.`). Without labels or pre-trained feature extractors (except on VOC and COCO), our method learns useful object-centric features from scratch. Thanks to the slot-based representation, we can perform image editing and compositional generation in a flexible and interpretable way.\n\n\n**2. Evaluation on the CelebA-Mask dataset [1].**\n\nA: Thanks for bringing up this dataset. We select four classes from the provided labels, the original “*hair*”, “*cloth*”, and “*background*” classes, and a new “*face*” class by merging masks of all other facial attributes such as eye, nose, mouth. The segmentation results of SlotDiffusion and baselines are shown below. Our method consistently outperforms baselines in all metrics.\n\n|CelebA-Mask|FG-ARI &uarr;|mIoU &uarr;|mBO &uarr;|\n|-|-|-|-|\n|Slot Attention|32.13|26.26|28.79|\n|SLATE|40.23|35.72|37.44|\n|**SlotDiffusion**|**44.01**|**38.46**|**41.35**|\n\n[1] Lee, Cheng-Han, et al. \"Maskgan: Towards diverse and interactive facial image manipulation.\" CVPR. 2020.\n\n\n**3. Basic comparison with LSD [2].**\n\nA: Thanks for this great suggestion. Since LSD has not released their code, we can only compare with it on CLEVRTex and MOVi-E where we both conduct experiments. Also, for fair comparison, we adopt the same pre-trained VAE as LSD does. LSD treats MOVi-E as an image dataset and trains the model on sub-sampled video frames. We follow this protocol and train a SlotDiffusion model on the same video frames.\n\nThe results are summarized in **Table 2 of the submitted PDF file**. On CLEVRTex, we achieve better FG-ARI, and comparable results in mIoU, mBO, and FID. On MOVi-E, we outperform LSD in both segmentation and generation metrics. However, since LSD’s code is not yet publicly available, there could be minor differences in the implementation details (parameter size, data pre-processing, etc.). Thus, our numbers can only serve as a preliminary comparison.\n\n[2] Jiang, Jindong, et al. \"Object-centric slot diffusion.\" arXiv preprint arXiv:2303.10834 (2023).\n\n\n**4. Performance gap between SlotDiffusion and baselines at different diffusion steps $T$.**\n\nA: We plot the best baseline results on each dataset as dashed lines in **Figure 2 of the submitted PDF file** to compare with our method using different $T$. We do not plot MSE as it favors blurry results which does not align with human perception. SlotDiffusion outperforms baselines in LPIPS, FG-ARI, and mIoU across different $T$, demonstrating robustness against this hyper-parameter.\n\n\n**5. Additional discussions about related works.**\n\nA: We thank the reviewer for pointing us to these papers. They are absolutely relevant as they also learn conditional diffusion models on datasets without any labels. We will include them in the Related Work section in the camera-ready version of the paper. Here is our extended discussions relating to these works:\n\n> **Conditional diffusion model without labels**. Several works have explored learning conditional DMs in an unsupervised manner. Hu et al. [3] and Bao et al. [4] utilize self-supervised feature extractors to obtain image features, and then perform clustering to create pseudo labels for training a conditional DM. Harvey et al. [5] show that conditional DMs usually outperform unconditional DMs in terms of FID. Inspired by this, they leverage CLIP embeddings [6] of unlabeled images to train conditional DMs without manual annotations. Similarly, SlotDiffusion also learns a conditional DM without labels. Instead of relying on pre-trained feature extractors, we build upon the unsupervised Slot Attention framework, and can learn both object-centric representations and conditional DMs from scratch.\n\n[3] Hu, V. T., et al. \"Self-Guided Diffusion Models.\" CVPR. 2023.\n\n[4] Bao, Fan, et al. \"Why Are Conditional Generative Models Better Than Unconditional Ones?.\" arXiv preprint arXiv:2212.00362 (2022).\n\n[5] Harvey, William, and Frank Wood. \"Visual Chain-of-Thought Diffusion Models.\" arXiv preprint arXiv:2303.16187 (2023).\n\n[6] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" ICML. 2021.\n"}]

Author rebuttal:
We would like to thank the reviewers for their helpful feedback and insightful comments.

We are glad that the reviewers find our paper “*well written*” (QakT), “*easy to follow*” (GFM9), and “*clear*” (jjQP), our diffusion-based slot decoder as “*sound*” (GFM9) and “*novel*” (jjQP). Also, our experiments are considered “*dense*” (QakT), “*rigorous*” (GFM9), and “*provides fruitful results that effectively validate the proposed method from various perspectives*” (JU9y).

From the view of object-centric learning, reviewer GFM9 remarks that our work “*fills a gap in the literature*” and “*is a valuable contribution*” to the community, and highlights our insight of treating slots as text features as “*novel and potentially quite powerful*”. Reviewer jjQP finds SlotDiffusion “*comparing favourably against*” other slot-based models “*in their respective settings*”. Finally, reviewer JU9y agrees that “*This direction of slot-based generation is promising*”.

We further thank the reviewers for their constructive feedback. We have uploaded a PDF file which includes figures and large tables to address the reviewers’ feedback. **Overall, we want to highlight two new experiments we conducted:**

**1. Combining SlotDiffusion with recent improvements in slot-based methods.** (suggested by reviewer QakT and reviewer jjQP)

We combine SlotDiffusion with two recent improvements on slot-based models, Slot-TTA [1] and bi-level optimization (dubbed BO-SlotDiffusion) [2], and report the results on CLEVRTex in the tables below. 

|CLEVRTex|FG-ARI &uarr;|mIoU &uarr;|mBO &uarr;|
|-|-|-|-|
|SlotDiffusion|68.39|57.56|61.94|
|SlotDiffusion + Slot-TTA|**69.66**|**58.67**|**62.82**|

|CLEVRTex|FG-ARI &uarr;|mIoU &uarr;|mBO &uarr;|MSE &darr;|LPIPS &darr;|FID &darr;|
|-|-|-|-|-|-|-|
|BO-QSA|**80.47**|64.26|66.27|268.2|0.38|98.93|
|SlotDiffusion|68.39|57.56|61.94|237.5|0.13|32.07|
|BO-SlotDiffusion|78.50|64.35|**68.68**|240.3|0.13|**30.06**|

See our response to individual reviews for detailed analysis (Slot-TTA: **point 5** to reviewer QakT; bi-level optimization: **point 2** to reviewer jjQP). Overall, our method is able to benefit from recent advances in slot-based methods, and further boost its performance.

[1] Prabhudesai, Mihir, et al. "Test-time adaptation with slot-centric models." ICML. 2023.

[2] Jia, Baoxiong, et al. "Improving object-centric learning with query optimization." ICLR. 2023.


**2. Basic comparison between SlotDiffusion and concurrent work LSD [3]** (suggested by reviewer JU9y)

Since LSD has not released its code, we can only compare with it on CLEVRTex and MOVi-E where we both conduct experiments. We try our best to re-train and test SlotDiffusion under the same setting as LSD. We report the results in **Table 2 of the submitted PDF file**.

On CLEVRTex, we achieve better FG-ARI, and comparable results in mIoU, mBO, and FID. On MOVi-E, we outperform LSD in both segmentation and generation metrics. Please see our response to reviewer JU9y (**point 3**) for more details.

[3] Jiang, Jindong, et al. "Object-centric slot diffusion." arXiv preprint arXiv:2303.10834 (2023).

For other questions raised by the reviewers, please see our response (with text and other new experiments) to individual questions below each review. We will incorporate all our responses and additional results in the final version of the manuscript.


Meta-review: All reviewers reached a consensus to accept the paper with review scores of 6/7/6/7.

The paper proposes a diffusion-based decoder for slot-centric models, which results in better generation, segmentation and video prediction. All reviewers acknowledged the novelty and importance of the work. They also found the paper solid and well-written.

The authors provided additional experimental results and clarifications in the rebuttal, which the reviewers found very helpful. They are suggested to include the new extensions (Slot-TTA and BO-SlotDiffusion) and the discussion about scaling the decoders from prior work in their main paper. They are also encouraged to add the comparisons with CutLer in their supplementary.
Decision: Accept (spotlight)