Title: Object-Centric Slot Diffusion
Venue: NeurIPS 2023
Accepted: yes
Keywords: Object-Centric Representation Learning

Diffusion Models

Unsupervised Representation Learning
Abstract: The recent success of transformer-based image generative models in object-centric learning highlights the importance of powerful image generators for handling complex scenes. However, despite the high expressiveness of diffusion models in image generation, their integration into object-centric learning remains largely unexplored in this domain. In this paper, we explore the feasibility and potential of integrating diffusion models into object-centric learning and investigate the pros and cons of this approach. We introduce Latent Slot Diffusion (LSD), a novel model that serves dual purposes: it is the first object-centric learning model to replace conventional slot decoders with a latent diffusion model conditioned on object slots, and it is also the first unsupervised compositional conditional diffusion model that operates without the need for supervised annotations like text. Through experiments on various object-centric tasks, including the first application of the FFHQ dataset in this field, we demonstrate that LSD significantly outperforms state-of-the-art transformer-based decoders, particularly in more complex scenes, and exhibits superior unsupervised compositional generation quality. In addition, we conduct a preliminary investigation into the integration of pre-trained diffusion models in LSD and demonstrate its effectiveness in real-world image segmentation and generation. Project page is available at https://latentslotdiffusion.github.io

Object-Centric Slot Diffusion

Abstract

The recent success of transformer-based image generative models in object-centric learning highlights the importance of powerful image generators for handling complex scenes. However, despite the high expressiveness of diffusion models in image generation, their integration into object-centric learning remains largely unexplored in this domain. In this paper, we explore the feasibility and potential of integrating diffusion models into object-centric learning and investigate the pros and cons of this approach. We introduce Latent Slot Diffusion (LSD), a novel model that serves dual purposes: it is the first object-centric learning model to replace conventional slot decoders with a latent diffusion model conditioned on object slots, and it is also the first unsupervised compositional conditional diffusion model that operates without the need for supervised annotations like text. Through experiments on various object-centric tasks, including the first application of the FFHQ dataset in this field, we demonstrate that LSD significantly outperforms state-of-the-art transformer-based decoders, particularly in more complex scenes, and exhibits superior unsupervised compositional generation quality. In addition, we conduct a preliminary investigation into the integration of pre-trained diffusion models in LSD and demonstrate its effectiveness in real-world image segmentation and generation. Project page is available at https://latentslotdiffusion.github.io

1 Introduction

The fundamental structure of the physical world is compositional and modular. While this structure is naturally revealed in some data modalities like language in the form of tokens or words, in other modalities such as images, it is elusive how one may discover this structure. However, this representational compositionality and modularity is essential for various applications that require the systematic manipulation of knowledge pieces to achieve high-level cognitive abilities. This includes reasoning [46, 5], causal inference [68], and out-of-distribution generalization [2, 22].

Object-centric learning [27] aims to discover the latent compositional structure from unstructured observation by learning to bind relevant features, thereby forming useful tokens in an unsupervised way. For images, one of the most popular approaches is to auto-encode the image via the Slot Attention [53] encoder. Slot Attention applies competitive spatial attention to partition the image into separate local areas and then obtain a representation, called a slot, from each area. Then, a decoder generates the image from the slots with the aim of minimizing the reconstruction error. Due to the limited capacity and competition between slots, each slot is encouraged to capture a reusable and compositional entity, such as an object.

The primary challenge that remains in the framework of unsupervised object-centric learning is making it work for complex naturalistic scene images. Until recently, most object-centric models have adopted a special type of decoder known as the mixture decoder . While a weak slot-wise decoder [81] is predominantly employed in this mixture decoder due to its efficacy in promoting the emergence of object-centric representation in relatively simple scene images, further studies [70, 73] have shown that this strong prior can make handling complex naturalistic scene images more challenging. Contrary to conventional belief, Singh et al. [70] recently proposed departing from this low-capacity mixture-decoder approach and suggested to use an expressive transformer-based autoregressive image generator in object-centric learning [70, 73]. It was shown that increasing the decoder capacity is crucial for handling complex and naturalistic scenes in this framework [8, 7, 69, 84, 71].

The success of transformer-based image generative modeling in object-centric learning naturally leads to the question: can diffusion models, another pillar of modern deep generative modeling known for their highly expressive generation capabilities, also benefit object-centric learning ? Diffusion models [75, 31] are based on a stochastic denoising process and have demonstrated impressive performance in a variety of image generation tasks [61, 58, 65, 14, 63, 66, 56], sometimes surpassing transformer-based autoregressive models. Moreover, diffusion models possess unique modeling capabilities that transformer-based autoregressive models cannot provide [75]. However, despite their potential, the applicability of diffusion models to unsupervised object-centric learning remains largely unexplored. Consequently, it is crucial to examine the feasibility of this approach and to identify the associated benefits and limitations.

In this paper, we address this question by introducing a novel model called Latent Slot Diffusion (LSD). The LSD model can be interpreted from two perspectives. From the perspective of objectcentric learning, LSD can be viewed as the first model substituting conventional slot decoders with a conditional latent diffusion model, in which the conditioning is object-centric slots provided by Slot Attention. From the diffusion model perspective, our approach is the first unsupervised compositional conditional diffusion model. While traditional conditional diffusion models [61, 63, 65, 52] require supervised annotations, such as a text description of an image for compositional generation, LSD is a diffusion model that enables the construction of such compositional descriptions in terms of visual concepts extracted from images through unsupervised object-centric learning.

In our experiments, we evaluate the proposed model across various object-centric tasks, including unsupervised object segmentation, downstream property prediction, compositional generation, and image editing. We show that the LSD model delivers significantly superior performance compared to the state-of-the-art model, namely, the transformer-based autoregressive generative model. A notable attribute of the proposed model is that LSD's performance advantage over autoregressive transformers increases as the scene complexity increases. In particular, LSD enables exploring, for the first time, the applicability of object-centric model to the FFHQ dataset [41], a collection of high-resolution and high-quality face images that surpasses the generative capabilities of existing object-centric models. Additionally, we discuss the overfitting issue faced by LSD on very simple scene images, such as those in the CLEVR dataset [37], and offer suggestions for addressing this problem.

2 Latent Slot Diffusion

In this section, we outline the auto-encoding framework of our proposed model, Latent Slot Diffusion or LSD 2 , beginning with our object-centric encoder and subsequently describing our proposed decoder.

2.1 Object-Centric Encoder

Given an input image x ∈ R H × W × C , our object-centric encoder seeks to decompose and represent it as a collection of N vectors or slots S ∈ R N × D , where each slot (denoted as s n ∈ R D ) is encouraged to represent a compositional entity in the image. For this, we adopt Slot Attention, an architecture that is also used in the current state-of-the-art object-centric learning approaches [53, 70, 43]. We now describe how Slot Attention works in our model.

In Slot Attention, we first encode the input image x as a set of M input features E ∈ R M × D input via a backbone network f backbone ϕ , i.e. , E = f backbone ϕ ( x ) . The network f backbone ϕ is implemented as a CNN

InputImage

Slots

Compose

Slots

GeneratedImage

K slots

SlotAttention

VisualConceptLibrary

Image

Image

Encoder

Decoder

Zo

Denoising

ZT

Zt

Denoising

Zt-1

Zo

Network

Network

Et

MinimizeSquaredError

N(0, 1)

→

Sample

Add Noise

N(0, 1)

Training

Compositional Image Generation

(detailed in Appendix D.2) whose final output feature map is flattened to form a set. Next, the features in E are grouped into N spatial groupings and the information in each grouping is aggregated to produce a slot . The grouping is achieved via an iterative slot refinement procedure. At the start of the refinement procedure, the slots S are filled with random Gaussian noise. Then, they are refined via competitive attention over the input features, where the N slots act as the queries and the M input features act as the keys and values. The queries and keys undergo a dot-product to produce N × M attention proportions. Next, on these attention proportions, the softmax function is applied along the axis N to produce attention weights A that capture the soft assignment of each input feature to a slot. Then, for each n , all input features are sum-pooled weighted by their attention weights A n, 1 , . . . , A n,M to produce an attention readout u n ∈ R D . These steps can be formally described as follows:

where, q, k, v are linear projections that map the slots and input features to a common dimension D . Using the bottom-up information captured by the readout u n , the slots are updated by an RNN as s n = f RNN ϕ ( s n , u n ) . In practice, competitive attention and RNN update are performed iteratively several times and slots from the last iteration are considered the final slot representation S .

2.2 Latent Slot Diffusion Decoder

In this section, we describe our proposed decoding approach called Latent Slot Diffusion Decoder or LSD decoder for reconstructing the image given the slot representation S . The design of the LSD decoder takes advantage of the recent advances in generative modeling based on diffusion [63, 31]. An overview of our decoding approach is provided in Figure 1.

2.2.1 Pre-Trained Image Auto-Encoder

One of the key design components of the LSD decoder is a pre-trained auto-encoder (AE) [21, 63], which provides a way to map an image x to a lower-dimensional latent representation z 0 via an encoder f AE ϕ . This enables LSD to reduce the computational burden of reconstructing high-resolution images by using the lower-dimensional latent z 0 as an intermediate reconstruction target. It also allows us to later obtain the original resolution image without compromising image content and fidelity by decoding the latent z 0 using the AE decoder g AE θ . These can be summarized as:

In our model, we employ an auto-encoder pre-trained on OpenImages 3 .

2.2.2 Slot-Conditioned Diffusion

In LSD, we leverage diffusion modeling to reconstruct the image latent z 0 conditioned on the slots S . This modeling approach has been primarily explored in supervised contexts, e.g. , for text-to-image generation in Latent Diffusion Models (LDM) [63]. However, unlike LDM, in this work, instead of conditioning the decoder on embeddings of supervised labels, we condition it on slots where the process of obtaining the slots themselves, i.e. , Slot Attention, is jointly trained with the decoder without supervision. Following LDM, our decoder works by training a decoding distribution p θ ( z 0 | S ) to maximize the log-likelihood log p θ ( z 0 | S ) of the image latent z 0 given the slots S . This decoding distribution p θ ( z 0 | S ) is modeled as a T -step denoising process:

where p ( z T ) = N ( 0 , I ) , p θ ( z t -1 | z t , t, S ) is a one-step denoising distribution, and z T , z T -1 , . . . , z 0 is a sequence of progressively denoised latents. The one-step denoising distribution p θ ( z t -1 | z t , t, S ) is parametrized via a neural network g LSD θ in the following manner:

β 1 , . . . , β T is a linearly increasing variance schedule, α t = 1 -β t , and ¯ α t = ∏ t i =1 (1 -β i ) .

Sampling Procedure. To sample a z 0 ∼ p θ ( z 0 | S ) , we adopt an iterative denoising procedure as in [63, 31]. The sampling process starts with a latent representation z T ∼ N ( 0 , I ) filled with random Gaussian noise. Next, conditioned on the slots, we denoise it T times by sampling sequentially from the one-step denoising distribution z t -1 ∼ p θ ( z t -1 | z t , t, S ) for t = T, . . . , 1 . This produces a sequence of latents z T , z T -1 , . . . , z 0 that become progressively cleaner. Finally, z 0 can be considered as the reconstructed latent representation.

Training Procedure. Following LDM [63], the training of p θ ( z 0 | S ) can be cast to a simple procedure for training g LSD θ as follows. Given an image x , its slot representation S , and its image latent z 0 , we first randomly choose a noise level t ∈ { 1 , . . . , T } from a uniform distribution. Given the t , we corrupt the clean latent z 0 and obtain a noised latent z t as follows:

The noised latent z t is then given as input to g LSD θ along with the slots S and denoising time-step t to predict the noise ϵ t . The network g LSD θ is trained by minimizing the mean squared error between the predicted noise ˆ ϵ t and the true noise ϵ t :

2.2.3 Denoising Network

We implement the denoising network g LSD θ as a variant of the conventional UNet architecture adapted to incorporate slot-conditioning. Our denoising network consists of a stack of L layers where each layer l is a UNet-style CNN layer followed by a slot-conditioned transformer:

where h 0 = z t is the input, h 1 , . . . , h L -1 are the hidden states and ˆ ϵ t = h L is the output.

CNNLayers. Following UNet [64], the convolutional layers CNN 1 θ , . . . , CNN L θ downsample the feature map via the first L 2 layers and then upsample it back to the original resolution via the remaining layers. Following standard UNet, these CNN layers also receive inputs via skip connection from an earlier layer denoted by skip ( l ) . This network design has also been explored in LDM [63].

Slot-Conditioned Transformer. The role of the transformer is to incorporate the information from the slots into the UNet-based denoising process. For this, in each layer l , the intermediate feature map ˜ h l produced by the CNN layer is flattened into a set of features. To this, positional embeddings p l are added and the resulting features are provided as input to the transformer. Within the transformer, these features interact with each other and with the slots S , thus incorporating the information from the slots into the denoising process. The transformer output is then reshaped back to a feature map h l .

3 Compositional Image Synthesis

In this section, we describe how a trained LSD model can be used to compose and synthesize novel images. The conventional approach to composing novel images is commonly supervised and rely on text prompts. This involves using words from a vocabulary to create a sentence, which is then given to a text-to-image model to synthesize the desired image [62, 61, 65, 63, 58]. However, in a fully unsupervised setting like ours, we first need to build a library of visual concepts by simply observing a large set of unlabelled images. Then, similarly to composing a sentence prompt using words, we compose a concept prompt by selecting concepts from the visual concept library. By providing this concept prompt to the LSD decoder, we can then synthesize a desired novel image. This approach of unsupervised compositional image synthesis was also explored in [70].

Unsupervised Visual Concept Library. To build a library of visual concepts from unlabelled images, we first take a large batch of B images x 1 , . . . , x B . We then apply slot attention to obtain slots for these images S 1 , . . . , S B . Next, we gather all these slots into a single set S and perform K -means on it. The K -means procedure assigns each slot in S to one of the K clusters. We consider the set of slots assigned to a k -th cluster as a visual concept library V k . With K as the number of clusters, this method results in K visual concept libraries V 1 , . . . , V K . Our experiments will demonstrate that this basic K -means technique can generate semantically meaningful concept libraries. For example, on a dataset of human face images like FFHQ [41], the K libraries correspond to practical concept categories such as hair style, face, clothing, and background.

Novel Image Synthesis. Given libraries V 1 , . . . , V K , we can compose a concept prompt S compose by picking K slots, each from the corresponding k -th library, and stacking them together:

We then give the composed prompt S compose to the LSD decoder to generate the image latent: z compose ∼ p θ ( z 0 | S compose ) . Applying the image decoder on z compose generates the desired novel scene image x compose = g AE θ ( z compose ) . For instance, in the FFHQ dataset, the concept prompt can be a collection of chosen hair style, face, clothing, and background. Decoding this prompt would generate a face image that conforms to this prompt.

4 Related Work

Unsupervised Object-Centric Learning. Object-centric representation learning aims to decompose multi-object scenes into meaningful object entities. A common approach to this is by auto-encoding. In this line, the focus has been to design an appropriate decoder that supports good decomposition. The most widely used decoders include the mixture-decoder [6, 25, 53, 26, 19, 78, 1, 17, 18, 39, 89, 80], spatial transformer decoder [20, 12, 50, 35, 13, 9, 36, 49, 72, 86], Neural Radiance Fields (NeRF) [77, 83, 74], transformer decoder [70, 73, 71, 84, 7, 67, 23], energy-based models [16] and complexvalued functions [55]. Among these, the mixture decoder has been shown to struggle on complex scene images [70, 73] while the spatial transformer decoder requires careful tuning of hyperparameters which limits their applicability in realistic scenes. Neural Radiance Fields require camera poses, and the learning setting involves 3D scenes unlike ours. Another line of work seeks object-centric scene decomposition without reconstruction. This includes works such as [69, 82, 28, 79, 3, 54, 87]. However, unlike ours, these approaches lack the ability to generate images. Preceding object-centric learning, several works pursued disentanglement within a single-vector representation of the scene and use it for compositional image generation [11, 30, 42, 45, 10]. However, lacking spatial binding mechanisms, these methods struggle in multi-object scenes [25] unlike ours.

Diffusion Models. Diffusion models (DMs) are a recent class of generative models that can produce high-quality images by reversing a stochastic process that gradually adds noise to an image [31, 76]. DMs have been applied to various tasks in computer vision, such as class-conditioned generation, textto-image generation, image editing, super-resolution, and inpainting [14, 33, 61, 58, 65, 56, 32, 66]. Recently, [63] proposed Latent Diffusion Models (LDM). By virtue of operating on a low-dimensional latent space, LDM reduces the computational demands of DMs significantly. [51] introduced a method for multi-object scene generation by combining signals from multiple text-conditioned denoising networks. However, to achieve controllable generation, many of these existing DMs require additional labels such as text descriptions to train and control the generation process. In contrast, our model can generate images compositionally using object concepts directly extracted from images. Moreover,

Datasets

LSD (Ours)

SLATE

SLATE

!

CLEVRTex

MOVi-E

FFHQ

DMs have also been used for representation learning. [4] demonstrated that the intermediate features of a learned DM are useful representations for label-efficient semantic segmentation. [60] introduced an image encoder that compresses an image into a latent representation, jointly trained with the denoising objective. Nevertheless, these representations are unstructured and not modular like ours.

Concurrent Research. A concurrent study in [85] also explores a similar idea of object-centric learning using DMs and its downstream applications such as the image editing tasks akin to our experiments. Just like LSD, they employ the Latent Diffusion Model (LDM) as the slot decoder. However, unlike our method, [85] independently train a latent encoder for each dataset, whereas LSD employs a pre-trained latent encoder shared across all input data.

5 Experiments

We extensively evaluate our proposed Latent Slot Diffusion (LSD) model on various object-centric tasks, including unsupervised object segmentation, downstream property prediction, compositional generation, and image editing. As will be shown, our model demonstrates substantial improvements over the state-of-the-art on multiple challenging datasets with complex texture and background, including FFHQ [41] which has been beyond the generative capability of object-centric models. In addition, we also include a preliminary exploration into the potential of LSD using pre-trained diffusion models, discussing its performance in both multi-object datasets and unconstrained realworld images in Section 5.4 and Appendix C.

Datasets. We evaluate our model on five datasets. Four of them are synthetic multi-object datasetsCLEVR [37], CLEVRTex [40], MOVi-C, MOVi-E [24]. They present increasing levels of difficultyCLEVRTex adds texture to objects and backgrounds, MOVi-C uses more complex objects and natural backgrounds, and MOVi-E contains large numbers of objects (up to 23) per scene. Furthermore, we explore the applicability of object-centric models to FFHQ [41], a dataset of high-quality face images. Unlike previous works in this line that only investigate low-resolution images, e.g. , up to 128 × 128 , we use a resolution of 256 × 256 for all datasets in our experiments.

Baselines. We compare our model against SLATE, the state-of-the-art object-centric learning and unsupervised compositional image generation approach. We use its improved version [73], which is more robust in complex scenes. For a fair comparison with LSD that leverages pre-trained auto-encoder, we also develop a variant of SLATE denoted as SLATE + , where its low-capacity dVAE [70] is replaced with a pre-trained VQGAN model [21]. For all models in this work, we use OpenImages-pretrained auto-encoders [63].

5.1 Object-Centric Representation Learning

In line with previous work [53], we use unsupervised object segmentation and downstream property prediction to evaluate the object-centric learning capability and representation quality. Within each dataset, all models we compare use the same number of slots.

(a) CLEVRTex

Unsupervised Object Segmentation. Following [53, 69], to measure segmentation quality, we report the foreground adjusted rand index (FG-ARI), the mean intersection over union (mIoU), and the mean best overlap (mBO). These metrics are computed based on the attention masks generated by Slot Attention. Specifically, we utilize the attention weights described in Section 2.1, where attention weights A are computed as A = softmax ( q ( S ) · k ( E ) T √ ) .

N

D

Our results in Table 1 suggest that LSD is particularly strong in visually complex scenes. For example, on the most challenging MOVi-E dataset, LSD outperforms the strongest baseline by > 8% in mBO, > 9% in mIoU, and > 6% in FG-ARI. On CLEVRTex and MOVi-C featuring complex textures, LSD also achieves > 9% and > 6% gain respectively, in both mBO and mIoU. We visually show the superior segmentation quality of LSD in Figure 2. LSD obtains tighter object boundaries, less object splitting, and cleaner background segmentation. The advantages are most noticeable on CLEVRTex and MOVi-E, where the baselines over-segment the objects more frequently, or exhibit a common failure mode that divides images into approximately uniformly distributed block masks.

We also note that the FG-ARI score is sometimes not an ideal metric for evaluating segmentation quality, because it only considers the foreground pixels and disregards the correctness of the mask shape, as also highlighted by [19, 40]. In the MOVi-E experiment, even though SLATE + partitions the images arbitrarily into uniform patches, as shown in Figure 2, it still achieves a FG-ARI score comparable to SLATE, which produces significantly more reasonable object masks. This highlights the need for additional metrics, such as the mIoU score that we also measure for evaluating the segmentation quality in this work.

Downstream Property Prediction. Following [15, 53], we evaluate the quality of the learned objectcentric representations through downstream property prediction. Specifically, for each property, we train a network to predict the property given a frozen slot representation as input. The correspondence between the slot and its true label is determined by Hungarian matching [44] using masks. We use linear heads and 2-layer MLP as the prediction networks for discrete and continuous values, respectively. We report prediction accuracy for discrete properties ( e.g. , shape and material), and mean squared error for continuous properties ( e.g. , position).

As shown in Table 1, LSD is competitive with or better than the strongest baseline on CLEVRTex, MOVi-C, and MOVi-E, which is consistent with our finding in unsupervised object segmentation that

LSD (Ours)

SLATE

!

Datasets

SLATE

CLEVRTex

MOVi-E

FFHQ

25

LSD shines in visually complex scenes. For example, LSD obtains > 4% gain in predicting object category on MOVi-E and object material on CLEVRTex.

5.2 Compositional Generation with Visual Concept Library

Like text-to-image generative models, LSD is able to take unseen slot-based prompts at test time and compose new images. Unlike text-to-image models, however, LSD obtains the compositional generation ability solely from images without relying on additional supervision like text inputs. As described in Section 3, we first build a concept library. Then, we sample one slot representation from each visual concept library and concatenate them into a sequence to form a slot-based prompt. We then feed the slot-based prompts to the LSD decoder to generate the images. This produces scenes with novel object layouts and faces with unseen attribute combinations.

We report in Table 2 the FID score [29] as a measure of the compositional generation quality. Following standard practice [14], we compute the FID score using 2K generated images and the full training dataset. Across all datasets, LSD achieves significantly better FID scores than SLATE and SLATE + . We further demonstrate the superior compositional generation quality of LSD in Figure 3. We observe that on the more challenging MOVi-E and FFHQ datasets, LSD generates images with substantially more clear details and better coherence. In contrast, SLATE is limited by its decoder and produces images with missing details in the objects or human faces. While the details can be improved by the pre-trained auto-encoder in SLATE + , some samples still exhibit severe distortions.

5.3 Slot-Based Image Editing

In addition to generating new images from randomly sampled slot-based prompts, LSD also allows editing existing images by directly modifying their slot representations. Our experiments showcase LSD's capability in slot-based image editing, including object removal, single-object segmentation, object insertion, background extraction, and background swapping. Remarkably, we demonstrate, for the first time in object-centric generative models, the ability to perform face editing in real-world images, as illustrated in Figure 4.

Original

Removal

Extract

BG

BG Swap

Source A

Source B

Face Replacement

We perform slot manipulation on the CLEVRTex dataset, including object removal, single-object extraction, object insertion, background extraction, and background swapping. To remove an object or extract the background, we discard the relevant slot or all object slots, respectively. Our findings indicate that with near-perfect object decomposition, removing an object can be achieved by simply discarding its slot. Additionally, the background image can be rendered from a single background slot, even without single-slot conditioning during training. In single-object extraction, we render an object with the same background using the respective object and background slots. For object insertion and background swapping, we split the image into top and bottom pairs (Figure 4 left ) and introduce a slot from another image or swap background slots. This demonstrates that an image's background can be entirely altered while preserving its original objects.

We further explore face replacement on the FFHQ dataset. LSD decomposes each image into four slots, corresponding to face, hairstyle, clothing, and background. Our results show that by replacing the face slots of the images, we are able to coherently change the image while maintaining the hairstyle, clothing, and background. The resulting images look realistic, suggesting that LSD can effectively blend various attributes even when given novel combinations.

Input

Slots

Sample 1

Sample 2

Sample 3

Sample 4

Sample 5

5.4 Pre-Trained Diffusion Models and Real-World Object-Centric Learning.

In recent studies of DMs, much attention has been given to controllable image generation using largescale pre-trained diffusion models such as Stable Diffusion [47, 57, 88]. One intriguing question is whether such pre-trained diffusion models can facilitate the learning of object-centric representation. In this regard, we present a preliminary investigation in Appendix C, where we test two LSD variants utilizing the pre-trained Stable Diffusion. Our findings indicate that (1) although the pre-trained Stable Diffusion model has not been trained on any multi-object datasets, it still provides a learning signal for reasonable object-centric learning, (2) with the strong generative capabilities of pre-trained DMs, LSD can scale up object-centric learning to real-world images; however, we also note the part-whole ambiguity issue in slot-based object learning in real-world scenarios, and (3) remarkably, the propose LSD variant achieves conditional generation with unconstrained real-world objects by applying classifier-free guidance [33] to the learned slot, which also provide a visualization of the semantic information captured in the learned representations. We show some image samples in Figure 5, please refer to Appendix C for additional details. We believe these preliminary attempts to be valuable explorations toward large-scale object-centric learning and hope our finding will inspire future research in this area.

6 Discussion

LSD on Simple Images. While LSD shows significant gains in complex naturalistic scenes, our preliminary investigation suggests a somewhat diminished performance in terms of segmentation and representation when the dataset consists of only visually simple and monotonous scene images like CLEVR. We conjecture that LSD may suffer from an overfitting problem when its high expressiveness encounters visually simple images. To address this issue, additional complexities can be introduced during training, such as integrating images from other datasets. In Appendix B, we offer a more detailed analysis of this challenge using the CLEVR dataset and introduce a training scheme that includes data samples from CLEVRTex to alleviate the problem. Our findings indicate that by integrating greater complexities into the training set, LSD can be effectively employed on simple datasets as well.

Computation Requirement. We compare the computation requirement of LSD and the baseline models in the CLEVRTex setting. LSD requires approximately 32 GB of training memory, compared to 31 GB and 36 GB for SLATE and SLATE + , respectively. Notably, LSD does not require much higher memory consumption than regular unconditional latent diffusion, which takes 28 GB under the same setting. We train LSD on 2 NVIDIA RTX 6000 GPUs for 4.5 days, while SLATE and SLATE + are trained in 1 day and 2.7 days using the same GPU setup. For test-time generation, LSD takes 50.7 seconds to generate 50 images, while SLATE and SLATE + takes 86.1 and 87.6 seconds, respectively.

7 Conclusion

The integration of diffusion models into unsupervised object-centric learning has been largely unexplored, but holds significant potential. To assess the feasibility of this approach and to identify its strengths and weaknesses, we introduced the Latent Slot Diffusion model in this study, which serves two purposes: (1) the first model integrating diffusion models into unsupervised object-centric learning, and also (2) the first unsupervised compositional diffusion model which does not require supervised annotations like text. Our key findings indicate that the proposed model surpasses state-ofthe-art transformer-based object-centric models in multiple tasks, with its performance advantage increasing as image complexity grows. Furthermore, the unsupervised compositional generation quality of our model exceeds that of transformer-based counterparts. Finally, we also introduce LSD variants that utilize pre-train Diffusion Models and demonstrate their ability in real-world object-centric learning and conditional generation with real-world objects. We believe that this work represents a significant advancement in object-centric learning towards dealing with complex naturalistic images.

References

Titas Anciukevicius, Christoph H Lampert, and Paul Henderson. Object-centric image generation with factored depths, locations, and appearances. arXiv preprint arXiv:2004.00642 , 2020.

Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: what is required and can it be learned? arXiv preprint arXiv:1811.12889 , 2018.

Federico Baldassarre and Hossein Azizpour. Towards self-supervised learning of global and object-centric representations. In ICLR 2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality , 2022.

Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In International Conference on Learning Representations , 2022.

Léon Bottou. From machine learning to machine reasoning. Machine learning , 94(2):133-149, 2014.

Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390 , 2019.

Michael Chang, Alyssa L. Dayan, Franziska Meier, Thomas L. Griffiths, Sergey Levine, and Amy Zhang. Hierarchical abstraction for combinatorial generalization in object rearrangement. In NeurIPS 2022 Workshop on All Things Attention: Bridging Different Perspectives on Attention , 2022.

Michael Chang, Thomas L Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. arXiv preprint arXiv:2207.00787 , 2022.

Chang Chen, Fei Deng, and Sungjin Ahn. ROOTS: Object-centric representation and rendering of 3D scenes. Journal of Machine Learning Research , 22(259):1-36, 2021.

Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances In Neural Information Processing Systems , 2018.

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems , pages 2172-2180, 2016.

Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of AAAI , 2019.

Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In International Conference on Learning Representations , 2020.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. arXiv preprint arXiv:2107.00637 , 2021.

Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems , 34, 2021.

Yilun Du, Kevin Smith, Tomer Ulman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3d physical objects from video. arXiv preprint arXiv:2007.12348 , 2020.

Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS-V2: Inferring unordered object representations without iterative refinement. arXiv preprint arXiv:2104.09958 , 2021.

Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations , 2020.

SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems , pages 3225-3233, 2016.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition , 28(1-2):3-71, 1988.

Anand Gopalakrishnan, Kazuki Irie, Jürgen Schmidhuber, and Sjoerd van Steenkiste. Unsupervised learning of temporal abstractions with slot-based transformers. arXiv preprint arXiv:2203.13573 , 2022.

Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam H. Laradji, Hsueh-Ti Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator. arXiv preprint arXiv:2203.03570 , 2022.

Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv:1903.00450 , 2019.

Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems , pages 6691-6701, 2017.

Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208 , 2020.

Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelovi´ c. Object discovery and representation networks. In ECCV , pages 123-143. Springer, 2022.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR 2017 : International Conference on Learning Representations 2017 , 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances In Neural Information Processing Systems , 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded Diffusion Models for High Fidelity Image Generation. Journal of Machine Learning Research (JMLR) , 23:47:1-47:33, 2022.

Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. DGMs and Applications @NeurIPS 2021 Poster , 2021.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021.

Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. In Advances in Neural Information Processing Systems , 2020.

Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In International Conference on Learning Representations , 2019.

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2901-2910, 2017.

Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. https://github.com/deepmind/multiobject-datasets/, 2019.

Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P Burgess. Simone: View-invariant, temporallyabstracted object representations via unsupervised video decomposition. arXiv preprint arXiv:2106.03849 , 2021.

Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint arXiv:2111.10265 , 2021.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv:1802.05983 , 2018.

Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv preprint arXiv:2111.12594 , 2021.

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955.

Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations , 2017.

Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences , 40, 2017.

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint arXiv:2301.07093 , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV) , pages 740-755. Springer, 2014.

Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In International Conference on Machine Learning , pages 4114-4124, 2020.

Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations , 2020.

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua Tenenbaum. Compositional Visual Generation with Composable Diffusion Models. In European Conference on Computer Vision (ECCV) , 2022.

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII , pages 423-439. Springer, 2022.

Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.

Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. arXiv preprint arXiv:2011.10287 , 2020.

Sindy Lowe, Phillip Lippe, Maja R. Rudolph, and Max Welling. Complex-valued autoencoders for object discovery. arXiv preprint arXiv:2204.02075 , 2022.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided Image Synthesis and Editing with Stochastic Differential Equations. International Conference on Learning Representations , 2022.

Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 , 2023.

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In International Conference on Machine Learning (ICML) , pages 16784-16804, 2022.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion Autoencoders: Toward a Meaningful and Decodable Representation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10619-10629, 2022.

A. Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical TextConditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125 , 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In Advances In Neural Information Processing Systems , 2022.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image Super-Resolution Via Iterative Refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence , PP:1-14, 2022.

Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Paveti´ c, Mario Luˇ ci´ c, Leonidas J. Guibas, Klaus Greff, and Thomas Kipf. Object Scene Representation Transformer. Advances In Neural Information Processing Systems , 2022.

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612-634, 2021.

Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, CarlJohann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Scholkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209.14860 , 2022.

Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In International Conference on Learning Representations , 2022.

Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural Systematic Binder. In International Conference on Learning Representations , 2023.

Gautam Singh, Skand Peri, Junghyun Kim, Hyunseok Kim, and Sungjin Ahn. Structured world belief for reinforcement learning in pomdp. In International Conference on Machine Learning , pages 9744-9755. PMLR, 2021.

Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. arXiv preprint arXiv:2205.14065 , 2022.

Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Frédo Durand, Joshua B. Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint arXiv:2205.03923 , 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. International Conference on Learning Representations (ICLR) , 2021.

Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01148 , 2021.

Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. arXiv preprint arXiv:2004.12906 , 2020.

Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. arXiv preprint arXiv:2301.11320 , 2023.

Yanbo Wang, Letao Liu, and Justin Dauwels. Slot-vae: Object-centric scene generation with slot attention. arXiv preprint arXiv:2306.06997 , 2023.

Nick Watters, Loic Matthey, Chris P. Burgess, and Alexander Lerchner. Spatial Broadcast Decoder: A Simple Architecture for Disentangled Representations in VAEs. ICLR 2019 Workshop LLD , 2019.

Xin Wen, Bingchen Zhao, Anlin Zheng, X. Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. arXiv preprint arXiv:2205.15288 , 2022.

Yizhe Wu, Oiwi Parker Jones, and Ingmar Posner. Obpose: Leveraging canonical pose for object-centric scene inference in 3d. arXiv preprint arXiv:2206.03591 , 2022.

Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05861 , 2022.

Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Objectcentric generative modeling with diffusion models. arXiv preprint arXiv:2305.11281 , 2023.

Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object-centric representations for reinforcement learning. arXiv preprint arXiv:2302.04419 , 2023.

Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for realworld videos by predicting temporal feature similarities. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023) , 2023.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023.

Ruixiang Zhang, Tong Che, B. Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, and Liam Paull. Robust and controllable object-centric learning through energy-based models. arXiv preprint arXiv:2210.05519 , 2022.

A Limitations and Broader Impact

Limitations LSD faces several limitations common to unsupervised object-centric representation learning. First, it struggles with part-whole ambiguity, meaning it has trouble distinguishing between individual parts of an object and the whole object itself, especially when objects have complex textures in real-world images. Second, the quality and level of detail in LSD's segmentation results are sensitive to the number of slots it uses. If there are too few slots compared to the actual objects in an image, the model tends to produce generalized masks instead of distinct object segments. Conversely, when there are too many slots, the model tends to over-segment the image, splitting objects into multiple parts. This sensitivity to slot numbers is a problem in scenarios where we don't know the exact number of objects in advance. Finally, even with a strong Diffusion decoder, LSD struggles when dealing with real-world images that have highly diverse object appearances. To address this final limitation, we conduct a preliminary exploration in Appendix C, where we propose two LSD variants and demonstrate their effectiveness in handling real-world data.

Broader Impact The proposed Latent Slot Diffusion (LSD) model is a generative model for objectcentric representation learning. The main potential negative social impacts is associated with its application in image editing and novel image synthesis. By enabling the extraction of object-centric or component-based representations from images, LSD allows for the generation of entirely new scenes with previously unseen configurations. This capability raises concerns related to image manipulation and privacy. For example, the capability to synthesize new human faces introduces potential privacy issues like deepfake generation and identity theft. Moreover, this ability could potentially cause digital misinformation, as it can generate and manipulate images with high accuracy in object-based level, making it increasingly difficult to distinguish between real and fake or edited visual content. Additionally, the image composition task gives rise to concerns about ethnic bias, ageism, or other forms of misrepresentation. This is due to the uniform sampling process of the learned slots within the same cluster and their independence across clusters. This can result in two significant problems: (1) the sampled distribution may reflect the biases present in the collected data, potentially leading to ethnic, age, or other population biases, and (2) when performing slot swapping for applications such as face replacement, mismatches in gender, age, and skin tone between existing parts and introduced parts may occur. Therefore, further application of this method should be proceeded under strict ethical guidelines and regulation.

B LSD on Simple Images

In the paper, we demonstrate that LSD achieves considerable performance improvements for complex images. However, our early experiments reveal that when applied to visually simplistic datasets like CLEVR, LSD exhibits reduced performance in terms of segmentation and representation quality, as illustrated in Table 3. To gain further insights into these results, we provide visualizations of the segmentation outputs in Figure 6.

As illustrated by the LSD samples in Figure 6, certain parts of the background, such as the upper regions of the images, are split and assigned to multiple object slots. This behavior leads to information mixing between objects and the background, which substantially reduces the performance of slot segmentation and representation. We conjecture that this issue may stem from overfitting in the model. Due to the simplicity of the texture and the absence of objects in these areas, the powerful decoder of LSD can easily memorize the rendering process of those regions independently of the slot-conditioning. As a result, it does not provide a sufficient training signal for the encoder to group those background parts into a meaningful slot.

In light of these findings, we tested a straightforward approach to address this problem. We propose introducing additional complexity to the background by incorporating data samples from the CLEVRTex dataset into the training process. By integrating the CLEVRTex samples, we introduce 60 different background textures, which makes implicit memorization difficult. This effectively provides the learning signal to capture the background component into the slot-conditioning. The results of this approach, labeled as LSD(Mix), can be seen in Figure 6. As clearly illustrated, including CLEVRTex data samples in the training results in cleaner background segmentation and alleviates the information mixing issue. As a result, LSD(Mix) achieves approximately 22% gains of improvement in both mIoU and mBO compared to the original model trained solely on CLEVR samples, as shown in Table 3. However, we also see that the FID score indicated a decrease in the generation quality of

Dataset

LSD

LSD(Mix)

SLATE

®

SLATE+

LSD(Mix) compared to the original LSD, likely due to the increased demand of model capacity to model both datasets. Overall, this experiment confirms that introducing complexity into the training dataset can mitigate the overfitting problem and allow LSD to effectively perform representation learning on simple datasets.

C LSD with Pre-Trained Diffusion Models

One of the main focuses of recent research in diffusion models have been on adapting pre-trained models for additional control signals [47, 57, 88]. Specifically, these studies propose the integration of a learnable adapter network into large-scale pre-trained diffusion models, such as Stable Diffusion, to enable controllable generation based on various types of conditioning such as sketches, semantic layouts, and depth maps. An inherent advantage of these approaches is their ability to bypass the resource-intensive processes of training the entire generative model, while still enabling the production of high-quality controllable images through minimal training of the adapter network.

We note that such a model design can also be utilized for slot-based conditioning. However, unlike existing works that primarily focus on learning the adapter to integrate the provided conditioning into the pre-trained diffusion model, our focus lies in training an adapter to achieve two objectives simultaneously: (1) we aim to learn how to extract object representations (slots) from input images in an unsupervised manner, and (2) we aim to learn how to incorporate these slot representations into the pre-trained diffusion model to facilitate slot-based conditional generation. This leads to an intriguing question: can learning through such a model, which has not been specifically trained on multi-object

CLEVR

CLEVR(Mix)

CLEVRTex

MOVi-E

COCO

The CLEVR(Mix) corresponds to the model trained on both CLEVR and CLEVRTex samples.

scenarios, provide the necessary learning signal to acquire object-centric representations? In this section, we provide a preliminary exploration on this question by introducing two LSD variants: Control-LSD and Latent Slot Stable Diffusion (Stable-LSD).

C.1 Control-LSD

Model Design Control-LSD utilizes a similar design of the adapter structure proposed in ControlNet [88] and consists of three main components: (1) a pre-trained Stable Diffusion model that is frozen throughout the training, (2) a "control block" that is a trainable copy of the unet encoder from Stable Diffusion, and (3) a object-centric encoder from the LSD model. Similar to LSD, the objectcentric encoder takes an image as input and generates a set of slots, which are then conditioned on using a Slot-Conditioned Transformer. However, instead of providing the slots to the diffusion decoder, Control-LSD feeds the slots into the trainable control block while solely employing null text conditioning in the Stable Diffusion. To integrate the slot control signal, the outputs of each intermediate control block are element-wise added to the intermediate blocks and skip-connections within the frozen Stable Diffusion model. Following LSD, we use a input resolution of 256 × 256 and attention map resolution of 64 × 64 for all datasets. We also use the same number of slots as in LSD for multi-object dataset, and 10 slots for the COCO dataset.

Results We evaluate Control-LSD on three multi-object datasets: CLEVR, CLEVRTex, and MOVi-E, as well as one real-world dataset, COCO [48]. Following the proposed LSD, we also include a CLEVR(Mix) variant for the CLEVR dataset which was trained on both CLEVR and CLEVRTex samples. The segmentation results are presented in Figure 7. We can see that Control-LSD struggles to achieve satisfactory object segmentation on the CLEVR dataset. The model has difficulties in distinguishing between the background and the objects. We also note that the model demonstrate

more reasonable results on CLVERTex and introducing samples from CLEVRTex to the training process of CLEVR (i.e., CLEVR(Mix)) also led to an improvement on CLEVR, but the results are still inferior to those achieved by LSD. Interestingly, we also observe that Control-LSD achieve better segmentation quality when applied to more realistic scenes, such as those in the MOVi-E dataset. We conjecture that this is attributed to the fact that the complexity of MOVi-E aligns more closely with real-world images used for pre-training the Stable Diffusion model, as compared to the samples from CLEVR and CLEVRTex datasets. This indicates that the distribution gap between the data used for pre-training Stable Diffusion and the multi-object datasets used for learning object-centric representation might be a potential contributing factor to the lower performance of Control-LSD on the multi-object datasets.

Finally, we also evaluate Control-LSD on a real-world multi-object dataset COCO [48]. Notably, Control-LSD showcases its ability to capture semantically meaningful entities in the slots, as illustrated in the segmentation results in Figure 7. However, we find it challenging to obtain accurate object segmentations aligned with COCO annotations. In this regard, we would like to share two key observations when applying the model on real-world scenes: (1) Without human annotations, unsupervised models have difficulty distinguishing between discrete parts of an object and the entire object. This is known as the part-whole ambiguity. For instance, while COCO annotates an entire human as one segment, Control-LSD might segment it into two distinct parts: the upper body and the legs. It is important to emphasize that neither of the two segmentation results are incorrect, as each could be beneficial for specific downstream tasks. (2) The segmentation granularity is sensitive to the number of slots. This sensitivity can be attributed to the model's tendency to always utilize the entire set of slots to segment the image. Consequently, if the slot count is less than the number of entities in an image, the model tends to produce semantic masks rather than object-level segments. Conversely, if the slots outnumber the entities, the model tends to over-segment the image, i.e., splitting some of the entity into multiple slots. This behavior limits the applicability of Control-LSD in real-world data, where the true number of objects is not known to the model. Therefore, we believe further investigation is needed for applying the model to real-world applications.

C.2 Stable-LSD

While Control-LSD bypasses the need to train a Diffusion decoder, the computational cost of training the adapter network-which includes both the control block and the object encoder-is still substantial. As an illustration, the memory demands are so significant that even with four 48GB GPUs, our experiments could only handle a training batch size of 32, which also leads to significant training latency. To address this problem, we introduce the the second LSD variant which further reduce trainable modules, we name it Latent Slot Stable Diffusion, or Stable-LSD for short. We show that despite its simple architecture, Stable-LSD demonstrate strong performance on real-world scenarios. We will start with describing the model design. 4

Model Design Firstly, following the DINOSAUR model [69], we employ a pre-trained image encoder, DINOv2 [59], as the CNN backbone for the object encoder. This encoder remains frozen during training. It is important to emphasize that unlike DINOSAUR, Stable-LSD is a generative model capable of generating images. Secondly, the learned slots are provided directly to the cross attention layers of pre-trained DMs as special text tokens. This means that the slot attention module acts dual purposes - as an object encoder extracting object-centric representations and as a feature adapter converting input tokens from the image to the text domain. Notably, within the Stable-LSD framework, only the slot attention module requires training. The resulting design significantly reduces the computational demands observed with Control-LSD. In our experiment, we were able to train the model with a batch size of 128 using two 48GB GPUs.

Real-World Segmentation Results Despite its simplicity, Stable-LSD demonstrate its effectiveness in real-world object-centric learning. To evaluate its object-centric learning ability, we compare Stable-LSD with DINOSAUR on instance-level FG-ARI and mBO. The result can be find in Table 4. The results show that Stable-LSD achieves comparative performance to DINOSAUR with relatively superior FG-ARI value and a diminished mBO value. We show the mask visualization on Figure 8.

STOP

PCOYHNE

Input

Slots

cfg=1.0

cfg=1.1

cfg=1.3

cfg=1.5

cfg=2.0

cfg=3.0

Real-World Image Generation Results We further evaluate the image generation quality of StableLSD. In Figure 8, we show the image reconstruction results (denoted by cfg = 1 . 0 ). The results demonstrate that the model output suffers from visual artifacts which constrained both the image reconstruction and generation capabilities. To further improve the generation results, we explore the technique of classifier-free guidance which is widely employed in text-to-image diffusion models.

Classifier-free guidance [34] uses a mixing weight (abbreviated as cfg) to control the weighted sum of noise predictions between a conditional and an unconditional diffusion model. With cfg = 1 , it operates similarly to standard conditional generation. When cfg > 1 , the model generates images that are more align to the text input, but comes with a cost of reduced image diversity. Practically speaking, an optimal cfg > 1 can yield images that not only better match the conditions but also exhibit enhanced quality. Since Stable-LSD provides object slots as text tokens to the pre-trained DMs, classifier-free guidance can be applied directly to the input slot, akin to how it's applied to input text in standard DMs, without any model modifications. And our experiment shows that a cfg > 1 for slots can also significantly improve the image generation quality.

In Figure 8, we tested various cfg values with Stable-LSD to optimize image generation quality. As we can see in the figure, from cfg = 1 . 0 to 1.3, the images appear cleaner with less visual artifacts,

STOP

STOP

CBD

EOAIHK

Input

Slots

Sample 1

Sample 2Sample 3

Sample 4S

Sample 5

delivering conceptually more consistent images with the input slots. Nevertheless, we also observe that higher cfg values strip away intricate visual details, leading to overly simplified images. We settled on a cfg value of 1.3 for following experiments.

In Figure 9, we show image samples generated based on the same set of learned slots. The variance in these samples comes from differing the initial noise maps in the denoising process. These results demonstrate that with a proper cfg value, Stable-LSD achieves slot-based conditional image generation with unconstrained real-world objects, which is for the first time in object-centric generative models. In addition, beyond showing the diversity of the generated images, these results also underscore a crucial point from the representation learning prospective: the learned slots contain rich semantic information rather than merely performing texture-based clustering. An evidence to this is the model's ability to maintain overall semantic consistency while exhibiting diverse presentations of the same semantic content across samples. For instance, even if the background of the stop sign images shows different tree shapes, they all clearly have trees in the background. Achieving this level of consistency would be challenging if the slots lacked an abstract semantic understanding of the objects in the scene. We show additional real-world segmentation and generation results in Figures 18 - 23.

We believe the prelimilary attempts detailed in this section to be valuable explorations toward largescale object-centric learning, and we hope our insights pave the way for further research in this area.

(a) CLEVRTex

(b) MOVi-C

(c) MOVi-E

D Additional Implementation Details

To ensure reproducibility, we are sharing additional implementation details of LSD in this section. Our implementation will be made available at https://github.com/JindongJiang/ latent-slot-diffusion

D.1 Experiment Setup

Datasets. In our experiments, we create three data subsets for each dataset: training, validation, and testing. For CLEVR, we utilize the official split for training and validation sets. However, for evaluation, we do not use the official test set as CLEVR does not provide ground-truth object segmentation masks. Instead, we use the "clevr_with_mask" subset from the Multi-Object Datasets [38] for evaluation. This subset is designed to have the same object distribution as CLEVR while providing ground-truth masks and attributes annotations for each object in the image, which allows us to evaluate the segmentation and representation quality of each model. We use the full subset as the test set for evaluation. For CLEVRTex, there is no official split provided, so we allocate 80% of the data for training, 10% for validation, and 10% for testing. Regarding MOVi-C and MOVi-E, we use 90% of the training set data for training and reserve 10% for validation. We use the official validation set for testing. It is important to note that the test sets in MOVi-C and MOVi-E are designed for out-of-distribution (OOD) evaluation, so we do not use them for test time evaluation. For the FFHQ dataset, we use 86% of the dataset ( ∼ 60K images) for training and 7% ( ∼ 5K images) for validation. To ensure consistency across all models, we shuffled the dataset and utilized a fixed random seed when splitting the data.

Segmentation Metrics. To evaluate the unsupervised object segmentation, we use three metrics: the foreground adjusted rand index (FG-ARI), the mean intersection over union (mIoU), and the mean best overlap (mBO). We predict the object mask by using the argmax along the slot dimension in the attention map. In computing the mIoU, we use Hungarian matching to obtain the ground-truth and slots assignment. Following [15], we use the negative cosine similarity as the matching loss. In mBO, we assign the ground-truth segment to the slot with the largest overlap. Note that the mBO metric is generally simpler than mIoU since it does not require a strict one-to-one mapping between the ground-truth and predicted masks.

D.2 Implementation Details

We will provide an overview of the implementation details in this section. The hyperparameters used in our approach are listed in Table 6.

Auto-Encoding CNN backbone. The object-centric encoder in our model includes a CNN backbone, which transforms the raw image input into feature vectors that can be processed by the slot attention

module. Previous studies have used multi-layer CNN networks as image encoders, as demonstrated by [53, 70, 73]. However, in our experiments with Latent Diffusion Models, we have found that such encoders tend to produce low-level features that mainly represent pixel position or color information. This leads to the slot attention module grouping pixels based on these low-level features, rather than on high-level object information. We hypothesize that this problem may arise from the early stages of training, where the model lacks a semantic understanding of the input image. This results in the LSD Decoder relying only on low-level and local information from slots to denoise the noisy input, without encouraging the object-centric encoder to learn semantic-level pixel grouping. As a result, without additional guidance, the model may become trapped in a suboptimal solution. We have observed this issue only when a Diffusion Decoder is used.

To overcome this issue, we have incorporated a UNet architecture [64] as the image encoder in the object-centric encoder. The main advantage of this approach is that the auto-encoding structure of UNet allows high-level global context information to be blended into the output features and slots, facilitating the learning process and helping the model escape from suboptimal pixel groupings. Additionally, the skip connections in UNet allow the output features to retain fine-grained low-level details from early CNN layers, ensuring that the resulting representations are rich in both high-level object information and low-level texture information. These improvements to the object-centric

encoder result in more accurate object-centric representations and enable better image generation results. We provide detailed design information in Table 6. Prior to applying the UNet structure, we first employ a single CNN layer with a kernel size of 4 and stride of 4 to downsample the image from 256 × 256 × 3 to 64 × 64 × S , where S is the base channel size in UNet. In Appendix E.1, we provide an additional ablation study for the UNet encoder.

Slot-Conditioned Cross Attention. We implement the conditioning mechanism between the object slots and decoder features through a transformer network as in LDM [63] and SLATE [70]. This conditioning mechanism consists of two transformer layers, including a self-attention layer that computes self-attention within the features generated by the UNet in the LSD Decoder, followed by a cross-attention layer that computes cross-attention between the object slots and UNet features.

In the cross-attention layer, we use the object slots S as key-value pairs, and the queries are intermediate layers of the diffusion network augmented with positional embedding ( ˜ h l + p l ) . We then apply multi-head cross attention for each head as Attention ( Q , K , V ) = softmax ( QK T √ d ) · V , where

Here, i is the head index, ˜ h l represents the intermediate layers of the diffusion network, p l is the positional embedding, S is the set of object slots computed from the object-centric encoder, and W ( i ) V , W ( i ) Q , W ( i ) K are learnable projection matrices.

Additional Details for Object-Centric Encoder. For all multi-object datasets, we set the number of slots to be the maximum number of objects per image plus one (intended for background). For the FFHQ dataset, we segment the image into 4 slots. We use the 256 × 256 resolution images for all experiments and, if necessary, we use bilinear interpolation for re-scaling and center cropping to get square images. We do not introduce additional data augmentations during pre-processing. When computing the object segmentation, we use the attention map from the slot attention as the mask prediction. The resolution of the mask is designed to be 64 × 64 for all models.

Additional Details for LSD Decoder. We add learnable positional embeddings into the self and cross-attention layers of UNet, which are found to greatly improve the stability of the training process for the CLEVR and CLEVRTex datasets. However, we have observed that this modification has a minor impact on the MOVi-C/E and FFHQ datasets, but have applied it across all models to maintain consistency. For the image auto-encoder, we use the KL-8 version provided in the LDM repository. During the generation processes for all tasks, we employ the DDIM sampler with η = 1 and run it for 200 steps.

Baseline Implementation. Our baseline implementations are closely based on the official releases. For the SLATE baseline, we adopt the improved version proposed in [73] 5 . We empirically find this version to be more robust to complex scenes in terms of both object segmentation and generation quality. The key difference of this version from the original SLATE is in the slot encoder. The original SLATE compute object slots from the latent space of the dVAE. Therefore, the attention resolution and information granularity of the slots are constrained by the dV AE latents. The improved SLATE addresses this issue by using a jointly trained CNN backbone to directly learn the slots from the original image. As shown in [73], this improved version of SLATE achieves improved performance on complex datasets such as MOVi-C and MOVi-E. For SLATE + , we replace the jointly trained dVAE with a VQGAN model pre-trained on OpenImages 6 .

E Additional Experiments

E.1 Ablation on CNN backbone in Object-Centric Encoder

This section presents an ablation study on the CNN backbone used in the object-centric encoder. To investigate the impact of the CNN backbone choice, we introduce a variant of LSD named LSD-CNN. Specifically, LSD-CNN uses a multi-layer CNN network as the image encoder in the object-centric encoder, following the same CNN design as in the improved SLATE model [73]. We compare the

(a) CLEVRTex

results of this variant with the proposed LSD, which utilizes the UNet structure as the image encoder in the object-centric encoder, on CLEVRTex, MOVi-C, and MOVi-E. The results are presented in Table 7. Overall, we observe that LSD outperforms LSD-CNN across nearly all tasks and datasets, suggesting that the UNet CNN backbone is critical for achieving the high performance of our model.

To further clarify the architecture choice, we also explore a LSD variant named LSD-Latent that employs an image latent encoder as its CNN backbone. We evaluate LSD-Latent on MOVi-E dataset and compare its object segmentation quality with LSD. The results can be seen in Table 8. The sub-optimal performance of LSD-Latent underscores the importance of having a separate CNN encoder in the object encoder.

F Additional Image Samples

We include additional generation samples in Figures 10 - 23. The additional samples demonstrate that LSD achieves more accurate object segmentation and generates more detailed and coherent scenes compared to the baselines. We provide additional observations and insights in the following sections.

F.1 Visualization of Visual Concept Prompts

In this section, we present visualizations of the visual concept prompts employed for generating image samples in the CLEVRTex and FFHQ datasets, as illustrated in Figure 10. To generate these visualizations, we apply the attention map corresponding to each specific slot as a mask over the original image, which highlights the regions captured by that slot. We observe that the LSD model

Compose

Compose

Concept Prompts

Concept Prompts

can seamlessly combine components from distinct images to create a coherent image sample while preserving the attributes of each component in the final image. Note that each concept library is associated with one k-means cluster within the extracted slots, and each prompt displayed in Figure 10 is sampled from one such library. The visualization of the prompts suggests that the concept libraries is able to capture practical concept categories, such as objects and backgrounds in the CLEVRTex dataset and facial attributes, hairstyles, clothing, and backgrounds in the FFHQ dataset.

F.2 Impact of Pre-Trained Auto-Encoder on Generation Quality

We investigate the effect of the pre-trained image auto-encoder in the comparison between SLATE and SLATE + in Figure 12 and 13. Our results show that the use of the pre-trained auto-encoder leads to less blurry images and significantly more details, such as the floor and object texture in CLEVRTex and face detail in FFHQ. However, we also observe that applying the pre-trained auto-encoder does not necessarily improve the coherence of the generated scenes. For instance, in some SLATE + samples, the shape of the objects appear distorted in CLEVRTex and the background and hair style of FFHQ images are rendered as color patches. These findings indicate the limitation of the transformer decoders in SLATE and SLATE + models and highlights the crucial role of LSD's diffusion decoder in achieving optimal generation quality.

F.3 Impact of Segmentation Quality on Slot-Based Editing

In Figure 14 and 15, we investigate the effect of object segmentation on the slot-based editing task. Our result show that when the object segmentation is almost perfect, editing tasks such as object removing can be easily accomplished by discarding the corresponding slot during image decoding. It is important to note that the object segmentation masks are derived from the attention weights on image features. Therefore, a perfect segmentation mask indicates that the object information is completely assigned to a single slot. However, we also observe cases when the object assignments are incomplete, such as when one object is divided into multiple slots. In such cases, removing an object would require removing all associated slots. These observations emphasize the importance of accurate object segmentation in achieving effective slot-based editing. Although LSD has demonstrated superior performance compared to existing approaches, further exploration in object segmentation is necessary to extend it to real-world applications.

CLEVRTex

LSD (Ours)

SLATE

SLATE

!

Dataset

MOVi-E

LSD (Ours)

SLATE

SLATE

!

Dataset

FFHQ

LSD (Ours)

SLATE

SLATE

!

Dataset

CLEVR

FFHQ

LSD (Ours)

SLATE

SLATE

!

MOVi-E

LSD (Ours)

SLATE

SLATE

!

CLEVRTex

LSD (Ours)

SLATE

SLATE

!

LSD (Ours)

SLATE

SLATE

!

®

LSD (Ours)

Source A

Source B

Source A

Source B

SLATE

Source A

Source B

Source A

Source B

SLATE

!

Source A

Source B

Source A

Source B

一

+

Input

Progressive Object Removal

Masks

Slot Attention

Input

Progressive Object Removal

Masks

Slot Attention

Background Extraction

Source A

Source B

Background Extraction

Source A

Source B

Single Object Extraction

Source A

Source B

Single Object Extraction

Source A

Source B

PCOHNE

EOAH

Input

Slots

cfg=1.0

cfg=1.1

cfg=1.3

cfg=1.5

cfg=2.0

cfg=3.0

STOP

PMorgan

PMor

SAAE

Input

Slots

cfg=1.0

cfg=1.1

cfg=1.3

cfg=1.5

cfg=2.0

cfg=3.0

Input

Slots

cfg=1.0

cfg=1.1

cfg=1.3

cfg=1.5

cfg=2.0

cfg=3.0

Input

Slots

Sample 1

Sample 2S

Sample 3

Sample 4S

Sample 5

STOP

CB0

发灸足

PMorgan

P.Mor

TeOSSDO

POLO

Input

Slots

Sample 1

Sample 2S

Sample 3S

Sample 4Sample 5

Input

Slots

Sample 1

Sample 2

Sample 3

Sample 4S

Sample 5

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Latent Slot Diffusion (LSD), an object-centric learning framework combining the Slot Attention module and a Latent Diffusion Model (LDM) based slot decoder. The model is trained in an auto-encoding manner, where the loss is the slot-conditioned denoising loss in the LDM-based decoder. Extensive experiments demonstrate the effectiveness of LSD in 1) unsupervised scene decomposition  2) object property prediction  3) image generation/editing, surpassing previous SOTA SLATE which uses a auto-regressive Transformer-based slot decoder.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow. The figures are of high quality\n- The use of LDM as the slot decoder is very intuitive and reasonable, considering the recent trends in object-centric models (CNN decoder --> dVAE+Transformer decoder --> KL-VAE+LDM decoder). The results verify this design choice\n- The experimental results cover lots of different tasks, and show very strong performance. The improvement in generation capacity is very impressive compared to baselines\n\nWEAKNESSES:\nI don't see any big issue with the paper. One might consider LSD as \"simply replacing the slot decoder with an LDM\", but I think this modification is reasonable, and works pretty well across datasets and tasks. SLATE/STEVE also just replace the CNN decoder in Slot Attention with Transformer decoders, and they have been widely used due to their strong unsupervised segmentation performance. I believe LSD's improvement in generation quality will also facilitate future research in object-centric generative models.\n\nThat being said, I think the paper should have more discussion of its limitations. See the `Questions` and `Limitations` sections below.\n\nQUESTIONS:\n1. Regarding the pre-trained VAE model:\n- I appreciate the authors' efforts in having SLATE+ with the same VAE for a fair comparison. Have you tried training a VAE from scratch, and comparing this LSD variant with SLATE (which also trains dVAE from scratch)?\n\n2. Regarding the LSD + Stable Diffusion (SD) experiments in the Appendix (**not affecting paper decision**):\n- This is an interesting experiment, especially the results on the real-world COCO dataset. Do you have any numerical results to compare with the SOTA object-centric model DINOSAUR [1]?\n- Can the authors show some generation or reconstruction results? I am curious because I believe even LSD cannot generate realistic unconstrained real-world images (FFHQ images are constrained as they only capture human faces). Will using a pre-trained SD decoder help here?\n\n[1] Seitzer, Maximilian, et al. \"Bridging the gap to real-world object-centric learning.\" ICLR. 2023.\n\n3. Minor question (**not affecting paper decision**): a [concurrent work](https://arxiv.org/abs/2305.11281) you might want to cite, but I understand that paper comes out after the submission deadline.\n\nLIMITATIONS:\nThe authors discuss limitations in Section 6. However, there are more limitations of this work (and the entire object-centric learning field):\n- LSD is still unable to generate realistic unconstrained real-world images\n- LSD suffers from the part-whole ambiguity issue, as can be seen from the COCO examples\n\nI'd like to see the authors discuss these points in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevious work has shown that transformer-based image generative models can be trained for object-centric learning which can handle complex scenes. That is, given unstructured observation, transformer-based models learn to find latent compositional structures to bind relevant features.\n\nThis work makes an attempt to explore the feasibility and potential of integrating diffusion models into object-centric learning.\n\nA major contribution of this work is a novel Latent Slot Diffusion (LSD) model which combines conventional slot decoders with a conditional latent diffusion model, conditioned on object-centric slots provided by Slot Attention.\n\nThe authors have shown the effectiveness of the proposed model by evaluating several object-centric tasks including unsupervised object segmentation, downstream property prediction, compositional generation, and image editing.\n\nSTRENGTHS:\n- **Scope and relevance**:  Considering the growing interest in diffusion models, this paper is exceptionally timely as it broadens their application towards object-centric learning.\n\n- **Significance of contributions**:  This paper presents a novel diffusion model for object-centric learning without the requirement of supervised annotations. It answers the raised question about how diffusion-based generative modeling can benefit object-centric learning properly. \n\n- **Experimental results**:  The experiments in this paper are enough to prove the effectiveness of the proposed model.\n\n- **Clarity**:  The main body of the paper is written very well.\n\nWEAKNESSES:\n- **Limited technical contribution**:  The proposed slot-contained diffusion is a trivial extension of existing text-to-image generation in Latent Diffusion Models (LDM) [59], which replaces text inputs with latent slots.\n\n- **Unclear model details**:  To my understanding,  the whole model is trained for image generation objectives. How to apply such a generative model to unsupervised object segmentation is not clear.\n\n- **Unclear training details**: if I am not mistaken, the object-centric encoder is trained independently rather than jointly with the latent slot diffusion decoder. I would suggest the authors clarify this. Thus, control experiments might be needed to study the effect of training strategies.\n\n- **Some implementation details are missing**:  Without providing the source code, some important hyperparameters are not presented to justify the reproducibility of this work, eg., the number of $K$ for k-means cluster,  noise schedule $\\alpha$, and steps $T$.\n\nQUESTIONS:\nIn addition to the above weaknesses, here are two more questions:\n\n- In line 231, given that the object segmentation masks are derived from the attention masks of Slot Attention, why using diffusion-based generative modeling can help unsupervised object segmentation?\n\n- The LSD model has been evaluated across multiple tasks and outperforms existing state-of-the-art methods. However, it raises the question: to what extent does the enhanced performance truly stem from the diffusion-based generative modeling rather than the improved image encoder? It's an intriguing point to ponder.\n\nOverall, this paper is a good effect. I will raise my rating if the authors can address my concerns.\n\nLIMITATIONS:\nThe authors did not adequately address the limitations of this work.\n\nA significant constraint of this study is the requirement for two distinct visual encoders: a pre-trained image auto-encoder and an object-centric encoder. This dual requirement results in a relatively heavy model for inference. This leads to the question: Is there any potential to share certain components between these two encoders to enhance efficiency?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

comments: SUMMARY:
The paper studies the diffusion models into object-centric learning. The authors introduce the concept of latent slot diffusion (LSD) that can replace slot decoders conditioned on object slots. The model can work in an unsupervised compositional mode without requiring annotations such as text. Their experiments show that LSD performs better in complex scenes comparing with other methods that do unsupervised compositional generation.

STRENGTHS:
The proposed method LSD can be viewed as a model substituting conventional slot-decoders with a conditional latent diffusion model, where the conditioning is done via the object slot attentions. It can also be viewed as a unsupervised conditional compositional diffusion based generation. The ablation experiments are helping in explaining the crux.

WEAKNESSES:
Object segmentation using LDS is yet to be perfected, leading to suboptimal downstream applications.

QUESTIONS:
What are potential ways to improve segmentation using LSD?

LIMITATIONS:
As a generative model, the method needs to consider the privacy and impacts of image manipulation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a methodology for using diffusion-based models to obtain object-centric representations. \nThis is done using slot representations, obtained from the Slot Attention module applied to the original image, as a conditioning variable for a latent diffusion model. The diffusion model and the Slot Attention module are trained end-to-end.\n\nThe proposed model is called Latent Slot Diffusion (LSD) and is evaluated on unsupervised object segmentation, downstream property prediction, compositional image generation, and image editing. The datasets used for evaluation are ClevrTex, MOVi-E, MOVi-C and, for the first time in the field, FFHQ. The performance of LSD is evaluated quantitatively on the first 4 tasks and is compared against SLATE and SLATE+. The proposed model outperforms both baselines in the tasks according to most metrics.\n\nSTRENGTHS:\n**Originality**. \nThe paper proposes a novel way to combine two existing models: slot attention and latent diffusion. The presentation clearly shows what are the novel elements.\n\n**Quality**. \nThe method shows a successful way to leverage diffusion in object-centric learning. Especially the results on property prediction show that it has concrete benefits for the representation learning itself, which might be useful for several different tasks, not only the ones shown in the paper.\n\n**Clarity**. \nThe submission neatly shows all the experiments that were carried out and the description of the underlying method is clear.\n\n**Significance**. \nThe work is a necessary exploration of leveraging the generative power of the diffusion approach in the object-centric setting, highlighting the complexities related to having too-strong of a decoder and providing clear examples of its strengths, especially by using a very complex dataset (FFHQ).\n\nWEAKNESSES:\n**Quality**. \nThe lack of error bars (i.e. standard deviation) in the analysis makes the quantitative analysis weaker. Additionally, it would be interesting to further explore the problems with FG-ARI, as it is currently the standard metric used in the field, and although I agree with the statements, this is only based on experience and intuition, and not a proper scientific analysis, which has not been carried out yet. Lack of comparison with traditional models such as the improved Slot Attention architecture proposed in [1] makes the performance of the model harder to evaluate. Are the good results obtained primarily due to the improved architecture (e.g. larger encoders, better decoder), or is there something fundamentally good about using diffusion (e.g. the iterative improvement) that results in better object representations?\n\n**Clarity**\nThe description of the method used for unsupervised object segmentation is lacking. There is only a reference to the use of the attention masks from slot attention, it would be much better to refer directly to the nice mathematical notation used before in the text.\n\n*References*\n1) Biza, Ondrej, et al. \"Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames.\" arXiv preprint arXiv:2302.04973 (2023).\n\nQUESTIONS:\nAs object segmentation is performed using the masks from the slot attention module, it is clear that it is not possible to obtain masks with the full resolution of the images in many cases. Could this be an area of improvement for future work or is it possible to already try using some unsupervised super-resolution technique to get higher-quality masks? Could be worth considering object segmentation as a visualization of what the model is doing instead of a separate task?\n\nDo you have any insights into how the model would perform if slot attention is applied to the latent representation of the latent diffusion model instead of the original image? This would help clarify the choices made during the development of the method.\n\nLIMITATIONS:\nThe authors have sufficiently addressed the limitations of the proposed model, as well as the broader impact of their approach. However, this can be improved further, by being more explicit about direct effect of ethnic bias, ageism or other forms of mis-representation that the model could lead to, which is standard practice for modern diffusion model, considering the extreme high-quality of the generated images. Direct manipulation of certain characteristics enables these effects in a much easier way.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}