Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generation
Authors: Kaan Akan
Yucel Yemez
Venue: ICLR 2025 Poster (2025)
Keywords: object-centric learning
slot attention
unsupervised object discovery
object segmentation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A novel object-centric learning approach integrates slot attention with pretrained diffusion models, enhancing object discovery and image generation while reducing computational costs using adapters and attention mask guidance.

Abstract: We present SlotAdapt, an object-centric learning method that combines slot attention with pretrained diffusion models by introducing adapters for slot-based conditioning. Our method preserves the generative power of pretrained diffusion models, while avoiding their text-centric conditioning bias. We also incorporate an additional guidance loss into our architecture to align cross-attention from adapter layers with slot attention. This enhances the alignment of our model with the objects in the input image without using external supervision. Experimental results show that our method outperforms state-of-the-art techniques in object discovery and image generation tasks across multiple datasets, including those with real images. Furthermore, we demonstrate through experiments that our method performs remarkably well on complex real-world images for compositional generation, in contrast to other slot-based generative methods in the literature. The project page can be found at https://kaanakan.github.io/SlotAdapt/

Published as a conference paper at ICLR 2025
SLOT-GUIDED ADAPTATION OF PRE-TRAINED DIFFUSION MODELS FOR OBJECT-CENTRIC LEARNING AND COMPOSITIONAL GENERATION
Adil Kaan Akan"
Yucel Yemez
Department of Computer Engineering, Koc University KKUIS AI Center, Koc University
ABSTRACT
We present SlotAdapt, object-centric learning method that combines slot at- tention with pretrained diffusion models by introducing adapters for conditioning. Our method preserves the generative power of pretrained diffusion models, while avoiding their text-centric conditioning bias. We also incorporate an additional guidance loss into architecture align cross-attention from adapter layers with slot attention. This enhances the alignment of our model with the objects the input image without using external supervision. Experimental re- sults show that our method outperforms state-of of-the-art techniques in object dis- covery and image generation tasks across multiple datasets, including those with real images. Furthermore, we demonstrate through experiments that our method performs remarkably well on complex real-world images for compositional gen- eration, in contrast to other slot-based generative methods in the literature. The project page can be found at https://kaaaakann wtwpaaiii.buu..oootaaiiiiooo
INTRODUCTION
The real world is inherently structured with distinct, composable parts and objects that can be com- bined in various ways; this compositional characteristic is essential for cognitive functions like rea- soning, understanding causality, and ability to generalize beyond training data (Lake et al. 2017] Bottou 2014, Schölkopf al.) 2021 Bahdanau et al.] 2019: Fodor & Pylyshyn 1988). While language clearly reflects this modularity through sentences made up of distinct words and tokens, the compositional structure is less obvious in visual data. Object-centric learning (OCL. offers a promising approach to uncover this latent structure by grouping related features into coherent object representations without supervision (Kahneman et al. 1992, Greff et al. 2020). By decomposing complex scenes into separate objects and their interactions, OCL mimics how humans interpret their environment (Spelke & Kinzler 2007), potentially improving the robustness and interpretability of Al systems (Lake al.] 2017 Schölkopf et al. 2021). This approach shifts from traditional pixel- based feature extraction to more meaningful segmentation of visual data, which is key for better generalization and supporting high-level reasoning tasks.
Recent advances in OCL have shown the potential to incorporate powerful generative models, such as transformers and diffusion models, into the OCL framework as image decoders. Notably, models such as Latent Slot Diffusion (LSD) (Jiang et al. 2023) and SlotDiffusion (Wu et al. 2023b have considerably improved performance in object discovery and visual generation tasks real-world settings by employing slot-conditioned diffusion models. A concurrent work, GLASS (Singh et al., 2024). has made further progress handling complex natural scenes by using generated captions as external guiding signals.
There exist two main approaches of integrating diffusion models into slot-based OCL methods in order to deal with real-world images. The first one, seen in GLASS and LSD, directly uses pretrained stable diffusion model as decoder. While this approach attempts to fully harness the generative power of pretrained diffusion models, it relies on cross-attention layers to achieve slot conditioning, which were however optimized to receive text input. contrast, the second approach, as in SlotDiffusion, trains the diffusion model from scratch on the target dataset, thereby eliminating any such text- coxdiitodiionnng related biases. However, this limits the generation capacity the slot- conditioned model, making less effective in handling complex real-world images.

---

Published as a conference paper at ICLR 2025
Enabling slot-based methods fully exploit the generative capabilities of pretrained diffusion mod- els while avoiding biases due to text-trained conditioning is a major challenge in object-centric learning, which hinders the effective handling of complex real-world images, particularly in compo- sitional image generation tasks (Jung et al. 2024p. In this work, we propose a novel approach that addresses this challenge by combining slot attention (I (Locatello et al. 2020) with adaptive condi- tioning in diffusion models (Mou et al., [2244] Rombach et al. 2022 Sohl- Dickstein et al.. 2015) Ho 2020 Our method, that we refer to as SlotAdapt, introduces the use of adapter layers (Mou al.
et al.
2024g for slot conditioning, which eliminates text-trained conditioning bias in pretrained dif-
fusion models and allows the slots to diverge from representations in the text embedding space while retaining the generative power of the pretrained diffusion model.
Another challenge for OCL methods dealing with the part-whole hierarchy problem the diffi- culty of deciding whether to segment an object into its parts or as a whole (Hinton 1979). especially in real-world settings. The GLASS model (Singh et al. 2024h mitigates this problem with the aid of external supervision and at the cost of increased complexity, whereas Jung et al. (2024) intro- duce compositional representation learning; however the applicability of their method on real-world datasets remains limited. In order address this challenge, we propose using cross- attesttot masks from adapter layers as pseudo-ground truth to guide slot attention maps. This self-suuervissory signal enhances the alignment between learned slots and actual objects without external supervision, facil- itating the learning of meaningful object representations. In fact, in our architecture, the guidance is mutual, as the cross-attention masks in the diffusion model are also guided and refined jointly with the slot attention masks, improving the image generation process.
In summary, we have the following main contributions: I) We introduce adapters for slot-based con- ditioning to combine slot attention with pretrained diffusion models, 2) We propose an additional guidance loss align pross-attention from adapter layers with slot attention without using exter- nal supervision, and 3) We present compositional image generation results on complex real-world images.
Through extensive experiments on various datasets, we demonstrate that our method, SlotAdapt, out- performs previous approaches in object discovery and compositional image generation tasks QDang et al. 2023 Wu et al. 2023b Seitzer et al. 2023). especially complex real-world image datasets. The performance of SlotAdapt in segmentation task is similar with the concurrent work (Singh et al.) 2024). yet achieve this with significantly reduced computational requirements and without any external supervision. Note that GLASS does not address the compositional generation task. To the best of our knowledge, our work is the first to present successful results for compositional genera- tion on COCO (Lin et al. 2014, a large complex real-world image dataset with 80 different object
classes.
RELATED WORK
2.1 UNSUPERVISED CENERTCC LEARNING
Unsupervised object-centric learning aims to decompose visual scenes into meaningful object rep- resentations without need for annotation. Early methods used iterative inference and CNN decoders to reconstruct scenes from object-specific feature vectors (slots) (Eslami et al. 2016 Burgess et al.
2019 Greff et al./ 2019 Lin et al. 2019, Jiang et al. 2019 Lin et al.] 2020). Attend-Infer-
(AIR) (Eslami et al.] 2016) and Sequential AIR (SQAIR) (Kosiorek et al. 2018)) reconstructed ob- jects in canonical poses using patch-based decoders. Slot attention (Locatello al.
2020) and
SAVi (Kipf et al. 202T) employed the spatial broadcast decoder (Watters et al. 2019) to predict images and segmentation masks from slots, combined via alpha compositing. While effective on synthetic datasets like CLEVR @Jonsoon et al. 2017) and 3D Shapes (Burgess & Kim 2018), 8), these methods struggle with complex, real-world images. Transformer-basee decoders, such as SLATE (iighh et al.] 202T) and STEVE (Singh et al. (022), pre-train a discrete VAE (dVAE) to tokenize images and use alot-conditioned transformers for autoregressive reconstruction. Despite improve- ments, they face challenges in the high-quality reconstruction of complex real images (Singh et al., 2021 Wu et al. 2023a). Methods like DINOSAUR (Seitzer et al. 2023) bypass reconstruction by using self-supervised learning to discover objects but lack image generation capabilities. To improve slot-based autoencoders, Kakogeorgiou et al. 2024) propose self- training and patch-order permuta- tion strategies, enhancing segmentation in complex real-world images. A relatively recent research direction in OCL is to employ diffusion models as slot decoders, enhancing scene decomposition and visual fidelity in weal-world scenarios (Jiang et al. 2023 Wu et al. 2023b). Several approaches
2

---

Published as a conference paper at ICLR 2025
have tackled compositional scene generation through different mechanisms: combining global and symbolic latent variables (iang & Ahn 2020), using hierarchical VAEs with slot attention (Wang
et al. 2023), and leveraging hierarchical discrete representations with autoregressive transformers (Wu et al. (024)) While these works focus on synthetic datasets, our work can achieve composi- tional generation in real-world scenarios.
2.2 DIFFUSION MODELS
Diffusion models (Sohl- Sickstin al. 2015 Ho et al.
2020 have shown remarkable versatility
in tasks such as class conditioned generation, text-to image-maaeeeee synthesis, and image editing (Dhariwal
& Nichol 2021 Ho & Salimans 2021 Meng et al.] 202T Ho et al. 2022 Saharia al. 2022b
Ramesh et al. 2022 Nichol et al. 2022 Saharia et al. 2022a). Latent Diffusion Models (LDMs)
(Rombach et al., 2022 address the computational complexity of diffusion models by operating in
a lower-dimensional latent space through the use of pre-trained autoencoder. This approach sig- nificantly reduces computational load while maintaining generation quality. LDMs also introduce flexible conditioning mechanism through Pross-attention layers, enabling integration of various conditioning information, such as text embeddings. Recent advances such as T2I adapters (Mou et al. 2024 Biner et al. 2024] further enhance the adaptability of LDMs. By introducing additional adapter cross-attention layers, these methods allow fine-tuning for new conditional tasks while keep- ing the base LDM frozen, reducing computational costs and data requirements for model adaptation.
2.3 OBJECT-CENTEN LEARNING METHODS WITH DIFFUSION MODELS
Recently, several works have investigated the use of diffusion models in object-centric learning, such as Latent Slot Diffusion (LSD) (hang et al. 2023). SlotDiffusion (Wu et al., 2023b), (Jung et al.
2024 and GLASS (iighh et al. 2024p. All these methods employ LDMs (Rombach et al. 2022) as
slot decoders. Their approaches are similar in that they all use slot attention (Locatello al. 2020 to extract slot representations from input image and then condition the diffusion model with these slots through cross-attention modules. These methods primarily differ in whether they use pretrained diffusion models, fine-tune them, train them from scratch. LSD and GLASS use pretrained diffusion models to be able to deal with complex rorl-world images, assuming an inherent alignment between slot representations and the text-trained cross aots--attnnionn within the diffusion model, which hinders their performance, especially on image generation tasks. SlotDiffusion, on the other hand, opts to retrain the diffusion model on the target dataset, thereby avoiding biases due text-trained conditioning, though at the cost of increased computational complexity and, more importantly, at the loss of the generative power of pretrained diffusion models.
GLASS, which is actually a concurrent work (Singh et al. 2024). attempts to mitigate the issues due to text-trained conditioning in pretrained diffusion models by utilizing cross-attention masks as useudo-ground truth to guide slot attention, in a manner similar to our approach. However, it neces- sitates an external image captioner or class labels to do this and requires two forward passes: one for generating the training images and pseudo masks, and another for slot attention and image recon- struction. Despite this, GLASS still struggles, particularly in differentiating object instances of the same class, and does not present any compositional image generation results. In contrast, SlotAdapt incorporates additional cross-attention layers into the pretrained diffusion model as adapters, en- abling the slots focus primarily on object semantics, rather than being constrained within a text- centric embedding space.
Jung et al. 2024) have recently introduced novel framework for compositional learning, which incorporates an additional compositional path into their architecture alongside the slot-conditioned diffusion model. Their approach processes two images simultaneously, composing objects from these images into a single output to apply a diffusion- based generative prior loss. This facilitates the learning of compositional information; however, the applicability of the resulting model on complex real-world images is limited; more specifically, they present results on BDD 100K (Yu et al. 2020). an autonomous driving dataset with limited context and relatively small number of object classes.
SLOT-BASED SOJECT-CENTRES LEARNING
3.1 BACKGROUND
Slot Attention:
Slot Attention (Locatello et al. 2020 provides a robust framework for segmenting
input data into discrete, interpretable components. operates by dynamically allocating a set of learnalle vectors, termed "slots", to represent distinct entities within the input data. These slots
3

---

Published as a conference paper at ICLR 2025
Modified Unot Block
guidhors
DINO
Slot Attention
Mals
Pooting
Takan
Oupst Poac ko
SD
Belf Atwriion
Tlag pler Tokan
Figure 1: SlotAdapt Architecture We extract object-centric information from the input image us- ing visual backbone, which combines DINO and slot attention. Stable Diffusion VAE is used to encode the image into latent space and then noise is added to the latent. Diffusion process condi- tioned on the generated slots well the register token which is generated by (mean) pooling the slots. We use the original cross attention layers diffusion model to condition on the register token, and additional adapter attentions condition the learned slots. The overall objective is predict the noise added to the image. Additionally, we introduce a guidance loss between the slot attention masks and adapter cross attention masks, which encourages the similarity between them. The guid- ance is only applied in the third upsampling block, while slot conditioning is applied throughout all downsampling and upsampling blocks.
are often initialized randomly and then iteratively updated via an attention mechanism, enabling the model to bind each slot to different parts of the input corresponding to individual objects or object-like like features. The iterative update rule for each slot is given by.
U(m) Attention
(S()) k(f)),
S(m+1) (RU(S(")) (("))
where q, k, learnable linear functions, respectively corresponding the query, key and value in attention computation; denotes image features; S is the set slots; U represents the update generated by the attention operation; and m is iteration number for the GRU (Gated Recurrent Unit). The initial slots, (0)), can be initialized from Gaussian distribution using initial object locations (Kipf et al. 2021 Elsayed et al., 2022). Attention is computed over the slot dimension, causing slots to compete for pixels. As result, each slot attempts bind distinct object each iteration. The effectiveness of Slot Attention lies in its capacity to disentangle and encode complex scenes into structured representations without supervision. This is achieved through the ability of the mechanism to assign representational capacity where needed.
Diffusion Models:
powerful class of generative models that simulate the gradual addition of noise to data and then learn to reverse this process. Once trained, they generate high-quality samples from pure Gaussian noise, and have shown remarkable success in various domains, particularly in computer vision for challenging tasks such image generation and editing.
Diffusion models (Sohl-Dickstein al. 2015 Ho al. 2020
The generative process involves a series of reverse diffusion steps from a noise distribution p(z) the data distribution p(x), defined by conditional probabilities p(x)-1(x)) The training objec- tive minimizes the difference between the actual and generated distributions, typically using a loss function L that penalizes the expected error of noise prediction each time step t:
(()) Exx=p(x),, (.01)) [|]e; cx L, 1)
(2)
where eg is the noise prediction of the neural network with parameters O, and y is conditioning signal for the model such as class or text. When dealing with images, the common practice is to perform training and sampling in low-dimensional latent space, which is achieved by first trans- forming the input via pretrained variational encoder and then to use the standard U-Net architecture the denoising neural network ég (Rombach al. 2022
4

---

Published as a conference paper at ICLR 2025
Previous OCL methods which employ diffusion models as slot decoders replace the condition y in Eq. 2 with slot representations generated by Slot Attention (Locatello et al. 2020) and use recon- struction loss learning signal.
3.2 SLOTADAPT
Object-centric Visual Encoding: We start with an input image E R and transform it through a visual backbone combined with slot attention in a manner similar to recent works Jiang al. 2025 Wu et al. 2023b). The visual backbone serves a feature extractor, reducing the image size to create a set of visual feature vectors f E R by flattening. This reduction makes compu- tations more efficient and condenses key features of the image into more compact representation. The slot attention mechanism works on these vectors, using a competitive process to generate N slots, represented as S E Rxd Each slot contains information about separate object entity in the scene. The resulting slot representations are then used to condition the diffusion- -ased decoder as explained the following section (see also Fig.
Slot Conditional Decoding: We employ a pretrained Stable Diffusion UNet model as a decoder conditioned the extracted slot representations. This UNet is primarily composed of residual, self-attention, and cross-attention layers stacked together. While self-attention layers capture spa- tial information, cross-attention layers model semantic relationships between text embeddings and intermediate features. Inspired by recent works (Mou et al. 2024 Ye et al. 2023) we extend this architecture by introducing an adapter layer after each existing cross-attention layer in all downsam- pling and upsampling blocks of the UNet. These adapter layers are dedicated to conditioning the model on the extracted slots and are essentially cross-attention layers which function similarly to their text-based counterparts in the UNet, with few minor differences (see Appendix for imple- mentation details).
Our odapter-based conditioning strategy is substantially different from the conventional use of text- trained cross-attention for slot conditioning. Our rationale is that by including these additional cross-attention layers, enable the slots to focus primarily on object semantics, rather than being constrained within a text-centric embedding space. This is particularly crucial, given that the cross- attention layers in pretrained diffusion modules typically optimized for text embeddings and hence expect textual input.
We also introduce extra token that leverages the unused text-conditioning modules of the pre- trained UNet to better capture context. This extra token is computed by (mean) pooling either the generated slots or the image features from the visual backbone, and then fed as input into the text cross-attention layers of the UNet model. The main motivation here is to create a 'register' token, similar the idea presented by Darcet et al. (2024). This register acts information within the diffusion model. By giving this task to dedicated token, we allow the indi- vidual slots to concentrate more on specific objects instead of diluting their focus with background or contextual details.
storage for overall scene
We freeze the pretrained diffusion model and train only the adapter layers and the slot attention com- ponent of our architecture. The training process minimizes a reconstruction objective, formulated as a noise prediction problem. Following (Rombach et al. 2022). at each training iteration (assuming a mini-batch of size 1), time step f, latent image and noise e are first sampled from their respective distributions. The noisy image X; is then calculated via forward diffusion and fed into the denoising UNet cg. The UNet predicts the noise e,, conditioned the extracted slots S and the register token r, and is updated based on the gradient of the following loss function (see also Eq. 2b:
()) = ||e, (,,
(3)
Attention Guidance: The attention mask generated by slot attention serves affinity measure
between image features and slot vectors, effectively segmenting objects in the image under the as- sumption that the slots capture object representations. We enhance this framework by leveraging the cross-attention mask extracted from the adapter layers in the diffusion model as a self-supervisory signal to guide slot attention, and/or vice versa. These dual attention masks, one from slot attention and the other from the diffusion model, encode similar semantics regarding the relationship between slot representations and image features. Normally, there are as many adapter cross- coss-attention masks as there are UNet blocks, but we focus only on the one from the third upsampling block (the second- lastast) since this layer is very close to the output and empirically provides an attention mask most
5

---

Published as a conference paper at ICLR 2025
aligned with the objects in the image. We denote the slot attention mask by ASi and the diffusion attention mask with Aри- In both cases, the attention masks are computed through the dot product between queries and keys, and normalized over the query dimension. In slot attention, slots act as queries and image features as keys, whereas in the attention mechanism of the diffusion model, this relationship is inverted. Formally we can write
ASa Softmax (AsA(fan (ffa))sss (S)
KPM (S) (FOm)
ADM Softmax
(4)
VD
vD
where OSA. ASs. 9om- and kuM are learnable linear functions, and fsx and fum represent image fea- tures. The cross-attention layer in the adapter employs a multi-head structure, resulting in multiple attention masks. We average these masks over the head dimension and use the result as the diffusion attention mask ADM-
The slot attention mask, ASa E R(b-w)) encodes how each pixel in the image features relates to the slot representations. Ideally, this attention mask should converge to an instance segmentation mask, with each slot representing a distinct object. Conversely, the diffusion attention mask, ADM R. R.Nxx (h-w)) shows the inverse relationship. In the optimal scenario, expect this attention map to converge to the transpose of the instance segmentation mask, enabling the diffusion model to generate the imput image accurately. We formulate guidance loss enforce the alignment of these dual attention masks by
Lguidance BCE( (AAA AJu)
(5)
where BCE is the binary cross-entropy loss. The overall training objective combines this guidance loss with the primary loss function, weighted by a hyperparameter X:
L Le + guidance
(6)
There are different design choices for implementing the guidance loss: 1) guiding only the slot attention mask ASa with AnM (stopping gradient for diffusion model), 2) guiding only the diffusion attention mask Apm with ASa (stopping gradient for slot attention), and 3) joint guidance, that is, guiding both Asa and Aps simultaneously (no gradient stopping). So in the first two options, one attention mask serves as a pseudo ground-truth for the other, whereas in the last option, the two masks are jointly enforced for alignment.
Another alternative we have considered guiding the attention masks through multiplication. In this scenario, there no explicit auxiliary loss: each of the cross-attention masks from multiple attention heads in the adapter layer is simply multiplied by ASa using the Hadamard product. Since each cross-attention mask ideally represents a part of the object bound by the corresponding slot, the multiplication confines the adapter attention to the region defined by the slot attention mask.
All the guidance alternatives described above aim, one way or another, to enhance the alignment of the attention matrices which represent the same semantics in the architecture, thereby improving their alignment with the objects in the input image. We compare these different alternatives for attention mask guidance in the experiments section.
EXPERIMENTS
Datasets: Our evaluation framework covers both synthetic and real-world datasets, aligning with recent works in the field (hang et al. 2023 Wu et al, 2023bb. We assess our method SlotAdapt on the synthetic MOVi-E dataset (Greft et al. 2022) and two widely-recognized real-world datasets: VOC (Everingham et al. 2010) and COCO (Ln et al. 2014). While our primary focus is lever- aging pretrained diffusion models for real-world scenarios, we include MOVi-E our evaluation due to its complexity, featuring scenes with up to 23 objects. This dataset serves challenging benchmark for object-centric learning in controlled environments. Both real-world datasets, VOC and COCO, have emerged as popular benchmarks for object discovery tasks. They present signif- icant challenges due to their multi-object nature and the large number of foreground classes they contain-22 20 and 80, respectively. These datasets have been instrumental in recent evaluations of
various object-centric learning methods on real-world images (Jiang et al. 2023
Wu et al. 2023b
Seitzer al. 2023).
Implementation Details: Following the previous works @iang et al. 2023
Wu et al.
2023b). we
convolutional backbone for MOVi-E and DINOv2 (Oquab et al. 2023) with VIT-B and a patch

---

Published as a conference paper at ICLR 2025
size of 14 as the encoder model for VOC and COCO. To serve decoder, incorporate pretrained Stable Diffusion (SD) model, v1.5 for MOVi-E; and COCO and v2.1 for VOC (Rombach et al. 2022), augmented with an additional cross -oos-attentionn layer adapter. For MOVi-E, jointly optimize the image backbone, slot attention mechanism, and adapter layers. For VOC and COCO, the slot attention and adapter layers. We maintain consistency with previous work (Jiang et al. 2023 Wu et al. 20233) in terms of dataset selection and preparation. Our models are trained for approximately 150K to 250K iterations. While this training duration is shorter compared to some previous works, we demonstrate that our approach achieves competitive performance.
focus our training exclusively
Table 1: Architectural ablations on MOVi-E. We examine the effects of architectural choices on segmentation and representation performance. We present block combinations the left and regis- ter token choice on the right. Up. Down and Mid refer to all upsampling blocks, all downsampling blocks and middle block in the diffusion model.
Conditioning Blocks
Register Token
Up- Only Up
Only Down
Up+Mid Up+Down
No Token
Stot Pooling
Feature Pooling
Segmentation (%)
FG-ARI
56.89
57.38
57,93
57.39
56.45
54.38
56.27
57.18
mBO (7)
39.59
43.05
40,20
39.96
43.38
40.07
41.65
43.98
mloU (1)
37.75
41.53
38.77
38.83
41.86
40.07
40.10
39.83
Representation
Category (1)
43.92
43.82
45.88
41.54
43,91
42.42
43.54
42.63
Position (L)
1.92
1.82
1.61
1.92
1.72
1.89
1.75
1.78
3D B-Box ())
3.94
3.78
3.48
3.95
3.75
3.83
3.77
3.78
We compare SlotAdapt with unsupervised state-of-the object-centric methods. On the MOVi-E dataset, we compare with SLATE (Singh et al. 2021). SLATE where SLATE's low capacity d'VAE is replaced by a pre-trained VQGAN model (Esser al. 202T and Latent Slot Dif-
Baselines:
fusion (LSD) (Jiag et al. 2023). On real-world datasets, we compare with Slot attention (SA) (Lo-
catello al. 20200, SLATE (Singh
al.
202Tp. DINOSAUR (Seitzer al. 2023). ) LSD JTang
et al. 2023) and SlotDiffusion (Wu et al.
2023b). In all our experiments on real-worke datasets, we
use frozen DINOv2 (Carnn et al. 202T Oquab et al.
2023)
the visual encoder for all models.
Metrics: Following previous work (Locatello et al.
2020 Jiang et al. 2023), we employ a set
of standard metrics assess the performance of our method on unsupervised object segmentation. These include the foreground adjusted rand index (FG-ARI), mean intersection over union (mloU), and mean best overlap (mBO). Our evaluation is performed on the slot attention masks, Aslat com- puted as in Equation g We use two different versions of the mloU and mBO metrics: one computed over instance-level masks, and the other over class-level masks. Note that instance-level metrics account for whether objects of the same class in image differentiated as separate instances in the resulting segmentation; hence, they are more informative on the representational and generative capabilities of an object-centric learning method. On the other hand, the FG-ARI, metric designed primarily for object discovery task, does not account for object masks larger than the ground-truth, which may be problematic, especially when assessing the generative capability.
GT
No Guidance
With Guidance
No Guidance
With Guidance
Figure 2: Qualitative comparison: with vs. without guidance. We visualize generated images and predicted segments on COCO dataset.
7

---

Published as a conference paper at ICLR 2025
Table Evaluation of guidance strategies. We present the segmentation performance on COCO for different guidance strategies. Joint guidance gives the best scores and significantly improves over no guidance option.
FG-ARI
mBO'
mBO
mioU
mioU
No guidance
42.3
31.5
34.8
31.7
38.5
Slot Guidance
41.2
33.4
36.9
33.1
37.9
DM Guidance
42.0
31.2
34.6
32.0
38.4
Joint Guidance
41.4
35.1
39.2
36.1
41.4
Multiplication Guidance
43.3
31.9
35.3
31.7
36.4
4.1 ABLATION STUDIES
To assess the impact of contributions, conduct series of experiments. All the experiments in this ablation study are conducted with register token (slot pooling), joint guidance of attention masks and conditioning through all downsampling and upsampling blocks, unless stated otherwise.
We first investigate the optimal block combination in the UNet architecture for conditioning the diffusion model on slots, where the options we consider all downsampling blocks, all upsampling blocks, mid-block and their certain combinations. The results obtained on MOVi-E dataset are presented in the left part of Table ! We observe that conditioning on either the upsampling blocks alone or both the downsampling and upsampling blocks yield superior performance, likely due to their proximity to the input and output in terms of structure and resolution.
Next, we evaluate the effect of incorporating register token DDacett et al. 2024) into the textual cross-attention layer in the diffusion model. The results obtained on MOVi-E are given in the right part of Table M where the options are register token, slot pooling and feature pooling. We observe that inclusion of register token, with slot or feature pooling, yields consistent improvements in all metrics, including segmentation accuracy and performance in downstream tasks.
Lastly, we examine the effectiveness of our guidance strategies on COCO dataset Table 2 We observe that joint guidance yields the best performance all metrics except the FG-ARI score, where the improvements are substantial compared to the no- guigguncce case. For FG-ARI metric, multiplication guidance gives the best performance, which is sufficient metric for object discovery but not necessarily for generative tasks. Moreover, in Fig. - we visually demonstrate the impact of the joint guidance strategy on the generated segmentation masks. We observe that the inclusion of guidance yields a significant improvement in segmentation quality, mitigating the part-whole hierarchy problem and producing segmentation masks better aligned with the objects in the scene, rather than with partial or fragmented objects. In turn, the generated (reconstructed) images are more faithful the original input images, as also observed in Fig. 2
4.2 EVALUATION RESULTS
the evaluation experiments for SlotAdapt are conducted with register token, joint guidance and conditioning on all downsampling and upsampling blocks, unless stated otherwise.
Figure 3: Unsupervised Object Segmentation. We show visualizations of segments on COCO (left) and VOC (right). SlotAdapt accurately binds distinct instances belonging to the same class.
Synthetic Dataset:
We evaluate the object discovery and segmentation performance of our method on MOVi-E in comparison to the baseline methods in Table 5] which is presented in Appendix, based on the FG-ARI, instance-level mBO and instance-level mloU metrics. We observe significant
8

---

Published as a conference paper at ICLR 2025
Table 3: Model Evaluation Comparison of FID and KID scores for reconstruction (left) and com- positional generation (right) across different methods.
Method
FID
KIDx 1000
Method
FID
KIDx 1000
LSD 35.537
19.086
LSD
167.232
103,482
SlotDiffusion
19.448
5.852
SlotDiffusion
64.213
57.309
Ours
10.857
0.388
Ours
40.568
34.381
improvements in almost all metrics, particularly with approximate 10% enhancement in object discovery and segmentation accuracy.
Real-World Dataset: We evaluate the object discovery and segmentation performance of our method on the COCO and VOC datasets in Table A using the FG-ARI, instance-level mBO and class-level mBO metrics, in comparison to baseline methods including StableLSD (Jiang et al. 2023), SlotDiffusion (Wu et al. 2023b), and DINOSAUR (Seitzer al. 2023).
We observe that SlotAdapt, when used with guidance, outperforms all the baselines for almost all metries, including the highly competitive DINOSAUR method. The only exception the class-level mBO score VOC, where SlotAdapt performs worse than SlotDiffusion. Notably, the improve- ment in mBO scores on COCO particularly significant, about 10% better than the next best base- line. The impact of guidance is also substantial COCO, which is more challenging and much larger dataset with complex multi-object scenes and varying object sizes, when compared to VOC.
In Fig 4 we visually compare the segmentation results of LSD, SlotDiffusion and SlotAdapt COCO. We observe that SlotAdapt successfully differentiates individual object instances as evi- denced by its superior instance-level mBO score, whereas LSD and Slot Diffusion struggle with this challenge. Moreover, SlotAdapt produces more complete segmentations of objects without dividing them into parts, which reflected by its higher FG-ARI score. These results highlight the robustness and versatility of SlotAdapt in handling the complexities of real-world data.
Generation and Compositional Editing: We first demonstrate the ability of our model gen- erate realistic images on COCO in Fig. [ We see that, when conditioned on slots, our model can reconstruct the input image with high quality, realism and notable fidelity. Regarding composi- tional generation and editing capabilities, Figure shows series of image edits by modifying input slots, including object replacement, removal and addition. We observe that the editing operations highly successful and scamless with only slight yet realistic changes the image background, while all maintaining high quality. To assess our model's capabilities, we conducted extensive ex- periments evaluating both its reconstruction fidelity and compositional generation performance. The comprehensive results are presented Table 3 For additional qualitative and quantitative results, refer readers to Section A.3 in the appendix.
Figure 4: Qualitative comparisons with other methods on COCO. We visualize predicted seg- ments SlotAdapt in comparison LSD and SlotDiffusion. SlotAdapt can more effectively dif- ferentiate between object instances of the same class compared to other methods.
Ours
Figure 5: Generation Results. We show sample images reconstructed by SlotAdapt on COCO (left) and VOC (right). SlotAdapt generates reconstructions highly faithful to the original input images.

---

Published as a conference paper at ICLR 2025
Object Removal
Object Replacement
Object Addition
STOP
Figure 6: Compositional Editing. We demonstrate object removal, replacement and addition edits on COCO images by using slots. Removing highlighted slots (top row) yields realistic and successful generations (first 4 examples). Replacing highlighted objects in the 3rd and 4th images with the cow object from the 5th and 6th images results in highly accurate edits, yet with small changes in the original images. Finally, adding the cow (5th image) and the person (3rd image) slots to the last two images, respectively, generates meaningful examples of complex scenes.
Table 4: Unsupervised object segmentation on real-world datasets. We compare SlotAdapt with state-of-the-ar methods VOC (left) and COCO (right). We present versions of our method, with and without guidance loss.
VOC
FG-ARI
mBO'
mBO
COCO
FG-ARI
mBO
mBO
SA + DINO ViT
12.3
24.6
24.9
SA + DINO ViT
21.4
17.2
19.2
SLATE + DINO VIT
15.6
35.9
41.5
SLATE + DINO ViT
32.5
29.1
33.6
DINOSAUR
23.2
43.6
50.8
DINOSAUR
34.3
32.3
38.8
LSD
18.7
40.5
43.5
LSD
33.8
27.0
30.5
SlotDiffusion
17.8
50.4
55.3
SloiDiffusion
37.2
31.0
35.0
Ours
28.8
51.6
52.0
Ours
42.3
31.5
34.8
Ours Guidance
29.6
51.5
51.9
Ours Guidance
41.4
35.1
39.2
5 CONCLUSION
We have targeted the object-centric learning problem, particularly on complex real-world images. To this end, we have presented a method that combines slot attention with pretrained diffusion mod- els. Our architecture, AlotAdapt, has three main novelties; adapters in the diffusion model for slot conditioning, the use of a register token to represent background in images, and attention guidance to align slot attention with cross attention from the diffusion model. We have conducted extensive experiments, particularly on COCO dataset, to validate our approach. Our experiments show that leveraging the generative capabilities of pretrained diffusion models is erucial for tackling the chal- lenging tasks such as object discovery, unsupervised segmentation and compositional generation and editing on complex real-world images. Our method takes step forward in achieving this goal, outperforming the state of the art methods on the challenging COCO dataset without relying on any external supervision in object discovery, segmentation and, particularly compositional generation and editing. We are the first to present successful compositional editing experiments on the COCO dataset.
A limitation of our method from the perspective of compositional generation is that the edited or re- constructed images, when conditioned on slots, may exhibit slight changes with respect the source image, though the generations are mostly highly realistic and of very good quality. One remedy for this can be the incorporation
additional training objectives to enforce fidelity to the input. An- other related issue is how to make use of this framework for image editing in practice, which needs further adjustments to the architecture possibly to accept user prompts and associate them with slot representations. Lastly, method still has issues for under- and over-segmentation, which may potentially be mitigated through slot merging and splitting. Additionally, our approach relies on pre-trained diffusion models that are predominantly trained real-world data, which may limit their adaptability to synthetic domains. While the model performs well on natural images, its effec- tiveness may be reduced when handling synthetic datasets such CLEVR and CLEVRTE CLLEVRTexJohnso et al. (2017).
Ack Acknowledgements: We thank Aykut Erdem and Erkut Erdem for their insightful discussions.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Adil Kaan Akan and Fatma Güney. Stretchbev: Stretching future instance prediction spatially and temporally. In European Conference on Computer Vision, pp. 444 460. Springer, 2022.
Adil Kaan Akan, Erkut Erdem, Aykut Erdem, and Fatma Güney. Slamp: Stochastic latent ap- pearance and motion prediction. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 14728-14737, 2021.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can be learned? In Proc. of the International Conf. Learning Representations (ICLR), 2019.
Burak Can Biner, Farrin Marouf Sofian, Umur Berkay Karakaş, Duygu Ceylan, Erkut Erdem, and Aykut Erdem. Sonicdiffusion: Audio-driven image generation and editing with pretrained diffu- sion models. arXiv preprint S.Ki::2005000875 2024.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 1.Kiv/8001.00 01111,101110000 2018.
Léon Bottou. From machine learning to machine reasoning. Machine Learning, 94(2):133-149, 2014.
Chris Burgess and Hyunjik Kim. 3D shapes dataset. http://ggiiiooooonddeepm/////hhs 2018.
Christopher Burgess, Loic Matthey Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representa- tion. arXiv preprint 19X1v:9001111300 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In Proc. of the International Conf. on Learning Representations (ICLR), 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8780-8794, 2021.
Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Cur- tis Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real- world videos. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 12873- 12883, 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams. John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision (IJCV), 2010.
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: critical analy- sis. Cognition, 28(1-2)33-71, 1988.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. Proc. of the International Conf. on Machine Leam- ing (ICMI.), 2019.
11

---

Published as a conference paper at ICLR 2025
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, al. Kubric: A scalable dataset generator. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 3749- 3761, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Geoffrey Hinton. Some demonstrations the effects of structural descriptions in mental imagery. Cognitive Science, pp. 231,550,1 1979.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshops, 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv: 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffuusin models for high fidelity image generation. Journal of Machine Learning Research (JMLR), 23:47:1-47:33. 2022.
Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimiza- tion. In Proc. of the International Conf. on Learning Representations (ICLR), 2023.
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. Advances in Neural Informa- tion Processing Systems, 33:12522-12582, 2020.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. SCALOR: Generative world models with scalable object representations. In Proc. of the International Conf. on Learning Representations (ICLR), 2019.
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. In Ad- vances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 3563-8601, 2023.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition ( (CVPR), 2017.
Whie Jung, Jaehoon Yoo, Sungjin Ahn, and Seunghoon Hong. Learning to compose: Improving ob- ject centric learning by injecting compositionality. In Proc. the International Conf. on Learning Representations (ICLR), 2024.
Daniel Kahneman, Anne Treisman, and Brian J. Gibbs. The reviewing of object files: Object- specific integration of information. Cognitive Psychology, pp. 175-219, 1992.
Ioannis Kakogeorgiou, Spyros Gidaris, Konstantinos Karantzalos, and Nikos Komodakis. Spot: Self-training with patch-order permutation for object-centric learning with autoregressive trans- formers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 22776-22786, June 2024.
Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In Proc. of the International Conf. on Learning Representations (ICLR), 2021.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems (NeurIPS). 2018.
12

---

Published as a conference paper at ICLR 2025
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects context. In Proc. of the European Conf. Computer Vision (ECCV), pp. 740-755. Springer, 2014.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. In Proc. of the International Conf. on Learning Representations (ICLR), 2019.
Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving gener- ative imagination in object-centric world models. In Proc. of the International Conf. on Machine Learning (ICML), 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. In Advances in Neural Information Processing Systems (NeurIPS), pp. 11525-15538 2020.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In Proc. of the International Conf. on Learning Representations (ICLR), 2021.
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proc. of the Conf. on Artificial Intelligence (AAAI), volume 38, pp. 2296-43044 2024.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proc. of the International Conf. on Machine Learn- ing (ICML), 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learn- ing robust visual features without supervision. In Transactions on Machine Learning Research (TMLR), 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), pp. 1-14, 2022b.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. IEEE Trans. on Patlern Analysis and Machine Intelligence (PAMI), 109(5))612-634, 2021.
Maximilian Seitzer, Max Hom, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. In Proc. of the International Conf. on Learning Representations (ICLR), 2023.
13

---

Published as a conference paper at ICLR 2025
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In Proc. of the International Conf. on Learning Representations (ICLR), 2021.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for com- plex and naturalistic videos. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Krishnakant Singh, Simone Schaub-Meyer, and Stefan Roth. Guided latent slot diffusion for object- centric learning. arXiv preprint arXiv:2407.17999 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proc. of the International Conf. Machine Learning (ICML), 2015.
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental Science, 2007.
Yanbo Wang, Letao Liu, and Justin Dauwels. Slot-vae: Object-centric scene generation with slot attention. In International Conference on Machine Learning, pp. 36020-36035. PMLR, 2023.
Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in VAEs. arXiv preprint 1xXiv:9011.0011, 2019.
Yi-Fu Wu, Minseuug Lee, and Sungjin Ahn. Structured world modeling via semantic vector quan- tization. Proc. of the International Conf. on Learning Representations (ICLR), 2024.
Ziyi Wu, Nikita Dvoriik, Klaus Greff, Thomas Kipf, and Animesh Garg. SlotFormer: Unsupervised visual dynamics simulation with object-centric models. In Proc. of the International Conf. on Learning Representations (ICLR), 2023a.
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object- centric generative modeling with diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), pp. 50932-50958, 2023b.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint w:Xiv:2308.06722 2023.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Mad- havan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2020.
14


Reviews:
[{"review_id": "xAo55Fehl3", "reviewer": "Reviewer_uBXM", "summary": "This paper aims to incorporate object-centric information into the pre-trained diffusion models. Specifically, they argue that previous works have struggled with text-conditioning bias even though they can leverage the potential of pre-trained diffusion models. To eliminate the text-conditioning-related bias, the authors propose SlotAdapt, which utilizes the adapter layers for the condition with slot, to enable the incorporation of diverse slot conditions. For their design choice, T2I adapter design and register tokens-like approaches are introduced. Then, they also use cross-attention masks for better conditioning, which enables a dual path of guidance for the diffusion models and masks. Through their experiments, they present various benchmarks and comparisons with previous object-centric methods to validate the effectiveness of their method.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The major strength is the performance of the proposed method. In their benchmark, their method significantly outperforms the baseline.\n- The paper is well-written and easy to follow.", "weaknesses": "- W1: A significant concern with the proposed method is its novelty. While the object-centric approach may offer a fresh perspective within the realm of object-centric learning, the use of adapter layers is a well-established technique in diffusion fields, widely known to be effective. This familiarity may make the method appear somewhat trivial or lackluster in terms of innovation for diffusion model applications.\n\n- W2: Additionally, the paper lacks an ablation study, which is critical for understanding the impact of the guidance loss on the model’s performance. Specifically, it would be insightful to see how the model behaves without the guidance loss, as this could further clarify the contribution of each component in the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2LtuJkssPM", "reviewer": "Reviewer_Hqxz", "summary": "The paper introduces SlotAdapt, an object-centric learning method that integrates slot attention with pre-trained diffusion models using adapter layers for slot-based conditioning. This work can significantly avoid biases generated from the text-centric conditioning models. The proposed adapter layers with slot attention can enhance the model's alignment with image objects without external supervision.", "questions": "How can the authors demonstrate that the unique contributions of their model extend beyond the use of adapters, which are becoming increasingly common in the LDM？\nThe paper's outcomes appear heavily reliant on segmentation quality. Could the authors clarify if the model's success is primarily due to the performance of segmentation, and whether integrating a superior segmentation method could achieve similar results?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Introducing adapter layers for slot-based conditioning enables the model to deviate from text embedding space representations while preserving the generative capabilities of pre-trained diffusion models.\n\nThe compositional image generation results on complex real-world images demonstrate the effectiveness of the model in handling compositional generation tasks.", "weaknesses": "The paper inconsistently uses both \"UNet\" and \"Unet\" to refer to the U-Net architecture. It is advised that the authors adhere to the standard capitalization \"UNet\" consistently across the document. Additionally, it is recommended to introduce the term \"register token\" earlier in the paper to establish its significance and reducing confusion. It is crucial for the authors to perform a thorough review and correct grammar and syntax problems.\nThe reconstructed images may exhibit slight changes or artifacts comparing to the source image. This issue might be attributed to over-segmentation, which could lead to the misrepresentation of certain image components during the generation process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6PmEBTJMbU", "reviewer": "Reviewer_8w5Y", "summary": "This paper introduces an object-centric learning approach, SlotAdapt, with the focus on scaling to real-world dataset. Instead of directly frozening or fine-tuning the pretrained stable diffusion model, SlotAdapt adds an adapter module on top of pretrained stable diffusion so that the learned slots are not strictly limited to lie in the text embedding space, enabling better slot-object alignment for real world images. The resulted flexibility of slots is traded with an additional module, Adapter, needing to be trained when compared with the frozen pretrained stable diffusion case. SlotAdapt also self-supervises slot attention maps with cross-attention maps for semantic alignment. Experiment results show improvements on the segmentation performance.\n\nThe main contribution of this paper comes from the architecture design.", "questions": "1. Does the number of slots used in the experiments have an impact on the part-whole results? For COCO dataset, do you simply assume the slot number is 80? In that case, is the training slow?\n\n2. Looks like the quantitative scores of baselines in Table 3 are directly copied from LSD paper. In that case, it should be guaranteed that the experimental settings of yours and that of LSD paper are identical for fair comparison.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The writing is smooth and easy to follow\n\n2. The paper is tackling an important problem in the object-centric learning community, i.e., pushing the boundary of object-centric learning approaches towards real world images, and shows some non-trival improvements on the segmentation performance.\n\n3. The proposed approach is straightforward and effective by borrowing and combining ideas from LSD and T2I adapters without requiring additional supervision. It largely benefits from the prior knowledge stored in pretrained stable diffusion models while allows flexible representation learning ability of slots.", "weaknesses": "1. This paper claims, \"Our method performs remarkably well on complex real-world images for compositional generation, in contrast to other slot-based generative methods in the literature.\" However, only qualitative results are provided, even without a comparison with existing approaches. It should be noted that existing approaches like LSD [1] and Slate [5] can already perform compositional generation on real world datasets like FFHQ in Figures 3, 4, and 10 https://arxiv.org/pdf/2303.10834. Comparing your approach with LSD and/or other approaches on Movi-E or COCO is necessary to support your claim. Otherwise, the sentence should be revised. Similarly, quantitative scores like FID are helpful to demonstrate the statements on high fidelity generation in the section \"Generation and Compositional Editing\".\n\n2. For the guidance strategy, the FG-ARI in Table 2 shows that guidance doesn't help a lot for segmentation, while Fig 2 qualitatively illustrates that guidance might help a bit. Are the qualitative results of Fig 2 possibly due to random seeds used in the experiments? Is there any intuition that guidance can help avoid over-splitting? After all, there is no external supervision, it is possible that both slot attention map and cross attention map oversplit. Furthermore, in section 4.1, it says \"In turn, the generated (reconstructed) images are more faithful to the original input images\", which is not that persuasive depending on what you focus on (the clothes of base ball player, the closets and ceiling lamp).\n\n\n3. There are several overclaims in the paper. For example, in Figure 5, it says \"SlotAdapt generates nearly perfect reconstructions\" while the figure shows that details are still largely different between GT and reconstruction. In Abstract and Conclusion, it claims outperforming baselines in compositional generation; however, there is no qualitative and quantitative comparison in any section in compositional generation.\n\n4. Now that the paper is titled with Compositional Generation, some relevant reference, such as [2][3][4] etc., are suggested to be discussed in related works though not directly compared.\n\n5. Limitation discussion. (1)  LSD paper finds that their approach shows significant gains in complex naturalistic scenes but a somewhat diminished performance in terms of segmentation and representation when the dataset consists of only visually simple and monotonous scene images like CLEVR. Does SlotAdapt share similar limitation when it comes to simple scenes like CLEVR or CLEVRTex? (2) Although adopting pretrained stable diffusion models allows using prior knowledge to improve segmentation performance on real-world images like COCO,  it on the other hand might limit the model's ability to discover per-dataset components, for example facial slots in LSD [1].\n\n\n[1] Jiang, \"Object-Centric Slot Diffusion\", NeurIPS 2023\n\n[2] Jiang, \" Generative neurosymbolic machines\", NeurIPS 2020\n\n[3] Wang, \"Slot-VAE: Object-Centric Scene Generation with Slot Attention\", ICML 2023\n\n[4] Wu, \"NEURAL LANGUAGE OF THOUGHT MODELS\", ICLR 2024\n\n[5 Singh, \"Illiterate DALL-E Learns to Compose\", ICLR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6zz1x7M8ea", "reviewer": "Reviewer_qbBW", "summary": "This paper proposes SlotAdapt, a novel approach to leverage large-scale pre-trained diffusion models (DMs) for the task of object-centric learning (OCL). There are priors work trying to combine OCL with DMs, yet they usually just treat slots as text embeddings, and directly input them to the cross-attention layer in the diffusion U-Net. SlotAdapt instead inserts a new adapter layer to the U-Net to condition on slots. With this design, slots do not need to lie in the text embedding space, leading to better object-centric representations. Authors conduct extensive experiments including both segmentation and generation on real-world datasets COCO and VOC. The results are impressive and the generation result is clearly better than previous works.", "questions": "1. What will happen if you add the guidance loss from the beginning of training? Will the training diverge?\n2. Why using SD 1.5 on COCO and SD 2.1 on VOC? In SlotDiffusion Appeidix E.5, they do mention that using the v-prediction objective sometimes leads to degenerate results. Is it also the case here as SD 2.1 is using v-prediction?\n3. Have you tried classifier-free guidance (CFG)? I am curious if using a larger CFG scale, will the reconstruction result be even better.\n4. Instead of pooling all slots to form the register token, I guess an interesting alternative is 1) add an additional slot in your slot attention module, and 2) only input this new slot to the cross-attention layer. This slot is dedicated to capture global information and may perform better than pooling object slots.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The adapter design is reasonable, and the motivation behind it is clear -- slots capture object semantic information, which might be incompatible with the text embedding space. With this new adapter layer, one can freeze the pre-trained DM while not sacrificing the expressiveness of learned object slots.\n- The guidance loss is novel. While there are concurrent work (e.g. GLASS) applying a similar loss, they usually require external supervisions such as captioner (to get text from input images). I am surprised that just letting slot attention mask and cross-attention mask to guide each other can work so well (I would expect some stop gradient operation to stabilize training). The ablation studies in Table 2 are comprehensive and answer my concerns very well.\n- As far as I know, SlotAdapt is the first work showing compositional generation ability on real-world images (prior work only show segmentation results). I find the results in Fig.6 impressive.", "weaknesses": "I do not find any big weakness of this paper. There is one claim that sounds not precise to me and I hope the author can revise it.\n- In line 468 and Fig.5 caption, authors claim that SlotAdapt \"generates nearly perfect reconstructions\". I do not think this is the case, as apparently the reconstructed images are clearly different from the input image. This result itself is not a weakness of the method, as slots are compressed representations and cannot preserve all visual details. However, I do believe the statement itself is wrong.\n\nMinor:\n- Please unify the name of U-Net in the paper. In Fig.1 and line 230 it is called \"Unet\", in line 207 \"U-Net\", in line 223 and 224 \"UNet\".\n- Line 320 \"v1.5 for MOVi-E; and COCO\", I think this \";\" is redundant.\n- The generation results of SlotAdapt are impressive, yet authors only put small figures in the main paper (Fig. 5 and 6). I understand this is due to page limit. Please add text to refer readers to large figures in Appendix Fig. 8 and 10.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)