SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Concurrent Research. A concurrent study in [85] also explores a similar idea of object-centric learning using DMs and its downstream applications such as the image editing tasks akin to our experiments. Just like LSD, they employ the Latent Diffusion Model (LDM) as the slot decoder. However, unlike our method, [85] independently train a latent encoder for each dataset, whereas LSD employs a pre-trained latent encoder shared across all input data.
We extensively evaluate our proposed Latent Slot Diffusion (LSD) model on various object-centric tasks, including unsupervised object segmentation, downstream property prediction, compositional generation, and image editing. As will be shown, our model demonstrates substantial improvements over the state-of-the-art on multiple challenging datasets with complex texture and background, including FFHQ [41] which has been beyond the generative capability of object-centric models. In addition, we also include a preliminary exploration into the potential of LSD using pre-trained diffusion models, discussing its performance in both multi-object datasets and unconstrained real-world images in Section 5.4 and Appendix C.
Datasets. We evaluate our model on five datasets. Four of them are synthetic multi-object datasets—CLEVR [37], CLEVRTex [40], MOVi-C, MOVi-E [24]. They present increasing levels of difficulty—CLEVRTex adds texture to objects and backgrounds, MOVi-C uses more complex objects and natural backgrounds, and MOVi-E contains large numbers of objects (up to 23) per scene. Furthermore, we explore the applicability of object-centric models to FFHQ [41], a dataset of high-quality face images. Unlike previous works in this line that only investigate low-resolution images, e.g., up to  $128 \times 128$ , we use a resolution of  $256 \times 256$  for all datasets in our experiments.
Baselines. We compare our model against SLATE, the state-of-the-art object-centric learning and unsupervised compositional image generation approach. We use its improved version [73], which is more robust in complex scenes. For a fair comparison with LSD that leverages pre-trained auto-encoder, we also develop a variant of SLATE denoted as  $\mathrm{SLATE}^+$ , where its low-capacity dVAE [70] is replaced with a pre-trained VQGAN model [21]. For all models in this work, we use OpenImages-pretrained auto-encoders [63].
5.1 Object-Centric Representation Learning
In line with previous work [53], we use unsupervised object segmentation and downstream property prediction to evaluate the object-centric learning capability and representation quality. Within each dataset, all models we compare use the same number of slots.
Table 1: Segmentation Performance and Representation Quality. Left: We evaluate the segmentation quality and report mBO, mIoU, and FG-ARI scores across various datasets and baselines. Right: We measure the representation quality by learning a probe to predict object properties given frozen slots. For position and 3D bounding box, we report MSE. For shape, material, and category, we report accuracy. The results are averaged over three random seeds, and the complete table, including standard deviations, can be found in Figure 5 in appendix.
(a) CLEVRTex
Segmentation	SLATE	SLATE+	LSD (Ours)	Representation	SLATE	SLATE+	LSD (Ours)
mBO (↑)	50.88	54.90	63.93	Shape (↑)	74.24	71.63	80.23
mIoU (↑)	49.54	52.96	62.52	Material (↑)	69.73	63.61	75.56
FG-ARI (↑)	44.19	70.71	64.41	Position (↓)	1.28	1.26	1.13
(b) MOVi-C
Segmentation	SLATE	SLATE+	LSD (Ours)	Representation	SLATE	SLATE+	LSD (Ours)
mBO (↑)	39.37	38.17	45.57	Position (↓)	1.37	1.28	1.14
mIoU (↑)	37.75	36.44	44.19	3D B-Box (↓)	1.48	1.44	1.44
FG-ARI (↑)	49.54	52.04	51.98	Category (↑)	42.45	45.32	46.11
(c) MOVi-E
Segmentation	SLATE	SLATE+	LSD (Ours)	Representation	SLATE	SLATE+	LSD (Ours)
mBO (↑)	30.17	22.17	38.96	Position (↓)	2.09	2.15	1.85
mIoU (↑)	28.59	20.63	37.64	3D B-Box (↓)	3.36	3.37	2.94
FG-ARI (↑)	46.06	45.25	52.17	Category (↑)	38.93	38.00	42.96
Unsupervised Object Segmentation. Following [53, 69], to measure segmentation quality, we report the foreground adjusted rand index (FG-ARI), the mean intersection over union (mIoU), and the mean best overlap (mBO). These metrics are computed based on the attention masks generated by Slot Attention. Specifically, we utilize the attention weights described in Section 2.1, where attention weights  $\mathbf{A}$  are computed as  $\mathbf{A} = \operatorname*{softmax}_{N}\left(\frac{q(\mathbf{S})\cdot k(\mathbf{E})^{T}}{\sqrt{D}}\right)$ .
Our results in Table 1 suggest that LSD is particularly strong in visually complex scenes. For example, on the most challenging MOVi-E dataset, LSD outperforms the strongest baseline by  $>8\%$  in mBO,  $>9\%$  in mIoU, and  $>6\%$  in FG-ARI. On CLEVRTex and MOVi-C featuring complex textures, LSD also achieves  $>9\%$  and  $>6\%$  gain respectively, in both mBO and mIoU. We visually show the superior segmentation quality of LSD in Figure 2. LSD obtains tighter object boundaries, less object splitting, and cleaner background segmentation. The advantages are most noticeable on CLEVRTex and MOVi-E, where the baselines over-segment the objects more frequently, or exhibit a common failure mode that divides images into approximately uniformly distributed block masks.
We also note that the FG-ARI score is sometimes not an ideal metric for evaluating segmentation quality, because it only considers the foreground pixels and disregards the correctness of the mask shape, as also highlighted by [19, 40]. In the MOVi-E experiment, even though  $\mathrm{SLATE^{+}}$  partitions the images arbitrarily into uniform patches, as shown in Figure 2, it still achieves a FG-ARI score comparable to SLATE, which produces significantly more reasonable object masks. This highlights the need for additional metrics, such as the mIoU score that we also measure for evaluating the segmentation quality in this work.
Downstream Property Prediction. Following [15, 53], we evaluate the quality of the learned object-centric representations through downstream property prediction. Specifically, for each property, we train a network to predict the property given a frozen slot representation as input. The correspondence between the slot and its true label is determined by Hungarian matching [44] using masks. We use linear heads and 2-layer MLP as the prediction networks for discrete and continuous values, respectively. We report prediction accuracy for discrete properties (e.g., shape and material), and mean squared error for continuous properties (e.g., position).
As shown in Table 1, LSD is competitive with or better than the strongest baseline on CLEVRTex, MOVi-C, and MOVi-E, which is consistent with our finding in unsupervised object segmentation that
Table 2: Compositional Image Generation. On various datasets, we generate images using object slots randomly sampled from the dataset. We compare the fidelity of the generated images via the FID score. The lower the FID score, the better the fidelity. We find that our model produces the best fidelity scores compared to the baselines.
FID ↓	SLATE	SLATE+	LSD (Ours)
CLEVRTex	105.83	69.23	29.53
MOVi-C	170.83	148.27	69.12
MOVi-E	169.32	126.51	64.76
FFHQ	112.38	98.76	27.83
Figure 3: Compositional Generation Samples. We qualitatively compare the quality of image samples generated by our model with the baselines. We observe that LSD provides significantly higher fidelity and more clear details compared to the other methods.
LSD shines in visually complex scenes. For example, LSD obtains  $>4\%$  gain in predicting object category on MOVi-E and object material on CLEVRTex.
5.2 Compositional Generation with Visual Concept Library
Like text-to-image generative models, LSD is able to take unseen slot-based prompts at test time and compose new images. Unlike text-to-image models, however, LSD obtains the compositional generation ability solely from images without relying on additional supervision like text inputs. As described in Section 3, we first build a concept library. Then, we sample one slot representation from each visual concept library and concatenate them into a sequence to form a slot-based prompt. We then feed the slot-based prompts to the LSD decoder to generate the images. This produces scenes with novel object layouts and faces with unseen attribute combinations.
We report in Table 2 the FID score [29] as a measure of the compositional generation quality. Following standard practice [14], we compute the FID score using 2K generated images and the full training dataset. Across all datasets, LSD achieves significantly better FID scores than SLATE and  $\mathrm{SLATE^{+}}$ . We further demonstrate the superior compositional generation quality of LSD in Figure 3. We observe that on the more challenging MOVi-E and FFHQ datasets, LSD generates images with substantially more clear details and better coherence. In contrast, SLATE is limited by its decoder and produces images with missing details in the objects or human faces. While the details can be improved by the pre-trained auto-encoder in  $\mathrm{SLATE^{+}}$ , some samples still exhibit severe distortions.
5.3 Slot-Based Image Editing
In addition to generating new images from randomly sampled slot-based prompts, LSD also allows editing existing images by directly modifying their slot representations. Our experiments showcase LSD's capability in slot-based image editing, including object removal, single-object segmentation, object insertion, background extraction, and background swapping. Remarkably, we demonstrate, for the first time in object-centric generative models, the ability to perform face editing in real-world images, as illustrated in Figure 4.
Figure 4: Slot-Based Image Editing. Left: Our model demonstrates image editing capabilities, including object removal, extraction, insertion, background extraction, and swapping. Right: In the FFHQ dataset, we showcase face replacement by combining face slots from Source-B images with hairstyle, clothing, and background slots from Source-A images.
We perform slot manipulation on the CLEVRTex dataset, including object removal, single-object extraction, object insertion, background extraction, and background swapping. To remove an object or extract the background, we discard the relevant slot or all object slots, respectively. Our findings indicate that with near-perfect object decomposition, removing an object can be achieved by simply discarding its slot. Additionally, the background image can be rendered from a single background slot, even without single-slot conditioning during training. In single-object extraction, we render an object with the same background using the respective object and background slots. For object insertion and background swapping, we split the image into top and bottom pairs (Figure 4 left) and introduce a slot from another image or swap background slots. This demonstrates that an image's background can be entirely altered while preserving its original objects.
We further explore face replacement on the FFHQ dataset. LSD decomposes each image into four slots, corresponding to face, hairstyle, clothing, and background. Our results show that by replacing the face slots of the images, we are able to coherently change the image while maintaining the hairstyle, clothing, and background. The resulting images look realistic, suggesting that LSD can effectively blend various attributes even when given novel combinations.
Figure 5: Unsupervised Object-Centric Learning and Conditional Generation on Real-World Images with Pre-Trained Diffusion Models. Each image sample is conditioned on the same set of learned slots and with a different initial noise map in the denoising process. We use  $\mathrm{cfg} = 1.3$  for all samples. Additional details are available in Appendix C.
5.4 Pre-Trained Diffusion Models and Real-World Object-Centric Learning.
In recent studies of DMs, much attention has been given to controllable image generation using large-scale pre-trained diffusion models such as Stable Diffusion [47, 57, 88]. One intriguing question is whether such pre-trained diffusion models can facilitate the learning of object-centric representation. In this regard, we present a preliminary investigation in Appendix C, where we test two LSD variants utilizing the pre-trained Stable Diffusion. Our findings indicate that (1) although the pre-trained Stable Diffusion model has not been trained on any multi-object datasets, it still provides a learning signal for reasonable object-centric learning, (2) with the strong generative capabilities of pre-trained DMs, LSD can scale up object-centric learning to real-world images; however, we also note the part-whole ambiguity issue in slot-based object learning in real-world scenarios, and (3) remarkably, the propose LSD variant achieves conditional generation with unconstrained real-world objects by applying classifier-free guidance [33] to the learned slot, which also provide a visualization of the semantic information captured in the learned representations. We show some image samples in Figure 5, please refer to Appendix C for additional details. We believe these preliminary attempts to be valuable explorations toward large-scale object-centric learning and hope our finding will inspire future research in this area.
LSD on Simple Images. While LSD shows significant gains in complex naturalistic scenes, our preliminary investigation suggests a somewhat diminished performance in terms of segmentation and representation when the dataset consists of only visually simple and monotonous scene images like CLEVR. We conjecture that LSD may suffer from an overfitting problem when its high expressiveness encounters visually simple images. To address this issue, additional complexities can be introduced during training, such as integrating images from other datasets. In Appendix B, we offer a more detailed analysis of this challenge using the CLEVR dataset and introduce a training scheme that includes data samples from CLEVRTex to alleviate the problem. Our findings indicate that by integrating greater complexities into the training set, LSD can be effectively employed on simple datasets as well.
Computation Requirement. We compare the computation requirement of LSD and the baseline models in the CLEVtex setting. LSD requires approximately 32 GB of training memory, compared to 31 GB and 36 GB for SLATE and  $\mathrm{SLATE^{+}}$ , respectively. Notably, LSD does not require much higher memory consumption than regular unconditional latent diffusion, which takes 28 GB under the same setting. We train LSD on 2 NVIDIA RTX 6000 GPUs for 4.5 days, while SLATE and  $\mathrm{SLATE^{+}}$  are trained in 1 day and 2.7 days using the same GPU setup. For test-time generation, LSD takes 50.7 seconds to generate 50 images, while SLATE and  $\mathrm{SLATE^{+}}$  takes 86.1 and 87.6 seconds, respectively.
The integration of diffusion models into unsupervised object-centric learning has been largely unexplored, but holds significant potential. To assess the feasibility of this approach and to identify its strengths and weaknesses, we introduced the Latent Slot Diffusion model in this study, which serves two purposes: (1) the first model integrating diffusion models into unsupervised object-centric learning, and also (2) the first unsupervised compositional diffusion model which does not require supervised annotations like text. Our key findings indicate that the proposed model surpasses state-of-the-art transformer-based object-centric models in multiple tasks, with its performance advantage increasing as image complexity grows. Furthermore, the unsupervised compositional generation quality of our model exceeds that of transformer-based counterparts. Finally, we also introduce LSD variants that utilize pre-train Diffusion Models and demonstrate their ability in real-world object-centric learning and conditional generation with real-world objects. We believe that this work represents a significant advancement in object-centric learning towards dealing with complex naturalistic images.
This work is supported by Young Researcher Program (No. 2022R1C1C1009443) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT. The GPU computing used for this work is partly supported by KI Cloud of Division of National Supercomputing Center, Korea Institute of Science and Technology Information (KISTI). We would like to thank Junmo Cho for managing of specific experiments, Yi-Fu Wu for his contributions to the initial draft of this paper, and Ligong Han and Yuxiao Chen for their valuable feedback on the manuscript.
[1] Titas Anciukevicius, Christoph H Lampert, and Paul Henderson. Object-centric image generation with factored depths, locations, and appearances. arXiv preprint arXiv:2004.00642, 2020.
[2] Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: what is required and can it be learned? arXiv preprint arXiv:1811.12889, 2018.
[3] Federico Baldassarre and Hossein Azizpour. Towards self-supervised learning of global and object-centric representations. In ICLR 2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
[4] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In International Conference on Learning Representations, 2022.
[5] Léon Bottou. From machine learning to machine reasoning. Machine learning, 94(2):133-149, 2014.
[6] Christopher P Burgess, Loic Matthew, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390, 2019.
[7] Michael Chang, Alyssa L. Dayan, Franziska Meier, Thomas L. Griffiths, Sergey Levine, and Amy Zhang. Hierarchical abstraction for combinatorial generalization in object rearrangement. In NeurIPS 2022 Workshop on All Things Attention: Bridging Different Perspectives on Attention, 2022.
[8] Michael Chang, Thomas L Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. arXiv preprint arXiv:2207.00787, 2022.
[9] Chang Chen, Fei Deng, and Sungjin Ahn. ROOTS: Object-centric representation and rendering of 3D scenes. Journal of Machine Learning Research, 22(259):1-36, 2021.
[10] Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances In Neural Information Processing Systems, 2018.
[11] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pages 2172-2180, 2016.
[12] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of AAAI, 2019.
[13] Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In International Conference on Learning Representations, 2020.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[82] Xin Wen, Bingchen Zhao, Anlin Zheng, X. Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. arXiv preprint arXiv:2205.15288, 2022.
[83] Yizhe Wu, Oiwi Parker Jones, and Ingmar Posner. Obpose: Leveraging canonical pose for object-centric scene inference in 3d. arXiv preprint arXiv:2206.03591, 2022.
[84] Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05861, 2022.
[85] Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Objectcentric generative modeling with diffusion models. arXiv preprint arXiv:2305.11281, 2023.
[86] Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object-centric representations for reinforcement learning. arXiv preprint arXiv:2302.04419, 2023.
[87] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real-world videos by predicting temporal feature similarities. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023), 2023.
[88] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[89] Ruixiang Zhang, Tong Che, B. Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, and Liam Paull. Robust and controllable object-centric learning through energy-based models. arXiv preprint arXiv:2210.05519, 2022.
A Limitations and Broader Impact
Limitations LSD faces several limitations common to unsupervised object-centric representation learning. First, it struggles with part-whole ambiguity, meaning it has trouble distinguishing between individual parts of an object and the whole object itself, especially when objects have complex textures in real-world images. Second, the quality and level of detail in LSD's segmentation results are sensitive to the number of slots it uses. If there are too few slots compared to the actual objects in an image, the model tends to produce generalized masks instead of distinct object segments. Conversely, when there are too many slots, the model tends to over-segment the image, splitting objects into multiple parts. This sensitivity to slot numbers is a problem in scenarios where we don't know the exact number of objects in advance. Finally, even with a strong Diffusion decoder, LSD struggles when dealing with real-world images that have highly diverse object appearances. To address this final limitation, we conduct a preliminary exploration in Appendix C, where we propose two LSD variants and demonstrate their effectiveness in handling real-world data.
Broader Impact The proposed Latent Slot Diffusion (LSD) model is a generative model for object-centric representation learning. The main potential negative social impacts is associated with its application in image editing and novel image synthesis. By enabling the extraction of object-centric or component-based representations from images, LSD allows for the generation of entirely new scenes with previously unseen configurations. This capability raises concerns related to image manipulation and privacy. For example, the capability to synthesize new human faces introduces potential privacy issues like deepfake generation and identity theft. Moreover, this ability could potentially cause digital misinformation, as it can generate and manipulate images with high accuracy in object-based level, making it increasingly difficult to distinguish between real and fake or edited visual content. Additionally, the image composition task gives rise to concerns about ethnic bias, ageism, or other forms of misrepresentation. This is due to the uniform sampling process of the learned slots within the same cluster and their independence across clusters. This can result in two significant problems: (1) the sampled distribution may reflect the biases present in the collected data, potentially leading to ethnic, age, or other population biases, and (2) when performing slot swapping for applications such as face replacement, mismatches in gender, age, and skin tone between existing parts and introduced parts may occur. Therefore, further application of this method should be proceeded under strict ethical guidelines and regulation.
B LSD on Simple Images
In the paper, we demonstrate that LSD achieves considerable performance improvements for complex images. However, our early experiments reveal that when applied to visually simplistic datasets like CLEVR, LSD exhibits reduced performance in terms of segmentation and representation quality, as illustrated in Table 3. To gain further insights into these results, we provide visualizations of the segmentation outputs in Figure 6.