SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To study how much randomness our LDM decoder has, we run it to decode the same set of slots given six different random seeds. The results are shown in Figure 17 of the submitted PDF file, where our model generates images with minor differences (mainly local textures and surface reflections), preserving the identity of objects.
Figure 15: Qualitative results of slot-to-video reconstructions on MOVi-D (top) and MOVi-E (bottom).
Figure 16: Compositional generation results on CLEVRTex (a), CelebA (b), and MOVi-E (c). Compared to baselines, SlotDiffusion produces results with higher fidelity and more details, such as textures on the objects and backgrounds, and hairs and facial features of humans, which are visually closer to the dataset samples.
E.4 Video Prediction and VQA
We show more qualitative results of Physion video prediction in Figure 18. PredRNN produces blurry frames with objects disappearing while still achieves a low MSE, proving that MSE is a poor metric in generation tasks. Although VQFormer generates sharp images, the generated frames are almost static, lacking scene dynamics. STEVE with SlotFormer is able to predict the rough motion of objects, yet the results are of low quality. In contrast, SlotDiffusion learns highly informative slots and can decode high-fidelity images. Therefore, our method predicts videos with the correct object motion and consistent scenes, even for the challenging dynamics of deformable cloth.
Figure 17: Images sampled from SlotDiffusion's LDM decoder with different random seeds.
Figure 18: Qualitative results of video prediction on Physion. SF stands for SlotFormer.
E.5 Failed Attempts
Here, we record some failed model variants we tried.
Image-space diffusion model. At the early stage of this work, we adopt an image-space DM as the slot decoder without the VQ-VAE tokenizer. This works well with low resolution inputs  $(64\times 64)$ . However, when we increase the resolution to  $128\times 128$ , we observe color shifting artifacts which is a common issue among high resolution DMs [76, 78]. Switching to an LDM-based decoder solves this problem, as the VQ-VAE decoder always maps latent codes to images with natural color statistics. In addition, since LDM consumes less memory, we can use a larger U-Net as the denoiser, which further improves the performance.
Prediction target of the diffusion model. By default, our DM adopts the traditional noise prediction  $(\epsilon)$  target. Recent works in DM propose two other formulations, namely, data prediction  $\left(\mathbf{x}_0\right)$  [71] and velocity prediction  $(\mathbf{v})$  [35], and claim to have better generation quality. We tried both targets in SlotDiffusion, but both model variants degenerate to the stripe pattern solution (similar to the SAVi result on MOVi-D in Figure 14) under multiple random seeds.
Classifier-free guidance [33]. We also tried classifier-free guidance (CFG for short) to improve the generation quality of SlotDiffusion. In order to apply CFG during inference, we need to randomly drop the conditional input (i.e., slots) during model training. However, with a high dropping rate, we observe severe training instability, where the model degenerates to the stripe pattern solution. When trained with a low dropping rate, we do not observe clear improvement in the generation quality.
Limitations. The ultimate goal of unsupervised object-centric models is to perform image-to-slot decomposition and slot-to-image generation on real-world data without supervision. With the help of pre-trained encoders such as DINO ViT [7], SlotDiffusion is able to segment objects from naturalistic images [21, 54]. However, the images reconstructed from slots are still of low quality, preventing unsupervised object-centric generative models from scaling to these data. In addition, objects in real-world data are not well-defined. For example, the part-whole hierarchy causes ambiguity in the segmentation of articulated objects, and the best way of decomposition depends on the downstream task. Finally, our learned slot representations are still entangled, while many tasks require disentanglement between object positions and semantics.
Future Works. From the view of generative modeling, we can explore better DM designs and techniques such as classifier-free guidance to improve the modeling capacity of fine local details. Moreover, our LDM decoder currently generates video frames individually without considering the temporal interactions. We can try the temporal attention layer in video DMs [27, 36] to improve the temporal consistency. To generate high-quality natural images, one solution is to incorporate pre-trained generative models such as the text-guided Stable Diffusion [73]. As discussed in the main paper, object slots in an image are similar to phrases in a sentence. We may try to map the slots to the text embedding space or via lightweight adapters [107].
From the view of object-centric learning, we want to apply SlotDiffusion to more downstream tasks such as reinforcement learning [95, 105] and 3D modeling [86, 104]. It is interesting to see if task-specific supervision signals (e.g., rewards in RL) can lead to better scene decomposition. To address the part-whole ambiguity of objects, Segment Anything [47] proposes to output three levels of masks per pixel, which cover the whole, part, and subpart hierarchy. We can apply such a design to slot-based models to support richer scene decomposition. Finally, there can be new protocols for evaluating these object-centric models. For example, MAE [29] leverages supervised fine-tuning to verify the effectiveness of their learned representations. We can also consider SlotDiffusion with labels such as class labels or object bounding boxes [46].
G Full Experimental Results
To facilitate future research, we report the numeric results of all bar charts used in the main paper in Table 7, Table 8, and Table 9.
Table 7: Full results on CLEVRTex and CelebA datasets. FG-ARI, mIoU, and mBO are given in %. * we use the masks provided by the CelebA-Mask dataset [51] to evaluate the segmentation results on CelebA. We select four classes from the provided labels, the original "hair", "cloth", and "background" classes, and a new "face" class by merging masks of all other facial attributes such as eye, nose, mouth. The data split of CelebA-Mask is slightly different from the original CelebA dataset, so we do not report this result in the main paper.
Method	CLEVBRTex	CelebA*
FG-ARI ↑	mIoU ↑	mBO ↑	MSE ↓	LPIPS ↓	FID ↓	FG-ARI ↑	mIoU ↑	mBO ↑	MSE ↓	LPIPS ↓	FID ↓
SA	67.92	53.70	57.43	212.4	0.41	134.57	32.13	26.26	28.79	243.3	0.28	89.63
SLATE	67.50	56.82	59.35	313.3	0.39	88.73	40.23	35.72	37.44	744.8	0.32	78.95
Ours	68.39	57.56	61.94	237.5	0.13	32.07	44.01	38.46	41.35	439.0	0.21	27.72
Table 8: Full results on MOVi-D and MOVi-E datasets. FG-ARI, mIoU, and mBO are given in %.
Method	MOVi-D	MOVi-E
FG-ARI ↑	mIoU ↑	mBO ↑	MSE ↓	LPIPS ↓	FVD ↓	FG-ARI ↑	mIoU ↑	mBO ↑	MSE ↓	LPIPS ↓	FVD ↓
SAVi	39.63	17.76	18.49	237.2	0.53	1123.8	46.70	23.49	22.23	255.2	0.55	1110.2
STEVE	49.43	29.51	29.16	673.2	0.51	857.0	56.04	27.86	27.97	603.2	0.51	796.4
Ours	52.09	31.65	31.24	632.7	0.40	493.2	59.99	30.16	30.22	586.5	0.39	353.3
Table 9: Full results on MOVi-Solid and MOVi-Tex datasets. FG-ARI, mIoU, and mBO are given in %.
Method	MOVi-Solid	MOVi-Tex
FG-ARI ↑	mIoU ↑	mBO ↑	MSE ↓	LPIPS ↓	FVD ↓	FG-ARI ↑	mIoU ↑	mBO ↑	MSE ↓	LPIPS ↓	FVD ↓
SAVi	44.95	30.74	29.88	158.1	0.44	1752.1	26.40	7.60	7.01	193.1	0.40	1139.2
STEVE	60.16	40.72	42.67	525.7	0.45	1245.2	39.02	34.27	34.89	476.6	0.32	704.6
Ours	69.13	46.55	48.68	240.3	0.17	361.9	42.32	38.33	39.03	208.0	0.11	242.2