SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Additional Details for Object-Centric Encoder. For all multi-object datasets, we set the number of slots to be the maximum number of objects per image plus one (intended for background). For the FFHQ dataset, we segment the image into 4 slots. We use the  $256 \times 256$  resolution images for all experiments and, if necessary, we use bilinear interpolation for re-scaling and center cropping to get square images. We do not introduce additional data augmentations during pre-processing. When computing the object segmentation, we use the attention map from the slot attention as the mask prediction. The resolution of the mask is designed to be  $64 \times 64$  for all models.
Additional Details for LSD Decoder. We add learnable positional embeddings into the self and cross-attention layers of UNet, which are found to greatly improve the stability of the training process for the CLEVR and CLEVRTex datasets. However, we have observed that this modification has a minor impact on the MOVi-C/E and FFHQ datasets, but have applied it across all models to maintain consistency. For the image auto-encoder, we use the KL-8 version provided in the LDM repository. During the generation processes for all tasks, we employ the DDIM sampler with  $\eta = 1$  and run it for 200 steps.
Baseline Implementation. Our baseline implementations are closely based on the official releases. For the SLATE baseline, we adopt the improved version proposed in [73]  $^{5}$ . We empirically find this version to be more robust to complex scenes in terms of both object segmentation and generation quality. The key difference of this version from the original SLATE is in the slot encoder. The original SLATE compute object slots from the latent space of the dVAE. Therefore, the attention resolution and information granularity of the slots are constrained by the dVAE latents. The improved SLATE addresses this issue by using a jointly trained CNN backbone to directly learn the slots from the original image. As shown in [73], this improved version of SLATE achieves improved performance on complex datasets such as MOVi-C and MOVi-E. For  $\mathrm{SLATE}^{+}$ , we replace the jointly trained dVAE with a VQGAN model pre-trained on OpenImages  $^{6}$ .
E.1 Ablation on CNN backbone in Object-Centric Encoder
This section presents an ablation study on the CNN backbone used in the object-centric encoder. To investigate the impact of the CNN backbone choice, we introduce a variant of LSD named LSD-CNN. Specifically, LSD-CNN uses a multi-layer CNN network as the image encoder in the object-centric encoder, following the same CNN design as in the improved SLATE model [73]. We compare the
<sup>5</sup>https://github.com/singh Gautam/steve
<sup>6</sup>We use the pre-trained weights from https://ommer-lab.com/files/latent-diffusion/vq-f8.zip
Table 7: Ablation Study on CNN backbone. This table presents the results of an ablation study on the CNN backbone in the object-centric encoder. Specifically, we compare the performance of the LSD method that uses a UNet backbone to the performance of LSD-CNN, which utilizes a multi-layer Convnet backbone.
(a) CLEVRTex
Segmentation	LSD-CNN	LSD	Representation	LSD-CNN	LSD
mBO (↑)	55.61	66.56	Shape (↑)	75.00	81.60
mIoU (↑)	53.82	65.02	Material (↑)	84.43	77.77
FG-ARI (↑)	43.88	61.74	Position (↓)	1.50	1.10
(b) MOVi-C
Segmentation	LSD-CNN	LSD	Representation	LSD-CNN	LSD
mBO (↑)	27.43	46.29	Position (↓)	1.25	1.14
mIoU (↑)	26.31	44.99	3D B-Box (↓)	1.46	1.44
FG-ARI (↑)	42.83	50.53	Category (↑)	41.29	46.71
(c) MOVi-E
Segmentation	LSD-CNN	LSD	Representation	LSD-CNN	LSD
mBO (↑)	22.00	39.63	Position (↓)	2.01	1.92
mIoU (↑)	20.71	38.28	3D B-Box (↓)	3.33	2.94
FG-ARI (↑)	42.14	53.40	Category (↑)	35.82	43.15
results of this variant with the proposed LSD, which utilizes the UNet structure as the image encoder in the object-centric encoder, on CLEVRTex, MOVi-C, and MOVi-E. The results are presented in Table 7. Overall, we observe that LSD outperforms LSD-CNN across nearly all tasks and datasets, suggesting that the UNet CNN backbone is critical for achieving the high performance of our model.
To further clarify the architecture choice, we also explore a LSD variant named LSD-Latent that employs an image latent encoder as its CNN backbone. We evaluate LSD-Latent on MOVi-E dataset and compare its object segmentation quality with LSD. The results can be seen in Table 8. The sub-optimal performance of LSD-Latent underscores the importance of having a separate CNN encoder in the object encoder.
F Additional Image Samples
We include additional generation samples in Figures 10 - 23. The additional samples demonstrate that LSD achieves more accurate object segmentation and generates more detailed and coherent scenes compared to the baselines. We provide additional observations and insights in the following sections.
F.1 Visualization of Visual Concept Prompts
In this section, we present visualizations of the visual concept prompts employed for generating image samples in the CLEVRTex and FFHQ datasets, as illustrated in Figure 10. To generate these visualizations, we apply the attention map corresponding to each specific slot as a mask over the original image, which highlights the regions captured by that slot. We observe that the LSD model
Table 8: Ablation Study on CNN backbone using Diffusion Latent Encoder. We compare LSD with LSD-Latent on segmentation quality in the MOVi-E dataset.
Segmentation	LSD-Latent	LSD
mBO (↑)	5.52	39.63
mIoU (↑)	4.04	38.28
FG-ARI (↑)	10.11	53.40
Figure 10: Compositional Image Generation with Concept Prompts. In this visualization, we show a concept prompt constructed by composing arbitrary slots from our visual concept library and the corresponding generated image by LSD.
can seamlessly combine components from distinct images to create a coherent image sample while preserving the attributes of each component in the final image. Note that each concept library is associated with one k-means cluster within the extracted slots, and each prompt displayed in Figure 10 is sampled from one such library. The visualization of the prompts suggests that the concept libraries is able to capture practical concept categories, such as objects and backgrounds in the CLEVRTex dataset and facial attributes, hairstyles, clothing, and backgrounds in the FFHQ dataset.
F.2 Impact of Pre-Trained Auto-Encoder on Generation Quality
We investigate the effect of the pre-trained image auto-encoder in the comparison between SLATE and  $\mathrm{SLATE^{+}}$  in Figure 12 and 13. Our results show that the use of the pre-trained auto-encoder leads to less blurry images and significantly more details, such as the floor and object texture in CLEVRTex and face detail in FFHQ. However, we also observe that applying the pre-trained auto-encoder does not necessarily improve the coherence of the generated scenes. For instance, in some  $\mathrm{SLATE^{+}}$  samples, the shape of the objects appear distorted in CLEVRTex and the background and hair style of FFHQ images are rendered as color patches. These findings indicate the limitation of the transformer decoders in SLATE and  $\mathrm{SLATE^{+}}$  models and highlights the crucial role of LSD's diffusion decoder in achieving optimal generation quality.
F.3 Impact of Segmentation Quality on Slot-Based Editing
In Figure 14 and 15, we investigate the effect of object segmentation on the slot-based editing task. Our result shows that when the object segmentation is almost perfect, editing tasks such as object removing can be easily accomplished by discarding the corresponding slot during image decoding. It is important to note that the object segmentation masks are derived from the attention weights on image features. Therefore, a perfect segmentation mask indicates that the object information is completely assigned to a single slot. However, we also observe cases when the object assignments are incomplete, such as when one object is divided into multiple slots. In such cases, removing an object would require removing all associated slots. These observations emphasize the importance of accurate object segmentation in achieving effective slot-based editing. Although LSD has demonstrated superior performance compared to existing approaches, further exploration in object segmentation is necessary to extend it to real-world applications.
Figure 11: Visualization of Unsupervised Object Segmentation. LSD achieves more accuracy object segmentation when comparing to the other methods.
Figure 12: Compositional Generation Samples using Visual Concept Library. LSD provides significantly higher fidelity and more clear details when comparing to the other methods.
LSD (Ours)
SLATE+
Figure 13: Slot-Based Image Editing: Face Replacement. We show face replacement in the FFHQ dataset, where we compose new images by combining the face slots from Source-B images with the hairstyle, clothing, and background slots from Source-A images.
Progressive Object Removal
Slot Attention
Figure 14: Slot-Based Image Editing: Object Removal and Segmentation Mask. Sample 1. Top: We show object removal by discarding the corresponding slots. Bottom: We show the segmentation masks of each slot.
Input
Progressive Object Removal
Masks
Slot Attention
Figure 15: Slot-Based Image Editing: Object Removal and Segmentation Mask. Sample 2.
Figure 16: Slot-Based Image Editing: Background Replacement. The background replacement task is performed by replacing the background-associated slots.
Figure 17: Slot-Based Image Editing: Object Insertion. The background replacement task is performed by simply adding the new slot to the existing set of slots.
Figure 18: Random Samples for Real-World Object-Centric Learning and Conditional Generation 1.
Figure 19: Random Samples for Real-World Object-Centric Learning and Conditional Generation 2.
Figure 20: Random Samples for Real-World Object-Centric Learning and Conditional Generation 3.
Figure 21: Random Samples for Real-World Image Generation 1.
Figure 22: Random Samples for Real-World Image Generation 2.
Figure 23: Random Samples for Real-World Image Generation 3.