Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling
Authors: Jiatao Gu
Ying Shen
Shuangfei Zhai
Yizhe Zhang
Navdeep Jaitly
Joshua M. Susskind
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion models
autoregressive models
latent variable models
text-to-image
Primary area: generative_models
TL;DR: We combine diffusion model with autoregressive priors for improved output diversity, interpretability, and controllability

Abstract: Diffusion models have emerged as a powerful tool for generating high-quality images from textual descriptions. Despite their successes, these models often exhibit limited diversity in the sampled images, particularly when sampling with a high classifier-free guidance weight. To address this issue, we present Kaleido, a novel approach that enhances the diversity of samples by incorporating autoregressive latent priors. Kaleido integrates an autoregressive language model that encodes the original caption and generates latent variables, serving as abstract and intermediary representations for guiding and facilitating the image generation process.
In this paper, we explore a variety of discrete latent representations, including textual descriptions, detection bounding boxes, object blobs, and visual tokens. These representations diversify and enrich the input conditions to the diffusion models, enabling more diverse outputs.
Our experimental results demonstrate that Kaleido effectively broadens the diversity of the generated image samples from a given textual description while maintaining high image quality. Furthermore, we show that Kaleido adheres closely to the guidance provided by the generated latent variables, demonstrating its capability to effectively control and direct the image generation process.

Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling
Jiatao Gu Ying Shen Shuangfei Zhait, Yizhe Zhang', Navdeep Jaitly', Josh Josh Susskind
AAplee University of Illinois Urbana- Chaam--a [jgu3,, szhai, yizzhang, njjitly,, @susskind)@apple.... com com "ying22eillino"" edu
equal contribution
(a) Standard diffusion model
A cat sat on the mat.
Diffusion
Model
Autoregressive Model (Prior)
Latent tokens (e.g.. captions, bbox, seg, visuell tokens)
Diffusion Model
(b) diffusion model with augo-regressive priors
Comparison of the generated image samples given the caption cat sat on the mat". Our models generate more diverse images with the help of autoregressive latent modeling.
Figure I:
Abstract
Diffusion models have emerged as a powerful tool for generating high-quality im- ages from textual descriptions. Despite their successes, these models often exhibit limited diversity in the sampled images, particularly when sampling with high classifier-fi free guidance weight. To address this issue, present Kaleido, novel approach that enhances the diversity of samples by incorporating autoregressive latent priors. Kaleido integrates autoregressive language model that encodes the original caption and generates latent variables, serving as abstract and intermediary representations for guiding and facilitating the image generation process. In this paper, we explore variety of discrete latent representations, including textual descriptions, detection bounding boxes, object blobs, and visual tokens. These representations diversify and enrich the input conditions to the diffusion models, enabling more diverse outputs. Our experimental results demonstrate that Kaleido effectively broadens the diversity of the generated image samples from a given textual description while maintaining high image quality, Furthermore, we show that Kaleido adheres closely to the guidance provided by the generated latents, demonstrating its capability to effectively control the image generation process.
Introduction
Diffusion models have become pervasive in many text-to-image generation tasks for their ability generate high-quality images based on textual descriptions. A pivotal mechanism in these models is
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

classifier-ff free guidance (CFG) [Ho and Salimans, 2021], which effectively steers the sampling process towards better alignment to textual prompts and improved sampling quality at the same time. CFG can be interpreted as tuning the temperature of the conditional distribution, whereas increasing the guidance scale sharpens the conditional distribution. This guides the generation to focus on regions of high conditional probability, effectively reducing sampling noise which is typically of lower density, However, while high CFG improves sampling quality, it simultancously narrows the diversity in the generated samples. This manifests in the models' inability to produce diverse images from the same caption, even when there are variations in the initial noise that seeds the generation process. For instance, given fixed textual description, "a cat sits mat", existing text-to-image diffusion models predominantly produce image samples depicting cats with similar colors and patterns, as illustrated in Figure 1. Such limited visual diversity hinders the practical application of diffusion models in scenarios where a wide range of creative and diverse visual interpretations are desired from identical textual inputs. It also poses challenges in scenarios demanding the representation of underrepresented data or accommodating a wide range of user preferences. Therefore, enhancing diversity in diffusion models without compromising the quality remains critical research problem.
To tackle this, we introduce Kaleido, a general framework that improves diffusion models with autoregressive priors. Kaleido first defines a discrete encoding of images (eg, detailed captioning, bounding boxes), which captures desirable abstractions of images that's not included in the default text prompts. Next, Kaleido integrates encoder-decoder language model that encodes the original text caption and autoregressively predicts the discrete latent tokens. Lastly, the diffusion model is conditioned on both the original text prompt and the autoregressively generated discrete latents and generates an image. This enriched conditioning allows Kaleido to produce a more diverse array of high-quality images, even high guidance scales. We explore various forms of latents, including textual descriptions, detection bounding boxes, object blobs, and abstract visual tokens all designed to refine and guide the conditional image generation process.
We experiment on both class and text conditioned image generation benchmarks 1 Kaleido not only outperforms standard diffusion models in terms of diversity but also maintains the high quality of the generated image. Additionally, the generated latents effectively control the characteristics of the generated images, ensuring that the image samples closely align with the intended latent variables. This modeling of latent tokens not only increases the diversity of image outputs but also provides degree of interpretability and control over the image generation process.
We show that
To summarize, Kaleido exhibits the following advantages:
1. Kaleido promotes the diversity in generated image samples even with high CFG, allowing the image generation of both high quality and diversity.
2. The generated latent variables are interpretable, offering an explainable mechanism behind the image generation process, and facilitating an understanding of how different latents affect the outputs.
3. Kaleido provides a fine- grainer editable interface that allows users adjust the discrete latent codes before final image production, granting greater flexibility and control over the output.
2 Preliminaries
Autoregressive Image Generation The success of large language models (LLMs) in NLP has demonstrated their scalability and universality of modeling any complex data, motivating the develop- ment of using autoregressive models for image generation. Typically, autoregressive image generation operates on discrete image tokens obtained from vector-quantization (VQ) [Van Den Oord et al., 2017]. More precisely, given an image x E xx we first obtain sequence of discrete tokens Zi:N E(x) which approximately reconstructs the input with a learned decoder D(zI.) æ. Then, an autoregressive model is learned to predict the discrete tokens one after another, mirroring the sequential language modeling:
N log Po 20m-11c)),
CAR
Σ
where e is the condition (e.g., class, text prompt, etc.), and Zμ is a special start token. At inference time, we first sample from the learned distribution, and then pass the sampled latents the decoder
the class conditioning setting can be considered as special case of text conditioning.

---

(D) to get the final output. Such VQ-based paradigm has been the foundation for various text-to- image [Esser et al., 2021. Yu et al., 2021, Zheng al., 2022, Yu et al., 2022] and multi-modal generation [Team et al., 2023, Team, 2024].
However, these methods share common limitation: they primarily rely on discretization, which struggles to capture all the nuances of an image when using limited length of discrete image token sequence. To generate higher-resolution images, longer sequence of image tokens is necessary. Yet, this inherently leads to increased capacity demands. For instance, Yu et al. [2022] requires 20B parameters work properly. Additionally, the left-te to-gh properties of these autoregressive models prevent the rewriting of previously generated image tokens, resulting in suboptimal image quality.
Diffusion-based Image Generation Diffusion models [Sohl-Dickstein et al., 2015, Ho et al., 2020] are latent variable models with pre-determined posterior distribution and are trained using a denoising objective, which has quickly become the new de-facto approach for image generation. Unlike autoregressive models which predict images as sequence, diffusion-baseee generate the whole image in a non-autoregressive fashion. Specifically, given an image 2 E IR x and signal-noise schedule {a,, where the signal-to-nosse ratio (SNR) (a /o7) decreases monotonically with f, we define a series of latent variables Zt, t = O...., T that adhere to: q( a xz) (Z ayæ, I). and q( (x,) N ( E. o
models iteratively
.
where vo - æ, Atis = a and o7)-- = o oi for < The model then learns to reverse this process using a backward model po( c), which reformulates a denoising objective:
CBM =
(a) [wi 11 c)- =x12
(3)
where xn (xt, c) is neural network (typically a UNet [Ronneherger et al., 2015] or Transformer [Pee- bles and Xie, 2022]) that maps the noisy input 2; its clean version æ. based the time step and conditional input C; wi E IRt is a loss weighting factor. In practice, De can be re-parameterized with noise- or v-prediction [Salimans and Ho, 2022] for enhanced performance, and can be applied on raw pixel space [Saharia et al., 2022, Gu et al., 2023] or latent space [Rombach et al., 2022].
Classifier-free Guidance An intriguing property of conditional diffusion models that we can easily guide the iterative sampling process for better sampling quality. For instance, Ho and Salimans [2021] introduced Classifter-free Guidance (CFG), which utilizes the diffusion model itself to perform guidance at test time. More specifically, we perform sampling using the following linear combination:
Tgg ( e) = 7 (ang(ar,, c) ag (ax)) + my (ax)).
where 7 is the guidance weight, and (2x)))) = we (Z,, 0) is the unconditional denoising output. During training, we drop the condition c with certain probability Puncond to facilitate unconditional prediction. When y > 1, CFG takes effect and amplifies the difference between conditional and unconditional generation, leading to global control of high-quality generation.
Compared to autoregressive models, diffusion models are more flexible in adjusting sample steps, allowing for the utilization of noise schedules to learn different frequencies. Additionally, with the use of CFG, diffusion models can achieve higher quality images with much fewer parameters than autoregressive models. However, it's notable that CFG can significantly impact the diversity of the diffusion output, which motivates us to revisit the basics and combine the strengths of both.
3 Kaleido Diffusion
We propose Kaleido, general framework that integrate autoregressive prior with diffusion model to enhance image generation. As illustrated in Fig. 2, Kaleido comprises two major components: an AR model that generates latent tokens abstract representations, and a latent-augmentedd diffusion model that iteratively synthesizes images based on these latents together with the original condition. For following sections, we first describe the importance introduce additional latents in standard diffusion models ($ 3.1). and show how we can model them with AR models ($ 3.2). The training and inference procedure are described in $ 3.3 and $ 3.4.
3.1 Lagent-augmented Diffusion Models
As demonstrated in Ho and Salimans [2021], diffiusion with CFG (Eq, (4)) is equivalent to follow V. a log (x(e) 7 VV a (log po (x/c) log pe (x))] + Va Po()),
(5)
3

---

Discrete latents
Discrete latents
Autoregressive Model (e.g., T5-decoder)
Context Encoder
Latent Extractors (e.g.. MLLMs)
]
Panda eating Pizza
Panda eating Pizza
Diftusion Model (e.g., MDM)
Stage Latent tokens Extraction
Stage Autoregressive and Diffusion Joint Training
Figure 2: Training pipeline of the proposed Kaleido diffusion.
which can be interpreted as sampling from a "temperature-adjiste" distribution:
Po (x) pe(x)) (po(c))))' where po(c)z>) po(zzcc)/po(z).
(6)
Here can be seen as inverse temperature, which sharpens the conditional distribution pa (a(c)) when
y > 1. That is say, CFG is crucial as it guides the generation to only focus on nigh-probability regions, avoiding sampling noise (which tends to have low density). However, sharpening the distribution also reduces the diversity, causing undesirable phenomena like "mode collapse". This is because (e.g., class label, text prompt, etc.) normally does not contain all the information that describes æ. Suppose we introduce a hypothetical variable z to represent the "modes" which (2 lc), and leave po (ar c) model other variations including local noise. In this case, CFG will simultaneously sharpen both distributions, considering:
we care most (=
pa pa(x)c)
Σ
(z c)
pe (a c)
mode selection
image variation
where standard diffusion models implicitly learn mode selection step together with generation.
Therefore, natural solution is to explicitly model "mode selection" before applying diffusion steps so that the mode distribution will not be distorted by guidance. In this way, the sampling procedure (Eq. (6)) is modified as steps: z (2 (2)c), x (1=l22. c), where CFG be applied after sampled. From the perspective of score function, we rewrite Pe (zx c) as po (al 2) in Eq. (5):
V Po c, 2) 7 V (log pa (œ) + log e)
log
}] + V log (or))( (8)
Compared to standard diffusion process, the highlighted term above pushes the updating direction towards the sampled modes at each step. This ensures diverse generation as long pa (z (z(c) is diverse.
A Toy Example We visualize the effect of explicitly introducing latent priors using toy dataset with two main classes, each containing two modes. We compare two models: a standard diffusion model conditioned on the major class ID, and latent-augmented model incorporating subclass ID as priors. Fig. 3 shows that while the standard diffusion model tends to converge to one mode (subclass) with increased guidance, the latent-augmented model captures all modes, showing the benefit of latent priors for improving diversity under high guidance. In practice, given the challenge of identifying all "modes" in real-world data distribution, next propose to employ autoregressive model to universally model various latent modes.
Autoregressive Latent Modeling
To capture the complex distribution of real images, it is clearly impossible to assign classes for each mode. However, it non-trivial to determine (1) the best representations
modes 2; (2) the suitable generative model that can model (z (z)c). Fortunately, the modes that humans can perceive from an image are largely abstract, and such abstract semantics are easily represented in discrete symbols. For example, we can easily describe content differences through natural language, create composite

---

CFG-1
CFG-5
CFG-8
CFG-25
Closs Class
Class
Class
Class
Class
Closs
Closs
Class
Class
Class
Class
Figure 3: Effect of augmented latents. The first row displays the sampling results from the standard diffusion model, while the second row shows the results from the latent-augmented diffusion models.
images based on spatial locations, and imagine novel visual concepts from experience. Therefore, is logical to use such abstract discrete tokens, i.c., x = [Z ZNI. Naturally, the most convenient way to model such distribution is using an autoregressive model pe (=|cc) Note that this is distinct from the conventional autoregressive image generation ($ 2), as we only learn such models as "latent modes". where the sampled 21:N are not supposed to reconstruct an image directly. As a result, eases the modeling difficulty and improves the sampling performance.
In this work, explore four types of abstract latents: textual descriptions (texf), defection bounding boxes (bbox), object blobs (blob), and visual tokens (voken). all of which can be predicted from multi- modal large language models (MLLMs) [Bai al., 2023, Liu et al., 2024, Ge al., 2023a] given the condition-image pair (c, Each type aims to enrich the mode-to-image correspondence, covering different aspects of image formation. These extracted tokens can either be predicted separately or modeled together with single autoregressive model. Fig. 4 shows the examples of these four generated latent variables. The methodology for constructing the training dataset for these variables is detailed in Appendix A.
3.3 Joint Learning of Autoregressive and Diffusion Models
Similar to other latent variable models like VAEs [Kingma and Welling, 2013], Kaleido can be trained to maximize the evidence lower bound (ELBO) as follows:
max log pa (a (x|c) 2 E [=zag(=)e,c)
[log pa + log pa (2/2 c)] + H [g(=|0,)))
CAR El)(1)
COM Eq) (3)
where q is the inference model, and H((q) is the entropy. In this paper, we always assume a fixed inference process (as explained $ 3.2). Therefore, the entropy term can be omitted, and we can efficiently sample and store z for the entire dataset before training starts. We illustrate the training pipeline in Fig. 2. Compared to standard diffusion models which typically involves a context encoder and a denoising network, Kaleido integrates the additional autoregressive decoder for modeling the discrete latents. Such decoder uses cross-attention to gather the encoder states at every step, and the final decoder layer states are concatenated with the encoder as the inputs for diffusion. Following common practices, we freeze the context encoder during training, and jointly optimize the autoregressive decoder together with the denoising model. The training objectives (Eq. (9)) is equivalent to the combination of both models, denoted L _DM + π LAR with n hyperparameter for balancing the contributions of the autoregressive and diffusion models in practice.
3.4 Interpretable and Controllable Generation
During the inference stage, given the provided textual description, the autoregressive model will first predict the discrete latents before image generation. These latents, being predominantly human- readable, add a layer of interpretability to the image- generation process, allowing humans to observe its internal "thought" process. This transparency also provides users with the flexibility to modify the latents desired. To incorporate user modification, the altered latents are re-input into the autoregressive decoder to obtain the modified final hidden states. The latent-augmented diffusion model then synthesizes the final image conditioned on the updated representation.
5

---

Textual Descriptions
A person in blue sweater and jeans is sitting the floor top gray couch with their laptop their lap. They have yellow Lubrador Retriever in their lap, who is looking at the camera. The dog has its tongue out and is laying down on the person's 's lap.
Object Blobs
Detection
Visual Tokens
Bounding
Boxes
Caption:
13
7
9
Dog laying a human's lap
Figure 4: A Variety of Discrete Tokens. Original caption:
"Dog laying a human's 's lap"
4 Experiments
4.1
Experimental Setups
Basel me
0.6
Baseline Ours
Dataset We validate approach on both class- and text-conditioned image generation benchmarks. For the former, we use ImageNet [Deng et al., 2009], and we learn the text- to-image models on CC12M [Chang- pinyo al., 2021], large image-text pair dataset where each image is ac- companied by descriptive alt-text. All models are trained to synthesize at 256 256. We generate all four types of latents as discussed in Appendix for both datasets.
15
Ours
0.5
10
2
2
6
Guidance weight
weight
Figure 5: Comparison with guidance weights.
Evaluation Metrics To assess the performance of our models, we employ Fréchet Inception Distance (FID) [Heusel et al., 2017] to capture the overall performance (considering both quality and diversity) of the generated images, and use Recall [Kynkäänniemi et al., 2019] to specifically measure the diversity of the generated images. Furthermore, we employ two additional quantitative assessments of diversity: Mean Similarity Score (MSS) and Vendi scores [Friedman and Dieng]. We use SSCD [Pizzi et al., 2022] as the pretrained feature extractor for calculating both MSS (SSCD) and Vendi (SSCD). Additionally, we utilize DiNOv2 [Oquab et al.] as the feature extractor for Vendi (DiNOv2), based on evidence from Stein et al. [2024] that suggests DiNOv2 provides a richer evaluation of generative models.
Implementation Details and Baseline We implement Kaleido with Matryoshka Diffusion Models (MDM) [Gu et al., 2023], a recently proposed approach that generates images directly in the raw pixel space with efficient training. The default MDM consists of frozen T5-XL [Radford et al., 2021] context encoder and a nested UNet-based denoiser. We initialize the additional autoregressive decoder with the decoder of T--XI. and make the parameters trainable. The vocabulary is resized to adapt special visual tokens. For fair comparison, we use MDM with the same hyper-parameters as our baseline model, and train both types in almost identical settings on 64 A100 GPUs. Additionally, we compare Kaleido with the Condition Annealed Diffusion Sampler (CADS) [Sadat et al.), a general sampling strategy that enhances the diversity of diffusion models by annealing the conditioning signal during inference. Given that CADS applicable to different model architectures, we also evaluate CADS integrated with both baseline model MDM (MDM CADS) and our model (ours CADS).
4.2
Quantitative Results
Fig. quantitatively compares Kaleido with the baseline diffusion models (MDM) with various guidance scales on ImageNet. Both metrics are evaluated with 50K samples against the full training set, where both our models and the baseline use DDPM sampling with 250 steps. Our findings reveal that Kaleido consistently enhances the diversity of samples without compromising their quality across different CFG, evidenced by the general improvement in both FID and Recall. Moreover, while the baseline's FID increases and Recall decreases significantly with higher CFG, Kaleido demonstrates a steadier performance profile.

---

Model
FID-50K
Precision t
Recall f
MSS (SSCD) 4
Vendi (SSCD) 1 Vendi (DiNOv2)
MDM
15.5
0.93
0.22
0.21
8.42
3.04
MDM CADS
10.6
0.60
0.62
0.12
9.28
4.72
Ours
9.0
0.85
0,42
0.16
8.82
3.79
Ours CADS
5.9
0.76
0.52
0.12
9.21
4.83
Table I: Comparison of quality and diversity on ImageNet. FID-50K, Precision, and Recall are
evaluated on 50K samples, while MSS and Vendi scores assess diversity on 1K x 10 samples.
To further investigate, we examine image quality and diversity between Kaleido and baseline models. As shown in Table 1, Kaleido outperforms the MDM + CADS combination in terms of FID-50K and precision, demonstrating that our method more effectively maintains high image quality while generating diverse samples. Furthermore, integrating CADS with our model yields the best FID-50K results. Note that precision cannot accurately evaluate models with diverse outputs since model producing high-quality but non-diverse samples could artificially achieve high precision [Sadat et al.].
Moreover, we assess the diversity of the generated images using 10K samples. Following CADS, we select 1, 000 random classes from ImageNet and generate 10 samples per class. Table shows that both Kaleido and CADS significantly enhance sample diversity. While CADS achieves better performance in diversity, our model maintains superior image quality. Additionally, the methodologies used in CADS complementary to ours, suggesting potential benefits from integrating CADS with our Kaleido. In fact, incorporating CADS into our model not only further improves image quality but also improves diversity, achieving the best scores in FID-50K, MSS (SSCD), and Vendi (DiNOv2).
Lastly, we provide visual comparisons for class- and text-conditioned image generation in Fig. 11. Notably, we observe that MDM + CADS fails to generate cats of diverse breeds from the prompt "a cat sleeping on the bed." In contrast, Kaleido can produce images of cats from various breeds with more diverse surrounding environments, showcasing its superior diversity capabilities. This observation contrasts with the trend of diversity scores in Table 1, suggesting that these diversity metrics may not fully capture certain aspects of diversity.
4.3 Qualitative Results
Diversity of Generated Images We present a comparative analysis of the images generated by Kaleido against baseline models (MDM). Fig. 7 demonstrates the comparison between baseline models and Kaleido on two conditional generation tasks: the class-conditioned image generation and the text-to-image generation. In both tasks, Kaleido consistently produces more diverse images from identical condition (class or textual description) across varying CFG scales. For instance, in the task of class-to-image generation, the baseline diffusion models generate predominantly frontal views of a "husky" at high CFG, while Kaleido produces diverse images depicting huskies in various poses and numbers. A similar improvement in diversity is observed in the text-to-image generation as well, highlighting the robustness of Kaleido in generating diverse images under identical conditions.
Control from Latent Tokens We show the efficacy of latent variables in guiding the image generation process in Fig. 6. Fig. 6 demonstrates images generated with different types of latent variables: (a) textual descriptions, (b) object blobs, (c) detection bounding boxes, (d) visual tokens, and (c) combined latents, which integrate textual descriptions, detection bounding boxes and visual tokens. We visualize the generated latents tokens alongside the resulting images, showing how closely the images generated by Kalcido align with the latent tokens. Such alignment is evident in fine-grained visual information such as object appearance, background, and atmosphere -, spatial location and orientation of different objects, and the stylistic elements of generated images. This alignment confirms that Kaleido can effectively interpret and utilize generated latent variables to guide and refine the image generation process.
Latent Editing Fig. 8 showcases the impact of latent editing in image generation. The first row displays images generated using autoregressively produced latent tokens. In the second row, we demonstrate the effect of manual modifications to the textual descriptions: changing "log" to "cobblestones" and "a body of water" to "forest". These changes result in a modified image where a frog is now positioned on cobblestones with a forest background. Additionally, by further augmenting the bounding box of a cup different position, we observe that the cup's position in the image changes accordingly, while most other visual elements remain unchanged. The precise control of image characteristies via latent editing underscores Kaleido's 's flexibility and controllability, offering
7

---

Inpat: ceddy wearing glasnes
mughh mndy
uoid loe woiig pimss
cron bume lor bocing 1ef love. Rabadgred leme Frow and gyv your
dely nen balggrend ta lpling woddy oaldy teeghs wariig
plase
ueretty
under driszly each kolds umbralla
(
Impuc: photo a blue Day standing lergo basket strewbberiis
(4)
Imput: squiral wearing croun
aseu dile sulh mypua dicneedl
75 (o eraeerrriieigg p'onelieenn weoeeneerralllleee eroerarraa celeedeergpperrr
erphentienaa sioppn denes toep eroune of sucleaatie
Imput: oul with knnited hot holding boand with "Thank You" written on
halding
Thank You
(4)
backpwand.
VON
salo eessage stand some Crnall imagria grpundd
THANKK YOU
YOUR
ThAnk TRU
Figure 6: Example of generation with various latents.. This figure showcases images generated with different types of latents: (a) textual descriptions, (b) object blobs, (c) detection bounding boxes, (d) visual tokens, and (e) combined latents (textual descriptions + detection bounding boxes + visual tokens). Each row shows two sets of generated images sampled with one type of latents. Each set displays visualization of the generated latents tokens (left) and collage of images (right) sampled using the same latent tokens but different noises. The image tokens capture visual details difficult to convey through text, such artistic style.
powerful interactive interface for users customize the generated images. Furthermore, the high fidelity of the re-generated images to their original versions indicates Kaleido's 's potential for applications requiring personalization or customizations.
Related Work
Augmenting Diffusion Models Various enhancements have been proposed to improve the versatil- ity and controllability of diffusion models with augmented latents. Innovations such Diffusion AE [Preechakul et al., 2022] integrates diffusion models with learnalle encoder that extracts high-level semantics and enables the diffusion model add details directly image space. Further efforts have focused on incorporating specific control signals, such bounding boxes, layout, and segmentation masks to guide and control the image generation process. [Balaji et al., 2022, et al., 2023, Zheng al., 2023a, Hu et al., 2023]. Recently, BlobGen [Nie et al., 2024] proposes to ground existing text-to-image diffusion models on object blobs tilted ellipses that capture spatial details of the objects for compositional generation. While these approaches improve the models' capacity to

---

"Siberian husky" (Class-to-lmage Generation)
bald eagle made of chocolate powder, mango, and whipped cream." (Text-to-lmage Generation)
y=10 (No CFG)
y=1.5
y=3.0
Y=5.0
Y=7.0
(a) Baseline
(b) Ours
(c) Baseline
(d) Ours
Diversity comparison to standard diffusion model. Images sampled under varying CFG scales (y). Panels (a) and (c) display images from the baseline models, while panels (b) and (d) show images from Kaleido. From top to bottom, as the CFG increases, the standard diffusion models exhibit reduced diversity, while Kaleido consistently maintains diversity across guidance scales.
Figure 7:
adhere to specified spatial layouts, they often necessitate modifications to the attention mechanism. potentially limiting their generality. In contrast, our method enhances the generative capabilities of diffusion models without altering the model architecture.
Connecting Diffusion Models with LLMs The remarkable success of Large Language Models (LLMs) and diffusion models has spurred interest connecting these models, aiming to leverage capabilities of L.LMs in understanding and generating complex data and combine it with the powerful image synthesis capabilities diffusion models [Ge al., 2023b, Zheng et al., 2023b, Sun al., 2023]. Ge et al. [2023b], Zheng al. [2023b] propose image tokenizers that encodes images into

---

Input: a photo a frog drinking coffee
Autoregressive Generation:
Diffusion Generation:
sipping a cup of soffee, seemingly erjoying a relazing break. The frog is pratiioned alog. with los eyes closed and a small inlll
imago
flavor
cup effeffeee
to the Tog, and backgrowrd
featunte a. bady wanerr The frog*
and yelloy coloration standa against nan seeting, making for chamsin whonoizal
Editl:
Diffusion Re-generation:
image, freg is seen sipping a cup of sofeffel seemiingyy enjoying a relaxing freat The frog is pretloned closed and small siile face, cup coffee placed to the freg, and the backgrourd featurse foreet, The frog's green and yelleu coloration standa our againet the setting, making charming
cobaltstores with
coffee,
Edita
Diffusion Re-generation:
image, frog is aears sipping on a cup of coffee, asemingly enjoying relaxing break. The frog is posttioeed cobalastones, with syea deased and small smile eavering flavor cup coffee plazed to the frog, and the backgrounrd fentures forest, The frog's green and yellou coloration stande againes the setting, making charming
it's
coffee,
Figure Effect of sequential latent editing. The top row displays images generated with autoregres- sively produced latent tokens. The middle row shows the re-generated images after applying latent editing to the textural description, and the bottom row presents re- e-ggenerated images after further edits to the bounding box, showing the impact of step-by-step latent editing.
visual tokens, enabling multimodal language modeling. This line of work focuses empowering LLM with image generation ability by aligning its output embedding space with the pre-trained diffusion models. Our work leverages the LLMs' robust capabilities textural understanding and generation to model the generation of abstract latents from the original text. These latents are then integrated with latent-augmented diffusion model, enabling more interpretable and diverse image generation process.
Our approach also distinguishes itself from the re-captioning method introduced in DALL-E [Betker et al., 2023]. Unlike re-captioning, which typically replaces the original captions with more descriptive captions, method retains the original condition and supplements it with latent variables of various forms (beyond textual captions like bbox, blob and "vokens").. The sampled latents serves as a unifying interface for various types of inputs, and introduce diversity compared to recaptioning where no sampling is involved at inference time.
6 Conclusion
In this work, we address the challenge of improving sample diversity under high CFG diffusion models. We introduce Kaleido Diffusion, which combines an autoregressive prior with latent- augmented diffusion model. Results show Kaleido increases diversity without compromising quality, even at high CFG. With human interpretable latent tokens, Kaleido offers an explainable mechanism behind the image generation process and provides a fine-grained editable interface, enabling precise user control over the generated images.
10

---

References
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
Yogesh Balaji, Seungjun Nah, Xun Huang. Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: toxt-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0132 2022.
James Betker, Gabriel Goh, Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/uall--3. pdf. 2(3):8, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3566 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-scale Hierarchical Image Database. IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang. and Ying Shan. Making Ilama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01211 2023a.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Planting seed of vision in large language model. In The Twelfth International Conference on Learning Representations, 2023b.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M Susskind, and Navdeep Jaitly. Matryoshka diffuuion models. In The Twelfth International Conference on Learning Representations, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffuuson guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Vincent Tao Hu, David W Zhang, Yuki M Asano, Gertjan J Burghouts, and Cees GM Snoek. Self- guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18413-18422, 2023.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 2013.
arXiv preprint
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yes Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances Neural Information Processing Systems, 32, 2019.
11

---

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded lext-to-image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22511-22521, 2023,
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Weili Nie. Sifci Liu, Morteza Mardani, Chao Liu, Benjamin Eckart, and Arash Vahdat. Compositional text-to-image generation with dense blob representations. arXiv preprint arXiv:2405.08244 088446 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:22/2.09774. 2022.
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A self- supervised descriptor for image copy detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14532-14542, 2022.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Dif- fusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619-10629, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 103.002003.00020, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-6) 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net Convolutional Networks for Biomedi- cal Image Segmentation. International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 234-241, 2015.
Scyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M Weber. Cads; Unleashing the diversity of diffiusion models through condition-annen sampling. In The Twelfth International Conference Learning Representations.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-3666 936444, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.000 10512, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pages 2256-2265. PMLR, 2015.
George Stein, Jesse Cresswell, Rasa Hosseinzadch, Yi Sui, Brendan Ross, Valentin Villecroze. Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Loaiza-Ganemi Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312. 2023.
12

---

Chameleon Team. Chameleon: Mixed-modaa early-fusion foundation models, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: family highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge. and Yonghui Wu. Vector-quantized image modeling with improved vogan. In International Conference on Learning Representations, 2021.
Jiahui Yu, Yuanzhong Xu. Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. Transactions on Machine Learning Research, 2022.
Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 02223412-23425, 2022.
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi. Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedingy of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 22490-22499, 2023a.
Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: 5: Interleaved vision-ande aoooon-nn--annnnn and-language generation via generative vokens. arXiv preprint arXiv:2310.02229, 2023b.
13

---

Appendix
A
Auto-regressive Latent Modeling
In this work, explore four types of abstract latents, including textual descriptions (text), detection bounding boxes (bbox), object blobs (blob), and visual tokens (voken), Each type is designed to enrich the mode-to-image correspondence, covering different aspects of image formation. Examples of these abstract latents are illustrated in Fig. 4. In the following paragraphs, we detail the methodology employed in constructing the training dataset for these abstract latents. Additionally, Fig. outlines the pipeline for the step-by-step generation of these abstract latents.
Textual descriptions Typical text-image datasets often provide captions that fail to fully capture the details of the image. For instance, as shown in Fig. 4, the original caption "Dog laying on a human's 's lap" omits crucial details such as the presence of "laptop", which is essential for accurate image
Textual descriptions (caption)
Original caption: (
Using the information provided in the caption above, Please provide detailed description of the image in 50-80 words, incorporating relevant information from the caption and expanding on the visual elements:
Include names, objects, events, and locations mentioned in the caption
Do not include placeholders like <PERSON> in the caption
Describe people, characters, animals, and notable entities
Mention the setting, background, and overall environment
Note colors, lighting, composition, and style aspects
Refer to any text, symbols, or logos in the image
Combine caption details with your observations to create comprehensive description of the key elements and overall scene, focusing on the most salient aspects of the image.
Textual descriptions (label)
Object labels: {}
Using provided object labels, generate detailed description of the image 50-80 words, incorporating the relevant information about prominent objects identified and expanding on the visual elements:
Describe each labeled object, including its size, shape, and placement in the scene Depict relations between objects, such proximity arrangement Highlight interactions or functions implied by the objects
Describe the surrounding environment context that complements the labeled objects
Any notable features or characteristics the objects, like color, texture, or design elements
Describe people, characters, animals, and notable entities
Mention the setting, background, and overall environment Note colors, lighting, composition, and style aspects Refer to any text, symbols, or logos in the image
Craft comprehensive depiction of the scene based on the identified objects, utilizing both the labels and contextual observations enrich the description.
Captions with grounding
Generate the caption in English with grounding:
Table 2: The instruction for prompting Qwen-VI. generate detailed textual descriptions and captions with grounding.
14

---

generation. To address this, we employ detailed textual descriptions as latent variables. These textual descriptions supplement the original captions by providing additional information that might be missing from the original captions. Specifically, we leverage Qwen-VI--Chee Vhh-Chat [Bai et al., 2023]. a large visual language model, designed for effective catructtionfoolowinng across variety of multimodal tasks. We instruct Qwen-Vi Vh-Chat to produce detailed textual description given the original caption and corresponding image. The specific instructions used for generating the textual descriptions are detailed in Table under the section Textual descriptions (caption). Fig. 4 shows an example of the generated detailed textual description that provides a more comprehensive depiction of the scenes than the original captions, thus allowing for a richer image generation.
Additionally, for the ImageNet dataset, which consists of label-image pairs for class-to-image generation, we instruct Qwen- VL-Chat to generate detailed descriptions based on the class label and corresponding image. The instruction for this procedure is similarly documented Table 2 under the section Textual descriptions (label).
Detection bounding boxes
The spatial location of objects within an image is also crucial infor- mation for accurate representation of the image, yet such information is typically absent in textual descriptions. To incorporate this spatial information into the image generation process, we use detection bounding boxes as one type of abstract latents. Specifically, 2023] to prompt the model to "Generate the caption in English with grounding:". This approach results in captions where the spatial locations of objects are explicitly annotated within the text. For instance, as shown in Fig. 4, the caption with grounding for this example is: "Dog (1, 33, 995, 995) resting head on owner's lap (I, 630, 785, 998) while they work on laptop (39, 336, 999, 972)." Each bounding box is described in a string format "x1, 1/1 I2. 1/2" where 11, 1/1 and 22, 1/2 are the coordinates the top-left and bottom-right corner, respectively. All the coordinates are normalized to [0, 1000] range. The coordinates string is treated as part of the text, obviating the need for an additional positional vocabulary.
Qwen-VI. [Bai et al.,
Object Blobs Inspired by Nie et al. [2024], we utilize object blobs as the abstract latents that contain more advanced spatial information. An object blob defined as a tilted ellipse that specifies the position, size, and orientation of object within image. Specifically, a blob is represented "(æe- Yes "major. Tminor, 0)" where (ze, ye) denotes the center point of the ellipse, Tmajor and mminor are the radii of its semi-major and semi-minor axes, respectively, and 0 E: [0, 180) denotes the orientation angle of the ellipse. To extract the blobs for meaningful objects, we leverage the results from bounding box detection and employ SAM [Kirillov et al., 2023] to generate the segmentation maps using the bounding boxes as prompts. Subsequently, an ellipse fitting algorithm is applied to these segmentation maps to determine the blob parameters for each identified object. This method allows for a more precise representation of objects' spatial characteristics, thus improving the integration of spatial and structural information within the image generation process.
Visual Tokens Representing images via discrete visual tokens, especially using technologies like Vector Quantized Variational Autoencoder (VQ-VAS VA-V [Van Den Oord al., 2017], has become a prevalent technique in generative modeling due to its ability to encode high-dimensional image data into a more manageable, discrete space. In this work, we utilize SEED [Ge et al., 2023b], a VQ-based image tokenizer, encode an image into sequence of abstract discrete image tokens. These tokens encapsulate high-level semantic information of the visual elements in the image, serving as potent latent variables for guiding the diffusion model. The visual tokens are concatenated with the delimiter "#", forming a sequence of visual tokens represented as "I, "I #Ig" where each "I," denotes the image token id.
B Implementation Details
B.1 Architecture
In this paper, we use the following NestedUNet architecture proposed in Gu et al. [2023] to implement the denoising model. The total number of parameters is about 500M. For the autoregressive prior, employ T5-XL [Raffel et al., 2020] for all experiments regardless of the input latent types. Both the denoiser and T5 decoder receive gradients and are trained end-to-end.
15

---

config:
resolutions==256 128 664]
reodittio_.chhmeeleef...111122 ,26]
inner_configt
resolutions=116..22,11
resolutionn.chhamell---3333 512, 768]
auw.aca._aayrra.err.booo-----
num_heads=8,,
schedule="cosinee
[um_res.block==[22..1]
bu..str..layerssppr.bbcck [0,
schcuule='cosiie--siitt emb
mbb_channels-1024
num num_im_atta_layere_
1m in..fature-pppeetttlllaaamall-o.
B.2 Training
For all experiments, we share all the following training parameters for both the baseline model and the proposed Kaleido Diffusion.
default training config:
batch_size=511
wum_updates=400_00
optimizer-'adant
adam_betal-0.9
9dam_beta2-0.99
adam_ops=1.e-8
learning.rat==le44
weight_decay=0.00
gradient_clip.noo==221
ema_decay=0.9999
mixel_pprciaaioo..rraaiiigggggp.
All experiments performed 64 A100 GPUs which takes roughly weeks for training 400k steps for both ImageNet and CC12M datasets. For text-to-image models, we perform an additional 400k steps progressive training 64 x 64 resolution, while train the entire model from scratch directly at 256 x 256 for ImageNet. Due to the memory cost of the T5-decoder, we can only fit 4 ~ 8 images per GPU, causing at least x3 slower training compared the original MDM models.
B.3 Learned Models
To demonstrate the effectiveness of various latents, train model with 5 types including text, bbox, blob, voken, and combined for text-to-image generation. For combined setting, we use the autoregressive model to predict
combined = text bbox voken
sequential way such that the latter latents will be controlled by earlier latents. We also trained models on ImageNet using combined latents for quantitative comparison.
C Limitations
Training Complexity: The enhanced diffusion model may require more complex and extended training processes compared to standard models. This could lead to increased computational costs and longer development times, potentially limiting accessibility for smaller organizations or individual researchers.
16

---

Difficulty in Finding Optimal Latents: Identifying the most effective latent variables to achieve the desired output diversity can be challenging. This process might involve extensive experimentation and fine-tuning, which can be time-consuming broader range of modes, such as depth and semantic maps, adds another layer of complexity to the model development, requiring sophisticated techniques to integrate these diverse forms of data effectively.
and resource-intensive Additionally, covering
Memory Usage: The improved diffusion model, with its increased output diversity, might demand higher memory usage due to the integration of the heavy language models. However, potential strategies such as partial training or joint training with LLMs could be explored to mitigate this issue. These methods could help distribute the computational load more effectively and reduce the memory footprint during the training process.
D Impact Statement
The proposed method to enhance diffusion models and increase output diversity has significant social implications. By advancing the diversity and accuracy of generated outputs, this technology can be leveraged in various fields such as art, media, and content creation, providing more inclusive and representative outputs that reflect a broader spectrum of human experiences and creativity, Moreover, in areas like healthcare and education, diverse and precise models can lead to more personalized and effective solutions, addressing the unique needs of individuals and communities. This innovation also promotes ethical AI practices by reducing biases in model outputs, fostering more equitable digital landscape. Ultimately, the enhanced diffusion models will contribute to the democratization of AI, making sophisticated tools accessible to a wider range of users and applications, thereby driving societal progress and innovation.
E
Color Cluster as Additional Latent
We have included alternative approach that constructs latent tokens without relying additional knowledge. Specifically, we train a model using color clusters as latent tokens. For each color channel (R, G, and B) within the range of 255, equally segment it into eight clusters, resulting in total of 8 x 8 8 = 512 color clusters. Given an image, we resize it to 4x4 pixels and assign a color cluster ID to each pixel based on its RGB value. The image is then encoded into sequence of color cluster IDs (e.g.. "C, #C+ "), with each C; representing a color cluster ID. This sequence serves the condition for training our Kaleido diffusion.
In Fig. 10, we showcase images generated using color clusters as latent tokens on ImageNet. Our results demonstrate that, compared to the baseline MDM, our Kalcido diffusion can generate more diverse images with latent tokens derived purely from color clustering. This highlights that Kalcido diffusion's 's capability to generate diverse images is independent of distilled external knowledge, confirming that our approach can produce varied images without the aid of any other pre-trained models.
F Qualitative Comparison with CADS
Fig. 11 shows the visual comparisons for class- and text-conditioned image generation with CADS.
G Additional Results
We show additional results randomly sampled from our models. For all results including the baseline model, we use DDPM sampling with 250 steps.
17

---

Textual Descriptions
A person a blue sweater and jeans sitting on the floor on top of gray couch with their laptop in their lap. They have yellow Labrador Retriever in their lap, who looking at the camera. The dog has its tongue out and is laying down on the person's lap.
QWen-VL-Che
(Vision Language Model)
Dog laying on lap
Instruction:...
Detection Bounding Boxes
Dog (1, 33, 995, 995) resting head owner's lap (1, 630, 785, 998) while they work on laptop (39, 336, 999, 972)
QWen-VL (Vision Language Model)
Dog laying on human's 's lap
Instruction:
Object Blobs
Dog (1, 33, 995, 995) resting head on owner's lap (1, 630, 785, 998) while they work laptop (39, 336, 999, 972)
(1, 33, 995, 995)
(1, 630, 785, 998)
(39, 336, 999, 972)
Segment-Anythnn (SAM)
Segment-Anythng (SAM)
Segment- Segment-Anything (SAM)
Fit Ellipse
Fit Ellipse
Fit Ellipse
(120, 164, 112, 130, 84)
(384, 881, 138, 486,
(562, 632, 238, 468, 78)
Dog (120, 164, 112, 130, 84) resting head on owner's lap (384, 881, 138, 486, 92) while they work on a laptop (562, 632, 238, 468, 78)
Visual Tokens
SEED (VQ-bused Image Tokenizer)
Figure 9: Pipeline for generating various discrete latents.
18

---

Baseline MDM ImageNet
Kaleido-MDM with Color Clusters Latents (Ours)
AR predicted Color Clusters
Diffusion Generated Image with CFG
Figure 10: Kaleido-MDM with color clusters as latents on ImageNet (class: lemon)
MIM--DPMM
MDM-CCD6
Raloh-DDMM
Kalsh--AAT
(a) Class-conditioned Image Generation (class: Bald
(b) Text-conditioned Image Generation (prompt: cat sleeping the bed).
Eagle).
Figure 11: Qualitative Comparison with CADS
19

---

Baseline Diffusion Model (MDM)
Kaleido Diffusion Model (Ours)
Figure 12: Uncurated samples for both the baseline (MDM) and the proposed Kaleido Diffusion on ImageNet 256 256. The guidance scale is 4.0.
20

---

Baseline Diffusion Model (MDM)
Kaleido Diffusion Model (Ours)
Figure 13: Uncurated samples for both the baseline (MDM) and the proposed Kaleido Diffusion on ImageNet 256 256. The guidance scale is set 4.0.
21

---

teddy wearing glasses
Baseline Diffusion Model (MDM)
Kaleido Diffusion Model (cexcts as latents)
Kaleido Diffusion (bbox as latents)
Kaleido Diffusion Model (blob latents)
Kaleido Diffusion Model (voken latents)
Uncurated samples for both the baseline (MDM) and the proposed Kaleido Diffusion (using text, bxx,bbox,blohh voken latents) on CC12M 256 x 256 given the same condition. We visualize the generated bounding
Figure 14:
moxeding-bbxss boxes and blobs for the ease of visualization. The guidance scale is set 7.0.
22

---

squirzel wearing croum
ShuR
Baseline Diffusion Model (MDM)
Kaleide Diffusion Model (cazzta latents)
tsck
KING YING
LMA
YOU ROIREP
Kaleido Diffusion Model (bbox as latenes)
Kaleido Diffusion Model (blob as latents)
GOD SWED KERT
Kaleido Diffusion Model (voken as latents)
Figure 15: Uncurated samples for both the baseline (MDM) and the proposed Kaleido Diffusion
(using text. text.bbor www.bboxbbblrl voken latents) on CC12M 256 256 given the same condition. We visualize the generated bounding
sunding-boxes boxes and blobs for the ease of visualization. The guidance scale is set 7.0.
23

---

Input: dog reading book
Autoregressivve
Diffusion Generation:
traps, small pag dog etting
beok the
dig sppecrs be neading The dng. beuse in slzned uundmariii
ongespalgg
+
planes anch The Indegrund femoree a larg unndow etgenape math
armeephen.
dog
Diffusion Re- re-ggeerration:
lmags, small pog dog and spp resemblee book. book
undy setotoo Aideefeget
is spen and the dog appeau to la plaudd
org4 oega gollggeeaaeeeenn pogpressnpepeee ppeet gown wwwwrrateraandegrggo
reading
+
auroundad by
sacked www.toroegogggoooooooooo
ou each sida. The bockground fecavaa lerpe window digccape mizoed
yed to peeggres of Beeeeegee--eeee Seepe-l-phlli
Edita: Switch their locations
Diflusion Re-generatio::
the trvege, small prur dog and vop dog resemblre book.
wnly Slowe peree oegeggedoooeeigee oygieet stwirg
and dog reading dng hiuse in plared
popteterneorjjeigggoo goowing yed gooeggreeg perg
euch side The beok bereeged fostures lerg wineine diysoope
Edits: Change book, dog breed, background
Diffusion Re-generation:
In the teags, aitio dog and on oul dtting o dig thet
# pord syersyndorrp forgedige enoomsseetinneg bigeenge baggeeeers tphacilnge Spepliy
readin The fig hause la glocdd on sinndrunti,
backgroand festurss
lerpe window uiit a anoo mevezsin
Edits: Change vokens from Figure (d) Sormer
Diffusion Re-generation:
In the tmags, altbo day and on pul ere anzing o dig thos
the
readin oi The fngg house to gloced sitetrunti
backgrend fersvess
lere windew uett a amoo wenersin
Figure 16: Interactive example of editing the generation process by manipulating the autoregressive predicted latents. The top displays images generated using autoregressively produced latent tokens, and the subsequent rows show the images re-generated after applying editing on the latents. The guidance scale is set 7.0.
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: We made the main claim of "improving diffusion models with autoregressive latent modeling" both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Due to page limits, include discussion of limitations in appendix section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
This is not theory paper
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include the model hyperparameters both in main paper and appendix for people to reproduce. Both datasets used open-sourced.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [No]
Justification: Code will be released after acceptance and internal review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ Selliiiiguides//ooeeeee
icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
CCodeeubeiieeeeeee / //iips.cooc hpppppulluuddeeeeeeeeppppppbpeee httpl///ggiiiee//eeeuubbiiiiii cerbbbmssssinnn ssionnollll seiiinyylcy)) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specified the details about training and testing
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the costly nature of each run, we did not include error bars
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
27

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention the compute requirements in the experiments and appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : http///eerrpp.e//ee cc hccppubicc/tthhicsuudell....
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: This paper conducted in code of ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Due page limits, include societal impacts in the appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
28

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We did not plan to release pretrained models therefore risk for misuse
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited all the papers for codes and datasets we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi sneerswwthcodeee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
29

---

Answer:
[NA]
Justification:
There are no newly released assets
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: There crowdsourcing steps.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "qhcz67hicq", "reviewer": "Reviewer_7a2N", "summary": "In this paper, the authors propose a principled pipeline called Kaleido Diffusion for text-to-image generation with better mode coverage and diversity. The main intuition is that, conventional DM requires large CFG to make samples locate at high-likelihood modes, which would somehow constrain the diversity by only generating those limited modes. To solve this problem, the authors propose to first use AR to capture the distribution of such latent tokens, then use diffusion model to take them as extra conditions for generation. Experiments on both quantitative numeraical results and qualitative visualizations are used to evaluate the effectiveness of this work.", "questions": "They are mainly elaborated in the weakness section. Overall I really like the intuition of this work. I'd like to see the authors' responses on my questions.", "limitations": "The limitations are discussed in the appendix.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The intuition of this work makes sense to me. The excessively large CFG would limit generated samples to certain modes, where using an extra model to capture such text-irrelevant information is a good practice.\n2) The visual quality and especially the teaser of this work can well demonstrate the effectiveness of this paper.\n3) The experiments are thorough to me.", "weaknesses": "1) I still don't well comprehend why choosing AR as the first-stage distribution learner. One of the reason I guess here is that, the CFG in AR model is only modulating the predicted logits, so that we can still sample across multiple reasonable modes for the next-stage DM through temperature parameter or top-k/top-p sampling. However, I'm still a little bit confused by this part: what if we use a diffusion model w/o CFG sampling to learn such an intermediate distribution? Is the model architecture design choice mainly for the fact that the authors use detailed textual descriptions as the latent token, so that it's natural to choose AR? If so, then this is a little bit ad-hoc to me.\n\n2) Choosing these four specific types of latent tokens is also kind of ad-hoc to me. For example, somebody may also say the excessive CFG could result in similar artistic styles in generation. In this way, we have to additionally extract the style token as ground truth, then laboriously train the whole pipeline again to enable it. Another drawback of such design is that: the training cost is large and it's hard to scale up, every time we want to solve the mode collapse of new categories, we have to first extract/tokenize that new aspect, then jointly train the diffusion model and AR part, which is too ad-hoc and hard to scale up to me.\n\n3) I hope the authors could also discuss about the diffusion decoder idea and the difference between that and this work. Specifically, AR + diffusion decoder would similarly take the latent tokens predicted by AR and feed into DM decoder as conditions. The major difference is that their setting doesn't need the ground truth for those latent tokens, which means the latent tokens are implicitly learned via end2end training. My question is what's the benefit of Kaleido compared to their pipeline, and won't their pipeline better at scaling up?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **W1: Reason for choosing AR in modeling mode selection distribution**\n\nWe appreciate your insights and queries regarding our choice of employing an AR model to model $p_\\theta(z | c)$. Our choice is grounded in several pivotal considerations:\n\n- **Why discrete?** \nThe modes $z$ that humans can perceive from an image are largely categorical, abstract, semantic, and high-level information. Such abstract semantics are more easily represented in **discrete** symbols. Moreover, modeling the modes as discrete latents further provides an explainable and editable mechanism for the image generation process. It allows the user to adjust the discrete latent codes before final image production, granting greater flexibility and control over the output. This capability is particularly beneficial in scenarios requiring detailed customization or iterative design processes.  We demonstrate the impact of sequential latent editing in Fig. 8.\n- **Why autoregressive?** Given our objective to model $z$ as abstract discrete tokens, an AR model emerges as the most suitable and convenient method for handling such discrete structures. The inherent design of AR models, which sample one token at a time conditioned on previous tokens, naturally supports the generation of diverse modes.\n\nUsing a diffusion model w/o CFG to learn $p_\\theta(z \\mid c)$ could also serve as an alternative for learning intermediate distributions, however, it presents several challenges. \n- First, employing a diffusion model to model discrete abstract latents is a challenging and ongoing research area. Alternatively, representing these abstract semantics (i.e., modes) with continuous tokens raises fundamental questions about the characteristics and truth distribution of these abstract continuous latents. The challenge lies in accurately defining a distribution that authentically captures the complex, abstract semantics underlying the mode using continuous token.\n- Moreover, even if such a distribution could be defined, diffusion models typically demand high CFG to model it effectively, which circles back to our original challenge with high CFG. Nevertheless, exploring the potential of using diffusion models without CFG as the mode-selection learner remains an intriguing avenue for future research. \n\nAdditionally, we would like to clarify that the AR model does not employ CFG during the sampling of latent tokens. The integration of $p_\\theta(z \\mid c)$ in Equation 8 works to push the updating direction towards the sampled modes $z$ at each step.\n\n\n\n## **W2: Reason for choosing the four specific types of latent tokens**\n- We would like to clarify that we employ various types of latent tokens in order to explore the best ways for representing the modes, each type of latent tokens does not serve as a restricted representative of a particular \"subset\" of modes. It is important to clarify that our goal is **not** to exhaustively cover every conceivable category of diversity for image generation.  In our experiments, we choose text, bbox/blob, and vokens because they are useful for showing different controls.\n- Sometimes, a single representative type of latents like text is sufficient to generate samples that are diverse enough in terms of various aspects. For instance, Fig. 12 in the Appendix.\n\nOur proposed model is a general tool that can cover most of the aspects of diversity. \n- If users seek to create images that focus on the diversity of specific artistic styles, they can manually adjust the discrete latent tokens to reflect desired styles. \n- Alternatively, like other general text-to-image generation models, users may opt to employ techniques such as LoRA to fine-tune the model to achieve enhanced diversity within specific artistic styles.\n\nWe will keep enhancing the clarity of these points in our revised paper until the methodological approach is clearly understood.\n\n## **W3: Difference from AR + diffusion decoder; Scalability**\n\nIn comparing our work with MLLMs using diffusion models as decoders, such as EMU [1][2] and MiniGPT5 [3], \n- A fundamental difference is that their \"autoregressive\" generation of latent tokens is based on regression, which is **deterministic** and typically produces similar images from the same input $c$. This deterministic nature in EMU and MiniGPT5 arises from their training objectives. EMU predicts visual representations $z’$ from the text input $c$, applying an image regression loss with encoded visual embeddings of image $x$ as the ground truth. MiniGPT5 uses an MSE loss to minimize the distance between generated image features $z’$ and the encoded caption feature of text input $c$.\n\n- In contrast, our Kaleido diffusion allows for greater variability by explicitly modeling $p_\\theta(z \\mid c)$ for diverse mode selection. This ability to generate diverse latents distinguishes our work, addressing challenges in generating varied, high-quality images under high CFG.\nFig. R.5 in our rebuttal PDF shows that, unlike EMU, which produces nearly identical images for a given $c$, Kaleido generates varied samples from the same text condition, demonstrating superior diversity.\n\n- Unlike models that jointly train the encoder and diffusion model decoder, Kaleido uses pretrained discrete encoders, offering flexibility and efficiency by reducing training costs and complexity.\n\nRegarding scalability, \n- the total parameter count for our AR model and diffusion model is approximately 1.5B and 500M, respectively, which enables low computational cost.\n- Both AR and diffusion can be trained in parallel jointly, with ground-truth latents pre-extracted. The training cost is similar to the standard language model and diffusion model training.\n\n\n[1] EMU: Generative Pretraining in Multimodality\n\n[2] Generative Multimodal Models are In-Context Learners\n\n[3] MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EKFegDvWFC", "reviewer": "Reviewer_d7gp", "summary": "This paper improves the diversity of diffusion generation by incorporating autoregressive latent priors. It leverages the autoregressive model to generate specific discrete latent features, and then concat them with the original extracted text features to serve as the condition of diffusion model. Experiments show that the method can achieve high quality and diversity even with high CFG.", "questions": "I hope the authors can answer my questions in Weakness section point by point. I am willing to raise my score if my concerns are well addressed.", "limitations": "See Weakness.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "The investigated problem is interesting and critical. By incorporating the intermediate \"mode\" representation, the method can apply CFG after the \"mode\" is sampled, which alleviates the \"mode collapse\" phenomenon. The author introduces four specific modes which can also support fine-grained editing.", "weaknesses": "(1) The writing lacks training and inference details. During training, whether all the modes are used in each step or they are randomly selected? For the visual tokens, whether the latent features from SEED are directly used or the sequence IDs are re-embedded? During inference, the autoregressive model is responsible for generating the latent modes if I understand correctly. Then how different latent modes are generated in control especially the combination of them?\n\n(2) Evaluation is poor. In the experiment section, only a figure is provided without providing numerical numbers. The main experiment setting is confused (class conditioned or text conditioned). The construction of the toy example is also not clearly stated.\n\n(3) Comparison with related works is not sufficient, making the position of this work unclear. There are related works also trying to improve the generation diversity of diffusion models such as [a].\n\n[a] CADS: Unleashing the diversity of diffusion models through condition-annealed sampling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **W1: Training and Inference details**\n\nWe appreciate your inquiry into the specifics of our training and inference methodologies. \n\n- During training, we investigate both isolated and combined uses of different latent tokens, including text, bounding boxes (bbox), blobs, and vokens, to highlight the unique contributions of each token type. Specifically, we train separate models focusing on individual latent types and a combined model that integrates text, bbox, and voken tokens for text-to-image generation.\n\n- Regarding the visual tokens, the latent tokens are formulated as a sequence of discrete image token IDs (\"$I_1$#$I_2$#...#$I_{32}$\"), where each $I_i$ denotes an image token ID (L754-756). Our autoregressive (AR) model re-embeds these image token IDs, resizing its vocabulary to accommodate the special image tokens (L203-204). The original SEED features are not used in our formulation.\n\n- In the combined setting, we include all latent tokens in a shared vocabulary and train them jointly. The AR model predicts **text | bbox | voken** sequentially, allowing later latents to be controlled by earlier ones. We observed that the \"text\" token first expands the semantic aspects (e.g., objects, behaviors) of the generation, the \"bbox\" specifically controls the spatial allocation of described objects, and the visual tokens control the global styles of the image. We plan to explore other combinations in future work.\n\nWe will provide more detailed descriptions of the training and inference details in the revised manuscript to eliminate any ambiguity.\n\n\n## **W2: Quantitative results and Toy examples**\n\n- The main experiment setting for the quantitative experiments is class-conditional generation on ImageNet 256x256, which aligns better with existing works. We also explored training text-to-image models with our methods with mainly qualitative comparisons. \n\n- In response to the concerns regarding quantitative evaluation, we expand our quantitative evaluation to include metrics like the Mean Similarity Score (MSS) and Vendi score, which are used for measuring image diversity following CADS. (**See general response**)\n\n- Regarding the toy example, we construct the toy dataset with two primary classes, each comprising two subclasses with a predefined weight (30% of samples in the first subclass and 70% in the second subclass). Each subclass is sampled from a Gaussian distribution. We train two models for comparison: a standard conditional diffusion model that uses the major class ID as conditions, and a latent-augmented conditional diffusion model that takes both the major class ID and subclass ID as conditions, with the subclass ID serving as latent priors. Both models are trained with classifier-free guidance. We design this toy experiment to show the benefit of latent priors for improving diversity under high guidance.\n\nWe will include these elaborated details and the results from these models in the revised version of our paper.\n\n\n\n## **W3: Comparison with related works**\n\nThank you for suggesting a comparison of our work with relevant methodologies like CADS. \n- We have now included a comparison with CADS in our study for both class-to-image and text-to-image settings. \n- Our results demonstrate that both Kaleido and CADS can effectively enhance the generation diversity of diffusion models, particularly for class-to-image generation tasks like ImageNet, in a relatively orthogonal manner. Additionally, we show that the improvements from Kaleido and CADS can be complementary.  \n- **Please refer to the general response to all reviewers for detailed results and discussion.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5EuJMjTzby", "reviewer": "Reviewer_6rHP", "summary": "This paper introduced Kaleido Diffusion which leverages an autoregressive model to first model the latent mode and then generate latents based on the sampled mode. The proposed method is reasonable. The authors explain the insight from a classifier-free guidance perspective. Several experimental results can support the authors' claim.\n\nIn addition, the paper with the same contents is published as a workshop paper in ICML24.", "questions": "- The dataset section claimed the usage of ImageNet and CC12M. However, the results of ImageNet are missing in the entire paper.\n- Which dataset was used for the results reported in Fig 5?\n- Can you quantitatively measure the diversity?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper investigates how to improve the diversity of generated images with additional mode controls, which is interesting.\n- The writing is clear and the mathematical explanation seems reasonable.", "weaknesses": "- The quantitative results are very limited (only Figure 5). \n- What is the context extractor (MLLM) used in the experiments? Although the authors claimed the mode selection is the major contribution of this work, the additionally enrolled pseudo labels perhaps also contribute to the performance improvements. \n- I am not convinced that the proposed approach is closely connected to the CFG explanation. Several previous studies, eg, ControlNet (ICCV23) and MaskComp (ICML24) have proven that dense controls will improve image quality. The proposed method is more like distilling knowledge from the pre-trained MLLM to obtain more detailed semantic information about the original dataset.\n- It would be better to discuss more implementation details to enhance reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **W2: Use of MLLM/context extractor; Introduction of pseudo labels**\n\nThe context (latent) extractor is employed to extract different types of abstract latents given the condition-image pair (Sec. 3.2). In practice, for different types of latents, we utilize different methods as the context (latent) extractors. The construction of the training dataset for these latent tokens is explained in Appendix A.\n\nWe appreciate the reviewer's concern regarding the potential impact of pseudo labels on our results. However, it is crucial to clarify that the improvements in our work are not solely due to the introduction of pseudo labels. Simply introducing pseudo labels, without modeling them as latents, would not alone lead to the observed improvements in image diversity.\n\n- To illustrate this point, we have conducted additional training with the MDM (baseline) model, using the generated synthetic textual description (i.e., pseudo tokens) as the condition for training. In Fig.R.3 of the rebuttal PDF, we demonstrate that merely using pseudo labels in this manner does not enhance the diversity, underscoring that the effectiveness of pseudo labels depends significantly on the thoughtful integration into the modeling process. For both models, all images are generated using DDPM with 250 steps and a CFG of 5.\n\n----\n\n## **W3: The CFG explanation & knowledge distillation**\n\nWe appreciate the reviewer’s reference to prior studies, such as ControlNet [1] and MaskComp [2]. While these contributions are significant, they are not designed to improve image diversity given the same input condition $c$.\n- In ControlNet, the text condition and dense control (e.g., canny edge) together form the user input $c$. Although using different additional controls (considered as different modes, $z′$ can produce diverse outputs, this approach differs from our focus. Our objective is to enhance diversity without requiring additional user-provided information. \n- Similarly, MaskComp [2] addresses the object completion task through iterative generation and mask segmentation but does not prioritize diverse generation of partial masks. \n\nOur design is closely connected to the CFG explanation:\n- In a standard diffusion model, increasing CFG sharpens the conditional distribution $p_\\theta(c \\mid x)$, leading to a reduction in diversity. To address this, we introduce an additional variable $z$, representing various “modes” of $z$. We propose explicitly modeling “mode selection” $p(z \\mid c)$ before applying CFG in diffusion steps, ensuring that the mode distribution is not distorted by guidance (see Equation 8).\n- We propose using an AR model to learn mode selection $p(z \\mid c)$, leveraging a synthetic condition-latent pair $(c, z)$ dataset. This can be viewed as **a form of knowledge distillation from pre-trained models**. However, we kindly ask the reviewer to consider the broader design and intent of our entire framework. Introducing the abstract latent variable $z$ strategically enhances the utilization of distilled knowledge. Without the latent modeling and sampling in our proposed framework, merely distilling from pre-trained models would not alone lead to the observed improvements in diversity. Our work provides a theoretical explanation from the CFG perspective on how explicitly modeling \"mode selection\" helps mitigate the issue of mode collapse under high CFG (Sec. 3.1).\n\nTo address concerns about the reliance on distilled knowledge, we have included an alternative approach that constructs latent tokens without relying on additional knowledge. \n- Specifically, we train a model using **color clusters** as latent tokens. For each color channel (R, G, and B) within the range of 0-255, we equally segment it into eight clusters, resulting in a total of $8 \\times 8 \\times 8 = 512$ color clusters. Given an image, we resize it to 4x4 pixels and assign a color cluster ID to each pixel based on its RGB value. The image is then encoded into a sequence of color cluster IDs (e.g., \"$C_1$#$C_2$#...#$C_{512}$\"), with each $C_i$ representing a color cluster ID. This sequence serves as the condition for training our Kaleido diffusion.\n- In Fig.R.4 of our rebuttal PDF, we showcase images generated using color clusters as latent tokens on ImageNet. Our results demonstrate that, compared to the baseline MDM, our Kaleido diffusion can generate much more diverse images with latent tokens derived purely from color clustering. This highlights that Kaleido diffusion's capability to generate diverse images is **independent of distilled external knowledge**, confirming that our approach can produce varied images without the aid of any other pretrained models.\n\n\n[1] Adding Conditional Control to Text-to-Image Diffusion Models\n\n[2] Completing Visual Objects via Bridging Generation and Segmentation\n\n----\n\n## **W1: Quantitative Results / Q3: Quantitative measure of diversity**\n(**See general response**)\n\n----\n\n## **W4: Implementation Details**\nThank you for your valuable feedback. To enhance the reproducibility of our work, in the revised version of the paper, we will provide more implementation details including the model parameters, training/inference hyperparameters, and any additional setup information necessary to replicate our results in the revised version.\n\n----\n## **Q1: Missing results on ImageNet?**\nThe results in ImageNet can be found in Fig. 5, the top row of Fig. 7, and Fig. 10 and 11 in the Appendix. We will make this clear in the revised version that these figures illustrate the performance on the ImageNet.\n\n----\n## **Q2: The dataset used in Fig. 5**\nThe dataset utilized for the results presented in Fig. 5 is ImageNet. We will ensure that this information is explicitly stated in the revised version", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "16rAulcJr3", "reviewer": "Reviewer_PSab", "summary": "Diffusion models generate high-quality images from text but often lack diversity, especially with high classifier-free guidance. Kaleido addresses this by using autoregressive latent priors, which generate diverse latent variables from captions. It enriches input conditions, resulting in more diverse outputs while maintaining quality. Experiments confirm Kaleido's effectiveness in increasing image diversity and adherence to guidance.", "questions": "1. What is the motivation behind introducing visual tokens in the autoregressive prior? From the overall narrative of your paper, it seems that the autoregressive prior is intended to provide more customized control for users. However, visual tokens are essentially uninterpretable, making them unusable for user control.\n2. The generation process involves an autoregressive process at the beginning. How does this process affect the efficiency of your method?", "limitations": "More results on the efficiency and quantitative performance of your method are encouraged.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Due to the tendency of samples to converge towards the direction indicated by the condition under high CFG settings, the approach of generating abstractions first to expand the diversity of conditions and then proceeding with generation seems a very reasonable approach. Moreover, these abstractions are controllable, enhancing interpretability and customization possibilities, which is also favorable.\n2. The qualitative results also appear promising.\n3. The approach of constructing fine-grained prior via autoregression is novel to my knowledge", "weaknesses": "1. The formulation in Section 3.1 seems somewhat unintuitive. From my perspective, both text descriptions and the additional autoregressive priors you construct are forms of condition signals for diffusion models. Therefore, I do not see a compelling reason to modify or complicate the original classifier-free guidance formulation. Why not simply regard your method as an extension of the condition signal for classifier-free guidance, while still adhering to the existing CFG formulation?\n2. The quantitative results are somewhat limited, with MDM being the only baseline. More comparisons with state-of-the-art diffusion models are encouraged.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **W1: Explanation of the formula in Sec. 3.1**\n\nThank you for your valuable feedback.\n- In our practical implementation, we indeed adhere to the existing classifier-free guidance (CFG) formulation. Both text descriptions and our introduced autoregressive priors indeed serve as conditioning signals for the diffusion models. Our sampling method can indeed be seen as an extension of the conditioning signal in CFG, specifically through the incorporation of autoregressive priors.\n- The formulation presented in Section 3.1 is designed to elucidate how these autoregressive priors effectively address diversity issues when operating under high CFG settings. Therefore, this is indeed a mathematical motivation rather than an actual implementation.\n\nWe will ensure these points are more clearly articulated in the revised version.\n\n\n## **W2: More Quantitative Results and comparison with sota diffusion models / L1: More quantitative Results:**\n\nThank you for your feedback! \n- Our proposed framework integrates an autoregressive prior with a diffusion model to enhance image generation. Theoretically, this approach is compatible with a variety of backbone diffusion models. In this paper, we focused on comparing our model with the MDM baseline in pixel space to achieve a fair comparison and clearly demonstrate the specific improvements our framework offers. As part of future experiments, we will also apply our approach to other model types, such as latent diffusion and flow matching.\n- **Please refer to the general response to all reviewers for detailed results and a discussion** of the comparison between our method and CADS—a state-of-the-art, training-free approach that enables diverse generation from diffusion models.\n\n\n## **Q1: Motivation Behind Introducing Visual Tokens**\n\n- The primary motivation for incorporating the autoregressive prior (i.e., latent tokens) is to explicitly model the mode selection distribution $p(z \\mid c)$, enabling the generation of diverse image samples from the same condition, even under high CFG settings. This approach also introduces an explainable and editable mechanism into the image generation process. Our experiments demonstrate that visual tokens (vokens) complement text tokens and are particularly effective at capturing visual details difficult to convey through text, such as artistic style, as shown in Fig. 6 of the main paper.\n- We would also like to clarify that vokens are **interpretable** and **can be manipulated by users for image editing**. The training dataset of visual tokens is constructed by encoding the images into discrete image tokens using SEED [1], a VQ-VAE-based image tokenizer [2] (L748-756). Therefore, we can interpret the generated visual tokens by decoding them back into images using the corresponding image de-tokenizer (i.e., decoder). User control over the image editing process can be achieved by replacing the vokens with different visual tokens. In Fig. R.6 of our rebuttal PDF, we show that by replacing the visual tokens, users can alter the style and characteristics of the re-generated image.\n\n[1] Planting a Seed of Vision in Large Language Models  \n[2] Neural Discrete Representation Learning\n\n## **Q2: Model Efficiency / L1: Results on Efficiency**\n\nAR sampling is performed only once before the diffusion steps, making its running cost negligible. \n- Specifically, in text-to-image settings for generating 256x256 images with classifier-free guidance (batch-size=32), the AR part takes **6 seconds** while the MDM part takes **110 seconds** on a single H100 GPU when using DDPM with 250 steps, which dominates the majority of the time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **Quantitative Comparison with CADS:**

In response to the request for more quantitative results and comprehensive baseline comparisons, we have conducted additional experiments, specifically comparing our Kaleido diffusion model with CADS [1].

- **Condition Annealed Diffusion Sampler (CADS)** is a general sampling strategy that enhances the diversity of diffusion models by annealing the conditioning signal during inference.

- Following CADS, we employ two additional quantitative assessments of diversity: Mean Similarity Score (MSS) and Vendi scores. We use SSCD [2] as the pretrained feature extractor for calculating both MSS (SSCD) and Vendi (SSCD). Additionally, we utilize DiNOv2 [3] as the feature extractor for Vendi (DiNOv2), based on evidence from [4] suggesting that DiNOv2 provides a richer evaluation of generative models.

- Given that CADS is a training-free strategy applicable to different model architectures, we integrate CADS with both the baseline model MDM and our Kaleido-MDM. We emphasize that the contribution of CADS is **orthogonal** to our work, and its application is independent and complementary to the core methodologies in our research.
-----
## **Results**
We report the results for class-conditional generation on ImageNet 256×256 in Table 1 and 2, and for text-conditional generation on the MSCOCO [5] validation set in Table 3. All models use DDPM sampling with 250 steps. 
- Table 1 presents a quantitative comparison based on evaluations from 50K samples. Our Kaleido diffusion outperforms the MDM + CADS combination in terms of FID-50K and precision, demonstrating that our method more effectively maintains high image quality while generating diverse samples. Furthermore, when we integrate CADS with our model, we achieve the best FID-50K results. Note that Precision cannot accurately evaluate models with diverse outputs since a model producing high-quality but non-diverse samples could artificially achieve high Precision [1].
- In Table 2 and 3, following CADS, we assess the diversity of the generated images using 10K samples. For Table 2, we select 1,000 random classes from ImageNet and generate 10 samples per class. For Table 3, we use 1,000 random text prompts from the MSCOCO validation set and generate 10 samples for each prompt. Our findings indicate that **both our Kaleido model and CADS significantly enhance sample diversity**. Although CADS achieves better performance in diversity, our model maintains superior image quality, as shown in Table 1. 
- Additionally, **the methodologies used in CADS are complementary to ours, suggesting potential benefits from integrating CADS with our Kaleido model**. In fact, incorporating CADS into our model not only further improves image quality but also improves diversity, achieving the best scores in FID-50K, MSS (SSCD), and Vendi (DiNOv2) in class-conditioned image generation, and best Vendi (DiNOv2) in text-conditioned image generation.
- Lastly, our **rebuttal PDF** includes visual comparisons of these models for class- and text-conditioned image generation in Fig.R.1 and 2, respectively. All images are generated using DDPM with 250 steps. Specifically, in Fig.R.2, we observe that MDM + CADS fails to generate cats of diverse breeds from the prompt "a cat sleeping on the bed." In contrast, our Kaleido diffusion model excels, producing images of cats from various breeds with more diverse surrounding environments, showcasing its superior diversity capabilities. This observation contrasts with the trend of diversity scores in Table 2, suggesting that these diversity metrics may not fully capture certain aspects of diversity.
-----

**Table 1: Comparison on 50K samples of ImageNet, CFG=5.0**

| Model |                     FID-50K ↓ | Precision ↑ | Recall ↑ |
|----------|----------|----------|----------|
| MDM                              | 15.5       | **0.93**  | 0.22   |
| MDM + CADS                 | 10.6      | 0.60        | **0.62**   |
| Kaleido (ours)                  | 9.0       | 0.85        | 0.42  |
| Kaleido (ours) + CADS    | **5.9**  | 0.76       | 0.52 |

-----
**Table 2: Diversity Comparison on 1K x 10 Samples of ImageNet**

| Model               | MSS (SSCD) ↓ | Vendi (SSCD) ↑ | Vendi (DiNOv2) ↑ |
|------------------|----------|----------|----------|
| MDM                             | 0.21         | 8.42        |  3.04   | 
| MDM + CADS                | **0.12**   | **9.28**  | 4.72   | 
| Kaleido (ours)                 | 0.16        | 8.82        | 3.79   | 
| Kaleido (ours) + CADS   | **0.12**   | 9.21       | **4.83**  | 

-----
**Table 3: Diversity Comparison on 1K x 10 Samples of COCO Val**

| Model               | MSS (SSCD) ↓ | Vendi (SSCD) ↑ | Vendi (DiNOv2) ↑ |
|------------------|----------|----------|----------|
| MDM                             |    0.29           | 7.55            | 3.39  | 
| MDM + CADS                |   **0.18**      | **8.65**      | 4.60   | 
| Kaleido (ours)                 |  0.20           | 8.52            | 4.59   | 
| Kaleido (ours) + CADS   |  0.19          | 8.61           | **4.75**  | 

------
------

[1] CADS: Unleashing the diversity of diffusion models through condition-annealed sampling.

[2] A Self-Supervised Descriptor for Image Copy Detection.

[3] DINOv2: Learning Robust Visual Features without Supervision.

[4] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.

[5] Microsoft COCO: Common Objects in Context

Meta-review: This paper has received consistent feedback from four reviewers, all of whom are in favor of acceptance. The reviewers engaged in thorough discussion and rebuttal, ultimately reaching a consensus. This paper proposes Kaleido Diffusion for text-to-image generation. The method enhances the diversity of generated images, and the resulting latent variables are interpretable. Therefore,  the AC  has decided to accept this paper.
Decision: Accept (poster)