Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models
Authors: Samuel Lavoie
Michael Noukhovitch
Aaron Courville
Venue: NeurIPS 2025 poster ()
Keywords: Generative model
diffusion model
representation
discrete
Primary area: deep_learning
TL;DR: We show that discrete representation of images improve unconditional and compositional generation

Abstract: We argue that diffusion models' success in modeling complex distributions is, for the most part, coming from their conditioning. This paper investigates the representation used to condition diffusion models from the perspective that ideal representations should improve modeling the data distribution, be easy to generate, and be compositional to allow generalizing outside the training distribution. We introduce Discrete Latent Code (DLC), an image representation derived from Simplicial Embeddings trained with a self-supervised learning objective. DLCs are sequences of discrete tokens, as opposed to the standard continuous image embeddings. They are easy to generate and their compositionality enables sampling of novel images beyond the training distribution.
Diffusion models trained with DLCs
improve generation fidelity, establishing a new state-of-the-art for unconditional image generation on ImageNet. Additionally, we show that composing DLCs allows the image generator to produce interesting out-of-distribution samples that coherently combine the semantics of images in diverse ways.
Finally, we showcase how DLCs can enable text-to-image generation by leveraging large-scale pretrained language models. Using only 9M image-caption pairs, 
we efficiently finetune a text diffusion model to generate novel DLCs that produces samples outside of the data distribution used to train the image generator.

Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models
Samuel Lavoie" Michael Noukhovitch Aaron Courville Mila, Université de Montréal
Abstract
We argue that diffusion models' success in modeling complex distributions is, for the most part, coming from their input conditioning. This paper investigates the representation used to condition diffusion models from the perspective that ideal representations should improve sample fidelity, be easy generate, and be compositional to allow out-of-training samples generation. We introduce Discrete Latent Code (DLC), an image representation derived from Simplicial Embeddings trained with a self-supervised learning objective. DLCs are sequences of discrete tokens, opposed the standard continuous image embeddings. They easy to generate and their compositionality enables sampling of novel images beyond the training distribution. Diffusion models trained with DLCs have improved generation fidelity, establishing new state-of-the-ar for unconditional image generation on ImageNet. Additionally. show that composing DLCs allows the image generator to produce cit-of-diitribution samples that coherently combine the semantics of images in diverse ways. Finally, we showcase how DLCs can enable text-to- text-to-image generation by leveraging large-scale pretrained language models. We efficiently finetune text diffusion language model to generate DLCs that produce novel samples outside of the image generator training distribution. Code available: http:://ghhbbbcooommaoeemsssssee
Introduction
Denoising diffusion models [Ho et al., 2020] have demonstrated incredible capabilities for generating high-fidelity images [Peebles and Xie, 2023]. To achieve this feat, state-of-the-art methods [Ramesh al., 2022, Rombach et al., 2022] generally train by conditioning on image labels or text captions [Radford et al., 2021]. Yet, strong diffusion models still generate images that exhibit low diversity or don't 't' realistically reflect complex input prompts [Astolfi al., 2024]. We posit that these failures of diffusion models are rooted in their inability to fully model the data distribution and can be alleviated by conditioning diffusion models on a better representation of the data. We argue that an ideal representation should (1) lead to hightfidelity generation of the data and (2) be compositional [Fodor and Pylyshyn, 1988, Hadley, 1997] to enable a generative model to produce novel images outside the training distribution by recomposing parts of images seen during training; i.e. enables productive generation [Edelman and Intrator, 2000].
A common choice for conditioning diffusion models is with a text prompt or caption. Natural language flexible representation of the world [Whorf, 1956, Wittgenstein, 1953, Niu et al., 2024] that easy to transmit and learn [Tomasello, 1999, Smith and Kirby, 2008], and is compositional [Johnson, 2004]: producing novel meanings by composing known words new ways. However, text captions are poor descriptors of images [Foucault, 1966, Lavoie et al., 2024] as they only capture a few concepts of an image while excluding everything else e.g. background subject, contextual items, quality/resolution of the image itself. Consequently, diffusion models conditioned directly on a text
*Correspondncce to lamocc. www.iil.mmmmmmmmmmmmmmm
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Unconditional generation.
(b) Semantic compositional generation.
Figure 1: Selected samples generated from a DiT-XL/2 with DLC512 for both in-distribution and out-of-distribution (OOD). Model trained on ImageNet 256 x 256 conditioned on Discrete Latent Code of 512 tokens. Left:: Samples from unconditional generation. Right: OOD samples of semantic compositional generation by conditioning on diverse compositions of two DLCs corresponding to (1) jellyfish and mushroom, (2) komodor and carbonara and (3) tabby cat and golden retriever.
representation [Radford et al., 2021], such Stable Diffusion [Esser et al., 2024], often struggle to produce images consistent with the prompt [Huang et al., 2025]. Though modern text-to-image models have leveraged text create intricate, novel images [Ramesh al., 2022], they often miss the desired semantics of an image (e.g. ignoring word order [Yuksekgonul et al., 2023]). One solution enhance the captions to include more information about the image [Urbanek al., 2024], but this is an expensive labelling task with humans and induces hallucinations when done with another model [Liu et al., 2024].
An alternative to condition the generative model on a learned image representation, Specifically, image embeddings trained with self-supervised learning (SSL) [Hjelm et al., 2019, Oquab al., 2024] are structured and more expressive than captions. But the standard embedding is continuous, and has two issues: (1) it is difficult to learn continuous distribution in order to sample from it (Section 3), and (2) they are generally not flexibly composable and cannot combine the semantics of two images in diverse ways (Subsection 5.2).
We propose to condition diffusion models with a representation that combines the benefits of both image and text representation. We condition generative model with Discrete Latent Code (DLC), a sequence of discrete image tokens. DLCs are derived from Simplicial Embeddings (SEMs) [Lavoic et al., 2022] that SSL method. We show that DLCs improve data modeling and are easy to learn, achieving the state- of-the-art FID for unconditional generative modeling ImageNet (example shown in Figure 1a). By leveraging a discrete representation extracted from a SSL representation, DLCs are compositional, such that DLCs can be composed and conditioned on by diffusion model to produce diverse novel OOD images as composition of image features (shown in Figure lb). Finally, we connect DLCs to large-scale pretrained language models create a -ottt------tt-aaaeee pipeline. We show that DLCs conditionally generated from text prompt can be used to generate images outside of the image generative model's 's training distribution.
sequence of distributions over vocabulary of image tokens Icarned with an
2 Background
Continuous diffuusin model. Throughout this paper, denote the observation data as x Pdata (2)). Denoising diffusion probabilistic models (DDPM) [Sohl-Dickstein et al., 2015, Ho et al., 2020] learns to iteratively reverse pre-defined stochastic noising process that transforms data into unstructured noise distribution over time. Typically, this process is defined via a Gaussian transition kernel that gradually perturbs input sample by following schedule over T timesteps in [0, 1]. Assume that Po() = (2') is our data distribution and p1 = =N I) is an isotropic Gaussian. DDPM defines the forward noising process follows:
Ct ag + OFC,
Pdnta (X), Pl.
(I)

---

(a) Training: 441 mixtures
(b) Unconditional generation
(c) Conditional generation
Uncond.
0.8
4.5
KLDiPamm [Po]
Oracle
4.0
GMM
8
10
0.6
3.0
250
500
750
0.2
2.5
Number of mixtares
(d) KLD wrt. mixtures
(e) Unconditional
(f) Conditional e
Figure 2: Unconditional diffusion gets worse at fitting distribution the number of modes increases. (a) Samples from the training data distribution Pdata with 121 mixtures. (b) Samples from unconditional diffusion model trained with Puat- (c) Samples from conditional diffusion model trained trained with Paata and the ground-trume truth mixture index. (d) KL divergence between Pdata and the modeled distribution py degrades as the number modes increase. Generations conditioned on an oracle index representing the mixture centroid and generation conditioned (GMM) have good fit to highly modal distributions. (e, f) Heatmaps of the magnitude of the estimated score 8o and vector fields with respect to the coordinate for the unconditional and the conditional generative models respectively. For d), we condition the score network on mixture index c 6.
increase the number of mixtures. Unconditional' 's fit of the distribution
index inferred from Gaussian Mixture Model
where og and og are defined according to a noise schedule (e.g. [Nichol and Dhariwal, 2021
Continuous diffusion models train parameterized score network (2',, c) whose input is the noise sample It, the timestep and an optional conditioning input (e.g. a label, a vector, a discrete code) c related to the input sample. The score networks are trained to estimate the noise vector with the denoising score matching objective [Hyvärinen, 2005, Vincent, 2011] at all time steps. The DDPM objective is defined mean squared error loss between the estimated noise vector time step t and the actual noise vector:
L(0)= = (z,, (10 cl|2.
(2)
The score network may be used to define the reverse process which transforms sample from the prior distribution Pı into sample from the data distribution using SDE solvers.
Generating highly modal continuous distributions hard
Diffusion generative models work remarkably well on datasets with low diversity [Ho et al., 2020] such as LSUN [Yu et al., 2016] or when conditioned on label [Peebles and Xic, 2023], or a text caption [Ramesh et al., 2021, Rombach et al., 2022]. However, large-scale diffusion models struggle to fit datasets with high diversity such as ImageNet [Li et al., 2024a], without conditioning, section, demonstrate that this issue be replicated with a simple toy dataset. We show that unconditional diffusion models exhibit degenerate fit to the data distribution as the number of modes in the distribution increases. More precisely, given Paata ground truth training distribution and pe the model distribution. For a fixed model capacity and compute budget, the of pa to Paata deteriorates the number of modes in Pdata increases.
this
Dataset and model. Our data a mixture of N Gaussians on square grid. For all N, we keep the variance fixed at 0.1 and the distance between the center of two neighbouring Gaussians fixed at

---

DLC Encoder
Composing DLCs
SEM encoder
length L
DIT
Argmax
Image
vocab
Discrete Latent Code
DLC
Texte Generation
Unconditional Generation
Bonsai Tree
LLADA
DIT
SEDD
DIT
Prompt
DLC
Mask
DLC
Figure 3: Discrete Latent Codes (DLCs) are Top Left: the output of finetuned DINOv2 with SEM,
followed by argmax over the vocabulary. Top Right: can generate semantically compositional images from a composition of two DLCs by selecting tokens from either code. Bottom Left: we enable text-to-image generation by finetuning text diffusion model for leXt-to-DLC sampling, Bottom Right: we sample unconditionally by first sampling DLC with SEDD then conditionally sampling image with DIT.
1.0. Our diffusion models are 4-layers MLPs trained using the improved DDPM procedure [Nichol and Dhariwal, 2021] with a batch size of 2048 and trained for 300M samples.
Quantifying the degrading fit. We provide example of the training data Pdata for N = 400 in Figure 2a and observe a very poor to the data distribution from unconditional generation as shown in Figure 2b. Meanwhile, conditional generative models have good fit to the training data as show in Figure 2c. We quantify the fit using the KL Divergence (KLD) between the ground truth distribution and the modeled distribution in Figure 2d. We estimate the density of the modeled distribution using the Gaussian Kernel Density estimation with bandwidth estimated using the Silverman rule. We find that the fit of the unconditional diffusion model quickly degrades (KLD) increases) with the number of mixtures. Yet the same diffusion model trained with the same setup but conditioned oracle index representing the centroid of the mixture leads to good fit for highly modal distribution. Conditioning on a learned index inferred via a Gaussian Mixture Model (GMM) nearly identically good. We demonstrate in Figure 2f that the vector fields of conditional diffusion are simpler, linearly directing toward the mode. Comparatively, unconditional diffusion model learn more complex flow depicted in Figure 2e.
Discussion. These results illustrate how training unconditional diffusion models is harder on diverse, continuous datasets, and that conditioning continuous distributions with a large number of modes is difficult for diffusion models as it requires more complex generative process than conditional generative models. Generating high-fidelity samples could be possible we could represent the data to make easier to learn and sample. In the next section, show that images can be represented as a sequence of discrete tokens and that such representation is easy to learn and provide a good conditioning to image generative models.
representation, is effective solution. Modeling
4 Generative models with Discrete Latent Code
Related work. While earlier approaches on conditioning diffusion model primarily relied on labels text captions [Dhariwal and Nichol, 2021, Rombach al., 2022], recent works have explored conditioning on image embeddings [Preechakul et al., 2022, Bordes et al., 2022, Harvey and Wood, 2023, Pernias et al., 2023]. Like DL.. some works have conditioned generative models on discrete latent codes of an image LLaviieMMrrchille et al., 2020, Bao et al., 2022, Hu et al., 2023, Wang et al., 2023, Xu al., 2024] but they are generally using short codes of small dimensionality. contrast,
Forr sufficiently small variances distances between Gaussians, we found that diffusion models struggle to capture the fine detail extant in the data [Karras et al., 2024].

---

we represent images as a longer, high- -igee-iiniiioall codes and leverage ofate-ofthhe-at SSL objectives for learning the image codes. This enables the code to represent more fine-grained features than prior work and to generally improve on image modelling. Several approaches have been proposed for learning discrete encoding [Oord et al., 2018] for image generation [Esser et al., 2021, Chang et al., 2022, Yu et al., 2022, Xu et al., 2024]. However, these approaches require back- propppropgg through the dinnttiscetiiztionn, necessitating a gradient estimator. Instead, we learn the discrete code using SEMs [Lavoie et al., 2022] which does not necessitate gradient estimator. Recently, Li et al. [2024a] improved unconditional generative models by conditioning them on a continuous latent image representation. Their approach projects embeddings via a randomly initialized and frozen network, which makes them easily learnable but not compositional. Tangentially, REPA [Yu et al., 2025] also leverages DINOv2 but improves label-conditioned diffusion by aligning its latent representation with DINOv2.
Inferring Discrete Latent Codes. We leverage SEM encoder [Lavoie et al., 2022] trained via a distillation objective [Zhou et al., 2022, Oquab et al., 2024]. Let co(2:) E Rd an encoded representation. Each simplicial embedding S o, (ev(z) W4) is projection of the encoded representation onto the V-dimensional simplex, with a learnable linear projection W; E R followed by temperature
softmax σ,, Given a sequence of SEMs (S1. S2 ..., SL). we infer a discrete latent code defined, by taking the argmax of each SEM. The DLC is thus defined as: T; = arg max S,, i E [r., c = (Ti, T2,.... Tr.) where T; is a token that takes a value in NY. We show an overview in Figure 3.
Improving unconditional generation with DLC. As discussed in Section 3, learning generative model on p(x) may be hard, specifically in cases where p(æ) is highly modal. This work proposes to model p(x) as the product of two generative models that are easier learn. Specifically,
a
p p(xlc) p(c)
(3)
hard
easy
easy
Practically, sampling from p(æ) can be achieved by ancestral sampling. First, sample from p(c). Conditioned on the sampled code e, sample the image p(æ]cc.
DLC conditioned diffusion models. Given an image æ and its associated discrete latent code c, we train conditional denoising score matching network sæ using Equation 2 to model p( p(z|c). As discussed in Section 3, p( x cc) can be easy to model when c facilitates the conditional generation; i.e. is an expressive and well structured representation of x. For image generation, we argue that such representation of an image, can be obtained from a SOTA SSL encoder.
Unconditional generation of DLC. For Equation 3 to hold true, p(c) has to be easy to model. Highly modal continuous distributions are hard to model with diffusion models as discussed in Section 3. Thus, a continuous code extracted from an SSL encoding of the image may also be equivalently hard to model. In contrast, large language models have shown the ability to model internet scale natural language, which are discrete and compositional codes representing highly diverse semantics. Thus, we argue and show that a discrete and compositional representation, such as DLC, of a diverse dataset of images p(c) is easy to generate too. Given that DLC non autoregressive, we propose to model p(c) with a discrete diffusion model [Austin et al., 2021]. specifically SEDD-Absorb [Lou et al., 2024]. SEDD-Absort samples discrete code by iteratively unmasking fully masked-sequence. The token to be unmasked is determined via a learned concrete score s/p C x R + RR V which estimates diffusion matrix that controls the mass transition from the mask token to the DLC token. Thus, &o allows us to estimate the transition probability P:- Arts (Ct (e--sllc) (c)) and sample from p(c) via the reverse diffusion process using e.g. the Tweedie r-leaping [Lou et al., 2024] simulation algorithm.
Remasking DLC. Recently, has been shown that remasking tokens improves sampling of discrete diffusion models (Wang et al., 2025]. While remasking is not required for sampling DLC, we found that remasking improves the generation quality of images. Thus, we also introduce a remasking strat- egy for SEDD-Absorb. Given the approximated posterior Pt- Pr-Aej((c-Aal C:). Let n, the probability of remasking a token. Following [Wang et al., 2025], we apply the remasking on unmasked token (i.e. ct F m) in the interval & [fo, f1] and = n if t [to, f1] and a O otherwise. We define the posterior with re-masking:
p Pi-setjje(C--s (cb--)44))))))) Cat((1 - o) + om) if ct 4 m else vi- A4 111 (c)))))))
(4)
where 5 = o. (1 -t) t) a correction term the step to take into account the remasked tokens.
5

---

FID SFID IST PRET RECT
10
DiT
Unconditional
RCG
3.44
186.9
SIT
DLC512
2.00
4.65
260.7
0.79
0.61
8
Conditional
MAR-H
2.35
227.8
0.79
0.62
SIT + REPA
SiT-XL/2
8.61
6.32
131.7
0.68
0.67
6-
REPA
5.90
DiT-XL/2
9.62
6.85
121.5
0.67
0.67
Uncond. w/ CFG
4
DisCo-Difft
3.70
RCG
2.15
253.4
DiT DLCs12
DL.C512
1.59
4.16
255.4
0.81
0.63
2-
2
Cond w/ CFG:
Training iteration (M)
SiT-XL/2
2.07
4.49
277.5
0.83
0.59
REPA
1.42
4.70
305.7
0.80
0.65
Figure 4: DLC greatly improves training ef- ficiency for FID without CFG on ImageNet.
DiT-XL/2
2.27
4.60
278.2
0.83 0.57
Evaluating FID w/o CFG during intermediate Table I: DLC achieves SOTA FID, sFID steps, DLC is already improving on vanilla DiT conditional ImageNet generation while reducing performance at 1/4 of the steps. Bascline num- reliance CFG. Bascline numbers taken from pub- bers taken from Yu et al. [2025]
lished works. On ImageNet 64 x 64.
Image generation with DLCs
We investigate the impact of conditioning image diffusion models on Discrete Latent Code (DLC) inferred from a SEM encoder [Lavoie et al., 2022]. We demonstrate that diffusion models DL.C:
1. push the state-of-the-art on unconditional ImageNet generation (Table 1, Figure 4),
2. outperform generative model conditioned with continuous SSL embeddings (Table 2),
3. exhibit compositional generation (Figure 6),
4. exhibit increased image generation FID as we increase the sequence length (Figure 5).
Setup for inferring DLC. Following REPA [Yu et al., 2025], we leverage pre-trained DINOv2 ViT- L. encoder [Oquab et al., 2024] and a randomly initialized linear projection layer, projecting its output into SEMs. Additionally, we use a randomly initialized DINO head that takes the concatenated SEMs as input. We investigate three SEMs configurations with the same total number of dimension (131 072): (1) 32 tokens of dimension 4096 (32 x 4096), (2) 128 tokens of dimension 1024 (128 x 1024) and (3) 512 tokens of dimension 256 (512 256), to systematically understand the trade-off between the number of tokens, the vocabulary size and their effective capacity. For the same number of tokens, larger sequence length represent more combinations (i.e. 512256 1024 409632).
We use the DINOv2 codebase with minimal modifications to support SEM training and we re-use the same hyper-parameter used for DINOv2 pre-training. We train the SEM encoders on ImageNet- 1K [Russakovsky et al., 2015] for 100 epochs. After pre-training, we associate each image with its DLC by taking the argmax across the SEMs outputs, shown in Figure 3.
Setup for training image diffusion. Our image diffusion experiments build upon the DiT code- base [Peebles and Xie, 2023]. We use the DiT-XL/2 architecture for latent diffusion on VAE-encoded images [Rombach et al., 2022]. We minimally modify the code to replace class label conditioning with either DLCs or continuous self-supervised embeddings, both of which are pre- pre-computed. DiTs are trained on ImageNet for up 800 epochs, using the same optimizer, learning rate, global batch size, sampling strategy, and other hyperparameters as in the original DiT implementation. We embed the tokens of the discrete latent code with an embedding matrix, as commonly done for embedding the label [Dhariwal and Nichol, 2021]. These embedded tokens are averaged to form the conditioning input to the DiT [Peebles and Xie, 2023]. We add an additional drop-token to the embeddings to also train unconditional model and to leverage classifier-fre free guidance [Ho and Salimans, 2022].

---

14
632xx4006
DEC pope shpp
4086
32 4095
stape
SED0 snakt sloc
emall
2-
1.3
+125 x 1024
512 256
1034
50
128 1024
20
3.8
SID0 foroudd FLLDDs
Triiing neation (M)
Thiining hemitio [K]
(a) DLC generator.
(b) Image generator.
(c) DLC generator.
(d) DLC generator.
Figure 5: Scaling analysis of DLC: trade-off between performance and compute controlled via the sequence length. (a) FID with respect to compute FID and compute scale with the sequence length of DLE. (b) and (c) Training a generative model to generate long sequence length and training an image generative model conditioned on long sequence length converge sequence length are more sensitive to the model size and attain lower FID. Results obtained without CFG nor remasking. Unless mentioned otherwise, DiTs are trained for 500 epochs.
lower FID. (d) Larger
Setup for latent embeddings diffusion. We use SEDD [Lou et al., 2024], a discrete diffusion model, to generate DLCs corresponding to ImageNet images, We closely follow their training recipe but instead of text tokens, our model is trained on DLC tokens. We accordingly adjust the sequence length and the vocabulary size of the model to match each DLC configuration. For example, the 512 256 DLC setup will be modeled by network that outputs sequence of size 512 with vocabulary size of 256. We use the re- re-masking strategy during sampling, introduced in Section 4, in our main results. Additional analysis on the impact of remasking in Appendix A. All other decision for training and sampling SEDD, including the model definition, training optimizer, training code, and hyperparameters remain unchanged from the original SEDD setup for text diffusion.
To compare with discrete, we also train continuous diffusion model to generate DINOv2 embeddings. We strictly follow the procedure used to train RDM [Li et al., 2024a] which leverages the DDPM objective with a U-NET backbone. We train the diffusion model with the same depth as the medium SEDD model to ensure a fair comparison between continuous and discrete embeddings.
5.1 DLC improves sampling fidelity
DLC improves unconditional generation. DiT [Peebles and Xie, 2023] and SiT [Ma et al., 2024] are strong and widely used backbones for cabel-conditioning ImageNet generation. In contrast, unconditional generation of ImageNet typically yields lower quality samples, as measured by FID, which aligns with our observations in Section 3. ImageNet is diverse dataset with lots of modes (e.g. different classes, environments, etc.) This feature of ImageNet hinders continuous diffusion models' ability to accurately fit the data distribution effectively. Prior works improved unconditional generation by conditioning generative models on learned representations, but underperform compared to label conditioning [Li et al., 2024a, Xu et al., 2024]. In Table 1, present system-level comparison of the state-of-the-arr conditional and unconditional generative models. We find that DiT-XL/2 model with DLC considerably improves the FiD compared to the same DiT-XL/2 with label-conditioning. Additionally, DiT trained with DLC pushes the unconditional generation stateoof-tee-al with FID of 1.59 and closes the gap with label-conditioned generative models.
DLC pushes the SOTA for generation without guidance. Figure 4 compares the performance of various diffusion methods training iterations without classifier-free guidance. We observe that DiT trained with DLC achives significantly lower FiD than conditiond basclines. Additionally, achives lower FiD than the baseline methods in considcrably fewer training iterations. In Table 1, we report MAR-H [Li et al., 2024b], the current SOTA generative model without guidance. We find that DLC-conditioned DiT outperforms MAR-H with a FID of 2.00 achieving the SOTA ImageNet generation without any guidance including label conditional and unconditional generation.
DLCs enable a trade-off between performance and compute. The plots in Figure 5 depicts the trade-off between the sequence length and the vocabulary size in DLCs. Although all configurations are designed to have the same total dimensionality, short sequence length (i.e. fewer tokens but larger vocabulary) are cheaper to obtain but lead to worst performance compared to long sequence length shown in the scaling plot in Figure 5a. Figure 5b and Figure 5c further show that both image
7

---

Linear probe %
Gen. FID!
Enc- Dec FID
Vendi
Unconditional
27.30
RCG
4.2 = 0.9
RCG
0.1*
4.89
DINOv2
9.6 + 1.3
DINOv2
85.9
37.90
2.12
DLCs12
13.2 + 0.8
DLC512
85.3
4.21
4.09
Table 2: DiT + DLC attains both high linear probe and low generative FID. Comparing DiT-XL/2 trained for 400 epochs with no CFG nor remasking. Unconditional and RCG conditional generation numbers are taken from [Li et al., 2024a]. We evaluate the classification accuracy (%) of linear probe trained on the input representation given to the diffusion model, the generative FID and the encoding-decodee decoding FID obtained by encoding 50 000 ImageNet samples to condition the generative model. *Using the encoder of RCG's released model.
Table 3: DLC produces more diverse generata- tions. Vendi score, mea- suring diversity of 8192 generated composed sam- ples. Avg. across 10 randomly chosen pairs of image embeddings.
(a) Source images
(b) DIT XL/2 DLCa12
(c) Mage-L + RCG
(d) DIT DLL- DINOv2
Figure 6: DLC enables high quality, diverse generations for semantic composition We generate compositions of (a) Komondor (dog breed) and Carbonara (b) DiT DLC generates diverse images of dog with pasta-like fur (c) Mage-L + RCG fails to generate dog (d) DiT conditioned on the averaged DINOv2 embedding produces reasonable but not-diverse images.
and DLC diffusion models take more iterations to converge for longer sequence but ultimately reach lower FID. Lastly, Figure 5d shows that models trained with longer sequences benefit more from larger DLC generators compared to those trained with smaller embeddings.
DLC improves over continuous embeddings conditioning. natural alternative DLC is to condition the generative model on continuous SSL embeddings. However, as discussed in Section 3, continuous distribution that lie a complex, multi-modal manifold, are hard for diffusion models to faithfully learn. The distribution of SSL embeddings may also have have lot of modes and thus also be hard to learn. RCG addresses this issue by projecting the SSL embedding using randomly initialized MLP, and report strong unconditional generation FID. Such projection likely results in latent space with simpler distribution and thus easier to learn. But, also leads to a representation without structure that is unable to produce novel OOD samples.
In Table we guantitatively compare unconditional diffusion models. Consistent with Section 3, both unconditional DiT-XL/2 model and condd--codd DřT-XL/2 model yield high generative FID. However, the DINOv2 conditioned generative model provided with ImageNet-encoded continuous embeddings, instead of generated, achieves significantly lower FID compared to all of the methods. This results suggests that the poor generative FID of DINOv2 conditioned generative model stems from the inability of correctly modeling the DINOv2 embeddings. Conversely, RCG which conditions its generative model on SSL embedding projected through randomly initialized network can attain decent generative FID, likely because its randomized embedding distribution is easier to model. However, these embeddings lack semantic structure, as illustrated by the low linear probe accuracy of classifier trained on the RCG embedding, which has negative consequences for its compositionality.

---

(a) bonsai.
(b) teapot on mountain.
(c) painting of flower.
Figure 7: DLC enables further OOD generalization with text-to-image generation. We use text-to-DLC model to enable generation of (a) bonsai, despite the label "bonsai" not being part of ImageNet, though there are handful of images of bonsai. (b) teapot mountain and (c) painting of a flower though neither exists in ImageNet's training set. We finetune LLaDA-8B to generate DLCCs for frozen D/T-X1.22 DL.C512 trained ImageNet-1k. The DLC tokens are generated by finetuning LLaDA [Nie et al., 2025], a text-conditional discrete diffusion model, on a random subset of 9M image-text pairs from LAION [Schuhmann et al., 2022].
5.2 DLC enables compositional diverse generation.
Compositional representations allows parts of representations to be composed in order to create novel concepts. A productive generative model can then leverage the composed representation to produce novel samples. In Figure 6, we explore the ability of diffusion models to produce novel images by composing the representations of reference images from different classes (e.g. komodor and carbonara). We compare models conditioned on Discrete Latent Code with baseline continuous embeddings (RCG and DINOv2). We compose the continuous embeddings by averaging the embeddings of the reference samples. For DLCs, we construct hybrid sequence by randomly sampling each token position from one of the two source DLCs, allowing for diverse combinations of semantic features. We show results in Figure 6a and observe that samples generated from composed RCG embeddings predominantly depict pasta, indicating a failure to represent both source concepts. DINOv2-based compositions do exhibit some degree of compositional generation, but the outputs show limited diversity and tend recombine contrast, DLC-based compositions successfully integrate visual features from both reference images and exhibit greater sample diversity (c.g. pasta on dog), as shown in Figure 6b. We quantify this diversity using the Vendi Score [Friedman and Dieng, 2023] in Table and find that DLC compositions consistently outperform continuous embeddings in generating diverse and semantically blended samples.
same dominant features (e.g. dog in pasta). In
Text-conditioned DLC for image generation
The dominant approach for conditioning image generative models is via text prompt [Ramesh et al., 2022]. However, training these text-conditioned models typically necessitates hundred of million of image-text pairs [Schuhmann al., 2022] and inaccessible compute for most practitioners. We show that these issues can be reduced by leveraging large-scale pretrained language models (LLM).
Discrete Latent Codes offer promising alternative for text-conditioned image generation by directly leveraging LLMs. Since DLCs are discrete tokens, they can be treated as part of language model's vocabulary and jointly be trained with text. Concretely, generation as a Markov Chain: text > c - æ. First, a text-to-DLC model samples a DLC from text prompt using a language model, p(cltext). Next, we generate image from DLC via a pre-trained image diffusion model p(z|c). Together, we have a flexible text-to-image pipeline p(=x|text) Σ. p( e) p(c[text) that leverages pre-trained LLMS and pre-trained image generators.
propose view the text-to-image
To obtain a text-to-DLCC generative model we build on top of LLLADA [Nie et al., 2025], a pre-trained text diffusion language model with 8B parameters. We extend its vocabulary by adding V + new

---

tokens corresponding to our DLC vocabulary and a <START-DLC> token. We learn to generate DLC's tokens conditioned on text by finetuning with masked prediction, as in pretraining LLADA. We train on image-caption pairs from LAOIN ISchuhmann et al., 2022] using a simple prompt of Saaage--DL--SSTTT--DD>><>---
following the template obtain a text-to-image generative model with 9M image-caption pairs, orders of magnitude less than than the standard lext-conditionin model.
Notably, we show that we can
We evaluate this pipeline's ability to generate novel images using text prompts out-of-distribution (OOD) relative the image diffusion model's s ImageNet training. As shown in Figure 7, we generate images with prompts that are clearly not ImageNet classes. To assess novelty, we retrieve the nearest example from ImageNet each generated sample (in DINOv2 embedding space) and show them in Appendix D. Though some "bonsai" images exist in the dataset, they are not labelled as such, so our text-to-DI.C C demonstrates how we can leverage even unlabelled images.
teapot on a mountain" shows a composition of two existing classes that are never seen together as we find no meaningful matches for teapots on a mountain ImageNet. and decomposition no paintings of flowers exist but they are painted shown on other objects e.g. plates. Altogether, the fine-tuned LLADA has effectively learned to map text prompts to plausible DLCs only having seen 9M image-caption pairs-and that the frozen diffusion model can interpret these DLCs to produce visually coherent and novel images outside of its training set.
painting of flower" demonstrates style transfer
Overall, this experiment shows that DLCs can serve as compact and compositional bridge between LLMs and image diffusion models. Our pipeline both effective and flexible, leveraging pretrained language and vision components without requiring joint training from scratch. By enabling L.Mss to "speak in DL.Cs" we open up new avenue for text-to-image generation that is scalable, modular, and generalizes beyond fixed label vocabularies.
7 Conclusion
We present Discrete Latent Code, compositional discrete representation learned solely from images. DLC improves on the state-of-ther thr-ttt for unconditional image generation on ImageNet, and enables productive generation capabílities leading to diverse compositional image synthesis and novel text-to-image paradigm. Diffusion model research generally focuses on improving the diffusion process, assuming label or CLIP embeddings as a given. We believe that focusing on the structural properties of representations, such as compositionality, can enable greater progress in the field. Furthermore, discrete image embeddings are an underexplored direction but are becoming more relevant as large scale pretrained diffusion language models offer vision for a unified text-image generation interface.
Acknowledgments
This research is supported by Samsung, NSERC. The authors thank Oumar Kaba, Sébastien Lachapelle, David Dobre, Muqeeth and Paul Barde for the insightful discussions. MN is supported by the Fonds de recherche du Québec, Nature et Technologies. This research was enabled by compute resources and software provided by Mila (mila.quebec), Calcul Québec (www.calculqueber.ca) and the Digital Research Alliance of Canada (alliancecan.ca).
References
Pietro Astolfi, Marlene Careil, Melissa Hall, Oscar Mañas, Matthew Muckley, Jakob Verbeek, Adriana Romero Soriano, and Michal Drozdzal. Consiseency-divariity-eeiii Pareto fronts of conditional image generative models, June 2024. URL UtL:ttp///axxi orggo/tg/// abs//406...../////// 10429. arXiv:2406. 10429 [cs].
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured Denoising Diffusion Models Discrete State-Spaces. Advances in Neural Information Processing Systems, volume 34, pages 17981-17993. Curran Associates, Inc., 2021. URI. ttctps://proceediigg www.nneuriiccmm httpppps.ccpppppp/eeeeee 455C5005478800888722777767887 Abstract html.
10

---

Fan Bao, Chongxuan Li, Jiacheng Sun, and Jun Zhu. Why Are Conditional Generative Models Better Than Unconditional Ones?, December 2022. arXiv:2212.00362 [es].
wwwpaawwww..rggb/////aa22 00362.
Florian Bordes, Randall Balestriero, and Pascal Vincent. High Fidelity Visualization of What Your Self-Supervised Representation Knows About, August 2022. URL http://arxiv, org. abs/ [es].
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked Generative Image Transformer, February 2022. URL http://arxiv http::///axxi.or/aaaaa/ orggabs//2000 20b.//20...2000.. 04200. arXiv:2202.04200 [es].
Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis, June 2021.
URL http://arxiv. arggabs/21055.o 05233. arXiv:2105.05233 [es].
Shimon Edelman and Nathan Intrator. A Productive, Systematic Framework for the Repre- sentation of Visual Structure. In Advances in Neural Information Processing Systems, vol- ume 13. MIT Press, 2000. URL https://papers. nips. wwtpccpppppppeeeee fiaperrfiles//ppe//000/ -at.aaaaaaaooAA--aaaata... html.
Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming Transformers for High-Resolution Image Synthesis, June 2021. URL aattp://arxiv wrw. hts//bbb/221222 09841. arXiv:2012.09841 [cs].
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English. Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling Rec- tified Flow Transformers for High-Resolution Image Synthesis, March 2024. URL http: arXiv:2403.03200 [cs].
//arxiv. org/ 2403.
Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1))3-71, March 1988. ISSN 0010-0277. doi: 10. 10.1016600010000777883033333 URL https://www sciencedirect wwwps/oinnc/ccccccccccccccctccct /ttp:///aaaaaatttooooo//o//////a http:///rr...ao////////00///0010
Michel Foucault. Les mots et les choses. Gallimard, Paris, 1966.
Dan Friedman and Adji Bousso Dieng. The Vendi Score: A Diversity Evaluation Metric for Machine Learning, July 2023. URL
02410. arXiv:2210.02410 [cs].
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock. Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang, Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Cantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang,
11

---

Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman. Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpiere Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth I Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presami, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriclla Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg. Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina. Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan. Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad. Sumit Gupta, Summer Deng, Sungmin Cho, Sunny
12

---

Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li. Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta,, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang. Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Llama 3 Herd of Models, November 2024. URL [cs].
butp://arxaa org/ 2xggabs//407. 21783. arXiv:2407.22)
Robert F. Hadley. Cognition, Systematicity and Nomic Necessity. Mind & Language, 12(2): 137-153, 1997. ISSN 1468-0017. doi: (111j_/4668-0001199...ooo 1110/ji11jj-...
URL https://
onlinelibrary. cectrrarrwwlleeee wiley. wwt.toioooccaacccctc
x. _eprint:
William Harvey and Frank Wood, Visual Chain-of-Thoughh Diffusion Models, June 2023. URL
http://arxiv.cm org/ aat///bb////..//////////////333/ 16187. arXiv:2303.16187 [cs].
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization, February 2019. URL htttt://axxiiii wwrg/abs/1888... [stat].
arXiv:1808.0667
Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance, July 2022. URL. https:
//arxiv. . org/ ab2/2207 12598v1.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Cur- ran Associates, Inc., 2020. URL https: -bw.aaa001...caaaaatatatt..cc html. citckey: ddpm.
www.nneeuuceppppeeeeeee cc/papprr202///ee
Vincent Tao Hu, David W. Zhang, Yuki M. Asano, Gertjan J. Burghouts, and Cees G. M. Snoek. Self-Guided Diffusion Models, November 2023. http:/tp://artt wrwg/ba//////eeeeeeooooo abs. 06462. arXiv:2210.0644 [cs].
Kaiyi Huang, Chenggi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T21-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation March 2025. URL http://arxixi:
06350. arXiv:2307.06335 [es].
Aapo Hyvärinen. Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research, 6(24):695-709, 2005. ISSN 1533-7928. URL http://mm org/
papers/v6/ hatae////yyrrrrrnnnneee html.
Chuanyang Jin, chuanyanginn/fatt-Din May 2025. URL https::///gtuu httt::://ggthbb...m////aaaaaaaap com/chhanyangg in/
fast-DiT. orieinal-date: 2023-05-16721:044 :4:04Z.
Kent Johnson. On the Systematicity of Language and Thought. Journal of Philosophy, 101(3):
111-139, 2004. doi: 10.58440jphh12004111332 Publisher: Journal of Philosophy Inc.
Tero Karras, Miik Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen. Timo Aila, and Samuli Laine. Guiding Diffusion Model with Bad Version of Itself. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, November 2024. URL /tttpp:://pperreee forum?id=bb6ffPPN
net/
Diederik Kingma and Ruiqi Gao. Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation. September 2023. URL http://arxiv. www.aawwvw...aaaaaeeaaaaaao arg/abs/2300 abb/22303000848 RxXiv:2303.0084 [cs].
Samuel Lavoie, Christos Tsirigotis, Max Schwarzer, Ankit Vani, Michael Noukhovitch, Kenji Kawaguchi, and Aaron Courville. Simplicial Embeddings in seff-Supervised Learning and Downstream Classification. The Eleventh International Conference on Learning Representations, September 2022. URL https: : wpw.reveeeeeeeeeeeeeeeeee .et/ nww/ Www.iireerd.rrrrrrreeeeer
13

---

Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mido Assran, Andrew Gordon Wilson, Aaron Courville, and Nicolas Ballas. Modeling Caption Diversity in Contrastive Vision- Language Pretraining. In Forty-first International Conference Machine Learning, June 2024. URL. htttps://openreviee 200f//20haro///300rrrr111......2 (12Fprof (12Fprof A Courvilee......
Samuel Lavoie-Marchildon, Faruk Ahmed, and Aaron Courville. Integrating Categorical Semantics into Unsupervised Domain Translation. In International Conference Learning Representations, October 2020. URL htttps://openrevieee net/ frrmmiddmmdd -d-
Tianhong Li, Dina Katabi, and Kaiming He. Return Unconditional Generation: A Self-supervised Representation Generation Method, November 2024a. URL http://arxix: orgg arggabs//20 ab////b/22212 03701. arXiv:2312.03701 [es].
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive Image Generation without Vector Quantization, November 2024b. URL 2406. 11838. arXiv:2406.1 11838 [es].
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang. Liping Hou, Rongjun Li, and Wei Peng. A Survey on Hallucination in Large Vision-Language Models, May 2024. URL http::ttt///iii org/ aaaaab/// 2202. 00253. arXiv:2402.00253 [cs].
Aaron Lou, Chenlin Meng. and Stefano Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution, June 2024. URL http://arxxiii http::///w/w/wwooo//////////a www.aa//.....eaeeeaeeeeeaaateeee 16834. arXiv:2310. 16834 [es, stat].
Nanye Ma. Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijndenn and Saining Xie. SIT: Exploring Flow and Diffusion-basee Generative Models with Scalable Interpolant Transformers, September 2024. URL http://///aa/ org/abs/24011 08740. arXiv:2401.08740 [cs].
Alex Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models, February 2021. URL http://arxixti org/abs//// aa/aa/bb//....///////////////2// 09672. arXiv:2102.09672.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models, February 2025. URL arXiv:2502.09992 [cs].
Qian Niu, Junyu Liu, Ziqian Bi, Pohsun Feng, Benji Peng. Keyu Chen, Ming Li, Lawrence KQ Yan, Yichao Zhang, Caitlyn Heqi Yin, Cheng Fei, Tianyang Wang, Yunze Wang. Silin Chen, and Ming Liu. Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges, December 2024. URL http://arxiv. orggaba// ab2/2409. 02387. arXiv:2409.02387 [es].
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning, May 2018. URL, htt.h/tp//aaaii.orr///////////// 1tt.//71..000 00937. arXiv:1711.00937 [cs].
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacidin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Ya Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision, February 2024. URL http:// arxiv httg/abs/23000 07193, arXiv:2304.07193 [cs].
William Peebles and Saining Xie. Scalable Diffusion Models with Transformers, March 2023, URL http://arxiv. htrg/abs/22113 09748. arXiv:2212.09748 [cs].
Pablo Pernias, Dominic Rampas, Mats L. Richter, Christopher J. Pal, and Mare Aubreville. Wuer- stchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models, September 2023. URL http://arxizs org/abs////eeeeooooo..go abs//3////2/66/ 00637. arXiv:2306.00637 [cs].
14

---

Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Dif- fusion Autoencoders: Toward Meaningful and Decodable Representation. March 2022. URL
http:///axxiv..o/ab///ss/sss 2tt.///1............111 arXiv:211 11.22111.11640 [es].
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http://arxix /atpiv./axivvv//a htt//bs/22000 000200000000.......0200222022000 arXiv:2103.000 [es].
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Imae Image Generation, February 2021. URL http: //arxiv. org/ absa ht////bb/222... 12092. arXiv:2102.12092 [cs].
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents, April 2022. URL http://arxiv, org/ /ab//////// 2204. 06125. arXiv:2204.061 06122 [cs].
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models, April 2022. URL http://arxiv. hrggabs/2112. 10722 1XXi:2212210055: [cs].
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge, January 2015. URL http://arxiv http:aawwwi.....aaaaaa/aae hatgg/ab//440 0575. arXiv: 1XXiv:1409.05 409.0575 [cs].
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models, October 2022. URL
org/abs hts//bb/2222/ 2210. 08402. arXiv:2210.08.4 [cs].
Kenny Smith and Simon Kirby. Cultural evolution: implications for understanding the human language faculty and its evolution. Philosophical Transactions of the Royal Society B: Biological Sciences, wwbpwwwwwwwwwwwc........ww.w...w a.m..ih.....mm www..ii..orrpm////aarea....eeaee htt.://///mmmaattccee///////////
November 2008. ISSN 0962-8436. doi: 10. 10.109/sstb.2008. 2008.015.
URL htw.::ttp:///ee
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsu- pervised Learning using Nonequilibrium Thermodynamics. In Proceedings of the 32nd In- ternational Conference Machine Learning, pages 2256-2265. PMLR, June 2015. URL html. ISSN; 1938-7228.
htttp::/pproceeding: mlr. /ttp:eecennaaaaa..rrrreee/////aa
Michael Tomasello. The cultural origins of human cognition. The cultural origins of human cognition. Harvard University Press, Cambridge, MA. US, 1999. ISBN 978-0-674-00070-4, Pages: vi, 248.
Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A Picture is Worth More Than 77 Text Tokens: Evaluating Captions, June 2024. URL http: arxiv. org/ abs/2312 08578. arXiv:23 12.00878 [cs].
Models on Dense
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need, August 2023. URL http: // //arxiv....////////////a///ii org/aa//aao abs/aabs...//////////////// 03762. arXiv:1706.0376: [cs].
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7))1661-1674, July 2011. ISSN 0899-7667, 1530-888X. doi: 10.1162/NECO.a. 00142. URL https: / /wttt/trrtttt//ec//ee mit http://eccoarriccce///////ee// 1674/77777
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking Discrete Diffusion Models with Inference-Tinee Scaling, March 2025. URL. http://arxiv/ wwwaariiio.o/raaaaa org/abs/ abs/ 2503. 00307. arXiv:2503.0033 [cs].
15

---

Yingheng Wang, Yair Schiff, Aaron Gokaslan, Weishen Pan, Fei Wang, Christopher De Sa, and Volodymyr Kuleshov. InfoDiffusion: Representation Learning Using Information Maximizing Diffusion Models, June 2023. htt.::ttp///iiiii
08757. arXiv:2306.08757
[es].
Benjamin Lee Whorf. Language, Thought, and Reality: Selected Writings of Benjamin Lee Whorf. MIT Press, 1956.
Ludwig Wittgenstein. Philosophical Investigations. Wiley-Blackwell, New York, NY, USA, 1953.
Yilun Xu, Gabriele Corso, Tommi Jaakkola, Arash Vahdat, and Karsten Kreis. DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents, July 2024. URL http://arxix: org/ abs/ 2407. 03300.0000000 arXiv:2407.03300 [es].
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans the Loop. June 2016. URL http: //rxivv. org/ aww/aa/s///0 03365. arXiv:1506.00335 [cs].
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content- Rich Text-to-lmage Generation, June 2022. arXiv:2206. 10789 [cs].
URL attt:://axzs: org/ harg//ba//a aba/a206...222/////a// 2206 10789.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Sain- ing Xie. Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, February 2025. URL. http:://ax/tt http:/aaaaa.r.raa//a/a/aaaaa///a herrg//bs/241..... 2111000994400 arXiv:2410.06940 [es].
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bfgs-of-words. and what to do about it?, March 2023.
URL http://arxivv www./t///////////////.......ee// aaaaab//22 22s00 2210.01936. arXiv:2210.01936 [cs].
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image BERT Pre- Traiirann with Online Tokenizer, January 2022. URL http: //ttx///a//vv///a org/abs/ 2111.07833 arXiv:211 1.07832 [cs].
16

---

A Resampling DLC with SEDD-absorb
Resampling tokens in discrete diffusson models is akin to adding noise in DDPM sampling. Generally, in MCMC sampling, such noise may prevent the sampling chain from ending up into a local minima. In Equation 4, we introduced remasking scheme for SEDD-absorb. In Figure 8, we explore the effect of the remasking ratio parameter 1 on the ImageNet generation FID. We find U-shape curve where a resampling ratio that is too low or too high cause degraded image generation quality.
In Figure 9 we present uncurated results with E 0.5). that are too low or too high produce samples that lower quality, but for different reasons. "/ close will result in sampling with too little steps to produce good DLC (i.e. it is equivalent to having very small total number of time-steps). 7) that is too small will not allow the sampling of the DLC correct mistakes made early in the sampling and will result in a local minima. In contrast to classifier-fre guidance, we don't 't find that very high 1) cause the model to generate weird artefacts or to reduce the diversity. This implies that remasking DLCs with SEDD-abson or other remasking methods could be used as strategy wholly replace classifier-free guidance. That said, remasking and classifier guidance are compatible using one strategy do not prevent from using the other strategy.
3.4
3.0
2.8
2.6
2.2
10
Resampling ratio (rj) - Symmiggygg scale
Figure 8: Effect of the resampling ratio in SEDD-absorb. Model trained on ImageNet 256 x 256 with DLC512 sequence of 512 DLC with 256 tokens each. We sample the tokens for 4096 steps and activate the remasking for steps in [0.3, 0.55). Generation without classifier-fre guidance. We find a U-shape curve with an optimal resmapling ratio of n = 0.01.
R
(a) 0.
(b) 1) 0.01.
(c) 1) 0.5.
Figure 9: Uncurated unsupervised generation for ressampling ratio 7) E (0, 0.01, 0.5). Generation without CFG.
17

---

B Comparing DLC to Stable Diffusion
To demonstrate the benefit of DLC, we compare the state-of-the-ar open-source diffusion model, Stable Diffusion, which is both larger and trained more data. We aim to reproduce one of our productive examples, Carbonara Komondor, from Figure 1b. First, we generate using only text prompt "Komondor made of Carbonara". Next, we aim to compare as closely to our average- image-enbedding conditioning. We leverage IP-Adapter [Ye et al., 2023], the standard tool for image-conditioned generation. We get the CLIP embeddings from our komondor and carbonara images and generate with IP-Adapter conditioning on the average of the two image embeddings. For all generations, use aP-adapter's 's recommended version Stable Diffusion 1.5, set IP-adapter scale to 0.6, CFG to default 7.5, and generate with 100 timesteps.
We show results in Figure 10. We find that text-only conditioning is not sufficient to generate our carbonara dog, supporting our claim that text embeddings can not always sufficiently capture image semantics. Generating from the average embedding is slightly better, though lacking in diversity and failing generate dog all in one case. Finally, combining both text and image conditioning allows Stable Diffusion to approach our method, though it is clearly heavily leaning towards Komondor and only changed the fur to be more pasta-like.
(a) Only text "Komondor made of (b) Average of CLIP embeddings Carbonara"
(c) Average embedding and text
with IP-Adapter
combination of Komondor and Carbonara but a) conditioning on purely text prompt insufficient and we require b) conditioning on the average CLIP embedding of image of Komondor and Carbonara to achieve reasonable combination the classes, though one failure mode. Conditioning on c) both average image embedding and text prompt works best, though shows distinct lack of diversity compared to DLC-based method.
Figure 10: Stable Diffusion somewhat reproduce
C Additional evidences showing inability of diffusion models to generate highly modal continuous distributions
Section 3 demonstrated that diffusion models struggle model highly modal distributions. For completeness, Figure 11 showcases the full generations resulting from 9 mixtures, 121 mixtures, 400 mixtures and 900 mixtures for unconditional. oracle conditioned and GMM conditioned diffusion models. Unconditional generation observes degrading fit as the number of modes in the dataset increase. Conditional generative model demonstrate good fit even for very large number of modes. Interestingly, inferring the mixtures scales relatively well with the number of modes.
18

---

Frainin
Uncandii
Oracie
GMM
30
Figure 11: Comparing a m x m grid of mixture of Gaussian P with samples from model distributions Qs unconditional, conditioned Gaussian Mixture Model (GMM). Unconditional generative models cannot samples highly modals distributions. Meanwhile, conditional generative models have no problem sampling highly modal distribution. Moreover, inferring mixture indices via GMM also scales to highly modal distributions.
oracle mixture index and mixture index inferred from
D Evaluating Novelty of Text-to-Image samples outside ImageNet
Our rext-to-DLC-to-e tion outside of the ImageNet distribution. Evaluating the novelty generations a fundamentally difficult task [Kingma and Gao, 2023]. To give qualitative sense of our pipeline's 's ability, we find the closest examples to our generation the ImageNet training set. To do so, we use the DINOv2 embedding space and find the 5 nearest neighbours in that space. We show results in Figure 12.
pipeline enables generation of interesting images that reflects generaliza-
We show three types of generalization found in our model. First, we show that this model can generate samples of images that occur in ImageNet, such bonsai, but for which no label of bonsai exists. Such generalization showease the utility of open-ended generation. Second, we show compositional generalization where we generate teapot on mountain. Notably, there are no images of teapots on mountains in the dataset. Our model clearly manages to learn and combine the semantics of separate foreground object and background setting. Thus, this results demonstrate that the generative image model can produce novel samples by extracting attributes in ImageNet and recomposing them in novel ways. Finally, we denote the generation of painting of flowers. While some painting of flower does exists, specific painting of the flower generated do not exists in the dataset. For example, there not painting of the white flower in ImageNet.
19

---

Generated
ImageNet's Nearest neighbours
A bonsai.
Generated
ImageNet's Nearest neighbours
(b) teapot mountain.
imageNat's Nearest neighbours
(c) Painting of flower.
Figure 12: Text-to-image generated samples from Figure 7 and their semantic nearest neighbours in ImageNet's training We find the semantic nearest neighbours with respect to the cosine similarity of DINOv2-vit/l encoding. While bonsai is not part of the labels of ImageNet, we find some bonsai in ImageNet's training that similar to the generated samples. However, there are teapot a mountain nor painting of a flower that resemble those generated by our model. This result shows the productive capability of our image generative model.
E Experimentation details
Unconditional image generation All models were trained with a batch size of 512 and trained according to the byper-parameters specified in Table 5. For unconditional generation of SEMs, use the Tweedie denoiser. We use remasking for generating the samples in Figure 4, Table and Figure Otherwise, remasking is used. We generate 50000 SEMs that then used to generate images using the image diffusion model.
The image diffusion models are all trained with DiT-XL/2 model [Peebles and Xie, 2023] and the hyper-parameters specified Table 4. Following Peebles and Xie [2023], we use the EMA VAE model from mable-Diffusion. The generation DDPM [Ho et al., 2020]. We only use
20

---

N blocks
28
N blocks
24
Hidden size
1152
Hidden size
1024
Patch size
2
Num heads
16
Num heads
16
Optimizer
Adam W
Optimizer
AdamW
Learning rate
3e-4
Learning rate
le-4
Batch size
512
Batch size
256
Warmup
2500
Weight decay
0
Gradient clipping
1.0
Num sampling steps
250
Weight decay
0
CFG scale (Table 1)
1.4
Num sampling steps
4096
Training epochs (Table 1)
1200
Resampling ratio y (Table 1)
le-4
Image size
256x256
Training epochs (Table 1)
200
Table 4: DiT-XL/2 hyper-parametert for training
Table 5: SEDD-medium hyper-parameters for training and sampling
and sampling
Model
8B BB-base
# sample seen Optimizer
9M
AdamW
Learning rate
le-5
Total batch size
128
Grad. accum. steps DLC shape
8
128 x 1024
Table 6: LLADA text-and-DLC haper-parameterrs
for fine-tuning.
classifier-free guidance for reporting results Table For fair comparison, we re-use the same CFG scheme DiT and apply CFG only on the first three-channel. For FID computation, generate 50000 samples conditioning on the pre-sampled SEMs.
Text-and-DD DL DLC fine-tuning We fine-tune L.LDA--B-- -B--bse [Nie al., 2025], large diffusion language model parameterized as 8B parameters Llama [Grattafiori et al., 2024] transformer [Vaswani al., 2023]. Contrary to SEDD, which predicts the concrete score, LLADA predicts the probability of every tokens directly pe (z (z).. The training objective to train the transformer is the cross-entropy
loss:
L
L( = Et,
Σ
1 MM]og pe (2//az))
(5)
For fine-tuning, we re-use the same Equation 5. However, instead of considering text tokens only in our objective, we consider pairs of text and DLC tokens. The DLC tokens comes from encoded images and the pairs text and images are randomly sampled from LAION [Schuhmann et al., 2022]. As proof-of-concept, randomly subsample 9M image-text pairs from LAION that are used for fine-tuning.
For sampling the DLC, we provide masked sequence for 128 mask tokens followed by separator token and the prompt. We follow Nie al. [2025] protocol with low-confidence remasking strategy.
Computational resources. We fine-tune the DINO + SEM encoders on two A 100 GPUs, with each training taking approximately one day. We train all image and discrete diffusion models on single node of 4x H100. The training speed (iteration per second) for our image diffusion model is constant across sequence length and label conditioning DiT at about 5.2 it/sec. Training 800 epochs of the image generator takes approximately 10 days. In contrast, our discrete diffusion SEDD training speed scales with the sequence length (see Figure 5). Our large sequence length model, SEDD-medium with L=512, takes about two days to train.
F
License
The compilation of assets used in the reproduction this work is presented in Table
21

---

Asset
License
Source
ImageNet
imagenet
htpps//www.image-ee..orgg
LAION
MIT
D2
Apache 2.0
http://gfhbbboommmcccccooerrerrr
Fast-DiT [Jin, 2025]
CC-BY-NC-4.0
http://ghhhbb..ommuuuuuuuuggnngg
SEDD
MIT
LLADA
MTT
Pytorch 2.5
Pytorch
Transformers
Apache 2.0
This work
MIT
Table 7: Compilation assets used in the production of this work along with their license and the
source location of each asset.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are that DLC:
Improves unconditional generation ImageNet. Demonstrated in Figure 4 and Table 1.
Enables diverse compositional generation. Demonstrated In Figure 6 and Table 3. Enables text-to-image generation of images not in ImageNet despite training the image generator only on ImageNet (showing importance of productivity). Demonstrated in Figure 7.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The main limitation of using discrete code is the computational tradeoff as discussed in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximation only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
22

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We release the code for review and provide the experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
23

---

reproducible, they While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release the code at submission time. We will release the DLC representation of ImageNet HuggingFace dataset when publicly releasing this work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
public/ mublic/guides/ wttgggiiiccdddsiieee patteeSummsssionoolool
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. cc/pcc/ppl.oo ccc/ppblic///// wgwpc//g/////////eoocooococcsse/ CCaeee//doeeSbbmeeeeeee iettuubmiiiiiooio wesiisssosssiiii onPoliccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5 provides an overview of the settings and the Appendix provide the details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
24

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: While we report std. for the vendi score, most of the main results do not include emoor bars. The error bars are generally not report at ImageNet scale due to the computation resources required to train these models and due to the robustness of the training. Therefore, follow the current convention when reporting results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Presented in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpps:////uipppp/
Answer: [Yes]
Justification:
25

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We briefly discuss broader impacts in the conclusion (Section 7).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deploymennt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The data and the models we will release are product of ImageNet, a dataset thoroughly studied.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
26

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Presented in Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com /on/aaateeee has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the ImageNet DLC representation and the models checkpoints at camera ready Huggingface. Alongside, we will release the model and data cards.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
27

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
cc cc//oonferncccs/2255///////////e
28


Reviews:
[{"review_id": "tINm3P1vUX", "reviewer": "Reviewer_RBRg", "summary": "This paper explores the discrete latent representation for conditional/unconditional image generation with diffusion models. Specifically, authors utilize the SEM encoder and argmax operation to extract discrete latent codes (DLC), which are used to condition the diffusion models. The proposed method is compared with various unconditional generative methods using the ImageNet benchmark.", "questions": "- For the compositionality of continuous representations, I wonder whether using random position-wise sampling (as done with DLCs) instead of averaging would yield similar results.\n- For unconditional generation, since DLCs are sampled with SEDD in a non-autoregressive manner, have you observed quality-diversity trade-offs when varying the sampling temperature or related parameters?", "limitations": "See weaknesses.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strenghts\n\n- The paper is well-written, and the motivation for utilizing discrete latent code (DLC) for dense description of images is reasonable.\n- Trained DLC can be widely applied to various tasks, including unconditional/text-conditional generation and compositional image synthesis.\n- Thoughtful experiments demonstrate the effectiveness and scalability of DLC.\n\nWeaknesses\n\n- While the DLC is promising, experiments are only conducted using the SEM encoder. More detailed analysis on choosing the SSL encoder model, or an ablation study, would strengthen the claims.\n- DLCs demonstrate diversity through random position-wise sampling (Figure 6). However, the visual quality of generated samples appears relatively low. Moreover, although the paper emphasizes compositionality, it is unclear whether DLCs support controllable interpolation. I wonder if DLC also shows smooth semantic transfer by adjusting the sampled token ratios as continuous embeddings do via weighted averaging. It would also be helpful to evaluate the diversity-quality trade-off using FID, in addition to the Vendi score, to better assess both aspects jointly.", "was_revised": true, "final_justification": "I have read the author's response. While I still have some concerns, such as the smooth semantic interpolation of the DLS tokens due to rebuttal policy, considering the paper's quality and impact on the community, I decided to keep my rating at 4: weak accept.", "rebuttal": "We thank the reviewer for their thoughtful assessment of our work. We are pleased that they appreciated the quality of the paper and the generality of DLC. Below, we respond to their comments and questions\n\n## Motivation for the SEM encoder.\nOur motivation for using the SEM encoder is informed from Table 2 of [[0](https://arxiv.org/abs/2204.00616)] where discretized SEM achieves a higher linear probe accuracy. Importantly, as shown in Figure 5.a of [[1](https://arxiv.org/abs/2410.06940 )], the downstream linear probe accuracy strongly correlated to the downstream FID. While we agree valuable to provide an ablation over different discrete encoder methods, the cost of running such an experiment is high given our resources.\n\n## Visual quality evaluation of semantic compositional generation\nTo assess visual quality in compositional generation, we reproduce the setup from Table 2 of the paper. Specifically, we compare continuous and discrete representations under equal training budgets and model sizes. We generate 50,000 composed samples using:\n* DLC (token sampling): FID = 34.34\n* DINOv2 (embedding averaging): FID = 41.71\n\nThese results indicate that DLC supports more coherent compositional generation. We will include this quantitative result and expand discussion of the diversity-quality tradeoff in the main paper as suggested.\n\n## Position-wise sampling of continuous representation leads to degenerate generation.\nQualitatively, the image generated lacks any structure found in natural images and quantitatively we observe a FID of 59.65 in this setting. While position-wise sampling of continuous representation would improve the diversity of the image generation, the continuous embeddings are global and do not allow the same token-wise decomposition of DLC. This aligns with the observation that continuous embeddings lack the localized semantics needed for token-wise decomposition, highlighting a key advantage of DLCs.\n\n## Continuous interpolation with DLC\nFollowing the reviewer’s suggestion, we implemented interpolation by adjusting the token sampling ratios between two DLCs, instead of fixing the sampling to be uniform across DLCs. As we interpolate the sampling ratio of the DLCs, we observe meaningful semantic transitions of semantic features from one image to the other. We will include qualitative examples in the appendix.\n\n## SEDD generation parameters ablation\nAs detailed in Appendix A, we study the effect of the resampling ratio in our sampling procedure. We observe that increased resampling improves generation quality while having limited impact on diversity. Our sampling is based on the Tweedie denoising framework [[2](https://arxiv.org/abs/2310.16834); algorithm 2], which does not include a temperature parameter.\n\n* [0] https://arxiv.org/abs/2204.00616\n* [1] https://arxiv.org/abs/2410.06940 \n* [2] https://arxiv.org/abs/2310.16834"}, {"review_id": "gIgazruHv4", "reviewer": "Reviewer_KXs6", "summary": "This paper introduces Discrete Latent Code, which is an image representation derived from simplicial embeddings trained with self-supervised learning and diffusion models. It aims to generate discrete token sequences that are learnable, compositional, and efficient for modeling complex and diverse distributions. The experiments demonstrate that the DLC improves generation quality in unconditional image generation on the ImageNet dataset in terms of FID. Besides, DLCs can do semantically compositional generation by combining tokens from other images. It can also support text-to-image generation by finetuning the large language models. The authors claim that DLC is a representation that bridges the gap between image expressivity and language compositionality.", "questions": "1. Could the authors provide visualizations of what individual DLC tokens represent, such as attention maps?\n2. Can DLC be used on other datasets or other domains? Can DLC be used in image editing?\n3. How does text-to-DLC perform when the prompt is very complex?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The motivation is straightforward. The paper aims to learn an image representation to model the complex data distribution, which also enables image generation and composition to improve the representation. It also addresses limitations of continuous embeddings in generative modeling.\n2. The method is original. The paper combines the simplicial embeddings with discrete diffusion in token generation using a unified pipeline.\n3. The experiment is clear. The paper provides plenty of experiments. DLC-conditioned diffusion models achieve better performance on ImageNet in terms of FID. Besides, the paper provides extensive ablation studies and compares them with both discrete and continuous baselines.\n\n\nWeaknesses:\n1. The computation cost is large. The method needs to train both DLC and diffusion models, which requires large computing resources. It limits the application and reproduction.\n2. Text-to-image evaluation is limited. The text-conditioned DLC model is only evaluated on a small dataset. The human or user studies are also missing.\n3. Some analysis about interpretability is missing. The reason why compositional discrete tokens can help generalization in diffusion is unclear. Besides, the semantics of individual DLC tokens are not explored.", "was_revised": true, "final_justification": "The authors have provided detailed responses to my concerns regarding computation cost, applicability to other datasets, and potential application in image editing. However, text-to-image evaluation remains only partially addressed. Therefore, I maintain my original score.", "rebuttal": "Thank you for your comments and feedback. We are glad that you appreciated the originality of the method, the clarity of the experiments and the motivation behind the method. Below, we respond to your comments and questions\n\n## Computation cost of DLC training\nThe overall computation is comparable to baselines methods (e.g. RCG, REPA). Moreover, the largest computational cost is the image generator (see paragraph Computational resources). As shown in Figure 4, training the image generator with DLC converges much faster than SOTA baselines. Thus, DLC may enable institutions with less compute to participate in image generation research.\n\n## Reproducibility and applicability of the work\nTo enable reproduction and application of the work, we release every model (the SEM encoder, the DiT, the SEDD and the fine-tuned LLADA) and the ImageNet DLC dataset, all available on huggingface. We also provide the code for training and evaluation of every model and for generating the ImageNet-DLC dataset.\n\n## Can DLC be used on other datasets or other domains?\nYes, DLC is not restricted to ImageNet. We used DLC with LAION allowing us to sample images that are not in ImageNet (e.g. “a teapot in Antarctica” is not in ImageNet as discussed in Appendix D, Figure 12). DLC could likely be extended to other domains such as audio, though such exploration is left for future work.\n\n## Can DLC be used in image editing?\nDLC enables semantic-level image editing through token manipulation, as shown in our compositionality experiments. Unlike pixel-level edits (e.g., inpainting), our method allows for structural changes at the representation level, enabling meaningful semantic transformations.\n\n## On text-to-image evaluation \nWhile our primary focus is improving image representations for image generation, we also explored text-to-image generation as a proof-of-concept on how such representation may be used in that setup (i.e. as tokens for a LLM). Due to compute and data limitations, we did not aim to outperform existing SOTA text-to-image models. Instead, we aimed to demonstrate the potential of DLC as a bridge between language and vision, and we hope our initial results inspire future work by the broader community.\n\n## Interpretability of DLC – visualization of DLC\nWe refer the reviewer to Section 4.4 of the SEM paper [[0](https://arxiv.org/pdf/2204.00616)]. The authors demonstrate that SEMs lead to an interpretable, disentangled representation. Our semantic token mixing experiments support this, as meaningful compositions arise from swapping tokens across images. This result indicates that individual tokens carry localized semantically consistent information.\n\n[0] https://arxiv.org/pdf/2204.00616"}, {"review_id": "bbPwaIVRAx", "reviewer": "Reviewer_v32v", "summary": "This paper proposes Discrete Latent Codes (DLCs) as a new conditioning representation for diffusion-based image generation. A DLC is a tokenized image representation obtained from a Simplicial Embedding model trained self-supervised, which maps an image into a sequence of discrete tokens instead of a continuous embedding. The authors argue that DLCs are easier to model (since one can learn a discrete token distribution) and inherently compositional, meaning parts of different images can be combined to generate novel outputs.", "questions": "Could you elaborate on the limitations and consistency of the DLC compositionality experiments?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:** The paper is well-executed with strong experimental results. The proposed DLC conditioning yields **state-of-the-art** unconditional ImageNet generation quality, significantly improving over prior approaches. For instance, a diffusion transformer (DiT-XL/2) conditioned on a 512-token DLC achieves an FID of *~2.2* without guidance, outperforming previous representation-conditioned models (e.g. a recent baseline “RCG” had FID 3.44). With classifier-free guidance, the method further reaches FID 1.47, closing the gap to class-conditioned generators. \n\n**Weaknesses**: In terms of **significance**, the work is strong for unconditional generation research, but one could question how broadly impactful that is, given that *conditional* (especially text-conditional) generation is the main focus of the field now. The paper’s answer is that DLCs can improve those too, but the current evidence for text-to-image is not state-of-the-art (since only 9M pairs were used, the text-conditioned results are likely below the best alternatives).", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns.", "rebuttal": "Thank you for your comments. We are encouraged that you found our experimental results strong and have a positive assessment of our paper’s clarity and execution.\n\n## Significance of this work\n\nText-to-image is indeed a major focus of the field but all current SOTA models (Midjourney, DALLE, Imagen) are closed source and for us to create a SOTA model would require significant compute, data, and engineering resources which are unavailable to most organizations (including ours).\n\nOur work is impactful since SOTA models may still use basic, continuous representations, e.g. Stable Diffusion 3.5 uses CLIP embeddings. We examine the underexplored direction of discrete image representation and show they are superior to their continuous counterpart for image generative modeling. We believe that SOTA on unconditional generation and DLC’s scaling behaviour in both performance (Figure 5) and efficiency (Figure 4) demonstrates strong promise for improving text-to-image at SOTA scale and compute. For that reason, we believe that this work is of great interest for the NeurIPS community and hope that it will inspire researchers at labs with the appropriate amount of resources to experiment with DLC.\n\n## Compositionality experiment limitations\nWe acknowledge that the compositionality experiments are qualitative. This is consistent with prior work (e.g. [[0](https://proceedings.neurips.cc/paper_files/paper/2024/hash/e304d374c85e385eb217ed4a025b6b63-Abstract-Conference.html\n)]), as we are unaware of standard quantitative benchmarks for compositionality in natural images.\n\nDeveloping such metrics is non-trivial due to the open-ended nature of visual composition, but is an important direction for future work. Our qualitative results suggest that DLCs enable diverse and coherent recombination of semantic parts, which we believe provides a meaningful step forward.\n\n## Compositionality experiment consistency\nThe same mix of tokens (e.g. AABBA) consistently results in the same outcome (e.g. dog in pasta vs dog made of pasta). Different token mixes result in different semantic mixes and the variability at the semantic level is controlled by the DLC. The variability from the prior of the image generative model captures fine-grained details that are not captured by the DLC such as slight difference in object positioning.\n\n[0] https://proceedings.neurips.cc/paper_files/paper/2024/hash/e304d374c85e385eb217ed4a025b6b63-Abstract-Conference.html"}, {"review_id": "V8sFKMGQ8G", "reviewer": "Reviewer_JhZC", "summary": "The authors propose DLC as a novel image representation derived from Simplicial Embeddings, trained with a self-supervised learning objective. DLCs consist of sequences of discrete tokens, contrasting with traditional continuous image embeddings. Diffusion models trained with DLCs achieve enhanced generation fidelity, setting a new state-of-the-art for unconditional image generation on the ImageNet dataset. The compositional nature of DLCs allows for the generation of novel, out-of-distribution images by creatively combining the semantics of different images. The authors also shows the ability to generate image with large LLMs with promising results.", "questions": "1. I am confused by the sections \"Setup for training image diffusion\" and \"Setup for latent embeddings diffusion\". Can the authors further explain what is the difference between those two and what you are trying to accomplish here?\n\n2. Runtime analysis. I would like to ask the inference time for the proposed method.\n\nOthers see weakness", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength:\n1. The idea is novel and interesting. I wonder if the discrete latent code could really separate the concepts with a deeper analysis.\n2. The experiment results are strong.\n\nWeakness:\n1. The images generated by the proposed method (Figure 6 (b)) shows undesired results where the identity of the dog is not preserved (top two images) while the traditional method (d) can adhere to the prompt. \n2. I feel like the compositional property is not well explored, does it matters how you concat two DLCs together (interleaving/shuffle/A|B or B|A)? This is interesting explore since you mention the bag of words problem posed by the VLMs.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal and I am satisfied with the results. Since I have already given accept, I will not change my score.", "rebuttal": "We sincerely thank the reviewer for their detailed and constructive assessment of the work. We are pleased that you find our method novel and impactful, and we appreciate your insightful questions and suggestions. We respond to the comments and questions below\n\n## Compositionality: How are DLCs combined? (interleaving/shuffle/A|B/B|A)\nThe way that DLCs are composed certainly matters as found in Figure 6b. Interleaving, A|B and B|Compositions are possible combinations that could be sampled and would lead to different image production. Since the DLC position is important then shuffling the DLC and then composing them would result in nonsensical image generation.\n\n## Preservation of identity in generated images: \nSome of the dogs in Figure 6b are not identical to the reference dog in Figure 6a. E.g. The dog in the top right corner has a piece of meat in place of its head and has pasta in place of its fur. We argue that this is a desired composition demonstrating that the image generator can coherently combine features from either reference images. We note that other combinations will more closely preserve the dog identity and this may be forced by sampling more tokens from the source dog image instead of uniformly sampling the tokens as we did in this experiment.\n\n## Runtime analysis\nThe inference compute cost can be controlled by changing the number of sampling steps of either DiT or SEDD. Following the reviewer’s comment, we performed an additional experiment where we control the runtime of images generated with DLC by reducing the number of SEDD steps. We report the comparative results (without CFG) below\n* RCG, runtime = 1.52s/it, FID = 3.44\n* DLC (# SEDD steps = 128), runtime = 1.32 s/it, FID = 4.14\n* DLC (# SEDD steps = 256), runtime = 1.91 s/it, FID = 2.86\n* DLC (# SEDD steps = 1024), runtime = 5.33 s/it, FID = 2.71 (updated result)\n* DLC (# SEDD steps = 2048 + remasking), runtime = 9.89 s/it, FID = 2.0 (updated result)\n\nWe find that DLC is competitive with baselines for similar inference runtime but strongly outperform them when allowed for more runtime, showing strong test-time scaling.\n\n## Clarification: “Setup for training image diffusion” vs “Setup for latent embeddings diffusion”\nThese two paragraphs aim at giving the reader more context about the training of the DLC-to-image conditional generative model (DiT) and the training of the DLC generative model (SEDD) respectively. We will revise Section 4.1 to make this distinction clearer, renaming the paragraph headers to “Pixel-space Diffusion Training” and “DLC-space Diffusion Training.”"}, {"review_id": "AXoHLCxbQR", "reviewer": "Reviewer_VouL", "summary": "This paper introduces Discrete Latent Codes (DLC)—a compositional, discrete image representation used to condition diffusion models. DLCs are derived from self-supervised Simplicial Embeddings and represent images as sequences of discrete tokens. The key contributions are: (1) state-of-the-art FID for unconditional ImageNet generation, (2) compositional generation by mixing codes from multiple images, and (3) a text-to-image pipeline where a language model is trained to produce DLCs from text, allowing novel image generation beyond the training distribution.", "questions": "1. Why average DLC token embeddings rather than use cross-attention or per-token conditioning? Could more expressive strategies improve performance?\n2. How consistent and controllable is compositional generation? Have structured mixing or multi-object compositions been tested? An interesting test would be on part-level generation (e.g., PartCraft [1] / PiT [2]) -- treating part as a discrete species etc.\n3. A systematic evaluation on how well does it generalize to unseen prompts or multi-object prompts? \n4. How can DLCs generalize to datasets beyond ImageNet, such as fine-grained species, scenes, high-res images, or non-natural data? The discrete latent code can be treated as \"concept\" too. \n5. How would you apply DLCs on existing T2I model (e.g., SD1.5/3 or Flux)?\n6. Can we do the reverse on the text-to-DLC -- DLC to text and see how well the image representation can be converted to text prompt (image captioning)?\n\n[1] Ng et al. PartCraft: Crafting Creative Objects by Parts. ECCV 2024.\n[2] Richardson et al. Piece it Together: Part-Based Concepting with IP-Priors. arxiv 2025.", "limitations": "Yes. The main limitation of using discrete code is the computational tradeoff as discussed in Section 5.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Quality:\n- The paper is technically robust and presents a well-executed system that integrates several recent advances (self-supervised vision encoders (SEMs), discrete diffusion models, and DiT). \n- SOTA performance on unconditional ImageNet generation, surpassing both unconditional and conditional baselines. The authors run extensive experiments comparing discrete vs. continuous conditioning, ablate different DLC configurations (e.g., token count vs. vocabulary size), and show clear improvements in both fidelity and diversity. \n- The compositional generation capability, enabled by simple token-level mixing, is a novel and effective demonstration of the model's expressiveness. Furthermore, the text-to-image results -- though limited in scope -- demonstrate that a pretrained LLM can be repurposed to generate discrete image codes, offering a flexible and low-resource alternative to traditional text-conditioned diffusion pipelines.\n\nClarity:\n- The writing is generally clear, and the paper does a good job tying its contributions back to its stated motivations (i.e., improving fidelity, ease of sampling, and compositionality). \n\nSignificance:\n- The work presents a meaningful shift in focus: instead of improving diffusion model architectures, it emphasizes improving the conditioning representation. By doing so, it achieves not only higher fidelity but also new generative functionality, such as compositional image synthesis and a bridge between LLMs and image diffusion. The idea of using discrete tokens as a modular interface between modalities (text and image) is compelling and could influence future work in scalable, controllable generation.\n\nOriginality:\n- The paper presents a novel integration of known techniques to produce a new capability. While discrete latent representations and self-supervised embeddings are not new, using them as conditioning signals for diffusion models, and demonstrating their compositionality via token-level mixing, is a novel and creative contribution. \n- Additionally, the use of a language model to generate image tokens directly is an original twist on text-to-image generation, enabling a lightweight and modular alternative to large-scale CLIP-based pipelines.\n\nWeaknesses:\n- The pipeline involves multiple stages (encoder training, token inference, discrete diffusion prior training, image diffusion), which demands substantial compute. This may hinder reproducibility and adoption in resource-constrained environments.\n- The methodology is presented in a logical progression. However, some technical details could benefit from more intuitive explanation --especially the use of Simplicial Embeddings, the SEDD discrete diffusion prior, and the rationale for averaging DLC token embeddings instead of using more expressive conditioning mechanisms like cross-attention.\n- The choice to average token embeddings for conditioning, rather than using sequence-aware mechanisms (e.g. cross-attention), may limit the expressiveness and spatial control of generation.\n- The text-to-image pipeline involving LLaDA is quite dense and could be clarified further for readers unfamiliar with diffusion LMs.\nEvaluation scope: All experiments are on ImageNet (256×256), with limited exploration of more complex datasets (e.g., multi-object scenes, higher resolutions). It’s unclear how well DLCs generalize beyond the object-centric domain.\n- While promising, the text -> DLC -> image pipeline is evaluated on just a few hand-picked prompts. There's no quantitative comparison to standard text-to-image models, nor is it clear how broadly the LLM generalizes to unseen concepts.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal. Most concerns are addressed except the systematic evaluation. Overall the paper is good and I am keeping my score as borderline accept.", "rebuttal": "Thank you for your thorough review and constructive feedback. We are glad you appreciate our contributions and the overall proposed system. We respond to comments and questions below\n\n## Computation cost of training DLC\nThe overall computation is comparable to baselines methods (e.g. RCG, REPA). Moreover, the largest computational cost is the image generator (see paragraph Computational resources). As shown in Figure 4, training the image generator with DLC converges much faster than SOTA baselines. Thus, DLC may enable institutions with less compute to participate in image generation research.\n\n## Reproducibility and applicability of the work\nTo enable reproduction and application of the work, we release every model (the SEM encoder, the DiT, the SEDD and the fine-tuned LLADA) and the ImageNet DLC dataset. Everything is released as a huggingface model or dataset, which makes it straightforward for researcher and practitioner to build on top. We also provide the code for training and evaluating every model and generating the ImageNet-DLC dataset.\n\n## Using the cross-attention mechanism instead of averaging the token embedding\nThis is a great suggestion. We could use a cross-attention mechanism, similar to the one used in SOTA text-to-image models; e.g. stable-diffusion. This mechanism would enable a contextual conditioning at every attention block which would likely improve performance and a more straight-forward application of DLCs to existing T2I. For this work, we argue that simple averaging leads to a closer apple-to-apple comparison to the baseline methods for unconditional generation.\n\n## On text-to-image evaluation\nWhile our primary focus is improving image representations for image generation, we also explored text-to-image generation as a proof-of-concept on how such representation may be used in that setup (i.e. as tokens for a LLM). Due to compute and data limitations, we did not aim to outperform existing SOTA text-to-image models. Instead, we aimed to demonstrate the potential of DLC as a bridge between language and vision, and we hope our initial results inspire future work by the broader community.\n\n## How can DLCs generalize to datasets beyond ImageNet?\nAs shown in Figure 6, Figure 7 and Figure 12 in the Appendix, DLCs can generalize to sample images outside of ImageNet. For example, the image generator can generate teapot in Antarctica, despite ImageNet not having any images of teapots in Antarctica. However, as we see in Figure 12, ImageNet has images of teapots and images of snowy mountains (presumably Antarctica). Thus, certain components of the DLC must encode concepts about teapots while others must encode concepts about snowy mountains, as noted by the reviewer. This shows that a SSL objective + an inductive bias for learning discrete representation (i.e. SEM) is enough for learning such concepts. We refer the reviewer to Section 4.4 of [[0](https://arxiv.org/abs/2204.00616)] for an qualitative and quantitative analysis of the SEM representation itself.\n\n## How would you apply DLCs on existing T2I models?\nWe believe that it would be straight-forward to use DLCs in existing T2I models using the cross-attention module proposed by the reviewer. A straightforward approach is to apply the architecture of [[1](https://arxiv.org/abs/2112.10752)] and replace (or add) DLCs conditionally generated from a text prompt as conditioning. Such DLCs may be projected onto a vector of appropriate dimension using a learnable linear projection.\n\n## Can we do DLC->text\nYes, the fine-tuned LLM can output text tokens given a DLC. Qualitatively, the outputs are reasonable and we believe that direction is exciting as it enables to combine both image and text tokens within the same generative model.\n\n[0] https://arxiv.org/abs/2204.00616\n[1] https://arxiv.org/abs/2112.10752"}]

Author rebuttal:

Meta-review: This paper presents a novel method that explores learning discrete latent representations from images in a self-supervised way for conditional/unconditional image generation with diffusion models. Reviewers acknowledged the motivation, method design, and strong performance of the proposed method, while initially raising some concerns, such as high computation cost, unclear generalizability to other datasets, and limited text-to-image generation results.

After the rebuttal, the authors addressed most of the concerns, and all reviewers agreed to accept this paper. AC read all the reviews, author rebuttals, and the paper, and believes this is a good paper for generative modelling and recommends acceptance.
Decision: Accept (poster)