Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling
Authors: Viacheslav Meshchaninov
Egor Chimbulatov
Alexander Shabalin
Aleksandr Abramov
Dmitry Vetrov
Venue: NeurIPS 2025 poster ()
Keywords: latent diffusion
text generation
language models
autoencoders
Primary area: deep_learning
TL;DR: We propose Cosmos, a diffusion-based text generation method that compresses text representations by 8×, generating high-quality text over twice as fast as standard diffusion models.

Abstract: Autoregressive language models dominate modern text generation, yet their sequential nature introduces fundamental limitations: decoding is slow, and maintaining global coherence remains challenging. Diffusion models offer a promising alternative by enabling parallel generation and flexible control; however, their application to text generation is hindered by the high dimensionality of token-level representations.
  We introduce Cosmos, a novel approach to text generation that operates entirely in a compressed, smooth latent space tailored specifically for diffusion. This space is learned using an autoencoder trained simultaneously for token-level reconstruction and alignment with frozen activations from a pretrained language encoder, providing robust semantic grounding and enabling effective perturbation‑based augmentations.
  Empirically, we demonstrate that text representations can be compressed up to $8\times$ while maintaining generation quality comparable to token‑level diffusion models. Furthermore, increasing the latent sequence length allows \textsc{Cosmos} to surpass both diffusion‑based and autoregressive baselines. We evaluate Cosmos on four diverse generative tasks including story generation, question generation, summarization, and detoxification and compare it with various generative paradigms.
  Cosmos achieves comparable or superior generation quality while offering more than $2\times$ faster inference. Code is released at https://github.com/MeshchaninovViacheslav/cosmos.

CosMOS: Compressed and Smooth Latent Space for Text Diffusion Modeling
Viacheslav Meshchaninov
Egor Chimbulatov
HSE University Constructor University ymeshchaninov@hse.. ru
HSE University
echimblaatov@ss..
Alexander Shabalin HSE University
Aleksandr Abramov DaluteDevices andri17720gmail.com com
Dmitry Vetrov
Constructor University
Constructor University amshabal in@hse. ru
dvetzov@cousttuucor university
Abstract
Autoregressive language models dominate modern text generation, yet their se- quential nature introduces fundamental limitations: decoding is slow, and main- taining global coherence remains challenging. Diffusion models offer promising alternative by enabling parallel generation and flexible control; however, their application to text generation is hindered by the high dimensionality of token-level representations, We introduce COSMOS, novel approach text generation that operates entirely in a compressed, smooth latent space tailored specifically for diffusion. This space learned using autoencoder trained simultaneously for token-level reconstruction and alignment with frozen activations from a pretrained language encoder, providing robust semantic grounding and enabling effective perturbation-basee augmentations. Empirically, we demonstrate that text rep- resentations can be compressed up 8x while maintaining generation quality comparable to token-level diffusion models. Furthermore, increasing the latent se- quence length allows COSMOS to surpass both diffusion- based and autoregressive baselines. We evaluate COSMOS on four diverse generative tasks including story generation, question generation, summarization, and detoxification and compare it with various generative paradigms. COSMOS achieves comparable or superior generation quality while offering more than 2x faster inference. Code released at GitHub.
Introduction
Autoregressive (AR) language models are facto gold standard for text generation [29, 30, 6]. By factorizing the probability of sequence into product of conditional token probabilities, they transform the global generation task into a series of local next-token prediction tasks that can be optimized efficiently via teacher forcing [42].
The same sequential factorization, however, creates several structural bottlenecks. First, decoding is inherently sequential: the time to produce a sequence grows linearly with its length because each token must await the completion of its predecessor. Second, the process suffers from exposure bias: single early mistake contaminates the context for all subsequent predictions and cannot be corrected without restarting generation [32, 2, 11]. Third, maximizing local log-likelihood encourages the
Corresponding author: vmeshchaninov @ @hsc.ru
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

model to privilege fluency over factual accuracy, resulting in the well-documented phenomenon of hallucination [21, 13]. Finally, because decisions are made token-by baten-by-token, the model lacks an explicit global plan and therefore struggles to maintain long-range logical or narrative coherence [44].
In stark contrast, computer vision has been reshaped by diffusion models [12, 27], particularly by la- tent diffusion [33, 5, 4J, which first compresses a high-resolution image or video into compact latent representation and then performs the diffusion process in that space. Operating in a low-dimensional latent manifold reduces computational cost by orders of magnitude and enables breathtaking advances in image and video synthesis. Yet recent works [14, 37] also show that diffusion is highly sensitive to the geometry of the latent space: poorly designed latent representations can destabilize training and degrade sample quality, underscoring the need for principled mechanisms to construct diffusable' representations.
Motivated by these insights, we revisit the foundations of text representation. We hypothesize that the token-level encoding used in contemporary language models is heavily overparameterized for the purpose of sequence generation. Building on the successes of latent diffusion in vision, we pose the following question: How far can we compress textual information into a compact latent space while still matching or even exceeding the generative fidelity of conventional token-level representations?
To answer this question, develop an autoencoder that maps text into lower-dimensions space and then train a diffusion model directly in that space. We show empirically that naively minimizing token-reconstruction loss yields brittle latent geometry that hampers diffusion, whereas introducing robustness- and amoothhess-orientee objectives produces a well-behaved manifold conducive to high-quality diffusion synthesis. Our experiments demonstrate that with the right training regime, latent diffusion not only rivals but in several settings surpasses traditional token-level baselines.
These findings challenge the prevailing assumption that token-level autoregression is indispensable for language generation and position latent-space diffusion as a powerful alternative paradigm for future large-scale language models. Our key contributions are as follows:
We propose COSMOS - a training recipe for a COmpressed and SMOoth latent Space, that allows train a diffusion model in more compact latent space while matching the quality of token-level diffusion baselines.
We demonstrate that modestly scaling the number of latent vectors enables latent-space diffusion to surpass both token-level diffusion and autoregressive models on an unconditional text generation task.
On standard text-generation benchmarks, the proposed latent diffusion achieves up to 2x faster sampling than conventional token-level diffusion models, while matching or slightly exceeding them in quality and diversity.
2 Related work
Early efforts to bring latent diffusion models into natural language generation treat the diffusion module mainly as a pre-processor that supplies conditioning vectors for an autoregressive decoder. LD4LG [19] trains a diffusion model on compressed hidden states from BART, then feeds the generated latent vectors into the BART decoder to produce text. PLANNER [47] adopts a similar two-stage recipe: a fine-tuned BERT encoder produces 16-token variational latent code, diffusion model refines that code, and a GPT-2 decoder generates text from this code. While both systems improve controllability, their heavy reliance powerful autoregressive decoders makes it difficult to isolate and evaluate the intrinsic generative capacity of the latent-space diffusion itself.
A complementary line of work applies diffusion directly to the continuous embeddings of pretrained encoders. TEncDM [36] demonstrates that full-length BERT representations can be modeled with Gaussian diffusion and decoded into coherent text without an intermediate autoregressive step. Crucially, however, TEncDM retains the original sequence dimensionality, leaving open question: How aggressively can such representations be compressed before generation quality collapses?
Our study closes this gap. We devise training procedure that shrinks BERT-level representations by up to 8x while preserving, and in some cases enhancing, their suitability for latent-space diffusion.
Diffusable representations refer to latent spaces that permit effective modeling by diffusion process.

---

Our study builds upon the approaches reviewed here. For additional context and discussion of related topies please see Appendix B.
3 Preliminary
Diffusion models [12, 27] learn data distribution by reversing progressive noising process. Given a clean sample 20 ~ Pdata- the forward dynamic corrupts the imput with Gaussian noise whose magnitude is controlled by continuous time index t [0, 1]:
Zt Von 20
o4,
NO,, I),
where the noise schedule at is monotonically decreasing with ap 1 and 01 0. A neural denoiser 20(2, t) with learnable parameters 0 trained to invert this corruption by minimising the denoising- -e----coooeemiinngg objective
CDM
Ezorpen, t- e~,1. [|
=o (= ()
At inference time the model is applied iteratively from t = 1 to t 0, gradually transforming pure noise into a realistic sample.
3.1 Latent diffusion
Latent Diffusion Models (LDMs) [33, 4] improve both compute efficiency and quality by learning the diffusion process in compact latent space rather than in pixel or token space. An autoencoder with en- coder E and decoder D is first trained to achieve high-fidelity reconstruction, w = D ((E(w) = w, where z = ([w) denotes the low-dimensional latent. Eqs. (1)(2) yield a diffusion model trained in latent space whose network can be shallower, and faster than counterpart operating in the original space, yet still achieves comparable perceptual quality after decoding D(z).
In the remainder of the paper adopt this framework for textual data: compress contextualised text representations into the latent space, and train a diffusion model that operates within this space.
Methodology
4.1 Overview
Frozen text encoder. We initialise the pipeline with a pretrained contextual encoder Fxes (BERT- base [7] by default) that remains frozen during all subsequent training stages. For input sequence of tokens w = (Wi,..., W the encoder produces a matrix of hidden states h = Evsu (w) E R where d 768. Each row supplies a semantically rich representation of its corresponding token [38].
These representations provide high-fidelity starting point for the compression stage.
Compressor. To distill the variable-length hidden states into compact of latents, we employ Perceiver Resampler architecture [1]. It is 12-layer transformer [39] which every block replaces self-attention with cross-attention. Concretely, let E gx be set of learnable vectors initialized randomly and kept at a fixed length N < L. In each block, these vectors act as queries (Q uW while keys and values are formed by projecting a concatenation of u with the text encoder outputs (K, V = u: h [W.y). Cross-attention therefore allows every vector u, gather information from the entire sequence of hidden states and from other vectors u, gradually refining u into semantically organized compressed representation. After the final block we obtain a fixed-lengee latent matrix z R Xxd which composes latent space for the diffusion process. For better interpretability of results we compress only along the sequence axis. Altering the embedding dimension d would require architectural changes in the diffusion network itself, conflating encoder effects with diffusion capacity and obscuring the variables we aim to isolate.
Latent normalization. Before starting the Gaussian diffusion process, we estimate global mean and standard deviation (14, o) E R xd on held-out corpus, and normalize each latent feature so that has zero mean and unit variance, z +- (z - 11//a. This step allows us to run pariance-preserving diffusion process.
3

---

CE loss
MSE loss
noise
001
W
002
W2
Comp
De- Comp
W
103
W1
BERT
masking
201
feature extraction
augmentation
compressor-decompresoo
decoding
Figure Overview of our training pipeline. A frozen BERT encoder extracts features, which are augmented before compression. A lightweight compressor-decompres decompressor pair is trained with both token reconstruction (CE) and MSE objectives to produce compact and perturbatton--rsilient latent representations.
Latent diffusion model. Eqs. (1)- Because N is small, the diffusion model runs faster in this space.
We train Gaussian diffusion model is space formed by z, following
Decompressor. Decompressor mirrors compressor in terms of architecture. It expands the fixed- length latent back to a sequence of L vectors h E IR 1xxd capped at Lmax 512 by default.
Token predictor. Finally, linear projection followed by softmax converts each vector in h into probability distribution over the vocabulary, yielding the generated text.
In our work, we refer to the combination of the text encoder and compressor as the encoder, while the decompressor together with the token predictor constitutes the decoder.
4.2 Learning a compact text latent space
The previous Section 4.1 outlined our end-to-end pipeline: frozen contextual encoder supplies token representations, compressor distils them into fixed-length latent matrix, and a diffusion model operates solely this latent space. We now zoom on the compression and decompression stages.
4.2.1 How compact can text latents be?
Figure 2 reports token-level reconstruction accuracy on WIKIPEDIA dataset
1.00
function of the latent sequence
Accuracy
32x lossless
length N. The compressor outlined in $4.1 is trained using token-level cross-entropy objective. Remarkably, lightweight compressor equipped with only 12 transformer blocks is able encode 512-token sequences into only N 16 latents with 100% reconstruction accuracy, achieving 32 compression relative to initial 512 hidden states produced by BERT.
text compression
0.50
0.00
512
Number of Latent Vectors (N)
These results paint an encouraging picture: such a compact latent manifold can be modeled generatively, text generation could proceed in a space whose dimensionality much smaller than that of the original token embeddings, promising dramatic speed-ups. However, as we show further, high reconstruction accuracy by itself does not imply that the latent manifold suitable for generative modeling.
Figure 2: Token-level reconstruc- tion accuracy on WIKIPEDIA (512 tokens) function of N.
4.2.2 Learning robust compact text representations
The reconstruction study ($4.2.1) reveals aggressive text compression can loss-free. However, for robust diffusion generation compressed space should satisfy additional requirements. Empiri- cally, we observe that if the latent manifold lacks smoothness and robustness (Section 5), a Gaussian diffusion model fails sample latents that yield high-quality texts. To improve diffusability of

---

the latent space, we employ three complementary strategies to the autoencoder training. Figure 1 provides an overview of the proposed approach.
MSE regularisation on encoder activations. Alongside the cross-entropy loss between the original tokens and their reconstructions, add a mean-squared error penalty between the frozen text encoder outputs h and their reconstructions h. This auxiliary objective forces the compressor to preserve the semantics carried by the contextual representations h.
Activation-space perturbations. In order to teach compressor to extract additional features from h instead just preserving its semantics, we apply perturb-and-reoover training procedure. Concretely, we sample augmented view h', pass it through the compressor-decompper reconstructed representation h', and minimise MSE(h, hi), forcing both compressor and decom- pressor to remain invariant to the perturbation. Two perturbations are applied with equal probability within every batch:
pipeline to obtain
Random masking: 30% of the vectors of are set to zero, and (b) Gaussian noise: after normalising h with the pre-computed statistics, we inject noise via
h' = ā v1 - 82 with 8 = 0.7 and & ~ ~VO I).
These augmentations force the autoencoder tolerate partial information loss and promote smooth interpolation between adjacent latents (see Section 5.1).
Latent-space augmentation. We also apply augmentation directly to the latent matrix z. Since the number of latent vectors N can be order of magnitude smaller than the token count L,, masking randomly
whole latent vectors would annihilate too much information. Instead, during training zero out a fixed proportion p of the individual features inside each latent vector. This fine-grained sparsification encourages neighbouring features within every latent vector to store redundant cues about one another, so that the representation remains intelligible even when a subset of features is excised, thereby making the manifold more robust to small latent perturbations..
5 Latent-space properties that facilitate diffusion training
This section embarks exploration of the text-latent manifold, asking which of its intrinsic features govern the performance of diffusion model. All experiments reported here employ autoencoders that compress 128-token texts into 16 latent vectors. The autoencoders are trained on WIKIPEDIA dataset, whereas the diffusion models are optimised on the smaller yet high-quality ROCSTORIES [23] dataset. Across all experiments, we generate 1 0000 samples and report each metric as the average over these samples. Our experiments spotlight two indispensable qualities manifold smoothness and resilience perturbations. The remainder of the section introduces analysis that quantifies both attributes.
5.1 Smoothness of the latent manifold
Once the autoencoder has been trained, each text corresponds set of latent vectors that can be decoded back into the original text, w = Dzz). Nonetheless, the latent space far from fully charted: broad regions contain vectors to which no text has ever been assigned, leaving the behaviour of the data density there unknown. Because our diffusion model captures the distribution of texts via the distribution of these latents, yet observes only finite subset during training, its ability to generalize depends critically on how smoothly that density varies across the manifold. Put differently, the degree to which locally estimated score function extends unexplored territory is dictated by manifold smoothness. Inspired by PLANNER [47], we investigate this property by conducting the following experiment, which simultaneously evaluates performance at points the model does not encounter during training.
1. Select two random texts from the training corpus and encode them latent vectors (1)) (2)) E Xdd
Form a linear interpolation of the endpoints = (z) + (1 11) 2(2), where μ E [0, 1] and mid-range μ values steer z into regions unseen during training.
5

---

3. Apply diffusion noise (Equation (1)) to get z at different time steps to measure how unseen regions influence diffusion different noise levels.
4. Predict a clean latent 2g from 2 and decode into text w. Evaluate text plausibility with GPT-2 perplexity (PPI.), averaged over 1 000 random endpoint
pairs.
By analysing how perplexity (PPL) evolves with the in- terpolation coefficient μ and the diffusion timestep f, we obtain a fine-gra traeggriied picture of manifold smoothness and model robustness far beyond the training support. Figure 3 shows that autoencoder trained only with cross-entropy (CE) loss and our robustness robustness-orientted alternative behave identically when μ - 0 or μ + confirming that both models successfully reconstruct train-time input latents. As we move away from the endpoints, perplexity rises for both encoders, but it escalates far more rapidly for the CE baseline- - clear evidence that the latent manifold learned by our autoencoder is markedly smoother.
Time
CE Bascline
400
0.0
Cosmos
0.1
0.2
200
0.3
0.4
0.5
0.6
0.7
Figure 3: PPL of texts decoded from an interpolation of two latents for CoSMOS and CE baseline.
5.2 Reducing the train-inference mismatch
We empirically observe that when a Gaussian diffusion model is trained in the text latent space, a persistent train-inference mismatch often emerges: the latent vector 2 produced at sampling time differ markedly from the latent vector the encoder would assign to its own decoded text, E ((D(2) * å. This gap has undesirable effects. First, the decoder becomes unreliable because it must interpret latent vectors contaminated by the diffusion model. Second, the diffusion model repeatedly feeds itself inputs it saw during training, compounding errors over time. To enable high-quality text generation under this mismatch, both the decoder and the diffusion model must be robust to perturbations in the latent space. In the following sections, we demonstrate through experiments how our training strategy ($4.2.2) fosters such robustness, effectively decreasing train-inference mismatch.
Decoder robustness. To assess how well the decoder
1.00
CE(W, W)
tolerates perturbations, we inject Gaussian noise into latent vectors of real texts: Zooised a a e, where ~ NN00, I), decode the perturbed vector to obtain w, and measure its BLEU score against the original text w. Figure 4 shows that two components contribute most to decoder stabil- ity: (i) an explicit MSE loss the decompressor's 's output ($4.2.2), which prevents the final-layer norm from explod- ing, a common effect observed when training solely with cross-entropy loss [8], and (ii) latent masking ($4.2.2), which forces the compressor distribute information more evenly among latent features and remain resilient to partial dropout. Notably, the decoder supplemented with described train-tima modifications can reconstruct text almost perfectly even under substantial noise (a = 1).
0.75
MSE(h, fi)
0.50
Activation aug.
Latent aug.
0.25
0
Figure 4: Decoder robustness to latent noising with sequential addition of train- ing modifications.
Diffusion robustness during generation. Next, we test the diffusion model's sensitivity to small perturbations introduced mid-trajectory. For late generation step f, chosen from the second half the reverse process, where latent representations are already semantically meaningful, we add Gaussian noise with magnitude v and continue sampling. The deviation from the original trajectory is quantified by the MSE between the final estimates, || z 2hhified 1/2-
CE B Baseline
10-2
0.05
Cosmos
MSE
0.1
10-5
0.5
10-4
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Figure 5: Evaluating diffusion model ro- bustness under mid-trajectory noise in- jection.
As illustrated in Figure 5, the diffusion model trained in our latent space is visibly more stable.

---

Direct mismatch measurement. Table tracks the train- rein-inferenne mismatch, quantified as the MSE between 2 and ( (2))). Each proposed refinement tightens the gap. The consistent drop in error confirms that the en- hanced training procedure makes the autoencoder more resilient to diffusion perturbations. Together, these re- sults demonstrate that our training recipe simultaneously strengthens the decoder and the diffusion model, thereby shrinking the train inference gap.
Table 1: Train-inferenne mismatch for different autoencoder objectives.
Configuration
MSE (2, E E (2)))
CE(W, th)
0.721
+MSE h)
0.619
Activation aug.
0.582
Latent aug.
0.499
6 Empirical study of latent representations for diffusion modeling
This section investigates how the design of autoencoder training strategies and the degree of latent space compression influence the quality of text generation via diffusion.
6.1 Empirical analysis of autoencoder training regimes
In this part of our study we assess the contribution of each proposed latent space augmentation to the final quality the diffusion model. For the ablation study, we compress text from 128 BERT representations down to 16 latent vectors. All autoencoders trained on the WIKIPEDIA dataset, while the diffusion models are trained on the ROCSTORIES [23] dataset.
Evaluation metries. We evaluate the uncon- ditional generation capacity of models with three complementary metries that, together, cap- ture textual quality, lexical diversity, and dis- tributional fidelity. First, we report perplex- ity (PPL), computed with GPT-2 LARGE [30]. Second, we quantify diversity using the score
Table Comparison of autoencoder training regimes text generation quality. Features are added cumulatively from top to bottom. The gray- highlighted row indicates the selected configura- tion used in the final model.
Configuration MAUVE f
PPL
Div 1
div(y) II # unique n-grams in 2, where y de-
# n-grums in notes the set of generated texts. Their, to ensure that the model does not reproduce the training dataset during the generation evaluate the Memorization (MEM). Finally, because low per- plexity be achieved by simple repetitive
CE(w, W)
0.294.cm 036
71.5,9
0.298 .011
MSE(h, h)
0.464.024
57.1.7
0.344.002
+ Random masking (rate)
0.1
0.465.
54.5.4
0.356 003
0.2
0.565 023
50.5.s
0.342,
texts, we also compute the MAUVE score [28],
0.3
0.586 015
42.8.7
0.335 0033
which measures the distributional alignment be-
0.4
0.514,011 011
46.4.
0.344
tween generated and reference texts and thus of-
0.5
0.461,025
51.8.7
0.362,
fers broader view of generation quality. For ev- ery model we generate 1,000 samples and, when calculating MAUVE, draw an equally sized ref- erence subset from the held-out test split. The entire evaluation procedure repeated ten times with different random seeds, and we report the mean and standard deviation of the resulting scores across runs.
+ Gaussian noising (6)
0.5
0.679 .01
36.8.3
0.310 .003
0.6
0.724.013
35.6.7
0.324 .u4
0.7
0.767 u11
33.6.3
0.328 002
0.8
0.725.0m o18
36.4.7
0.333 0002
0.9
0.664 _23
44.3.s
0.294 _03
Latent dropout (rate)
0.1
0.771. ,014
33.9.7
0.325.pm
Results. Table demonstrates that each pro-
0.2
0.781 029
32.6.s
0.322, .322.004 0004
posed feature contributes significantly to the
0.3
0.812
31.9.5
0.328, _003
model performance. Remarkably, adding MSE
0.4
0.836 009
30.2.5
0.322, .00
penalty between the predicted and reference BERT activations alone boosts all quality met-
0.5
0.724.012
34.8.4
0.320.om
rics by about 50%. Subsequent augmentation of the BERT activations yields further leap, underscoring the value of intermediate contextual, semantically grounded representations. This stage rises MAUVE to 0.767, already achieving the performance obtained when the diffusion model trained on the full-length, uncompressed BERT representations (0.767 0.762 for TEncDM [36]). Finally, dropping random features from la-
7

---

tent vectors pushes MAUVE beyond 0.83, lowers perplexity to 30.2, keeping diversity essentially unchanged.
These results provide a clear answer to the question posed in the introduction of this study: text repre- sentations indeed be mapped into more compact latent space, where trained diffusion model performs comparably to traditional token-level counterparts without sacrificing the text generation quality. For all subsequent autoencoders we lock in the configuration highlighted in Table 2: the MSE penalty is kept, the random-maskine rate is fixed at 0.3, Gaussian noising is applied with d = 0.7, and latent masking rate is set to 0.4.
Additionally, we explore the use of variational prior in the latent space and observe that offers no clear advantage. A thorough analysis is provided in Appendix D.I.
6.2 Impact of scaling the number of latent vectors
In this section, examine how diffusion generation qual- ity varies when varying the number of latent vectors N.
1.0
BERT Representations
We keep the training pipeline fixed according to the hy- perparameters detailed in Section 6.1. We do not modify the embedding dimension d, because would necessitate architectural modifications to the diffusion model, making difficult to isolate the impact of latent space diffusability from changes in model capacity and structure.
MAUVE
0.6
0.4
CE Baseline Cosmos
0.0
2
8 16
32
64 128
Number of Latent Vectors (N)
Figure 6 compares baseline cross-entropy compressor to the robustness-orienta compressor introduced in Sec- tion 4.2.2. We observe that for CE compressor decrease of N leads to rapid degradation in quality. In contrast, the proposed autoencoder maintains high generation quality even under substantial compression. achieves an 8x reduction in latent sequence length while surpassing the quality of uncompressed representations.
Figure 6: Impact of scaling N on diffu- sion generation quality.
Table 3 provides further insights by comparing generation quality across different numbers la- tent vectors N. Although configuration N- 16 matches the quality of BERT representations baseline, observe notable quality gap com- pared to the configuration with N=128: the
Table 3: Impact of scaling the number of latent vectors N on unconditional generation quality.
N
MAUVE 1
PPL +
Div 1
Source
0.953
21.7
0.403
MAUVE score decreases from 0.940 0.836,
BERT repr.
0.762
29.19
0.295.0u
while perplexity increases from 26.3 to 30.2.
However, adopting a less aggressive compres- sion to - (a 4x compression) results in only minor quality degradation, still signifi- 8
2
0.172.018
109.5.
0.344 o04
4
0.197 .016
70.1
0.354.0o 003
0.403 .077
51.9.7
0.327 002
cantly outperforming the baseline. Based on these observations, together with results from
16
0,836 009
30.2.5
0.322
32
0.901 pos
27.3.4
0.346 on4
Section 7, we recommend limiting latent com-
64
0.923 8o9
26.7.2
0.347 003
pression to approximately 4x to optimally bal- 128
0.940 ø11
26.3.4
0.346,gu
ance quality and computational efficiency.
Comparison across generative paradigms
We evaluate the performance of COSMOS on four distinct text generation tasks: unconditional story generation (ROCStories [23]), text summarization (XSum [25]), detoxification (ParaDetox [17I). and question generation (SQuAD2.0 [22]). For unconditional story generation we employ metrics discussed in Section 1. For text summarization and question generation, we report BERTScore (BS) [46]. The detoxification task is evaluated using BLEU-4 (BLEU). We further assess models' efficiency by comparing inference time for unconditional generation on sequences with 128 and 512 tokens. Additional metrics and implementation details are provided in the Appendix D.2.
Baselines. We benchmark our proposed method against a diverse set of generative paradigms. These include Gaussian diffusion embeddings (DiffuSeq [10], SeqDiffuSeq [45], AR- IR-Diffusion [43]),
8

---

Table Comparison with autoregressive and diffusion baselines across four generative tasks. Infer- ence time (in seconds) is reported for sequence lengths of 128 and 512 tokens. The best-performing scores are shown bold, while the second-best scores are underlined.
ROCStories
XSum ParaDetox SQuAD2.0
Time (s)
Method
MAUVE
P + Div f Mem 4
BS f
BLEU
BS t
128
512
Source text
0.953
21.7
0.403
0.365
GPT2
0.789
20.5
0.252
0.455
0.690
0.677
0.680
1.2
42.8,
GPT Neo
0.720
19.9
0.258
0.469
0.621
0.610
0.665
2.6,1
45.2,
AR-Diffusion
0.066
41.8
0.101
0.540
0.568
0.647
0.569
226.4.
6022.1
DiffuSeq
0.086
50.5
0.124
0.516
0.588
0.679
0.563
215.91.2
1565.7
SeqDiffuSe
0.103
29.3
0.137
0,663
0.617
0,688
0,574
100.5
601.5
TESS
0.061
22.4
0.163
0.550
0.627
0.693
0.667
1441,
5984.0
SEDD
0.598
70.8
0.336
0.325
0.576
0.666
0.443
15.01.1
60.31.0
LD4LG
0.716
30.6
0.331
0.432
0.702
0.708
0.641
27.9.5
102,2
TEncDM
0.762
29.1
0.295
0.438
0.699
0.619
0.703
29.6.1
180,3
COSMOSN
0.836
30.2
0.322
0.394
0.654
5.8.
COSMOS
0.940
26.3
0.346
0.383
0.704
0.694
0.708
35.1,1
36.6.
troplex-basedd diffusion (TESS [20]), masked diffusion (SEDD [18]), alternative latent diffusion baselines (LD4LG [19], TEncDM [36]), and autoregressive models (GPT-2 [30] and GPT-Neo [3]). All models operate within comparable parameter scale, ranging from approximately 100M to 200M. To ensure fairness, all baselines were faithfully reimplemented trained on generative tasks using training protocols closely aligned with those described by their original authors.
Autoencoder setup across tasks. To perform latent- diffusion, begin by training au- toencoders on WIKIPEDIA dataset, varying the input sequence length. For tasks with relatively short input, such unconditional story generation and text detoxification, of the autoencoder that map 128-token input into either 16 or 128 latent vectors, resulting in the COSMOS = 16 and COSMOS 128 configurations, respectively. In contrast, conditional generation tasks such as summarization and question generation require substantially longer input contexts. For these settings, we employ a third autoencoder configured compress 512-token sequences into 128 latent vectors. Accordingly, we do not report results for long-context tasks for COSMOSN sych aggressive compression fails to yield plausible outputs in these long-context scenarios. We adopt 1- (4x compression) for long-context tasks as it strikes a favorable balance between generation quality and computational efficiency, as demonstrated in the scaling analysis in Section 6.2.
design two variants
as
Results. shown in Table 9, COSMOS 16 achieves strong performance relative its closest latent diffusion baseline, TEncDM. It consistently outperforms TEncDM across most evaluation metrics, with the exception of unconditional perplexity, while offering significantly faster generation. Scaling up to bigger latent representation, COSMOS = 128 further advances generation quality. On ROCStories, it achieves a MAUVE score 0.940, closely approaching the human reference score of 0.953, and substantially outperforming the best autoregressive baseline (GPT2), which reaches only 0.789. Although COSMOS 128 lags behind GPT2 in perplexity, this gap reflects an evaluation bias: perplexity inherently favors autoregressive models, particularly when assessed using the same decoding objective they were trained on.
In generation tasks that involve longer input contexts, COSMOS = 128 consistently matches or slightly exceeds the performance of both diffusion- based and autoregressive baselines, while sub- stantially reducing inference time. This efficiency stems from core advantage of latent diffusion: unlike autoregressive decoding, where inference time grows linearly with sequence length, diffusion models operate with a fixed number of sampling steps. Although this cost is less favorable for short sequences, the benefits become increasingly pronounced as input length grows. This favorable scaling behavior makes latent diffusion as a promising direction for future research.

---

8 Experiments on OpenWebTexx
In this section, we present empirical evaluation of proposed method on the large-scale OpenWebText (OWT) [9] dataset.
8.1 Impact of Diffusion Model Depth
We train three diffusion models with vary- ing depths of 12, 24, and 48 layers, which correspond approximately 0.12B, 0.25B, and 0.5B parameters, respectively. For these experiments, the autoencoder and la- tent space configuration are held constant. We selected our best- cest-peforming autoen- coder from preliminary experiments, which utilizes the largest latent space of 128 vec- tors for 128-token sequences.
Table 5: Scaling results on OpenWebTers for 128-token generation. Increasing the number of layers in the dif- fusion model transformer improves generation quality and diversity. Best results in bold.
Model Size
MAUVE
PPL 4
Div
Mem |
0.12 0.25
0.849
97.6
0,492
0.135
0.914
91.2
0.546
0.124
0.5
0.923
89.7
0.554
0.125
The results presented in Table clearly indicate positive scaling trend. As the number of layers in the diffusion model increases, both generation quality and diversity improve. This demonstrates that increasing the capacity of the diffusion component is effective strategy for enhancing performance, even when the autoencoder's architecture remains fixed.
8.2 Comparison on Long-Sequence Generation
We further assess our model's capabilities on the challenging task of generating 512- token sequences. We compare our model, COSMOS, in two configurations: without compression (N 512 latent vectors) and with 2x compression (NV 256). These benchmarked against TEncDM [36]. latent diffusion baseline, and GIDD [40], strong discrete diffusion model. The sults, presented Table 6, offer several key insights into the advantages of our ap- proach.
Table 6: Comparison with the GIDD masked diffusion model on Open WebText for 512-token generation. Our model (CosmOsN=512) shows significantly better qual- ity and coherence.
Model
MAUVE
PPL L
Div
Mem |
Source text
0.968
23.2
0.465
0.036
GIDD
0.286
228.3
0.588
0.112
TEncDM
0.228
118.6
0.324
0.102
COSMOS
0.263
84.3
0.368
0.156
COSMOSN=512
0.702
55.0
0.319
0.180
First, our primary model, COSMOS demonstrates significant performance gap over the
discrete diffusion paradigm. It outperforms GIDD by large margin, achieving a MAUVE score that is nearly three times higher (0.792 vs. 0.286) and a perplexity that is almost four times lower (55.0 vs. 228.3), indicating superior text quality and coherence.
Second, the results confirm the expected trade-off between compression and generation fidelity. While our compressed model, COSMOSN=256 shows a degradation in performance compared to its uncompressed counterpart, it still outperforms TEncDM, a baseline that works with full-length, uncompressed textual representations. This result powerfully highlights the efficiency of our latent diffusion framework for text generation.
Conclusion
In this paper, we present COSMOS, novel approach to text generation that shifts the focus from high-dimensional token-level representations to compact latent space tailored for diffusion modeling. By learning a compressed and smooth latent representation through carefully designed autoencoder objectives, COSMOS enables high-quality generation while reducing inference time. Our empirical results show that, COSMOS matches or outperforms traditional token-level diffusion and autoregres- sive baselines across diverse tasks. Our findings challenge the dominance of token-level models and highlight latent diffusion promising direction for building fast, high-quality language models.
10

---

Acknowledgements
The work was supported by the grant for research centers in the field of Al provided by the Ministry of Economic Development of the Russian Federation in accordance with the agreement 10000CC31392$P400002 and the agreement with HSE University No 139-15-2025-009, This research was supported in part through computational resources of HPC facilities at HSE University.
References
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira,
O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: visual language model for few-shot learning, 2022.
[2] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence predic- tion with recurrent neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
[3] S. Black, L. Gao, Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow, Mar. 2021.
[4] A. Blattmann et al. Align your latents: High-resolution video synthesis with latent diffusion models. 2023.
[5] A. Blattmann al. Stable video diffusion: Scaling latent video diffusion models large datasets. 2023.
[6] T. B. Brown, B. Mann, N. Ryder, and et al. Language models are few-shot learners. In Advances Neural Information Processing Systems (NeurIPS), 2020.
[7] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo- gies, volume I (long and short papers), pages 4171-4186, 2019.
[8] T. Gao, X. Yao, and D. Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.0882) 2021.
[9] A. Gokaslan and V. Cohen. Openwebtext corpus.
http://Skyylinn007, www.:0007gggtttt github. 1o/
OpenWebTextCorpuss 2019.
[10] S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.0880 08.0333 2022.
[11] T. He, J. Zhang, Z. Zhou, and J. Glass. Exposure bias versus self-recovery: Are distortions really incremental for autoregressive text generation? arXiv preprint 190.v 2019.
[12] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[13] Z. Ji, G. Wang, B. Wang, and et al. Survey of hallucination in natural language generation. ACM Computing Surveys, 2023. to appear.
[14] Kouzelis, Kakogeorgiou, S. Gidaris, and N. Komodakis. Eq-vae: Equivariance regularized latent space for improved generative image modeling. arXiv preprint arXiv:2502.09509 2025.
(15) H. Laurençon, I.. Saulnier, T. Wang, C. Akiki, A. Villanova del Moral, T. Le Scao, I.. Von Werra, C. Mou, E. González Ponferrada, H. Nguyen, et al. The bigscience roots corpus: 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:3 1809- 3 31826, 2022.
[16] C.-Y. Lin. Rouge: package for automatic evaluation of summaries. Text summarization branches out, pages 74-81, 2004.

---

[17] V. Logacheva, D. Dementieva, S. Ustyantsev, D. Moskovskiy, D. Dale, I. Krotova, N. Semenov, and A. Panchenko. ParaDetox: Detoxification with parallel data. In Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 6804-6818, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[18) A. Lou, C. Meng, and S. Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310. 16834, 2023.
[19] J. Lovelace, V. Kishore, C. Wan, E. Shekhtman, and K. Q. Weinberger. Latent diffusion for language generation. Advances in Neural Information Processing Systems, 36:56998-57025, 2023.
[20] R. K. Mahabadi, H. Ivison, J. Tae, J. Henderson, I. Beltagy, M. E. Peters, and A. Cohan. Tess: Text-to-text self-conditioned simplex diffusion. arXiv preprint arXiv:2305.08379 2023.
[21] J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1906-1919, 2020.
[22] S. Minace, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao. Deep learning- based text classification: a comprehensive review. ACM computing surveys (CSUR), 54(3):1-40, 2021.
[23] N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, 2016.
[24] N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In K. Knight, A. Nenkova, and O. Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 839-849, San Diego, California, June 2016. Association for Computational Linguistics.
[25] S. Narayan, S. B. Cohen, and M. Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint 18X8:1800800845 2018.
[26] S. Narayan, B. Cohen, and M. Lapata. Don't give me the details, just the summary! Topic- aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018.
[27] A. Q. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional Conference on Machine Learning (ICML), 2021.
[28] K. Pillutla, S. Swayamdipta, R. Zellers. J. Thickstun, S. Welleck, Y. Choi, and Z. Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816 2021.
[29] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018
[30] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 18):9, 2019.
[31] P. Rajpurkar, R. Jia, and P. Liang. Know what you don't know: Unanswerable questions for SQuAD. In Gurevych and Y. Miyao, editors, Proceedings the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, July 2018 Association for Computational Linguistics.
[32] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. In International Conference Learning Representations (ICLR), 2016.
12

---

[33] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[34] S. S. Sahoo, M. Arriola, A. Gokaslan, E. M. Marroquin, A. M. Rush, Y. Schiff, J. T. Chiu, and V. Kuleshov. Simple and effective masked diffusion language models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[35] Y. Schiff, S. S. Sahoo, H. Phung, G. Wang, S. Boshar, H. Dalla-torre B. P. de Almeida, A. Rush, T. Pierrot, and V. Kuleshov. Simple guidance mechanisms for discrete diffusion models. arXiv preprint arXiv:2412.10199 2024.
[36] A. Shabalin, V. Meshchaninov, Chimbulatov, V. Lapikov, R. Kim, G. Bartosh, D. Molchanov, S. Markov, and D. Vetrov. Tenedm: Understanding the properties of the diffusion model in the space of language model encodings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 251 10-25118, 2025.
[37] I. Skorokhodov, S. Girish, B. Hu, W. Menapace, Y. Li, R. Abdal, S. Tulyakov, and A. Siarohin. Improving the diffusability of autoencoders. arXiv preprint arXiv:2502. 14831, 2025.
[38] I. Tenney, D. Das, and E. Pavlick. BERT rediscovers the classical NLP pipeline. In A. Korhonen, D. Traum, and L.. Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593 4601, Florence, Italy, July 2019. Association for Computational Linguisties.
[39] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[40] D. von Rütte, J. Fluri, Y. Ding, A. Orvieto, B. Schölkopf, and Hofmann. Generalized interpolating discrete diffusion. arXiv preprint arXiv:2503.04482, 2025.
[41] G. Wang. Y. Schiff, S. S. Sahoo, and V. Kuleshov. Remasking discrete diffusion models with inference-ting scaling. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy.
[42] R. J. Williams and D. Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270-280, 1989.
[43] T. Wu, Z. Fan, X. Liu, H.-T. Zheng, Y. Gong, J. Jiao, J. J., Guo, N. Duan, W. Chen, et al. Ar- diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems, 06:39957-39977 2023.
[44] J. Ye, J. Gao, S. Gong, I.. Zheng, X. Jiang, Z. Li, and L.. Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. arXiv preprint arXiv:2410.14117, 2024.
[45] H. Yuan, Z. Yuan, C. Tan, F. Huang, and S. Huang. Seqdiffuseq; Text diffusion with encoder- decoder transformers. arXiv preprint arXiv:2212.103255, 2022.
[46] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. arXiv preprini arXiv: 1904099675 2019.
[47] Y. Zhang. J. Gu, Z. Wu, S. Zhai, J. Susskind, and N. Jaitly. Planner: Generating diversified paragraph via latent language diffusion model. Advances in Neural Information Processing Systems, 36:801 78 80190, 2023.
13

---

Appendix
A Dataset descriptions
15
Additional Related Work
15
B.1
Gaussian Diffusion on Embeddings
15
B.2
Simplex-Base Diffusion
15
B.3
Masked Diffusion
15
B.4
Latent Diffusion
16
Implementation details
16
C.1 Latent diffusion pipeline
16
C,2 Training and inference configuration
16
C,3 Inference time benchmarking configuration
16
D Additional autoencoder training analysis
16
D.I Marginal utility of variational prior in autoencoder
16
D.2 Additional details of comparison across generative paradigms
18
Limitations
18
Societal Impact
18
G Generation examples
18
14

---

A Dataset descriptions
This dataset [24] consists of five-sentence stories and serves as a well-established. small-scale, unconditional benchmark in language diffusion research, It contains a total of 98, 181161 instances, of which 88,161 are used for training, 10,000 for validation.
ROCStories
Wikipedia
For large-scale experiments, we use the English Wikipedia subset from the ROOTS
corpus [15]. The resulting dataset comprises 8.6 million sequences, with 50,000 held out for validation.
This dataset [26] is used for abstractive summarization and comprises 204,000 BBC news articles paired with summaries covering diverse topics (e.g., sports, politics). It includes 204,045 training instances, 11,332 validation instances, and 11,334 test instances. We truncate input articles to 512 tokens and limit reference summaries to 50 tokens.
XSum
This question-ansee answering dataset [31] requires identifying the text span that answers given question or indicating that no answer is possible from the provided context. For generative modeling, we reverse the task: given context and an answer, the model generates the corresponding question. The dataset contains 130,319 training and 11,873 test instances. Contexts are truncated to 512 tokens.
SQuAD2.0
ParaDetox For small-scale conditional generation experiments, we use the ParaDetox dataset [17], which comprises 19,766 pairs of toxic and neutral comments. Both context and target sequences are truncated to 40 tokens.
B Additional Related Work
In addition to the works discussed in the main paper, a significant body research has explored diffusion models for text generation. These can be broadly categorized follows.
B.1 Gaussian Diffusion on Embeddings
One line of work focuses on applying Gaussian diffusion directly to word embeddings. Models like DiffuSeq [10] and SeqDiffuSeq [45] are designed for toweencc-t--equence text generation tasks. DiffuSeq employs a partial noising strategy and trained end-to-end a classifier-fres free manner. SeqDiffuSeq utilizes encoder-decoder Transformer architecture and incorporates self- conditioning and an adaptive noise schedule to enhance performance. AR-Diffusion [43] integrates autoregressive principles into the diffusion process to better capture the sequential dependencies in language, allowing tokens on the right to depend on the generated tokens on the left. This achieved by using a dynamic number of denoising steps that varies with the token's 's position.
B.2 Simplex-Based Diffusion
Another direction explores alternative spaces for the diffusion process. TESS [20], for instance, applies fully non-autoregressive diffusion process on the logit simplex space. This approach, combined with a novel celf-conditioning technique, demonstrates strong performance on various natural language generation tasks while requiring fewer diffusion steps.
B.3 Masked Diffusion
Masked-token diffusion models adopt BERT-style prediction objective with iterative unmasking, enabling fast and parallel generation. For natural language, SEDD [18] established connection between score-based models and ELBO maximization, demonstrating the effectiveness of the ap- proach. Subsequent work, such as simple absorbing mask diffusion with a clean training recipe, has successfully narrowed the perplexity gap to autoregressive language models and supports flexible semi-autoregressivve decoding [34]. The quality of generation is also significantly boosted by various inference-time techniques. These include remasking for iterative correction [41], discrete guidance for improved controllability [35]. and hybrid methods that expand self-correction capabilities [40].
15

---

B.4 Latent Diffusion
In the realm of latent diffuusion, several alternative baseline models have been proposed. LD4LG [19] learns a continuous diffusion model in the latent space of pretrained mecoder-decoder model. This method samples continuous latent representations which are then decoded into natural language. proving effective for both unconditional and conditional text generation. Similarly, PLANNER [47] adopts a two-stage recipe where fine-tuned BERT encoder produces variational latent code, which is then refined by diffusion model before being translated into text by GPT-2 decoder. TEncDM [36], on the other hand, operates in the space of pretrained language model encodings, which, unlike embeddings, integrate contextual information.
C Implementation details
C.1 Latent diffusion pipeline
With diffusable latent manifold in hand ($4.2.2), now describe the diffusion model that operates entirely within this space. Our setup follows Eq. (2): we train a denoiser that receives a noisy latent 2, at diffusion time E [0, 1] and predicts the corresponding clean latent Zg. The loss is the simple mean-squared error / zo (zr,
I). where za shares parameters across all time-steps. In line with earlier work [19, 36], we employ self-conditioning:
50% of training updates the model is fed its own previous estimate of Zo as an additional input. This iterative refinement enables the model, inference time, reuse its past predictions, producing markedly crisper and more coherent text. We adopt the noise schedule introduced by TEncDM [36]:
I=tanntta /2 where parameter d controls he rate at which noise is injected during the diffusion steps. At inference time Euler solver with 200 uniform steps. The denoiser itself inherits the TEncDM [36] architecture. is 12-layer Transformer with 12 attention heads and hidden size of 768. In total, the network comprises roughly zs 130M parameters.
Conditional generation. For sequence- sequence-to-sequee to-sequence generation, train diffusion model in conditional setting. The model learns to reconstruct a noisy latent representation of the target text, conditioned the latent representation of the source generated by our autoencoder. Consequently, the source text's vector representation is also leveraged in a compressed format, enhancing both training and inference efficiency. The diffusion model conditioned on the source latent, which is first processed by 12-layer transformer encoder. This encoder extracts relevant features and derives suitable conditioning representation from the source latent. These encoded source representations are then injected into the diffusion model using cross-attention mechanisms. For conditional tasks, we initiate the training of our diffusion model by initializing its denoiser weights with those pretrained on an unconditional task.
C.2 Training and inference configuration
All models are trained on NVIDIA A100 GPUs. Detailed training configurations and approximate durations for both COSMOSN 128 and COSMOS are provided in Table 7.
C.3 Inference time benchmarking configuration
We measure the generation time for a batch of 512 samples per model on single NVIDIA A GPU using the bfloat16 data type. Inference settings for all baseline models follow the default configurations of their respective repositories, except for SEDD, where we adopt 32 diffusion steps in accordance with the original paper's 's setup for matching autoregressive quality. We report the mean and standard deviation across five independent runs.
D Additional autoencoder training analysis
D.1 Marginal utility of variational prior in text autoencoder
In this section, we explain why incorporating a variational prior does not significantly enhance the performance of our text autocncoder.
16

---

ROCStories
Wikipedia
XSum
SQuAD2.0
Paradetox
Optimizer
AdamW
Learning Rate
2e-4
(3,,82
(0.9, 0.98)
Warmup Steps
1000
Learning Rate Schedule
Constant
Weight Decay
0.01
Gradient Clipping
1
EMA Decay
0,9999
Batch Size
1024
Training Steps
200k
500k
100k
100k
10k
Max Seq Length
80
512
64
64
40
Max Context Length
512
512
40
Sampling steps
200
Schedule parameter
3
7
7
9
Autoencoder Training Time
14h
Diffusion Training Time
1d 5h
2d 12h
1d 1h
1d 9h
14h
Table 7: Training details for COSMOS across different datasets.
In a classical Variational Autoencoder (VAE), the encoder maps each input to the parameters of Gaussian latent distribution, i.e., mean vector μ and a (diagonal) variance vector o². Training minimises reconstruction term together with the Kullback-Leiblee (KL) divergence between the encoder distribution and an isotropic prior A(O,, , I). For latent matrix = E Nxd the KI. term is
N
Dru(N(E:H, (O, 1)) =
2 +
+ - llog o - 11.
Our text autoencoder trained with the following objective:
C = CE(w, w) + MSE(h, h)) + 3 (ra(N(ss,
where CE(w, w) is the token-level cross-entropy. MSE(h, h) matches reconstructed contextual representations, and 320 0 balances reconstruction fidelity against latent regularisation.
During training, the decoder receives stochastic latent sample
+ T O IO,,1)
so that the KL term nudges (n, σr) towards the prior. Increasing strengthens this pressure, producing a smoother latent geometry at the cost of higher reconstruction error; decreasing does the opposite as encoder tends to push latents away from each other.
In our experiments, we observed that the performance is highly sensitive to the choice of Table presents the results for different values of 3.
The best perplexity is achieved at 0.01, but the gain over the baseline (3 = 0) is marginal (33.6 + 32.6). For larger 3 the model collapses: at 3 = 1, perplexity explodes and MAUVE falls.
Table Impact the KI. weight on diffusion generation quality. Arrows indicate the preferred direction.
This limited improvement can be attributed to the decoder's robustness to Gaussian noise in the latent space. As illustrated in Figure 4, the decoder maintains reasonable performance even when sampling from the prior with 1, indi- cating that the latent space is sufficiently robust without strong KL regularization.
MAUVE 1
PPL 4
DIV
0.767
33.6
0.328
0.0001
0.733
39.6
0.329
0.001
0.764
33.3
0.328
0.01
0.765
32.6
0.326
0.1
0,658
39.6
0.329
0.011
677.1
0.587
Alternative regularisers, notably latent masking. yield larger and more robust improvements. We therefore omit the KI. term in the final model and rely solely on latent masking to shape the representation space.
17

---

D.2 Additional details of comparison across generative paradigms
Table Comparison with autoregressive and diffusion baselines across three generative tasks. The best-performing scores are shown in bold, while the second-best scores are underlined.
ParaDetox
Xsum
SQuAD2.0
Method
BLEU 1
J-Score +
R-1/2/L 1
BS
1
R-L 1
BS
GPT2
0.677
0.604
0.283/0.082/0.218
0.690
0.332
0.680
GPT Neo
0.610
0.492
0.231/0.04550.1171
0.621
0.245
0.665
AR-Diffusion
0.647
0.465
0.268/0.059/0.206
0.568
0.185
0.569
DiffuSeq
0.679
0.475
0.189/0.130/0.136
0.588
0.186
0.563
SeqDiffuSeq
0.688
0.486
0.286/0.067/0.213
0.617
0.194
0.574
TESS
0.694
0.587
0.317/0. 16/0.264
0.627
0.339
0.667
SEDD
0.666
0.001
0.200/0.033/0.133
0.576
0.086
0.443
LD4LG
0.708
0.580
100/0.00
0.702
0.211
0.641
TEncDM
0.619
0.496
0.319/..10700.55
0.699
0.323
0.703
COSMOSN=16
0.649
0.497
COSMOSN=128
0.694
0.554
0.328/0.11470.255
0.704 0.339
0.708
We also present extended results for conditional generation tasks using broader variety of evaluation metrics. For XSUM and SQUAD2.0, we report ROUGE scores [16], standard metric that evaluates text quality based on n-gram overlap with reference outputs. For PARADETOx, additionally include the J-score, which is defined as the product of style accuracy, fluenc,, and content preservation. Extended results are summarized in Table 9.
E
Limitations
There several limitations to study that point to promising directions for future work. First, jointly training the autoencoder and diffusion model remains an open research direction. Such an approach may significantly improve training efficiency. Second, the latent. dimensionality d is held fixed in all experiments, because changing d would necessitate redesigning diffusion backbone; a systematic sweep of this hyper--arameter is therefore left to dedicated follow-up work. Finally, report results with relatively small backbones of roughly 130M parameters to keep ablations rapid and compute budgets fair; scaling the architecture is orthogonal engineering effort that is expected to reinforce the empirical trends observed here.
F Societal Impact
Our research introduces an alternative modeling approach for language, centered on latent continuous diffusion. We hold the view that this method does not bring about significant societal risks exceeding those already connected with current language models.
G Generation examples
To give a qualitative sense of model behaviour, we first present representative unconditional gen- erations: Table 10 shows random samples from COSMOS and the TEncDM baseline on ROCStories, while Table 11 does the same for Wikipedia. We then move to the conditional setting, providing pepueecc-to-sqqueenn outputs for three benchmarks: summaries for XSum in Table 12,
18

---

Table 10: Randomly generated samples for ROCStories dataset.
TEncDM
COSMOSN=128
Amy and her friend Sue were excited about sixth grade together. The three of them visited Amy's s online shoe store. Amy's s friend Beth arrived at the shoe and Sue tried some dresses. The girls checked out and looked felt like she was being an idiot.
Amy wanted to buy new dress for the dance! She shopped around ten minutes. She tried on dress that was too big. Amy loved that the dress was too grab for her size. She found that was little too long wear!
the collection. Amy
Maggie was nasty girl in school. When she moved to small town, she had Mae friends. After school, sheared out and made no friends. Sometimes they called her back, and they left her alone. She finally understood why she needed to be friends.
Jay was sitting at home bored. was looking for an activity to keep him active. He decided to play game of haskethall. He played haskethall for an hour. He was able to have a fun time that afternoon.
My car was becoming very worn out. went to my mechanic get checked out. He told that had flat tire. went NCaven to get it fixed. still made it to work in no time.
Sam was working late. He didn't have money to pay all his bills. He was having trouble getting paid. Sam decided quit his work job. He able to get another job to pay bills.
My daughter moving to new school next year. She was nervous about moving this new location. afraid of leaker of all of her relationships. We figured it out and made some alterations the team. This is where she will graduating high school.
Debra hasn't 't written check in weeks. She's been very stressed and frustrated with paying bills. Last week, she acted horribly during her check. bank quickly ran a check up. To her shock, she found out that she owed nearly million in cash.
Tina never thought she needed her son enough to swim. So she begged him to do so while he was supposed to. So he practiced and swam all the time. But when it was time to go outside, he constantly got sore. Tina swim hours.
cool
John the library. The librarian told him read more books. John went the booktore. He tried to read 6 books. He couldn't 't decide which to buy.
stressed to let
detoxified rewrites for ParaDetox in Table 14, and Table 13.
examples for SQuAD 2.0 in
19

---

Table 11; Randomly generated samples for the Wikipedia dataset using COSMOSN=128-
The press said working conditions for the match did not have to be finalised. Even the organizers had planned for the date of the match to be 19 August 2017 to accommodate crowd of some 500, 000 people watching the match.
Bob Hall served Foundation founding president from 1990 1991. In 1993, Hansen was elected President of the College Bymphoschipelgeons Upon his retirement from this position, he also served as the chairman of the Scientific Advisory Committee of the College of Medical Surgeonsgeons (IMS) ). In 1996, publisher magazine selected Bob Hall as his candidate for the next U. S. presidential nomination for 2000 and 2001. He died October 26, 2013, at his home Tampa, Florida from the causes kidney failure at age of 71, caused by a cerebral kidney failure in 1980. He is buried at the
was excluded from the committee because Johnson refused to approve the text of the bill, and two senators refused to attend discussing the legal provisionss of the Seventh Amendment, but invited senior officials to hear it. Both the Congress House and the Weed Hermans challenged the committee's request to have the Seventh Amendment be argued in the Supreme Court's decision during the subsequent legislative hearings. Both attributed the premise premisel of Johnson's Amendment to internal law, which allocated the Congress to adopt executive decrees beginning in January 1699. The law, for its part, was however applied by the majority of the
Folk music typically contains a mix of German, Russian, Russian, Iranian, Egyptian, Norwegian, Turkish, Jewish and Tabad music, though typically the songs are from specific religion or back- ground. IcelandmardOedk for instance, distinguish genderly distinguishes folk music from the multi- diverse genres, primarily rock and jazz.. Folk music consequently not subegoaticly distinct from rock, which turn, which consists primarily of music of folk including jazz, blues, nor rock, and industrial rock. Due to the diversity of genres in these genres, certain new genres of music also exist. Pandit music, such
In 1974, Parker became the Director of the Office of Virginia's Chamber of Trade and Commerce. position he would hold under increased pressure from his fellow Democratic opponent. He oversaw the J. Howard administration, which lasted for years headed by William J. P honorter.
20

---

Table 12:
Samples from XSum summarization dataset. Parts of the articles are omitted for brevity.
Article: Two-year-old Lane Thomas Graves had been playing in the sand near the resort's Seven Seas Lagoon when he was dragged underwater by the creature.. The lighthouse has been installed near to where the attack occurred... A Disney spokesperson said they hoped the monument would spread awareness for the Lane Thomas Foundation, which also uses the lighthouse its logo. Who is liable for alligator boy's death? "The lighthouse sculpture has been installed to help spread awareness the Lane Thomas Foundation. which was established to provide assistance and support to families whose children need organ transplants, Walt Disney World said in a statement.
Reference: Walt Disney World has unveiled a lighthouse memorial for a young boy who was killed by an alligator while on holiday at the Florida theme park.
TEncDM
CosMOSN=126
Walt Disney Disney has unveiled a lighthouse statue in memory of young boy who died when he was stabbed by an alligator at Florida
A statue of Florida boy who died after being rescued from a beachqua- rina during water has been honoured by US television provider Disney.
resort in Florida.
Article:
The Sky Blues currently play in Coventry's 's Ricoh Arena hut had a long dispute with the stadium's 's previous owners... In statement, Rughy Borough Council said its leader and the council's 's executive director and head of planning had met with Coventry City March. "The cluh requested the meeting to understand how the council would deal with any planning application for potential stadium sites in the horough of Rugby." it said. It said the plans would need to be finalised by September to be included in the council's s local plan, but added that site had yet to be identified. Peter Ward, from Sky Blues Supporters' Consultative Group, said he was pleased to hear that things were "moving" with the club's 's search for new home. "It's good that finally there is some evidence things
Reference: Planners in Rugby have revealed they have been in talks with Coventry City Football Club about building stadium in the borough.
TEncDM
CoSMOSN=128
Coventry City Council says it is looking on whether potential plans for a 0m Super League stadium in Coventry.
Coventry City fans say they will meet with the city council over a proposed move from the club club new stadium.
Table 13: Samples from SQuAD2.0 question generation dataset.
Context: Temporal measurement has occupied scientists and technologists, and was a prime motivation navigation and astronomy. Periodic events and periodic motion have long served as standards for units of time. Examples include the apparent motion of the sun across the sky. the phases of the moon, the swing of pendulum, and the beat of a heart. Currently, the international unit of time, the second, is defined by measuring the electronic transition frequency of caesium atoms (see below). Time is also of significant social importance, having economic value (tiee is moncy) as well as personal value, due to an awareness of the limited time in each day and in human life spans.
Answer: Temporal measurement.
Question: What has been prime motivation in astronomy and navigation?
TEnCDM
COSMOSN-128
What has a significant role in sur-
What have technologists filled time in astronomy?
veying units of time?
Context: In Book 11 of his Confessions, St. Augustine of Hippo ruminates on the nature of time, asking, What then is time? If no one asks me, I know: if I wish to explain it to one that asketh, know not. He begins define time by what it not rather than what is, an approach similar to that taken in other negative definitions. However, Augustine ends up calling time a 201edistention201 of mind (Confessions 11.26) by which we simultaneously grasp the past in memory, the present by attention, and the future by expectation. Answer: St. Augustine of Hippo.
Question: Who commented the nature of time in Book 11 of his confessions?.
TEnDDM
COSMOSN=128
Who wrote nature of timefes- sions? ?
Which philosopher ruminates about explaining the nature of time?
21

---

Table 14: Random samples for the detoxification task the ParaDetox dataset.
Input: fucking imagine obama just put the hands up and keep good distance between he and cruz.
Reference:
Imagine Obama just put the hands and distanced himself with Cruz.
TEncDM
CoSMOSN=126
2018 obama just put the hands up and keep good distance between him and cruz.
imagine obama just put the hands up and keep good distance between he and cruz.
Input: you just summed up how fucking stupid politics is in one comment
Reference: You just summed up how polities is in one comment.
TEncDM
COSMOSN=128
you just sum up up how bad politics is in one comment
you just sum up up how how stupid politics one comment
Input: only churches give a crap it seems.
Reference: only churches care it seems.
TEncDM
COSMOSN=126
Only churches care it seems..
only churches give thing seems.
Input: second, maybe if you douches would read the damn edit summaries this wouldnt happen.
Reference: second, maybe if you would read the edit summaries this wouldnt happen.
TEncDM
CoSMOS
201d, maybe if you would read the editmaries.
you read edit summaries, this wouldn' 't happen.
Input:
what we witnessed here today was fucking miracle, and i want you to acknowledge that.
Reference: What we witnessed here today was miracle, and want you to acknowledge that.
TEncDM
COSMOSN =128
What witnessed here today miracle, and want you acknowl- edge that.
what witnessed here today a a miracle, and want to acknow]- edge that.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that COSMOS learns a compressed and smooth latent representation through carefully designed autoencoder objectives, enabling high-quality text generation with reduced inference time. Empirically, CoSMoS matches or surpasses traditional token- level diffusion and autoregressive baselines across diverse of generative tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The discussion of limitations is located in the Appendix E.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
We do not have theorems in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We thoroughly describe our method in Section 4.2.2 and provide a complete set of hyperparameters in Appendix C. We use publicly available datasets and will make our code publicly available upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes)
Justification: We use publicly available datasets. We add code to the supplementary materials and will make our code publicly available upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ httpssscccppbccccceceeceeecc //guidesss CooeeSubmi ssi aesoonooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details could be found in Appendix and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Following prior work, we do not report error bars for comparison results across all tasks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
These details could found Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https :ttops://neeee http///eeuripp.eee
Answer: [Yes]
Justification: Our research does not involve human participants and does not introduce any new datasets; all data used are publicly available.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss societal impact in Appendix F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not release any pretrained models datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide all the information about the datasets in Appendix A. We also cite the BERT paper whose representations we use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww/coo//aaooooo datastss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
We release only our source code which we thoroughly comment.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/coodemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our study does not rely on crowdsourced data and does not involve any research with human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our study does not involve crowdsourcing or any research involving human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are not utilized in any the scenarios described.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http::://neurppass http://s...ccooeeeeeeeeee/////// for what should or should not be described.
29


Reviews:
[{"review_id": "O4o8IQdocx", "reviewer": "Reviewer_AwHe", "summary": "The paper proposes COSMOS (COmpressed and SMOth latent Space), a latent diffusion model for text generation. COSMOS learns a compressed and smooth latent representation through carefully designed autoencoder objectives, enabling high-quality generation while reducing inference time.\n\nThe authors carefully demonstrate how their approach achieves desired latent-space properties for diffusion learning (Section 5). They also ablate and show how CE loss and MSE loss were important to learning latent space representations for COSMOS.", "questions": "Please see my comments in the Strength and Weakness section. \n\nWhat is the complexity of the compressor and the decompressor? Are they quadratic in the sequence length? \n\nMinor: Appendix were referred, but I could not find relevant appendix in the paper.", "limitations": "Please see my comments in the Strength and Weakness sections. I believe the paper will benefit from a honest discussion on its adaptability to real worlds tasks and the scaling limitations.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper adopts latent diffusion framework for textual data by learning contextualised text representations into the latent space, and trains a diffusion model that operates within this space. The proposed approach is very interesting, the authors have done a great job (specially in Section 5 and 6) to demonstrate how to effectively learn such models for text. However, I am afraid that the paper in its current form and with a limited set of experiments will not be very impactful. \n\nFor example, the paper will benefit from a more thorough related work and a detailed and honest discussion on its limitations, instead of directly claiming that the model is able to match or outperform autoregressive baselines across diverse tasks. \n\n> Our empirical results show that, COSMOS matches or outperforms traditional token-level diffusion and autoregressive baselines across diverse tasks. Our findings challenge the dominance of token-level models and highlight latent diffusion as a promising direction for building fast, high-quality language models.\n\n\nIn Table 4, it would be very important to report on significant differences among various models. Also it would be great to show how model predictions look both for Table 2 and Table 4. \n\nIt would be important to explain why only GPT2 and GPT Neo were chosen for autoregressive models. What are the SOTA performances on these tasks? \n\nI think the experiments were done with 100-200M model parameters. What are the limitations of scaling COSMOS to much larger models?", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and the valuable suggestions. We value your acknowledgement of our work and your interest in the latent‑diffusion training technique we propose for textual data. \n\nWe are sorry to hear you could not find the appendix. It is included with the full paper in the supplementary material (file: full_paper.pdf). We would be grateful if you could take a moment to review it, as we believe it addresses a number of the points you have raised. For your convenience, we have also duplicated answers to those questions immediately below.\n\n> **W1.** On related works. \n\nIn the related works section, we initially focused on latent diffusion for text, as these works form the direct foundation of our research. We then provide a broader discussion of text diffusion methods in Section 7, where we categorize them into four main groups: gaussian diffusion on text embeddings, simplex-based diffusion, masked diffusion, and latent diffusion. Following your suggestion, we will update the related works section to incorporate these alternative approaches for a more comprehensive overview.\n\n> **W2.** On limitations.\n\nWe discuss the limitations of our approach in Appendix D. We acknowledge that the two-stage training process is a significant limitation. Before the diffusion model can be trained, a text autoencoder must be prepared. This step, while enabling the diffusion to operate in a more meaningful latent space, introduces an additional training overhead not found in autoregressive models and complicates the overall workflow. Concurrently, we are not claiming our model outperforms large-scale autoregressive models. Nevertheless, our results demonstrate that it achieves performance competitive with autoregressive models of a similar size, specifically ~0.1B, on several generation tasks, albeit with the added training overhead.\n\n> **W3.** On significant differences among various models.\n\nThank you for this suggestion. To better contextualize our work, we will add a dedicated discussion in the appendix that explores the significant differences among various diffusion models for text, complementing the expanded related works section.\n\n> **W4.** On model predictions.\n\nPlease see Appendix F for sample outputs on each dataset we evaluated.\n\n> **W5.** On the choice of GPT-2.\n\nThe primary objective of Table 4 is to compare Cosmos with other diffusion-based approaches for text generation, such as Gaussian diffusion on text embeddings, simplex-based diffusion, and masked diffusion. All these methods select their own text representation, define a text corruption process, and train a model to refine the text using full bidirectional context. Our results validate our approach, showing that operating in a compressed latent space can substantially speed up sampling while maintaining or even improving quality, given a properly trained autoencoder. The autoregressive model was included as it represents the standard paradigm, making it an important benchmark for any alternative text generation approach. GPT-2 and GPT-Neo (an open-source alternative to GPT-3) were chosen as classic representatives of autoregressive modeling. \n\nFurthermore, the goal of this paper was not to develop a state-of-the-art model for specific generative tasks, as separate works focus on developing large, specialized models for such purposes [4, 5]. Instead, our primary contribution is a method for learning a compressed textual space specifically designed for diffusion modeling, which in turn allows for the efficient generation of high-quality text within the autoencoder's latent space.\n\n> **W6.** On Cosmos scaling.\n\nLatent diffusion for text is a relatively new paradigm, which makes the scaling process non-trivial. It requires additional research to determine how to best allocate resources: how much should one increase the autoencoder, the latent space size, or the diffusion model?\n\nIn this paper, we have partially answered some of these questions. For example, in Section 6.2, we showed that it is beneficial to increase the dimensionality of the latent space even without increasing the size of the autoencoder. Specifically, we investigated how the quality of generated text changes as the number of layers in the diffusion model's transformer is increased. According to preliminary results at a text length of 128, both quality and diversity of generated text improves significantly. This inspires confidence that the proposed approach can be scaled.\n\n| Model Size | $n_{layers}$ | MAUVE $\\uparrow$ | PPL $\\downarrow$ | Div $\\uparrow$  | Mem  $\\downarrow$ |\n|------------|----------|-------|------|-------|-------|\n| 0.12 B     | 12       | 0.849 | 97.6 | 0.492 | 0.135 |\n| 0.25 B     | 24       | 0.914 | 91.2 | 0.546 | **0.124** |\n| 0.5 B      | 48       | **0.923** |**89.7** | **0.554** | 0.125 |\n\nWe have also performed an experiment with an increased generation length of 512 tokens with our 0.12B model and compared it with the discrete diffusion model GIDD [3], which is of a similar size and was pretrained on the same dataset and length. This result shows that the latent diffusion paradigm for text generation is highly promising and poised to impact the field as a whole.\n\n| Model        | MAUVE$\\uparrow$ | PPL $\\downarrow$  | Div $\\uparrow$  | Mem $\\downarrow$  |\n|--------------|---------|--------|--------|--------|\n| Source text  | 0.968   | 23.2   | 0.465  | 0.036  |\n| GIDD         | 0.286   | 228.3  | 0.588  | 0.112  |\n| Cosmos       | 0.524   | 59.0   | 0.315  | 0.171  |\n\nWe hope our work provides a foundation for further research into latent language diffusion as a promising paradigm.\n\nThe final manuscript will incorporate these new experimental outcomes.\n\n> **Q1.** On the complexity of the compressor and the decompressor.\n\nThe compressor block is linear with respect to the sequence length $L$, with an attention complexity of O($NLd$). The decompressor block is quadratic with respect to the sequence length, with its attention complexity being O($L^2 d$). Here, \n$L$ denotes the sequence length, $N$ the number of latents, and $d$ the latent dimensionality.\n\nCrucially, we only use the computationally expensive decompressor once per inference, to map the final latent back to text. The primary efficiency gain stems from running the entire iterative diffusion process in the compressed latent space. Since attention complexity is quadratic, operating on this much shorter sequence of length N significantly accelerates generation compared to models working in the full sequence space.\n\n $\\newline$\n\nReferences:\n\n[1] Sahoo, Subham, et al. \"Simple and effective masked diffusion language models\" (2024).\n\n[2] Wang, Guanghan, et al. \"Remasking discrete diffusion models with inference-time scaling\" (2025).\n\n[3] von R{\\\"u}tte, Dimitri, et al. \"Generalized interpolating discrete diffusion\" (2025).\n\n[4] Liu, Yixin, et al. \"BRIO: Bringing order to abstractive summarization\" (2022).\n\n[5] Zhang, Tianyi, et al. \"Benchmarking large language models for news summarization\" (2025).\n\n---\n\nWe thank you for your thoughtful feedback and suggestions. We would be grateful if you would consider raising your score, in case we have addressed your concerns. Please let us know if any aspects still need clarification."}, {"review_id": "kAqP6mPaB1", "reviewer": "Reviewer_Kj8c", "summary": "This work explores latent diffusion for language generation. It focuses primarily on the design of the latent diffusion autoencoder. Two primary approaches exist from past work. The first approach involves compressing activations from a language model and then decoding them autoregressively (LD4LG and PLANNER). The second approach learns a diffusion model directly in the activation space of a pretrained language model and decodes them with a learnable non-autoregressive decoder. This work develops an autoencoder that both compresses the activations of a language model and decodes them non-autoregressively. It closely follows the architecture of the LD4LG autoencoder. They introduce a number of carefully ablated choices that improve the effectiveness of the latent space for diffusion modeling and decoding. They validate their approach across a number of tasks (unconditional generation, summarization, detoxification, and question generation) with a wide range of baselines.", "questions": "How is variable-length generation handled? This the main confusion about the paper that I would like to see addresses.\n\nCan you expand on the challenge of compressing the dimensionality? Adding a linear projection at the input/output layer should enable changing the latent dimensionality without significantly altering the diffusion model capacity. I think it's fine to limit the scope of the study, but do not think this is a legitimate challenge.\n\nIs “Latent-space augmentation” equivalent to dropout or is there some implementation difference?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nLatent diffusion is a relatively underexplored approach for language generation compared to autoregressive LMs and discrete diffusion LMs and work in this space is welcome.\n\nThe proposed design choices (e.g. dropout, noise augmentation, intermediate MSE regularization) in this work are well-motivated and the ablation studies are comprehensive. Prior work in the image domain has demonstrated the importance of the autoencoder design, so exploring it for language is natural. The analysis of the robustness of the latent space is interesting and insightful.\n\nThe baseline selection is comprehensive and covers a range of different approaches (autoregressive LMs, word embedding diffusion, discrete diffusion, latent diffusion). The authors validate their approach on a reasonable selection of tasks. Their approach consistently compares favorably to the set of baselines.\n\n\n**Weaknesses**\n\nThis work develops dataset-specific language diffusion models instead of trying to develop a general pretrained language model trained on web-text as is the current language modeling paradigm. I think that the authors’ setting is more computationally tractable and is reasonable given the early stage of the paradigm, but it is a limitation. It is an open question whether such approaches can scale to that setting.\n\nFor the unconditional language generation evaluation, the work does not quantify memorization of the training set. All of the metrics can simply be maximized by reproducing samples from the training set, so the memorization can be quantified in some way. Even just using a simple n-gram overlap metric such as was used by prior work (LD4LG). \n\nOne weakness of non-autoregressive approaches is that variable-length generation is not as natural as with autoregressive models, although I do not see discussion of this in the paper. My understanding is that the number of latents in the decoder must be specified. How is this done during inference?\n\nThe authors mention that they only focus on compressing the sequence length because reducing the dimensionality of the latents requires modifying the diffusion architecture. However, you can generally account for differences in input dimensions by simply introducing a linear projection at the input/output layer that represents a negligible change in capacity. This restriction limits the scope of their study.\n\nAs written, the “Latent-space augmentation” sounds like standard dropout. If it is the exact same operation, then just introducing it as dropout would be clearer.\n\nI would expect things such as the choice of text encoder to have a significant impact on downstream generation quality, but this choice is not explored in this work.", "was_revised": true, "final_justification": "The author response cleared up the confusion around how variable-length generation was handled. These details were omitted in the original submission. They also clarified my other questions about their “Latent-space augmentation” and committed to improving their regarding the dimensionality of the latent space.\n\nTheir promise to make the scope of this study and avenues for future work more explicit were also appreciated and will improve the quality of the final paper.", "rebuttal": "Thank you for your review and positive assessment of our work. We greatly appreciate that you recognized the motivation behind our design choices, the value of our ablation studies, and the insights provided by our analysis of latent space robustness.\n\n> **W1.** It is an open question whether such approaches can scale to that setting.\n\nWe thank the reviewer for highlighting this important point. As you rightly observe, latent diffusion for text is a relatively new and under-explored area. Before developing a large-scale model, it is essential to validate hypotheses regarding the training of the latent space and the diffusion model at a smaller scale. This is precisely what we accomplished in this work. We believe this foundational research is a prerequisite for successful large-scale development.\n\nTo that end, since submitting this work, we have taken the first steps toward a general pretrained language model on the OpenWebText dataset. Specifically, we investigated two questions: first, whether generation quality improves when increasing the diffusion model's size while keeping the autoencoder and latent space fixed, and second, how our 0.12B model performs at a length of 512 tokens compared to the GIDD discrete diffusion model pretrained under the same conditions.\n\nTo increase the diffusion model's size, we focused on increasing the number of layers. The results for 128-token sequences show that generation quality improves significantly in both quality and diversity.\n\n| Model Size | $n_{layers}$ | MAUVE $\\uparrow$ | PPL $\\downarrow$ | Div $\\uparrow$  | Mem  $\\downarrow$ |\n|------------|----------|-------|------|-------|-------|\n| 0.12 B     | 12       | 0.849 | 97.6 | 0.492 | 0.135 |\n| 0.25 B     | 24       | 0.914 | 91.2 | 0.546 | **0.124** |\n| 0.5 B      | 48       | **0.923** | **89.7** | **0.554** | 0.125 |\n\nFurthermore, the comparison with discrete diffusion at a generation length of 512 shows that the latent diffusion paradigm is highly promising for advancing the text generation field.\n\n| Model        | MAUVE $\\uparrow$ | PPL $\\downarrow$  | Div $\\uparrow$  | Mem $\\downarrow$  |\n|--------------|---------|--------|--------|--------|\n| Source text  | 0.968   | 23.2   | 0.465  | 0.036  |\n| GIDD         | 0.286   | 228.3  | 0.588  | 0.112  |\n| Cosmos       | 0.524   | 59.0   | 0.315  | 0.171  |\n\nThese results provide positive evidence that our approach can be successfully scaled to longer sequences and larger datasets. The final manuscript will incorporate these new experimental outcomes.\n\n> **W2.** On memorization of the training set.\n\nWe thank you for bringing this important point to our attention. We have calculated this metric for all methods involved in the comparison. The SEDD method shows the lowest value; however, its value is lower than the reference, which suggests that the generated texts may be less coherent. Cosmos achieved the values closest to the reference, which indicates that Gaussian latent diffusion is not prone to copying data from the dataset. Following Cosmos are the other two Gaussian latent diffusion methods, TencDM and LD4LG. We will add this metric to the final version of the paper.\n\n| Method             | Mem $\\downarrow$ |\n|--------------------|-------|\n| Source text        | 0.365 |\n| GPT2               | 0.455  |\n| GPT Neo            | 0.469  |\n| AR-Diffusion       | 0.540  |\n| DiffuSeq           | 0.516  |\n| SeqDiffuSeq        | 0.663  |\n| TESS               | 0.550  |\n| SEDD               | 0.325  |\n| LD4LG              | 0.432  |\n| TEncDM             | 0.438  |\n| Cosmos$_{N=16}$    | 0.394 |\n| Cosmos$_{N=128}$   | 0.383 |\n\n> **W3. Q1.** On variable length generation.\n\nWe acknowledge that non-autoregressive methods do not generate texts of arbitrary length as naturally as autoregressive models. The procedure in our work is consistent with other studies on text diffusion [1, 2], where the decoder reconstructs a sequence up to a maximum length with padding, and the output is then cut off after the first EOS token.  The number of latents is the same for all texts. \n\nIt is also worth noting that this challenge is a direct consequence of generating the text holistically in an iterative manner. This provides the key benefit of a full bidirectional context but also the drawback of less natural variable-length generation. A block-based approach could partially resolve this issue in future work.\n\n> **W4. Q2.** On changing the latent dimension.\n\nWe agree that adding input and output projection layers is a possible way to adapt the diffusion model to a changed latent dimensionality. However, this would not achieve our goal of accelerating inference. The computational complexity would remain the same because the internal operations would still process sequences of the original length and original dimension. This is exactly why we investigated sequence length compression instead, as this is a direct way to reduce attention complexity and speed up the generation process without changing the capacity of diffusion model.\n\n> **W5. Q3.** On latent-space augmentation.\n\nOur augmentation method differs from classic dropout [3] in a key respect. Standard dropout involves zeroing out a fraction of neurons and then scaling the remaining ones by 1/(1-p). Our technique omits this final scaling step. To prevent any potential confusion for the reader, we deliberately chose not to refer to it as dropout.\n\n> **W6.** The impact of text encoder on downstream generation quality.\n\nWe agree that the autoencoder is an important component. However, its specific architecture may be less critical than one might assume for general-purpose generation. This is substantiated by the work of TEncDM, who found that results across different downstream tasks did not vary significantly with the choice of autoencoder. While a highly-tuned autoencoder might be necessary to \"win\" a benchmark, a general one provides a robust and comparable baseline. \n\nWe believe a more dominant factor for downstream performance is the autoencoder's pre-training data, as shown by T5. Since our goal in this paper was not to achieve state-of-the-art results on downstream tasks, but rather to compare different diffusion paradigms for text generation, we did not select a text encoder for a specific task and instead focused on developing the compressor and decompressor for the diffusion model.\n\n $\\newline$\n\nReferences:\n\n[1] von R{\\\"u}tte, Dimitri, et al. \"Generalized interpolating discrete diffusion\" (2025).\n\n[2] Shabalin, Alexander, et al. \"Tencdm: Understanding the properties of the diffusion model in the space of language model encodings\" (2025).\n\n[3] Hinton, Geoffrey, et al. \"Improving neural networks by preventing co-adaptation of feature detectors\" (2012).\n\n[4] Raffel, Colin, et al. \"Exploring the limits of transfer learning with a unified text-to-text transformer\" (2020).\n\n---\n\nWe look forward to incorporating your feedback in our final version and thank you again for your time and insightful comments."}, {"review_id": "fnjjRTW26R", "reviewer": "Reviewer_ojHu", "summary": "This paper introduces COSMOS, an approach for text generation that leverages a compressed and smoothed latent space specifically tailored for diffusion models. Using a combination of a frozen language encoder, a lightweight compressor-decompressor architecture, and robustness-promoting training objectives, the method aims to achieve efficient, high-quality text generation with latent spaces significantly smaller than token-level representations. COSMOS is evaluated on a range of text generation tasks (such as story generation, summarization, question generation, and detoxification) and compared against both autoregressive and diffusion baselines. The experiments demonstrate that COSMOS achieves comparable or superior text quality while providing substantial speed-ups in inference.", "questions": "Q1. What is the fundamental difference between your method and LD4LG?  \nQ2. Public datasets and methodologies must include citations or sources.  \nQ3. In Section 4.2.2, there are two consecutive ending punctuation marks in the last line. Please fix this issue.  \nQ4. In Section 6.2, you state: \"It achieves an 8× reduction in latent sequence length while surpassing the quality of uncompressed representations.\" However, according to Figure 6, this phenomenon only holds for the comparison between N=128 and N=16, not universally. Recommend revising the wording to reflect this nuance.  \nQ5. Reproducible code and implementation details.  \nQ6. The evaluation metrics lack human evaluation results under few-shot settings.  \nQ7. Related work deserves a more detailed discussion.  \nQ8. Long-context experiments.  \nQ9. Direct experimental validation of exposure bias mitigation.", "limitations": "Refer to weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. Applying diffusion models to the field of text generation is an interesting and reasonable approach that can alleviate some of the issues caused by sequential decoding. \n2. The paper presents extensive ablation studies and evaluates the proposed method, COSMOS, on a variety of downstream tasks. The improvements over baselines provide compelling evidence for the method’s effectiveness.\n\n**Weakness**\n1. The paper lacks a clear and detailed comparison with related work, particularly with methods such as LD4LG. A more thorough discussion of conceptual and empirical differences would strengthen the contribution.\n2. All evaluated examples are relatively short in length. To better demonstrate the scalability and robustness of the proposed method, experiments on longer text sequences are recommended.\n3. Although the introduction highlights exposure bias as a major motivation, the experimental validation of this point is indirect. More explicit analysis would be beneficial.", "was_revised": true, "final_justification": "This paper introduces COSMOS, a diffusion model-based text generation method based on compressed and smoothed latent spaces. The core idea is to combine a frozen language encoder, lightweight compression–decompression modules, and robustness-oriented training objectives to significantly shorten the latent sequence length while maintaining or even improving generation quality, thereby enabling efficient inference across a variety of text generation tasks.\n\nIn the rebuttal, the authors addressed concerns about scalability and latent space compression by adding experiments on long-text generation (512 tokens) as well as corresponding compression (8x) ablations. They also committed to discussing the issues of variable-length generation and LLM-based evaluation in the Limitations section, and to fixing the typographical error in the final version. Overall, the authors’ responses resolved my main concerns; therefore, I assign a final rating of 4.", "rebuttal": "We sincerely appreciate your thoughtful feedback and are encouraged by your interest in diffusion models for language generation.\nWe are especially grateful for your positive recognition of our ablation studies and evaluation design.\n\n> **W1. Q7.** On related works. \n\nThank you for this valuable suggestion. While our related works section focuses primarily on latent diffusion methods for text—given their direct relevance to our approach—we agree that a more thorough comparison with a broader set of related methods would strengthen the manuscript.\n\nIn the revised version, we will expand our discussion to provide a clearer account of both conceptual and empirical differences between our method and prior work across the broader landscape of text diffusion models. This will include a more detailed analysis of methodological design choices, evaluation strategies, and task settings, in order to better contextualize our contributions within the existing literature.\n\n\n> **W2. Q8.** On longer contexts. \n\nAs also noted by Reviewer Kj8c, latent diffusion for text generation is a relatively new and under-explored area.Before developing a large-scale model, it is crucial to validate the underlying hypotheses for training the latent space and the diffusion model at a smaller scale. This work accomplishes precisely that, laying what we consider the essential groundwork for successful large-scale development.\nHaving established this foundation, we agree that evaluating our method on longer text sequences is a critical next step in assessing its scalability.\n\nTo this end, we have taken the first steps and made additional experiments on OpenWebText with extended generation lengths of up to 512 tokens with 512 latents using our 0.12B parameter model. We compared its performance against the discrete diffusion baseline GIDD [1], which has a comparable model size and was pretrained on the same dataset and sequence length. The results show that our latent diffusion approach maintains coherent and diverse text generation at this longer length, demonstrating its robustness beyond short examples.\n\nThese findings suggest that the method scales well to longer sequences, and we will include these results in the final version of the manuscript to provide a more comprehensive evaluation.\n\n\n| Model        | MAUVE $\\uparrow$ | PPL $\\downarrow$  | Div $\\uparrow$  | Mem $\\downarrow$  |\n|--------------|---------|--------|--------|--------|\n| Source text  | 0.968   | 23.2   | 0.465  | 0.036  |\n| GIDD         | 0.286   | 228.3  | 0.588  | 0.112  |\n| Cosmos       | 0.524   | 59.0   | 0.315  | 0.171  |\n\n> **W3. Q9.** On explicit exposure bias analysis. \n\nThank you for your insightful comment regarding exposure bias. Since our work investigates diffusion in a continuous latent space, the proposed framework inherently mitigates exposure bias. As the diffusion model updates the entire latent representation simultaneously, it can flexibly revise tokens that are out of place at any position.\n\nWhile extensive empirical evidence supports this behavior in the image domain, we acknowledge that corresponding evidence in the language domain is more limited. In response to your suggestion, we now provide several randomly selected predictions of our model on ROCStories at different time steps. These examples demonstrate the model’s capacity to revise previously predicted tokens. We appreciate your feedback and will include additional examples in the final version of the manuscript to further illustrate this behavior.\n\n| Timestep 0.65 | Timestep 0.05 |\n|---|---|\n| Yesterday I went **on** a walk in the woods. When I was on my walk I ran into a bear. I was **frightened.** I yelled. Thankfully the bear went away. | Yesterday I went **for** a walk in the woods. When I was on my walk I ran into a bear. I was **terrified.** I yelled. Thankfully the bear went away. |\n| Neil wanted to **fly** to California. He knew he had to save his money to go there. He bought **two** **tickets** from a **nice** **apartment.** The **week** **after** his **suitcase** was packed and he boarded his plane. Neil **arrived** in California and loved **California!** | Neil wanted to **go** to California. He knew he had to save his money to go there. He bought **a** **ticket** from a **sales** **man.** The **first** **day** his **bag** was packed and he boarded his plane. Neil **landed** in California and loved **it!** |\n| Jon was practicing playing his guitar. He decided to enter the talent show. **After** **shaking** **his** **nervous,** he went on stage and played **his** song **perfectly.** **The** **shocked** the judges ' s **c** **cheered.** **Jon** **Jon** won first place in the talent show! | Jon was practicing playing his guitar. He decided to enter the talent show. **Surprisingly,** he went on stage and played **the** song **beautifully.** **This** **cause** the judges ' s **shock** **and** **laughter.** **He** **had** won first place in the talent show! |\n| Anna wanted to help the community. She signed up to **be** volunteer at the soup kitchen. Every week, she met with many homeless. Every week, she was helping **homeless** people for food. Anna was very happy to help **such** **people** so **kindly!** | Anna wanted to help the community. She signed up to **go** volunteer at the soup kitchen. Every week, she met with many homeless. Every week, she was helping **hurt** people for food. Anna was very happy to help so **many** **in** **kind** **aid!** |\n\n> **Q1.** On the differences from LD4LG.\n\nWe appreciate your reference to the LD4LG paper, which we also found to be insightful and well-executed. While our work draws conceptual inspiration from LD4LG, there is a key methodological distinction. Specifically, LD4LG employs autoregressive language generation conditioned on latent representations obtained via diffusion. As a result, it inherits the limitations associated with autoregressive modeling, such as exposure bias and constrained token-level dependency.\n\nMoreover, LD4LG utilizes a fine-tuned BART-base model as the decoder for these latent representations. Given the strength of BART-base as a standalone model, it becomes difficult to isolate and quantify the specific contribution of the diffusion-generated latents to the overall generation quality. This design choice makes the impact of the latent diffusion component less transparent. In contrast, our primary contribution is a methodology for learning a textual space specifically designed for diffusion modeling, which in turn allows for the efficient generation of high-quality text within the autoencoder's latent space.\n\n> **Q2.** On the abscence of citations.\n\nThank you for bringing this issue to our attention. We have made every effort to properly cite all public datasets and methodologies used in our work, either in the main text or in the Appendix. However, it is possible that we may have inadvertently omitted a citation or failed to provide sufficient attribution in some instances.\n\nIf you could kindly point us to specific lines or sections where a citation appears to be missing, we would be grateful. We will be happy to correct any such oversights in the final version of the manuscript.\n\n> **Q3.** On the typographical error.\n\nThank you for noting this typographical error. We will correct it in the final version.\n\n> **Q4.** On the overly strong statement.\n\nThank you for the suggestion. We will revise and soften the statement accordingly in the final version.\n\n> **Q5.** On the supplementary materials.\n\nWe have included the working code in the supplementary material. Implementation details are provided in Appendix Section B.\n\n> **Q6.** On the lack of human evaluation.\n\nWe adopt an evaluation framework that is standard in the continuous language diffusion literature [2], [3]. The aggregated automatic metrics used for our selected tasks are well-established, and prior work has shown that they correlate reliably with human judgments. Additionally, conducting large-scale human evaluation for models with fewer than 0.3B parameters is resource-intensive and, unfortunately, beyond our current budget constraints.\n\n $\\newline$\n\nReferences:\n\n[1] von R{\"u}tte, Dimitri, et al. \"Generalized interpolating discrete diffusion\" (2025).\n\n[2] Lovelace, Justin et al., \"Latent Diffusion for Language Generation\" (2023).\n\n[3] Shabalin, Alexander, et al., \"TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings\" (2025).\n\n---\n\nThank you for your valuable feedback and suggestions. If our responses have addressed your concerns, we would kindly appreciate your consideration of revising your score. Please do not hesitate to let us know if any points require further clarification."}, {"review_id": "FU0Wq9FjCL", "reviewer": "Reviewer_QQwS", "summary": "The paper studies the use of a learned continuous and compressed latent spaces for text generation. A large part of the paper is focused on how to regularize the latent space to improve its “diffusability” and on evaluations to understand how each of these regularizations affect the learned latent space and the subsequent performance by the latent diffusion model. One of the main benefits of the proposed method comes from running a generative model in a lower dimensional space (typically 4x and 8x smaller), which leads to faster sampling.", "questions": "In the diffusion model evaluations, what is the decoded sequence length (that is, L)? How exactly can the method be used to generate longer outputs than the ones obtained by decoding the fixed length latent variables to a fixed output size. (Related to the next question.)\n\nI understand in this work the method is always trained on inputs of certain size L=512, and the latent space has L / c, where c is the compression factor >1. You could train this on sequences of varying lengths I suppose? By using not a fixed size latent space, but a latent space of size (L / c) x d, where L changes every batch.\n\nI was not totally clear on how you are introducing context (prompt) in the method, for instance, for short or long context tasks. \n\nWhy are other baselines, for instance MDLM, not considered? Is it due to the scale of the networks used? (This work focuses on architectures in the 100M-200M range).", "limitations": "The paper includes a limitation section. See weaknesses above for some questions regarding other potential limitations of the method.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The analysis and evaluations of the regularization techniques used are quite thorough and indicate that the regularizations used are quite relevant for the task at hand. I consider this to be quite nice, as learning the right latent space for these tasks (i.e. well suited for a diffusion model to operate in the latent space) is critical.\n- The model gets faster inference without sacrificing performance.\n- The fact that the method builds on continuous space diffusion models enables the use of tools developed in that space, such as distillation and other fast sampling techniques, which are not yet as developed for discrete diffusion models.\n\nWeaknesses:\n- Typical AR language models rely on low temperature sampling, which appears to be somewhat important for good performance. Moving to the continuous latent diffusion framework, it is unclear how this could be done. While there are techniques for low temperature sampling for diffusion models in continuous spaces, these are quite heuristic, and the final effect may be quite different to the one obtained when doing low temperature sampling for LLMs (the fact that the model operates on a latent space may exacerbate this).\n- I don’t find the empirical evaluation of the final diffusion models entirely convincing regarding the performance of the method at scale. Models are on the smaller side (100M-200M parameters), trained on small-to-medium size datasets (ROCStories, XSum, etc.), and on fairly small latent spaces (16 x dim). For longer sequences, the 16 should be increased? Or would this approach be used in an autoregressive manner generating blocks?\n- While I don’t consider this necessary, it could be very nice to have. One strength I mentioned above involves the use of distillation techniques that are only applicable in continuous space diffusion models. It would be great to see how these techniques interplay with the proposed method. This would be orthogonal to the paper’s contributions, but is aligned with the whole direction of increased efficiency. Maybe such techniques could give another boost in speed, which would be very nice.", "was_revised": true, "final_justification": "Please see discussion.", "rebuttal": "We sincerely thank the reviewer for the positive and encouraging feedback. We are pleased that you found our analysis of the regularization techniques thorough and relevant, and we appreciate your recognition of the efficiency gains and the advantages of building on continuous‑space diffusion models. Below, we address each point raised.\n\n> **W1.** On the analogue of temperature sampling in the continuous latent diffusion framework.\n\nTemperature sampling in autoregressive models allows for increasing or decreasing the confidence of token sampling, which in turn typically improves quality at the expense of diversity, and vice versa. In continuous Gaussian diffusions, there is indeed no direct concept of temperature sampling. However, we can propose a technique with an analogous effect. The work of TEncDM [1] shows that self-conditioning increases the network's confidence in its predictions. Based on this fact, we can control the prediction confidence using the following procedure to obtain the prediction $x_0$:\n\n$x_0^{t} = x_{\\theta}(x_t, t, \\emptyset) + w \\cdot \\bigg(x_{\\theta}(x_t, t, x_0^{t+1}) - x_{\\theta}(x_t, t, \\emptyset)\\bigg).$\n\nHere, the final prediction $x_0^t$ at step $t$ is an extrapolation from the unconditional prediction $x_{\\theta}(x_t, t, \\emptyset)$ in the direction of the self-conditioned prediction $x_{\\theta}(x_t, t, x_0^{t+1})$, controlled by a weight $w$.\n\nA larger $w$ increases model confidence, leading to higher-quality generations, while a smaller $w$ promotes diversity. Our key finding here is that this trade-off can be finely controlled; by tuning $w$, it is possible to achieve substantial improvements in quality with only a minor decrease in diversity. \n\n| $w$   | MAUVE $\\uparrow$ | PPL $\\downarrow$ | Div $\\uparrow$  |\n|-----|---------|--------|--------|\n| 0.9 | 0.781   | 33.5   | **0.326**  |\n| 1.0 | 0.836   | 30.2   | 0.322  |\n| 1.1 | 0.842   | 28.3   | 0.315  |\n| 1.2 | 0.845   | 27.4   | 0.315  |\n| 1.3 | **0.867**   | 27.3   | 0.314  |\n| 1.4 | 0.841   | 26.9   | 0.309  |\n| 1.5 | 0.843   | **26.3**   | 0.306  |\n| 1.6 | 0.842   | 27.3   | 0.306  |\n\nThis represents a very useful practical feature of our approach. We appreciate you raising this point and will add these results to the final version of the paper.\n\n> **W2.** On empirical evaluation.\n\nLatent diffusion for text generation is a relatively new and under-explored area, as also noted by Reviewer Kj8c. Prior to developing a large‑scale model, it is crucial to validate the underlying hypotheses for training the latent space and the diffusion model at a smaller scale. This work accomplishes precisely that, laying the groundwork we consider essential for successful large‑scale development.\n\nTo provide initial evidence of this scalability, we have taken the first steps on the OpenWebText dataset by investigating if quality improves when increasing the diffusion model's size, keeping other components fixed. By increasing the number of layers, we found that generation quality and diversity both improve significantly:\n\n| Model Size | $n_{layers}$ | MAUVE $\\uparrow$ | PPL $\\downarrow$ | Div $\\uparrow$  | Mem  $\\downarrow$ |\n|------------|----------|-------|------|-------|-------|\n| 0.12 B     | 12       | 0.849 | 97.6 | 0.492 | 0.135 |\n| 0.25 B     | 24       | 0.914 | 91.2 | 0.546 | **0.124** |\n| 0.5 B      | 48       | **0.923** | **89.7** | **0.554** | 0.125 |\n\nAdditionally, we trained an autoencoder using our strategy without compression and a diffusion model with a sequence length of 512 on OpenWebText, for comparison with the discrete diffusion method GIDD, which was trained under identical conditions. \n\n| Model        | MAUVE $\\uparrow$ | PPL $\\downarrow$  | Div $\\uparrow$  | Mem $\\downarrow$  |\n|--------------|---------|--------|--------|--------|\n| Source text  | 0.968   | 23.2   | 0.465  | 0.036  |\n| GIDD         | 0.286   | 228.3  | 0.588  | 0.112  |\n| Cosmos       | 0.524   | 59.0   | 0.315  | 0.171  |\n\nThese results are a strong positive indicator for scaling. \n\nRegarding generation length, we used a fixed number of latents for all texts in this work. Adapting the model for longer texts requires separate research as, you correctly noted, there are several viable strategies to explore\n\n> **W3.** On distillation of continuous diffusion model.\n\nWe agree that exploring the distillation of latent text diffusion models is a fascinating direction for future work. However, we are cautious about the direct application of techniques from the image domain, as we are not aware of any work that has successfully distilled a continuous Gaussian diffusion model into a one-step generator for text. As you rightly point out, such an acceleration technique is orthogonal to our core contribution of learning a compressed latent space for diffusion. We believe our model will become a good framework for beginning research in this direction.\n\n> **Q1.** On sequence length.\n\nAs we wrote in Section 7 in the paragraph \"Autoencoder setup across tasks,\" the decoder sequence length (L) is 128 or 512, depending on the experiment. To generate sequences of arbitrary length, we follow the same procedure as in other works on text diffusion [1, 2], where the decoder reconstructs a sequence up to a maximum length with padding, and the output is then cut off after the first EOS token. Consequently, the decoder in our framework does not produce sequences that exceed the predefined maximum length.\n\n> **Q2.** On using not a fixed size latent space.\n\nWe thank you for this insightful suggestion. To keep the core methodology focused, we employ a fixed latent space size in this work. We agree that using a variable number of latents for texts of varying lengths is a compelling idea for future exploration, though it requires additional research beyond the scope of this paper.\n\n> **Q3.** On introducing context in the method.\n\nTo encode the context before feeding it into the diffusion model, we first pass it through our autoencoder, which generates a compressed representation. It is this representation that is subsequently supplied to the diffusion model via cross-attention. Depending on the task and context length, we use a model configured for either 512 or 128 tokens for this process.\n\n> **Q4.** On baselines.\n\nOur comparative analysis was designed to benchmark our method against key paradigms in text diffusion: Gaussian diffusion on text embeddings, simplex-based diffusion, masked diffusion, and latent diffusion. From the many masked diffusion models available, we selected SEDD as a prominent and established baseline. For a more direct and controlled comparison, we also benchmark against GIDD (as detailed in our response to W2), as it was trained under the same conditions and text length as our model. We considered MDLM but opted against it because its training on single-line concatenated texts produces fragmented, non-comparable samples. To avoid re-training a method specifically for our setup, we decided to use GIDD, which was trained in the exact same setup as our model.\n\n $\\newline$\n\nReferences:\n\n[1] Shabalin, Alexander, et al. \"Tencdm: Understanding the properties of the diffusion model in the space of language model encodings\" (2025).\n\n[2] von R{\\\"u}tte, Dimitri, et al. \"Generalized interpolating discrete diffusion\" (2025).\n\n---\n\nWe sincerely appreciate your constructive feedback, which will greatly assist us in improving the final manuscript. Please let us know if there are any remaining points that require clarification."}]

Author rebuttal:

Meta-review: This paper presents COSMOS (COmpressed and SMOoth latent Space), a latent diffusion model for text generation that learns compressed and smooth latent representations through carefully designed autoencoder objectives. The key scientific claims include achieving inference speedups through operating in compressed latent spaces (4x-8x smaller) while maintaining or improving generation quality, and demonstrating that properly regularized latent spaces enable effective diffusion modeling for text. The paper's strengths include comprehensive ablation studies validating design choices for latent space regularization, thorough baseline comparisons across multiple text diffusion paradigms (autoregressive, embedding diffusion, discrete diffusion, and latent diffusion), and demonstrated efficiency gains without performance degradation. The work also introduces novel techniques like self-conditioning for quality-diversity trade-offs and provides detailed analysis of latent space robustness. However, initial weaknesses included limited experimental scale (100-200M parameters), evaluation primarily on short sequences (128 tokens), lack of comparison with some recent baselines, missing memorization metrics, and insufficient discussion of variable-length generation challenges. 

During the rebuttal period, authors significantly strengthened their submission with substantial new experiments and commitments. During the borderline discussion, Reviewer Kj8c raised their score after authors clarified variable-length generation implementation and committed to expanding related work sections. Reviewer QQwS maintained their borderline accept score, appreciating the additional scaling experiments while acknowledging that true scalability remains an open question. Reviewer ojHu changed their score from borderline reject to borderline accept (score 4) after authors provided new experiments on longer generation lengths (512 tokens), higher compression ratios (8x), memorization metrics, and commitments to expand limitations discussion. The authors conducted scaling experiments up to 0.5B parameters showing improved quality and diversity, demonstrated superior performance over discrete diffusion baselines (GIDD) on longer sequences, and introduced a novel self-conditioning technique analogous to temperature sampling. Reviewer AwHe did not participate in the discussion or rebuttal despite multiple author follow-ups, leading me to weigh their rating less heavily. The consensus among engaged reviewers, supported by the substantial improvements during rebuttal, justifies acceptance of this foundational work in latent text diffusion.

The primary reasons for acceptance are the solid technical contributions to an underexplored area, comprehensive experimental validation, and the foundational value for future research in latent text diffusion.
Decision: Accept (poster)