Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes
Authors: Bocheng Li, Zhujin Gao, Linli Xu
Accepted at: acl25

Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes

Bocheng Li*1,2, Zhujin Gao*1,2, Linli Xu†1,2
1School of Computer Science and Technology, University of Science and Technology of China
2State Key Laboratory of Cognitive Intelligence
{bccli,gaozhujin}@mail.ustc.edu.cn, linlixu@ustc.edu.cn

Abstract
Diffusion models have emerged as a promising approach for text generation, with recent works falling into two main categories: discrete and continuous diffusion models. Discrete diffusion models apply token corruption independently using categorical distributions, allowing for different diffusion progress across tokens but lacking fine-grained control. Continuous diffusion models map tokens to continuous spaces and apply fine-grained noise, but the diffusion progress is uniform across tokens, limiting their ability to capture semantic nuances. To address these limitations, we propose Non-simultaneous Continuous Diffusion Models (NeoDiff), a novel diffusion model that integrates the strengths of both discrete and continuous approaches. NeoDiff introduces a Poisson diffusion process for the forward process, enabling a flexible and fine-grained noising paradigm, and employs a time predictor for the reverse process to adaptively modulate the denoising progress based on token semantics. Furthermore, NeoDiff utilizes an optimized schedule for inference to ensure more precise noise control and improved performance. Our approach unifies the theories of discrete and continuous diffusion models, offering a more principled and effective framework for text generation. Experimental results on several text generation tasks demonstrate NeoDiff’s superior performance compared to baselines of non-autoregressive continuous and discrete diffusion models, iterative-based methods and autoregressive diffusion-based methods. These results highlight NeoDiff’s potential as a powerful tool for generating high-quality text and advancing the field of diffusion-based text generation.

1 Introduction
Diffusion models have demonstrated remarkable success in generating high-quality samples in various domains, including vision (Dhariwal and Nichol, 2021; Nichol and Dhariwal, 2021; Ho and Salimans, 2021; Rombach et al., 2022) and audio (Chen et al., 2020; Kong et al., 2020). Inspired by their achievements, there has been a growing interest in applying diffusion models to text generation tasks (Li et al., 2022; Gong et al., 2022; Gao et al., 2024; Zheng et al., 2023).

The core idea behind diffusion models is to corrupt the data through a forward process and then learn to reverse this process to generate new samples. In text generation, existing diffusion models can be broadly categorized into two classes: discrete and continuous diffusion models. Discrete diffusion models treat tokens as discrete random variables and perform state transitions independently for each token using a categorical distribution. While straightforward, this approach fails to capture the continuous and fine-grained nature of language, limiting the potential benefits of multi-step generation. Continuous diffusion models, on the other hand, operate in a continuous space by mapping tokens to continuous representations, enabling more fine-grained perturbations. However, these models typically apply diffusion at the sentence level, resulting in uniform noise levels across all tokens within a sentence, restricting the model’s ability to leverage contextual information and recover tokens with varying noise levels based on the surrounding context (Chen et al., 2023; Wu et al., 2024).

To address these limitations, we propose integrating the complementary strengths of discrete and continuous diffusion approaches, enabling fine-grained noise control at the token level. This unified approach aims to provide precise token-level control while maintaining continuous-valued noise distributions, which is absent in existing frameworks. While recent text diffusion models (Han et al., 2023; Gong et al., 2023; Wu et al., 2024) have made advances, they do not fully address
Figure 1: Comparison of the noising paradigms employed by Non-simultaneous Continuous Diffusion and two other diffusion models. The color intensity on the text tokens represents the token-level noising progress (intrinsic time \( \tau \)). Discrete diffusion applies an independent but coarse-grained noising paradigm to each token within a sentence. In contrast, continuous diffusion utilizes a fine-grained noising schedule but applies it uniformly across all tokens. NeoDiff distinguishes itself by assigning an independent, fine-grained intrinsic time \( \tau \) to each token, with finer noising schedule in extrinsic time \( t \).

this requirement, necessitating a unified theoretical framework that bridges discrete and continuous diffusion paradigms through a carefully designed forward process.

Furthermore, we observe that existing approaches primarily focus on enhancing the forward process, overlooking the inherent varying difficulties in denoising different tokens and the impact of generation context. In analyzing the reverse process, we recognize that tokens with lower noise levels can guide the recovery of more heavily corrupted tokens, thereby enhancing the overall text generation quality.

In response to these challenges, we present Non-simultaneous Continuous Diffusion Models (NeoDiff), which unifies discrete and continuous diffusion models through a bi-temporal framework. The key insight is to generalize the time variable in previous diffusion models into extrinsic time \( t \), representing the diffusion progress of the entire sentence, and intrinsic time \( \tau \), tracking the diffusion progress of each individual token. This generalization enables us to introduce a novel Poisson process as the forward process, seamlessly integrating the flexibility of discrete noise with the fine granularity of continuous noise. An overview of this noising paradigm is illustrated in Figure 1.

To optimize the reverse process, we develop a context-aware time predictor that estimates the intrinsic time \( \tau \) using an adaptive modulation function to guide the denoising process. The extrinsic time schedule is further calibrated through Bayesian optimization, providing precise control over the noise distribution.

NeoDiff achieves a fine-grained, improved diffusion process in both forward and reverse directions, naturally overcoming the constraints of previous discrete and continuous diffusion models, and exhibiting superior generation quality. We evaluate NeoDiff on a diverse set of NLP tasks, including machine translation, paraphrasing, text simplification, and question generation. NeoDiff consistently outperforms previous non-autoregressive diffusion-based and iteration-based methods, as well as autoregressive diffusion baselines. Specifically, our contributions can be summarized as follows:

• We introduce NeoDiff, a unified theoretical framework that combines the advantages of discrete and continuous noise, generalizing and unifying existing text diffusion models.

• We propose the Poisson diffusion process as the forward process, enabling fine-grained corruption of text data, a context-aware time predictor that adaptively modulates the reverse process based on semantic context, and an optimized extrinsic time schedule for precise noising control.

• We conduct extensive experiments to evaluate the effectiveness of NeoDiff and compare it to existing text diffusion models. Our results highlight the advantages of our unified framework and suggest its potential to advance diffusion-based text generation.

2 Background

2.1 Diffusion Models

Diffusion models assume a gradual noise injection process over time for data samples \( z_0 \in \mathbb{R}^{N \times d} \). The forward diffusion process forms a series of latent variables \( z_1, z_2, \cdots, z_T \) satisfying the Markov
property, and finally become pure Gaussian noise \( z_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) \):

\[
q(z_t | z_{t-1}) = \mathcal{N}\left( z_t; \sqrt{\alpha_t} z_{t-1}, \beta_t \mathbf{I} \right),
\]

where \( \alpha_t + \beta_t = 1 \), determining the degree of noising at time \( t \) and constituting the noise schedule. The reverse process is parameterized as

\[
p_\theta(z_{t-1}|z_t) = \mathcal{N}\left( z_{t-1}; \mu_\theta(z_t, t), \Sigma_\theta(z_t, t) \right),
\]

Here, \( \mu_\theta(\cdot) \) and \( \Sigma_\theta(\cdot) \) are the model’s estimates of the distribution mean and covariance matrix, respectively. The training objective is derived from the variational lower bound of the negative log-likelihood loss, and can be then simplified as an MSE loss (Ho et al., 2020; Li et al., 2022; Gao et al., 2024):

\[
\mathcal{L}_{VLB} = \mathbb{E}\left[ \| z_\theta(z_t, t) - z_0 \|^2 - \log p(z_0 \mid z_1) \right]
\]

2.2 Discrete Diffusion Models

Discrete diffusion models directly model the noise on categorical distributions, discarding the assumption that the noise in latent variables follows a normal distribution in continuous space. These models typically represent data as sequences of one-hot vectors and employ a transition matrix to add noise to the data. Among them, Hoogeboom et al. (2021a) proposed a multinomial diffusion model that employs a uniform noising method. Austin et al. (2021) introduced D3PM, which employs a noising method with an absorbing state. Specifically, they added an absorbing state [MASK] to the vocabulary, which can only be entered but not exited. The remaining states, at each diffusion step, either stay in the current state or enter the absorbing state with a certain probability. Recently, Lou et al. (2024) made progress by developing score entropy, which extends score matching to discrete spaces and demonstrates substantial performance improvements. While effectively adapting diffusion models to discrete data, these methods have limitations. The discrete nature of the noise limits its expressiveness, making it difficult to capture the nuances of continuous transitions between states. This restricts the model’s ability to represent gradual semantic changes or finely adjust individual token features, potentially limiting the benefits of multi-step generation.

2.3 Continuous Diffusion Models

Continuous diffusion models map discrete tokens to a continuous vector space using a mapping function, allowing the application of standard continuous diffusion processes. Analog Bits (Chen et al., 2022) uses a binary encoding scheme (int2bit : \( \mathbb{Z} \to 0, 1^{[\log_2 V]} \)) to represent token indices as binary sequences. After the reverse diffusion process, a quantization operation followed by binary decoding (bit2int : \( 0, 1^{[\log_2 V]} \to \mathbb{Z} \)) recovers the token indices. Han et al. (2023) proposed a mapping function logits-generation : \( \mathbb{Z} \to \mathbb{R}^V \), which transforms token indices into a probability simplex. Li et al. (2022) proposed Diffusion-LM, where the token sequence y is first mapped to a random representation \( z_0 \) using a word embedding as the mean. After the reverse diffusion process, the generated vectors are rounded back to discrete tokens. Gong et al. (2022) extended this approach to sequence-to-sequence generation with DiffuSeq, which concatenates the source and target sentences and utilizes an attention mechanism to leverage source information during generation. However, a key limitation of continuous diffusion models is the uniform noise injection applied to all tokens during the forward process. This uniform noise injection hinders the model’s ability to effectively leverage contextual information. Ideally, varying noise levels across tokens would allow the model to utilize less noisy tokens as context for restoring more corrupted ones, facilitating better contextual modeling.

2.4 Improvements over Previous Diffusion Models

Recent studies have explored various methods to address the limitations discussed above. Han et al. (2023) introduced a semi-autoregressive generation strategy that generates fixed-length blocks autoregressively while employing non-autoregressive iterative denoising within each block. Wu et al. (2024) proposed a hierarchical noise addition method, where noise levels increase monotonically from left to right within a sentence, enabling autoregressive generation. Gong et al. (2023) presented a hybrid approach that combines standard continuous noise with the probabilistic replacement of tokens with [MASK], integrating discrete and continuous noise. Although these studies have contributed to enhancing the forward diffusion process, their improvements did not fully achieve fine-grained noise at the token level, thus not completely addressing
the limitations of both continuous and discrete diffusion models. Also, these approaches typically employ a fixed reverse process that mirrors the forward diffusion process, without considering the varying difficulties in denoising different tokens and the impact of the actual generation context.

3 Non-simultaneous Continuous Diffusion Models

To address these limitations, we propose a unified diffusion framework called Non-simultaneous Continuous Diffusion Models (NeoDiff). Figure 2 presents an overview of NeoDiff, illustrating its architecture and key components. NeoDiff employs an Encoder-Decoder Transformer architecture (Vaswani et al., 2017), with the decoder serving as the primary component for denoising, and the encoder provides the embedding of the condition sentence \( \mathbf{x} \) to a transformer-decoder-based time predictor. In the following sections, we will provide a detailed formulation of NeoDiff and demonstrate how it addresses the limitations of previous approaches.

3.1 Unified Formulation and Training Objective

We present a unified framework for diffusion models by introducing two time dimensions: extrinsic time \( t \) and intrinsic time \( \tau \). The extrinsic time \( t \) represents the global diffusion progress of the entire sentence, while the intrinsic time \( \tau \) captures the diffusion progress of individual tokens.

This formulation generalizes existing approaches. We can easily derive discrete diffusion models by modeling \( \tau \) as a monotonically increasing random function of \( t \), with \( \tau_t \in \{0, 1\} \), where \( \tau_t = 0 \) and \( \tau_t = 1 \) signify original and fully corrupted tokens, respectively. And continuous diffusion can be obtained by setting \( \tau \) as a deterministic function that typically equals \( t \) (\( \tau_t = t \)). Furthermore, recent hybrid diffusion models, such as DiffuSeq-V2 (Gong et al., 2023), can also be formalized under this framework by setting \( \tau_t = \max(t + \tau_{\text{mask}}(t), 1) \), where \( \tau_{\text{mask}}(t) \sim \text{Bernoulli}(\gamma, \beta(t)) \) and \( \gamma \) is the ratio of tokens replaced by [MASK] when \( t = 1 \).

NeoDiff defines \( \tau_t \in [0, 1] \) as a continuous random function of extrinsic time \( t \in [0, 1] \), enabling fine-grained control over the diffusion process. We impose boundary conditions \( \tau_0 = 0 \) and \( \tau_1 = 1 \) to guarantee token preservation at initialization and complete corruption at termination of the diffusion process.

Let \( \mathbf{z} \in \mathbb{R}^d \) denote a token embedding and \( \mathbf{z}_t \) its latent representation at time \( t \), with initial and final conditions \( \mathbf{z}_0 = \mathbf{z} \) and \( \mathbf{z}_1 \sim \mathcal{N}(0, \mathbf{I}) \). The forward process defines the joint distribution as:

\[
q(\mathbf{z}_{>0}, \tau_{>0}|\mathbf{z}_0) := \prod_{t>0} q(\mathbf{z}_t, \tau_t|\mathbf{z}_0)
= \prod_{t>0} q(\mathbf{z}_t|\mathbf{z}_0, \tau_t)q(\tau_t),
\]

where

\[
q(\mathbf{z}_t|\mathbf{z}_0, \tau_t) := \mathcal{N}\left(\mathbf{z}_t; \sqrt{\bar{\alpha}(\tau_t)}\mathbf{z}_0, \bar{\beta}(\tau_t)\mathbf{I}\right),
\]

and \( \bar{\alpha}(\cdot) \) and \( \bar{\beta}(\cdot) \) denote noise schedules with their domains scaled to \([0, 1]\).

Given \( t' = t - \Delta t \), the reverse process is defined as

\[
p_\theta(\mathbf{z}_{0:1}, \tau_{0:1}) := p_\theta(\mathbf{z}_1, \tau_1) \prod_{t'<1} p_\theta(\mathbf{z}_{t'}, \tau_{t'}|\mathbf{z}_t, \tau_t)
= p_\theta(\mathbf{z}_1, \tau_1) \prod_{t'<1} p_\theta(\mathbf{z}_{t'}|\mathbf{z}_t, \tau_t, \tau_{t'})p_\theta(\tau_{t'}|\mathbf{z}_t, \tau_t).
\]

We further parameterize the distribution of \( \mathbf{z}_{t'} \) as

\[
p_\theta(\mathbf{z}_{t'}|\mathbf{z}_t, \tau_t, \tau_{t'}) = q(\mathbf{z}_{t'}|\hat{\mathbf{z}}_0(\mathbf{z}_t, \tau_t, t), \tau_{t'}),
\]

where \( \hat{\mathbf{z}}_0 \) is the model prediction of \( \mathbf{z}_0 \).

Following Ho et al. (2020) and Li et al. (2022), we derive NeoDiff’s training objective from the variational lower-bound \( \mathcal{L}_{\text{VLB}} \), and with the simplified \( \mathcal{L}_z \) and an anchor loss \( \mathcal{L}_{\text{anchor}} \) (Gao et al., 2024) as a regularization term to avoid collapse of the embedding space, the training objective of Neodiff can be written as

\[
\begin{align*}
\mathcal{L} &= \mathcal{L}_z + \mathcal{L}_\tau + \mathcal{L}_{\text{anchor}} \\
&= \mathbb{E}_q \Big[ \| \hat{\mathbf{z}}_0(\mathbf{z}_t, \tau_t, t) - \mathbf{z}_0 \|^2 \Big] \\
&\quad + \sum_{0<t'<1} \underbrace{\mathbb{E}_{\tau_{t'}} \left[ \text{KL}(q(\tau_{t'})||p_\theta(\tau_{t'}|\mathbf{z}_t, \tau_t)) \right]}_{\mathcal{L}_\tau} \\
&\quad + \underbrace{- \log p_\theta(y|\hat{\mathbf{z}}_0(\mathbf{z}_t, \tau_t, t))}_{\mathcal{L}_{\text{anchor}}} .
\end{align*}
\]

A detailed derivation can be found in Appendix A.
3.2 Fine-Grained Forward Process Using Poisson Diffusion

After establishing the unified formulation, we define a fine-grained forward diffusion process through intrinsic time \( \tau \). To quantify the diffusion progression within a single token, we introduce a discrete state function \( s_t \in \{0, 1, 2, \cdots, s_{\max}\} \), where uniformly divided states represent distinct levels of the diffusion process from \( s_t = 0 \) (noiseless) to \( s_t = s_{\max} \) (maximum noise). For an infinitesimal time interval \( \Delta t \), the transition dynamics follow a Poisson process characterized by:

\[
\mathbb{P}[s_t = s_{t'} + 1] = \gamma(t) \Delta t + o(\Delta t)
\]
\[
\mathbb{P}[s_t = s_{t'}] = 1 - \gamma(t) \Delta t + o(\Delta t),
\]

where \( \gamma(\cdot) \) is a hyperparameter function termed the transition schedule. This formulation yields a tractable distribution for \( s_t \):

\[
s_t \sim \text{Poisson} \left( \int_0^t \gamma(t') \, dt' \right) = \text{Poisson} \left( \lambda(t) \right).
\]

To ensure compatibility with the continuous-time framework of NeoDiff, we normalize the state function to \([0, 1]\) through normalization and clipping:

\[
\tau_t = \text{Clip} \left( \frac{s_t}{s_{\max}}, 1 \right) = \text{Clip} \left( s'_t, 1 \right),
\]

where \( \text{Clip}(\cdot, \cdot) \) denotes the truncation operation to maintain bounded noise levels. We choose \( s_{\max} \) sufficiently large to achieve fine-grained transitions between noise states, and set \( \lambda(t) = k s_{\max} t \) to maintain \( \mathbb{E}[s_t] = \lambda(t) \propto s_{\max} \). This design ensures that \( \tau_t \) remains independent of \( s_{\max} \) and reduces the process to a homogeneous Poisson process with constant transition schedule \( \gamma(t) \).

However, a critical limitation of this basic formulation emerges when examining the coefficient of variation (CV) of the normalized state function \( s'(t) \):

\[
\text{CV} = \frac{\sqrt{\mathbb{V}[s'_t]}}{\mathbb{E}[s'_t]} = \frac{1}{\sqrt{\lambda(t)}} \propto \frac{1}{\sqrt{s_{\max}}},
\]

which indicates that as \( s_{\max} \) increases, the relative variation between token states diminishes proportionally to \( \frac{1}{\sqrt{s_{\max}}} \). Consequently, when \( s_{\max} \) becomes sufficiently large, the discreteness of the process is lost as all tokens effectively share nearly identical \( \tau \) values, causing NeoDiff to degenerate into a continuous diffusion model.

To address this limitation, we further introduce a variance-controlled rescaling transformation:

\[
\tau_t = \frac{\text{Clip} \left( \text{Round} \left( \frac{s_t - \lambda(t)}{\sqrt{\lambda(t)}} \sigma(t) + \lambda(t) \right), s_{\max} \right)}{s_{\max}}
\]

Under this transformation, the variables within \( \text{Clip}(\cdot, \cdot) \) follow a distribution centered at \( \lambda(t) \) with variance \( \sigma(t) \). To ensure that the discrete characteristics of our process remain invariant to the choice of \( s_{\max} \), we set \( \sigma(t) = \lambda(t) \). Since the choice of \( \lambda(t) \) and \( \sigma(t) \) may result in \( \tau_1 \neq 1 \), we truncate \( \tau_t \) to 1 for \( t > t_{\max} \), where \( t_{\max} \) is a predefined threshold.

3.3 Context-aware Reverse Process with Time Predictor

We propose a context-aware reverse process that explicitly models the conditional distribution \( p_\theta(\tau_{t'}|z_{t'}, \tau_t) \), in contrast to previous approaches that simply mirror the forward process by assuming \( p_\theta(\tau_{t'}|z_{t'}, \tau_t) = q(\tau_{t'}) \). This explicit modeling enables adaptive denoising based on both semantic context and noise states.

![An overview of NeoDiff.](page_184_153_1079_312.png)

Figure 2: An overview of NeoDiff.
Time Predictor Design. When modeling a known distribution, researchers typically employ reparameterization tricks to model its parameters. However, in our case, the Poisson distribution’s sole parameter \( \lambda(t) \) is a deterministic function of \( t \) that measures the overall noise progress of the sample and is equivalent to \( t \). To obtain the noise progression \( \tau \) for each basic token, we directly treat both \( p_\theta(\tau_{t'}|z_t, \tau_t) \) and \( q(\tau_{t'}) \) as standard discrete distributions and learn them using cross-entropy loss, without using reparameterization tricks.

Model Input Design. While \( z_t \) could serve as an input to \( \tau_\theta \), this choice would enable the model to predict noise levels through direct comparison with all other embedding vectors. Such an approach would result in a reverse process that merely retraces the forward process, providing little value for generation quality control. To address this limitation, we propose using the generated sample \( z_\theta \) as input to \( \tau_\theta \). This design choice increases the modeling complexity of the prediction task while enabling \( \tau_\theta \) to serve dual purposes: noise level prediction and semantic quality assessment of the generated output. To provide temporal context, we incorporate \( t' = t - \Delta t \) as an additional input, ensuring the model’s awareness of the target time distribution. The complete formulation of \( p_\theta(\tau_{t'}|z_t, \tau_t) \) is expressed as \( \tau_\theta(z_\theta(z_t, \tau_t, t'), x) \), where \( x \) represents the conditioning sentence embedding.

Pseudo Label for Training the Time Predictor. The naive approach of using \( \tau_{t'} \) as the direct training label for the time predictor can introduce systematic bias in the learning process. While \( \tau_{t'} \) is derived from \( z_\theta \), this predicted quality measure may not accurately reflect the actual generation quality after the complete denoising process. For example, tokens initially assigned high noise levels might still produce high-quality outputs after denoising, making their initial \( \tau_{t'} \) assignment suboptimal. Instead, we propose a pseudo-labeling strategy for training the time predictor. More specifically, we first compute a confidence score for each generated output using the combined loss \( \mathcal{L}_z + \mathcal{L}_{\text{anchor}} \) from the denoised prediction \( z_\theta \). To ensure these confidence scores follow a distribution compatible with \( \tau_{t'} \), we apply inverse transform sampling. To accomplish this, we compute the normalized rank \( r \) for each token’s loss within the single sample and map these ranks through the inverse cumulative distribution function (ICDF) of the Poisson distribution: \( \tilde{s}(t) = F^{-1}(r; \lambda(t)) \), where \( F \) denotes the Poisson cumulative distribution function. The resulting \( \tilde{s}(t) \) values are then transformed via Eq. (7) to obtain the final pseudo labels.

3.4 Optimized Extrinsic Time Schedule
The choice of time schedule in diffusion models significantly impacts both generation quality and computational efficiency. While previous works such as Dhariwal and Nichol (2021); Chen (2023) focus on optimizing the noise schedule function with fixed extrinsic time steps, we propose to perform direct optimization on the schedule of extrinsic time \( t \). Our method builds upon Li et al. (2024), who introduced post-training Bayesian optimization to select optimal subsets of time steps for inference acceleration. However, where they treat time steps as discrete variables and optimize for subset selection, we formulate the problem as continuous optimization over the complete time schedule \( \{t_1, t_2, \ldots, t_K\} \), where \( K \) denotes the total number of diffusion steps. This continuous formulation enables more precise calibration through Bayesian optimization, effectively exploring the full space of possible time schedules. We evaluate candidate schedules using a trained model on the validation set via Bayesian optimization, optimizing for the BLEU score as our objective metric. This approach yields task-specific optimal time schedules that further enhances generation quality. The detailed optimization procedure is presented in Appendix B.4.

4 Experiments

4.1 Experimental Setup

Datasets and Metrics We evaluate our approach on several NLP tasks, including machine translation (WMT14 En-De (Bojar et al., 2014), WMT16 En-Ro (Bojar et al., 2016), IWSLT14 De-En (Cettolo et al., 2014)), paraphrasing (QQP), text simplification (Wiki-Auto (Jiang et al., 2020)), and question generation (Quasar-T (Dhingra et al., 2017)). Dataset splits are detailed in Table 16. We use BLEU score (Papineni et al., 2002) as the evaluation metric across all tasks, supplemented with SacreBLEU (Post, 2018) for translation tasks. For comprehensive evaluation, we employ LLM-based evaluation using DeepSeek-V3 685B (DeepSeek-AI, 2024) with specialized prompts, assessing accuracy, fluency, completeness, and task-specific criteria such as creativity for translation and phrasing diversity for paraphrasing. The evaluation pro-
<table>
  <tr>
    <th>T</th>
    <th>Model</th>
    <th>b</th>
    <th>IWSLT WMT14</th>
    <th>WMT16</th>
  </tr>
  <tr>
    <td rowspan="2">D</td>
    <td>Absorbing</td>
    <td>5</td>
    <td>28.32*</td>
    <td>21.62*</td>
    <td>30.41*</td>
  </tr>
  <tr>
    <td>Multinomial</td>
    <td>5</td>
    <td>21.28*</td>
    <td>6.94*</td>
    <td>25.25*</td>
  </tr>
  <tr>
    <td rowspan="2">C</td>
    <td>AR-Diffusion</td>
    <td>1</td>
    <td>26.78</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>AR-Diffusion</td>
    <td>10</td>
    <td>30.64</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="4">H</td>
    <td>SeqDiffuSeq</td>
    <td>1</td>
    <td>28.65†</td>
    <td>23.63†</td>
    <td>23.98†</td>
  </tr>
  <tr>
    <td>SeqDiffuSeq</td>
    <td>10</td>
    <td>30.03†</td>
    <td>24.24†</td>
    <td>26.17†</td>
  </tr>
  <tr>
    <td>Difformer</td>
    <td>1</td>
    <td>30.94</td>
    <td>22.32</td>
    <td>30.74</td>
  </tr>
  <tr>
    <td>Difformer</td>
    <td>10</td>
    <td>32.09</td>
    <td>23.80</td>
    <td>30.93</td>
  </tr>
  <tr>
    <td rowspan="2">H</td>
    <td><b>NeoDiff</b></td>
    <td>1</td>
    <td>32.39†</td>
    <td>24.41†</td>
    <td>30.87†</td>
  </tr>
  <tr>
    <td><b>NeoDiff</b></td>
    <td>10</td>
    <td><b>33.14†</b></td>
    <td><b>25.28†</b></td>
    <td><b>32.31†</b></td>
  </tr>
</table>

Table 1: Machine translation BLEU scores for NeoDiff and baseline methods. T: Model type (AR: Autoregressive, D: Discrete, C: Continuous, H: Hybrid). †: NeoDiff outperforms baselines with beam size \( \leq b \); bold: best result. *: Results from Zheng et al. (2023); ‡: Results from Yuan et al. (2024); remaining data reproduced.

<table>
  <tr>
    <th>T</th>
    <th>Model</th>
    <th>b</th>
    <th>IWSLT WMT14</th>
    <th>WMT16</th>
  </tr>
  <tr>
    <td rowspan="2">D</td>
    <td>CMLM</td>
    <td>5</td>
    <td>29.41*</td>
    <td>23.22*</td>
    <td>31.26*</td>
  </tr>
  <tr>
    <td>CMLM(MBR)</td>
    <td>5</td>
    <td>29.32*</td>
    <td>23.09*</td>
    <td>30.92*</td>
  </tr>
  <tr>
    <td rowspan="2">C</td>
    <td>DiffusionLM</td>
    <td>5</td>
    <td>26.61*</td>
    <td>15.33*</td>
    <td>27.01*</td>
  </tr>
  <tr>
    <td>DiffusionLM</td>
    <td>50</td>
    <td>29.11*</td>
    <td>17.41*</td>
    <td>29.39*</td>
  </tr>
  <tr>
    <td rowspan="2">C</td>
    <td>SeqDiffuSeq</td>
    <td>1</td>
    <td>30.16†</td>
    <td>19.16†</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SeqDiffuSeq</td>
    <td>10</td>
    <td>30.45†</td>
    <td>19.76†</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="2">C</td>
    <td>DiNoiSer</td>
    <td>5</td>
    <td>31.29*</td>
    <td>24.25*</td>
    <td>30.93*</td>
  </tr>
  <tr>
    <td>DiNoiSer</td>
    <td>50</td>
    <td>31.61*</td>
    <td>24.26*</td>
    <td>31.08*</td>
  </tr>
  <tr>
    <td rowspan="2">C</td>
    <td>Difformer</td>
    <td>1</td>
    <td>30.06</td>
    <td>22.13</td>
    <td>30.52</td>
  </tr>
  <tr>
    <td>Difformer</td>
    <td>10</td>
    <td>31.08</td>
    <td>23.26</td>
    <td>30.75</td>
  </tr>
  <tr>
    <td rowspan="2">H</td>
    <td><b>NeoDiff</b></td>
    <td>1</td>
    <td>31.50†</td>
    <td>24.09</td>
    <td>31.59†</td>
  </tr>
  <tr>
    <td><b>NeoDiff</b></td>
    <td>10</td>
    <td><b>32.20†</b></td>
    <td><b>24.64†</b></td>
    <td><b>32.21†</b></td>
  </tr>
</table>

Table 2: Comparison on SacreBLEU for machine translation tasks. *: Results from Ye et al. (2023); †: Results from Yuan et al. (2024); remaining data are reproduced. †: NeoDiff outperforms baselines with beam size \( \leq b \).

cess involved providing the LLM with source text, generated text from different models, and specific instructions tailored to each task. Figure 3 shows the prompt templates used. To rigorously assess the diversity of outputs of the model, We also included Inter-Sentence Div-4 as Gong et al. (2022), which measure diversity at the set-of-outputs-per-source level.

Baselines We compared NeoDiff against several strong baselines across multiple diffusion model categories. For discrete diffusion models, we included Absorbing Diffusion (Austin et al., 2021), Multinomial Diffusion (Hoogeboom et al., 2021b), and CMLM (Ghazvininejad et al., 2019). For continuous diffusion models, we benchmarked against DiffusionLM (Li et al., 2022), Dif-former (Gao et al., 2024), SeqDiffuSeq (Yuan et al., 2024), AR-Diffusion (Wu et al., 2024), DiNoiSer (Ye et al., 2024), Meta-DiffuB (Chuang et al., 2024), TESS (Karimi Mahabadi et al., 2024) and TEncDM (Shabalin et al., 2025). For hybrid approaches, we compared with DiffuSeq-V2 (Gong et al., 2023). We also included Transformer and fine-tuned GPT2 models as autoregressive baselines.

<table>
  <tr>
    <th>T</th>
    <th>Model</th>
    <th>b</th>
    <th>QQP</th>
    <th>QT</th>
    <th>WA</th>
  </tr>
  <tr>
    <td rowspan="3">AR</td>
    <td>Transformer</td>
    <td>1</td>
    <td>29.65*</td>
    <td>16.83*</td>
    <td><b>41.68*</b></td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>5</td>
    <td>30.83*</td>
    <td>16.45*</td>
    <td><b>43.86*</b></td>
  </tr>
  <tr>
    <td>GPT2-base FT</td>
    <td>-</td>
    <td>19.80*</td>
    <td>7.41*</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">D</td>
    <td>CMLM</td>
    <td>1</td>
    <td>24.02</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>CMLM</td>
    <td>10</td>
    <td>26.32</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Absorbing</td>
    <td>10</td>
    <td>23.82*</td>
    <td>17.38*</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">C</td>
    <td>Multinomial</td>
    <td>10</td>
    <td>20.70*</td>
    <td>16.96*</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SeqDiffuSeq</td>
    <td>1</td>
    <td>23.28†</td>
    <td>17.20†</td>
    <td>37.09†</td>
  </tr>
  <tr>
    <td>SeqDiffuSeq</td>
    <td>10</td>
    <td>24.34†</td>
    <td>17.46†</td>
    <td>37.12†</td>
  </tr>
  <tr>
    <td rowspan="3">C</td>
    <td>Difformer</td>
    <td>1</td>
    <td>28.52</td>
    <td>16.03</td>
    <td>40.37</td>
  </tr>
  <tr>
    <td>Difformer</td>
    <td>10</td>
    <td>30.43</td>
    <td>16.66</td>
    <td>40.77</td>
  </tr>
  <tr>
    <td>Meta-DiffuB<sup>D<sub>2</sub></sup></td>
    <td>-</td>
    <td>25.52†</td>
    <td>18.20†</td>
    <td>38.77†</td>
  </tr>
  <tr>
    <td rowspan="3">C</td>
    <td>Meta-DiffuB<sup>D<sub>2</sub></sup></td>
    <td>26.32†</td>
    <td>-</td>
    <td>39.57†</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Meta-DiffuB<sup>D<sub>3</sub></sup></td>
    <td>22.71†</td>
    <td>-</td>
    <td>24.71†</td>
    <td>-</td>
  </tr>
  <tr>
    <td>TESS</td>
    <td>30.20†</td>
    <td>19.50†</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">C</td>
    <td>TEncDM(BERT)</td>
    <td>30.20†</td>
    <td>-</td>
    <td>41.60†</td>
    <td>-</td>
  </tr>
  <tr>
    <td>TEncDM(T5)</td>
    <td>30.20†</td>
    <td>-</td>
    <td>41.60†</td>
    <td>-</td>
  </tr>
  <tr>
    <td>TEncDM(RoBERTa)</td>
    <td>30.00†</td>
    <td>-</td>
    <td>40.50†</td>
    <td>-</td>
  </tr>
  <tr>
    <td>H</td>
    <td>DiffuSeq-V2</td>
    <td>1</td>
    <td>22.10§</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="2">H</td>
    <td><b>NeoDiff</b></td>
    <td>1</td>
    <td><b>29.47†</b></td>
    <td><b>20.44†</b></td>
    <td><b>41.57</b></td>
  </tr>
  <tr>
    <td><b>NeoDiff</b></td>
    <td>10</td>
    <td><b>31.32†</b></td>
    <td><b>20.03†</b></td>
    <td><b>41.86</b></td>
  </tr>
</table>

Table 3: BLEU scores on QQP, QT, and WA(Wiki-Auto). *: Results from Zheng et al. (2023); †: Results from Yuan et al. (2024); §: Results from Gong et al. (2023); *: Results from Gao et al. (2024); o: Results from Gong et al. (2022); †: Results from Chuang et al. (2024). \( D_{\theta_1} = \text{DiffuSeq}, D_{\theta_2} = \text{SeqDiffuSeq}, D_{\theta_3} = \text{Dinoiser}; \): Results from Karimi Mahabadi et al. (2024); o: Results from Shabalin et al. (2025). Remaining results reproduced. †: NeoDiff outperforms baselines with beam size \( \leq b \).

We set the maximum noise state \( s_{\max} \) to 100 for all tasks and datasets, incorporating self-conditioning (Chen et al., 2022) and noise rescaling with \( DGS_{\max} = 0.2 \) (Gao et al., 2024). We used byte pair encoding (Sennrich et al., 2016) without knowledge distillation (Kim and Rush, 2016) to evaluate under challenging conditions. During decoding, we employed 2D parallel decoding (Gao et al., 2024) and selected the best candidate sentence using the minimum Bayes risk (MBR) method (Kumar and Byrne, 2004) based
<table>
  <tr>
    <th>Task</th>
    <th>Models</th>
    <th>b</th>
    <th>Semantic Faithfulness</th>
    <th>Fluency</th>
    <th>Completeness</th>
    <th>Phrasing Diversity</th>
  </tr>
  <tr>
    <td rowspan="4">QQP</td>
    <td>CMLM</td>
    <td>10</td>
    <td>72.86</td>
    <td>81.99</td>
    <td>75.60</td>
    <td>55.86</td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>1</td>
    <td>83.64</td>
    <td>92.56</td>
    <td>84.96</td>
    <td><b>57.19</b></td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>5</td>
    <td>83.70</td>
    <td><b>94.73</b></td>
    <td>86.05</td>
    <td>54.52</td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>10</td>
    <td>83.93</td>
    <td>94.64</td>
    <td>86.02</td>
    <td>54.55</td>
  </tr>
  <tr>
    <td></td>
    <td>NeoDiff</td>
    <td>10</td>
    <td><b>87.42</b></td>
    <td>91.87</td>
    <td><b>88.79</b></td>
    <td>45.83</td>
  </tr>
  <tr>
    <th colspan="2">Models</th>
    <th>b</th>
    <th>Accuracy</th>
    <th>Fluency</th>
    <th>Completeness</th>
    <th>Creativity</th>
  </tr>
  <tr>
    <td rowspan="3">WMT14</td>
    <td>Difformer</td>
    <td>10</td>
    <td>79.72</td>
    <td>80.31</td>
    <td>85.24</td>
    <td>75.12</td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>5</td>
    <td><b>85.66</b></td>
    <td><b>86.35</b></td>
    <td><b>90.81</b></td>
    <td><b>80.07</b></td>
  </tr>
  <tr>
    <td>NeoDiff</td>
    <td>10</td>
    <td>80.30</td>
    <td>80.81</td>
    <td>85.61</td>
    <td>76.20</td>
  </tr>
</table>

Table 4: LLM evaluation of text generation tasks using DeepSeek-v3 685B. We evaluate Paraphrasing (QQP Dataset) and Machine Translation (WMT14 En-De Dataset). (1) We access QQP on Semantic Faithfulness, Fluency, Completeness, and Phrasing Diversity. (2) We access WMT14 En-De on Accuracy, Fluency, Completeness, and Creativity. Detailed prompts are provided in Fig 3.

on the BLEU score. We also used post-training Bayesian optimization (Li et al., 2024) to calibrate the extrinsic time schedule, limiting the optimization to 100 rounds for all tasks. Details of the experimental settings are provided in Appendix B.

4.2 Results

Our experimental evaluation demonstrates NeoDiff’s effectiveness across multiple generation tasks. On machine translation benchmarks (Table 1 and 2), NeoDiff consistently outperforms existing non-autoregressive diffusion-based, iteration-based, and autoregressive diffusion approaches. As shown in Table 3, these improvements extend beyond translation to diverse generation tasks. Unlike baselines such as AR-Diffusion that rely heavily on MBR and show performance drops with single samples (\( b = 1 \)), NeoDiff maintains robust performance even in this constrained setting. NeoDiff also demonstrates strong performance in LLM-based evaluations (Table 4, prompts in Figure 3). Notably, on the QQP task (Table 4), NeoDiff achieves superior scores in semantic faithfulness and completeness. For the WMT14 task, NeoDiff achieves performance comparable to Difformer across multiple aspects. NeoDiff also demonstrates strong inter-sentence diversity (Inter-Sentence Div-4) when generating multiple candidates. Detailed comparisons against AR model on QQP dataset can be found in Appendix C (Table 8). Our results show that NeoDiff can balance the quality-diversity trade-off more effectively than autoregressive models like Transformer as the output space scales (i.e., with increasing \( b \)), a characteristic also observed in Gong et al. (2022). The Bayesian Optimization component introduces a manageable overhead (Appendix D.1, approximately 6% of training time on WMT14). Also, NeoDiff’s inference speed and memory usage are competitive with similar models (Appendix D.2).

<table>
  <tr>
    <th>#</th>
    <th>Poisson Diffusion Process</th>
    <th>Time Predictor</th>
    <th>Optimized t Schedule</th>
    <th>BLEU</th>
  </tr>
  <tr>
    <td>Base</td>
    <td></td>
    <td></td>
    <td></td>
    <td>32.09</td>
  </tr>
  <tr>
    <td>+P</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td>32.75</td>
  </tr>
  <tr>
    <td>+PT</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td>32.97</td>
  </tr>
  <tr>
    <td>Full</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td><b>33.14</b></td>
  </tr>
</table>

Table 5: Ablation study on the impact of proposed components on IWSLT14 De-En dataset with a \( b = 10 \).

4.3 Analysis

Our ablation studies (Table 5) demonstrate clear improvements from each component, with the full model achieving a substantial +1.05 BLEU improvement over the baseline. We further analyze each component’s impact on generation quality:

Poisson Process for Multi-token Coherence
The Poisson diffusion process enables more fine-grained control over multiple tokens by precise inter-token coordination. This advantage yields a substantial performance gain over standard continuous diffusion (\( \tau_t = t \)). As evidenced in Table 6A, this improved control manifests itself in better phrase-level coherence.

Time Predictor for Guided Denoising
By leveraging information from less-noised tokens to guide the denoising trajectory of noisier ones, the time predictor enhances the model’s ability of more contextually informed token generation. Table 6B demonstrates this through more natural word selections and verb choices that better preserve the original meaning.
A Src: das zeigt die enorm große rolle , die ein meeresschutzgebiet spielen kann.
Ref: and hence, the enormous role that a marine protected area can play.
Base: and this shows the enormously big role that a area can play with a sea protected
+P: so this shows the enormously big role that a marine protected area can play

B Src: er ist ganz glücklich darüber, weil er sie getäuscht hat.
Ref: he’ll be very happy because he’s deceived you.
+P: he’s very happy about it because he decaked her.
+PT: he’s very happy about it because he deceived her.

C Src: die korrelation ist also gering .
Ref: so the correlation is low .
+PT: so it’s a small of the correlation.
Full: so the correlation is small.

Table 6: Example outputs illustrating three key mechanisms of NeoDiff: (A) improved phrase-level coherence with Poisson process, (B) enhanced token-level refinements with time predictor, and (C) better sentence-level organization with optimized schedule.

Optimized Schedule for Global Coherence The optimized extrinsic time schedule enables dynamic adjustments to the diffusion trajectory, facilitating escape from sub-optimal samples where sequence order or overall structure significantly deviates from the target distribution. This global refinement allows for more substantial rewriting when needed, as demonstrated in Table 6C where entire phrases are better reorganized.

Additional examples demonstrating the impact of these components are provided in Table 11, 12, and 13. In Appendix E, we track the step-wise generation processes, demonstrating superior convergence speed and accuracy for NeoDiff compared to continuous diffusion baselines. We also compared NeoDiff against continuous diffusion baselines on token-level controlled generation, demonstrating its unique ability to perform targeted modifications while maintaining semantic consistency across translations (Appendix F).

5 Conclusion

In this work, we introduce Non-simultaneous Continuous Diffusion Models (NeoDiff), a novel diffusion-based text generation framework that unifies discrete and continuous diffusion models. NeoDiff generalizes the time variable, incorporates the Poisson diffusion process, adaptively modulates the reverse process based on semantic context, and uses an optimized extrinsic time schedule for inference. This unified framework enables fine-grained control and achieves superior performance across diverse natural language processing tasks. Our extensive experiments demonstrate the effectiveness of this unified framework, opening up new avenues for advancing diffusion-based text generation.

Limitations

While NeoDiff demonstrates strong performance across various Seq2Seq-based conditional generation tasks (e.g., machine translation, paraphrasing, text simplification, and question generation), we note some implementation considerations. The post-training optimization of extrinsic time schedules requires additional sampling iterations, though this overhead is negligible compared to the training time. The time predictor introduces a modest parameter increase to the backbone network.

Acknowledgements

This research was supported by the National Natural Science Foundation of China (Grant No. 62276245).

References

Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981–17993.

Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pages 12–58.

Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016. Findings of the 2016 conference on machine translation (wmt16). In First conference on machine translation, pages 131–198. Association for Computational Linguistics.

Eric Brochu, Matthew W. Hoffman, and Nando de Freitas. 2011. Portfolio allocation for bayesian optimization. Preprint, arXiv:1009.5419.

Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th iwtl evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pages 2–17.
Jiaoo Chen, Aston Zhang, Mu Li, Alex Smola, and Diyi Yang. 2023. A cheaper and better diffusion language model with soft-masked noise. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4765–4775, Singapore. Association for Computational Linguistics.

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. 2020. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations.

Ting Chen. 2023. On the importance of noise scheduling for diffusion models. Preprint, arXiv:2301.10972.

Ting Chen, Ruixiang ZHANG, and Geoffrey Hinton. 2022. Analog bits: Generating discrete data using diffusion models with self-conditioning. In The Eleventh International Conference on Learning Representations.

Yunyen Chuang, Hung-Min Hsu, Kevin Lin, Chen-Sheng Gu, Ling Zhen Li, Ray-I Chang, and Hung-yi Lee. 2024. Meta-diffuSBs: A contextualized sequence-to-sequence text diffusion model with meta-exploration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437.

Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794.

Bhuwan Dhingra, Kathryn Mazaitis, and William W. Cohen. 2017. Quasar: Datasets for question answering by search and reading. Preprint, arXiv:1707.03904.

Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. 2024. Empowering diffusion models on the embedding space for text generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4664–4683, Mexico City, Mexico. Association for Computational Linguistics.

Marjan Ghazvininejad, Omer Levy, Yinhai Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112–6121, Hong Kong, China. Association for Computational Linguistics.

Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2022. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations.

Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2023. Diffuseq-v2: Bridging discrete and continuous text spaces for accelerated seq2seq diffusion models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9868–9875.

Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. 2023. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11575–11596.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851.

Jonathan Ho and Tim Salimans. 2021. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forrê, and Max Welling. 2021a. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454–12465.

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forrê, and Max Welling. 2021b. Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems, volume 34, pages 12454–12465. Curran Associates, Inc.

Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural crf model for sentence alignment in text simplification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7943–7960.

Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, and Arman Cohan. 2024. TESS: Text-to-text self-conditioned simplex diffusion. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2347–2361, St. Julian’s, Malta. Association for Computational Linguistics.

Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327.

Zhifeng Kong, Wei Ping, Jiati Huang, Kexin Zhao, and Bryan Catanzaro. 2020. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations.

Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter
of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169–176, Boston, Massachusetts, USA. Association for Computational Linguistics.

Bocheng Li, Zhujin Gao, Yongxin Zhu, Kun Yin, Haoyu Cao, Deqiang Jiang, and Linli Xu. 2024. Few-shot temporal pruning accelerates diffusion models for text generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7259–7269, Torino, Italia. ELRA and ICCL.

Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. 2022. Diffusion-Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328–4343.

Dong C. Liu and Jorge Nocedal. 1989. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45:503–528.

Aaron Lou, Chenlin Meng, and Stefano Ermon. 2024. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834.

Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162–8171. PMLR.

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Belgium, Brussels. Association for Computational Linguistics.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.

Alexander Shabalin, Viacheslav Meshchaninov, Egor Chimbulatov, Vladislav Lapikov, Roman Kim, Grigory Bartosh, Dmitry Molchanov, Sergey Markov, and Dmitry Vetrov. 2025. Tenecdm: Understanding the properties of the diffusion model in the space of language model encodings. Preprint, arXiv:2402.19097.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. 2024. Ar-diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems, 36.

Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. 2023. Dinoiser: Diffused conditional sequence learning by manipulating noises. Preprint, arXiv:2302.10025.

Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. 2024. Dinoiser: Diffused conditional sequence learning by manipulating noises. Preprint, arXiv:2302.10025.

Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2024. Text diffusion model with encoder-decoder transformers for sequence-to-sequence generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 22–39, Mexico City, Mexico. Association for Computational Linguistics.

Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. 2023. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737.

A Detailed Derivation of the Training Objective of NeoDiff

Let \( z \) represent a token embedding and \( z_t \) its latent representation at time \( t \), with \( z_0 = z \) and \( z_1 \sim \mathcal{N}(0, \mathbf{I}) \). The joint distribution of the forward process is then given by:

\[
q(z_{>0}, \tau_{>0}|z_0) := \prod_{t>0} q(z_t, \tau_t|z_0) \tag{8}
\]
\[
= \prod_{t>0} q(z_t|z_0, \tau_t)q(\tau_t), \tag{9}
\]

where
\[
q(z_t|z_0, \tau_t) := \mathcal{N}\left(z_t; \sqrt{\alpha(\tau_t)}z_0, \beta(\tau_t)\mathbf{I}\right),
\]
and \( \bar{\alpha}(\cdot) \) and \( \bar{\beta}(\cdot) \) denote noise schedules with their domains scaled to \([0, 1]\).

Given \( t' = t - \Delta t \), the reverse process is defined as
\[
p_\theta(z_{0:1}, \tau_{0:1}) := p_\theta(z_1, \tau_1) \prod_{t'<1} p_\theta(z_{t'}, \tau_{t'}|z_t, \tau_t)
\]
\[
= p_\theta(z_1, \tau_1) \prod_{t'<1} p_\theta(z_{t'}|z_t, \tau_t, \tau_{t'}) p_\theta(\tau_{t'}|z_t, \tau_t).
\]

We further parameterize the distribution of \( z_{t'} \) as
\[
p_\theta(z_{t'}|z_t, \tau_t, \tau_{t'}) = q(z_{t'}|\hat{z}_0(z_t, \tau_t, t), \tau_{t'}),
\]
where \( \hat{z}_0 \) is the model prediction of \( z_0 \).

Following Ho et al. (2020), the training objective is derived from the variational lower-bound
\[
\mathcal{L}_{\text{VLB}} = \mathbb{E}_q \left[ -\log \frac{p_\theta(z_{0:1}, \tau_{0:1})}{q(z_{>0}, \tau_{>0}|z_0)} \right]
\]
\[
= \mathbb{E}_q \left[ -\log \frac{p_\theta(z_1, \tau_1)}{q(z_1, \tau_1|z_0)} \right]
\]
\[
+ \sum_{0<t'<1} -\log \frac{q(z_{t'}|\hat{z}_0(z_t, \tau_t, t), \tau_{t'})}{q(z_{t'}|z_0, \tau_{t'})}
\]
\[
+ \sum_{0<t'<1} -\log \frac{p_\theta(\tau_{t'}|z_t, \tau_t)}{q(\tau_{t'})}
\]
\[
- \log p_\theta(z_0, \tau_0|z_{\Delta t}, \tau_{\Delta t})
\]
\[
= \mathbb{E}_q \left[ \underbrace{\mathrm{KL}(q(z_1, \tau_1|z_0)\|p_\theta(z_1, \tau_1))}_{\mathcal{L}_1} \right]
\]
\[
+ \sum_{0<t'<1} \underbrace{\mathrm{KL}(q(z_{t'}|z_0, \tau_{t'})\|q(z_{t'}|\hat{z}_0, \tau_{t'}))}_{\mathcal{L}_z}
\]
\[
+ \sum_{0<t'<1} \underbrace{\mathrm{KL}(q(\tau_{t'})\|p_\theta(\tau_{t'}|z_t, \tau_t))}_{\mathcal{L}_\tau}
\]
\[
- \underbrace{\log p_\theta(z_0, \tau_0|z_{\Delta t}, \tau_{\Delta t})}_{\mathcal{L}_0}.
\]

Note that \( \mathcal{L}_1 \) is a constant and can be ignored, and \( \mathcal{L}_0 \) also becomes negligible when \( \Delta t \to 0 \). According to prior works (Ho et al., 2020; Li et al., 2022), the term \( \mathcal{L}_z \) can be simplified as
\[
\mathcal{L}_z = \| \hat{z}_0(z_t, \tau_t, t) - z_0 \|^2.
\]

We also add an anchor loss (Gao et al., 2024) \( \mathcal{L}_{\text{anchor}} = \mathbb{E}_q[-\log p_\theta(y|\hat{z}_0(z_t, \tau_t, t))] \) as a regularization term to avoid collapse of the embedding space. Finally, the training objective of the proposed NeoDiff can be written as
\[
\mathcal{L} = \mathcal{L}_z + \mathcal{L}_\tau + \mathcal{L}_{\text{anchor}}.
\]
B  Experimental Settings

B.1 Data Preprocessing

We used byte pair encoding (BPE) (Sennrich et al., 2016) for tokenization. Unlike previous work, we did not employ knowledge distillation (Kim and Rush, 2016) for preprocessing to evaluate our model’s performance under more challenging conditions.

B.2 Model Configuration

For our experiments, we set the maximum noise state \( s_{\max} \) to 100 and used the \( \sqrt{t} \) schedule for training, optimized schedule for inference. To enhance model performance, we applied self-conditioning (Chen et al., 2022). The transition schedule coefficient \( k \) was set to 2, and the maximum truncation time \( t_{\max} \) was set to 0.99. Following Gao et al. (2024), we also employed noise rescaling with a degradation score threshold \( DGS_{\text{MAX}} \) of 0.2.

Regarding the model architecture, we adopted the configuration from Gao et al. (2024) for the IWSLT14 De-En, WMT14 En-De, and WMT16 En-Ro datasets. For the QQP, Wiki-Auto, and QT datasets, we used the configuration from Gong et al. (2022) to enable a fair comparison with these models. Detailed settings are presented in Table 9.

B.3 Training and Generation

We trained our models using NVIDIA RTX 3090 24G GPUs on Ubuntu 18.04 with FairSeq 0.12 (Ott et al., 2019)(MIT-licensed). For the WMT14 En-De and WMT16 En-Ro datasets, training took nearly 4 days and 2 days, respectively, using 4 GPUs. For the IWSLT14 De-En dataset, training took approximately 1 day using a single GPU. The QQP, Wiki-Auto, and QT datasets each required around 8 hours of training on a single GPU. The training data splits are presented in Table 16.

During generation, we used 20 iteration steps (\( K = 20 \)) without early stopping for the IWSLT14 De-En dataset. For the other datasets, we employed 10 iteration steps without early stopping, which is faster than the 20 steps (\( k = 20 \)) used by Gao et al. (2024) across all datasets. We utilized 2D parallel decoding and selected the best sentence using the minimum Bayes risk (MBR) (Kumar and Byrne, 2004) method based on the BLEU score. The reported results are averaged over 3 runs. The random seed is set to 7.

B.4 Optimized Extrinsic Time Schedule

We propose a systematic approach to optimize the extrinsic time schedule \( S = \{t_1, t_2, ..., t_K\} \), where \( K \) denotes the number of diffusion steps and \( t_i \in [0, 1] \) with \( t_1 < t_2 < ... < t_K \). While previous works (Dhariwal and Nichol, 2021; Chen, 2023) focus on optimizing noise schedules with fixed time steps, we directly optimize the time schedule through Bayesian optimization. Our method extends Li et al. (2024)’s framework from discrete subset selection to continuous optimization over the complete schedule.

At its core, our approach is straightforward: we sample text using different time schedules on the validation set and select the schedule that achieves the highest BLEU score for inference. The optimization process (Algorithm 1) employs Gaussian Process-based Bayesian optimization with the GP-Hedge acquisition function (Brochu et al., 2011). Starting from a uniform time schedule, we iteratively propose candidate schedules using Limited-memory BFGS (Liu and Nocedal, 1989) and evaluate them using BLEU scores on the validation set. This approach enables precise calibration of the time schedule while maintaining the ordering constraint \( t_1 < t_2 < ... < t_K \). Following Li et al. (2024), we limit optimization to 100 iterations, keeping the computational overhead negligible compared to model training time (Li et al., 2024). The resulting task-specific schedules demonstrate improved generation quality while maintaining computational efficiency.

C Additional Diversity Analysis on QQP dataset

In this section, we provide a detailed comparison of NeoDiff and Transformer on the QQP task, specifically focusing on multi-candidate generation and inter-sentence diversity. The results presented in Table 8 complement the main paper’s Table 4 by offering a deeper look into how diversity metrics evolve with an increasing number of generated samples (\( b \)).

D Efficiency Analysis

D.1 Bayesian Optimization Overhead (WMT14 En-De):

• Training: 505.88 RTX3090 GPU Hours
• Bayesian Optimization: 28.1 RTX3090 GPU Hours (approximately 6% of training time)
<table>
  <tr>
    <th>Models</th>
    <th>K</th>
    <th>Speed (sentences/second)</th>
    <th>Memory Cost (MB)</th>
  </tr>
  <tr>
    <td>Transformer*</td>
    <td>n</td>
    <td>6.05</td>
    <td>-</td>
  </tr>
  <tr>
    <td>CMLM*</td>
    <td>10</td>
    <td>11.80</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DiffuSeq*</td>
    <td>2000</td>
    <td>0.06</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SeqDiffuSeq*</td>
    <td>2000</td>
    <td>0.05</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Difformer</td>
    <td>20</td>
    <td>6.49</td>
    <td>2034</td>
  </tr>
  <tr>
    <td>NeoDiff</td>
    <td>20</td>
    <td>5.12</td>
    <td>2080</td>
  </tr>
</table>

Table 7: Runtime Comparison on IWSLT14 De-En. *: Results from Gao et al. (2024). Others are reproduced.

<table>
  <tr>
    <th>Model</th>
    <th>b</th>
    <th>Semantic Faithfulness</th>
    <th>Fluency</th>
    <th>Completeness</th>
    <th>Phrasing Diversity</th>
    <th>Inter-Sentence div-4</th>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>1</td>
    <td>83.64</td>
    <td>92.56</td>
    <td>84.96</td>
    <td><b>57.19</b></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>5</td>
    <td>83.70</td>
    <td><b>94.73</b></td>
    <td>86.05</td>
    <td>54.52</td>
    <td>0.686</td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>10</td>
    <td>83.93</td>
    <td>94.64</td>
    <td>86.02</td>
    <td>54.55</td>
    <td>0.561</td>
  </tr>
  <tr>
    <td>NeoDiff</td>
    <td>1</td>
    <td>84.24</td>
    <td>88.95</td>
    <td>87.83</td>
    <td>39.18</td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>NeoDiff</td>
    <td>5</td>
    <td>85.63</td>
    <td>90.69</td>
    <td>88.39</td>
    <td>41.62</td>
    <td>0.684</td>
  </tr>
  <tr>
    <td>NeoDiff</td>
    <td>10</td>
    <td><b>87.42</b></td>
    <td>91.87</td>
    <td><b>88.79</b></td>
    <td>45.83</td>
    <td>0.631</td>
  </tr>
</table>

Table 8: Detailed comparison of NeoDiff and Transformer on the QQP task. Metrics include Semantic Faithfulness, Fluency, Completeness, Phrasing Diversity (single-sample), and Inter-Sentence Diversity (Inter-Sentence Div-4, multi-candidate).

Note: The cost of Bayesian optimization is directly proportional to the amount of data sampled in each iteration. While we used the entire WMT14 validation set, significantly reducing the sample size (e.g., to 20 samples) can drastically lower this overhead to less than 0.1 GPU Hours (Li et al., 2024).

D.2 Runtime Comparison (IWSLT14 De-En)

Table 7 presents a runtime comparison of NeoDiff and several baselines on the IWSLT14 De-En dataset. We measured inference speed (sentences/second) and memory cost (MB). NeoDiff demonstrates competitive inference speed, processing 5.12 sentences per second, which is comparable to Difformer’s 6.49 sentences per second. While significantly faster than diffusion-based models like DiffuSeq and SeqDiffuSeq, NeoDiff’s speed is lower than the highly optimized Transformer and CMLM models. In terms of memory usage, NeoDiff’s 2080 MB consumption is similar to Difformer’s 2034 MB.

E Step-wise Generation Examples on IWSLT14 De-En for NeoDiff and Difformer

Table 14 and 15 present a detailed comparison of the translation generation process on IWSLT14 De-En dataset between NeoDiff and Difformer(continuous diffusion model). After incorporating the three aforementioned components(Poisson process, time predictor and optimized schedule), NeoDiff demonstrates more accurate and faster convergence in translation on some sentences compared to Continuous Diffusion Model(Difformer), as illustrated by the step-by-step generation process. Specifically, NeoDiff avoids some of the common pitfalls of diffusion models, such as getting stuck in local optima or generating repetitive phrases.

F Fine-grained Controlled Generation through Token Manipulation

We demonstrate NeoDiff’s capability for token-level controlled generation while preserving semantic consistency across translations. Given a source sentence \( \mathbf{x}_{src} \) and its latent representation \( \mathbf{z}_0 \), we replace a single token to obtain a modified source \( \mathbf{x}'_{src} \). For translation, we initialize the process with \( \mathbf{z}_0 \) and set \( \tau = 1 \) only for the modified token position, maximizing noise specifically at that location. This targeted noise application enables precise semantic modifications in the output translation \( \mathbf{x}'_{tgt} \) while preserving the remaining content. As shown in Table 10, NeoDiff achieves localized modifications, whereas baseline methods like Difformer tend to alter substantial portions of the output sentence. This controlled generation capability stems from our fine-grained noise paradigm, enabling token-specific manipulation of the generation process.
Algorithm 1 Extrinsic Time Schedule Calibration via Bayesian Optimization

Require: Trained Diffusion Model \( M \),
    Initial Extrinsic Time Schedule \( S_{\text{init}} = \{t_1, t_2, ..., t_K\} \), where \( t_i \in \mathbb{R} \) and \( 0 \leq t_1 < t_2 < ... < t_K \leq 1 \),
    Optimization iterations \( n_{\text{iter}} \),
    Domain for elements in Extrinsic Time Schedule \( \mathcal{D} \subset [0, 1] \),
    Source Text \( T_{\text{src}} \),
    Target Text \( T_{\text{tgt}} \)

Ensure: Optimized Extrinsic Time Schedule \( S_{\text{opt}} = \{t'_1, t'_2, ..., t'_K\} \), where \( t'_i \in \mathbb{R} \) and \( 0 \leq t'_1 < t'_2 < ... < t'_K \leq 1 \)

1: Initialize \( S_{\text{init}} = \{t_1, t_2, ..., t_K\} \) such that \( t_i \) are uniformly spaced in \([0, 1]\).
2: Perform a sampling on \( T_{\text{src}} \) using diffusion model \( M \) and extrinsic time schedule \( S_{\text{init}} \), yielding predicted text \( T_{\text{pred}} \).
3: Compute the BLEU score \( \text{BLEU}(T_{\text{tgt}}, T_{\text{pred}}) \) using \( T_{\text{tgt}} \) and \( T_{\text{pred}} \).
4: Initialize the observation set for Bayesian optimization: \( O \leftarrow \{ (S_{\text{init}}, \text{BLEU}(T_{\text{tgt}}, T_{\text{pred}})) \} \).
5: for \( i = 1 \) to \( n_{\text{iter}} \) do
6:   Update the Gaussian Process posterior given observations \( O \).
7:   Generate a candidate set \( \mathcal{D}' = \{S'_1, S'_2, ..., S'_N\} \), where each \( S'_j = \{t'_{j1}, t'_{j2}, ..., t'_{jK}\} \) represents a candidate extrinsic time schedule with \( t'_{jk} \in \mathcal{D} \) and \( 0 \leq t'_{j1} < t'_{j2} < ... < t'_{jK} \leq 1 \). The candidate set \( \mathcal{D}' \) is generated by performing 20 iterations of Limited-memory BFGS (Liu and Nocedal, 1989) with 5 random initial points within \( \mathcal{D}^K \).
8:   Compute the acquisition function value \( \alpha_{\text{GP-Hedge}}(S'_j) \) (Brochu et al., 2011) for all \( S'_j \in \mathcal{D}' \).
9:   Select the next observation point \( S_i = \arg\max_{S'_j \in \mathcal{D}'} \alpha_{\text{GP-Hedge}}(S'_j) \).
10:  Perform a sampling on \( T_{\text{src}} \) using \( M \) and \( S_i \), yielding predicted text \( T'_{\text{pred}} \).
11:  Compute the BLEU score \( \text{BLEU}(T_{\text{tgt}}, T'_{\text{pred}}) \) using \( T_{\text{tgt}} \) and \( T'_{\text{pred}} \).
12:  Update the observation set: \( O \leftarrow O \cup \{ (S_i, \text{BLEU}(T_{\text{tgt}}, T'_{\text{pred}})) \} \).
13: end for
14: \( S_{\text{opt}} = \arg\max_{(S, \text{BLEU}) \in O} \text{BLEU} \)
<table>
  <tr>
    <th>Hyper-parameters</th>
    <th>WMT14 En-De</th>
    <th>WMT16 En-Ro</th>
    <th>IWSLT14 De-En</th>
    <th>QQP</th>
    <th>Wiki-Auto</th>
    <th>QT</th>
  </tr>
  <tr>
    <th colspan="7">Architecture</th>
  </tr>
  <tr>
    <td>\( d_{\text{model}} \)</td>
    <td>512</td>
    <td>512</td>
    <td>512</td>
    <td>768</td>
    <td>768</td>
    <td>768</td>
  </tr>
  <tr>
    <td>\( d_{\text{emb}} \)</td>
    <td>128</td>
    <td>128</td>
    <td>128</td>
    <td>128</td>
    <td>128</td>
    <td>128</td>
  </tr>
  <tr>
    <td>\( d_{\text{ffn}} \)</td>
    <td>2048</td>
    <td>2048</td>
    <td>1024</td>
    <td>3072</td>
    <td>3072</td>
    <td>3072</td>
  </tr>
  <tr>
    <td>Heads</td>
    <td>8</td>
    <td>8</td>
    <td>4</td>
    <td>12</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Encoder Layers</td>
    <td>6</td>
    <td>6</td>
    <td>6</td>
    <td>6</td>
    <td>6</td>
    <td>6</td>
  </tr>
  <tr>
    <td>Decoder Layers</td>
    <td>6</td>
    <td>6</td>
    <td>6</td>
    <td>6</td>
    <td>6</td>
    <td>6</td>
  </tr>
  <tr>
    <td>Time Predictor Layers</td>
    <td>3</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Activation</td>
    <td>ReLU</td>
    <td>ReLU</td>
    <td>ReLU</td>
    <td>ReLU</td>
    <td>ReLU</td>
    <td>ReLU</td>
  </tr>
  <tr>
    <th colspan="7">Diffusion</th>
  </tr>
  <tr>
    <td>Steps</td>
    <td>10</td>
    <td>10</td>
    <td>20</td>
    <td>10</td>
    <td>10</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Training Schedule</td>
    <td>sqrt</td>
    <td>sqrt</td>
    <td>sqrt</td>
    <td>sqrt</td>
    <td>sqrt</td>
    <td>sqrt</td>
  </tr>
  <tr>
    <td>Inference Schedule</td>
    <td>Optimized</td>
    <td>Optimized</td>
    <td>Optimized</td>
    <td>Optimized</td>
    <td>Optimized</td>
    <td>Optimized</td>
  </tr>
  <tr>
    <td>DGS<sub>MAX</sub></td>
    <td>0.2</td>
    <td>0.2</td>
    <td>0.2</td>
    <td>0.2</td>
    <td>0.2</td>
    <td>0.2</td>
  </tr>
  <tr>
    <td>Self-Conditioning</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <th colspan="7">Training</th>
  </tr>
  <tr>
    <td>Steps</td>
    <td>600K</td>
    <td>400K</td>
    <td>300K</td>
    <td>50K</td>
    <td>100K</td>
    <td>100K</td>
  </tr>
  <tr>
    <td>Batch Size (Tokens)</td>
    <td>32K</td>
    <td>24K</td>
    <td>8K</td>
    <td>8K</td>
    <td>12K</td>
    <td>16K</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
    <td>AdamW</td>
    <td>AdamW</td>
    <td>AdamW</td>
    <td>AdamW</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>Adam \( \beta \)</td>
    <td>(0.9, 0.98)</td>
    <td>(0.9, 0.98)</td>
    <td>(0.9, 0.98)</td>
    <td>(0.9, 0.98)</td>
    <td>(0.9, 0.98)</td>
    <td>(0.9, 0.98)</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>\( 5 \times 10^{-4} \)</td>
    <td>\( 5 \times 10^{-4} \)</td>
    <td>\( 5 \times 10^{-4} \)</td>
    <td>\( 5 \times 10^{-4} \)</td>
    <td>\( 2.3 \times 10^{-4} \)</td>
    <td>\( 2 \times 10^{-4} \)</td>
  </tr>
  <tr>
    <td>Warmup</td>
    <td>10K</td>
    <td>10K</td>
    <td>10K</td>
    <td>10K</td>
    <td>10K</td>
    <td>10K</td>
  </tr>
  <tr>
    <td>Clip Gradient</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>Dropout</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.3</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Length Predict Factor</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Label Smoothing</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
  </tr>
  <tr>
    <th colspan="7">Inference</th>
  </tr>
  <tr>
    <td>Steps</td>
    <td>10</td>
    <td>10</td>
    <td>20</td>
    <td>10</td>
    <td>10</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Bayesian Optimization Rounds</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
</table>

Table 9: The model architectures and hyper-parameters used in our experiments.
Evaluate this translation from {src_lang} to {tgt_lang} (0-100 score):
[Source] {source}
[Reference] {reference}
[Translation] {translation}
Score these aspects STRICTLY IN THIS ORDER:
1. **Accuracy**: Faithfulness to source meaning
2. **Fluency**: Naturalness in target language
3. **Completeness**: Information retention
4. **Creativity**: Handling of ambiguous or open-ended source content

Return ONLY 4 numbers separated by commas, NO text.

Evaluate this paraphrase generation (0-100 score):
[Original] {source}
[Reference] {reference}
[Paraphrase] {paraphrase}
Score these aspects STRICTLY IN THIS ORDER:
1. **Semantic Faithfulness**: Meaning preservation from original
2. **Fluency**: Naturalness in language
3. **Completeness**: Retention of all information
4. **Phrasing Diversity**: Variation in wording/structure while preserving meaning

Return ONLY 4 numbers separated by commas, NO text.

Figure 3: Prompt templates used for LLM-based evaluation. Top: Translation evaluation prompt. Bottom: Paraphrase evaluation prompt.
<table>
  <tr>
    <th>Model</th>
    <th>Generated Content</th>
  </tr>
  <tr>
    <td>NeoDiff <tt>tgt_pred</tt></td>
    <td>and the world we live in now , looks <span style="background-color:#b3d9f4">like this</span>.</td>
  </tr>
  <tr>
    <td>NeoDiff <tt>tgt'_pred</tt></td>
    <td>and the world we live in now , looks <span style="background-color:#e6b9b9">different</span>.</td>
  </tr>
  <tr>
    <td>Difformer <tt>tgt_pred</tt></td>
    <td>and the world we're living in now looks <span style="background-color:#b3d9f4">like this</span>.</td>
  </tr>
  <tr>
    <td>Difformer <tt>tgt'_pred</tt></td>
    <td>and the world that we're living in <span style="background-color:#e6b9b9">right now , it looks different</span>.</td>
  </tr>
  <tr>
    <td><tt>src</tt></td>
    <td>sein ganzer arbeitsprozess hat sich <span style="background-color:#b3d9f4">danach</span> geändert.</td>
  </tr>
  <tr>
    <td><tt>tgt</tt></td>
    <td>and his whole work process changed <span style="background-color:#b3d9f4">after</span> that.</td>
  </tr>
  <tr>
    <td><tt>src'</tt></td>
    <td>sein ganzer arbeitsprozess hat sich <span style="background-color:#e6b9b9">davon</span> geändert.</td>
  </tr>
  <tr>
    <td><tt>tgt'</tt></td>
    <td>His whole work process changed <span style="background-color:#e6b9b9">because of</span> that.</td>
  </tr>
  <tr>
    <td>Model</td>
    <td>Generated Content</td>
  </tr>
  <tr>
    <td>NeoDiff <tt>tgt_pred</tt></td>
    <td>his whole work process has changed <span style="background-color:#b3d9f4">after</span> that.</td>
  </tr>
  <tr>
    <td>NeoDiff <tt>tgt'_pred</tt></td>
    <td>his whole work process has changed <span style="background-color:#e6b9b9">from</span> that.</td>
  </tr>
  <tr>
    <td>Difformer <tt>tgt_pred</tt></td>
    <td>and his whole work process has changed <span style="background-color:#b3d9f4">after</span> that.</td>
  </tr>
  <tr>
    <td>Difformer <tt>tgt'_pred</tt></td>
    <td>and his whole work process has changed <span style="background-color:#e6b9b9">from this</span>.</td>
  </tr>
  <tr>
    <td><tt>src</tt></td>
    <td>der zweite faktor sind die dienste , die wir <span style="background-color:#b3d9f4">nutzen</span>.</td>
  </tr>
  <tr>
    <td><tt>tgt</tt></td>
    <td>the second factor is the services we <span style="background-color:#b3d9f4">use</span>.</td>
  </tr>
  <tr>
    <td><tt>src'</tt></td>
    <td>der zweite faktor sind die dienste , die wir <span style="background-color:#e6b9b9">kennen</span>.</td>
  </tr>
  <tr>
    <td><tt>tgt'</tt></td>
    <td>The second factor is the services we <span style="background-color:#e6b9b9">know</span>.</td>
  </tr>
  <tr>
    <td>Model</td>
    <td>Generated Content</td>
  </tr>
  <tr>
    <td>NeoDiff <tt>tgt_pred</tt></td>
    <td>the second factor is the services that we <span style="background-color:#b3d9f4">use</span>.</td>
  </tr>
  <tr>
    <td>NeoDiff <tt>tgt'_pred</tt></td>
    <td>the second factor is the services we <span style="background-color:#e6b9b9">know</span>.</td>
  </tr>
  <tr>
    <td>Difformer <tt>tgt_pred</tt></td>
    <td>the second factor is the services that we <span style="background-color:#b3d9f4">use</span>.</td>
  </tr>
  <tr>
    <td>Difformer <tt>tgt'_pred</tt></td>
    <td>the second factor is <span style="background-color:#e6b9b9">really</span> the services that we <span style="background-color:#e6b9b9">meet</span>.</td>
  </tr>
</table>

Table 10: Token manipulation example.
<table>
  <tr>
    <th>Source</th>
    <th>Reference</th>
    <th>Base Translation</th>
    <th>+P Translation</th>
  </tr>
  <tr>
    <td>nein war nie eine möglichkeit gewesen .</td>
    <td>no had never been an option .</td>
    <td>no one has never been a possibility.</td>
    <td>no had never been an opportunity.</td>
  </tr>
  <tr>
    <td>weiß jemand , was drei sekunden sind ?</td>
    <td>does anyone know what three seconds are ?</td>
    <td>does anybody know what three seconds?</td>
    <td>does anyone know what three seconds are?</td>
  </tr>
  <tr>
    <td>sie hatten ein konzept von blauem blut .</td>
    <td>they had a concept of blue blood .</td>
    <td>they had a idea of blue blood.</td>
    <td>they had a concept of blue blood.</td>
  </tr>
  <tr>
    <td>und raten sie was wir in dem angriffscode gefunden haben ?</td>
    <td>and guess what we found in the attack code ?</td>
    <td>and do you guess what we’ve found in the code of attack?</td>
    <td>and guess what we found in the code of attack?</td>
  </tr>
  <tr>
    <td>jetzt sehen sie den dalmatiner .</td>
    <td>now you see the dalmatian .</td>
    <td>now this is the dalmatinan.</td>
    <td>now you see the dalmatiner.</td>
  </tr>
  <tr>
    <td>denn die kategorien sagen mir , wie ich sie auseinander halten kann .</td>
    <td>because the categories tell me how to tell them apart .</td>
    <td>because the categories are telling me how i can keep it apart.</td>
    <td>because the categories tell me how to keep them apart.</td>
  </tr>
  <tr>
    <td>wie konnte es möglich sein , dass wir dies tun ?</td>
    <td>how could it be possible that we would do this ?</td>
    <td>so how could it possible for us to do this?</td>
    <td>how could it be possible that we could do this?</td>
  </tr>
  <tr>
    <td>aber es gab immer einen lebenszyklus in ihren präsentationen .</td>
    <td>but there was always a life cycle to their presentations .</td>
    <td>but there has always been a life cycle in your presentations.</td>
    <td>but there was always a life cycle in their presentations.</td>
  </tr>
  <tr>
    <td>wir reden zwiespältig davon .</td>
    <td>we talk about it ambivalently .</td>
    <td>we’re talking about it in elessly.</td>
    <td>we talk about it continually.</td>
  </tr>
  <tr>
    <td>was geschah also jahre danach ?</td>
    <td>so what happened years afterward ?</td>
    <td>so for years after that, what happened?</td>
    <td>so what happened years after that?</td>
  </tr>
</table>

Table 11: Additional examples showing the improvements from introducing the Poisson diffusion process on IWSLT14 De-En dataset. The Base model often produces unnatural word ordering and incorrect lexical choices, while +P shows better handling of complex phrases and more natural English constructions.

<table>
  <tr>
    <th>Source</th>
    <th>Reference</th>
    <th>+P Translation</th>
    <th>+PT Translation</th>
  </tr>
  <tr>
    <td>sie haben ihr telefon gemietet . sie haben es nicht gekauft .</td>
    <td>you rented your phone . you didn’t buy it .</td>
    <td>they’ve rented your phone. they didn’t buy it.</td>
    <td>you rented your phone. you didn’t buy it.</td>
  </tr>
  <tr>
    <td>ihr familie versammelte sich .</td>
    <td>and the family gathered .</td>
    <td>her family.</td>
    <td>her family gathered.</td>
  </tr>
  <tr>
    <td>dunkler urin . dunkel .</td>
    <td>dark urine . dark .</td>
    <td>dark up. dark.</td>
    <td>dark urine. dark.</td>
  </tr>
  <tr>
    <td>diese leute verdienen geld .</td>
    <td>these guys make money .</td>
    <td>these people are earing money.</td>
    <td>these people make money.</td>
  </tr>
  <tr>
    <td>er ist ganz glücklich darüber , weil er sie getäuscht hat .</td>
    <td>he’ll be very happy because he’s deceived you .</td>
    <td>he’s very happy about it because he decaked her.</td>
    <td>he’s very happy about it because he deceied her.</td>
  </tr>
  <tr>
    <td>er hatte 20 minuten herrlicher musik gehabt .</td>
    <td>he had had 20 minutes of glorious music .</td>
    <td>he’d had 20 minutes of god.</td>
    <td>he’d had 20 minutes of glorious music.</td>
  </tr>
  <tr>
    <td>... es dem crowdsourcing beachtung schenkt .</td>
    <td>... paying attention to crowdsourcing .</td>
    <td>... it’s adghting to the crowdsourcing.</td>
    <td>... it gives attention to the crowdsourcing.</td>
  </tr>
  <tr>
    <td>er zeigte immer hier hin .</td>
    <td>he kept pointing here .</td>
    <td>he always showed here.</td>
    <td>he always pointed over here.</td>
  </tr>
  <tr>
    <td>wenn man es verallgemeinert , passiert folgendes .</td>
    <td>if you generalize this , something like this happens .</td>
    <td>when you generate it, this is what happens.</td>
    <td>when you generalize it, this is what happens.</td>
  </tr>
  <tr>
    <td>man konnte manhattan sehen .</td>
    <td>you could see manhattan .</td>
    <td>see manhattan.</td>
    <td>you could see manhattan.</td>
  </tr>
</table>

Table 12: Additional examples demonstrating the impact of the time predictor module on IWSLT14 De-En dataset. The examples show how the time predictor enables finer-grained control primarily through word substitutions and better token-level refinements by leveraging information from less-noised tokens to guide the denoising process.
<table>
  <tr>
    <th>Source</th>
    <th>Reference</th>
    <th>+PT Translation</th>
    <th>Full Translation</th>
  </tr>
  <tr>
    <td>so wie es früher eben entsprechend auf dem dorf passierte .</td>
    <td>just like it used to happen in the village .</td>
    <td>in the same way that happened in the village, it just happened.</td>
    <td>just as it used to happen in the village.</td>
  </tr>
  <tr>
    <td>darum helfen sie da mit , fragen sie bei den leuten mal nach .</td>
    <td>so you’re helping out there , just ask the people .</td>
    <td>so you can help with there, ask about people.</td>
    <td>that’s why they help there with, ask people to ask.</td>
  </tr>
  <tr>
    <td>noch immer sind wir dem storytelling als informationsvermittlung sehr , sehr stark verhaftet .</td>
    <td>what’s left is storytelling .</td>
    <td>but we’re still very sted to storytelling as an information reation, very vivily arrested.</td>
    <td>we’re still very to storytelling as an information mediation, very, very arrested.</td>
  </tr>
  <tr>
    <td>wir wählen jedes jahr einige fellows aus und wir lassen sie mit stadtverwaltungen arbeiten .</td>
    <td>we select a few fellows every year and we have them work with city governments .</td>
    <td>we choose some fellows every year, and we have them work with city adminicies.</td>
    <td>we choose some fellows every year, and we let them work with urban management.</td>
  </tr>
  <tr>
    <td>also bot ich einen 10000 $ preis an software für die gewinner .</td>
    <td>so i offered a 10,000 dollar prize of software to the winning team .</td>
    <td>so i offered a $10,000 price for the winner software.</td>
    <td>so i offered a 100,000 price of software for the winners.</td>
  </tr>
  <tr>
    <td>dies ist in unserem ganzen land der zweitgrösste abfallhuss amerikas .</td>
    <td>this , all over the country , is the second largest waste stream in america .</td>
    <td>this is in our entire country, the two-largest waste river of america.</td>
    <td>this is the second est waste flow in america’s land in our entire country.</td>
  </tr>
  <tr>
    <td>wir haben eine art gleichgewicht erreicht .</td>
    <td>we have reached a kind of equipoise .</td>
    <td>we’ve reached some kind of equilibrium.</td>
    <td>we’ve reached some kind of balance.</td>
  </tr>
  <tr>
    <td>und das ist aber ganz im anfang .</td>
    <td>and that’s just the beginning .</td>
    <td>and that’s just at the very beginning.</td>
    <td>and that’s at the very beginning.</td>
  </tr>
  <tr>
    <td>ich bin überzeugt, dass man irgendwie zur nostalgie , zu wunschdenken hingezogen ist .</td>
    <td>i’m convinced that there’s some sort of pull to nostalgia , to wishful thinking .</td>
    <td>i’m convinced you’ve been drawn to nostalgia, sort of wokkthinking.</td>
    <td>i’m believe that there’s kind of moved to nostalgia, you’re moved to thinking.</td>
  </tr>
  <tr>
    <td>wir haben uns daran gewöhnt , dass dinge linear passieren .</td>
    <td>we no longer imagine the thing in images things in images , but codify them through language .</td>
    <td>we were used to make things happen to linear.</td>
    <td>so we’ve been used to linear that things happen.</td>
  </tr>
</table>

Table 13: Additional examples showing the impact of the optimized schedule on IWSLT14 De-En dataset. These examples demonstrate how the schedule primarily influences the overall sampling trajectory at the sentence level, leading to more natural sentence constructions and better semantic coherence.
<table>
  <tr>
    <th>Time Step</th>
    <th>Difformers Translation</th>
    <th>NeoDiff Translation (Ours)</th>
  </tr>
  <tr>
    <td colspan="3"><b>Source:</b> ihr problem ist , dass sie zu wenig haben .<br><b>Reference:</b> their problem is that they have too little .</td>
  </tr>
  <tr>
    <td>0</td>
    <td>dete@@ social tious falsche foot ere secu rity madeupword0000 sorry fold says write chri@@</td>
    <td>28 lar@@ electricity terms surface ting madeupword0001 madeupword0000 @ gen</td>
  </tr>
  <tr>
    <td>1</td>
    <td>your problem is that is that they have too little .</td>
    <td>their problem is they they have too little .</td>
  </tr>
  <tr>
    <td>2</td>
    <td>the problem of that is that they have too little .</td>
    <td>their problem is that they have too little .</td>
  </tr>
  <tr>
    <td>3</td>
    <td>the problem of that is that they have too little .</td>
    <td>their problem is that they have too little .</td>
  </tr>
  <tr>
    <td>4</td>
    <td>the problem your you is that they have too little .</td>
    <td>their problem is that they have too little .</td>
  </tr>
  <tr>
    <td>5</td>
    <td>the problem your problem is that they have too little .</td>
    <td>their problem is that they have too little .</td>
  </tr>
  <tr>
    <td>6</td>
    <td>and , your problem is that they have too little .</td>
    <td>their problem is that they have too little .</td>
  </tr>
  <tr>
    <td>7</td>
    <td>now , your problem is that they have too little .</td>
    <td>their problem is that they have too little .</td>
  </tr>
  <tr>
    <td>8</td>
    <td>now ; your problem is , they have too little .</td>
    <td>their problem is that they have too little .</td>
  </tr>
  <tr>
    <td>9</td>
    <td>now ; your problem is , you have too little .</td>
    <td>their problem is that they have too little .</td>
  </tr>
  <tr>
    <td>...</td>
    <td>...</td>
    <td>...</td>
  </tr>
  <tr>
    <td>20</td>
    <td>now , your problem is , you have too little .</td>
    <td>their problem is that they have too little .</td>
  </tr>
  <tr>
    <td>Final</td>
    <td>now , your problem is , you have too little .</td>
    <td>their problem is that they have too little .</td>
  </tr>
</table>

Source: denn die kategorien sagen mir , wie ich sie auseinander halten kann .
Reference: because the categories tell me how to tell them apart .

<table>
  <tr>
    <th>Time Step</th>
    <th>Difformers Translation</th>
    <th>NeoDiff Translation (Ours)</th>
  </tr>
  <tr>
    <td colspan="3"><b>Source:</b> denn die kategorien sagen mir , wie ich sie auseinander halten kann .<br><b>Reference:</b> because the categories tell me how to tell them apart .</td>
  </tr>
  <tr>
    <td>0</td>
    <td>es clay madeupword0002 ahead jobs involved line madeupword0001 fold <pad> <unk> giving bu@@ <unk> ers sa@@</td>
    <td>market@@ madeupword0003 madeupword0001 van mas price gun ba madeupword0000 <unk> 3 ator anima@@ once</td>
  </tr>
  <tr>
    <td>1</td>
    <td>because the categ@@ ories tell tell me how can can hold them apart .</td>
    <td>because the categ@@ ories tell me how to keep them apart .</td>
  </tr>
  <tr>
    <td>2</td>
    <td>because the categ@@ ories tell tell me how can can hold them apart .</td>
    <td>because the categ@@ ories tell me how to keep them apart .</td>
  </tr>
  <tr>
    <td>3</td>
    <td>because the categ@@ ories are tell me how i can hold it apart .</td>
    <td>because the categ@@ ories tell me how to keep them apart .</td>
  </tr>
  <tr>
    <td>4</td>
    <td>because the categ@@ ories are telling me how i can hold it apart .</td>
    <td>because the categ@@ ories tell me how to keep them apart .</td>
  </tr>
  <tr>
    <td>5</td>
    <td>because the categ@@ ories are telling me how i can hold it apart .</td>
    <td>because the categ@@ ories tell me how to keep them apart .</td>
  </tr>
  <tr>
    <td>6</td>
    <td>because the categ@@ ories are telling me how i can hold it apart .</td>
    <td>because the categ@@ ories tell me how to keep them apart .</td>
  </tr>
  <tr>
    <td>7</td>
    <td>because the categ@@ ories are telling me how i can hold it apart .</td>
    <td>because the categ@@ ories tell me how to keep them apart .</td>
  </tr>
  <tr>
    <td>8</td>
    <td>because the categ@@ ories are telling me how i can hold it apart .</td>
    <td>because the categ@@ ories tell me how to keep them apart .</td>
  </tr>
  <tr>
    <td>9</td>
    <td>because the categ@@ ories are telling me how i can hold it apart .</td>
    <td>because the categ@@ ories tell me how to keep them apart .</td>
  </tr>
  <tr>
    <td>...</td>
    <td>...</td>
    <td>...</td>
  </tr>
  <tr>
    <td>20</td>
    <td>because the categ@@ ories are telling me how i can keep it apart .</td>
    <td>because the categories tell me how to keep them apart .</td>
  </tr>
  <tr>
    <td>Final</td>
    <td>because the categories are telling me how i can keep it apart .</td>
    <td>because the categories tell me how to keep them apart .</td>
  </tr>
</table>

Table 14: Step-by-step generation process of Difformers(continuous diffusion model) and NeoDiff on IWSLT14 De-En dataset(Part 1/2). NeoDiff converges to the correct translation more quickly and accurately.
<table>
  <tr>
    <th>Time Step</th>
    <th>Difformers Translation</th>
    <th>NeoDiff Translation (Ours)</th>
  </tr>
  <tr>
    <td colspan="3"><b>Source:</b> ich sprach also einige monate später bei einer konferenz .<br><b>Reference:</b> so i spoke at a conference a couple months after that .</td>
  </tr>
  <tr>
    <td>0</td>
    <td>complex positive which affect o went attack@@ care@@ <pad> gers <pad> david fri@@ level</td>
    <td>madeupword0001 le@@ leaders news madeupword0003 sta@@ cannot än@@ madeupword0001 spin@@ published mes@@ exhi@@</td>
  </tr>
  <tr>
    <td>1</td>
    <td>so i i to a few months later at a conference .</td>
    <td>so i spoke at at a a few months later .</td>
  </tr>
  <tr>
    <td>2</td>
    <td>so i i to a few months later at a conference .</td>
    <td>so i spoke at a conference a few months later .</td>
  </tr>
  <tr>
    <td>3</td>
    <td>so i i about a few months later at a conference .</td>
    <td>so i spoke at a conference a few months later .</td>
  </tr>
  <tr>
    <td>4</td>
    <td>so i i about a few months later at a conference .</td>
    <td>so i spoke at a conference a few months later .</td>
  </tr>
  <tr>
    <td>5</td>
    <td>so i i talking a few months later at a conference .</td>
    <td>so i spoke at a conference a few months later .</td>
  </tr>
  <tr>
    <td>6</td>
    <td>so i i talking a few months later at a conference .</td>
    <td>so i spoke at a conference a few months later .</td>
  </tr>
  <tr>
    <td>7</td>
    <td>so i i talking a few months later at a conference .</td>
    <td>so i spoke at a conference a few months later .</td>
  </tr>
  <tr>
    <td>8</td>
    <td>so i was talking a few months later at a conference .</td>
    <td>so i spoke at a conference a few months later .</td>
  </tr>
  <tr>
    <td>9</td>
    <td>so i was talking a few months later at a conference .</td>
    <td>so i spoke at a conference a few months later .</td>
  </tr>
  <tr>
    <td>...</td>
    <td>...</td>
    <td>...</td>
  </tr>
  <tr>
    <td>20</td>
    <td>so i was talking a few months later at a conference .</td>
    <td>so i spoke at a conference a few months later .</td>
  </tr>
  <tr>
    <td><b>Final</b></td>
    <td>so i was talking a few months later at a conference .</td>
    <td>so i spoke at a conference a few months later .</td>
  </tr>
</table>

Table 15: Step-by-step generation process of Difformers(continuous diffusion model) and NeoDiff on IWSLT14 De-En dataset(Part 2/2). NeoDiff converges to the correct translation more quickly and accurately.

<table>
  <tr>
    <th>Splits</th>
    <th>WMT14 En-De</th>
    <th>WMT16 En-Ro</th>
    <th>IWSLT14 De-En</th>
    <th>QQP</th>
    <th>Wiki-Auto</th>
    <th>QT</th>
  </tr>
  <tr>
    <td>Training</td>
    <td>4,500,966</td>
    <td>608,319</td>
    <td>160,215</td>
    <td>144,715</td>
    <td>677,751</td>
    <td>116,953</td>
  </tr>
  <tr>
    <td>Validation</td>
    <td>3,000</td>
    <td>1,999</td>
    <td>7,282</td>
    <td>2,048</td>
    <td>2,048</td>
    <td>2,048</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>3,003</td>
    <td>1,999</td>
    <td>6,750</td>
    <td>2,500</td>
    <td>5,000</td>
    <td>10,000</td>
  </tr>
</table>

Table 16: The dataset splits used in our experiments.

Reviewer paper summaries: This paper introduces Non-simultaneous Continuous Diffusion Models (NeoDiff) for text generation, aiming to unify the strengths of both discrete and continuous diffusion approaches. Specifically, it employs a fine-grained forward “Poisson diffusion” process that allows each token to follow its own noise trajectory, paired with a time predictor that adaptively modulates reverse diffusion based on semantic context. The method further integrates a Bayesian-optimized time schedule to fine-tune when and how noisily corrupted tokens are denoised, thus enhancing generation quality. Experimental results on machine translation, paraphrasing, text simplification, and question generation show that NeoDiff outperforms conventional non-autoregressive diffusion and hybrid methods, highlighting the benefits of precise token-level control and adaptive scheduling.

The paper proposed a framework that integrates the strengths of both discrete and continuous diffusion generation progress for text. Three designs have completed this framework: the Poisson diffusion process as the forward process, context-aware time predictor adaptively modulating the reverse process, and optimized extrinsic time schedule for precise noising control.
Summary of strengths: - The proposed Poisson diffusion process enables fine-grained token-level noise injection, bridging the gap between purely discrete and uniform continuous methods.
- The context-aware time predictor is a compelling innovation, allowing the reverse process to dynamically adjust and leverage partially denoised tokens for improved generation.
- Empirical improvements on diverse benchmarks (e.g., WMT, IWSLT, QQP) suggest a robust and task-agnostic framework.

1. The non-simultaneous diffusion processes is well and carefully designed, which makes the work a certain innovation.
2. The paper and attachment materials are well prepared.
Summary of weaknesses: - Incorporating a time predictor and Bayesian-optimized scheduling heightens the model’s complexity, making it more challenging to implement and tune.
- The Poisson-based approach may become less interpretable under certain hyperparameter choices, as controlling the token-by-token noising distribution can require detailed monitoring.
- The paper mostly focuses on sequence-to-sequence tasks; potential impacts on very long-text scenarios or purely unconditional text generation remain underexplored.

1. It is demonstrated that “the continuous diffusion progress is uniform across tokens, limiting their ability to capture semantic nuances”. Is this a defect unique to continuous models? This also leads to a problem, the second argument of the paper, the time predictor, which is designed to adaptively modulate the denoising progress based on token semantics, seems to be an incremental module to the proposed unification method. Is it plug-and-play and also applicable for the other diffusion models?
2. The motivation and some arguments of the paper (especially in the Introduction), such as “lacking fine-grained control”, require theoretical or references support. 
3. The experimental verification are not convincing since the comparison systems are all from two or three years ago. Newly published works in the past two years are needed, such as [1] [2] and so on.
4. Although this paper concerns the limitations of addressing fine-grained control and capturing semantic nuances, I don't think the experimental tasks, such as machine translation, interpretation, and text simplification, fully demonstrate how the proposed model solves these limitations. The work [2] also combines discrete and continuous diffusion from a methodological perspective, and conducted experiments on five fine-grained control tasks presented by Diffusion-LM.

[1] Aaron Lou, Chenlin Meng, Stefano Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. ICML2024. (Best paper award)
[2] Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, Diyi Yang. A Cheaper and Better Diffusion Language Model with Soft-Masked Noise. EMNLP2023.
Comments, suggestions and typos: - Provide thorough ablation and interpretability studies on the Poisson diffusion’s token-level transitions and how specific hyperparameters affect noise distribution.
- Investigate performance and speed trade-offs for larger-scale or real-time scenarios where the overhead from schedule optimization may be more pronounced.

I am confused by the definition of “times” in this paper, see also lines 099-103, and the formula 300-301.
Overall assessment scores: 4

3.5