Can Diffusion Model Achieve Better Performance in Text Generation? Bridging the Gap between Training and Inference!
Year: 2023

Abstract:
Diffusion models have been successfully adapted to text generation tasks by mapping the discrete text into the continuous space. However, there exist nonnegligible gaps between training and inference, owing to the absence of the forward process during inference. Thus, the model only predicts based on the previously generated reverse noise rather than the noise computed by the forward process. Besides, the widely-used downsampling strategy in speeding up the inference will cause the mismatch of diffusion trajectories between training and inference. To understand and mitigate the above two types of training-inference discrepancies, we launch a thorough preliminary study. Based on our observations, we propose two simple yet effective methods to bridge the gaps mentioned above, named Distance Penalty and Adaptive Decay Sampling. Extensive experiments on 6 generation tasks confirm the superiority of our methods, which can achieve 100× → 200× speedup with better performance. Our code is available at https://github.com/CODINNLG/ Bridge_Gap_Diffusion.

heading: 1 Introduction
text: With the prevalence of AIGC (Artificial Intelligence Generated Content) in recent years, generative models (Kingma and Welling, 2013; Goodfellow et al., 2020) have been receiving more attention. As one of the representative generative models, diffusion models (Sohl-Dickstein et al., 2015; Song et al., 2020) have achieved great success on myriads of generation tasks with continuous data, such as image (Song et al., 2020; Ramesh et al., 2022; Rombach et al., 2022), audio generation (Kong et al., 2020), and molecule generation (Hoogeboom et al., 2022), by iteratively refining the input noise to match a data distribution. More recently, diffusion models have been successfully adapted to text
∗Equal contribution. †Corresponding Author.
generation (Li et al., 2022; Gong et al., 2022; Lin et al., 2022) by first leveraging an extra embedding module that maps the discrete data into the continuous space and then recovering the text from the continuous space with rounding strategy (Li et al., 2022) or logits projection (Strudel et al., 2022).
A typical diffusion-based text generation model contains one reverse process (from noise to data) and one forward process (from data to noise), which is shown in Figure 1. More concretely, both of the two processes can be viewed as Markov chains, where the forward process gradually perturbs the data into Gaussian Noise while the reverse process recovers the original data step by step conditioned on the correlated noise from the forward process. The training stage involves both of the above two processes, while the inference stage only consists of the reverse process, i.e., the model predicts based on the previous noise outputted by the model itself rather than the correlated forward noise. Such discrepancy between training and inference, also called exposure bias (Ranzato et al., 2015), leads to error accumulation as the denoising steps grow during the inference stage (Huszár, 2015; Wiseman and Rush, 2016).
Another drawback of the diffusion model is that it requires multiple iterative denoising steps to produce the final results since the reverse process should approximate the forward process (Ho et al., 2020), which usually involves thousands of steps. Numerous iterative reverse steps of diffusion models are inevitably time-consuming for text genera-
11359
tion. For instance, a diffusion model takes around 12 hours on one single NVIDIA A100 GPU to finish the inference of 10K sentences with a length of 128 while the CMLM-based non-autoregressive model (Ghazvininejad et al., 2019) only takes a few minutes1. To accelerate the inference speed in text generation, down sampling (Nichol and Dhariwal, 2021) is leveraged (Li et al., 2022; Gao et al., 2022; Gong et al., 2022), though much faster but at the cost of performance owing to the gap between the downsampled steps in inference and the full diffusion trajectory in the training stage.
To explore the insights and the potential improvement of the aforementioned training-inference gaps, we conduct a preliminary study with a diffusion model (Gong et al., 2022) on the story generation task and mainly observe that: (1) injecting the noise generated by the model itself into the training stage can improve the model performance, and (2) the uniform downsampling strategy in the inference that treats each step equally impairs the model performance, and adaptive sampling strategy should be applied for different generation stages. Accordingly, we propose two simple yet effective strategies: Distance Penalty and Adaptive Decay Sampling, to bridge the training-inference gaps and accelerate the inference process. Experiments on 6 generation tasks of 3 different settings (directed, open-ended, and controllable) show the superiority of our methods without changing the original architecture of the diffusion model or adding more parameters. Surprisingly, our methods can achieve 100× speedup with performance improvement or 200× acceleration with competitive results.

heading: 2 Background

heading: 2.1 Diffusion Model
text: Diffusion models are one of the prevalent generative models (Sohl-Dickstein et al., 2015; Song et al., 2020; Nichol and Dhariwal, 2021), which can transfer an arbitrary data distribution into the Gaussian noise with the forward process and recover the data from the pure noise with the reverse process and both two processes can be regarded as a Markov chain. Specifically, given the time steps T = {0, 1, · · · , T} and the original data distribution z0 at time step t = 0, the forward process gradually perturbs it into the Gaussian noise
1Both diffusion model and CMLM model share the same backbone model, i.e., Transformer (Vaswani et al., 2017).
zT ∼ N (0, I) at time step t = T :
q(zt | zt−1) = N (zt; √ 1− βtzt−1, βtI), (1)
where zt represents the intermediate noise at time step t and βt ∈ (0, 1) is the scaling factor, controlling the amount of added noise at time step t.
The reverse diffusion process recovers the initial data distribution z0 from the Gaussian noise zT by predicting the noise of current time step t and denoising it into the next reverse state zt−1:
pθ(zt−1 | zt) = N (zt−1;µθ(zt, t),Σθ(zt, t)), (2) where µθ and Σθ can be implemented by neural networks fθ, e.g., Transformer2:
µθ(zt, t) = 1√ αt (zt − βt√ 1− ᾱt fθ(zt, t)), (3)
where αt = 1− βt and ᾱt = ∏t i=1 αi.
Training The training objective of the diffusion model is to maximize the marginal likelihood of data log pθ(z0), and the simplified training objective can be written as (Ho et al., 2020):
Lsimple = T∑
t=1
E q(zt|z0)
||µθ(zt, t)− µ̂(zt, z0)||2, (4)
where µ̂(zt, z0) is the mean of q(zt−1 | z0, zt), and it is worth noting that each intermediate noise zt can be obtained directly without the previous history during the training stage (Equation 12).
Inference The inference stage only consists of the reverse process. To sample zt−1 ∼ pθ(zt−1 | zt) in Equation 2, reparameterization strategy (Kingma and Welling, 2013) is leveraged:
zt−1 = µθ(zt, t) + σtϵ, (5)
where ϵ ∼ N (0, I), σ2t = βt, and zt is initialized with pure Gaussian noise in the beginning. More details about the training and inference stages as well as the derivations are shown in Appendix A.

heading: 2.2 Diffusion Model for Text Generation
text: The core of applying diffusion models for text generation task is the transition between discrete space and continuous space. Existing works mainly introduce the embedding function (Li et al., 2022) E(·) to map the discrete text w = {w1, w2, · · · , wL}
2Σθ is often set as σ2t I (Ho et al., 2020), where σ2t = βt.
of length L into the continuous space E(w) = {E(w1), E(w2), · · · , E(wL)} ∈ RLd. Thus, the diffusion model can handle discrete text generation by adding an extra forward step before t = 0, denoted as q(z0 | w) = N (E(w), σ0I), and another step at the end of the reverse process, i.e., pθ(w | z0). More details are given in Appendix B.

heading: 2.3 Inference Speedup
text: One critical point that prevents the usability of diffusion models in text generation is their slow sampling speed during inference due to the long reverse trajectory, which makes each diffusion step simple and easy to estimate (Sohl-Dickstein et al., 2015). To accelerate the inference speed in text generation tasks, current works (Li et al., 2022; Gao et al., 2022) apply the downsampling strategy (Nichol and Dhariwal, 2021) that picks the subset T ′ = {t′1, t′2, · · · , t′k} from the full diffusion trajectory and each intermediate reverse step can be obtained by: z′t−1 = µθ(z ′ t, t ′) + σ′tϵ.

heading: 3 Gaps between Training and Inference
text: From the above description of diffusion models, we can summarize two gaps: (1) the reverse process at time step t in inference is conditioned on the predicted noise zt+1 by the model itself while zt+1 can be obtained directly with the forward computation q(zt+1 | z0) during training, and (2) the downsampled time subset T ′ in inference is inconsistent with the full diffusion trajectory T in training stage when applying the downsampling method for inference speedup. To calibrate the effects of these two types of training-inference gaps, we launch a study on the story generation task in this section.

heading: 3.1 Study Settings
text: We implement the diffusion model with the transformer model and select the ROC Stories (ROC) corpus (Mostafazadeh et al., 2016) for the story generation task. Specifically, given the prompt or the source sentence wx and the reference wy, we apply the partially noising strategy (Gong et al., 2022) for training (Appendix A). We utilize BLEU (B-2) score (Papineni et al., 2002) to reflect the generation precision (the higher, the better), Lexical Repetition (LR-2) score (Shao et al., 2019) to show the diversity of text (the lower, the better), ROUGE (R-2) to represent the recall of generation result (the higher, the better) and Perplexity (PPL) to reflects the fluency (the lower, the better). More
implementation details are in Appendix C.

heading: 3.2 Analysis
text: Training with Predicted Noise To mitigate the training-inference gap, it is natural to inject part of the predicted noises into the training stage by replacing the forward noise zt+1 in pθ(zt | zt+1) with the predicted noise z′t+1 from the (t + 1)-th step of the reverse process or injecting the predicted noise into zt by replacing ||µθ(zt, t)− µ̂(zt, z0)||2 in Equation 4 with γ1||µθ(zt, t) − µ̂(zt, z0)||2 + γ2||µθ(zt, t) − µ̂(z′t, t)||2, where zt ∼ q(zt | z0) and z′t ∼ pθ(zt | z′t+1). We report the evaluation results in Figure 2 with different settings of γ1 and γ2 and can mainly observe that replacing the forward noise with the predicted noise (γ2 = 1, γ1 = 0) does mitigate the training-inference gap by achieving a better performance than the vanilla training scheme (γ2 = 0, γ1 = 1), and the injecting strategy performs better than the replacing one. More details about noise replacement operation and evaluation results are shown in Appendix D.1.
Sampling Strategy Downsampling can accelerate the inference by uniformly selecting the subsets T ′ from the full diffusion trajectory T but at the cost of performance. Such a uniform sampling strategy treats each reverse step equally while neglecting the discrepancies among them in contribution to the final result. To explore whether such an equal-step sampling strategy brings the performance decrease, we simply compare different nonuniform sampling schemes. Along with the reverse steps, we split the reverse process into three stages [κ1, κ2, κ3] and downsample different numbers of steps for each stage but keep the total downsampled steps the same3. As shown in Figure 3, we can observe that when downsampling more steps from κ1 (orange curve), the model can achieve
3For total number of downsampled steps 20, we can sample {[12, 4, 4], [4, 12, 4], [4, 4, 12], [8, 4, 8]} steps as [κ1, κ2, κ3].
a better performance than other downsampling schemes (green curve, red curve, and purple curve) and even exceed the original full reverse steps (blue curve). In other words, the equal-step uniform downsampling scheme does limit the model capability, and the simple non-uniform downsampling strategy can mitigate such issue and meanwhile accelerate the inference speed.
Extensive Trials As mentioned above, the gap brought by the different diffusion trajectories in the inference stage, i.e., downsampled reverse steps v.s. the full reverse steps, further aggravates the training-inference discrepancy. In view that simply injecting the predicted reverse noise in training can effectively narrow the gaps between training and inference, it is also appealing to make such a strategy adapt to the downsampled diffusion trajectories, i.e., introducing the downsampled reverse noises in the training stage. For instance, we can inject the predicted reverse noise downsampled from the reverse steps of (t, t + δ] into the d-th (d ∼ (t, t+ δ]) forward noise to compute the t-th step reverse noise, i.e., replacing the forward noise zt+1 in pθ(zt | zt+1) with zd∼(t,t+δ].
Intuitively, adding a perturbation with a reasonable range of values in training can make the model more robust towards the perturbation during inference, while an unconstrained perturbation value might risk the model training, e.g., the training collapse in auto-regressive text generation mod-
els (Zhang et al., 2019b). For our purposes, the discrepancy before and after injecting the downsampled reverse noise in each training step should fall in a rational range, which mainly depends on the time step t and the choice of δ. To explore more insights, we depict the discrepancy between predicted reverse noises and forward noises along with 200 randomly selected continuous time steps with the Euclidean distance, which is consistent with the training objective in Equation 4. To simplify the study experiment, we downsample a time step for every twenty steps4. As shown in Figure 4, we can observe that (1) the discrepancy between predicted reverse noises and forward noises is getting larger along with the increase of time step t (red diagonal arrow), and (2) the differences between the forward noise at time step t and the predicted reverse noise from t to t+ δ are becoming larger along with the increase of time step (yellow horizontal arrow). Thus, the range of downsampled reverse noise steps should be gradually narrowed along with the increase of time step.

heading: 3.3 Potential Improvement
text: Based on the analysis mentioned above, we can conclude that: (1) injecting the predicted reverse noise into the training stage can mitigate the training-inference gaps, (2) the scheme of uniform downsampling in inference which treats each step equally harms the model performance, and a nonuniform adaptive method should be designed, and (3) inspired by (1) and (2), we can inject the downsampled reverse noises into the training stage while the range of downsampled steps should be gradu-
4We utilize the diffusion model trained with 240K steps. More implementation details are shown in Appendix D.2
ally narrowed as the time step increases.

heading: 4 Method
text: We propose two simple yet effective methods: Distance Penalty in the post-training stage and Adaptive Sparse Sampling in inference to bridge the gaps without introducing any architecture modification to diffusion models. Thus, it can be flexibly adapted to different diffusion model variants.

heading: 4.1 Distance Penalty
text: We first introduce the Distance Penalty strategy, which injects the Downsampled predicted reverse noise into the post-training stage of diffusion models that consists of T time steps.5 For better illustration, we utilize new symbols K = {0, 1, · · · ,K} for the time steps in the post-training stage to distinguish from the original diffusion trajectory T in the training stage. The overview of the Distance Penalty strategy is shown in Figure 5.
Downsampling Range in Training To obtain a rational predicted reverse noise for each step k, i.e., conduct the downsampling operation in the range Rk = {k−1, · · · , k−h}, and mitigate the traininginference gaps, we constrain the total amount of noises inRk with the threshold ωkadj :
ωkadj = √ 1− ᾱK k′ , (6)
where √ 1− ᾱK denotes the scaling factor that controls the variance of noise accumulated at step K (Appendix A), and k′ is the number of the predefined downsampled steps in inference.
Noise Injection After obtaining the downsampling range Rk = {k − 1, · · · , k − h} for step k, we can inject the predicted reverse noise into reverse step k with Equation 12 in Appendix A, by which we can acquire every predicted reverse noise zd∼Rk with the correlated forward noise zd+1:
   Ldis = K∑ k=1 |Rk|∑ h=1 ||µθ(zk, k)− µ̂(zk−h, k − h)||2
zk−h = µθ(zk−h+1, k − h+ 1) + σ2k−h+1I, (7)
where I ∼ N (0, 1) and σ2k−h+1 = βk−h+1. However, the random item I in zk−h makes the model difficult to converge, and the computation 5To make sure each reverse step can generate a rational noise for perturbation, we apply the Distance Penalty strategy to a converged model. We also apply such a strategy in different training stages in Appendix F.4.
of Ldis is complex. Thus, we apply the simplified Equation 15 in Appendix B, which approximates the model output fθ(zd) to the original data distribution directly, and rewrite the loss function into:
Ldissimple = K∑
k=1
|Rk|∑
h=1
||fθ(zk, k)− fθ(zk−h, k − h)||2, (8)
Considering that the step k is uniformly sampled during the training stage, to avoid the boundary condition k − h < 0, the final training objective is:
Lpost = { Lsimple + γLdissimple k ≥ h Lsimple k < h,
(9)
where Lsimple is the original training objective of diffusion model (Equation 4), and γ is the penalty weight that controls the degree of the constraint.

heading: 4.2 Adaptive Decay Sampling
text: We also apply the Adaptive Decay Sampling (ADS) strategy to mitigate the issues brought by uniform downsampling. More concretely, we split the reverse process into three stages [κ1, κ2, κ3] and adaptively adjust the downsampled steps in each stage according to the total amount of added noise of each stage during the training, i.e., more downsampled steps are required to decompose the large noise, which is controlled by ᾱ1:k (Equation 3):
ηi =    1√ 1− ᾱKi/3 − i−1∑ j=1 ηj i > 1
1√ 1− ᾱK/3
i = 1
(10)
Then, we can treat ηi as the weight to split the total downsampled steps T ′ into different subsets for each stage κi. Such a strategy is associated with the noise scheduler, which controls the calculation of ᾱ1:k, and we put more details in Appendix E.

heading: 5 Experiment

heading: 5.1 Settings
text: We describe the main settings of our experiments in this section, and more implementation details can be referred to Appendix C.
Tasks & Datasets We conduct the experiments on three different generation tasks, i.e., directed, open-ended, and controllable. For directed generation tasks, we utilize the WIkI-AUTO corpus (Jiang et al., 2020) for Text Simplification task and Quora
Question Pairs6 corpus for Paraphrase task. For open-ended generation tasks, we adopt the ROC Stories (ROC) corpus for Story Generation task and Quasar-T (Dhingra et al., 2017) dataset preprocessed by Lin et al. (2018) and Gong et al. (2022)7 for Question Generation task. For controllable text generation task, we utilize the E2E (Novikova et al., 2017) dataset and select Semantic Content control task and Syntax Spans control task. More statistics of datasets are listed in Table 8 of Appendix C.1.
Baselines We apply the DIFFSEQ (Gong et al., 2022) as the baseline for directed generation tasks and open-ended generation tasks and utilize Diffusion-LM (Li et al., 2022) for controllable generation tasks. For both of the above two baselines, we implement fθ with Transformer model and select the converged checkpoint for posttraining. The total diffusion steps T for training and K for post-training are both 2,000. We set the hyper-parameter γ of Ldissimple as 2 and utilize the square-root noise schedule for βt. Besides, we also compare the generation results with the autoregressive (AR) model BART (Lewis et al., 2020) and the none-autoregressive (NAR) model CMLM (Ghazvininejad et al., 2019), which are both implemented with the open library Fairseq toolkit8 (Ott et al., 2019). For open-ended generation task, we utilize nucleus sampling (Holtzman et al., 2019) (top-p=0.5) for the BART model. Meanwhile, for the controllable generation tasks, we apply PPLM (Dathathri et al., 2019) and FUDGE (Yang and Klein, 2021) for guidance.
6https://www.kaggle.com/c/ quora-question-pairs
7https://drive.google.com/drive/folders/ 122YK0IElSnGZbPMigXrduTVL1geB4wEW?usp=sharing
8https://github.com/facebookresearch/fairseq
Evaluation Metrics For open-ended generation tasks, we report BLEU (B-n) (Papineni et al., 2002), ROUGE (R-n) (Lin, 2004), Distinct (Dn) (Li et al., 2016), Lexical Repetition (Repn, 4-gram repetition for n-times) (Shao et al., 2019), BERTScore (Zhang et al., 2019a), Mauve score (Mav) (Pillutla et al., 2021), Perplexity (PPL), and Semantic Similarity (SIM, semantic similarity between generations and corresponding prompts) (Guan et al., 2021)9. We select part of the metrics mentioned above for the evaluation of directed generation tasks and utilize success rate (Ctrl) (Li et al., 2022) to evaluate the control effect. The setting of n is described in each subsection below, and the evaluation results (D-n, Rep-n, and PPL) of the golden text are reported for comparison. For fair comparison, we calculate the PPL and Mauve score with the same pre-trained GPT-2 (Radford et al.) model10. We also fine-tune the GPT-2 model on each downstream dataset and report their PPL and Mauve score in Appendix F.1.

heading: 5.2 Results
text: For all experiments, we compare the performance under the settings of full 2,000 reverse steps and uniformly downsampled 20 steps, aka Respace. We apply the Minimal Bayesian Risk method (Li et al., 2022) and set candidate size |S| as 10.
Open-ended Text Generation We report the open-ended generation results in Table 1 and observe that our method with 2,000 reverse steps can exceed the DIFFSEQ on most of the evaluation metrics (except for the Distinct and Lexical Repetition metrics), especially for PPL and Mauve scores that have significant improvement, which means
9https://huggingface.co/sentence-transformers/ bert-base-nli-mean-tokens
10https://huggingface.co/gpt2-large
our method can generate high-quality and fluency results. For downsampled 20 steps, we can find that our method still surpasses the Respace method except for the diversity metrics (D-2 and LR-2) and suffers from a smaller decrease when compared with DIFFUSEQ results of 2,000 steps. The reason for the high diversity of baselines is that the original DIFFSEQ model or Respace method can generate many meaningless texts, which leads to hard-toread sentences (high PPL score). More details can be referred to Case Study in Appendix H. Besides, our method also achieves better performance than language models, i.e., CMLM and BART.
Directed Text Generation Table 2 summarizes the directed text generation results. We can observe that the improvement in directed text generation tasks is significant that our method achieves better performance than both DIFFSEQ (2,000 steps) and Respace strategy (20 steps), especially the PPL score. We provide more evaluation results of directed generation tasks in Appendix F.2.
Controllable Text Generation The results of controllable text generation are listed in Table 3, where we follow the official setting to evaluate the PPL11. Our method can achieve a better control quality than baselines with higher Ctrl scores and generate more fluency results with lower PPL scores but suffers from low diversity.

heading: 5.3 Ablation Study
text: We conduct the ablation study on the ROC dataset and set candidate size |S| = 1 in this section.
Effect of Distance Penalty We first explore the influence of Distance Penalty by adjusting the hyper-parameter ω and report the results in Table 4. We can observe that when the constraint becomes larger, i.e., ω from 1 to 6, the model can generate more fluent and precise texts but at the cost of diversity. Besides, we also find that our method can surpass the simple post-training strategy, i.e., ω = 0, which means the improvement is brought
11https://github.com/XiangLi1999/Diffusion-LM/ blob/main/train_run.py
by the Distance Penalty rather than post-training, which leads to over-fitting on the training data.
Effect of Downsampling Range To explore the effect of downsampling range Rk in the training stage, we set the range with the constant and report the results in Table 5. We can observe that as the range becomes larger, i.e., more injected noises, the model can generate more fluent results (lower PPL) and more precise results (higher B-2 and R-2) with a smaller sampling range. Thus, adaptively adjusting the sampling range is essential for making a trade-off among different metrics.
Comparison of Sampling Strategies We compare our ADS method with the Respace and the DDIM strategies (Appendix E.3) and can observe that ADS can achieve a better performance (B-2 and R-2) and generate more fluency texts compared with Respace and more diverse texts compared with DDIM. Besides, the performance decline of ADS is smaller than the other two strategies, which shows the robustness of ADS (Appendix F.5).

heading: 5.4 Human Evaluation
text: We compare our method with the vanilla diffusion model on six tasks under 2000 and 20 inference step settings for human evaluation. For each setting, we randomly sample 10 comparison pairs for every task and hire three annotators to give their preferences (win, loss, and tie) for three evaluation criteria: fluency, coherence, and relevance. More details can be referred to Appendix G. To ensure consistency among the three annotators, we report the Fleiss’ kappa score (Fleiss, 1971). The
results are shown in Table 7, and we can observe that all the inter-annotator agreements are substantially consistent (ζ ∈ [0.6, 1]) and our method can achieve better performance than the vanilla diffusion model under both settings. More concretely, as the number of reverse steps decreases, our method can drive the model to generate much better results than the vanilla diffusion model.

heading: 6 Related Work

heading: 6.1 Text Generation via Diffusion Model
text: Denoising diffusion probabilistic models (Ho et al., 2020) have shown promising performance on text generation tasks (Yang et al., 2022; Li et al., 2022; Gong et al., 2022; Austin et al., 2021). There exist two main methodologies, including modeling on discrete state spaces or continuous state spaces (Sohl-Dickstein et al., 2015). Early works mainly focus on designing one discrete corrupting process on discrete space by introducing the absorbing tokens (Austin et al., 2021) or transforming the intermediate state into a uniform categorical base distribution (Hoogeboom et al., 2021). However, such discrete modeling suffers from the scaling of one-hot vectors (Gong et al., 2022), which can be only qualified for uncontrollable text generation. Li et al. (2022) propose the Diffusion-LM, which models the data in the continuous space with one mapping function connecting the continuous space
and discrete text space. Gong et al. (2022); Han et al. (2022) combine the diffusion model with iterative NAR model (Gu et al., 2019) and semi-AR model (Wang et al., 2018) to further improve the performance in text generation tasks. Nevertheless, the approaches above all suffer the inefficiency of inference (reverse process), and the quality of generated text decreases remarkably when applying less denoising steps (He et al., 2022).

heading: 6.2 Inference Acceleration of Diffusion Model
text: One critical drawback of Diffusion Models is that they require many iterations to produce highquality results. Song et al. (2020) propose one denoising diffusion implicit model (DDIM) and redefine the sampling function to accelerate the generation process. Jolicoeur-Martineau et al. (2021) devise a faster SDE solver for reverse diffusion processes, and Salimans and Ho (2021) distill a trained deterministic diffusion sampler into a new diffusion model, which only takes half of the sampling steps to generate a full image. Recent work (Kim and Ye, 2022) also proposes an orthogonal approach Denoising MCMC to accelerate the score-based sampling process of the diffusion model. Nevertheless, all the methods above are designed for the computer vision field, and the inference acceleration of diffusion for text generation is still unexplored.

heading: 6.3 Exposure Bias of Autoregressive Model
text: Exposure Bias is widely recognized as a central challenge in autoregressive models, primarily due to the discrepancies between training and test-time generation, which can result in incremental distortion during the testing phase (Bengio et al., 2015; Schmidt, 2019; He et al., 2021). To mitigate such issue, three mainstream methods have been adopted, including designing new training objectives (Ranzato et al., 2015; Shen et al., 2016; Wiseman and Rush, 2016; Zhang et al., 2019b), adding regularization terms to standard training objective function (Zhang et al., 2019c), as well as adopting reinforcement learning approaches (Bahdanau et al., 2016; Brakel et al., 2017) to minimize the expected loss with Minimum Risk Training.

heading: 7 Conclusion
text: This work focuses on bridging the training and inference gaps of the diffusion model. The result of the preliminary study shows that injecting predicted noise into the model can help mitigate the gaps, and
the uniform downsampling strategy for inference acceleration harms the model performance. Thus, we propose two simple yet effective strategies: Distance Penalty and Adaptive Decay Sampling, to mitigate the aforementioned gaps. Experiments on 6 text generation tasks of 3 different settings show that the model with our methods can achieve better performance and great inference speedup.

heading: 8 Limitation
text: Although our method can improve the performance as well as accelerate the inference speed, it suffers from two problems: (1) the diversity of generated results is low compared with language models (LMs) due to the clamp sampling strategy, and (2) the diffusion steps of post-tuning stage should stay consistent with the steps in the training stage, and there still exists the gaps between training and inference, i.e., |T | = |K| ̸= |T ′|, To mitigate the aforementioned two issues, we can explore a better post-training or training strategy to mitigate the training-inference gaps further. In addition, we found that the diffusion model does not perform well in open-ended generation tasks, such as generating incoherent sentences. This is closely related to the drawbacks of NAR models, which have a strong conditional independence assumption. We will attempt to address this issue in the future.
Ethics Statement
It is worth noting that all the data used in this paper are publicly available, and we utilize the same evaluation scripts to make sure that all the comparisons are fair. We have replaced the people names in the corpora with special placeholders to mitigate the problematic biases (Radford et al.) issue of generation results. Although we have taken some methods to mitigate the problematic biases, such a problem cannot be solved completely. We urge the users to cautiously apply our methods in the real world and carefully check the generation results.

heading: Acknowledgement
text: This work is supported by the National Science Foundation of China (NSFC No. 62206194 and No. 62106165), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), and the Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions. This work is also supported by Beijing Academy of Artificial Intelligence (BAAI).

heading: A Preliminary of Diffusion Model
text: In this section, we provide more details of training and inference.
Training Objective The training objective of the diffusion model is to maximize the marginal likelihood of distribution Ez0∼pdata [log pθ(z0)], and the variational lower bound (VLB) can be written as:
Lvlb = E q(z1:T |z0) [log q(zT | z0) pθ(zT )
+
T∑
t=2
log q(zt−1 | z0, zt) pθ(zt−1 | zt) − log pθ(z0 | z1)]
(11)
Training During the training stage, each intermediate noise zt−1 (1 ≤ t ≤ T + 1) of the forward process can be obtained directly by accumulative multiplication with Equation 1:
q(zt | z0) = N (zt; √ ᾱtz0, (1− ᾱt)I), (12)
where αt = 1− βt and ᾱt = ∏t
i=1 αi. It is worth noting that, according to the reparameterization method, the value of 1− ᾱt denotes the variance of accumulated noise of the current step zt−1, i.e., controlling how much noise should be added at the current step.
Combined with Equation 3, Equation 4 and Equation 12, the training process can be referred to Algorithm 1 (Ho et al., 2020).
Algorithm 1 Training Process 1: repeat 2: sample z0 ∼ q(z0) 3: sample t ∼ Uniform({1, · · · , T}) 4: sample ϵ ∼ N (0, I) 5: calculate zt = √ ᾱtz0 + √ 1− ᾱtϵ
6: gradient descent on∇θ||ϵ− ϵθ(zt, t)||2 7: until converged
Inference For the inference stage, there only exists the reverse process, and each intermediate state zt−1 is strictly conditioned on the previous history. It can be summarized into Algorithm 2:

heading: B Diffusion Models for Text Generation
text: In this section, we provide more details about Embedding Setting, Clamping Strategy, and Partially Noising Strategy.
Algorithm 2 Inference Process 1: sample zT ∼ N (0, I) 2: for t← T, · · · , 1 do 3: sample ϵ ∼ N (0, I) if t > 1,else ϵ = 0 4: zt−1 = 1√αt (zt − βt√ 1−ᾱt ϵθ(zt, t)) + θtϵ
5: end for 6: return z0
Embedding Setting As mentioned in Section 2.2, given the embedding function E(·), we can map the discrete text into the continuous space or transform the noise back to the discrete text. Specifically, such mapping strategy, also called rounding (Li et al., 2022), is achieved by selecting the most probable word in the embedding space by argmax operation: pθ(w | z0) = ∏L i=1 pθ(wi | zi0), where pθ(wi | zi0) is a softmax distribution and zi0 denotes the i-th position of z0 distribution. To train the embedding E , the simplified training objective (Equation 4) should be rewritten as:
L′simple =Lsimple + T∑
t=1
E q(z0:T |w)
[||E(w)−
µθ(z1, 1)||2 + log pθ(w | z0)] (13)
Clamping Strategy To make the rounding operation more precise, the diffusion model applies the Clamping strategy (Li et al., 2022), which forces each predicted vector to commit to its nearest word vector through the embedding E in each reverse step during the inference. Thus, combined with Equation 3, the sampling function of Equation 5 should be rewritten as:
zt−1 = √ αClamp(fθ(zt, t)) + √ 1− αϵ (14)
Besides, it also approximate the training objective of Equation 13 into Equation 15 by scaling the constants:
Ltextsimple = E q(z0:T |w)
[ ||µ̂(zT ; z0||2 + T∑
t=2
||µ̂(zt; z0)
− µθ(zt, t)||2 ] + E
q(z0:1|w)
[ ||E(w)− fθ(z1, 1)||2
− log pθ(w | z0) ] ,
(15) where each reverse diffusion step estimates the z0 directly rather than µ̂(zt, z0).
Partially Noising Strategy For sequence-tosequence text generation tasks, Gong et al. (2022)
propose the Partially Noising Strategy that simply perturbs the target text wx and recovers it conditioned on the source text wx. More concretely, we concatenate wx and wy, denoted as wx ⊕ y and utilize an anchor vector,i.e., E(wx), to replace the wx part after each perturbance during the forward diffusion process. Then, the training objective of the diffusion model can be rewritten as:
Lseq =Lsimple + T∑
t=1
E q(z0:T |w)
[||E(wx ⊕ y)−
µθ(z1, 1)||2 + log pθ(wx ⊕
y | z0)] (16)
C Implementation Details
This section provides more details on dataset processing, baseline settings, and evaluation metrics.
C.1 Dataset Processing
We provide the statistics of each corpus in Table 8. For the ROC dataset, we mask all the names with special placeholders (Guan et al., 2021) and only keep 4 sentences in the target. For directed and open-ended generation tasks, we apply the pretrained tokenizer12. For the E2E dataset, we apply the NLTK package13 for tokenization.
C.2 Baselines
We utilize Diffusion-LM (Li et al., 2022) and DIFFSEQ (Gong et al., 2022) as the diffusion model baselines, both of which are implemented with transformer model, which contains 12 layers and 12 attention heads. For a fair comparison, we utilize the language models CMLM (Ghazvininejad et al., 2019) and BART (Lewis et al., 2020) that have the same model architecture, i.e., Transformerbased model, and the number of model parameters. For the CMLM model, we set the iteration steps in inference as 10. The maximum sequence length
12https://huggingface.co/bert-base-uncased 13https://www.nltk.org/
is 64 for controllable generation and 128 for directed and open-ended generation tasks. All the models are trained from scratch, and we set total step T = 2000 of the diffusion model and apply a square-root noise schedule ᾱt = √ 1− t/T + s, where s is a small constant. We conduct the experiments on 4 NVIDIA A100 (40GB) GPUs (directed generation and open-ended generation) and 1 NVIDIA TITAN V GPU (controllable generation14). We select the best checkpoint according to the loss on the validation set (directed generation and open-ended generation) or test the PPL value at the end of each epoch (controllable generation). The total training steps and training time (second) are listed in Table 9. To stay consistent with the baselines, we use the Minimum Bayesian Risk decoding (MBR) method (Li et al., 2022) in all the experiments, setting the candidate size |S| = 10.
C.3 Evaluation Metrics
For Lexical Repetition (LR-n) score, we calculate the repetition times n of k-gram texts in the generation results and select the hyper-parameter n and k according to the average generation length. Specifically, we choose k = 4, n = 2 for openended generation task, k = 2, n = 2 for directed generation task, and k = 2, n = 1 for controllable generation task. Besides, for the Semantic Similarity metric, we utilize the Sentence-BERT model15 to compress the whole sentence into a vector and utilize cosine similarity to calculate the distance between two vectors. We apply the pre-trained GPT-2 model16 to calculate the PPL score for open-ended and directed generation tasks as well as utilize the fine-tuned GPT-2 model with E2E dataset for controllable generation task.
14Due to the limitation of the platform. 15https://huggingface.co/sentence-transformers/
bert-base-nli-mean-tokens 16https://huggingface.co/gpt2-large

heading: D Gaps between Training and Inference
text: We provide more detailed preliminary experimental results in this section.
D.1 Training with Predicted Noise
In this section, we provide more details of noise replacement operation. If we want to replace the conditioned noise zt+1 in pθ(zt | zt+1) with another noise zt+δ, we can utilize Equation 2 which transform the probability distribution into the Gaussian distribution, and replace µθ(zt+1, t + 1) in Equation 2 with µθ(zt+δ, t + δ). More experimental results of noise injection are shown in Figure 6.
D.2 Extensive Trials
This section shows more results of the distance between the predicted reverse noise with the forward noise. We plot the result of models trained with 40K, 80K, and 120K steps and show the results of the randomly initialized model, i.e., trained with 0 step, in Figure 7. We can observe that the results of the trained models share a similar trend with Figure 4, which indicates that we can inject the predicted reverse noise in the early stage of training (Appendix F.4).

heading: E Adaptive Decay Sampling
text: In this section, we introduce the concept of noise scheduler and explain the correlation between the Adaptive Decay Sampling (ADS) strategy and
noise scheduler to help better understand our method. Besides we also describe the implementation details of the DDIM method. The overview of the Adaptive Sparse Sampling strategy is shown in Figure 8, where we split the total down-sampled steps into different subsets for three denoising stages [κ1, κ2, κ3] with weight ηi(i ∈ {1, 2, 3})
E.1 Noise Scheduler
The noise scheduler controls the amount of noise added to each diffusion forward step parameterized by ᾱ1:T . As shown in Figure 9, we plot the sqrt noise scheduler (Li et al., 2022), which is defined by ᾱt = 1 − √ t/T + s, where s = 1e − 4 is a small constant that simulates the start noise. We can observe that the noise increases rapidly for the first 500 forward steps and slows down in the latter steps. When we split the total forward diffusion steps into three stages, we can find the model is trained to solve the high-noise with more steps during the training stage.
E.2 Correlation between ADS and Noise Scheduler
We also quantify the amount of remaining noise, i.e., the distance between z0 and zt, in each predicted reverse step t with √ 1− ᾱt and plot the denoising curves in Figure 10. We can observe that the ADS method pays more attention to solving the high-noise compared with Respace strategy, which treats the noise of each stage equally (yellow curve v.s. green curve), and amount of remaining noise decreases rapidly in the third stage (Stage 3), which is correspond to κ3 in three denoising stages [κ1, κ2, κ3] mentioned in Section 4.2. Besides, Figure 10 also confirms our preliminary study of sampling strategy in Section 3.2, i.e., more downsampled steps for the early denoising stage can improve the model performance.
E.3 Implementation of DDIM sampling
We apply the DDIM sampling strategy (Nichol and Dhariwal, 2021) for comparison, which transfers
the Markov inference process into the non-Markov one to speed up the inference, i.e., skip some reverse steps during the inference stage. Given zt, the sampling function can be written as:
zt−1 = √ ᾱt−1
( zt − √ 1− ᾱtfθ(zt, t)√
ᾱt
)
+ √ 1− ᾱt−1 − σ2t fθ(zt, t) + σtϵt,
(17)
where ϵt ∼ N (0, 1) and σt is the hyper-parameter. In this paper, we set σt = 0 for all time step t.

heading: F Main Result & Ablation Study
text: In this section, we provide more experimental results and implementation details.
F.1 Evaluation with Fine-tuned GPT-2 Model We report the Mauve and PPL scores calculated with fine-tuned GPT-2 model for each task in Table 10. Specifically, the GPT-2 model is fine-tuned with the language modeling task on each downstream dataset in 3 epochs and then employed to evaluate the generation results.
F.2 Directed Generation Results We provide the full evaluation results of directed generation tasks in Table 12.
F.3 Sampling Strategy Comparison We provide the full results of different sampling strategies on ROC and WIKI-AUTO datasets as
well as report the inference speed17 in Table 13.
F.4 Speed up the Training
To save the total training time, we explore the insights of post-training the diffusion model with the Distance Penalty method from its early training stage rather than from the end of its training stage. We conduct the experiment on the ROC dataset and set the candidate size |S| of MBR as 1 for convenience. As shown in Table 14, we can observe that post-tuning with the Distance Penalty method can bring massive improvement to the diffusion model, and it can still achieve a great performance when post-training the model with few warm-up training steps, i.e., 40K(Start) + 30K(Post). Besides, the improvement will be more significant when posttraining the model with more training steps.
17When calculating the inference speed, we set the length of generated results to the same and control the batch size as 1.
F.5 Robustness of Adaptive Decay Sampling To reflect the decrease of each evaluation metric along with fewer inference steps more clearly, we plot the rate of change for each metric in Figure 11. We can find that the change rate of our ADS strategy is lower than the Respace strategy, which means our method has better robustness as the number of down-sampled steps decreases.

heading: G Human Evaluation
text: We show the statistic of human evaluation data in Table 11 and human evaluation interface in Figure 12 and 13. We build the human evaluation interface with the open-source python web library Django 18. As shown in Figure 13, during the evaluation, each comparison pair contains one prompt and two corresponding outputs generated from two different models. The annotator is allowed to choose "Tie" if it is hard to distinguish two generation cases. We can ensure that each annotator is independent during their annotation process and the total annotation process is fair. We hired three annotators and payed each annotator $ 0.05 for comparing each pair. The payment is reasonable considering that it would cost average 30 seconds for an annotator to finish a comparison.

heading: H Case Study
text: In this section, we present part of the generated results of each task for better illustration. We randomly select the cases generated by the diffusion model of 2,000 steps and 20 steps, and the diffusion model post-trained with the Distance Penalty method (2,000 steps) and ADS strategy (20 steps). For clear representation, we utilize green color denotes the key phrase related to the prompt, red color locates the phrase contradicts the prompt, and blue
18https://www.djangoproject.com
color highlights the serious grammatical errors. We can observe that, with our methods, the model can generate more fluency and high-quality texts, while the original model generates many repetitions or meaningless tokens. It is worth noting that the language model (pre-trained GPT-2) may allocate a good PPL score to the sentence with many repetition tokens, and those texts with massive meaningless tokens or hard-to-read sentences may achieve a better Distinct and Lexical Repetition score.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 8
3 A2. Did you discuss any potential risks of your work? Section 8 and Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5 and Appendix C
3 B1. Did you cite the creators of artifacts you used? Section 5 and Appendix C
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 5
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 5 and Appendix C
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 5 and Appendix C
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix C.1 and Appendix G
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5, Appendix C, Appendix D
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5 and Appendix F
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5 and Appendix C
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 5.4 and Appendix G
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section 5.4 and Appendix G
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Section 5.4, Appendix G and Ethics Statement
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We will attempt to address this issue in the future