Venue: NAACL 2024

# Diffusion Glancing Transformer for Parallel Sequence-to-Sequence Learning

 Lihua Qian\({}^{1}\) Mingxuan Wang\({}^{1}\) Yang Liu\({}^{2}\) Hao Zhou\({}^{2,3}\)

\({}^{1}\) ByteDance \({}^{2}\)Institute for AI Industry Research (AIR), Tsinghua University

\({}^{3}\) Shanghai Artificial Intelligence Laboratory

{qianlihua,wangmingxuan.89}@bytedance.com

liuyang2011@tsinghua.edu.cn

zhouhao@air.tsinghua.edu.cn

###### Abstract

Previously, non-autoregressive models were widely recognized as being superior in generation efficiency but inferior in generation quality due to the challenges of modeling multiple target modalities. To enhance the multi-modality modeling ability, we propose the diffusion glancing transformer (DiffGlat), which employs a modality diffusion process and residual glancing sampling. The modality diffusion process is a discrete process that interpolates the multi-modal distribution along the decoding steps, and the residual glancing sampling approach guides the model to continuously learn the remaining modalities across the layers. Experimental results on various machine translation and text generation benchmarks demonstrate that DiffGlat achieves better generation accuracy while maintaining fast decoding speed compared with both autoregressive and non-autoregressive models.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 4846–4862

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

The Transformer Vaswani et al. (2017) has been the most widely used architecture in sequence generation, outperforming its counterparts in (almost) all downstream tasks, such as machine translation and question answering Brown et al. (2020); OpenAI (2023). The typical transformer architecture adopts the autoregressive decoding strategy (AR), generating tokens in a predefined order, i.e., left to right. Recently, non-autoregressive generation models (NAR) attract increasing attention for their fast generation speed, which are _considered_ to sacrifice generation quality by generating the outputs simultaneously instead of sequentially.

However, we argue that NAR has advantages compared to AR _beyond generation efficiency_ for the following reasons: 1) the parallel generation enables NAR to remove the _inductive bias_ of the _predefined generation order_, thereby liberating the potential of applications, such as molecules or proteins that have no well-defined orders; 2) furthermore, NAR can utilize _bi-directional context_ for sequence modeling and generation, whereas AR could only exploit context from one direction.

Although being promising, current state-of-the-art NAR models still fall behind the AR counterpart in terms of generation quality. The main drawback of NAR lies in the difficulty of modeling multimodal distributions Gu et al. (2018), i.e., one input has multiple valid outputs. For example, a source sentence could be translated into multiple different target sentences. As the NAR model decodes all the tokens in parallel, it may mix tokens from different translation targets. In contrast, the AR model can output consistency tokens more easily as every prediction is conditioned on previous ones. Recent progress of NAR has significantly improved the multi-modal learning ability of parallel generation, including knowledge distillation Kim and Rush (2016), iterative decoding Lee et al. (2018), latent variable modeling Ma et al. (2019) and revised learning objectives Libovicky and Helcl (2018); Ghazvininejad et al. (2020); Du et al. (2021); Huang et al. (2022). However, these NAR models still hardly outperform the AR baseline consistently.

In this paper, we propose DiffStat, which shows that NAR can outperform AR in both _efficiency_ and _accuracy_, without requiring knowledge distillation from AR. Generally DiffStat is designed within the denoising diffusion implicit model framework Song et al. (2021).

Figure 1: Our modality diffusion process. \(q(\mathbf{y_{t}})\) represents the marginal distribution of \(\mathbf{y_{t}}\), which visualizes the modality distribution. The number of modalities increases as the timestep \(t\) decreases.



First, to reduce the difficulty of learning multi-modalities for NAR, DiffGlat defines a discrete _modality diffusion process_ that _smoothly_ decomposes the modality learning in the data across many diffusion transition steps. With modality decomposition, each diffusion transition only includes a scheduled number of modalities, which makes the modality learning much easier for NAR. And our preliminary experiments confirm the effectiveness of modality decomposition (See Section 2.2). Note that each diffusion transition is learned by several adjacent NAR layers, and thus DiffGlat can stack a sufficient number of layers to model complex multi-modal distributions.

Besides, we proposes a _residual glancing sampling_ technique, which adaptively adjusts the learning difficulty of each diffusion transition in a layerwise and residual manner.

Experiments demonstrate that the proposed Diffglat significantly improves the quality of NAR generation on standard benchmarks. Using only 1 decoding iteration, DiffGlat can outperform all NAR baselines, including iterative ones. With 3 iterative decoding steps, DiffGlat beats the strong AR baseline with a moderate margin (+0.47 BLEU). Comparisons between DiffGlat and AR Transformer on 10 standard machine translation benchmarks, with both Transformer base and big settings, consistently verify the effectiveness of DiffGlat. Additionally, we also find that Diffglat can slightly outperform the AR baseline on image captioning and paraphrasing tasks. These results show the extensive potential of DiffGlat in other generation tasks.

## 2 Preliminary

Given the input sequence \({\mathbf{x}}=x^{1},x^{2},...,x^{m}\) and the target sequence \({\mathbf{y}}=y^{1},y^{2},...,y^{n}\), the NAR model factorizes the joint probability \(P({\mathbf{y}}|{\mathbf{x}})\) with the conditional independence assumption:

\[p({\mathbf{y}}|{\mathbf{x}};\theta)=\prod_{i=1}^{n}p(y^{i}|{\mathbf{x}};\theta), \tag{1}\]

where \(\theta\) is the parameter of the model, and each token \(y^{i}\) in \({\mathbf{y}}\) is independent from other target tokens. The conditionally independent factorization demands the model to capture the modalities for the combination of all the tokens in one step, which can be difficult when the number of modalities is large. In contrast, the AR model factorizes the joint probability in an autoregressive way:

\[p({\mathbf{y}}|{\mathbf{x}};\theta)=\prod_{i=1}^{n}p(y^{i}|y^{<i},{\mathbf{x}};\theta), \tag{2}\]

By conditioning on preceding tokens \(y^{<i}\), the autoregressive factorization divides the modality learning of the sequence into multiple steps where each step learns to predict one token. Similar to the autoregressive factorization, the denoising diffusion models also smooth the source-target transformation by interpolating intermediate distributions between the source and the target.

Inspired by the learning decomposition in AR and diffusion models, we employ training procedures as in diffusion models to decompose the complex modality learning of NAR into several easier diffusion transition steps. Since the process of adding Gaussian noise is not designed for tackling the multi-modality problem, we also explore new diffusion processes to address the problem. A preliminary study demonstrates that a learning process with a gradually growing number of modalities is beneficial for learning modalities in NAR.

### Denoising Diffusion Models

With a series of latent variables \({\mathbf{y}}_{1}{\mathbf{y}}_{2}...{\mathbf{y}}_{T}\), the diffusion process can be characterized by the posterior \(q({\mathbf{y}}_{1:T}|{\mathbf{y}}_{0})\). This process guides the generative process \(p({\mathbf{y}}_{0:T}):=p_{\theta}({\mathbf{y}}_{T})\prod_{t=1}^{T}p_{\theta}({\mathbf{y}} _{t-1}|{\mathbf{y}}_{t})\) in diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) to fit the data \({\mathbf{y}}_{0}\) step by step. Depending on the Markov property of the process defined by \(q({\mathbf{y}}_{1:T}|{\mathbf{y}}_{0})\), the diffusion processes can be divided into Markovian and non-Markovian ones.

Markovian Diffusion ProcessThe Markovian diffusion processes are employed in many previous work for diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020). In these work, the forward process is a Markov chain where the posterior for \({\mathbf{y}}_{t}\) can be determined by conditioning on \({\mathbf{y}}_{t-1}\):

\[q({\mathbf{y}}_{1:T}|{\mathbf{y}}_{0}):=\prod_{t=1}^{T}q({\mathbf{y}}_{t}|{\mathbf{y}}_{t-1}) \tag{3}\]

For example, Ho et al. (2020) propose a denoising diffusion probabilistic model (DDPM), which adds Gaussian noise with increasing variances \(\beta_{1:T}\in(0,1]^{T}\), and the forward transition probability is defined by: \(q({\mathbf{y}}_{t}|{\mathbf{y}}_{t-1}):=\mathcal{N}({\mathbf{y}}_{t};\sqrt{1-\beta_{t}}{ \mathbf{y}}_{t-1},\beta_{t}\mathbf{I})\). The posterior \(q({\mathbf{y}}_{t-1}|{\mathbf{y}}_{t},{\mathbf{y}}_{0})\) of DDPM can be computed in closed form, and the detailed derivation can be found in (Ho et al., 2020) 

Non-Markovian Diffusion ProcessBesides the Markovian process, the posterior \(q({\mathbf{y}_{1:T}|\mathbf{y}_{0}})\) can also be modeled as a non-Markovian process Song et al. (2021):

\[q({\mathbf{y}_{1:T}|\mathbf{y}_{0}}):=q({\mathbf{y}_{T}|\mathbf{y}_{0}})\prod_{t=2}^{ T}q({\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{y}_{0}}), \tag{4}\]

where each decomposition term depends on \(\mathbf{y}_{0}\). According to Song et al. (2021), non-Markovian processes are less stochastic than the Markovian process defined in Ho et al. (2020), leading to more deterministic and efficient generative models. Here, we propose to use the non-Markovian process to define the modality increasing process directly.

### Proof of Concept

In addition, we also include a synthetic experiment to demonstrate that learning with growing number of modalities across layers can effectively benefit the performance of NAR. Specifically, we create a synthetic dataset with the 4 modalities shown in Table 1, where each source has only one target but may come from one of four different modalities. We simulate the modality diffusion process by using the progressive modality learning (PML) inside NAR layers. In PML, we train the middle layers of NAR with only 2 modalities by transforming the target of modality I to modality II and the target of modality of III to modality IV. And we train the last decoder layer by the targets with 4 modalities (See Appendix A for more details).

The results in Table 2 show that the vanilla NAR model fails to output correct sequences, and both the middle and last layer of the NAR trained with PML achieve significant accuracy gains. From the modality distribution plot in Figure. 2, we can find that the NAR trained with PML learns only 2 modalities in the middle layer, and captures all the modalities in the last layer.

## 3 The Proposed Diffusion-GLAT

In this section, we will introduce DiffStat, which enables parallel sequence-to-sequence learning in a denoising diffusion implicit model DDIM, Song et al. (2021) framework. The primary goals of designing DiffStat are: a) decomposing the modality learning to reduce the training difficulty of NAR and b) achieving high generation quality with few denoising transitions to keep fast decoding.

In denoising diffusion models, the parameters \(\theta\) are trained to fit the data distribution \(q(\mathbf{y}_{0})\) by maximizing a variational lower bound Ho et al. (2020):

\[\mathcal{L}_{\text{VLB}}=\mathbb{E}_{q(\mathbf{y}_{0}|\mathbf{x})}\Big{[} -\log p_{\theta}(\mathbf{y}_{0}||\mathbf{x})\Big{]}\leq \tag{5}\] \[\mathbb{E}_{q(\mathbf{y}_{0:T}|\mathbf{x})}\Big{[}D_{\text{KL}}\big{(}q( \mathbf{y}_{1:T}|\mathbf{y}_{0},\mathbf{x})||p_{\theta}(\mathbf{y}_{0:T}|\mathbf{x})\big{)}\Big{]}\]

To achieve the two goals described in the beginning, we define a discrete diffusion process \(q^{\text{MDP}}\) that adds the modalities in the data gradually as the diffusion step \(t\) decreases. Additionally, the training process is equipped with the residual glancing sampling techniques \(p_{\theta}^{\text{RGS}}\), which further boosts the modality learning ability. Therefore, we can rewrite the training objective in Eq. 5 as:

\[\mathcal{L}_{\text{VLB}}= \mathbb{E}_{q}D_{\text{KL}}(q(\mathbf{y}_{1:T}|\mathbf{y}_{0},\mathbf{x})|p_ {\theta}(\mathbf{y}_{0:T}|\mathbf{x})) \tag{6}\] \[= \mathbb{E}_{q}D_{\text{KL}}(q^{\text{MDP}}(\mathbf{y}_{1:T}|\mathbf{y}_ {0},\mathbf{x})||p_{\theta}^{\text{RGS}}(\mathbf{y}_{0:T}|\mathbf{x}))\]

The modality diffusion process \(q^{\text{MDP}}\) gradually interpolates modalities among intermediate layers, which adaptively schedules sequence modalities across the NAR Transformer layers. And the residual glancing sampling (RGS) samples the target tokens that are not correctly captured across neural layers to enhance the modality learning process.

### Modality Diffusion Process: \(q^{\text{MDP}}\)

In order to reduce the difficulty of learning modalities for NAR, we introduce a discrete diffusion process that distributes modalities to multiple transitions, which is illustrated in Figure 1. As shown in Section 2.2, a reasonable denoising process for learning modalities is adding the modalities in \(\mathbf{y}_{0}\) gradually. But the modalities in the real world data

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline  & & & & & NAR & w/ PML \\ Model & DATA & AR & NAR & Middle & Last \\ \hline Token Acc. & 100 & 100.0 & 56.6 & 98.8 & 97.7 \\ Seq Acc. & 100 & 100.0 & 0.0 & 93.4 & 88.6 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results of synthetic experiments.

Figure 2: The output modality distributions.

 

cannot be explicitly manipulated, thus we need to divide the modalities in \({\mathbf{y}_{0}}\) implicitly. Previous work reveals that the trained NAR models could capture part of the modalities in the data Zhou et al. (2020). Therefore, we leverage the modality distribution captured by the model itself to construct the modality diffusion process.

Specifically, \(q^{\text{MDP}}\) should guarantee that the difficulty of the assigned modality learning task is appropriate for each transition. Thus, we define each transition of the modality diffusion process as the interpolation of distribution for target \(\mathcal{P}({\mathbf{y}_{0}})\) and the prediction distribution of the next step \(\mathcal{P}_{t-1}\). Mathematically, the posterior \(q({\mathbf{y}_{1:T}|\mathbf{y}_{0}},{\mathbf{x}})\) can be decomposed as:

\[q^{\text{MDP}}({\mathbf{y}_{1:T}|\mathbf{y}_{0}},{\mathbf{x}}):=\\ q({\mathbf{y}_{T}|\mathbf{y}_{0}},{\mathbf{x}})\prod_{t=2}^{T}q({\mathbf{y}_{t-1}|\mathbf{ y}_{t}},{\mathbf{y}_{0}},{\mathbf{x}}) \tag{7}\]

For simplicity, we omit \({\mathbf{x}}\) for \(q\) in subsequent elaboration. With the model \(p_{\theta}\) and the target \({\mathbf{y}_{0}}\), we interpolate \({\mathbf{y}_{0}}\) and \(p_{\theta}({\mathbf{y}_{T-1}|\mathbf{y}_{T}},{\mathbf{x}})\) to construct intermediate distributions whose numbers of modalities are between the modalities in \({\mathbf{y}_{0}}\) and those captured by \(p_{\theta}\):

\[q({\mathbf{y}_{t}|\mathbf{y}_{0}}):=\bigg{(}\gamma_{t}{\bf 1}+(1- \gamma_{t}){\mathbf{y}_{0}}\bigg{)}\odot\mathcal{P}_{t-1}/Z_{t} \tag{8}\] \[:=\bigg{(}\gamma_{t}\mathcal{P}_{t-1}+(1-\gamma_{t})\mathcal{P}_{ t-1}\odot{\mathbf{y}_{0}}\bigg{)}/Z_{t},\]

where \(\mathcal{P}_{t-1}=p_{\theta}({\mathbf{y}_{t-1}|\mathbf{y}_{t}},{\mathbf{x}})\), \({\mathbf{y}_{0}}\) is the one-hot vector sequence of the target in data, \(\gamma_{t}\in(0,1]\) is the hyper-parameter for controlling the interpolation, \(\odot\) represents element-wise multiplication and \(Z_{t}\in(0,1]^{n}\) is the factor for normalization.1 We can consider the definition of Eq. 8 as data corruption on the modality probability landscape of \(\mathcal{P}\) by extracting the \(\mathcal{P}\) term out of the parenthesis. To ensure the form in Eq. 8 holds for \(t\geq 1\) (See Appendix B), we define \(q({\mathbf{y}_{t-1}|\mathbf{y}_{t}},{\mathbf{y}_{0}})\) as:

Footnote 1: Because \(\sum_{\mathbf{v}}q({\mathbf{y}_{t}|\mathbf{y}_{0}})=1\) is not guaranteed after interpolation, we normalize the distribution with \(Z_{t}=\sum_{i=1}^{|V|}\hat{q}({\mathbf{y}_{t}^{w}|\mathbf{y}_{0}})\), where \(\hat{q}({\mathbf{y}_{t}|\mathbf{y}_{0}})=\gamma_{t}\mathcal{P}_{t-1}+(1-\gamma_{t}) \mathcal{P}_{t-1}\oplus\mathbf{y}_{0}\), \(|V|\) is the size of token categories and \({\mathbf{y}_{t}^{w}}\) is a sequence with the token \(w\) on all the positions.

\[q({\mathbf{y}_{t-1}|\mathbf{y}_{t}},{\mathbf{y}_{0}}):=\gamma_{t-1}\mathcal{ P}_{t-2}+\bigg{(}(1-\gamma_{t-1}) \tag{9}\] \[\mathcal{P}_{t-2}-\omega_{t}\mathcal{P}_{t-1}\bigg{)}\odot{\mathbf{y }_{0}}+\omega_{t}Z_{t}{\mathbf{y}_{0}}\odot{\mathbf{y}_{t}}\]

Here, \(\omega_{t}\) is a hyper-parameter. Similarly, we also re-normalize \(q({\mathbf{y}_{t-1}|\mathbf{y}_{t}},{\mathbf{y}_{0}})\). Intuitively, the sum of the first term with \(\mathcal{P}_{t-2}\) and the second term with \({\mathbf{y}_{0}}\) performs the modality interpolation as in Eq. 8, and adding the third term with \({\mathbf{y}_{0}}\odot{\mathbf{y}_{t}}\) attempts to preserve the modalities captured by \({\mathbf{y}_{t}}\) in the denoising transitions. Thus, \({\mathbf{y}_{t-1}}\) serves as a intermediate target, with fewer modalities than \({\mathbf{y}_{0}}\) but more modalities than \({\mathbf{y}_{t}}\), for smoothing the modality learning.

### Non-autoregressive Generative Process

We define the generative process similar to the iterative refinement process in NAR. Specifically, we use a generative process that only conditions on the input computed by the model itself for every step, removing the gap between training and inference. Besides, we also augment the training with residual glancing sampling (RGS) to further smooth the modality learning, which improves over the glancing training Qian et al. (2021) by layer-wise sampling and selecting tokens that are not captured in the input.

Self-Conditioned Generative ProcessTo keep consistency between the training and inference, we parameterized \(p_{\theta}({\mathbf{y}_{t-1}|\mathbf{y}_{t}})\) completely by the model itself. We set the decoder input sequence as the initial state \({\mathbf{y}_{T}}\), and the embedding of decoder inputs \(emb({\mathbf{y}_{T}})\) to be the initial representation \({\mathbf{H}_{T}}\). For the generative transitions \(p_{\theta}({\mathbf{y}_{t-1}|\mathbf{y}_{t}})\), we compute the next representation \({\mathbf{H}_{t-1}}\) with \({\mathbf{H}_{t}}\), and maps the hidden states representation to the softmax distribution for \(p_{\theta}({\mathbf{y}_{t-1}|\mathbf{y}_{t}},{\mathbf{x}})\):

\[p_{\theta}({\mathbf{y}_{t-1}|\mathbf{y}_{t}},{\mathbf{x}}) =\text{softmax}({\mathbf{H}_{t-1}\mathbf{V}^{T}}), \tag{10}\] \[\text{where}\quad{\mathbf{H}_{t-1}}=f_{\theta}^{(t)}({\mathbf{H}_{t}}, \operatorname*{Enc}({\mathbf{x}})),\]

Here, Enc is the encoder that maps the input \({\mathbf{x}}\) to hidden states, \({\mathbf{V}}\) is the vocabulary embedding matrix, and \(f_{\theta}^{(t)}\) is parameterized by neural layers. Since \({\mathbf{H}_{t}}\) contains the information for \({\mathbf{y}_{t}}\) and is consistent in both training and inference, we use \({\mathbf{H}_{t}}\) as the input for \(p_{\theta}({\mathbf{y}_{t-1}|\mathbf{y}_{t}},{\mathbf{x}})\). The self-conditioned generative process shares similar motivation with self-conditioning Chen et al. (2022) and step-unroll Savinov et al. (2022), but operates in a purer way as we directly transits the hidden states as in standard decoders.

Residual Glancing SamplingPrevious work demonstrate that adaptive sampling target tokens according to the model performance can improve 

the generation quality of NAR (Qian et al., 2021). For models training with diffusion processes, we strengthen the glancing training in a fine-grained way by layer-wise glancing and glancing remaining tokens. The overall residual glancing training procedure is shown in Figure 3.

For layer-wise glancing, we replace part of intermediate hidden states \({\mathbf{H}_{t}}\) with the embedding of tokens sampled from the intermediate target \(\tilde{\mathbf{y}}_{t-1}=\arg\max q({\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{y}_{0}})\), and predict the remaining target tokens \(\widehat{\mathbf{y}^{\text{RGS}}_{t-1}}\) using the glancing target tokens \(\hat{\mathbf{y}^{\text{RGS}}_{t-1}}\):

\[\begin{split}\mathcal{L}^{\text{RGS}}_{t-1}&=-\log p ^{\text{RGS}}_{\theta}({\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{x}})\\ &=-\log p_{\theta}(\widetilde{\mathbf{y}^{\text{RGS}}_{t-1}}|\hat{ \mathbf{y}^{\text{RGS}}_{t-1}},\mathbf{y}_{t},\mathbf{x})\end{split} \tag{11}\]

The sampling number of glancing tokens \(\tilde{\mathbf{y}}^{RSS}_{t-1}\) is determined by the distance between the model output \(\hat{\mathbf{y}}_{t-1}=\arg\max p_{\theta}({\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{x}})\) and the intermediate target \(\tilde{\mathbf{y}}_{t-1}\): \(S_{t-1}=\alpha\cdot d(\tilde{\mathbf{y}}_{t-1},\hat{\mathbf{y}}_{t-1})\), where \(\alpha\) is a hyper-parameter for adjusting the sampling number. Details for glancing could be found in Qian et al. (2021).

Besides layer-wise glancing, we propose to modify the scope for selecting glancing tokens as in Figure 4. Since the input captures part of the modalities, the model can learn only the remaining part that are not correctly captured in the input. Thus, we utilize \(\hat{\mathbf{y}}_{t}\) to schedule the learning, and modify the glancing sampling to only sampling \(S_{t-1}\) tokens that are different between \(\hat{\mathbf{y}}_{t}\) and \(\tilde{\mathbf{y}}_{t-1}\):

\[\mathbb{R}\mathbb{G}\mathbb{G}_{t-1}(\tilde{\mathbf{y}}_{t-1})=Random(\tilde{\mathbf{ y}}_{t-1}/\hat{\mathbf{y}}_{t}) \tag{12}\]

In residual glancing training, we randomly sample \(\tilde{\mathbf{y}}^{RSGS}_{t-1}\) from \(\tilde{\mathbf{y}}_{t-1}\) with the operation \(\mathbb{R}\mathbb{G}\mathbb{G}_{t-1}\).

### Implementation Details

In this section, we will introduce the implementaion details for DiffGlat. To enhance the ability of modality capturing, we employ the DA-Transformer (Huang et al., 2022b). The DA-Transformer (DAT) can strengthen each transition of DiffGlat by distributing the modalities to different positions in the expanded output sequences. We also elaborate the difference between the decoding iterations and the diffusion transitions in the parameterization part.

DA-TransformerIn training, the DAT model expands the output lengths to the predefined max length, and maps target tokens to multiple positions in the expanded sequence. Since the output length of DAT is not equal to the length of the target \(\mathbf{y}\), we use the best alignment of \(\tilde{\mathbf{y}}_{t}\) as the glancing target and for computing \(q^{\text{MDP}}\). The best alignment is obtained by maximizing the output probability:

\[\mathbf{y}^{align}_{t}=\arg\max_{\mathbf{a}\in\Gamma(\tilde{\mathbf{y}}_{t})}p_{t}(\mathbf{a}| \mathbf{x}), \tag{13}\]

where \(\Gamma\) expands \(\mathbf{y}\) to the expanded output length by inserting blanks. For the intermediate target \(\tilde{\mathbf{y}}\) to compute loss \(L^{\text{RGS}}_{t-1}\), we use the decoding result with the original target length \(|\mathbf{y}|\) rather than the aligned target. In inference, after parallel neural network computation, DAT uses links between positions to extract output tokens in the expanded sequence, where we use the Joint-Viterbi decoding proposed by Shao et al. (2022).

ParameterizationAs each transition of the generative process directly forward the hidden states

Figure 4: The details of residual glancing sampling.

Figure 3: Learning procedure of the layer-wise residual glancing sampling.

 

to the next transition, we can build fully non-autoregressive models by stacking the neural layers or iterative models by sharing the parameters of \(\theta_{t}\). And we can use part of the layers in the decoder to perform one denoising transition so that one forward inference of the decoder can perform multiple denoising transitions.

TrainingSince \(q(\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{y}_{0})\) is a distribution consists of multiple modalities, directly minimizing the KL-divergence in Eq. 6 introduces multi-modal targets for every source input. Thus, we optimize the model using the sequence with the highest probability: \(\tilde{\mathbf{y}}_{t-1}=\operatorname*{arg\,max}q(\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{y}_ {0})\). The KL-divergence can then be rewritten as:

\[\mathcal{L}^{\text{RGS}}_{t-1}=-\log p^{\text{ROS}}_{\theta}(\tilde{\mathbf{y}}_{ t-1}|\mathbf{y}_{t},\mathbf{x}) \tag{14}\]

To reduce the denoising steps needed to achieve high quality, we use a small number for the total diffusion steps \(T\) and attempt to fit more modalities with each generative transition \(p_{\theta}(\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{x})\). For training efficiency, we sample a diffusion timestep \(t\) to compute the loss for each training step.

## 4 Experiments

Different from most of the previous work for NAR, we directly train our models on raw data without using data distilled from AR. To verify the effectiveness of our method, we compare DiffGal with several strong NAR and AR baselines on several sequence generation tasks. Furthermore, analysis and ablation studies study are also conducted to demonstrate the effects of each component.

### Experimental Settings

BenchmarksWe conduct experiments on machine translation, paraphrase generation and image captioning. For machine translation, we use 10 machine translation benchmarks: WMT14 En\(\leftrightarrow\)De, WMT17 En\(\leftrightarrow\)Zh, WMT14 En\(\leftrightarrow\)Fr, WMT16 En\(\leftrightarrow\)Ro and WMT13 En\(\leftrightarrow\)Es. The preprocessing follows the procedure in Zhou et al. (2020) and Kasai et al. (2020). For paraphrase generation and image captioning, we use the Quora Question Pairs dataset2 (QQP), and MS-COCO dataset Lin et al. (2014) respectively. The data are tokenized and segmented into subwords using Byte-Pair Encoding Sennrich et al. (2016).

Footnote 2: [https://www.kaggle.com/c/quora-question-pairs](https://www.kaggle.com/c/quora-question-pairs)

Evaluation MetricsWe report the sacreBLEU3Post (2018) scores for machine translation, and the tokenized bleu results are shown in Appendix C. For speedups compared with the Transformer _base_, we follow previous work Gu et al. (2018) by evaluating on WMT14 En-De with batch size 1. Further comparison for decoding speedup are provided in Appendix D.

Hyper-parametersWe build our models based on the Transformer Vaswani et al. (2017) architecture and use the Transformer _base_ setting as default. For machine translation, we train all the models, including the autoregressive Transformer, with batches of 64k tokens and 300k training steps using the Adam optimizer Kingma and Ba (2015).

\begin{table}
\begin{tabular}{l l|r|r|r|r|r|r|r} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{1}{c|}{**Iter**} & \multicolumn{1}{c|}{**WMT14**} & \multicolumn{1}{c|}{**WMTT17**} & \multicolumn{1}{c|}{**WMIT17**} & \multicolumn{1}{c|}{**Average**} \\  & & **En-De** & **De-En** & **En-Zn** & **Zh-En** & **Gap** & **Speedup** \\ \hline AR & Transformer _base_ (Ours) & \(M\) & 27.18\({}^{*}\) & 31.48\({}^{*}\) & 34.65\({}^{*}\) & 23.39\({}^{*}\) & 0 & 1.0x \\ \hline \multirow{4}{*}{Iterative Models} & Diff-LM Li et al. (2022) & 20 & 17.41 & 19.69 & – & - & -10.78\({}^{\circ}\) & 0.6x \\  & CDCD Dickeman et al. (2022) & 100 & 20.0 & 26.0 & – & - & -6.33\({}^{\circ}\) & – \\  & Diffformer Gao et al. (2022) & 20 & 23.80 & – & - & - & -3.38\({}^{\circ}\) & – \\  & DiNotSE Chen et al. (2023) & 20 & 24.26 & 29.05 & - & - & -2.68\({}^{\circ}\) & – \\  & SUNDAE Savivov et al. (2022) & 10 & 25.99 & 30.24 & - & - & -1.36\({}^{\circ}\) & 2.2x \\  & DAT+literative refinement\({}^{\dagger}\) & 3 & 25.67\({}^{*}\) & 30.85\({}^{*}\) & – & - & -1.07\({}^{\circ}\) & 7.2x \\ \hline \multirow{4}{*}{Non-iterative Models} & CTC Alnevidsky and Heller (2018) & 1 & 17.73\({}^{*}\) & 21.48\({}^{*}\) & 25.77\({}^{*}\) & 12.33\({}^{*}\) & -9.85 & 14.3x \\  & GLATT-CTC Qian et al. (2021) & 1 & 24.85\({}^{*}\) & 28.37\({}^{*}\) & 30.20\({}^{*}\) & 17.57\({}^{*}\) & -3.92 & 14.3x \\  & DAT T\({}^{\dagger}\) Huang et al. (2022) & 1 & 26.47\({}^{*}\) & 30.22\({}^{*}\) & 33.27\({}^{*}\) & 23.21\({}^{*}\) & -0.88 & 13.0x \\  & DAT+DSLP \({}^{\dagger}\) & 1 & 26.08\({}^{*}\) & 30.34\({}^{*}\) & - & - & -1.12\({}^{\circ}\) & 12.6x \\ \hline Ours & DiffGalAT \({}^{\dagger}\) & 1 & 26.72 & 31.35 & 34.12 & 23.74 & -0.19 & 13.0x \\  & DiffGLTAT \({}^{\dagger}\) & 3 & 27.91 & 31.55 & 35.09 & 24.02 & +0.47 & 7.2x \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results on WMT14 En\(\leftrightarrow\)De and WMT17 Zh\(\leftrightarrow\)En. The average gap is computed against our Transformer implementation. * represents the results are obtained from our re-implementation, and \(\diamondsuit\) indicates that the average gap is only computed with available results. For models with \(\dagger\), we use the Joint-Viterbi decoding proposed by Shao et al. (2022) for inference. The average gap is computed against the results of our implemented Transformer _base_.

 

The training of DiffGal with iterative decoding takes about 58 hours on 16 NVIDIA A100-80G GPUs. We average the best 5 checkpoints for BLEU scores on the validation set to get the final model. All the models are built with 6 Transformer decoder layers, and we use 3 decoder layers for each transition \(p_{\theta}(\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{x})\). Thus, each decoding iterations performs 2 denoising diffusion steps. For paraphrasing and image captioning, we use smaller architecture and shorter training steps. The detailed hyper-parameters can be found in Appendix E.

### Main Results

From the results in Table 3, we can find that DiffGal achieves considerable improvement over strong baselines. With a process that gradually adds modalities in the denoising pass, DiffGlat reduces the number of modalities to learn for each transition, enhancing the ability for capturing modalities in data. Depending on whether the model parameters are reused iteratively, /method can be trained for non-iterative or iterative decoding. For the non-iterative setting, DiffGlat only modifies the training procedure, thus can keep the inference the same as the process of the base model. Compared with several strong baselines, we highlight the advantages of DiffGlat:

* Our method can achieve better generation quality than strong NAR or even AR. With only one iteration of parallel decoding, our model achives higher BLEU scores than that of previous non-autoregressive models. And when applying iterative decoding, our model can even outperform the Transformer with a margin of 0.47 BLEU on average. For comparison, directly applying iterative refinement to DAT does not improve the performance.
* DiffGlat overcomes the slow sampling of diffusion models and achieves high decoding efficiency. DiffGlat completely keeps the fast speed of parallel decoding for the non-iterative setting, and can still achieve a \(7.2\times\) speedup with 3 decoding iterations.
* We can also use DiffGlat without DAT or combine DiffGlat with CTC, where our approach achieves more improvements over the baselines. The results in Figure 5 show that our method improves 1.5\(\sim\)4 BLEU over GLAT. The CTC results are shown in Appendix C.

**Results on 10 Machine Translation Benchmarks** To evaluate the performance more comprehensively, we compare the autoregressive Transformer with DiffGlat on 10 Benchmarks, and list the results in Table 4. We conduct experiments for both the _base_ and the _big_ setting. Our results show that DiffGlat outperforms or achieves comparable results to the Transformer on the 10 benchmarks.

#### Paraphrasing and Image Captioning

Besides machine translation, we also conduct experiments for two other text generation tasks: paraphrase generation and image captioning. For paraphrase generation, DiffGlat can outperform the Transformer with about 1.7 BLEU scores and 2 ROUGE scores. As for image captioning, DiffGlat greatly improves over the DAT baseline and achieves slightly better results on the 4 metrics compared with the Transformer. Results on paraphrasing and image captioning shows that DiffGlat generalizes well for text generation tasks.

### Ablation Study

Comparison of Different Diffusion ProcessesWe substitute the modality process in Section 3.1

Figure 5: Results with GLAT



to compare the performance of different diffusion processes. For comparison, we use two typical discrete diffusion processes proposed in Austin et al. (2021): D3PM absorbing and D3PM uniform. In each forward step, the D3PM absorbing process masks each token of \({\mathbf{y}_{t-1}}\) with some given probabilities \(\beta_{t}\), while the D3PM uniform process substitutes tokens in \({\mathbf{y}_{t-1}}\) with any other tokens uniformly. In our experiment, \(\beta_{t}\) is set to \(1/(T-t+1)\). From the results in Table 6, we can find that our modality gains improvement over the absorbing and uniform process.

Effectiveness of Residual Glancing TrainingTo verify the effectiveness of the residual glancing, we remove our proposed modification for the glancing training for comparison. The results are shown in Figure 7, note that the result of DiffStat without RGS uses the original glancing training. We can find that removing residual glancing causes a performance decline, indicating that our residual glancing improves the original glancing training.

### Analysis

Progressive Modality CapturingTo measure the performance of modality capturing, we compute the modality coverage percentage under the corresponding thresholds, and the coverage curve is presented in Figure 8. Since the exact modalities for real world data is unavailable, we compute the coverage for the data point \(({\mathbf{x}},{\mathbf{y}})\) as the modality is covered when the related data points are covered. Specifically, we compute the normalized loss for each data point with the trained model: \(L_{\theta}({\mathbf{x}},{\mathbf{y}})=-\log p_{\theta}({\mathbf{y}}|{\mathbf{x}})/|{\mathbf{y}}|\). And the data point is considered covered by the model if \(L_{\theta}({\mathbf{x}},{\mathbf{y}})\leq\tau\), where \(\tau\in\mathbb{R}\) is the threshold. For models with DAT, we compute \(L_{\theta}({\mathbf{x}},{\mathbf{y}})\) with the best aligned path \({\mathbf{a}^{*}}=\arg\max_{{\mathbf{a}}\in\Gamma}P({\mathbf{y}},{\mathbf{a}}|{\mathbf{x}})\). As depicted in Figure 8, the value of \(\tau\) ranges from 0 to 10, and each point represents the percentage of data points with \(L_{\theta}({\mathbf{x}},{\mathbf{y}})\) less than or equal to \(\tau\).

In the figure, we can find that the curves of DiffStat is overall on top of the DAT curve, indicating our model captures more modalities. And the three curves of DiffStat shows that DiffStat captures more modalities as the model performs more denoising steps.

Analysis for Decoding Iterations and LayersWe also conduct experiments to analyze the performance with different decoding iterations and the effect of layer numbers for each denoising transition. The results are provided in Appendix F.

## 5 Related Work

The work for non-autoregressive neural machine translation can be divided into non-iterative and iterative methods. Previous non-iterative methods attempt to enhance the generation quality by earning from autoregressive models Wei et al. (2019); Guo et al. (2020), incorporating light sequential decoding Sun et al. (2019); Huang et al. (2022b), and introducing latent variables Bao et al. (2019); Ma et al. (2019); Song et al. (2021); Bao et al. (2022). Besides, models with iterative decoding have also been developed Lee et al. (2018); Ghazvininejad et al. (2019); Gu et al. (2019); Huang et al. (2022a); Saharia et al. (2020); Savinov et al. (2022); Huang et al. (2022c).

DiffGalot also has connections with diffusion models for discrete data but differs from them in both probabilistic modelling and design motivation. More discussions can be found in Appendix I.

## 6 Conclusion

In this work, we propose DiffStat, a parallel sequence generation model trained with the modality diffusion process and residual glancing sampling. By smoothing the learning of modalities in the diffusion model framework, DiffGlat greatly improves the generation quality of parallel generation. Compared with the autoregressive Transformer, DiffGlat achieves superior performance in both accuracy and efficiency for multiple se 

quence generation tasks, demonstrating the potential of the parallel generation paradigm.

## Acknowledgements

This work is supported by the National Key R&D Program of China (2022ZD0160501), Natural Science Foundation of China (62376133). Hao Zhou is the corresponding author of this paper.

## References

* Austin et al. [2021] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rainne van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. In _Advances in Neural Information Processing Systems_, volume 34, pages 17981-17993. Curran Associates, Inc.
* Banerjee and Lavie [2005] Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_, pages 65-72.
* Bao et al. [2019] Yu Bao, Hao Zhou, Jiangtao Feng, Mingxuan Wang, Shujian Huang, Jiajun Chen, and Lei Li. 2019. Non-autoregressive transformer by position learning. _CoRR_, abs/1911.10677.
* Bao et al. [2022] Yu Bao, Hao Zhou, Shujian Huang, Dongqi Wang, Lihua Qian, Xinyu Dai, Jiajun Chen, and Lei Li. 2022. latent-GLAT: Glancing at latent variables for parallel text generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8398-8409, Dublin, Ireland. Association for Computational Linguistics.
* Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Torn Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems_, volume 33, pages 1877-1901. Curran Associates, Inc.
* Chen et al. [2022] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. 2022. Analog bits: Generating discrete data using diffusion models with self-conditioning. _arXiv preprint arXiv:2208.04202_.
* Dieleman et al. [2022]Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. 2022. Continuous diffusion for categorical data. _arXiv preprint arXiv:2211.15089_.
* Du et al. [2021] Cunxiao Du, Zhaopeng Tu, and Jing Jiang. 2021. Order-agnostic cross entropy for non-autoregressive machine translation. In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_, volume 139 of _Proceedings of Machine Learning Research_, pages 2849-2859. PMLR.
* Gao et al. [2022] Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. 2022. Difformer: Empowering diffusion model on embedding space for text generation. _arXiv preprint arXiv:2212.09412_.
* Ghazvininejad et al. [2020a] Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020a. Aligned cross entropy for non-autoregressive machine translation. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 3515-3523. PMLR.
* Ghazvininejad et al. [2019] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 6111-6120. Association for Computational Linguistics.
* Ghazvininejad et al. [2020b] Marjan Ghazvininejad, Omer Levy, and Luke Zettlemoyer. 2020b. Semi-autoregressive training improves mask-predict decoding. _CoRR_, abs/2001.08785.
* Gong et al. [2022] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. 2022. Diffuseq: Sequence to sequence text generation with diffusion models. _arXiv preprint arXiv:2210.08933_.
* Graves et al. [2006] Alex Graves, Santiago Fernandez, Faustino J. Gomez, and Jurgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In _Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), Pittsburgh, Pennsylvania, USA, June 25-29, 2006_, volume 148 of _ACM International Conference Proceeding Series_, pages 369-376. ACM.
* May 3, 2018, Conference Track Proceedings_. OpenReview.net.
* Gu et al. [2019] Jiatao Gu, Changnan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 

2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 11179-11189.
* Guo et al. (2020) Junliang Guo, Xu Tan, Linil Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2020. Fine-tuning by curriculum learning for non-autoregressive neural machine translation. In _Proceedings of the aaai conference on artificial intelligence_, volume 34, pages 7839-7846.
* He et al. (2022) Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. 2022. Diffusionbert: Improving generative masked language models with diffusion models. _arXiv preprint arXiv:2211.15029_.
* Helcl et al. (2022) Jindrich Helcl, Barry Haddow, and Alexandra Birch. 2022. Non-autoregressive machine translation: It's not as fast as it seems. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1780-1790, Seattle, United States. Association for Computational Linguistics.
* Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In _Advances in Neural Information Processing Systems_, volume 33, pages 6840-6851. Curran Associates, Inc.
* Hoogeboom et al. (2021) Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forre, and Max Welling. 2021. Argmax flows and multinomial diffusion: Learning categorical distributions. In _Advances in Neural Information Processing Systems_, volume 34, pages 12454-12465. Curran Associates, Inc.
* Huang et al. (2022a) Chenyang Huang, Hao Zhou, Osmar R. Zaiane, Lili Mou, and Lei Li. 2022a. Non-autoregressive translation with layer-wise prediction and deep supervision. _The Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022_.
* Huang et al. (2022b) Fei Huang, Hao Zhou, Yang Liu, Hang Li, and Minlie Huang. 2022b. Directed acyclic transformer for non-autoregressive machine translation. In _Proceedings of the 39th International Conference on Machine Learning_, volume 162 of _Proceedings of Machine Learning Research_, pages 9410-9428. PMLR.
* Huang et al. (2022c) Xiao Shi Huang, Felipe Perez, and Maksims Volkovs. 2022c. Improving non-autoregressive translation models without distillation. In _International Conference on Learning Representations_.
* Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 3128-3137.
* Kasai et al. (2020) Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. 2020. Non-autoregressive machine translation with disentangled context transformer. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 5144-5155. PMLR.
* Kasai et al. (2021) Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. 2021. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_, pages 1317-1327. The Association for Computational Linguistics.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _ICLR (Poster)_.
* November 4, 2018_, pages 1173-1182. Association for Computational Linguistics.
* Li et al. (2022) Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. 2022. Diffusion-lm improves controllable text generation. _arXiv preprint arXiv:2205.14217_.
* November 4, 2018_, pages 3016-3021. Association for Computational Linguistics.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In _European conference on computer vision_, pages 740-755. Springer.
* Ma et al. (2019) Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard H. Hovy. 2019. Flowseq: Non-autoregressive conditional sequence generation with generative flow. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 4281-4291. Association for Computational Linguistics.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for 

sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Demonstrations, pp. 48-53. External Links: Link Cited by: SS6.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. External Links: ISBN 978-1-4503-3213-X, Document, Link Cited by: SS6.
* M. Popovic (2015)chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisbon, Portugal, pp. 392-395. External Links: Link Cited by: SS6.
* M. Post (2018)A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, Belgium, Brussels, pp. 186-191. External Links: Link Cited by: SS6.
* L. Qian, H. Zhou, Y. Bao, M. Wang, L. Qiu, W. Zhang, Y. Yu, and E. Li (2021)Glancing transformer for non-autoregressive neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, Volume 1: Long Papers, Virtual Event, August 1-6, 2021, pp. 1993-2003. External Links: Link Cited by: SS6.
* R. Rei, C. Stewart, A. C. Farinha, and A. Lavie (2020)COMET: a neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 2685-2702. External Links: Link, Document Cited by: SS6.
* C. Saharia, W. Chan, S. Saxena, and M. Norouzi (2020)Non-autoregressive machine translation with latent alignments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 1098-1108. External Links: Link, Document Cited by: SS6.
* N. Savinov, J. Chung, M. Binkowski, E. Elsen, and A. van den Oord (2022)Stepen-unrolled denoising autoencoders for text generation. In International Conference on Learning Representations, External Links: Link Cited by: SS6.
* R. Sennrich, B. Haddow, and A. Birch (2016)Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers, D. Besson, G. Busso, P. de la Cruz, M. De Roeck, and J. Leskovec (Eds.), Berlin, Heidelberg, pp. 1-14. External Links: ISBN 978-3- 

Chunting Zhou, Jiatao Gu, and Graham Neubig. 2020. Understanding knowledge distillation in non-autoregressive machine translation. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.

 

Settings of the Synthetic Experiment

DataIn order to investigate the effect of modalities in learning, we create source-target pairs with known modalities for training. Suppose we have 4 modalities in the synthetic data, the source sequences can be transformed into target sequences with the rule of modality. We generate source sequences of length 32 with numbers uniformly chosen from \(1\sim 5000\), and randomly choose only one modality for each source sequence to create the target sequences. In total, we generate 100000 sequence pairs for training, and 5000 pairs each for validation and test.

Model SetupWe train AR and NAR models on the synthetic data, with all models built with 4 encoder layers and 4 decoder layers. To learn all the modalities gradually, we train a NAR model with a modality growing process. Specifically, besides the original NAR training loss that learns 4 modalities, we also train the NAR model to learn 2 modalities with the middle decoder layer. To capture only 2 modalities in the middle, we merge modalities by transforming the targets of modality I to modality II and the targets of modality of III to modality IV. With the merged source-target pairs with only 2 modalities, we train the middle decoder layer to fit modality II and IV.

EvaluationFor output quality measurement, we obtain the model outputs \(\hat{\mathbf{y}}\), and compare them with the closest targets \(\mathbf{y}\) in the 4 modalities for all the outputs. The number accuracy is the percentage of \(\hat{y}_{i}=y_{i}\) and the sequence accuracy is the percentage of \(\hat{\mathbf{y}}=\mathbf{y}\). To visualize the distribution of different modalities, we use the modality of the closest target as that of the output, and report the proportion for different modalities.

## Appendix B Proof for Definition Consistency of the Modality Diffusion Process

Lemma 1With \(q(\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{y}_{0})\) defined in the modality diffusion process \(q^{\text{MDP}}\), for \(t\geq 1\), we have:

\[q(\mathbf{y}_{t}|\mathbf{y}_{0})=(\gamma_{t}\mathcal{P}_{t-1}+(1-\gamma_{t})\mathcal{ P}_{t-1}\odot\mathbf{y}_{0})/Z_{t} \tag{15}\]

ProofAccording to the definition in Section 3.1, we have

\[q(\mathbf{y}_{t}|\mathbf{y}_{0}):=(\gamma_{t}\mathcal{P}_{t-1}+(1-\gamma_{t})\mathcal{ P}_{t-1}\odot\mathbf{y}_{0})/Z_{t} \tag{16}\]

\[q(\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{y}_{0}):=\gamma_{t-1}\mathcal{P}_{ t-2}+((1-\gamma_{t-1})\mathcal{P}_{t-2} \tag{17}\] \[~{}~{}~{}-\omega_{t}\mathcal{P}_{t-1})\odot\mathbf{y}_{0}+\omega_{t} Z_{t}\mathbf{y}_{0}\odot\mathbf{y}_{t}\]

Since our models predcit the tokens in the sequence independently, the posterior \(q(\mathbf{y}_{t}|\mathbf{y}_{0})\) can also be decomposed into the form of independent token distributions:

\[q(\mathbf{y}_{t}|\mathbf{y}_{0}) =\prod_{i=1}^{n}q(y_{t}^{i}|\mathbf{y}_{0}) where \tag{18}\] \[q(y_{t}^{i}|\mathbf{y}_{0}) =\begin{cases}\gamma_{t}\mathcal{P}_{t-1}^{i}/Z_{t}&y_{t}^{i}\neq y _{0}^{i}\\ \mathcal{P}_{t-1}^{i}/Z_{t}&y_{t}^{i}=y_{0}^{i}\end{cases}\]

Here, \(y_{0}^{i}\) is the \(i\)th token of \(\mathbf{y}_{0}\), and \(\mathcal{P}_{t-1}^{i}\) is the \(i\)th token output distribution of \(\mathcal{P}_{t-1}\). In the same way, we can rewrite \(q(y_{t-1}^{i}|\mathbf{y}_{t},\mathbf{y}_{0})\) as:

\[\begin{cases}\gamma_{t-1}\mathcal{P}_{t-2}^{i}&y_{t-1}^{i}\neq y_{0}^{i}\\ \mathcal{P}_{t-2}^{i}-\omega_{t}(\mathcal{P}_{t-1}^{i}-Z_{t}\mathds{1}(y_{0}^ {i}==y_{t}^{i}))&y_{t-1}^{i}=y_{0}^{i}\end{cases} \tag{19}\]

For any \(t\geq 2\), we can compute \(q(\mathbf{y}_{t-1}|\mathbf{y}_{0})\) by:

\[q(\mathbf{y}_{t-1}|\mathbf{y}_{0})= \int_{\mathbf{y}_{t}}q(\mathbf{y}_{t}|\mathbf{y}_{0})q(\mathbf{y}_{t-1}|\mathbf{y}_{t },\mathbf{y}_{0})\mathrm{d}\mathbf{y}_{t} \tag{20}\]

Thus,

\[q(y_{t-1}^{i}=y_{0}^{i}|\mathbf{y}_{0}) \tag{21}\] \[= (1-q(y_{t}^{i}=y_{0}^{i}|\mathbf{y}_{0}))q(y_{t-1}^{i}|y_{t}^{i}\neq y _{0}^{i},\mathbf{y}_{0})\] \[+q(y_{t}^{i}=y_{0}^{i}|\mathbf{y}_{0})q(y_{t-1}^{i}|y_{t}^{i}=y_{0}^{ i},\mathbf{y}_{0})\] \[= (1-\mathcal{P}_{t-1}^{i}/Z_{t})(\mathcal{P}_{t-2}^{i}-\omega_{t} \mathcal{P}_{t-1}^{i})\] \[+(\mathcal{P}_{t-1}^{i}/Z_{t})(\mathcal{P}_{t-2}^{i}-\omega_{t}( \mathcal{P}_{t-1}^{i}-Z_{t}))\] \[= \mathcal{P}_{t-2}^{i}-\omega_{t}\mathcal{P}_{t-1}^{i}+\mathcal{P} _{t-1}^{i}/Z_{t}\cdot\omega_{t}Z_{t}\] \[= \mathcal{P}_{t-2}^{i}\]

And from Eq.19, we can derive:

\[q(y_{t-1}^{i}|\mathbf{y}_{0})=\gamma_{t-1}\mathcal{P}_{t-2}^{i}\quad\text{for $y_{t-1}^{i}\neq y_{0}^{i}$} \tag{22}\]

Therefore, after normalization, we have:

\[q(\mathbf{y}_{t-1}|\mathbf{y}_{0}) =\prod_{i=1}^{n}q(y_{t-1}^{i}|\mathbf{y}_{0})\quad where \tag{23}\] \[q(y_{t-1}^{i}|\mathbf{y}_{0}) =\begin{cases}\gamma_{t}\mathcal{P}_{t-2}^{i}/Z_{t-1}&y_{t-1}^{i} \neq y_{0}^{i}\\ \mathcal{P}_{t-2}^{i}/Z_{t-1}&y_{t-1}^{i}=y_{0}^{i}\end{cases}\]

Similarly, we can prove that Eq.16 holds for \(t\geq 1\).

## Appendix C Additional Results on Machine Translation

For reference and further comparison, we also report the tokenized BLEU scores and the results based on CTC Graves et al. (2006) rather DAT Huang et al. (2022b) 

Results with Tokenized BLEUAs some previous work reports tokenized BLEU scores on the machine translation benchmarks, we also provide the results for tokenizer BLEU scores for direct comparison.

Combination with CTCWe also conduct experiments for DiffGalt with CTC, and the results are presented in Table 8.

The experimental results show that DiffGlat achieves improvements of 1\(\sim\)3 BLEU scores over GLAT+CTC, demonstrating the effectiveness of DiffGlat. DiffGlat can easily combine with various existing methods for parallel generation because DiffGlat maintains the decoding process or simply adds more iterations. Specifically, DiffGlat keeps the original inference process in the non-iterative setting or forwards the decoder multiple times without intermediate decoding in the iterative setting.

## Appendix D Inference Time Comparison

To provide a more comprehensive comparison of the decoding speedup, as discussed in Helcl et al. (2022), we measure the inference latency on the WMT14 test set with 1 Nvidia-V100 GPU, and report the inference latency with batch size 1 in Table 9. Following the setting in Kasai et al. (2021), we also measure the inference latency of a Transformer with 12 encoder layers and 1 decoder layer.

Comparing with the Transformer (12-1), DiffStat with 3 decoding iterations still has a \(2.6\times\) speedup. Although the Transformer with deep encoder and shallow decoder can achieve faster inference, the depth of decoder is important for capability of decoder-only models Brown et al. (2020); OpenAI (2023). Thus, the comparison for models with 6 decoder layers is also useful.

## Appendix E Hyper-parameters for Experiments

We train our models with fairseq Ott et al. (2019)4. For machine translation, the dropout is set to 0.1 except En-Ro/Ro-En and the Transformer _big_ setting, where the dropout is 0.3. For paraphrasing and image captioning, we use the dropout of 0.3. Since our modality diffusion process requires the model to capture part of the modality, for the first 100k steps, we train the model to predict the target \({\mathbf{y}}_{0}\) at all steps \(t\). For the subsequent training transitions, we train the model with the modality diffusion process. For DAT, the decoding length is set to be 8 times the source length for non-iterative models and 4 times for iterative models.

Footnote 4: [https://github.com/facebookresearch/fairseq](https://github.com/facebookresearch/fairseq)

For the hyper-parameters of the modality diffusion in Eq. 9, we use a simplified implementation with the interpolation between \(\mathcal{P}_{t-2}\) and \({\mathbf{y}}_{0}\):

\[q({\mathbf{y}}_{t-1}|{\mathbf{y}}_{t},{\mathbf{y}}_{0})=\gamma_{t-1}\mathcal{P}_{t-2}+(1- \gamma_{t-1})\mathcal{P}_{t-2}\odot{\mathbf{y}}_{0} \tag{24}\]

Here, we set \(\gamma_{t-1}\in(0,1]\) to be selected from a pre-defined set of values. To achieve a similar effect for interpolating with \({\mathbf{y}}_{0}\odot{\mathbf{y}}_{t}\), we choose \(\gamma_{t-1}\) as the maximum value that preserves 90% of the tokens in \({\mathbf{y}}_{0}\odot{\mathbf{y}}_{t}\). With such interpolation, \({\mathbf{y}}_{t-1}\) includes most of the target tokens that is correctly predicted in the previous \({\mathbf{y}}_{t}\). Thus, the difference between \({\mathbf{y}}_{t}\) and \({\mathbf{y}}_{0}\) gradually reduces as \(t\) decreases.

In terms of residual glancing training, we set the hyper-parameter of glancing schedule \(\mu=1/T\). And the hyper-parameter for computing sampling numbers \(\alpha\) decrease from 0.5 to 0.1 periodically for iterative DiffStat.

For paraphrase generation, we utilize a Transformer architecture with 4 encoder layers and 4 decoder layers, with a hidden dimension of 256. For image captioning, we use the image features extracted by fast R-CNN 5 with a Transformer _base_ setting. The maximum training steps are set to 100k for both paraphrase generation and image captioning.

Footnote 5: [https://github.com/peteanderson80/bottom-up-attention](https://github.com/peteanderson80/bottom-up-attention)

## Appendix F Additional Analysis

To further study our approach, we analyze the impact of the number of decoding iterations, the number of layers for one reverse transition, and the total diffusion steps in training.

Performance with Iterative DecodingThe BLEU scores for different decoding iterations are presented in Figure 8. Here, the evaluated models are trained with generative processes of 3 iterations. Our findings indicate that the BLEU scores increases as the number of iterations increases to 3, and the scores do not increase when the iterations exceeds 3. Note that the decoding iterations in inference can be different from that in training. Since the model is trained using a generation process of 3 iterations, it also achieves the best scores 

with 3 decoding iterations, which is consistent with training.

The Number of Decoder Layers for each TransitionWe use 3 decoder layers to model one transition \(p_{\theta}(\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{x})\). Thus, one decoding iteration of the 6-layers decoder performs 2 denoising transitions. Without changing the total number of the decoder layers and iterative decoding, we conduct experiments to study the influence of the number of layers for one transition. The results in Figure. 9 show that the model achieves the best performance when using 2 or 3 layers for one transition. But the scores with only 1 layer for 1 transition decreases, which may caused by insufficient modelling capacity with only 1 layer.

The Effect of Diffusion Process LengthsWe also investigate the effect of diffusion process steps \(T\) and the results are illustrated in Figure 10. Note that we perform each reverse denoising step with 3 decoder layers, so every iteration corresponds to 2 steps in the diffusion process. We find that the performance grows on WMT14 datasets until \(T\) reaches 6, and declines when the \(T\) is 8. We think the reason why the performance stops growing with the increasing iterations is that large iterations makes part of the denoising transitions too easy to learn. The easy learning task can lead to capability degradation, as the issues caused by adding

Figure 8: The BLEU score curves with iterative decoding.

Figure 10: The effect of different diffusion process lengths.

\begin{table}
\begin{tabular}{l l|c c c c|c|c|c} \hline \hline  & Iter & **WMT14** & **WMT14** & **WMTT17** & **WMTT17** & **Average** \\  & & En-De & **De-En** & **En-Zh** & **Zn-En** & **Gap** & **Speedup** \\ \hline \multirow{2}{*}{AR} & Transformer [5] & \(M\) & 27.6 & 31.4 & 34.3 & 23.7 & -0.35 & 1.0x \\  & Transformer _base_ (Ours) & \(M\) & 27.81\% & 31.96\% & 34.65\% & 23.98\% & -0 & 1.0x \\ \hline \multirow{2}{*}{Iterative Models} & CMLLM [5] & 24.61 & 29.40 & - & - & -2.88\% & 2.2x \\  & Imputer [1] & 8 & 25.0 & - & - & -2.96\% & 2.7x \\  & SUNDAE [1] & 10 & 26.25 & 30.80 & - & - & -1.36\% & 2.2x \\  & CMLLM [5] & 10 & 26.40 & 30.92 & - & -1.23\% & 1.7x \\  & DNINGSE [10] & 20 & 24.48 & 29.40 & - & -2.68\% & - \\ \hline \multirow{2}{*}{Non-iterative Models} & CTC [1] & 18.42 & 23.65 & 26.84 & 12.23 & -9.31 & 14.3x \\  & QaZE [12] & 1 & 22.4 & 26.8 & - & -5.28\% & 14.2x \\  & GLAT+CTC [14] & 1 & 25.02 & 29.14 & 30.65 & 19.92 & -3.42 & 14.3x \\  & DAT\({}^{\dagger}\)[14] & Huang et al., 2022b & 1 & 26.95\% & 30.73\% & 33.27\% & 23.60\% & -0.96 & 13.0x \\ \hline \multirow{2}{*}{Ours} & DiffGLAT \({}^{\dagger}\) & 1 & 27.40 & 31.94 & 34.12 & 24.23 & -0.18 & 13.0x \\  & DiffGLTAT \({}^{\dagger}\) & 3 & 28.57 & 32.08 & 35.09 & 24.86 & +0.55 & 7.2x \\ \hline \hline \end{tabular}
\end{table}
Table 7: The tokenized BLEU scores on WMT14 En\(\leftrightarrow\)De and WMT17 Zh\(\leftrightarrow\)En. The average gap is computed against our Transformer implementation. * represents the results are obtained from our re-implementation, and \(\diamondsuit\) indicates that the average gap is only computed with available results. For models with \(\dagger\), we use the Joint-Viterbi decoding proposed by Shao et al. (2022) for inference. The average gap is computed against the results of our implemented Transformer _base_.

 

small noises in diffusion models for text (Li et al., 2022).

## Appendix G The chrF and COMET Scores

Besides the commonly used BLEU Papineni et al. (2002) metric, we compute the additional chrF Popovic (2015) and COMET Rei et al. (2020) scores for more comprehensive evaluation. We use the wmt22-comet-da for computing the COMET score.

DiffGlat achieves better chrF scores on WMT17 En\(\leftrightarrow\)Zh and comparable scores on WMT14 En\(\leftrightarrow\)De compared to the Transformer. And DiffGlat still outperforms DAT in terms of COMET but falls behind the Transformer baseline. The gap in COMET may caused by the distribution mismatch between the NAR outputs and the training data for COMET.

## Appendix H Data Statistics and Evaluation Metrics

The statistics of the data we used in experiments are listed as follow: WMT14 En\(\leftrightarrow\)De (4.5m), WMT17 En\(\leftrightarrow\)Zh (20m), WMT14 En\(\leftrightarrow\)fr (35m), WMT16 En\(\leftrightarrow\)Ro (0.6m) and WMT13 En\(\leftrightarrow\)Es(12m). For paraphrase generation and image caption, we use the Quora (145k) dataset and MS-COCO (113k) dataset respectively. For MS-COCO, we use the Karpathy split Karpathy and Fei-Fei (2015).

For paraphrase, we report tokenized BLEU and ROUGE-L Lin (2004)6. For image caption, we also report METEOR Banerjee and Lavie (2005) and CIDEr Vedantam et al. (2015), and use the official evaluation tools for evaluation7.

Footnote 6: The script is ROUGE-1.5.5.pl

Footnote 7: [https://github.com/cocodataset/](https://github.com/cocodataset/) cocoapi

## Appendix I Connections with Diffusion Models and Iterative NAR

Diffusion Models for Discrete DataPrevious work studies continuous or discrete process for modelling discrete data with diffusion models. For continuous diffusion processes, a series of work explores adding Gaussian noise in the word embedding space Li et al. (2022), converting the discrete data to 0/1 bits Chen et al. (2022), the design space of diffusion modelsDieleman et al. (2022), and conditional text generation with continuous diffusion Gong et al. (2022). For discrete diffusion processes, Hoogeboom et al. (2021) study a multinomial diffusion process where each state transits to other states uniformly, while Austin et al. (2021) explore more types of state transitions, including masking and increasing the transition frequency for similar states. He et al. (2022) propose a noise schedule based on token information. Zheng et al. (2023) derive a discrete diffusion framework with route mechanism via reparameterization.

Discussion for comparison with Step-UnrolledModels and Semi-Autoregressive Training Savinov et al. (2022) introduces a denoising procedure related to diffusion models but different from ours. Specifically, SUNDAE corrupts the target and uses the corrupted sequence as the input for learning denoising. In contrast, DiffGal employs a diffusion process for dividing modalities, and uses the corrupted sequence as the intermediate training target. Besides, SUNDAE heavily relies on multiple decoding iterations, while our method can achieve competitive quality without iterative decoding.

Compared with SMART Ghazvininejad et al. (2020), our method learns to predict the intermediate targets rather than replacing decoder inputs. For SMART, the model first generates outputs with several iterations and uses the outputs as the decoding inputs to learn mistake correction. In contrast, DIFFGLAT uses the intermediate targets to capture the multi-modality in data gradually. Although both methods can be used for iterative refinement, they work on input and target, respectively

## Appendix J Limitations

This work does not include large-scale pre-training experiments, which cannot be directly compared with the large autoregressive GPTs. The aim of this work is to establish the foundational framework

\begin{table}
\begin{tabular}{l|r r r r} \hline  & WMT14 En\(\leftrightarrow\)De & WMT14 De\(\leftrightarrow\)En & WMT17 En\(\leftrightarrow\)Zh & WMT17 Ze\(\leftrightarrow\)Ze \\ \hline Transformer & 0.8603 & 0.9832 & 0.3078 & 0.5230 \\ DAT & 0.9621 & 0.5669 & 0.2995 & 0.5900 \\ DWTGLAT & 0.5790 & 0.5815 & 0.3136 & 0.5310 \\ \hline \end{tabular}
\end{table}
Table 10: The chrF scores on WMT14 En\(\leftrightarrow\)De and WMT17 En\(\leftrightarrow\)Zh 

and potential of non-autoregressive text generation models, leaving the more detailed and expansive comparative study as a future endeavor.