SlowGuess/NeurIPS_2023_OCR huggingface dataset:
AR-DIFFUSION: Auto-Regressive Diffusion Model for Text Generation
Tong Wu $^{1*†}$ , Zhihao Fan $^{2*†}$ , Xiao Liu $^{3}$ , Hai-Tao Zheng $^{1,8‡}$ , Yeyun Gong $^{3‡}$ , Yelong Shen $^{4}$ , Jian Jiao $^{5}$ , Juntao Li $^{6}$ , Zhongyu Wei $^{2}$ , Jian Guo $^{7‡}$ , Nan Duan $^{3†}$ , Weizhu Chen $^{4‡}$
<sup>1</sup> Shezhen International Graduate School, Tsinghua University, <sup>2</sup> Fudan University,
<sup>3</sup>Microsoft Research Asia, <sup>4</sup>Microsoft Azure AI, <sup>5</sup>Microsoft,
$^{6}$ Soochow University,  $^{7}$ IDEA Research,  $^{8}$ Pengcheng Laboratory
{yegong, yeshe, nanduan, wzchen}@microsoft.com,
zheng.haitao@sz.tsinghua.edu.cn, guojian@idea.edu.cn
Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing language models are trained with a left-to-right auto-regressive approach. To account for the inherent sequential characteristic of natural language, we introduce Auto-Regressive Diffusion (AR-DIFFUSION). AR-DIFFUSION ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right. In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-DIFFUSION clearly demonstrated its superiority over existing diffusion language models and that it can be  $100 \times \sim 600 \times$  faster when achieving comparable results. Our code is available at this https URL.
Text generation is a fundamental task within the field of natural language processing (NLP). Pre-trained language models like GPT-4 [OpenAI, 2023], LLaMA [Touvron et al., 2023], and Alpaca [Taori et al., 2023] have garnered significant attention with their ability to generate fluent and human-like textual content. These models utilize the auto-regressive (AR) Transformer decoders [Vaswani et al., 2017] to emit generated tokens one-by-one in sequential order from left to right. By leveraging the power of position dependency, AR models are able to enhance the naturalness, coherence, and adherence to human language conventions in the generated text [Brown et al., 2020].
Recent studies have shown the remarkable performance of diffusion models in image generation [Ho et al., 2020], motivating researchers to extend diffusion to text generation [Li et al., 2022a, Gong et al., 2022, Dieleman et al., 2022, Yuan et al., 2022, Ye et al., 2023]. By introducing timestep, these methods progressively regulate the interpolation between the original tokens and Gaussian noise, then iteratively denoise for text generation. At each timestep, the diffusion-based text generator predicts
*Work done during an internship at Microsoft Research Asia.
These authors contributed equally to this work.
Corresponding author.
Figure 1: Model behaviors illustrated on a two-dimensional coordinate system, where the horizontal axis stands for the position and the vertical axis represents the diffusion timestep. In the inference stage, different models will behave differently. (a) For the typical Diffusion-LM [Li et al., 2022a], each token share the identical movement speed  $v(n_{1}, t_{i}, t_{i+1}) = v(n_{2}, t_{i}, t_{i+1}) = |t_{i+1} - t_{i}|$ . (b) For AR from the perspective of diffusion models, the tokens have two states based on the degree of interpolation between the original tokens and Gaussian noise: to be decoded (at timestep  $t = T$ ) and already decoded (at timestep  $t = 0$ ). Specifically, we have  $v(n_{1}, t_{i}, t_{i+1}) = 0$  and  $v(n_{2}, t_{i}, t_{i+1}) = T$ . (c) In AR-DIFFUSION,  $(n_{e}, t_{e})$  is the coordinate of anchor point. Tokens in different positions exhibit varying movement speeds, such as  $v(n_{1}, t_{i}, t_{i+1}) > v(n_{2}, t_{i}, t_{i+1})$  when  $n_{1} < n_{2}$ .
all tokens simultaneously following Non-Auto-Regression (NAR) [Lewis et al., 2020, Qi et al., 2020, 2021, Li et al., 2022b], leading to faster decoding speed compared to AR. However, it also inherits the drawback of NAR, namely the sacrifice of inter-token position dependency [Li et al., 2022c] and the drop of generation performance [Bao et al., 2021].
To conduct a comprehensive analysis, we introduce a two-dimensional coordinate system to track the diffusion timestep of tokens  $f(\cdot)$  positioned at various locations. As illustrated in Figure 1, the system assigns the token position  $n \in [1, N]$  to the horizontal axis and the diffusion timestep  $t \in [0, T]$  to the vertical axis. Diffusion-LM [Li et al., 2022a], which is followed by existing diffusion-based text generation models, is shown in Figure 1(a). It assigns a uniform timestep  $t$  to all tokens. In contrast, tokens in the AR model depicted in Figure 1(b) exhibit distinct timesteps within a generation step  $(t_i)$ . For instance, the already decoded token at position  $n_1$  has a timestep of 0, while the to-be-decoded token at position  $n_2$  has a timestep of  $T$ . This approach effectively captures the sequential dependency. Motivated by this observation, we introduce AR-DIFFUSION, an auto-regressive diffusion method, for the disparity in token positions and the principle of sequential token identification.
In AR-DIFFUSION, we propose a multi-level diffusion strategy that includes both sentence-level and token-level diffusion. We randomly choose a sentence-level timestep  $t$ , and assign dynamic movement speeds  $v(\cdot)$  by determining position-sensitive token-level timestep  $f(n, t)$  for each token. This enables tokens at the left of a sentence to undergo faster movement from random Gaussian noise to token embedding, while those at the right of the sentence experience slower movement to better utilize information from previously denoised tokens. During inference, to reduce the significant number of inference steps (e.g., 2,000) required in Diffusion-LM [Li et al., 2022a], SeqDiffSeq [Yuan et al., 2022] and GENIE [Lin et al., 2023], we introduce a skipping mechanism that collaborates with the multi-level diffusion strategy to accelerate the process.
Experimental results across various text generation tasks, such as text summarization, machine translation, and common sense generation, have consistently demonstrated that AR-DIFFUSION surpasses existing text diffusion models, including AR methods in terms of both quality and diversity. Moreover, our verification reveals that AR-DIFFUSION requires fewer resources during decoding while maintaining superior performance. It achieves  $100 \times$  faster than SeqDiffSeq [Yuan et al., 2022] in machine translation and  $600 \times$  faster than GENIE [Lin et al., 2023] in text summarization while delivering comparable results. Furthermore, it demonstrates promising results even in a challenging scenario where decoding is limited to only two steps.
2.1 Conditional Generative Language Models
In the field of natural language generation, conditional generative models are commonly implemented using either auto-regressive (AR) or non-auto-regressive (NAR) methods. In AR [Vaswani et al., 2017], tokens on the right are predicted based on visible left tokens. The likelihood is given by  $p_{\mathrm{AR}}(\pmb {y}|\pmb {x}) = \prod_{i = 1}^{N}p(\pmb {y}_i|\pmb {y}_{1:i - 1};\pmb {x})$ , where  $y_{i}$  denotes the  $i$ -th token of  $\pmb{y}$ . On the other hand, NAR [Gu et al., 2017] assumes conditional independence among tokens and generates them uniformly without distinction during decoding, resulting in the likelihood  $p_{\mathrm{NAR}}(\pmb {y}|\pmb {x}) = \prod_{i = 1}^{N}p(\pmb {y}_i|\pmb {x})$ . This parallel generation approach is of lower quality compared to AR, although it offers a substantial speed advantage.
2.2 Diffusion Models for Text Generation
Recently, Li et al. [2022a] propose a natural language generation model based on the diffusion process, which is typically divided into a forward noising process and a reverse denoising process.
Specifically, the forward process is a fixed linear Gaussian model, which gradually perturbs the random variable  $z_0$  until it becomes the standard Gaussian distribution. This can be formalized as:
$$
q \left(\boldsymbol {z} _ {t} \mid \boldsymbol {z} _ {0}; \boldsymbol {x}\right) = \mathcal {N} \left(\boldsymbol {z} _ {t}; \sqrt {\bar {\alpha} _ {t}} \boldsymbol {z} _ {0}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right), \tag {1}
$$
where,  $\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$ , and  $\alpha_i$  is a coefficient that monotonically decreases with timestep  $t$ ,  $z_t$  is the latent state at timestep  $t$ .
The reverse process is to initiate from standard Gaussian noise and progressively utilize the denoising transition  $p_{\theta}(z_{t - 1} | z_t; \boldsymbol{x})$  for generation.
$$
p _ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t - 1} \mid \boldsymbol {z} _ {t}; \boldsymbol {x}\right) = \mathcal {N} \left(\boldsymbol {z} _ {t - 1}; \mu_ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t}, t; \boldsymbol {x}\right), \Sigma_ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t}, t; \boldsymbol {x}\right)\right), \tag {2}
$$
where the mean  $\mu_{\theta}$  and variance  $\Sigma_{\theta}$  are learned from the model. In particular, we follow Li et al. [2022a]'s approach of using predefined variance without trainable parameters.
To extend the continuous diffusion process to discrete text generation, Li et al. [2022a] introduce an additional Markov transition from the discrete tokens  $\pmb{y}$  to the latent variable  $\pmb{z}_0$ . In practice, we add an embedding step  $q_{\phi}(\pmb{z}_0|\pmb{y}) = \mathcal{N}(\pmb{z}_0; \mathrm{Emb}(\pmb{y}), (1 - \alpha_0)\mathbf{I})$  in the forward process, and use a trainable rounding step which is parametrized by  $p_{\theta}(\pmb{y}|\pmb{z}_0; \pmb{x}) = \prod_{i=1}^{N} p_{\theta}(y_i | z_0^i; \pmb{x})$  in the reverse process. In each timestep, we utilize an encoder-decoder model  $\pmb{g}_{\theta}(\pmb{z}_t, t; \pmb{x})$  to approximate  $\pmb{z}_0$  [Lin et al., 2023] in a NAR manner and then estimate  $\mu_{\theta}(\pmb{z}_t, t; \pmb{x})$ .
In consequence, combined with maximizing the evidence lower bound (ELBO) of  $\log p_{\theta}(\pmb{y}|\pmb{x})$ , our training objective of the conditional diffusion language model is:
$$
\mathcal {L} = \mathbb {E} _ {q _ {\phi} \left(\boldsymbol {z} _ {0: T} \mid \boldsymbol {y}\right)} \left[ - \log p _ {\boldsymbol {\theta}} (\boldsymbol {y} \mid \boldsymbol {z} _ {0}; \boldsymbol {x}) + \sum_ {t = 1} ^ {T} \| \boldsymbol {z} _ {0} - \boldsymbol {g} _ {\theta} \left(\boldsymbol {z} _ {t}, t; \boldsymbol {x}\right) \| ^ {2} \right]. \tag {3}
$$
3.1 Multi-Level Diffusion
In the typical diffusion process, every token in the text sequence has the same diffusion timestep. In order to leverage the sequential nature of language, we enable tokens to have different diffusion timesteps during the forward and reverse pass. To accomplish this, we propose a multi-level diffusion strategy that includes both sentence-level and token-level diffusion. Firstly, at the sentence-level, we follow Diffusion-LM [Li et al., 2022a] to randomly select a timestep  $t$ . Secondly, at the token-level, we incorporate positional information  $n \in [1, N]$  based on the sentence-level timestep to regulate the diffusion timestep for the current token. The procedure is illustrated as:
$$
\boldsymbol {z} _ {t} = \left(\boldsymbol {z} _ {f (1, t)} ^ {1}, \boldsymbol {z} _ {f (2, t)} ^ {2}, \dots , \boldsymbol {z} _ {f (N, t)} ^ {N}\right), \tag {4}
$$
where  $N$  is the given target sentence length,  $\mathbf{z}_t$  is the sentence representation at timestep $^4$ $t$ ,  $\mathbf{z}_{f(n,t)}^n$  is the latent representation for the  $n$ -th token at sentence-level timestep  $t$ , and  $f(n,t)$  is a token-level
4Please note that if we talk about a "阶梯" without explicitly indicating that it is for token-level, it should be for sentence-level.
timestep function that denotes the token-level diffusion timestep determined by token position  $n$  and sentence-level timestep  $t$ .
We visualize the token-level timestep  $(n, f(n, t))$  onto a two-dimensional coordinate system as Figure 1, which takes the token position as the horizontal axis and the sentence-level timestep as the vertical axis. Furthermore, to provide a more profound description of the characteristics of movement, we define the speed of movement as the following equation.
$$
v \left(n, t _ {i}, t _ {i + 1}\right) = f \left(n, t _ {i + 1}\right) - f \left(n, t _ {i}\right), \tag {5}
$$
where  $t_i$  and  $t_{i+1}$  are the start and end sentence-level diffusion timesteps. It can be observed that tokens in Diffusion-LM share the same movement speed, while those in AR exhibit different speeds.
3.2 Token-Level Diffusion with Dynamic Movement Speed
Based on the speed of movement, we propose a fundamental principle, dynamic movement speed, for designing the token-level diffusion timestep function  $f(n,t)$  to take advantage of AR in diffusion. Specifically, elements on the left side of a sentence undergo higher movement speed from random Gaussian noise to token embedding, while those on the right side experience lower movement speed, thereby they can be generated in the later sentence-level timestep and utilize information from previously generated tokens more effectively.
Algorithm 1 Training Process of AR-DIFFUSION.
Input: Dataset  $\{(\pmb {x},\pmb {y})\}$  , maximum timestep number  $T$  and maximum target length  $N$
Output: Optimized model parameters  $\theta$
1: Define an anchor point  $(n_e, t_e)^5$ .
2: repeat
3: Sample  $(\pmb{x},\pmb{y})$  from the dataset and embed  $\pmb{y}$  into  $z_0$
4: Sample a sentence-level timestep  $t$  from the interval  $[0, N + T]$ , then the start point is determined by the following equation:
$$
\left(n _ {s}, t _ {s}\right) = \left(\operatorname {c l i p} (N - t, 0, N), \operatorname {c l i p} (t - N, 0, T)\right) \tag {6}
$$
5: Use the point-slope linear function to determine the token-level timestep  $f(n, t)$  in position  $n$ :
$$
f (n, t) = \operatorname {c l i p} \left(\frac {t _ {e} - t _ {s}}{n _ {e} - n _ {s}} \left(n - n _ {s}\right) + t _ {s}, 0, T\right) \tag {7}
$$
6: Sample  $\mathbf{z}_{f(n,t)}^n$  for each  $n$  in different positions with Gaussian reparameterization.
7: According to equation (3) and equation (9), employ gradient descent to optimize the objective:
$$
\min  _ {\theta} \left[ - \log p _ {\boldsymbol {\theta}} (\boldsymbol {y} \mid \boldsymbol {z} _ {0}; \boldsymbol {x}) + \sum_ {n = 1} ^ {N} \left\| \boldsymbol {g} _ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {f (n, t)} ^ {n}, f (n, t); \boldsymbol {x}\right) - \boldsymbol {z} _ {0} \right\| ^ {2} \right] \tag {8}
$$
8: until converged
Following the guidance of the principle, we develop a token-level diffusion strategy with the linear function, which is shown in Figure 1(c). In particular, the procedure is illustrated in Algorithm 1, where  $\mathrm{clip}(x,\min ,\max)$  function is to clamp all elements in  $x$  into the range [min,max]. Specifically, in the forward process of diffusion, the start point goes to the left from  $(N,0)$  to  $(0,0)$  along the horizontal axis and then moves up to  $(0,T)$  along the vertical axis. Therefore, the entire range of sentence-level timestep is extended to  $[0,N + T]$ .
In the reverse diffusion process, the multi-level diffusion follows the formula:
$$
\left. \boldsymbol {g} _ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t}, t; \boldsymbol {x}\right) = \boldsymbol {g} _ {\boldsymbol {\theta}} \left(\left(\boldsymbol {z} _ {f (1, t)} ^ {1}, f (1, t)\right), \left(\boldsymbol {z} _ {f (2, t)} ^ {2}, f (2, t)\right), \dots , \left(\boldsymbol {z} _ {f (N, t)} ^ {N}, f (N, t)\right); \boldsymbol {x}\right), \right. \tag {9}
$$
where  $\pmb{g}_{\theta}(\pmb{z}_{f(n,t)}^n, f(n,t); \pmb{x})$  denotes the  $n$ -th element.
In particular, the anchor point is set as  $(2\times N,T)$  in our implementation. The impact of different choices of the anchor point is discussed in supplementary material E.
3.3 Inference with Skipping
Typically, the generation process needs to go through all the sentence-level timesteps from  $T + N$  to 0. To reduce the decoding time, we introduce a skipping mechanism that allows us to traverse a subset of timesteps.
Algorithm 2 Inference Process of AR-DIFFUSION with the Skipping Mechanism.
Input: Source condition  $\mathbf{x}$ , number of decoding steps  $M$  and model parameters  $\theta$ .
Output: Predicted target embedding  $\hat{y}$
1: Define an anchor point  $(n_e, t_e)$ .
2: Uniformly select a decreasing sequence of timesteps  $\{t_i\}_{i=0}^M$  ranging from  $T + N$  to 0.
3: Sample  $\mathbf{z}_{t_0} \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ .
4: for  $i = 0$  to  $M - 1$  do
5: Calculate the start point  $(n_s, t_s)$  using equation (6).
6: Based on the current sentence-level inference steps  $t_i$  and the next one  $t_{i+1}$ , assign token-level timesteps  $f(n, t_i)$  and  $f(n, t_{i+1})$  to token in position  $n$  using equation (7).
7: Reverse sample  $\mathbf{z}_{t_{i+1}} = \left( \mathbf{z}_{f(1, t_{i+1})}^1, \mathbf{z}_{f(2, t_{i+1})}^2, \dots, \mathbf{z}_{f(N, t_{i+1})}^N \right)$  from  $p_\theta(\mathbf{z}_{t_{i+1}} \mid \mathbf{z}_{t_i}; \mathbf{x})$  with the following formulas:
$$
p _ {\theta} \left(\boldsymbol {z} _ {t _ {i + 1}} \mid \boldsymbol {z} _ {t _ {i}}; \boldsymbol {x}\right) = \prod_ {n = 1} ^ {N} p _ {\theta} \left(\boldsymbol {z} _ {f (n, t _ {i + 1})} ^ {n} \mid \boldsymbol {z} _ {f (n, t _ {i})} ^ {n}; \boldsymbol {x}\right) \tag {10}
$$
$$
p _ {\theta} \left(\boldsymbol {z} _ {f (n, t _ {i + 1})} ^ {n} \mid \boldsymbol {z} _ {f (n, t _ {i})} ^ {n}; \boldsymbol {x}\right) \sim \mathcal {N} \left(\boldsymbol {z} _ {f (n, t _ {i + 1})} ^ {n}; \lambda \boldsymbol {z} _ {f (n, t _ {i})} ^ {n} + \mu \boldsymbol {g} _ {\theta} \left(\boldsymbol {z} _ {f (n, t)} ^ {n}, f (n, t); \boldsymbol {x}\right), \sigma \mathbf {I}\right) \tag {11}
$$
8: end for
9: Map  $\mathbf{z}_{t_M}$  to the nearest embedding  $\hat{\mathbf{y}}$ .
In practice, we propose an algorithm for the inference, illustrated in Algorithm 2.
$$
\lambda = \frac {\sqrt {\frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}}} \left(1 - \bar {\alpha} _ {f (n, t _ {i + 1})}\right)}{1 - \bar {\alpha} _ {f (n , t _ {i})}}, \mu = \frac {\sqrt {\bar {\alpha} _ {f (n , t _ {i + 1})}} \left(1 - \frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}}\right)}{1 - \bar {\alpha} _ {f (n , t _ {i})}}, \sigma = \frac {\left(1 - \alpha_ {f (n , t _ {i})}\right) \left(1 - \bar {\alpha} _ {f (n , t _ {i + 1})}\right)}{1 - \bar {\alpha} _ {f (n , t _ {i})}} \tag {12}
$$
In equation (10), the conditional distribution of  $\mathbf{z}_{t_{i+1}}$  is inferred by  $p_{\theta}(\mathbf{z}_{t_{i+1}} | \mathbf{z}_{t_i}; \mathbf{x})$ , and then we decompose it by positions due to the independent forward process of elements at different positions. From equation (11) to equation (12), we establish the relationship between tokens at different timesteps, and the detailed derivation can be found in supplementary material F.
4.1 Tasks and Datasets
Text Summarization This task involves taking a long document as input and generating a concise sentence as output. This requires models with the ability to identify important content and rewrite it in a condensed form. In our experiments, we use the publicly available XSUM [Narayan et al., 2018] and CNN/DAILYHermann et al. [2015] on GLGE<sup>6</sup>, which is also named as GLGE-Easy.
Machine Translation Translation is a widely used sequence-to-sequence task. The input is a sequence of words in the source language, and the output is a sequence of corresponding words in the target language. We choose the IWSLT 2014 dataset and the data processing method is to follow the scripts provided by fairseq<sup>7</sup>.
Common Sense Generation In this task, the model is provided with a concept set consisting of objects and actions as input. The objective is to generate a sentence that incorporates these concepts and describes a realistic scenario. We use COMMONGEN<sup>8</sup> dataset for evaluation.
$^{6}$ https://microsoft.github.io/glge/
<sup>7</sup>https://github.com/facebookresearch/fairseq/tree/main/examples/translation
<sup>8</sup>https://inklab.usc.edu/CommonGen/
The results on different datasets are shown in Table 1, Table 2, Table 3 and Table 4. The best result is **bolded** and the second-best result is **underlined**. "k" indicates the number of generated candidate samples<sup>9</sup>. It can be seen from the results in each table that AR-DIFFUSION achieves the best performance.
During the inference process, we utilize 20 inference steps and employ Minimum Bayes Risk (MBR) [Kumar and Byrne, 2004] decoding to select the best sample, following [Li et al., 2022a]. We choose MBR instead of the selection approach in GENIE, as GENIE picks up the best sample by calculating the maximum score for each generated one using ground truth, which introduces unfairness. To ensure a fair comparison, we re-implement GENIE using our configuration and perform inference with 20 steps. More experimental details can be found in the supplementary material B.
Table 1: Results on XSUM test set. The results of NAR and Semi-NAR are from Qi et al. [2021], and the results of AR are from GLGE [Liu et al., 2021].
Methods	Pattern	ROUGE-1	ROUGE-2	ROUGE-L
NAT [Gu et al., 2017]	NAR	24.0	3.9	20.3
iNAT [Lee et al., 2018]	24.0	4.0	20.4
CMLM [Ghazvininejad et al., 2019]	23.8	3.6	20.2
LevT [Gu et al., 2019]	24.8	4.2	20.9
InsT [Stern et al., 2019]	Semi-NAR	17.7	5.2	16.1
iNAT [Lee et al., 2018]	27.0	6.9	22.4
CMLM [Ghazvininejad et al., 2019]	29.1	7.7	23.0
LevT [Gu et al., 2019]	25.3	7.4	21.5
LSTM [Greff et al., 2017]	\( AR^{10} \)	25.1	6.9	19.9
Transformer [Vaswani et al., 2017]	30.5	10.4	24.2
GENIE [Lin et al., 2023] (k=50)	Diffusion	29.3	8.3	21.9
AR-DIFFUSION (k=50)	31.7	10.1	24.7
AR-DIFFUSION (k=500)	32.2	10.6	25.2
Table 2: Results on CNN/DAILYMAIL test set. The results of AR are from GLGE Liu et al. [2021].
Methods	Pattern	ROUGE-1	ROUGE-2	ROUGE-L
LSTM [Greff et al., 2017]	AR	37.3	15.7	34.4
Transformer [Vaswani et al., 2017]	39.5	16.7	36.7
GENIE [Lin et al., 2023] (k=50)	Diffusion	34.4	12.8	32.1
AR-DIFFUSION (k=50)	39.6	16.3	37.1
AR-DIFFUSION (k=500)	40.2	17.1	37.7
Text Summarization The results presented in Table 1 and Table 2 clearly demonstrate that AR-DIFFUSION outperforms the existing NAR and Semi-NAR approaches across all metrics. Moreover, AR-DIFFUSION consistently achieves significant improvements over GENIE in terms of all indicators. Furthermore, in comparison to Transformer, AR-DIFFUSION outperforms it on both ROUGE-1 and ROUGE-L, while achieving comparable performance in terms of ROUGE-2. Notably, when the sample number is 500, AR-DIFFUSION demonstrates superiority over Transformer across all the measures.
Machine Translation Table 3 presents the BLEU score implemented by SeqDiffuSeq setting<sup>11</sup>. AR-DIFFUSION outperforms the non-auto-regressive CNAT in greedy search for a single sample, and achieves a substantial gain. Moreover, the BLEU score of AR-DIFFUSION surpasses GENIE by a large margin and shows a slightly better performance than the AR Transformer. Specially, AR-DIFFUSION achieves a more powerful result at  $k = 500$ .
<sup>9</sup>The relationship between sample number and results is discussed in supplementary material E.
10 Notably, although AR's beam search has a small beam, the search space may be larger than 50 or even 500.
<sup>11</sup>We also report SacreBLEU in supplementary material C to compare with DINOISER.