Title: AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: text generation

diffusion model

auto-regression

sequential dependency
Abstract: Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. 
However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing language models are trained with a left-to-right auto-regressive approach.
To account for the inherent sequential characteristic of natural language, we introduce  Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right.
In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-Diffusion clearly demonstrated its superiority over existing diffusion language models and that it can be $100\times\sim600\times$ faster when achieving comparable results. Our code is available at https://github.com/microsoft/ProphetNet/tree/master/AR-diffusion.

AR-DIFFUSION: Auto-Regressive Diffusion Model for Text Generation

Abstract

Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing language models are trained with a left-to-right auto-regressive approach. To account for the inherent sequential characteristic of natural language, we introduce Auto-Regressive Diffusion (AR-DIFFUSION). AR-DIFFUSION ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right. In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-DIFFUSION clearly demonstrated its superiority over existing diffusion language models and that it can be 100 × ∼ 600 × faster when achieving comparable results. Our code is available at this https URL.

1 Introduction

Text generation is a fundamental task within the field of natural language processing (NLP). Pretrained language models like GPT-4 [OpenAI, 2023], LLaMA [Touvron et al., 2023], and Alpaca [Taori et al., 2023] have garnered significant attention with their ability to generate fluent and humanlike textual content. These models utilize the auto-regressive (AR) Transformer decoders [Vaswani et al., 2017] to emit generated tokens one-by-one in sequential order from left to right. By leveraging the power of position dependency, AR models are able to enhance the naturalness, coherence, and adherence to human language conventions in the generated text [Brown et al., 2020].

Recent studies have shown the remarkable performance of diffusion models in image generation [Ho et al., 2020], motivating researchers to extend diffusion to text generation [Li et al., 2022a, Gong et al., 2022, Dieleman et al., 2022, Yuan et al., 2022, Ye et al., 2023]. By introducing timestep, these methods progressively regulate the interpolation between the original tokens and Gaussian noise, then iteratively denoise for text generation. At each timestep, the diffusion-based text generator predicts

𝑡 𝑖+1

T

T

𝑡 𝑖+1

𝑡

𝑖

𝑡

𝑖

Timestep

Timestep

Position

Position

𝑓(𝑛2, 𝑡

𝑖+1 )

𝑓(𝑛2, 𝑡

𝑖

)

(a) Diffusion-LM

(c) AR-Diffusion

Token-Level Diffusion Timestep Function

Forward Diffusion Process

Reverse Diffusion Process

(𝑛𝑒 , 𝑡

𝑒

)

𝑛1

𝑛2

N

N

𝑛2

T

𝑡 𝑖+1

𝑡

𝑖

Timestep

Position

(b) AR

N

𝑛2

𝑛1

𝒗 𝒏𝟐, 𝒕 𝒊

, 𝒕 𝒊+𝟏

= |𝒕𝒊+𝟏 -𝒕𝒊|

𝒗 𝒏𝟏,𝒕𝒊, 𝒕 𝒊+𝟏

= |𝒕𝒊+𝟏 -𝒕𝒊|

𝑓(𝑛1, 𝑡

𝑖+1 )

𝑓(𝑛1, 𝑡

𝑖

)

𝑛1

𝑓(𝑛2, 𝑡

𝑖+1 )

𝑓(𝑛2, 𝑡

𝑖

)

𝒗 𝒏𝟐,𝒕𝒊, 𝒕 𝒊+𝟏

= 𝑻

𝒗 𝒏𝟏, 𝒕 𝒊

, 𝒕 𝒊+𝟏

= 𝟎

𝑓 𝑛1, 𝑡

𝑖

= 𝑓(𝑛1, 𝑡 𝑖+1 )

𝑓(𝑛2, 𝑡

𝑖+1 )

𝑓(𝑛2, 𝑡

𝑖

)

𝒗(𝒏𝟐, 𝒕 𝒊

, 𝒕 𝒊+𝟏

)

𝒗(𝒏𝟏, 𝒕 𝒊

, 𝒕 𝒊+𝟏

)

𝑓(𝑛1, 𝑡

𝑖+1 )

𝑓(𝑛1, 𝑡

𝑖

)

all tokens simultaneously following Non-Auto-Regression (NAR) [Lewis et al., 2020, Qi et al., 2020, 2021, Li et al., 2022b], leading to faster decoding speed compared to AR. However, it also inherits the drawback of NAR, namely the sacrifice of inter-token position dependency [Li et al., 2022c] and the drop of generation performance [Bao et al., 2021].

To conduct a comprehensive analysis, we introduce a two-dimensional coordinate system to track the diffusion timestep of tokens f ( · ) positioned at various locations. As illustrated in Figure 1, the system assigns the token position n ∈ [1 , N ] to the horizontal axis and the diffusion timestep t ∈ [0 , T ] to the vertical axis. Diffusion-LM [Li et al., 2022a], which is followed by existing diffusion-based text generation models, is shown in Figure 1(a). It assigns a uniform timestep t to all tokens. In contrast, tokens in the AR model depicted in Figure 1(b) exhibit distinct timesteps within a generation step ( t i ). For instance, the already decoded token at position n 1 has a timestep of 0 , while the to-be-decoded token at position n 2 has a timestep of T . This approach effectively captures the sequential dependency. Motivated by this observation, we introduce AR-DIFFUSION, an auto-regressive diffusion method, for the disparity in token positions and the principle of sequential token identification.

In AR-DIFFUSION, we propose a multi-level diffusion strategy that includes both sentence-level and token-level diffusion. We randomly choose a sentence-level timestep t , and assign dynamic movement speeds v ( · ) by determining position-sensitive token-level timestep f ( n, t ) for each token. This enables tokens at the left of a sentence to undergo faster movement from random Gaussian noise to token embedding, while those at the right of the sentence experience slower movement to better utilize information from previously denoised tokens. During inference, to reduce the significant number of inference steps (e.g., 2,000) required in Diffusion-LM [Li et al., 2022a], SeqDiffSeq [Yuan et al., 2022] and GENIE [Lin et al., 2023], we introduce a skipping mechanism that collaborates with the multi-level diffusion strategy to accelerate the process.

Experimental results across various text generation tasks, such as text summarization, machine translation, and common sense generation, have consistently demonstrated that AR-DIFFUSION surpasses existing text diffusion models, including AR methods in terms of both quality and diversity. Moreover, our verification reveals that AR-DIFFUSION requires fewer resources during decoding while maintaining superior performance. It achieves 100 × faster than SeqDiffSeq [Yuan et al., 2022] in machine translation and 600 × faster than GENIE [Lin et al., 2023] in text summarization while delivering comparable results. Furthermore, it demonstrates promising results even in a challenging scenario where decoding is limited to only two steps.

2 Preliminary

2.1 Conditional Generative Language Models

In the field of natural language generation, conditional generative models are commonly implemented using either auto-regressive (AR) or non-auto-regressive (NAR) methods. In AR [Vaswani et al., 2017], tokens on the right are predicted based on visible left tokens. The likelihood is given by p AR ( y | x ) = ∏ N i =1 p ( y i | y 1: i -1 ; x ) , where y i denotes the i -th token of y . Onthe other hand, NAR [Gu et al., 2017] assumes conditional independence among tokens and generates them uniformly without distinction during decoding, resulting in the likelihood p NAR ( y | x ) = ∏ N i =1 p ( y i | x ) . This parallel generation approach is of lower quality compared to AR, although it offers a substantial speed advantage.

2.2 Diffusion Models for Text Generation

Recently, Li et al. [2022a] propose a natural language generation model based on the diffusion process, which is typically divided into a forward noising process and a reverse denoising process.

Specifically, the forward process is a fixed linear Gaussian model, which gradually perturbs the random variable z 0 until it becomes the standard Gaussian distribution. This can be formalized as:

where, ¯ α t = ∏ t i =1 α i , and α i is a coefficient that monotonically decreases with timestep t , z t is the latent state at timestep t .

The reverse process is to initiate from standard Gaussian noise and progressively utilize the denoising transition p θ ( z t -1 | z t ; x ) for generation.

where the mean µ θ and variance Σ θ are learned from the model. In particular, we follow Li et al. [2022a]'s approach of using predefined variance without trainable parameters.

To extend the continuous diffusion process to discrete text generation, Li et al. [2022a] introduce an additional Markov transition from the discrete tokens y to the latent variable z 0 . In practice, we add an embedding step q ϕ ( z 0 | y ) = N ( z 0 ; Emb( y ) , (1 -α 0 ) I ) in the forward process, and use a trainable rounding step which is parametrized by p θ ( y | z 0 ; x ) = ∏ N i =1 p θ ( y i | z i 0 ; x ) in the reverse process. In each timestep, we utilize an encoder-decoder model g θ ( z t , t ; x ) to approximate z 0 [Lin et al., 2023] in a NAR manner and then estimate µ θ ( z t , t ; x ) .

In consequence, combined with maximizing the evidence lower bound (ELBO) of log p θ ( y | x ) , our training objective of the conditional diffusion language model is:

3 Methodology

3.1 Multi-Level Diffusion

In the typical diffusion process, every token in the text sequence has the same diffusion timestep. In order to leverage the sequential nature of language, we enable tokens to have different diffusion timesteps during the forward and reverse pass. To accomplish this, we propose a multi-level diffusion strategy that includes both sentence-level and token-level diffusion. Firstly, at the sentence-level, we follow Diffusion-LM [Li et al., 2022a] to randomly select a timestep t . Secondly, at the token-level, we incorporate positional information n ∈ [1 , N ] based on the sentence-level timestep to regulate the diffusion timestep for the current token. The procedure is illustrated as:

where N is the given target sentence length, z t is the sentence representation at timestep 4 t , z n f ( n,t ) is the latent representation for the n -th token at sentence-level timestep t , and f ( n, t ) is a token-level

timestep function that denotes the token-level diffusion timestep determined by token position n and sentence-level timestep t .

We visualize the token-level timestep ( n, f ( n, t ) ) onto a two-dimensional coordinate system as Figure 1 , which takes the token position as the horizontal axis and the sentence-level timestep as the vertical axis. Furthermore, to provide a more profound description of the characteristics of movement, we define the speed of movement as the following equation.

where t i and t i +1 are the start and end sentence-level diffusion timesteps. It can be observed that tokens in Diffusion-LM share the same movement speed, while those in AR exhibit different speeds.

3.2 Token-Level Diffusion with Dynamic Movement Speed

Based on the speed of movement, we propose a fundamental principle, dynamic movement speed, for designing the token-level diffusion timestep function f ( n, t ) to take advantage of AR in diffusion. Specifically, elements on the left side of a sentence undergo higher movement speed from random Gaussian noise to token embedding, while those on the right side experience lower movement speed, thereby they can be generated in the later sentence-level timestep and utilize information from previously generated tokens more effectively.

Algorithm 1 Training Process of AR-DIFFUSION.

Input : Dataset { ( x , y ) } , maximum timestep number T and maximum target length N .

Output : Optimized model parameters θ

1: Define an anchor point ( n e , t e ) 5 .

2: repeat

3: Sample ( x , y ) from the dataset and embed y into z 0 .

4: Sample a sentence-level timestep t from the interval [0 , N + T ] , then the start point is determined by the following equation:

5: Use the point-slope linear function to determine the token-level timestep f ( n, t ) in position n :

6: Sample z n f ( n,t ) for each n in different positions with Gaussian reparameterization.

7: According to equation (3) and equation (9), employ gradient descent to optimize the objective:

8: until converged

Following the guidance of the principle, we develop a token-level diffusion strategy with the linear function, which is shown in Figure 1(c). In particular, the procedure is illustrated in Algorithm 1, where clip( x, min , max) function is to clamp all elements in x into the range [min , max] . Specifically, in the forward process of diffusion, the start point goes to the left from ( N, 0) to (0 , 0) along the horizontal axis and then moves up to (0 , T ) along the vertical axis. Therefore, the entire range of sentence-level timestep is extended to [0 , N + T ] .

In the reverse diffusion process, the multi-level diffusion follows the formula:

where g θ ( z n f ( n,t ) , f ( n, t ); x ) denotes the n -th element.

.

3.3 Inference with Skipping

Typically, the generation process needs to go through all the sentence-level timesteps from T + N to 0 . To reduce the decoding time, we introduce a skipping mechanism that allows us to traverse a subset of timesteps.

Algorithm 2 Inference Process of AR-DIFFUSION with the Skipping Mechanism.

Input : Source condition x , number of decoding steps M and model parameters θ . Output : Predicted target embedding ˆ y .

1: Define an anchor point ( n e , t e ) .

2: Uniformly select a decreasing sequence of timesteps { t i } i M =0 ranging from T + N to 0 .

3: Sample z t 0 ∼ N ( 0 , I ) .

4: for i = 0 to M -1 do

5: Calculate the start point ( n s , t s ) using equation (6).

6: Based on the current sentence-level inference steps t i and the next one t i +1 , assign token-level timesteps f ( n, t i ) and f ( n, t i +1 ) to token in position n using equation (7).

7: Reverse sample z t i +1 = ( z 1 f (1 ,t i +1 ) , z 2 f (2 ,t i +1 ) , · · · , z N f ( N,t i +1 ) ) from p θ ( z t i +1 | z t i ; x ) with the following formulas:

8: end for

9: Map z t M to the nearest embedding ˆ y .

In practice, we propose an algorithm for the inference, illustrated in Algorithm 2.

In equation (10), the conditional distribution of z t i +1 is inferred by p θ ( z t i +1 | z t i ; x ) , and then we decompose it by positions due to the independent forward process of elements at different positions. From equation (11) to equation (12), we establish the relationship between tokens at different timesteps, and the detailed derivation can be found in supplementary material F.

4 Experiments

4.1 Tasks and Datasets

Text Summarization This task involves taking a long document as input and generating a concise sentence as output. This requires models with the ability to identify important content and rewrite it in a condensed form. In our experiments, we use the publicly available XSUM [Narayan et al., 2018] and CNN/DAILYMAIL Hermann et al. [2015] on GLGE 6 , which is also named as GLGE-Easy.

Machine Translation Translation is a widely used sequence-to-sequence task. The input is a sequence of words in the source language, and the output is a sequence of corresponding words in the target language. We choose the IWSLT 2014 dataset and the data processing method is to follow the scripts provided by fairseq 7 .

Common Sense Generation In this task, the model is provided with a concept set consisting of objects and actions as input. The objective is to generate a sentence that incorporates these concepts and describes a realistic scenario. We use COMMONGEN 8 dataset for evaluation.

4.2 Main Results

The results on different datasets are shown in Table 1, Table 2, Table 3 and Table 4. The best result is bolded and the second-best result is underlined . ' k ' indicates the number of generated candidate samples 9 . It can be seen from the results in each table that AR-DIFFUSION achieves the best performance.

During the inference process, we utilize 20 inference steps and employ Minimum Bayes Risk (MBR) [Kumar and Byrne, 2004] decoding to select the best sample, following [Li et al., 2022a]. We choose MBR instead of the selection approach in GENIE, as GENIE picks up the best sample by calculating the maximum score for each generated one using ground truth, which introduces unfairness. To ensure a fair comparison, we re-implement GENIE using our configuration and perform inference with 20 steps. More experimental details can be found in the supplementary material B.

Text Summarization The results presented in Table 1 and Table 2 clearly demonstrate that ARDIFFUSION outperforms the existing NAR and Semi-NAR approaches across all metrics. Moreover, AR-DIFFUSION consistently achieves significant improvements over GENIE in terms of all indicators. Furthermore, in comparison to Transformer, AR-DIFFUSION outperforms it on both ROUGE-1 and ROUGE-L , while achieving comparable performance in terms of ROUGE-2 . Notably, when the sample number is 500, AR-DIFFUSION demonstrates superiority over Transformer across all the measures.

Machine Translation Table 3 presents the BLEU score implemented by SeqDiffuSeq setting 11 . AR-DIFFUSION outperforms the non-auto-regressive CNAT in greedy search for a single sample, and achieves a substantial gain. Moreover, the BLEU score of AR-DIFFUSION surpasses GENIE by a large margin and shows a slightly better performance than the AR Transformer. Specially, AR-DIFFUSION achieves a more powerful result at k = 500.

Common Sense Generation As depicted in Table 4, AR-DIFFUSION achieves superior performance compared to the current AR, NAR, and other diffusion methods across all the metrics on the COMMONGEN dataset.

4.3 Inference Efficiency

First, we use the number of function evaluations (NFE) as a measure to compare inference efficiency [Ye et al., 2023] in machine translation. From Table 3, it is evident that even when the NFE is reduced to 1% of SeqDiffuSeq (equivalent to 100 × faster), AR-DIFFUSION still outperforms SeqDiffuSeq. Moreover, increasing the number of generated candidate samples ( k = 500 ) leads to further performance improvements, albeit with increased time consumption.

Second, we conduct experiments with an extremely limited number of inference steps (2 and 3) 12 and compare the performance with that of GENIE in XSUM. The results are presented in Table 5. When reducing the number of steps to 2, GENIE experiences a significant decline, with an average score of 4.20 in the AVG Drop column, while AR-DIFFUSION exhibits a comparatively smaller

decrease of 1.34. Furthermore, with 3 steps, although the performance deterioration of GENIE is somewhat reduced, the average score still shows a decline of 2.81. In contrast, AR-DIFFUSION maintains a high performance level, with an average score differing from the 20-step result by only 0.64. Notably, the results of AR-DIFFUSION at 3 steps are comparable to the results of GENIE at 2,000 steps. Therefore, compared to GENIE, the inference speed of AR-DIFFUSION can be accelerated by up to 600 × .

4.4 Analysis

Diversity of Samples Diversity is a key advantage of diffusion models. To measure the diversity of generated samples, We adopt the SELF-BLEU [Zhu et al., 2018] metric, in which a lower score indicates higher diversity. In Lin et al. [2023], various sampling methods were applied to the pretrained auto-regressive model BART 13 . As shown in Table 6, AR-DIFFUSION achieves significantly higher diversity compared to the auto-regressive model. Furthermore, the diversity can be comparable to GENIE with a better performance.

(a)

(b)

(c)

21.2

21.2

21.0

21.0

20

20.8

20.8

19

20.6

20.6

18

20.4

20.4

20.2

17

20.2

20.0

16

20.0

AR-Diffusion w/ skipping

19.8

AR-Diffusion w/ skipping

GENIE w/skipping

19.8

GENIE w/skipping

AR-Diffusion w/ DDIM

15

GENIE W/DDIM

19.6

2

4

10

20

40

100

2

4

10

20

40

100

2

4

10

20

40

100

Ablation Study To demonstrate the effectiveness of our proposed method, we perform ablation experiments on the XSUM dataset. Our results show that both our proposed multi-level diffusion and skipping mechanism are essential for achieving the high performance of AR-DIFFUSION.

Maintaining the skipping inference method, we remove the token-level diffusion during the training process, which degenerates to GENIE w/ skipping. The comparison results are shown in Figure 2(a). It can be observed that after removing, the AVG-ROUGE score is greatly lower after 2 steps.

The performance of applying our proposed skipping mechanism and DDIM [Song et al., 2021] to AR-DIFFUSION is shown in Figure 2(b). The results demonstrate that the skipping mechanism consistently outperforms DDIM in various inference steps. Additionally, the skipping mechanism can be easily applied to GENIE. As depicted in Figure 2(c), DDIM suffers a significant drop in performance when the number of inference steps is less than 40. In contrast, the skipping mechanism consistently maintains good performance across all inference steps.

12.0

operatives  time ##vy β service ##loading [unused71]  less ##† ##

ल

cfl

ء

travelling  northern  ##ᴰ ##″ earlier failed

a government plan to lift with ##

モ

getting  ##ß e to 50 - public

غ

cleaning  selangor shared intended dublin college

a government plan to petition with public than prices to research  and public leiden cleaningp

selangor discovered

a government plan to deal with public spending cuts to research  and public sectorcare

, figures obtained by the tata

a government  plan to deal with public spending cuts to research  and public sector , figures  obtained  by the bbc .

14.0

16.0

18.0

20.0

Case Study By mapping the state to the token with the highest logits, we visualize the intermediate states of AR-DIFFUSION. As depicted in Figure 3, AR-DIFFUSION undergoes a denoising process, transforming the random Gaussian noise into a coherent sentence over 20 steps, and we present 5 of them. With the progression of each timestep, compared to the tokens on the right side of the sentence, the tokens on the left side demonstrate faster determination and a rapid increase in the corresponding logits. This behavior is consistent with our principle of dynamic movement speed from left to right.

5 Related Work

AR and NAR Language Models AR models have been the dominant approach for text generation [OpenAI, 2023, Touvron et al., 2023, Dong et al., 2023], but their token-by-token generation nature often leads to unsatisfactory inference speed. To address this issue, NAR models have been developed in recent years. The NAR method is initially proposed by Gu et al. [2017], its objective is generate the entire output sequence in parallel, thereby improving generation speed and efficiency. Subsequently, LevT [Gu et al., 2019] adopts insertion and deletion to address the lack of flexibility in NAR generation, CMLM [Ghazvininejad et al., 2019] utilizes a masked language model to improve the quality of NAR generation through a constant number of iterations, and CNAT [Bao et al., 2021] introduces latent variables to represent the category information of the target word to make full use of the latent representation. However, these NAR methods are hard to model inter-token position dependency and deficient in generation performance.

Continuous Text Diffusion The application of diffusion models to continuous text space is first introduced by Li et al. [2022a]. Through the embedding and rounding processes, the direct integration of continuous noise into word embeddings was accomplished. After that, more people attempt to adopt continuous text diffusion model to solve sequence-to-sequence tasks. DiffuSeq [Gong et al., 2022] divides the input into two parts, utilizing one part as a condition, and perturbs the other part with noise. CDCD [Dieleman et al., 2022] proposes score interpolation and time warping to allow diffusion model and Euclidean embedding to share the same loss function for training. SeqDiffuSeq [Yuan et al., 2022], GENIE [Lin et al., 2023] and DINOISER [Ye et al., 2023] incorporate diffusion model into the encoder-decoder structure through cross-attention mechanisms.

It is important to highlight the differences between our method and both ARDMs [Hoogeboom et al., 2022] and TimeGrad [Rasul et al., 2021], despite the common references to autoregression and diffusion in all these. ARDMs employ an order-agnostic technique, leveraging masking and prediction for generation in arbitrary orders. On the other hand, TimeGrad integrates RNN and diffusion to model the conditional distribution of future steps of multivariate time series. In contrast, our research focuses on implementing the diffusion process within a continuous embedding space, with the primary aim of generating text in a left-to-right sequence.

6 Conclusion

This paper introduces AR-DIFFUSION, which exhibits AR-like generation behavior but enables efficient parallel decoding. Embracing the inherent sequential nature of language, we propose a multi-

level diffusion model, consisting of sentence-level and token-level components, to assign dynamic movement speeds to tokens. Consequently, compared to those on the right, the left tokens undergo fewer denoising steps and generate earlier to subsequently influence the later ones. Furthermore, we introduce a skipping mechanism to facilitate parallel generation within the multi-level diffusion framework. The experimental results across various tasks demonstrate that AR-DIFFUSION surpasses existing diffusion models in terms of quality while maintaining diversity. Additionally, compared to existing diffusion language models, AR-DIFFUSION achieves comparable results while being 100 × ∼ 600 × faster.

References

OpenAI. GPT-4 technical report. CoRR , abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774 .

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023. doi: 10.48550/arXiv.2302.13971. URL https://doi. org/10.48550/arXiv.2302.13971 .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html .

Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. DiffusionLM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022a. URL https://openreview.net/forum?id=3s9IrEsjLyk .

Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. CoRR , abs/2210.08933, 2022. doi: 10.48550/ arXiv.2210.08933. URL https://doi.org/10.48550/arXiv.2210.08933 .

Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089 , 2022.

Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. Seqdiffuseq: Text diffusion with encoder-decoder transformers, 2022.

Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. Dinoiser: Diffused conditional sequence learning by manipulating noises, 2023.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 7871-7880. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.703. URL https://doi.org/10.18653/v1/2020.acl-main.703 .

Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 2401-2410, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. findings-emnlp.217. URL https://aclanthology.org/2020.findings-emnlp.217 .

Weizhen Qi, Yeyun Gong, Jian Jiao, Yu Yan, Weizhu Chen, Dayiheng Liu, Kewen Tang, Houqiang Li, Jiusheng Chen, Ruofei Zhang, et al. Bang: Bridging autoregressive and non-autoregressive generation with large scale pretraining. In International Conference on Machine Learning , pages 8630-8639. PMLR, 2021.

Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. ELMER: A nonautoregressive pre-trained language model for efficient and effective text generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 1044-1058. Association for Computational Linguistics, 2022b. URL https://aclanthology.org/2022.emnlp-main.68 .

Yafu Li, Leyang Cui, Yongjing Yin, and Yue Zhang. Multi-granularity optimization for nonautoregressive translation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 5073-5084. Association for Computational Linguistics, 2022c. URL https://aclanthology.org/2022.emnlp-main.339 .

Yu Bao, Shujian Huang, Tong Xiao, Dongqi Wang, Xinyu Dai, and Jiajun Chen. Non-autoregressive translation by learning target categorical codes. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5749-5759, Online, June 2021. Association for Computational Linguistics. doi: 10. 18653/v1/2021.naacl-main.458. URL https://aclanthology.org/2021.naacl-main.458 .

Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise, 2023.

Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281 , 2017.

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 1797-1807, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1206. URL https://aclanthology.org/D18-1206 .

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper_ files/paper/2015/file/afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf .

Shankar Kumar and William Byrne. Minimum bayes-risk decoding for statistical machine translation. Technical report, JOHNS HOPKINS UNIV BALTIMORE MD CENTER FOR LANGUAGE AND SPEECH PROCESSING (CLSP), 2004.

Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, et al. Glge: A new general language generation evaluation benchmark. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 408-420, 2021.

Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. arXiv preprint arXiv:1802.06901 , 2018.

Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 6112-6121, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1633. URL https://aclanthology.org/D19-1633 .

Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In Advances in Neural Information Processing Systems , pages 11181-11191, 2019.

Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. arXiv preprint arXiv:1902.03249 , 2019.

Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. LSTM: A search space odyssey. IEEE Trans. Neural Networks Learn. Syst. , 28(10):2222-2232, 2017.

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 1823-1840, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.findings-emnlp.165. URL https://aclanthology.org/2020.findings-emnlp.165 .

Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. Incorporating copying mechanism in sequence-to-sequence learning. In ACL (1) . The Association for Computer Linguistics, 2016.

Raymond Hendy Susanto, Shamil Chollampatt, and Liling Tan. Lexically constrained neural machine translation with levenshtein transformer. In ACL , pages 3536-3543. Association for Computational Linguistics, 2020.

Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 1097-1100, 2018.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=St1giarCHLP .

Chenhe Dong, Yinghui Li, Haifan Gong, Miaoxin Chen, Junxin Li, Ying Shen, and Min Yang. A survey of natural language generation. ACM Comput. Surv. , 55(8):173:1-173:38, 2023. doi: 10.1145/3554727. URL https://doi.org/10.1145/3554727 .

Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=Lm8T39vLDTE .

Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8857-8868. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/rasul21a.html .

Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18-2012. URL https://aclanthology.org/D18-2012 .

Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970 , 2022.

Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie-Yan Liu. A survey on non-autoregressive generation for neural machine translation and beyond. CoRR , abs/2204.09269, 2022.

Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. CoRR , abs/1610.02424, 2016.

Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Typical decoding for natural language generation. CoRR , abs/2202.00666, 2022.

Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In ACL (1) , pages 889-898. Association for Computational Linguistics, 2018.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR . OpenReview.net, 2020.

A Limitation

A primary limitation of our work lies in the requirement of generating a large number of candidate samples for optimal performance. As an illustration in Table 2 of CNN/DAILYMAIL dataset, ARDIFFUSION ( k = 50) achieves a 0.8 lower ROUGE-2 score compared to AR-DIFFUSION ( k = 500). We anticipate exploring more efficient sampling strategies to minimize the number of generated samples without performance drop.

B Experimental Details

Model Setup Our model configuration is implemented based on Transformer-base [Vaswani et al., 2017]. In particular, For XSUM and CNN/DAILYMAIL, we set the diffusion embedding dimension to 128. For IWSLT14, we use 64-dimensional diffusion embedding, 4 attention heads and 1024-dimensional feed-forward layers. For COMMONGEN, we adopt 64-dimensional diffusion embedding, 8 attention heads and 512-dimensional feed-forward layers.

Training and Inference In the training phase, we employ a square-root noise schedule and 2,000 diffusion steps [Li et al., 2022a]. Specially, we use the tokenizer and vocabulary constructed by Byte Pair Encoding (BPE) 14 [Kudo and Richardson, 2018] for translation tasks. For other tasks, we adopt the tokenizer and vocabulary of bert-base-uncased .

Baselines We set four groups of baselines:

NAR: NAT [Gu et al., 2017], iNAT [Lee et al., 2018], CMLM [Ghazvininejad et al., 2019], LevT [Gu et al., 2019] and CNAT [Bao et al., 2021];

Semi-NAR: InsT [Stern et al., 2019], iNAT [Lee et al., 2018], CMLM [Ghazvininejad et al., 2019] and LevT [Gu et al., 2019];

AR: bRNN [Gu et al., 2016], LSTM [Greff et al., 2017] and Transformer [Vaswani et al., 2017];

Diffusion: DiffusionLM [Li et al., 2022a], CDCD [Dieleman et al., 2022], SeqDiffuSeq [Yuan et al., 2022], DINOISER [Ye et al., 2023] and GENIE [Lin et al., 2023].

Metrics We follow the approach of Qi et al. [2020] 15 to evaluate the ROUGE-1/2/L of the summarization task. For the evaluation of translation tasks, we adopt the setting of SeqDiffuSeq [Yuan et al., 2022] to report BLEU score. In addition, we also calculate the SacreBLEU score according to the setting of DINOISER [Ye et al., 2023] for comparison. For COMMONGEN, we employ ROUGE-2/L , BLEU-3/4 , METEOR and SPICE under the evaluation methods of Lin et al. [2020] 16 .

Training Parameters Our training parameters on different datasets are shown in Table 7. Our linear schedule warm up steps is 4,000 × N gc , where N gc denotes gradient accumulation number. In addition, we use the AdamW (weight decay = 0.0) optimizer and dropout is 0.2. All experiments are implemented on 8 Tesla V100-32G. It takes about 20 hours to train XSUM and CNN/DAILYMAIL, about 5 hours to train IWSLT14, and about 2 hours to train COMMENGEN.

C More Results on IWSLT14

In Table 8 we give the SacreBLEU score according to the setting of DINOISER. AR-DIFFUSION has notable improvements over non-auto-regressive CMLM. Besides, AR-DIFFUSION achieves excellent performance among text diffusion models for both EN → DE and DE → EN tasks. Specifically, AR-DIFFUSION is far superior to GENIE and comparable to the newly proposed DINOISER at n = 50. Nevertheless, the performance is stronger than DINOISER when k = 500 17 .

D Comparison of Runtime Speed

We also compare the runtime speeds of GENIE and AR-DIFFUSION by measuring the time taken for each sample generation, denoted as seconds per iteration (s/it). To ensure precision in our measurements, we sample 50 cases from the CNN/DAILYMAIL dataset. The generation process speed comprises two main components: function evaluation (FE, which is equivalent to the model forward computation) and MBR. All experiments are conducted on an A100-40G GPU and 50 CPUs.

The results are presented in Table 9. It is evident that for k ≤ 50 , the MBR duration is minimal and can be negligible relative to NFE. Furthermore, even when k = 500 and step = 20, AR-DIFFUSION remains approximately twice as efficient as GENIE ( k = 10 and step = 2000).

E Impact of Minimum Bayes Risk and Anchor Point

Minimum Bayes Risk To investigate the relationship between the number of generated candidate samples ( k ) and the quality of generation, we generate varying numbers of samples, ranging up to 1,000, on the IWSLT14 De → En test set and present the results in Figure 4. The curve demonstrates an initial gain of approximately 0.5 SacreBLEU within the first 200 samples, after which the gain becomes insignificant with generating more samples.

Anchor Point We conduct experiments on AR-DIFFUSION using different anchor points ( n e , t e ) . These anchor points vary in terms of n e values, namely 1 . 0 × N , 2 . 0 × N and 3 . 0 × N , where N denotes the target sentence length. Additionally, they share a common t e value of T , which represents

the total time step of diffusion. We present the results in Table 10, and determine that the best result is achieved at ( n e , t e ) = (2 . 0 × N,T ) .

32.4

SacreBLEU

32.2

32.0

31.8

200

400

600

800

1000

Sample Number (n)

F Proof of Inference with Skipping

During the inference process, skipping strategy requires the model g θ to infer the state z n 2 t i +1 at a far-off timestep t i +1 compared to the current state z n 2 t i , where t i +1 ≪ t i . In our model, due to the dynamic speed setting, token z n 1 t i +1 with smaller timestep t i +1 ≤ t i , which is closer to t i +1 , and positions n 1 ≤ n 2 can provide stronger auxiliary information than z n 1 t i . This reduces the difficulty of inferring states for tokens in the end, making our multi-level diffusion model particularly suitable for accelerating the generation process.

Through maximizing the evidence lower bound (ELBO) of p ( z 0 ) , the training object is equivalent to minimize the divergence between q ( z t | z t -1 , z 0 ) and p θ ( z t -1 | z t ) following [Luo, 2022].

By converting the joint probability distribution into a conditional probability distribution, we obtain the following formula for q ( z t i +1 | z t i , z 0 ) .

Similarly, we reach the same conclusion regarding p θ ( z t i +1 | z t i ) .

Based on equation (13), which consists of q ( z t | z t -1 , z 0 ) , and the interchangeability between q ( z t | z t -1 , z 0 ) and p θ ( z t -1 | z t ) , we can decompose q ( z t i +1 | z t i , z 0 ) by incorporating z t i and z 0 , and utilize our estimated z 0 to determine the expression of p θ ( z t i +1 | z t i ) .

Next, we obtain the explicit expression q ( z n f ( n,t i +1 ) | z n f ( n,t i ) , z n 0 ) through linear interpolation between z n f ( n,t i ) and z n 0 .

where we have the following notations for simplification.

Building upon equation (15), we substitute z n 0 with g θ ( z n f ( n,t ) , f ( n, t ); x ) , yielding the final formula for p θ ( z n f ( n,t i +1 ) | z n f ( n,t i ) ; x ) as the following equation.

G Different Sampling Methods of BART

The sampling methods used by Lin et al. [2023] including Greedy Search, Beam Search Xiao et al. [2022], Diverse Beam Search(diversity strength = 0.8) Vijayakumar et al. [2016], Typical Sample ( τ = 1.2) Meister et al. [2022], Top-k Sample ( k = 50) Fan et al. [2018] and Nucleus Sample ( p = 0.92) Holtzman et al. [2020].

Specifically, greedy search is to select the token with the highest probability at each step. Beam search is to select the largest token from among the beams with higher probability at each step. Diverse beam search is to divide the beams into multiple groups at each step and ensure the difference between groups by calculating the diversity score between groups. Typical sampling selects samples through a discrete random process. Top-k sampling is to randomly select one of the k candidate tokens with the highest probability at each step. Nucleus sampling is to randomly select one token at each step from the candidate tokens whose probability density is greater than p .

H More Cases

12.0

##ft hmm

る

northern hacker support by yells lion on [unused698]  tennis bars named

！

##s

つ

1898 1682ια limp

a british soldier who was killed by an army in 43 lifted losers requested ##ged a prosecutor  of verbal of rogers

য

.

a british soldier who was killed by an army in intersect has been named  upon a prosecutor  of deep ofise

ি

process.

a british soldier who was killed by an army in ash has been named by the ministry  of crucial  of championships  .

a british soldier who was killed by an army in afghanistan has been named by the ministry of defence of moddra .

14.0

16.0

18.0

20.0

12.0

proxy oxidation  ##ћ ##г

อ

ɬ ##ilaonate ##rwinpuri saskatoon  amplitude

ད

1702 ##

غ

the

ভ

##rvalstraß e ##

：

barcelona

a plaque for spain ' s first ##alis war amid been be in the historic  fleet of serumrnik say,  the ##ɹ of 2003 dso g fee . ##₍р

a plaque for spain ' s first world war has been added in the century city of liverpool say,  the certain of britain in g fee .

a plaque for spain ' s first world war has been be in the historic  city of liverpool say,  the amazing of serum in g post .

a plaque for spain ' s first world war has been unveiled in the historic  city of liverpool

, the first of britain in the years  .

14.0

16.0

18.0

20.0

12.0

##ₓ angelica

ნ

breach ##nostic [unused557]  fires sale why organizational  interception originates

ک

inial ##pa knots

china ' s prime minister creators  it concerning  the " 800

ک

in

མ

beach ' 1728 withdrawal  [unused308]  ##♦ attending

china ' s prime minister says it is the "ity emissions  " in the country  ' s [unused697]  foundation  , the engine of " .

china ' s prime minister says  it is the " emissions " in the country ' s tq crisis , the engine of " .

⺼

[unused887]√

china ' s prime minister says  it is the " emissions  " in the country  ' s economic  crisis , the engine of parliament  .

14.0

16.0

18.0

20.0

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a diffusion model on text generation. The idea is generally interesting. It learns to diffuse sentence-level and token-level diffusion, where the latter one is diffused with dynamic movement speeds.Its experiments are well-designed and its empirical results are strong.\n\nSTRENGTHS:\n1. The method is interesting  with a perspective to discuss autoregressive and non-autoregressive diffusion models.\n2. The skipping mechanism is useful to accelerate the generation process.\n3. The empirical results are strong.\n\nWEAKNESSES:\n1. The paper is hard to follow, and the writing should be improved.\n2. Limited diiffusion model baseline and missing some related baselines.\n\nQUESTIONS:\n1. I wonder if the skipping mechanism can also be applied to other diffusion models.\n2. In Algo2 line 6, where does n come from? Do you enumerate n? \n3. In Diffusion-LM, there is a rounding operation as Algo 2 line 9 at each diffusion step. It seems AR-Diffusion do not requires such rounding operation at each step. If true, would DPM-Solver help accelerating the diffusion process?\n4. GENIE is the main (almost the only) diffusion model baseline in the experiments, but some related baselines are mssing, such as DiffuSeq [1], CDCD[2], Difformer [3], DINOISER[4] and RDM[5]. It seems the authors already noticed some of them in the related work wonder, and I think it would be helpful to include more complete baselines.\n\nIf my conerns are addressed, I'm willing to raise my score. \n\nReferences\n1. Gong et al.  Diffuseq: Sequence to sequence text generation with diffusion models 2022\n2. Dieleman et al. Continuous diffusion for categorical data 2022\n3. Gao et al. Difformer: Empowering Diffusion Models on the Embedding Space for Text Generation, 2022\n4. Ye et al. DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises 2023\n5. Zheng et al. Reparameterized Discrete Diffusion for Text Generation 2023\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents AR-DIFFUSION, a diffusion model that displays auto-regression-like generation behavior. The primary contributions of this work can be summarized as follows:\n1)\tA multi-level diffusion strategy is proposed, encompassing both sentence-level and token-level diffusion.\n2)\tA skipping mechanism is introduced, which works in tandem with the multi-level diffusion strategy to expedite the process.\n3)\tThe superiority of the model over existing diffusion language models is verified in terms of text generation tasks and inference efficiency.\n\nSTRENGTHS:\nThe author presents an approach for integrating an auto-regressive-like mechanism into the diffusion model for text generation and has conducted comprehensive experiments to validate the efficacy of the proposed method. The idea of incorporating autoregressive dependency into the diffusion model is captivating.\n\nWEAKNESSES:\nI find the experiments in this paper insufficiently convincing.\n\nMy primary concern is that the main gains appear to be derived from MBR. With the skipping mechanism, the per-sentence generation step is reduced from 2000 to 20. This gives you chances to use K=500 for MBR, which is really large, as MBR is N^2 in computation.  The NFE metric is somewhat misleading, as it only considers the number of model forwards and does not account for the MBR process. It would be more appropriate to report the runtime speed of all methods for a fair comparison.\n\nThe comparison with baselines is not exhaustive. To substantiate the claim that their method benefits from introducing auto-regressive dependency back into text diffusion models, the authors should compare additional diffusion baselines, such as DIFFUSEQ and SeqDiffuSeq, with the same diffusion steps and candidates used in MBR, rather than just GENIE.\n\nMoreover, GENIE itself reported a Rouge-L score of 41.2 on CNN/Daily Mail, while the table in this paper shows only 32.1. This discrepancy should be clarified in the main text. If the lower score is due to a smaller parameter size, why not increase it to match GENIE's level? If the proposed method is effective, it could potentially be comparable to models like BART. As it stands, the parameter size of AR-Diffusion is too small to demonstrate its superiority.\n\nFurthermore, the comparison with NAR methods is not solid enough. Baselines such as CMLM and LevT were proposed three years ago. More recent methods like DA-Transformer should be included for a more comprehensive comparison.\n\nQUESTIONS:\nPlease answer the questions mentioned in the Weaknesses section.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper introduces a diffusion method optimized for the autoregressive text generation scheme. They employ different movement speeds for denoising with respect to the token positions. Specifically, they apply a lower movement speed to right-side tokens to guide models to reflect information in left-side tokens. Based on the dynamic movement speed method, they also propose a skipping mechanism during inference for efficient decoding. Experimental results show that the proposed method outperforms the previous diffusion-based approach at the same NFE, and the average performance drop is much lower in an extremely limited number of inference steps. In ablation experiments, they show that both AR-diffusion and the skipping mechanism are effective and the skipping mechanism can be applied to the other diffusion-based model.

STRENGTHS:
- The methodology is highly intuitive and well-motivated. 
- The proposed method is simple while mathematically supported and powerful. 
- They conduct experiments in various text-generation tasks to show not only consistent performance but also the efficiency of their method.
- The skipping mechanism can be effectively applied to the other diffusion model.

WEAKNESSES:
- Additional case studies comparing with GENIE or (N)AR models would provide further insights.

QUESTIONS:
- Have you compared the decoding speed between AR-Diffusion with an inference step of 2 or 3 and NAR models?
- Would AR-Diffusion also be robust to infilling tasks?

LIMITATIONS:
Since the model configuration of AR-Diffusion is based on Transformer-base in the paper, it would be possible to conduct a scalability study for various sizes in future work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces left-to-right sequential characteristics into diffusion models, enhancing the text generation performance of diffusion models. By considering the AR model as a diffusion model with two states: to be decoded and already decoded, AR-Diffusion defines a continuous diffusion model with decreasing diffusion speeds from left to right. Experiments on various text generation tasks show that AR-Diffusion achieves improvements over existing diffusion models.\n\nSTRENGTHS:\n1. The idea of introducing the left-to-right inductive bias into diffusion models for text generation is straightforward and reasonable.\n  2. By controlling the number of inference steps and generation candidates, AR-Diffusion can achieve tradeoff between quality and efficiency, which is more flexible than the Transformer.\n  3. Compared with the autoregressive model (BART), the generation of AR-Diffusion is more diverse\n\nWEAKNESSES:\n1. I think the authors overclaim the decoding speedup. First of all, most diffusion baseline models in the paper have no advantage in both generation quality and efficiency compared with the Transformer. Thus, AR-Difffusion should compare with the Transformer for decoding efficiency. Besides, existing diffusion models can achieve competitive results with much fewer steps. For example, Difformer[1] and Dinoiser[2] can achieve competitive scores with 20 steps, and Diff-GLAT[3] can even generate high quality sequences with only 3 steps. Therefore, I think more comprehensive experiments should be conducted to claim decoding speedup.\n  2. Although the AR-Diffusion achieves better BLEU scores than that of the Transformer in Table 3,  the results in Table 8 of Appendix C shows that the Transformer is still better than AR-Diffusion. Why are the results in the two tables contradictory? As SacreBLEU is a more standard metric for machine translation[4], does the results indicate that AR-Diffusion still lags behind the Transformer with a certain gap？\n\nQUESTIONS:\n1. Which decoding method does the Transformer use, greedy search or beam search? If beam search is used, what is the beam search size of each reported Transformer result?\n\nLIMITATIONS:\nAR-Diffusion requires a large number of candidates to achieve better results. Although generating hundreds of samples has large generation overhead, AR-Diffusion achieves promising results with MBR decoding.  I think the large number of candidates is not a serious issue in the current stage.\n\nReference  \n[1] Gao, Z., Guo, J., Tan, X., Zhu, Y., Zhang, F., Bian, J., & Xu, L. (2022). Difformer: Empowering diffusion model on embedding space for text generation. arXiv preprint arXiv:2212.09412.\n\n[2] Ye, J., Zheng, Z., Bao, Y., Qian, L., & Wang, M. (2023). Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.10025.\n\n[3] Qian, L., Wang, M., Liu, Y., & Zhou, H. (2022). Diff-glat: Diffusion glancing transformer for parallel sequence to sequence learning. arXiv preprint arXiv:2212.10240.\n\n[4] Post, M. (2018, October). A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers (pp. 186-191).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}