A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
Authors: Jiaao Chen
Aston Zhang
Mu Li
Alex Smola
Diyi Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Diffusion Language Model
Soft-masked Noise

Abstract: Diffusion models that are based on iterative denoising have been recently proposed and leveraged in various generation tasks like image generation. Whereas, as a way inherently built for continuous data, existing diffusion models still have some limitations in modeling discrete data, e.g., languages. For example, the generally used Gaussian noise can not handle the discrete corruption well, and the objectives in continuous spaces fail to be stable for textual data in the diffusion process especially when the dimension is high. To alleviate these issues, we introduce a novel diffusion model for language modeling, Masked-Diffuse LM, with lower training cost and better performances, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. Also, we directly predict the categorical distribution with cross-entropy loss function in every diffusion step to connect the continuous space and discrete space in a more efficient and straightforward way. Through experiments on 5 controlled generation tasks, we demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.

A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
Jiaao Chen Aston Zhang', Mu Li, Alex Smola, Diyi Yang Georgia Institute of Technology, Meet GenAl, "Stanford University
Abstract
more informative words to simpler or less infor- mative words through soft-masking. As a result, the models are encouraged to recover and generate the text following an easy-firs gesst-eenrattion nature (Dieleman et al., 2022) to improve the generation structure and quality with more flexibility. Also, during the diffusion process, we directly predict the discrete token with cross- loss that maps the continuous space discrete textual space to stabi- lize the intermediate diffusion steps. Through our proposed
Diffusion models that are based on iterative de- noising have been recently proposed and lever- aged in various generation tasks like image gen- eration. Whereas, as a way inherently built for continuous data, existing diffusion models still have some limitations in modeling discrete data, e.g., languages. For example, the generally used Gaussian noise can not handle the discrete corruption well, and the objectives in continu- ous spaces fail be stable for textual data in the diffusion process especially when the di- mension is high. To alleviate these issues, we introduce a novel diffusion model for language modeling, Masked-Diffusion LM, with lower training cost and better performances, inspired by linguistic features in languages. Specifically, we design linguistc-informed forward pro- cess which adds corruptions to the text through strategically soft-masking to better noise the textual data. Also, we directly predict the cat- egorical distribution with cross-entropy loss function in every diffusion step to connect the continuous space and discrete space in a more efficient and straightforward way. Through experiments on 5 controlled generation tasks, we demonstrate that our -Diffuuii LM achieve better generation quality than the state-of-the-an diffusion models with better ef- ficiency. Code is available at https: //g //gi thub. Com//AAL-NLPPP MLLLL--LLP/Masked__. Diwesseidddffuiiiinn___..
LM, the application- specific performance metrics as well as training efficiency are significantly improved over current diffusion language models based on experiments.
Our work is inspired by recent advances in dif- fusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021; Yang et al., 2022; Ramesh et al., 2022; Rombach al., 2022) that are introduced as a new generative modeling ap- proach based on iterative denoising and have achieved high-quality generations for visual and audio modalities (Ramesh et al., 2022; Rombach et al., 2022; Saharia et al., 2022; Nichol and Dhari- wal, 2021; Kong et al., 2020).
Although these approaches have received grow- ing attention and achieved impressive success, ap- plying diffusion models to textual domain is still challenging and under-explored due to the discrete nature of the text (e.g., one-hot vectors) compared to continuous data like images (e.g., RGB values) (Li et al., 2022). A few prior works (Li et al., 2022; Gong et al., 2022; He et al., 2022; Austin et al., 2021; Hoogeboom et al., 2021b) that explore using diffusion models on textual data can be divided into two lines. The first is to extend diffusion models to discrete state spaces (Austin et al., 2021; Hooge- boom al., 2021b,a), The second is perform the diffusion process and its reverse process in the continuous domain and bridge the continuous and the discrete domain through embedding and round- ing (Li et al., 2022; He et al., 2022), for example,
1 Introduction
We present a novel diffusion method for model- ing languages, Masked-Diffusion LM (language model), which uses strategic soft-masking in- formed by linguistic features to corrupt both the discrete and continuous space, and then iteratively denoise them back by predicting the categorical distribution. Specifically, strategic soft-masking process is designed that gradually adds perturba- tion to the input text in an order from harder or
Correspondence to Jiaao Chen <jiaaochen gatechh..uu and Aston Zhang <az @ aomonzhang..om

---

Diffusion-LM (Li et al., 2022). Despite the im- provements, most previous works fail to leverage the linguistic features (e.g., words in sentences are with different importance) to noise the input tex- tual data and recover it back in a more suitable way. Besides, they usually neglect or fail to adapt large pre-trained language models (PLMs) (Devlin et al.. 2019; Liu al., 2019; Yang et al., 2019; Joshi et al., 2019; Sun al., 2019; Clark et al., 2019; Lewis al., 2020; Bao et al., 2020; He al., 2020; Raffel et al., 2020), which is an unmissable treasure in the NLP community: their adopted k-nearest-nee neiihbo rounding technique that maps continuous space to discrete space cannot handle high-dimensional data in stable and efficient way (Li et al., 2022). As a result, a corruption process tailored for languages and the objective that allows efficient and straight- forward discrete and continuous space transforma- LM
images and audio (Ramesh et al., 2022; Rombach et al., 2022; Kong al., 2020; Savinov al., 2022). Despite their huge success, it is still challenging and under-explored to adapt diffusion models to dis- crete domains like languages. A few recent works have modified the diffusion models for textual data. For example, discrete forward processes. such categorical transition kernels (Hoogeboom et al., 2021a; Ye et al., 2023), uniform transition kernels, and absorbing kemnlss (Hoogeboom et al., 2021b), have been introduced. However, replacing con- tinuous diffusion with a discrete corruption pro- cess affords some flexibility (Dieleman et al., 2022; Zheng et al., 2023; Reid et al.. 2022). Other works have also made efforts to model text in the continu- ous embedding space and applied Gaussian noise uniformly to every token (Li et al., 2022; He et al., 2022; Chen and Yang, 2023), which is closer the settings in previous works of diffusion mod- els. However, they neglect the inherent linguistic features in the text (e.g., different words are play- ing different roles in sentences) so the generated text often lacks coherence (He et al., 2022). Be- sides, the k-nearest-neighhor neighbor rounding technique (Li et al., 2022; Gao et al., 2022) holds up the de- coding and convergence speed especially when the vocabulary is large or the hidden dimension is high, thus limiting the potential of combining large pre- trained language models (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Joshi et al., 2019; Sun et al., 2019; Clark et al., 2019; Lewis et al.. 2020; Bao et al., 2020; He et al., 2020; Raffel et al., 2020). To alleviate these issues, in our work, we introduce a linguistic-inforen cess corrupt the discrete and continuous space with structures, and then use linear projections and cross-entropy objectives to directly map the latent variables to textual data for better efficiency and generating better text.
tion is in great need. Our realizes this extension.
To demonstrate the effectiveness of our intro- duced Masked-Diffuusin LM, we perform experi- ments E2E dataset (Novikova et al., 2017) and 5 controllable generation tasks (Li et al., 2022) in- cluding Semantic Content, Pfrts-of-speech, Syntax Tree, Syntax Spans, and Length. We observe that our Diikkcd-iifuusin LM can (i) achieve the state- of-the-art performances compared to recent base- line models, and (ii) allow more efficient training and inference compared to previous Diffusion-LMM.
To summarize, our contributions are: W1) in- troduce a strategic masking noise strategy guided by linguistic features to corrupt the textual data in diffusion models for modeling languages. (2) We use linear layers and cross-entropy objectives to bridge the continuous and discrete spaces in the diffusion process for efficiency and stability. (3) We conduct experiments different controllable generation tasks demonstrate the effectiveness of our proposed methods compared to previous diffusion language models.
soft- soft-masking pro-
Non-Autoregressive Text Generation Most lan- guage models (Chowdhery et al., 2022: Brown et al., 2020) and text generation models (Vaswani et al., 2017a; Eikema and Aziz, 2021; Chen and Yang, 2020, 2021) follow a left-to-right autore-
2
Related Work
Diffusion Models for Language There has been growing attention in deep generative diffusion mod- cls, which is a latent variable generative method based on iterative denoising (Sohl-Dickstein et al., 2015; Ho al., 2020; Song al., 2021). Through a forward and diffusion process, diffusion models have shown state-of-fe of-the-arttt sample quality on gen- erating in the continuous domain such as producing
gressive manner. However, the fixed generation order prevents the models' flexibility in editing former text based on later generation results, espe- cially for global controllable generation settings. To overcome the limitations, non-autoregressive text modeling has been proposed (Ghazvininejad et al., 2019; Ren et al., 2020; Gu et al., 2018; Sa-

---

haria et al., 2020: Savinov et al., 2022) through masked language models (Ghazvininejad et al., 2019), iterative sequence alignment (Saharia et al..
This bridges the discrete space and continuous space. We will then add designed soft-masked noise to the tokens' representations in the later dif- fusion models.
2020), insertion and deletion (Gu et al., 2018), or unrolling the generation path (Savinov et al., 2022). Our Masked-Diffussien LM achieves the non-autoregressive generation through gradually recovering the intermediate latent variables in planned sequence from the forward process.
3.2 Forward Process with Soft-Masking
Different words in sentences play different roles. As result, when corrupting the sentences and re- covering the sentences, words with various impor- tance should be treated differently. Thus, in this work, instead of evenly adding Gaussian noise to all the token embeddings like in Diffusion-LMM (Li et al., 2022), we add soft-masked noise to different tokens in the input text in different stages to cor- rupt the text gradually with structures. Intuitively, more important words would be perturbed with soft-masks in an earlier stage so that the model could be encouraged generate them in the later phase to follow the -ir-firttggnerationn nature of language planning and generation.
Plug-and-Play Controllable Generation Our work is also closely related the line of research about souugandd-lay controllable generation meth- ods (Yang and Klein, 2021; Dathathri et al., 2020; Krause et al., 2021; Liu et al., 2021), which mod- ify the outputs based on extra guidance such as classifiers without changing or fine-tuning the pre- trained language models. Dathathri et al. (2020) used gradients to edit the autoregressive language model's hidden representations to fulfill the con- trol guidance. Yang and Klein (2021) proposed to reweight the predicted token from the language models while (Krause et al., 2021; Liu et al., 2021) further fine-tuned smaller LM to reweight the token predictions. In this work, we upply the gradient-based lung-and-payy approach to our Masked- Diffusion LM for controllable generation by making classifier-guided gradient updates the intermediate latent variables during the diffusion.
In this work, we consider the following aspects to measure and define the importance of words in one sentence:
Word Relevancy We use the tf-idf weights (Dessí et al., 2020), Wiriddr of the word as one way to measure the relevance of word to in one sentence d:
fwed
Wh--ar(w. (w. d) = Ewed
LM In this section, we describe our introduced Masked- Diffusion LM. The overall diagram shown Fig- ure I and Algorithm 1,2. Different from the recent diffusion models for languages, e.g., Diffusion-LMN (Li et al., 2022), which are based on continuous diffusion models, we propose to make corruptions in both discrete and continuous space to help mod- formulate
3 Method: the
N
log I+|ldED:
where the fw,d is the number of times that word occurs in sentence d, N is the number of sentences in the corpus, and D is the of sentences, and [d E D w d' is number of sentences where the word appears. A higher weight for word in sentence d in tf-idf means that the word might be more important in the sentence.
eling the textual data. Specifically, novel corruption process as an alternative to Gaus- sian diffusion (in Section 3.2) and we directly map continuous vectors to discrete inputs in every dif- fusion step with cross-entropy objectives (in Sec- tion 3.3). Moreover, our approach could easily inte- grate pre-trained language models (in Section 3.4).
We also consider measuring the amount of information with entropy H (Bentz and Alikani- otis, 2016; He et al., 2022) in the word w to reflect the importance of that word:
Entropy
3.1 Embedding
l1ww) -p (p ( log (p (w))
(3)
For the input sentence with I tokens = W1tt we first map the discrete tokens to the continuous space and form the initial latent variable, Xo. through a learnable embedding layer encoder e(.):
where (w) Ej=
represents the probability of word w and f is the word Reluency in the cor- pus. A word with lower entropy indicates that the word might contain less information and thus be
Xo = Whd = e(wix).
(1)

---

Diffusion Process po (xt-1 xx,
<m> <m> <m><>>
<m> <m>!
<m> is fun!
NLP is fun!
FFN
FFN
FFN
X
X
X
X
Encoder
<m> <M> <m><>>
<m> <m>!
<m> is fun!
NLP is fun!
Forward Process (xx+|xx+> -xt-))
Figure 1: The overall process of our Masked-Diffusion Diffusion LM. In the forward process, soft-mask is added to more informative words earlier to gradually corrupt the input text. For example, NLP is soft-masked prior to stop words like is. Then in the diffusion process, models learn to generate easy words like is first and then fill in more important words such as fun and NLP.
less important compared to the words with higher entropy.
Algorithm 1 Forward Process
Input A sentence X = [zo....., Fnl. Output Corrupted hidden representations Hr [ hnn.
In practice, we combine these two measures (with normalization) to decide the importance I of the word in one sentence d by:
I: Encode the sentence into hidden representa- tions via an encodere(.): Ho = e(X).
Fusarr(w d) [(w) = Ewred Wirsar(w' (w, d) + H(w) Eweed H(w))
2: for 1 = 1,.... K do
Add soft-masking noise to H based the importance of tokens (from higher- importance to lower-importance): H+41 soft-masking(H)) 4: end for
(4)
Based the introduced importance I of the words in a sentence, we first divide these words into m buckets [W The buckets with lower indices include words with higher importance. We will add most-masked noise to words with higher importance before words with lower importance. By doing this, models could learn to generate the easier words first and then generate harder words in the reversed denoising process for better generation quality. Specifically, every step
where s is a small constant that corresponds to the starting noise level. Thus, less noise would be added to harder words to stabilize the training. By performing the above noising steps, initial latent variable Xo is gradually corputed to series of noisy latent variables X1.T.
will add a small amount of Gaussian noise the hidden representation of the word w; in bucket W 1
3.3 Diffusion Process
g(wae (l = (( (11 - 3.).. B,T),
After the forward process to corrupt the input to- kens in sentences d into latent variables X we then gradually denoise Xr back to Xo through diffusion steps, 8,-1 = p(Xx10), where A is the learned parameter to model the state transition. In practice, model the transition with Transformers (Vaswani et al., 2017b).
(5)
where Si is the amount of noise added at diffusion step l.
We further apply a square-root noise schedule following Li et al. (2022) to gradually increase 3;:
After every diffusion step t E (0, T). instcad of minimizing the distance between the hidden rep-
3 = 1 vt/T+
(6)

---

Algorithm 2 Diffusson Process
Controllable Text Generation with Maaseed-iiftssion LM
Input Corrupted hidden representations H (ho....,huu Nn].
In this section, we illustrate how we apply our Masked-Diffusion LM to fulfill controllable text generation. Inspired by recent plug-and-play meth- ods (Yang and Klein, 2021; Dathathri et al., 2020; Krause et al., 2021; Liu et al., 2021), conduct controls from external modules (e.g., classifiers) directly on the latent variables X₁ in every inter- mediate step [O,T] T) in our Masked-Diffusion
Output sentence X = [zo...., [nn]
1: Utilize a transition network f(.) to recover the last state: H = [(H))
2: Utilize a linear layers to map hidden represen- tations to actual tokens X₁ g H 1)
3: Compute the loss Le and update the transition network.
4: Do the above steps until it recovers the sen- tence.
LM:
T
p (XoT c) = III p X), (7)
[=1
resentations of X and Xo (Li et al., 2022), we first directly map the continuous space to discrete space using learnable linear layer f(.) and then minimize weighted cross entropy between the predicted sentence and (i) the original sentence d and (ii) the masked sentence d time step /
We follow the conditional independence assump- tion (Yang and Klein, 2021; Dathathri et al., 2020; Krause et al., 2021; Liu et al., 2021) and decom- pose the above joint probability into sequence of control task at every time step t:
p (X.-1 Xx, e) x (X,-1 Xt) p(c X,-1,
p (X X₂) p(c X,-1).
L. = -==>(EE((8--))44 (1 (1=n))C((((-)))))).).. (0,
(8)
As a result, for the t-th step, we run gradient updates on X, to generate X1-1:
Here, ' = Izd. In other words, we put higher weights on the masked tokens that masked in this time step during the forward process and put lower weights the other tokens. So the models are learned to generate the corresponding masked tokens first at every time step.
v._, log (X,-1 X c) XV log (X X₂) + V Vx.- logp (c X(-1).
(9)
where both log (X, Xt) and log p(c are differentiable: the first term is parametrized by the transition Transformers, 0, in Masked-Diffuseee LM, and the second term is parametrized by extra neural network classifiers. Note that the extra clas- sifiers are trained with the diffusion latent variables as input to allow direct gradient updates on the la- tent space. Note that À is fluency regularization hyper-parameter to balance the fluency (gradient updates from Masked-Diffusiie LM) and control (gradient updates from classifiers) in order to fur- ther improve the generation quality.
3.4 Adapting Pre-trained Language Models
Our introduced Masked -Diffusionf LM also allows the use of large pre-trained language model (De- vlin et al., 2019; Liu et al., 2019; Yang al., 2019; Joshi al., 2019; Sun al., 2019; Clark al., 2019; Lewis et al., 2020; Bao al., 2020; He et al.,
2020; Raffel al., 2020). In this work, we use BERT (Devlin et al., 2019) as an example. To com- bine the prior knowledge in large language models, it is straightforward to directly replace the embed- ding layer e(.) with the pre-trained model and use the pre-trained model to get the hidden representa- tions of input tokens as the initial state in diffusion models. We the final linear layers in pre-trained models to predict the tokens. For efficiency, in our experiments, when using pre-trained models, we freeze the parameters in them and only learn the transition model in our Masked Diffussion LM.
For the decoding strategy, following Li et al. (2022), the Minimum Bayes Risk (MBR) decoding (Kumar and Bymee 2004) is used to aggregate and select the sample that has the lowest expected loss under the specified loss function from the Masked- Diffusion LMM
5 Experiments
5.1 Datasets
In this work, we train our Masked-Diffusiae LM on the E2E datasets (Novikova et al., 2017), which

---

Semantic Content Fluency
POS Fluency
Syntax Tree Fluency
Syntax Spans Fluency
Length Fluency
Methods
Acc
Acc
Acc
Acc
PPLM
9.9
5.32
FUDUGE
69.9
2.83
27.0
7.96
17.9
3.39
54.2
4.03
46.9
3.11
Diffusion-LM BERT
81.2
2.55
90.0
5,16
86.0
3.71
93.8
2.53
99.9
2.16
77.4
2.68
86.2
5.43
82.3
3.92
89.3
3.13
99.9
2,68
Masked-
Hasked-Diffusion Diffusion LM f
81.9
2.35
91.6
5.03
86.6
3.66
94.7
2.48
99.9
2.13
BERT
2.30
92.9
89.7
3.44
95.8
2.33
100
2.08
Table I: Main Results. The Accuracy (1) and the Fluency (L) of different methods on five controllable generation tasks including semantic content, POS, syntax tree, syntax spans and length. f indicates our methods.
Methods
Training (h)
Inference (s)
of the generated sentence should match the given syntactic category over the given spans. We evaluate the accuracy of the sentence by the exact match rate of the given spans.
Diffusion-Im
8.0
+BERT
15.2
Bosked-Diffusion LM
3,4
68
+BERT
4.8
700
Length. For a given target length (e.g.. 20). the models need to generate a sentence within # of the given target. We evaluate the accu- racy by the match rate of the sentence lengths.
Table 2:
Training time and inference time (generating 50 samples) for different models.
consists of 50K restaurant reviews together with the labels in terms food type, price, and customer ratings.
For every control task, we sample 200 control targets from the validation splits, and we gener- ate 50 samples for each control target. The first four tasks rely on a classifier guide the diffu- sion, and the last one task classifier free. To further evaluate the fluency of the generated sen- tences from models, we use a teacher LM (i.e., a carefully fine-tuned GPT-2 model) and report the perplexity of generated text under the teacher LM. A lower perplexity indicates better sample quality and fluency.
Following Li et al. (2022), we conduct 5 con- trol tasks to evaluate the learned Masked-Diffusion language model:
Semantic Content. For given field (e.g.. food) and value (e.g., Japanese), sentences that covers field=value need to be generated. We evaluate the accuracy of the generated sentence by examine the exact match rate of "value" (word mention).
5.2 Baselines
Parts-of-speech. For a given sequence of parts-of-speech (POS) tags (e.g., Noun Verb Determiner Noun), the models need to pro- duce the sentence with the same length and follow the exact given POS tag sequence (e.g.. Birds eat the warms). We evaluate the accu- racy of the generation by checking the word- level POS tag exact match (under an oracle POS tagger).
We compare our following state-of-the-ar baselines controllable generation tasks:
LM with the
PPLM (Dathathri al., 2020) runs gradient ascent on the pre-trained language models' hidden representations to increase the classi- fier probabilities and language model proba- bilities.
FUDGE (Yang and Klein, 2021) reweights the predicted tokens from the pre-trained lan- guage models by a discriminator which takes a prefix sequence and predicts whether the complete sequence would satisfy the con- straint.
Syntax Tree. For given syntactic parse tree, the generated sentence should have the same parse tree. We evaluate the accuracy by first parsing the generated sentence with an off-the- shelf parser and report the Fl scores compared to the given parse.
Diffusion-LM (Li et al., 2022) learns an em- bedding to map discrete text into the con- tinuous space where performs Gaussian
Syntax Spans, For a given (span, syntactic category) pair (e.g., (2, 5, VP)), the parse tree

---

Methods
Semantic Content
POS
Syntax Tree
Syntax Spans
Length
Diffusion-Im
2.89
2.76
3.16
2.88
2.46
+BERT
3.87
3.46
3.72
3.68
3.34
Mafsed-Diffusion LM
2.56
2.48
2.88
2.35
2.18
+BERT
1.32
1.28
1.16
1.55
1.86
Table 3: The average ranking every method receives from human evaluation (lower is better).
Semantic Content
5.4 Results
Noise Type
Acc
Fluency
We show the main results on five controllable gener- ation tasks in Table 1. When the diffusion process is engaged, the performances all the controlled generation tasks receives significant boosts (e.g.- 81.2 of Diffusion-L.M vs. 69.9 FUDUGE on Semantic Content task), suggesting the superior- ity of the diffusion model on controllable genera- tion tasks. While the previous Diffusion-LM can not be well combined with large language model like BERT (e.g., 5% drop on Semantic Content accuracy), largely due to the fact that their way (rounding) bridge continuous space and discrete space suffers from significantly higher dimensions. Compared to Diffusion-LLM, our proposed Masked- Diffusion LM consistently outperforms the previ- ous models in all tasks (c.g.. a 1.7% improvement on the POS task), indicating the effectiveness of our introduced linguistic-infor innioseme noise forward process. Also, when combined with large language models like BERT, our method significantly out- performs the previous methods, demonstrating that our approach can be well aligned with pre-trained models.
Gaussian
75.3
3.01
Random Mask
78.8
2.67
Mask w. POS
80.4
2.58
Mask w. Entropy
81.1
2.44
Mask w. Rel
80.8
2.52
Mask w. Entropy+Rel f
81.6
2.38
Table 4: Performances Semantic Content of Masked- Diffusion LM with different types of noise applied in forward noising process. indicates our method.
diffusion process, Also, a rounding step is designed to map the embeddings back into discrete texts. For every control task, the Diffusion-LM infuses the controlling signals in every diffusion step.
5.3 Experimental Setting
We use a Transformer with 80M parameters to pa- rameterize our -ifffud-iiffsii LM, with a se- quence length n = 64, diffusion steps = 500, and a square-root noise schedule. For Masked- Diffusion LM, we set the hidden dimension to 128. We set the number of word buckets 3. When combining pre-trained models, we in- corporate BERT-base (Devlin al., 2019) with about 110M parameters. We use BERT to en- code the input text into vectors with dimension of 768 and freeze the parameters in BERT. We learn Masked- Diffuusioifff LM with the AdamW op- timizer (Loshchilov and Hutter, 2019) for 20,000 steps with learning rate of 3c-4, dropout probabil- ity of 0.1, and batch size of 32. We use linear warmup schedule starting with 1,000 warmup steps. All experiments are conducted on NVIDIA A 100 Tensor Core GPUs. We use 4 GPUs for training and single GPU for sampling.
Efficiency We also display the training cost and inference cost Table Compared to the previous Diffusion-LM, our method requires significantly less training time to converge and needs less infer- ence time to generate sentences. This is because our introduced noise process is more stable and suitable for modeling languages. Besides, the ob- jectives introduced are more efficient than the rounding techniques in previous work.
Human Evaluation We then conduct human evaluation to evaluate the generated conversations qualitatively. We ask native speakers of English from Amazon Mechanical Turk to rank the qual- ity of 50 generated sentences (randomly sam- pled) from different models for every control task. Specifically, annotators need to rank different sys- tem outputs based on the (i) fluency (whether the

---

Semantic Content fluency
POS fluency
Syntax Tree fluency
Syntax Spans fluency
Length
Methods
Ace
Acc
Acc
Acc
Acc
fluency
L2 L2-BERT
81.1
2.44
90.6
5.17
86.2
3.68
94
2.51
99.8
2.14
80.1
2.48
89.4
5.82
84.1
3.91
93.2
2.88
99.9
2.89
CE 1 CE-BERT
81.9
2.35
91.6
5.03
86.6
3.66
94.7
2.48
99.9
2.13
82.9
2.30
92.9
4.78
89.7
3.44
95.8
2.33
100
2.08
Table 5: Performances of Masked-Diffusion LM trained with different objectvies controllable generation tasks. f indicates our method.
Case Study
Sentences
Input
{ = 500
[mask] [mask] [mask] [mask] [mask] [mask]] [mask] [mask] is an [mask] restaurant
f = 400
200
The [mask] is an Indian restaurant
t 0
The Mill is an Indian restaurant.
Input
name:
Travellers Rest Beefeater
500
[mask] [mask]] [mask] [mask] [mosk] [mask] [mask] [mask] [mask] [mask] [mask] [mask]
400
[mask] Rest [mask] [mask]] [mask] [mask] that is [mask]]
200
Travellers Rest [mask] is reasonably [mask] restaurant that awesome
Travellers Rest Beefeater is reasonably priced restaurant that awesome
Table 6: Examples of the intermediate generated text of our Content tasks.
LM on the Length and Semantic
given sentence is readable and fluent) and (ii) the controllability (whether the given sentence match the given control conditions). To increase anno- tation quality, we require turkers to have 98% approval rate with over 10,000 approved tasks for their previous work. The pay rate $0.15 per hit. Every example is assessed by 3 annotators, and the rank for every sentence aggregated by majority voting. The Intra-Class Correlation (ICC/k) was 0.63, indicating moderate agreement (Koo and Li, 2016). The results are shown in Table 3. As shows, our proposed Masked-Diffusion LM and its variation with BERT received the best average ranks, suggesting the effectiveness of our proposed diffusion modeling strategy for languages.
order. (ii) Mask with POS, where the soft-mask per- turbs the tokens in an order (noun + verb - other words) based on POS tags. Our introduced noise strategy (Mask with Entropy and Reluency) shows significantly better performances semantic con- tent generation. This indicates that our introduced noise strategy that considers the linguistic features in sentences is providing more uppropriate pertur- bation to the textual data for the diffusion process.
We further show the impact of differ- ent objectives in Table 5. We compare our used cross entropy objectives with the Le object that is used in Li et al. (2022) where they minimize the distance between latent intermediate variables and the initial latent variable instead of directly predicting the text. We observe that cross entropy objectives slightly perform better than L2 when the pre-trained model is not used. After combin- ing with large language models, CE-BERT signif- icantly outperforms the L2-BERT indicating the effectiveness of our introduced objectives in terms of incorporating large language models.
Objectives
5.5
Ablation Studies
We then perform ablation studies to demonstrate the effectiveness of our introduced linguistic- informed noise and the cross entropy objectives.
Noise Strategy We first demonstrate the per- formances on Semantic Content task of Masked- Diffusson LM with different types of noise strategy in Table Gaussian adds Gaussian noise all the tokens in the input sentence in the forward process following Li al. (2022). We also compare dif- ferent masking noise strategies: (i) Random Mask, where the soft-mask is added to tokens in a random
5.6
Case Studies
We also include some examples of intermediate steps of Misked-Diffusion LM in Table 6. In the de- noising diffusion process, casy words are generated first. For example, "is". "an", and "restaurant".

---

With more diffusion steps, sentences are enriched with more informative words such as "Mill" and "Indian". It shows that our Masked-Diffusion LM encourages the generation to follow an easy-first order for stable and better generation quality.
Yang. Xiaodong Liu, Yu Wang. Songhao Piao, Jian- feng Gao, Ming Zhou, et al. 2020. Unilmv2: Pseudo- masked language models for unified language model pre-training. arXiv preprint arXiv:2002.12804.
Christian Bentz and Dimitrios Alikaniotis. 2016. The word entropy of natural languages.
6 Conclusion
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Tiii Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner. Sam McCandlish, Alec Radford, Ily Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
In this work, we present a novel diffusion model for language, Deekku--iiffusion LM, which corrupts the discrete text with linguistic-ifforme soft- masking strategy and then itcratively denoises them back by directly predicting the text. Specifically, we gradually soft-mask the tokens in the sentence following an order from more informative words to less informative words in the forward process. This satisfies the flexibility for diffusion models, as well as encourages the casy-first gurst-eeneration nature in the denoising process for better generation quality. Also, we directly predict the discrete token during the diffusion process with the cross-entropy loss to stabilize the intermediate diffusion steps and make approach orthogonal large pre-trained language models. Experiments on E2E dataset and five controllable generation tasks including Semantic Content, Parts-of-speech, Syntax Tree, Syntax Spans, and Length show that our Masked- Diffusion LM can (i) achieve the state-of-the-an performances compared to recent baseline models and (ii) allow more efficient training and inference compared to the previous Diffusion-LM.
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence- to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 4106- 41 4118, Online. Association for Computational Lin- guistics.
Jiaao Chen and Diyi Yang. 2021. Structure- aware ab- stractive conversation summarization via discourse and action graphs. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 1380-1391, Online. As- sociation for Computational Linguistics.
Jiaao Chen and Diyi Yang. 2023. Controllable con-
versation generation with conversation structures via diffussion models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7238- 7251.
7 Limitations
In this work, we mainly leverage linguistic soft- masking such as word relevancy and word entropy. We encourage future work to explore how to in- corporate other linguistic structures design the nosing process. And we mainly test with smaller models like simple transformer models as well as BERT-based models. Future work might test with larger pre-trained models to evaluate whether dif- fusion methods would work better or not. Also, we focused on controllable generation to evaluate the models. Future work may study different down- stream tasks.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury. Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan. Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lec, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean. Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
References
Jacob Austin, Daniel D. Johnson, Jonathan Ho. Daniel Tarlow, and Rianne van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan

---

Kevin Clark, Minh-Thare Luong, Quoc V Le, and Christopher D Manning. 2019. Electra: fre-training text encoders as discriminators rather than generators. In International Conference on Learning Representa- lions.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models.
Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bast- ings, Ben Poole, Rianne van den Berg. and Tim Sali- mans. 2021a. Autoregressive diffusion models.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: simple approach to controlled text generation. In International Conference on Learning Representa- tions.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. 2021b. Argmax flows and multinomial diffusion: Learning categori- cal distributions.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predict- ing spans. Transactions of the Association for Com- putational Linguistics, 8:64-77.
Danilo Dessí, Rim Helaoui, Vivek Kumar, Diego Re- forgiato Recupero, and Daniele Riboni. 2020. Tf-idf vs word embeddings for morbidity identification in clinical notes: An initial study.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. NATCL-HLT.
Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. 2020. Diffwave: A versatile diffu- sion model for audio synthesis.
Terry K Koo and Mae Y Li. 2016. A guideline of selecting and reporting intraclass correlation coeffi- cients for reliability research. Journal of chiropractic medicine, 15(2):155-163.
Sander Dieleman, Laurent Sartran, Arman Roshan- nai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnand Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler 2022. Continuous diffusion for categorical data.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. GeDi: Gener- ative discriminator guided sequence generation. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 4929 4952, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Bryan Eikema and Wilker Aziz. 2021. Sampling-based approximations to minimum bayes risk decoding for neural machine translation.
Zhujin Gao. Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. 2022. Difformer: Empowering diffusion model on embedding space for text generation. arXiv preprint arXiv:2212.094/
Shankar Kumar and William Byrne 2004. Minimum Bayes-risk decoding for statistical machine transla- tion. In Proceedings of the Human Language Tech- nology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169-176. Boston, Mas- sachusetts, USA. Association for Computational Lin- guistics.
Marjan Ghazvininejad. Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel de- coding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing (EMNLP-IJCNLP), pages 61 12- 6121, Hong Kong, China. Association for Computa- tional Linguisties.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke Zettlemoyer. 2020. Bart: De- noising soguence-to-equemne pre-training for natural language generation, translation, and comprehension. SCL.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2022. Diffuseq: Sequence to sequence text generation with diffusion models.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani. Percy Liang, and Tatsunori Hashimoto. 2022. Diffusson- Im improves controllable text generation.
Jiatao Gu. James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Nonsaveoreeressive neural machine translation. In International Confer- ence on Learning Representations.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta. Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. DExperts: Decoding-tine time con- trolled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages Online. Association for Computational Linguistics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654
Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuan- jing Huang, and Xipeng Qiu. 2022. Diffusionbert: Improving generative masked language models with diffusion models.

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zet,lemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint vrrXiv:190..11 1669..
Jascha Sohk-Dickstein, Eric Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2013. Deep un- supervised learning using nonequilibrium thermody- namics. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Pro- ceedings of Machine Learning Research, pages 2256- 2265, Lille, France. PMI.R
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021. Denoising diffusion implicit models. In International Conference on Learning Representations.
Alex Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced represen- tation through knowledge integration. arXiv preprint (504.09900022
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end- to-end generation. In Proceedings of the 18th An- nual SIGdial Meeting on Discourse and Dialogue, pages 201-206, Saarbrücken, Germany, Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017a. Attention is all you need.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017b. Attention is you need. In Advances in neural information processing systems, pages
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu. and Mark Chen. 2022. Hierarchical text- conditional image generation with clip latents.
Kevin Yang and Dan Klein. 2021. FUDGE: Controlled text generation with future discriminators. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 351 1-335,,,, Online. Association for Computational Linguistics.
Machel Reid, Vincent Hellendoomn, and Graham Neu- big. 2022. Diffuser: Discrete diffusion via edit-based reconstruction. arXiv preprint arXiv:2210.1688
Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, and Tie-Yan Yan Liu. 2020. study of non- autoregressive model for sequence generation.
Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong. Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. 2022. Dif- fusion models: A comprehensive survey of methods and applications.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High- resolution image synthesis with latent diffusion mod- els. In 2022 JEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674- 10685.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ Salakhutdinov, and Quoc V Lc. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In Advances in neural informa- tion processing systems, pages 9754-5764.
Chitwan Saharia, William Chan, Saurabh Saxena. Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans, Jonathan Ho. David J Fleet, and Mohammad Norouzi. 2022. Pho- torealistic texi-to-image diffusion models with deep language understanding.
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. 2023. Dinoiser: Diffused con- ditional sequence learning by manipulating noises. arXiv preprint 2rxiv:2302.10005.
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. Non-autoregressive ma- chine translation with latent alignments. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1098-1108, Online. Association for Computational Linguistics.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong- 2023. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737.
Nikolay Savinov. Junyoung Chung. Mikolaj Binkowski, Frich Elsen, and Aaron van den Oord. 2022. Step- unrolled denoising autoencoders for text generation. International Conference on Learning Representa- tions.


Reviews:
[{"review_id": "X60NWhcobS", "reviewer": "Reviewer_Q6XL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and constructive reviews. \n\n### On the selection of the evaluation sets and testing multilingual datasets\nWe mainly follow the previous work [1] to evaluate and compare different methods on 5 controllable generation tasks. In terms of the multilingual settings, we think our methods would still show improvements because: (1) our introduced strategical masking noise are based on the word relevance and entropy which are agnostic to language variations, and (2) our methods would also be easily to different pre-trained multilingual models to handle multilingual cases such as machine translations.\n\n### On the comparison to other relevant work\nWe have discussed the major comparison to previous discrete and continuous diffusion models in the language model domain [1,2,5] in the introduction and related work section: our work utilize soft-masking informed by linguistic features to corrupt both the discrete and continuous space, and then iteratively denoise them back by predicting the categorical distribution. Compared to discrete diffusion models like [5,6,7], we model the corruption process in the continuous space which enables flexible corruption and controllable reconstruction by manipulating the hidden representations which allow classifier-free guidance or gradients-based updates for conditional/controllable generation. And our soft-masking noise also shares the advantages of the discrete diffusion process to guide the corruption by linguistic structures (e.g., soft-masking tokens based on the importance. ). Compared to continuous diffusion models like [1,2,3,4], our methods have two major improvements: (1) we introduce a more structured noise guided by linguistic features that noise the discrete and continuous space at the same time instead of random Gaussian noise [1,2,3,4]; (2) we directly predict the categorical distribution during the denoising process to bridge the continuous and discrete space instead of rounding techniques [1,3] which allows faster convergence and the ability to combine large pre-trained models. We would like to expand these comparisons in our revised version.\n\n[1] Li, Xiang, et al. \"Diffusion-lm improves controllable text generation.\" NeruIPS (2022).\n\n[2]Dieleman, Sander, et al. \"Continuous diffusion for categorical data.\" arXiv preprint arXiv:2211.15089 (2022).\n\n[3]Gao, Zhujin, et al. \"Difformer: Empowering diffusion model on embedding space for text generation.\" arXiv preprint arXiv:2212.09412 (2022).\n\n[4] Ye, Jiasheng, et al. \"Dinoiser: Diffused conditional sequence learning by manipulating noises.\" arXiv preprint arXiv:2302.10025 (2023).\n\n[5] He, Zhengfu, et al. \"Diffusionbert: Improving generative masked language models with diffusion models.\" ACL (2023).\n\n[6] Zheng, Lin, et al. \"A reparameterized discrete diffusion model for text generation.\" arXiv preprint arXiv:2302.05737 (2023).\n\n[7] Reid, Machel, Vincent J. Hellendoorn, and Graham Neubig. \"Diffuser: Discrete diffusion via edit-based reconstruction.\" ICLR (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The linguistic-informed masking is intuitive and shows clear benefits in ablations. Gradual masking encourages generating simple words first.\n2. Results show state-of-the-art performance on multiple controlled generation tasks compared to previous diffusion LMs and other methods. \n3. The approach is compatible with pretrained LMs.\n", "reasons_to_reject": "1. This paper only evaluate the proposed approach on controllable generation tasks. The paper would be strengthened by evaluating the proposed model in a multilingual setting, which has been commonly used in prior work on diffusion models for text generation. For example, several previous[1,2,3,5] studies have experimented with machine translation tasks like English-to-German translation on the WMT14 dataset. Applying the current model to translation or other multilingual datasets could further demonstrate its capabilities and allow for comparison to these prior methods. The lack of multilingual evaluation is a concern since the current benchmarks, while varied, are limited to monolingual English tasks.\n2. The paper's comparisons to other relevant work could be expanded. In the experiments, only a couple recent diffusion models were compared against. Given significant recent progress in text diffusion modeling, more in-depth discussion of how the proposed model relates to other continuous diffusion methods (e.g. CDCD[1], Difformer[2], DINOISER[3]) and discrete diffusion methods (e.g. DiffusionBERT[4], RDM[5], DiffusER[6]) would strengthen the paper. The contributions could be better situated by clarifying the advantages of the current model compared to these other related studies. \n\n---\n[1] Continuous diffusion for categorical data\n\n[2] Difformer: Empowering Diffusion Model on Embedding Space for Text Generation\n\n[3] DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises\n\n[4] DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models\n\n[5] A Reparameterized Discrete Diffusion Model for Text Generation\n\n[6] DiffusER: Discrete Diffusion via Edit-based Reconstruction", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new diffusion model for language modeling called Masked-Diffusion LM, where the authors designs a linguistic-informed soft masking noise process during the forward diffusion that gradually masks words from more informative to less informative. \nFor more efficient and stable training for text diffusion, they use cross-entropy loss to directly predict tokens instead of L2 loss on embeddings. The model also allows incorporating large pretrained language models like BERT by freezing BERT and learning the diffusion process.\nExperiments are mainly conducted on controllable text generation showing the effectiveness of the proposed approach over Diffusion-LM\n\n**Main contributions:**\nDeveloping a diffusion LM that is tailored for language with soft masking and direct discrete prediction, and showing strong performance on controllable text generation tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ImJqSfOeQE", "reviewer": "Reviewer_h85U", "summary": "", "questions": "Q1: In the diffusion forward process, the paper indicates that the hidden representation of tokens is added by a small amount of Gaussian noise at each time step t. Are latent variables $\\hat{X}_{t-1}$ corrupted by both Gaussian noise and MASK? An ablation study is needed to analyze their effects respectively. \n\nQ2: In the loss function $\\mathcal{L}_{t}$, the corrupted tokens are forced to learn two different targets, i.e. the original sentence $d$ and the masked sentence $\\hat{d}$, which is an ambiguous and unstable target for the model.\n\nQ3: When adapting BERT to the Masked-Diffusion LM, the parameters of BERT are frozen. Does it mean only the parameter of the final linear layer is updated or a new randomly initialized transformer (transition model) is built upon the BERT?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comment. \n\n### On the reverse process\nIntuitively, for our reverse process, different from DDPM, we directly predict the hidden representation of the target sequence at every step $t$, $X_{t-1} = f(X_{t})$ where $f(.)$ is a transition network such as a transformer, $X_{t}$ is the hidden representation of the sequence at step $t$. This can be viewed as predicting the representations of the tokens that are soft-masked at the step $t$ during the corruption process based on the recovered hidden representation by $f(.)$. And we further learn the feedforward layer $g(.)$ to predict the actual tokens based on the hidden representations $S_{t-1} = g(X_{t-1})$ to provide supervision to learn the transformer. We would elaborate more details on the reverse process in our revised version.\n\n### On the application to Non-bidirectional LMs\nOur methods can be also applied to Non-bidirectional LMs such as GPT. As illustrated in Section 4.4 and Figure 1, we could just replace the encoder with GPT-based models to map the discrete tokens into continuous hidden representations. Then we applied the soft-masked corruption process and the reverse process to reconstruct the sequence. In the reverse process, we learn the transition network from a randomly initialized transformer based on the token predictions. Such a process does not involve any pre-trained transformers except the final linear layers to predict the tokens. As a result, our method does not rely on bidirectional LMs and can also be combined with autoaggressive LMs.\n\n### On the questions about the forward process\nOur soft-masked noise can be viewed as a combination of both Gaussian noise and masking tokens. Specifically, in our forward process with soft-masked noise, compared to previous methods which add Gaussian noise to all the tokens representations at every time step, our method can be viewed as only adding a small amount of Gaussian noise to selected tokens at that time step (selected based on the word importance) which allows easy-first-generation nature in the reverse process. Also, instead of directly transforming the representations to 0s (the hard-masked noise adding to the discrete space) which create a large amount of noise at every time step, our soft-masking noise allows smaller corruption to the continuous space.\n\n### On the questions about the loss function\nOur loss function at time step $t$ can be viewed as two parts: predicting the tokens that are not masked at $t$ and predicting the tokens that are masked at $t$. And we put higher weights on the later part. So the models would learn to predict the masked tokens first at every time step.\n\n### On the questions about the learned parameter when adopting BERT\nWhen utilizing BERT, we freeze the pre-trained parameters in the BERT models and learn a new randomly initialized transformer which is built upon the BERT to serve as the transition model for the reverse process （recovering the corrupted hidden representations).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Well motivated. The hybrid diffusion model that makes corruptions in both discrete and continuous space is a novel method and worth further exploring.\n2. Integrating pre-trained LM with diffusion models is a promising exploration.\n3. The experiments comprehensively demonstrate the improvement of performance compared to baselines.", "reasons_to_reject": "1. The paper lacks a description of the reverse process of the proposed soft-masking diffusion LM. The proposed method corrupts the textual data with both continuous Gaussian noise and discrete MASK tokens, which is different from typical diffusion models like DDPM. It is obviously impossible to directly apply the DDPM's reverse process to this method. \n\n2. Non-bidirectional LMs like GPTs are not pre-trained with the Mask strategy, which is not compatible with the proposed soft-masking diffusion LM. The method could only be applied to LMs like BERT, which limits the possibility to scale up to LLM like GPT3.", "questions_to_authors": "Q1: In the diffusion forward process, the paper indicates that the hidden representation of tokens is added by a small amount of Gaussian noise at each time step t. Are latent variables $\\hat{X}_{t-1}$ corrupted by both Gaussian noise and MASK? An ablation study is needed to analyze their effects respectively. \n\nQ2: In the loss function $\\mathcal{L}_{t}$, the corrupted tokens are forced to learn two different targets, i.e. the original sentence $d$ and the masked sentence $\\hat{d}$, which is an ambiguous and unstable target for the model.\n\nQ3: When adapting BERT to the Masked-Diffusion LM, the parameters of BERT are frozen. Does it mean only the parameter of the final linear layer is updated or a new randomly initialized transformer (transition model) is built upon the BERT?", "paper_topic_and_main_contributions": "The paper introduces a linguistic-informed soft-masking strategy in diffusion models for text generation, where important words would be perturbed with masks in an earlier stage. The proposed diffusion strategy combines discrete and continuous space and corrupts both of them during the diffusion forward process to model the textual data. The authors also explore the integration of pre-trained LM like BERT to further improve performance.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eWrYysfhOn", "reviewer": "Reviewer_YAfc", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. We hope that our response can clarify the misunderstandings and you can consider our work more favorably in the ratings.\n\n### On the comparison to discrete diffusion techniques\nFirst, discrete diffusion methods can not allow  classifier-free guidance or gradient-based methods for conditional generation[1,2], which has been shown to be instrumental to the success of diffusion-based text-conditional image generators. In discrete methods, the corruptions are usually word masking/substitution in the discrete space. As a result, in the reverse process, it is infeasible to manipulate the controlling without the guidance of classifiers or perturb the hidden representation directly for controllable generation in the continuous space. Also, compared to discrete methods, continuous diffusion models provide more smoother transitions which provide smoother transitions between different states or levels of diffusion. This enables the examination of diffusion processes at any point in time, rather than being restricted to specific time intervals defined by discrete models. This allows better interpretation of the generation process as shown in our Table 6.\n\nBecause of this, previous studies on continuous diffusion models [1,2] do not consider discrete diffusion models as baselines on conditional/controllable generation tasks. We follow [1] to mainly utilize well-known controllable generation methods together with the Diffusion-LM as our baseline to compare. \n\nThird, our experiments provide several insights to show the effectiveness of our proposed methods: (1) The main results in Table 1,3 as well as the inference time in Table 2 demonstrate the better performance and  efficiency compared to previous Diffusion-LM. Also, our methods can be better combined with LLMs. (2) Our ablation studies in Table 4 and 5 clearly demonstrate the effectiveness of the key design (soft-masked noise and objectives to predict the categorical distribution). (3) Our case studies further provide visualization and interpretation of the diffusion process where tokens are generated from an easy to hard manner. \n\n### On the pre-trained model dependencies\nFirst, we would like to highlight that our methods boost the performance consistently with/without pre-trained models as shown in Table 1 in our paper in terms of automatic evaluations. We also emphasize the average rankings different methods receive from human evaluation (lower is better) in the table below (also in table 3). It is noteworthy that in both settings, our Masked-Diffusion-LM receives better ranks compared to Diffusion-LM. This suggests that our methods would improve the generation even without any pre-trained models. So our method does not depend on the use of pre-trained models. \n| Models              | BERT | Semantic Content | POS      | Syntax Tree | Syntax Spans | Length   |\n|---------------------|------|------------------|----------|-------------|--------------|----------|\n| Diffusion-LM        | No   | 2.89             | 2.76     | 3.16        | 2.88         | 2.46     |\n| Masked-Diffusion-LM | No   | **2.56**         | **2.48** | **2.88**    | **2.35**     | **2.18** |\n| Diffusion-LM        | Yes  | 3.87             | 3.46     | 3.72        | 3.68         | 3.34     |\n| Masked-Diffusion-LM | Yes  | **1.32**         | **1.28** | **1.16**    | **1.55**     | **1.86** |\n\nWhat’s more, in sharp contrast to previous work [1], our methods can be more effectively combined with pre-trained transformer models to utilize the advances in LLMs. Because we directly predict the categorical distribution instead of utilizing k-nearest-neighbor rounding technique to map continuous space to discrete space. This brings in better stability to handle high-dimensional representations from LLMs.\n\n[1] Li, Xiang, et al. \"Diffusion-lm improves controllable text generation.\" Advances in Neural Information Processing Systems 35 (2022): 4328-4343.\n\n[2]Dieleman, Sander, et al. \"Continuous diffusion for categorical data.\" arXiv preprint arXiv:2211.15089 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Here are some of the main strengths and benefits of this paper:\n\n- Novel diffusion modeling technique for text: The proposed masking and discrete prediction scheme provides a new way to adapt diffusion models to better suit textual data. This contributes to advancing diffusion models for NLP.\n\n- Improved controllable text generation: The experiments demonstrate state-of-the-art performance on several controllable text generation tasks. This is a useful generation capability for applications.\n\n- The proposed techniques allow faster training and inference compared to prior diffusion LMs, improving efficiency.\n\n- Well-written: The paper is clearly written and provides sufficient background and explanation of the new techniques. This makes it accessible to the NLP community.", "reasons_to_reject": "Here are some points to keep in mind:\n\n- No comparison to other discrete diffusion techniques: The paper compares mainly to continuous diffusion methods for text. Including comparisons to other discrete techniques could offer more informative insights.\n\n- Pre-trained model dependencies: The benefits of this paper rely on transformer-based pre-trained models. It may be helpful to test the robustness of different model choices.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel diffusion model for generating text, called Masked-Diffusion LM. This model has several key contributions. Firstly, it uses a linguistic-informed soft masking process during the forward diffusion process. This means that important words, based on TF-IDF and entropy, are masked earlier, so that the model learns to generate them later. This encourages an easy-first generation. Secondly, it employs a cross-entropy loss to directly predict tokens and map the continuous latent space to discrete tokens during diffusion. This is more efficient than previous rounding techniques. Finally, Masked-Diffusion LM achieves state-of-the-art performance compared to prior diffusion LMs in experiments on controllable text generation tasks, such as enforcing semantic content and POS tags. It also allows for more efficient training and inference.\n\nIn summary, this paper proposes a linguistically-motivated masking scheme for the text corruption process in diffusion models and a direct discrete token prediction technique during diffusion that improves efficiency. Evaluation on controllable generation tasks demonstrates improved performance over prior diffusion LMs for text.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JoSVb23uAk", "reviewer": "Reviewer_4Jws", "summary": "", "questions": "1. Do the authors try to build this method upon LLM? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive assessments. \n\n### On the questions about the use of LLM\nYes, our methods can be easily built upon LLMs like BERT as stated in the Section 4.4 and Table 4. We could just replace the encoder (shown in Figure 1) with any LLMs to map the discrete tokens into continuous hidden representations. Then we applied the soft-masked corruption process and the reverse process to reconstruct the sequence. In the reverse process, we learn the transition network from a randomly initialized transformer based on the token predictions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method for computing the importance of the word in a sentence is very interesting, and combining the importance of the word with the forward process is very ingenious.\n\n2. Mapping the continuous space to discrete space is very simple and elegant for alleviating the problem led by discrete corruption during the diffusion process.\n\n3. The analysis and comparison, like in Table 4, sufficiently show the effectiveness of the proposed approach.", "reasons_to_reject": "I do not see any risk.", "questions_to_authors": "1. Do the authors try to build this method upon LLM? ", "paper_topic_and_main_contributions": "This paper proposed a novel way of adding noise during the forward process of the diffusion language model, which first proposes a method to compute the importance of each word in a sentence, then weigh the noise that is added into word embedding by the corresponding importance of the word. Meanwhile, this paper proposes a simple and effective way to alleviate the discrete corruption problem. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the reviewers acknowledge the intuitive masking approach, improved efficiency, and novelty, they also suggest improvements such as conducting multilingual evaluation, expanding comparisons to other diffusion models, and providing more details on the reverse process and target ambiguity.
Decision: Accept-Main