InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation
Authors: Renzhi Wang
Jing Li
Piji Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Text generation
Diffusion model
Information entropy

Abstract: Diffusion models have garnered considerable interest in the field of text generation. Several studies have explored text diffusion models with different structures and applied them to various tasks, including named entity recognition and summarization. However, there exists a notable disparity between the "easy-first" text generation process of current diffusion models and the "keyword-first" natural text generation process of humans, which has received limited attention. To bridge this gap, we propose InfoDiffusion, a non-autoregressive text diffusion model. Our approach introduces a "keyinfo-first" generation strategy and incorporates a noise schedule based on the amount of text information. In addition, InfoDiffusion combines self-conditioning with a newly proposed partially noising model structure. Experimental results show that InfoDiffusion outperforms the baseline model in terms of generation quality and diversity, as well as exhibiting higher sampling efficiency.

InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation
Renzhi Wang JJin Li Piji Li
College of Computer Science and Technology,
Nanjing University of Aeronautics and Astronautics, China
MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China 3 Department of Computing, The Hong Kong Polytechnic University, China
(rzhwang, pjli yēnuaa edu cn 3 jjing-and laing-amelinn amelia. iu edu.
Abstract
to various text generation tasks, such as named entity recognition (Shen et al., 2023) and summa- rization (Zhang et al., 2023). However, these works have failed to recognize a fundamental difference between the process of generating text with diffu- sion models and the actual process of human text generation, which may be reason why text diffu- sion models have consistently fallen short in terms of generation efficiency and quality.
Diffusion models have gareered considerable interest in the field of text generation. Sev- eral studies have explored text diffusion mod- els with different structures and applied them to various tasks, including named entity recogni- tion and summarization. However, there exists a notable disparity between the "casy-first" first""tt text generation process of current diffusion models and the "keyword-firs"" natural text generation process of humans, which has received limited attention. To bridge this gap, propose In- FoDiffusion, a non-autoregreesivee text diffusion model. Our approach introduces "keyinfo- first" generation strategy and incorporates noise schedule based on the amount of text information. In addition, InfoDiffusion bines self-condee conditioning with newly proposed partially noising model structure. Experimen- tal results show that InfoDiffusion outperforms the baseline model in terms of generation qual- ity and diversity, as well as exhibiting higher sampling efficiency.
Previous research has found that the text diffu- sion models seem follow "easy-first" principle (Emelianenko al., 2019; He al., 2022) the decoding process. The "casy-first" principle means the model tends to generate tokens that are most frequently observed (and least surprising) in the training corpus, in order to achieve higher likeli- hood. As the context becomes richer, more details incorporated into the sequence. Figure/ 1 illus- trates the decoding process of some existing text diffusion models, where it can be observed that the model tends to prioritize generating simple, high- frequency, and semantically poor words like "the" and "of" before generating less frequent but more informative and semantically rich words like "struc- ture" and "rememher". This differs from the actual order in which humans process generate text. People tend prioritize the core parts of sentence or a paragraph, which contain crucial information (Grice, 1975). For example, when asked:" are your upcoming plans?", you would answer:"I am going to finish a research paper." this pro- cess, the words that come to your mind first are most likely "paper" or "finish", as they carry key information or have higher information entropy. rather than meaningless words like "a" or "the". It is difficult imagine how someone whose first reaction "the" would answer the question men- tioned above. Similarly, this also disastrous for a language model to which we expect to impart language abilities and even thoughts.
1 Introduction
Non-autoregressive (NAR) generation refers method of generating sequences where each el- ement is generated independently, without rely- ing on previously generated elements, allowing for faster parallel generation but potentially sacrificing the generation accuracy (Xiao et al., 2022). Re- cently, diffusion models have demonstrated pow- erful generative capabilities in image generation tasks, gradually becoming new paradigm in gen- erative models. The successful application of diffu- sion models to continuous data such images and audio has motivated researchers to introduce them to discrete data like text. Previous researches have attempted to incorporate diffusion models into non- autoregressive text generation, designing different text diffusion model structures and applying them
"Corresponding author http:////rwwwe////toofffuuuuooa ottoo//rg/////ttttao/oo////on/ra
'Code:https:/ //gittth://///t....././.
This inconsistent decoding order in text diffu-

---

D3PM
DiflusionBERI
DifluSeq
the man has also been errested by the police.
today, he will be remembered for that mistake.
want to become a good geologist.
the man has also been urrested by the police;
today he will be remembered for that mistake,
want to become a good geologist; want to become a good geologist.
the man
been arrested by the police.
Loday, he will be remembered for mistake, .
the man has also been amrested the police. the man has also been arrested by the police.
today he will be remembered for that mistake,
want to become a good geologist.
today he will be remembered for that mistake. I want to become e good geologist
Figure I: Inference process of three text diffusion models: illustrating the "casy-first" generation order. Each row represents one inference step.
sion models could lead to poor generation quality and low efficiency. On one hand, due to the fact that the core part of sentence (key information) is accurately generated in the later half of the sam- pling process, the model lacks comprehensive understanding of the overall semantic meaning of the sentence in the early stages, resulting in un- satisfactory generation quality. On the other hand, the lack of guidance from key information in the initial half of the sampling process leads to the gen- eration of many meaningless or irrelevant words from the beginning. Due to the presence of these meaningless even erroneous sampling steps, the efficiency of the model's sampling process is low. To address the aforementioned issues, we pro- pose a new non-autoregressive text generation model called InfoDiffusion. We devise a new noise schedule based on the information entropy of the text, enabling the model to aware the information carried by each word in a sentence. This guidance helps the model prioritize generating key informa- tion during the sampling process, thereby enhanc- ing the quality and speed of sampling. Furthermore. we have integrated celf-conditioning further im- prove the generated output's 's quality and utilized "partially noising and conditional denoising" tech- nique to realise sequence- ceateence-t-ssqueeeee to-sequence tasks.
Preliminaries
2.1 Diffusion Models
Diffusion models are class of latent variable models characterized by forward and reverse Markov process (Sohl-Dickstein et al., 2015; Ho et al., 2020). In this framework, given sample from the data distribution zo ~ g(ar)) the forward process generates a sequence of latent variables 1,,..., By by sampling from:
q(Z x1-1) = (2 v1 B, B,I) (1)
where 3e E (0.1) is noise schedule controlling the step size of adding noise. Based on the repa- rameterization trick, arbitrary intermediate latent variable xa can be sampled in a closed form:
ạ mn) Nx V Varzo v1-
(2)
where 1- a = I Qg. Following a pre- defined noise schedule, 83 increases (o, decreases) as the timestep grow and eventually corrupts zo into random noise. Tf 3r is small enough, the verse process q 1 is also a Gaussian, which is learned by a parameterized model:
Pa (#t-1 xx) N (xc-1i Me (xe, t) Ee (Zz, t)) (3)
where μ4 (xz, t) and Eg (Iz, t) can be implemented by a denoising networks fo (rre, t) like U-Net or Transformer (Li et al., 2023). During inference, the reverse process begins with sampling noise from a Gaussian distribution p(æT) = [N(r;; 0, I) and iteratively denoise it by Po (It-1 mz) until obtain- ing 20. The learning objective of diffusion models is derived from the variational lower bound of the negative likelihood of input æp. denoted as:
In summary, our contributions are as follows:
We propose new non-autoregresssve text gen- eration model called InfoDiffusion, and enables the model to aware the information entropy con- tained in the text to prioritize generating key in- formation during the sampling process.
We combine self-conditioning and "partially nois- ing and conditional denoising" to achieve high- quality sequence-tor -oonnce-----eueecce text generation. Experimental results demonstrate that InfoDif- fusion, which follows a "keyinfo-fire genera- tion order consistent with humans, achieves bet- ter generation quality and higher efficiency than baseline models across four text generation tasks.
Lulb E,y [DL (a (xT 200 Ippe (ΦT)))
T
Eg [KL (g (xz-1 wy, 20) || (xz-1 xx))]
E. [log po (#o x1)
(4)

---

Keyinfo-Fins
on mat
Self- Conditionnn
Pe (
Rounding
XT
Xt
Xt-
X
W
Embedding
Information Entropy Aware Noise Schedule
Figure 2: The overview of the proposed text diffusion model InfoDiffusion. Grey represents undecoded words, red underline indicates words decoded the current time step, and black represents words decoded in previous time steps.
where Eg denotes the expectation over the joint dis- tribution ((Zo:T). With additional condition on zo- the posterior of the forward process q(2; 2o) becomes tractable using Bayes theorem, then the simplified objective Iisiple can be expressed as:
to new end-to-end objective(Li et al., 2022):
S = E,[L + loggo w) logpo (x)))
(6)
which can be further simplified as:
Ce2c simple = =Bg simple + ||EMB(w) no 2011
logpe [r))]
Siimple Σ (1) - po 2
(7)
(5)
InfoDiffusion
where μr the mean of posterior q mo).
Through different parameterization strategies, the prediction objective can also be the noise (Ho et al.,
In this section, we introduce the detailed design InfoDiffusion. The overall model architecture of InfoDiffusion is depicted in Figure 2. InfoDiffu- sion incorporates Information Entropy Aware Noise Schedule, enabling the model to follow "keyinfo-fisst" generation strategy, thereby achiev- ing text generation that aligns with human-like pro- cesses. Additionally, InfoDiffusion combines self- conditioning and partially noising to achieve faster and superior text generation.
2020) or original data zo (Li et al., 2022).
2.2 Continuous Text Diffusion Model
To adapt diffusion models for discrete text data, straightforward approach employ word embed- ding, mapping discrete tokens to continuous word vector space before going through the continuous diffusion process. The continuous text diffusion model (Li et al., 2023), also known as the embed- ding diffusion model, introduces an embedding step (o(x)) . (EMB(w), ooT) in the forward process, where EMB(w) represents randomly initialized embedding function obtained from pre-trained model (such as BERT) that projects the discrete token w into the continuous word vector space. For the backward process, the text diffu- sion model maps continuous word vectors back to their corresponding actual words through the word rounding module po(w]ro) - I Po (wx/ar;) The inference process starts from random noise 2' and follows the typical continuous diffusion pro- cess mentioned in Section 2.1 combined with word rounding reconstruct the target word from the noise. To jointly learn the denoising network and the word embedding, the continuous text diffusion model extends the training objective in Equation
3.1 Information Entropy Aware Noise Schedule
In diffusion models, the noise schedule is crucial component. The noise schedule directly determines how the original data is gradually perturbed dur- ing the forward process and how the model learns to recover the target data from the noise during the reverse process. This also leads to the signif- icant influence of noise scheduling on the quality and diversity of generated samples. Previously, commonly used noise schedules, such the linear schedule (Ho et al., 2020) and the cosine schedule (Nichol and Dhariwal, 2021), have shown promis- ing results in image generation tasks. However, these schedules assume all tokens carry the same amount of information and does not consider the linguistic differences among the tokens in se- quence. This directly leads to a "shortcut" taken by

---

1.3
Seld-Candttioiing
Xe+1
X
Figure 4: An illustration reverse diffusion sampling steps with Self-Conditioning sampling directly based on its previously generated samples.
Figure 3: The noise schedule for each token se- quence is determined based on the information entropy.
where e(w") represents the normalized value of the information entropy of the i-th word in sentence w and A(t) control the effect of the informativeness at time step t. To ensure that the latent variables 2g retains all information at the beginning process (1 0) and zero information at the end process (t T), the noise schedule X(t) is designed be sinusoidal, satisfying X(0) X(T) 0, fol- lowing (He al., 2022). The H(w) represents the mean entropy of sentence w and max maa(HH(v) (w")) represents the maximum entropy in sentence w.
the model: it tends generate tokens that are most frequently appearing (and easiest) the training corpus to achieve a higher likelihood. However, this generation order contradicts the behavioral pat- tern of humans, who tend to prioritize thinking about and generating the core parts of text, which contain higher information content. We refer to this human strategy of prioritizing the generation of high-in inghhinformttion content parts in text as "key- information-fiite first", abbreviated as "keyinfo-first". In order to address the differences in the gen- eration process mentioned above and enable the model to generate text more like human, we have designed new noise schedule that can aware the informational entropy of words in sentence. That is, at the initial stage of the forward process, now-information words are perturbed, and high- information words are perturbed the final stage, thus guiding the model to prioritize generating key information during the reverse process.
Figure 3 shows how ã progresses during the for- ward process. For example, consider the sentence "The bus passes by the school", the school" carries higher information content. Therefore, we encourage masking it at the later stage, allowing our model to learn to restore it in the early place. It is worth noting that such noise schedule does not alter the training objective since it does not modify the conditional probability function ( 12,, 2o) in Equation 4.
Specifically, we first linearly interpolating the mutual information between the latent variables 2, and the original data 2n to O, i.e. [xx;; .n)) se (I- (zo). where H denotes the entropy, which measures the amount of information of random variable. this case, the noise function in the clas- sical diffusion model becomes 3 = T : (Equa- tion 1) and & 2021). Furthermore, to prioritize perturbing words with lower information before perturbing words with higher information, we design noise weights based on the information entropy of each word in a sentence w. The specific details are as follows:
3.2 Self-Conditioning
In the sampling process of classical diffusion mod- els, each time step L, the model generates the current prediction z (Iy, t, 0) through a denoising network fe(2,, l). However, this denoising net- work only relies on the updated Ie from the pre- vious time step and discards the estimated result z ,hicch means there is no connection between the predicted results of adjacent sampling steps. In the text generation process of the text diffusion model, this implies that the semantic information between the generated results of adjacent time steps is inconsistent and incoherent, inevitably leading to subpar text quality and low inference efficiency. To address the issue of semantic incoherence mentioned above, inspired by (Chen al., 2022), (9) employ the self-conditioning. As shown in Fig- ure 4, this technique considers different denois- ing functions (f,,,, 2/ //+1 t, 0), utilizes the previ- (10) ously estimated samples as auxiliary inputs. Self-
& (Equation 2) (Austin et al.,
L + x(t)e(w") E [0.1]
af = 1- T
(8)
X(t) = Asin( t (T)
H(w') (ww
c(w²)= max(H((i)) - min(HHW))

---

conditioning refine the denoising function based on previous estimations instead of starting from scratch with new estimations. By doing so, direct connections and dependencies are established be- tween the generation results of adjacent time steps, uchieving semantic consistency and coherence. For more efficient model training, we adopt the same training strategy Analog-Bit (Chen et al., 2022): with 50% probability, train # (31, -0t1 1, by setting the input #ff1 O, which brings back the model without seff-conditioning Otherwise, we first estimate z by (rr, O, t, 0) and then use it for celfitondii training. In the second case, we do not back-propagate through the first esti- mated 8/- Therefore, the increase in additional training time is less than 25%.
4 Experimental Setup
4.1 Tasks and Datasets
Following (Gong et al.. 2022), we conduct exper- iments on four typical and popular tasks: Open domain dialogue, Question generation, Text simpli- fication and Paraphrase. Open domain dialogue requires models to generate informative and mean- ingful responses given a dialogue context. We em- ploy the widely used Commonsense Conversation Dataset (Zhou et al., 2018), with over 3 million con- versational pairs covering wide range of everyday topics. Question generation aims to generate ques- tions which can be answered by the given contents. We utilize the Quasar-T dataset (Dhingra et al., 2017), processed by (Gong et al., 2022), containing 119K training samples of document-question pairs. Text simplification aims to modify complex text into simplified sequences by simplifying grammar and word choice. We use the corpus constructed by (Jiang al., 2020) consisting of 666K complex- simple sentences. Paraphrase involves rewriting sentence with the same semantic meaning but a different surface form. We adopt the widely used Quora Question Pairs (QQP), sourced from the community question-ame ansser platform Quora, which consists of 147K positive pairs.
3.3 Partially Noising and Conditional Denoising
In the classical requence-to-sequees task, given souree text kens, it generates target text sequence y {wj, .... w
(wi, wg...., ) with to-
A sequence generation model can achieve this by modeling the conditional probability: p(y]s). To accomplish sequence-tor sequence text generation tasks, we employ the Par- tially Noising and Conditional Denoising (Gong et al., 2022). This technique adds noise only to the target text y during forward process and applies denoising solely to y during the denoising process. Specifically,, given a pair of text: the source text w= and the target text w. we first perform word embedding and concatenate the text pair as EMB(w" Ov). Then, we obtain the initial state To of the forward process through (x w Ov) N (EEMB(w> Ov))), we use 8, and yt to represent the parts of x; belong- ing to w" and w at diffusion time step t, following (Gong et al., 2022). In forward process, we only add noise Yt while keeping 8, unchanged. In the reverse denoising process, 81 is still kept unchanged and treated as the demoising condition, controlling and guiding the model to generate the desired text from the noise. The training objective at this point can be simplified as (Gong et al., 2022):
4.2 Baseline Methods
Following (Gong et al., 2022), we compare InfoD- iffusion to four groups of baselines:
Encoder-decoder autoregressive model. We choose two popular models: GRU (Chung et al., 2014) with attention and Transformer (Vaswani al., 2017).
To simplify the notation,
Fine-tuned large pre-trained language model. We choose GPT-2 (Radford et al., 2019) and GP- VAE (Du al., 2022). GPT-2 is trained with language modeling and GPVAE augments T5 (Raffel al., 2020) with VAE.
Non-autoregressive model. we consider LevT (Cortes et al., 2015), a widely used, strong itera- tive NAR model. adopts inscrtion and deletion to generate and refine sequences iteratively.
T
Text diffusion model. We choose DiffuSeq (Gong et al., 2022). It is a recent text diffusion model, and the performance of other text diffu- sion models is similar to it.
ssimpl
10 (o(x1,1)13
1=2
(11)
+ +|EEMB(w") fo (21, 1)
logpo (w= On (aro)
We implement these models following their orig- inal papers.
where fo is the denoising network.

---

Table Evaluation results on four conditional text generation tasks. The best results are denoted by bold fonts, and the best results without pretrained language models are denoted by underline fonts.
Quality
Diversity
Dataset
Model
Length
BLEUt
ROUGE-LI
BERTScore"
Dist-11
Self-BLEU,
Diverse-4f
GRU-attention
0.0068
0.1054
0.4128
0.8998
0.8008
0.1824
4.46
Tansformer-base
0.0189
0.1039
0,4781
0.7493
0.3698
0.6472
19.5
Open
GPT2-base FT
0.0108
0.1508
0.5279
0.9194
0.0182
0.9919
16.8
Domain
GPT2-large large FT
0.0125
0.1002
0.5293
0.9244
0.0213
0,9938
16.8
Dialogue
GPVAE-T5
0,0110
0,1009
0,4317
0.5625
0.3560
0,5551
20.1
NAR-LevT
0.0138
0.0550
0.4760
0.9726
0.7103
0.1416
DiffuSeq
0.0139
0,1056
0,5131
0,9467
0.0144
0,9971
13.6
InfoDiffusion
0.0152
0.1272
0,5314
0.9497
0.0152
0.9810
15.3
GRU-attention
0.0651
0,2617
0.5222
0.7930
0.9999
0,3178
10.1
Transformer-base
0.0364
0.1994
0,5334
0.8236
0,8767
0.4055
12.1
GPT2-hase FT
0.0741
0.2714
0.6052
0,9602
0.1403
0,9216
10.0
Question Generation
GPT2-large
0.1110
0.3215
0.6346
0,9670
0,2910
0.8086
9.96
GPVAE-T5
0,1251
0.3390
0.6308
0.9381
0.3567
0.7286
11.4
NAR-LevT
0.0930
0.2893
0.5491
0.8914
0.9830
0,4776
6.93
DiffuSeq
0.1731
0.3665
0.6123
0.9056
0,2789
0,8103
11.5
InfoDiffusion
0.1924
0,3892
0.6310
0.9142
0,2625
0.8021
12.7
GRU-atlention
0.3256
0.5602
0.7871
0.8883
0.9998
0,3313
Transformer-buss
0.2445
0.5058
0.7590
0.8886
0,8632
0.4028
18.5
GPT2-base FT
0.3085
0.5461
0.8021
0.9439
0.5444
0.6047
16.1
Text Simplification
GPT2-large
0.2693
0,5111
0.7882
0,9464
0.6042
0.5876
15.4
GPVAE-T5
0,3392
0.5828
0.8166
0.9308
0.8147
0.4355
18.5
NAR-LevT
0,2052
0,4402
0.7254
0.9715
0.9907
0.3271
8.31
DiffuSeq
0.3622
0.5849
0.8126
0,9264
0.4642
0.6604
17.7
InfoDiflusion
0.3941
0,5997
0.8437
0.9323
0.4515
0.6741
15.3
GRU-attention
0.1894
0,5129
0.7763
0,9423
0.9958
0.3287
8.30
Transformer-hase
0.0580
0,2489
0.5392
0.7889
0.7717
0,4312
5.52
GPT2-base FT
0.1980
0,5212
0,8246
0.9798
0,5480
0,6245
9.67
Paraphrase
GPT2-large FT
0,2059
0.5415
0.8363
0,9819
0.7325
0.5020
9.53
GPVAE-T5
0.2409
0,5886
0.8466
0.9688
0,5604
0.6169
9.60
NAR-LevT
0,2268
0.5795
0,8344
0.9790
0.9995
0.3329
885
DiffuSeq
0.2413
0.5880
0.8365
0.9807
0.2732
0.8641
11.2
InfoDiflusion
0.2656
0.5928
0.8576
0.9815
0,2873
0.8972
11.4
4.3 Evaluation Metrics
thermore, we incorporate self- (Zhu et al., 2018), sentence-level metric that assesses over- lapping n-grams among generated texts. A lower Self-BLEU score and higher diverse-4 value indi- cate greater degree of diversity the generated outputs. Following (Gong et al., 2022). we gener- ate three samples per text condition to calculate the diversity metrics for each method.
When evaluating the generated sequences, both quality and diversity play vital roles. To assess quality, we employ BLEU (Papineni al., 2002) and ROUGE (Lin, 2004) as standard metries, mea- suring the overlapping n-grams between the gener- ated and gold texts. However, since string matching alone may not suffice for open-ended generation, we also utilize BERTScore (Zhang et al., 2020) to evaluate the semantic similarity at the embed- ding level. Greater scores in BLEU, ROUGE, and BERTScore indicate superior performance in text generation. In terms of diversity, we consider eval- uating distinet n-grams using Distinet (Li et al., 2016) and the ratio of distinct n-grams to total words using Diverse (Deshpande et al., 2019). Fur-
4.4 Implementation Details
InfoDiffusion is built upon the 12 layers of Trans- former with 12 attention heads and has approxi- mately 91M parameters. The maximum sequence length set to 128, with an embedding dimen- sion of d 128. We perform diffusion steps T = 2, 000. To address -focabuuuyy genera-

---

tion, utilize Byte Pair Encoding (Sennrich et al., 2016) to construct the vocabulary. The accuracy metrics of InfoDiffusion are evaluated using MBR (Minimum Bayes Risk) with a candidate sample size of ||| = 10. The experiment is deployed on NVIDIA RTX 3090 Tensor Core GPUs, and we use 4 GPUs on training and single GPU on sampling.
Text Simpiification
0.35
Informficcion
0.20
Difluseq IntoDittasion
0.5
Difuusq
-04
5 Results and Analysis
0.05
5.1 Text Generation Evaluation
250
500
750
1000
1250
1500
1750
2000
Sampling Steps
As shown in Table 1, we conclude that InfoDiffu- sion achieves comparable or even higher generation quality compared with strong baselines.
Figure 5: The curve of BLEU/div-4 score along with generation process.
First, compared to encoder-decoder autoregres- sive models and Non- Aotoorrregressivve models, In- foDiffusion exhibits absolute advantage in terms of quality and diversity. For instance, in question generation tasks, the quality metric BLEU has im- proved by more than threefold, while distinct has increased by +0.12. The improvement in diver- sity metries is cqually significant. For example, the value of diverse-4 increased from 0.64 to 0.98, representing an improvement of over 50%.
Table Ablation studies on QQP.
Model
BLEU
ROUGE-L?
BERTScore
Dist-1f
InfaDifiasion
0,2656
0.5928
0.8576
0
SciffCCoddrtooing
0.2531
0.5884
0.8462
0.9816
Noise Schodule
0,2480
0.5870
0.8413
0.9798
prove in the later stages of sampling (The decreas- ing trend in diversity metrics is due to the sam- pling process gradually generating the target text from noise and noise has a high level of diversity). But InfoDiftusion exhibits the opposite behavior, generating high-quality text in the early and mid- dle stages of sampling. Approximately halfway through the sampling process, the quality of the text generated by InfoDiffusion surpasses the final results of DiffuSeq. This indicates that InfoDif- fusion can converge to the target sentence more quickly and shorten the sampling time by half com- pared to DiffuSeq while maintaining almost the same generation performance.
Second, compared pre-trained models like GPT2, InfoDiffusion outperforms the base vari- ant and performs comparably to the large variant, which has 8 times more parameters than InfoDiffu- sion. In terms of diversity, InfoDiffusion leads in seven out of the twelve comparative scenarios, in- dicating a slight advantage over pre-trained models in generating diverse texts.
Last, compared the well-performing diffusion model DiffuSeq, InfoDiffusion demonstrates supe- rior text generation quality across all datasets. All quality metrics show an improvement of +0.01 to +0.03. On the other hand, although the score of self-BLEU lags behind DiffuSeq in text simplifi- cation tasks, there is a slight improvement in text diversity across the remaining datasets.
5.3 Ablation Analysis
To demonstrate the effectiveness of the proposed techniques in InfoDiffusion, conducted ablation studies on the QQP dataset. As shown in Table 2, when we removed the self-conditioning, the BLEU score decreased by 0.0126, while Dist-l remained almost the same. Furthermore, when we addition- ally removed the proposed noise schedule from InfoDiffusion and used the sqrt schedule proposed in DiffusionL.M (Li et al., 2022) instead, the BLEU score dropped by 0.0051 and Dist-1 dropped by 0.0018. This indicates that the proposed noise schedule and self-conditioning contribute to im- proving the quality of generated text, while the impact of self conditonnii on the diversity gen- erated text is minimal.
5.2 Inference Efficiency Comparison
One of the major concerns of diffusion models is the efficiency of Inference. We compare our In- foDiffusion with DiffuSeq in terms of inference efficiency. We conduct experiments on Text Sim- plification and set the inference batch size to 50 and diffusion time step to 2000 for both models. The quality (i.e.. BLEU) and diversity (i.e., div- 4) curves during the model generation process are shown in the Figure 5. The quality and diversity of the text generated by DiffuSeq gradually im-

---

Table 3: A sampling case from QQP dataset. We truncate the selected samples to the first 10 tokens and mark the generation process of each word with different colors.
Diffusion Step &
Generation Results of Intermediate Processes 3i
Input Text
What should i do to be great geologist?
[ 100
athan backlash swiped regentlated patrollingnine jennie ? chill [PAD]
130
athan backlash swiped regentlated spotiig geologist ? chilean [PAD]
200
clan patrice swiped regent carmelgrowth geologist? [unused288] [PAD]
t 230
glancing patrice can heringlated growth geologist navigable [PAD]
& 300
glance patrice can moscowgrowth geologist ? corporal [PAD] [PAD]
t 340
[CLS] how can i 1859 a 1765 geologist? mequeen [PAD] [PAD] [PAD]
t 400
[CIS]] how can [unused2.2]] sculpted geologist? [SEP] [PAD] [PAD]
490
[CLS] how
35th nueva geologist? [SEP] [PAD] [PAD]
600
[CLS] how 35th sculpted geologist? [SEP] [PAD] [PAD]
840
[CLS] how can become a good geologist? [SEP]
t 950
[CLS] how can
become a good geologist? [SEP]
1000
[CLS]] how can become a gond geologist? [SEP]
1600
[CLS] how can
become good geologist? [SEP]
2000
[CLS]
how can
become good geologist [SEP]
verb
adjective
adverb
CT
DruSal
250-
Drusa
300
200
300
250
202-
200
300
100
322
50
10
10
Figure 6: Comparison of distributions of different types of words relative generation order. The x-axis represents the diffusion step f, while the y-axis represents the number words of a certain type that are first decoded that diffusion step.
5.4
Case Study
sion decodes these high- high-inforration words much more earlier compared to DiffuSeq.
We select an illustrative cases and investigate the generation process of InfoDiffusion. There are more cases in the Appendix C. As shown in Table 3, the generation process reveals that the InfoDiffu- sion model follows the "keyinfo-fire firstt" generation order: it prioritizes generating nouns with higher in- formation content, such as "i" and "geologist", and then sequentially generates words with lower infor- mation content, such as "can", "how", "become", and "good" to complement the sentence.
Conclusion
This paper, we propose InfoDiffusion, a novel non- autoregressive text diffusion model. By designing an Information Entropy Aware Noise Schedule, we enable the diffusion model to follow "keyinfo- first" text generation process that is more aligned with human text generation, thereby achieving im- proved effectiveness and efficiency in text gener- ation. Experimental results on four benchmark datasets confirm the effectiveness of InfoDiffusion. This study is the first research on the decoding or- der of diffusion models and the first attempt to alter the decoding order of diffusion text models. Future work could explore the use of the proposed noise schedule to replace the existing noise in related tasks based on diffusion models, in order to further enhance the model's 's performance.
In order to illustrate more clearly the model's preference for generating key information, we se- lected four categories of words that generally have key information higher information content: nouns, verbs, adverbs, and adjectives (Clark and Weir, 2002; Emelianenko et al., 2019). We com- pared the decoding order of these words in InfoD- iffusion and DiffuSeq during the decoding process. As shown in Figure 6, it is evident that InfoDiffu-

---

Limitations
progress in statistical language modeling. In INTER- SPEECH 2014, 15th Annual Conference of the In- ternational Speech Communication Association, Sin- gapore, September 14-18, 2014, pages 2635-2639. ISCA.
Despite the strong performance of InfoDiffusion. it still has the following limitations. First, due to the strong preference of language for simple words, simple words may still appear early in the decoding process, Second, our evaluation relied solely on automatic metrics like BLEU, without assessing potential issues like hallucinations in the generated text. Future work could utilize both au- tomatic metries and human evaluation to compre- hensively assess text quality across dimensions in- cluding grammar, semantics, and more. This mul- ufaceted approach will facilitate the generation of truthful, logical, and reliable text.
Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, and Diyi Yang. 2023. A cheaper and better diffusion language model with soft-masked noise. CoRR. abs/2304.04746.
Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. 2022. Analog bits: Generating discrete data ing diffusion models with self-conditioning. CoRR, abs/2208. 042002.
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014, Empirical evaluation of gated recurrent neural networks sequence model- ing. CoRR, abs/1 411.3555..
Ethics Statement
The research presented in this paper on the diffu- sion text model adheres to ethical guidelines and principles. We have prioritized privacy, mitigated biases, ensured transparency, and promoted respon- sible use. Our commitment to accountability, re- sponsible governance, and continuous ethical as- sessment underscores our dedication to upholding the highest standards of integrity in the develop- ment and deployment of the diffusion text model.
Stephen Clark and David Weir. 2002. Class- Basss-Baed Prob- ability Estimation Using a Semantic Hierarchy. Com- putational Linguistics, 28(2):187-206
Corinna Cortes. Neil Lawrence, Daniel D. Lee. Masashi Sugiyama, and Roman Gamett, editors. 2015. Advances in Neural Information Processing Systems 28: Annual Conference on Neural Informa- tion Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada.
Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexan- der G. Schwing, and David A. Forsyth. 2019. Fast, diverse and accurate image captioning guided by part- of-speech. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 10695-10704 Computer Vision Foundation/ IEEE.
A Acknowledgements
This rescarch supported by the National Natural Science Foundation of China (N0.62106105), the CCF-Baidu Open Fund (No.CCF- CCB--aad2000307),, the CCF-Tencent Open Research Fund (N..RAGR20220122), the CCF-Zhipu AI Large Model Fund NN..CCT-Zhhpp2223355 the Fundamental Research Funds for the Central Uni- versities (No.NJ2023032), the Scientific Research Starting Foundation of Nanjing University of Aero- nautics and Astronautics (QR2122)) and the High Performance Computing Platform of Nanjing University of Aeronautics and Astronautics.
Bhuwan Dhingra, Kathryn Mazaitis, and William W. Cohen. 2017. Quasar: Datasets for question answer- ing by search and reading. CoRR, abs/ ht///777.03999 .099044
Wanyu Du, Jianqiao Zhao, Liwei Wang. and Yangfeng Ji. 2022. Diverse text generation via variational encoder-decoder models with gaussian process priors. CoRR, abs/2244.01227.
Dmitrii Emelianenko, Elena Voita, and Pavel Serdyukov. 2019. Sequence modeling with unconstrained gen- eration order. In Advances in Neural Information Processing Systems 32: Annual Conference on Neu- rul Information Processing Systems 2019, NeurIPS 2019, December 841 2019, Vancowvr, BC, Canada, pages 7698 7709.
References
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. In Advances Neural Information Processing Sys- tems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 17981-17993.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2022. Diffuseq: Sequence to quence text generation with diffusion models. CoRR, abs/2210.0897.
Ciprian Chelba, Tomás Mikolov, Mike Schuster, Qi Ge Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring
Herbert P Grice. 1975. Logic and conversation. Speech acts, pages 4 41-58, Brill.

---

Zhengfu He, Tianxiang Sun, Kuanning Wang. Xuan- jing Huang, and Xipeng Qiu. 2022. Diffusionbert: Improving generative masked language models with diffusion models. CoRR, 1ttp/2211.15022
on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8162-8171. PMLR.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation, In Proceedings of the 40th Annual Meeting of the Association for Compu- national Linguistics, July 6-12, 2002, Philadelphia. PA. USA, pages 311-318. ACL.
Jonathan Ho, Ajay Jain, and Pieter Abbecl. 2020. De- noising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6- 12, 2020, virtual.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural CRF model for sentence alignment in text simplification. In Proceed- ings of the 58th Annual Meeting of the Association for Compulational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7943-7960. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Rex., 21:140:1-140:67
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-prone prommttig ob- jective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 110-119, The Association for Computational Lin- guistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing. EMNLP 2016, Austin, Texas, USA, November 1-4, 2016. pages 2383-2392. The Association for Computational Linguistics.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. 2022. Diffusion- Im improves controllable text generation. In NeurlPS.
Machel Reid, Vincent J. Hellendoorn, and Graham Neu- big. 2022. Diffuser: Discrete diffusion via edit-based reconstruction. CoRR, abs/22 16886.
Yifan Li, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Diffusion models for non-autoregressive text generation: A survey. CoRR, abs/2303.06577
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine tramslation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computationa Lin- guistics, ACL 2016, August 7-12, 2016, Berlin, Ger- many, Volume 1: Long Papers. The Association for Computer Linguistics.
Chin-Yew Lin. 2004. Rouge:
A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't t give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empiricul Methods in Natural Lan- guage Processing, Brussels, Belgium, October 31- November 4, 2018, pages 1797-1807. Association for Computational Linguistics.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yucting Zhuang. 2023. Diffusion- ner: Boundary diffusion for named entity recognition. CoRR, 02.22305.13999
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2015. Deep un- supervised learning using nonequilibrium thermody- namics. Proceedings of the 32nd International Conference on Machine Learning, ICML /C15, Lille, France, 6-11 July 2015, volume 37 of JMLR Work- shop and Conference Proceedings, pages 2256-2265. JMLR.org.
Xuanfan Ni and Piji Li. 2023. A systematic evaluation of large language models for natural. In Proceedings of the 22nd Chinese National Conference on Com- putational Linguistics (Volume 2: Frontier Forum). pages 40-56, Harbin, China. Chinese Information Processing Society of China.
Xuanfan Ni, Piji Li, and Huayang Li. 2023. Unified text structuralization with instruction-tuncd language models. CoRR, has/2303.14956
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, 7, Long Beach, CA, USA, pages 5998-6008.
Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference

---

Yisheng Xiao, Lijun Wu, Junliang Guo. Juntao Li, Min Zhang, Tao Qin, and Tie Liu. 2022. A survey on non-autoorerressvve generation for neural machine translation and beyond. CoRR, abs/2204.092 09969..
A Related Works
A.1 Text Diffusion Models
Adapting diffusion models to non-autoregressive (NAR) text generation poses challenge due the discrete nature of text. Discrete tokens cannot be directly corrupted by continuous noise, neces- sitating the redesign of typical diffusion models for text data. In this section, we focus on diffu- sion models customized for text, which can either perform diffusion in discrete space or incorporate an additional step of mapping discrete tokens to latent space of token embeddings before applying continuous diffusion.
Jiasheng Ye. Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. 2023. DINOISER: diffused con- ditional sequence learning by manipulating noises.
CoRR, abs/2302. 10225
Congehi Yin, Piji Li, and Zhaochun Ren. 2023, Ctrl- Dialogue structure learning for open-domain response generation. In Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023- 4 May 2023, pages 1539 1550. ACM.
struct:
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2022. Seqdiffuseq: Text dif- fusion with encoder-decoder transformers. CoRR, abs/2212.103325.
Diserete Text Diffusion Models. These text diffusion models extend diffusion models to dis- crete state spaces by corrupting and refining the sentences at the token level. D3PM (Austin et al., 2021) employs Markov transition matrices instead of Gaussian noise to diffuse real-world distribu- tions. DiffusER (Reid et al., 2022) generates a sequence of edits that effectively transforms ran- dom noise into output. DiffusionBERT (He et al., 2022) combines diffusion models with pre- pre-trained Language Models to enhance their performance. Diffusion-NAT (Zhou et al., 2023) proposes an iter- ative self-prompting strategy for denoising process. Continues Text Diffusion Models. Continu- ous text diffusion models introduce an additional step where discrete tokens are mapped to the la- tent space of token embeddings, followed by the adoption of continuous diffusion. Diffusion-LM (Li et al., 2022) the first to proposes constructing diffusion models on continuous word embedding space. DiffuSeq (Gong et al., 2022) focuses on sequence-to-sequence generation using encoder- only Transformers and utilizes partial noising to define the diffusion process and learn the denoising function. SeqDiffuSeq (Yuan et al., 2022) proposes an encoder-decoder diffusion model architecture for conditional generation and uses adaptive noise schedule technique to improve generation quality. DiNoiSer (Ye et al., 2023) proposes an adaptive method to determine the range of noise scales sam- pled for model to leverage amplified noise scales from the source conditions during inference. Masked Dif- fuse LM (Chen et al., 2023) follows easy first gen- eration and designs a soft masking strategy based on tf-idf. Additionally, DiffuSum (Zhang et al., 2023) applics diffusion models to enhance extrac- tive summarization.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023. Generation enhanced extractive summa- rization with diffusion. CoRR, abs/2305.01735.
Diffusum:
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguisties. ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers. pages 2204-2213, Associa- tion for Computational Linguistics.
Tianyi Zhang, Varsha Kishore. Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- netw.net.
Hao Zhou, Tom Young. Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu, 2018. Commonsense knowledge aware conversation generation with graph attention. In Proceedings the Awvety-Sevents Inter- national Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages
ijcai.org.
Kun Zhou, Yifan Li, Wayne Xin Zhao, and Ji- Ji-Rong Wen. 2023. Diffusion-nas nat: Self- Self- prompting discrete diffu- sion for non-autoren autourrrrrrreivveee text generation. CoRR, abs/2305.04044.
training allowing the
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI. USA, July 08-12, 2018 pages 1097-1100. ACM.

---

Table 4: Examples of sampling process of InfoDiffusion.
Input Text
How do read and find my YouTube comments?
400
[CLS] how can aground dismissing youtube johnstone? hammond [PAD] [PAD]
= 800
[CLS] how can increase [unused336] youtube 1855? [SEP]
1200
[CL.S] how can read my youtube comments? [SEP]
1600
[CLS]] how read my youtube comments? [SEP]
L 2000
[CLS] how can read my youtube comments? [SEP]
Input Text
How i make friends?
400
[CLS] how bassett iseibner anthropologist casino? [SEP] [PAD]
= 800
[CLS] how do the my friends iidae? [SEP]IPAD]PPAD]
1200
[CLS] how do make my friends? [SEP]
1600
[CLS) how do make my friends? [SEP)
t = 2000
[CLS] how do make my friends? [SEP]
Input Text
How i can speak english fluently?
400
[CL.S] how can campos susceptible pauline and brien gunfire? [SEP] sarawak forts [PAD] [PAD]
800
[CLS] how can campos robyn english outlaws scrambled forts? [SEP]
1 1200
[CLS] how speak fluent english and get tammy? [SEP]
1 1600
[CLS] how can speak fluent english and get confident? [SEP]
2000
[CLS] how can speak fluent english and get confident? [SEP]
Table 5: Examples of sampling process of DiffuSeq.
Input Text
How do read and find my YouTTub comments?
400
[CLS) docks doo blessed [unused305] snoop sentencing creators [PAD] [PAD] nall [PAD]wangains [unused781]
= 800
[CLS] how nods helgrade 139 ratios 2 ? [SEP] [PAD] [PAD] [PAD] [PAD] heresy [PAD]
è 1200
[CLS] how gymnusium laguna [unused730] youtube lobbied? [SEP]]
/ 1600
[CI SJ how do 275 my [unused7330 30] comments? [SEP]
t 2000
(CLS) how do i read my youtube comments? [SEP]
Imput Text
make friends? stamp trinidad glint labyrinth [unused347] admiral [PAD] [unused481] joo rochester governors examines [PAD] [PAD]
400
f 800
[CLS] blackout charlton ames labyrinth? [SEP] [PAD]
t 1200
[CLS] howl i engraving guinness? [SEP]
1 1600
CC S how bonnet engraving guinness? [SEP]
t 2000
[CLS] how do make my friends? [SEP]
Input Text
How speak english fluently?
400
[CLS] tchran disputched northermost oswald lansing jennie fivb whistling routing (PADJikh [PAD] [PAD] sheds
= 800
[CLS] ton efficacy battled city and? [SEP] [PAD] [PADJikh [PAD] [PAD]
t 1200
[CLS] how [unused293] i discovers dario mudden english? [SEP)
t 1600
[CLS] how can oswald fluent [unused490] english? [SEP]
t 2000
(CLS) how can i speak fluent like english? [SEP]
A.2 Noise schedule
Mutual Information Schedule. D3PM (Austin et al., 2021) designs the mutual information sched- ule for discrete diffusion models with absorbing states. This schedule reduces to 8 = T ci as same as in (Sohl-Dickstein et al., 2015).
Various noise schedules have been proposed to con- trol the frequency of different input data in the denoising network. Existing methods either adopt popular noise schedules from vision tasks or design new schedules specifically tailored to text data.
Sqrt Schedule. Diffusion-LM (Li et al., 2022) presents the sgri schedule, which is defined asa;
Linear Schedule. The linear schedule (Ho et al., 2020) involves varying 3 from 10-4 to 0.02 in linear manner. This schedule is designed to main- tain a relatively low noise scale at the start.
Vt/T + s. This noise schedule is also adopted by DiffuSeq (Gong et al., 2022).
Adaptive Schedule. SeqDiffuSeq (Yuan et al., 2022) proposes an approach to increase the dif- ficulty of predicting To in correlation with the timestep. This schedule involves Icarning the rela- tionship between the noise scale and the loss using
Cosine Schedule. This schedule defines ā₂
((t) [lo): where f(t) = cos( 1/7'.. 4)2² to slows down 1+x the growing speed of the noise scale.

---

Table 6: Examples of sampling process on LMIB.
Model
Sampling Process
[MASK] [MASK] [MASK]
IMASK] [MASK] [MASK] [MASK] IMASKI [MASK] [MASK] [MASK] [MASK], , [MASK] IMASKI [MASK] [MASK] IMASK] that [MASK].
DiffusionBERT
today. [MASK] will be [MASK] [MASK] that [MASK]
today, [MASK] will remembered for that mistake.
today he will be remembered for that mistake.
[MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK]
[MASK] [MASK] [MASK] [MASK] [MASK] mistakes [MASK]
InfoDiffusion-disrrte
[MASK] [MASK] J [MASK] their [MASK] mistakes.
make up for their [MASK] mistakes.
I make up for their recent mistakes.
Table 7: The comparison for different models.
Table Experimental results on Quasar-T and QQP.
Model
Parameters
Learning Paradigm
Diversity Source
Dataset
Model
BLEU
Rouge-T.
GRU-artention
6SM
enender-daonder
DiffuSoq (Gong al., 2022)
0.1731
0.3665
Trunsformer buse
SIM
encoder-dees decoder
Temperature
Quasa-T
DiffusioaBERI (He al., 2022)
0.0971
0.3420
GPT2-base FT
117M
peerain-fuettue
Hyhrid straregy
InfoDiffiusio
0.1924
0.3892
GPT2-large FT
117M
pretrain- finetune
Hybrid strategy
DiffuSeq (Gong al., 2022)
0,2431
0,5880
GPVAE-
pretrins-fictune
Gaussian sampling
QQP
DiffusionBERT (He et al., 2022)
0,2420
0,5845
NAR-LevT
117M
pretrain-VAE VAE
InfoDiffusion
0.2656
0.5928
DiffuSeq
9IM
non-mutoregren mnnngrrtoeeeessi
Gauesian sampling
InfoDiitssion
non-suloreeressive
Gaussian sampling
Table 10: Experimental results on PersonaChat, XSUM
Table 8: Experimental results on LMIB.
and SQuAD.
Model
PPL.
BLEU
Seif-BLEUL
Model
PersonaChat
XSUM
SQuAD
D3PM (Austin al., 2021)
77.50
0.4241
0.2288
BLEU-11
BLEU-2
Rouge-L.
Rouge-L]
Diffusiin-I M (1 al., 2022)
118.62
0.3553
0.2668
DiffuScq InfoDifliusin
37.79
32.50
20.29
29.29
DiffusionBERT (He et al., 2022)
63.87
0.4358
0.2151
40.13
36.47
35.70
InfoDiffusion discrete
68.46
0.4529
0,2084
converge to the target sentence more quickly.
an existing schedule (such as the sqrt schedule).
D Additional Experiments
B Baselines Settings
Following the reviewers' advice and (Ni et al., 2023; Ni and Li, 2023; Yin et al., 2023), in this section, we conduct some additional experiments to further demonstrate the effectiveness and gener- alization of the proposed method.
We follow the same baseline settings as (Gong et al., 2022) and the results are also collected from their work. The settings are listed Table 7. The GRU- attention encoder-dccoder model lacks diversity search algorithms, resulting in limited sentence- level diversity. In the case of NAR-LevT, we ad- here to the original paper's methodology by setting the maximum iteration to 9 and upplying the speci- fied termination condition. As for GPVAE-T5, we assign a scalar value of all tasks.
We validate the efficacy of the proposed method on discrete text diffusion models. To implement our method on discrete diffusion models, we adopte some settings from DiffusionBBRTT and adjust the noise schedule and the decoding schemes. On one hand, we train our model on LM1B (Chelba et al., 2014) which was used by DiffusionBERT, and the experimental results are shown in Table 6 and Ta- ble 8. On the other hand, we also compare the per- formance of DiffusionBERT on Quasar-T (Dhin- gra al., 2017) and QQP datasets used in our paper, and the results are shown in Table 9. The experimental results show that the method we pro- posed is also effective in the discrete domain. The
Inference Cases
In this section, we provide more sampling exam- ples of InfoDiffusion and DiffuSeq to illustrate dif- ferent sampling processes of text diffusion models. From the Table 4 and Table 5, we can visually observe that InfoDiffusion has the ability to priori- tize the generation of high-ing inghomfformttonn words and

---

model can follow the "keyinfo-first" generation or- der while maintaining high quality, showing the general applicability our approach.
Meanwhile, we conduct experiments on another 3 datasets: PersonaChat (Zhang et al., 2018), XSUM (Narayan ct al., 2018) and SQuAD (Ra- jpurkar et al., 2016). PersonaChat is dataset for dialogue generation, with the goal of predicting re- sponses according the dialog history. XSUM is dataset for summarization, with the goal of sum- marizing the document into sentence. SQuAD is a dataset for question generation, with the goal generating questions based on given passages and answers. The results in Table 10 demonstrate that InfoDiffusion still achieves strong performance across these datasets.


Reviews:
[{"review_id": "di4upQ23W7", "reviewer": "Reviewer_xSvW", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide thoughtful feedback on our paper submission. We greatly appreciate you highlighting the strengths of our work, including the novelty of our approach for bridging the gap between diffusion models and human text generation, as well as the improvements shown over baseline models.\n\nRegarding your comments about missing incorporation of relevant studies. We will add a chapter in the appendix. For examlpe: \"SeqDiffuSeq[1] proposes an encoder-decoder diffusion model architecture for conditional generation and uses adaptive noise schedule technique to improve generation quality.  DiNoiSer[2] proposes an adaptive method to determine the range of noise scales sampled for counter-discreteness training allowing the model to leverage amplified noise scales from the source conditions during inference. Masked Diffuse LM[3] follows easy first generation and designs a soft masking strategy based on tf-idf. Diffusion-NAT[4] proposes a iterative self-prompting strategy for denoising process. GlyphDiffusion[5] render a target text onto a glyph image containing visual language content and generate an image containing the target text through diffusion decoding process.\"\n\nRegarding the \"distinctive aspects and innovative elements\", most prior works on designing different decoding techniques focus on sentence-level structure and process. For example, combining encoder-decoder structures or adjusting the overall noise level of the entire sentence. In contrast, we pay more attention to the decoding order at the word-level. We discover the phenomenon that diffusion models tend to generate meaningless words first, which differs from humans' language habit of thinking about key words first. Therefore, we propose the \"keyinfo-first\" decoding strategy, which is more reasonable and interpretable. On the other hand, our proposed keyinfo-first strategy can be combined with sentence-level decoding techniques to achieve better decoding performance. We are also actively conducting experiments to validate this point, such as adopting the \"keyinfo-first\" decoding strategy on top of the encoder-decoder model structure proposed in SeqDiffuSeq, and verifying the general applicability of our proposed strategy on discrete diffusion models.\n\nOverall, we found your feedback extremely constructive. Please let us know if you have any other suggestions for improving the completeness and quality of our submission. We appreciate you taking the time to provide such thoughtful and detailed comments.\n\n[1] [SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers](https://arxiv.org/abs/2212.10325)\n\n[2] [DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises](https://arxiv.org/abs/2302.10025)\n\n[3] [A Cheaper and Better Diffusion Language Model with Soft-Masked Noise](https://arxiv.org/abs/2304.04746)\n\n[4] [Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation](https://arxiv.org/abs/2305.04044)\n\n[5] [GlyphDiffusion: Text Generation as Image Generation](https://arxiv.org/abs/2304.12519)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A new trial to fill the gaps between text generation process of diffusion models with natural text generation process of humans.\n2. Introducing a “key info-first” generation strategy and incorporates a noise schedule based on the amount of text info. It combines self-conditioning with a newly proposed partially noising model structure.\n3. Experimental results show the improvements over the baseline model in terms of generation quality, diversity and sampling efficiency.", "reasons_to_reject": "1. The absence of an extensive incorporation of relevant studies concerning the exploration of decoding order in diffusion models, such as SeqDiffuSeq (Seqdiffuseq: Text diffusion with encoder-decoder transformers), DiNoiSer (Diffused Conditional Sequence Learning by Manipulating Noises), and Masked-Diffuse LM (A cheaper and better diffusion language model with soft-masked noise), which have similarly investigated alternative corruption and decoding processes.\n2. Inadequate comparison with other closely related methodologies that delve into the decoding order of diffusion models, coupled with a limited depth of analysis regarding distinctive aspects and innovative elements in comparison to other similar systems.\n\nDiffused Conditional Sequence Learning by Manipulating Noises, Ye et al., 2023\nSeqdiffuseq: Text diffusion with encoder-decoder transformers, Yuan et al., 2022\nA cheaper and better diffusion language model with soft-masked noise, Chen et al., 2023", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces InfoDiffusion, a non-autoregressive text diffusion model that aims to bridge the disparities between the text generation process seen in diffusion models and the innate text generation process of humans. InfoDiffusion presents a novel \"key info-first\" generation strategy and integrates a noise schedule that adapts based on the volume of textual information. This is achieved through the fusion of self-conditioning with a recently introduced partially noising model framework. The empirical findings illustrate enhancements over the baseline model in aspects of generation quality, diversity, and the efficiency of sampling.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Diffused Conditional Sequence Learning by Manipulating Noises, Ye et al., 2023\nSeqdiffuseq: Text diffusion with encoder-decoder transformers, Yuan et al., 2022\nA cheaper and better diffusion language model with soft-masked noise, Chen et al., 2023"}}, {"review_id": "LsaGWaW4VY", "reviewer": "Reviewer_SEBp", "summary": "", "questions": "Please refer to reasons to reject", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback on my paper submission. We appreciate you taking the time to provide constructive comments. We summarize your main concerns as follows:\n\n> Q1: Continuous diffusion is not an effective way and try the proposed approach on discrete diffusion model.\n\nA1: Regarding the use of continuous diffusion, you raise a fair point about discrete diffusion being potentially more effective for natural language generation. I agree it is an interesting area for exploration, and have already run experiments using discrete diffusion methods as you suggest. To apply the proposed method to discrete diffusion models, we adopted some settings from DiffusionBERT, and adjusted the noise schedule and decoding schemes. On one hand, we trained our model on LM1B which was used by DiffusionBERT, and the experimental results are shown in Table 1 and Table 2. On the other hand, we also compared the performance of DiffusionBERT on the Quasae-T and QQP datasets used in our paper, and the experimental results are shown in Table 3.\n\nTable 1 Experimental results on LM1B\n\n| Method                 | PPL$\\downarrow$ | BLEU$\\uparrow$ | Self-BLEU$\\downarrow$ |\n| ---------------------- | --------------- | -------------- | --------------------- |\n| D3PM[1]                | 77.50           | 0.4241         | 0.2288                |\n| Diffusion-LM[2]        | 118.62          | 0.3553         | 0.2668                |\n| DiffusionBERT          | **63.78**       | 0.4358         | 0.2151                |\n| InfoDiffusion-discrete | 68.46           | **0.4529**     | **0.2084**            |\n\nTable 2 Decoding process on LM1B\n\n| Method                 | Decoding Process                                             |\n| ---------------------- | ------------------------------------------------------------ |\n| DiffusionBERT          | [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] |\n|                        | [MASK] , [MASK] [MASK] [MASK] [MASK] [MASK] that [MASK] .    |\n|                        | today , [MASK] will be [MASK] [MASK] that [MASK] .           |\n|                        | today , [MASK] will be remembered for that mistake .         |\n|                        | today , he will be remembered for that mistake .             |\n| InfoDiffusion-discrete | [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK]      |\n|                        | I [MASK] [MASK] [MASK] [MASK] [MASK] mistakes [MASK]         |\n|                        | I [MASK] [MASK] [MASK] their [MASK] mistakes .               |\n|                        | I make up for their [MASK] mistakes .                        |\n|                        | I make up for their recent mistakes .                        |\n\nTable3 Experimental results on Quasar-T and QQP dataset\n\n| Dataset  | Method        | BLEU$\\uparrow$ | Rouge-L$\\uparrow$ |\n| -------- | ------------- | -------------- | ----------------- |\n| Quasar-T | DiffuSeq      | 0.1731         | 0.3665            |\n|          | DiffusionBERT | 0.0971         | 0.3420            |\n|          | InfoDiffusion | **0.1924**     | **0.3892**        |\n| QQP      | DiffuSeq      | 0.2431         | 0.5880            |\n|          | DiffusionBERT | 0.2420         | 0.5845            |\n|          | InfoDiffusion | **0.2656**     | **0.5928**        |\n\nThe experimental results show that in the discrete domain, the method we proposed is also effective. The model can follow the \"keyinfo-first\" generation order, and the generation quality is high. This demonstrates that our method has general applicability. \n\n[1] [Structured Denoising Diffusion Models in Discrete State-Spaces](https://arxiv.org/abs/2107.03006)\n\n[2] [Diffusion-LM Improves Controllable Text Generation](https://arxiv.org/abs/2205.14217)\n\n> Q2: The novelty of this work.\n\nA2: Concerning the novelty of this work, I understand your concern about similarities to prior work like Diffusion-BERT. However, our innovation lies more in the following: We discovered the phenomenon that diffusion models tend to generate meaningless words first while humans tend to think of key words first, and proposed the \"keyinfo-first\" generation strategy to prioritize generating key words first to address this issue. To measure whether a word is a key word, we hypothesize that words with higher information entropy are more critical and important to a sentence. Therefore, we use information entropy to control the priority. Our noise schedule is indeed similar to DiffusionBERT, but DiffusionBERT only utilizes information entropy to design noise while we explore how to guide diffusion models to generate in a more reasonable, effective and interpretable way. Following the \"keyinfo-first\" idea but using different computational formulas is also feasible. We also considered using tf-idf or PMI for the computation.\n\nRegarding \"self-conditioning\", we are not the first to adopt this technique. The earliest work introducing it to text generation can be traced back to [1]. However, as stated in lines 288-293 of the paper, we use this method because: in experiments we find that incorporating self-conditioning leads to more unified semantics and higher relevance between the texts generated at adjacent time steps. We are the first to discover this phenomenon and thereby utilize this technique accordingly. On the other hand, we not only tried self-conditioning, but also experimented with methods like anchor loss, noise factor, embedding normalization [2], etc. From the experimental results in the table below, self-conditioning achieves the best performance and is more interpretable, so we finally retained this technique.\n\n| Model                          | BLEU$\\uparrow$ |\n| ------------------------------ | -------------- |\n| InfoDiffusion + self-condition | 0.3941         |\n| + anchor loss                  | 0.3742         |\n| + noise factor                 | 0.3613         |\n| + embedding normalization      | 0.3602         |\n\n[1] [Self-conditioned Embedding Diffusion for Text Generation](https://arxiv.org/abs/2211.04236)\n\n[2] [Difformer: Empowering Diffusion Models on the Embedding Space for Text Generation](https://arxiv.org/abs/2212.09412)\n\n> Q3: Follow the used datasets and experimental settings in Diffusion-NAT.\n\nA3: For the experiments, you make a good recommendation to evaluate on more standard datasets like XSum, Persona-Chat and SQuAD. The reason we did not use those datasets is that our work was completed around March, when most text diffusion models at that time were trained on the datasets we chose (e.g. DiffuSeq, Seqdiffuseq[1]), also for convenient comparison with concurrent works. On the other hand, Diffusion-NAT came out in May, after our work. Moreover, works concurrent with and even after Diffusion-NAT also used the same datasets as ours, such as RenderDiffusion[2], DiffuSIA[3], TESS[4] etc. We are now training and evaluating our model on the datasets you mentioned as quickly as possible, but due to limited GPU resources etc., we do not yet have final results.\n\n[1] [SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers](https://arxiv.org/abs/2212.10325)https://arxiv.org/abs/2305.04044)\n\n[2] [GlyphDiffusion: Text Generation as Image Generation](https://arxiv.org/abs/2304.12519)\n\n[3] [DiffuSIA: A Spiral Interaction Architecture for Encoder-Decoder Text Diffusion](https://arxiv.org/abs/2305.11517)\n\n[4] [TESS: Text-to-Text Self-Conditioned Simplex Diffusion](https://arxiv.org/abs/2305.08379)\n\nThank you again for the helpful feedback. We sincerely hope the changes outlined in our rebuttal and planned for the revision will adequately address your concerns and result in an improved rating. Your thoughtful suggestions substantially enhance the quality of our work. We greatly appreciate reviewers like yourself who uphold rigorous standards and provide guidance to advance research. Please let us know if you have any other feedback on how we can further improve the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.the focused keyinfo-first generation manner is indeed an important habit in humans to represent the spoken language, and the proposed new schedule can alleviate this issue.", "reasons_to_reject": "1.continuous diffusion is not an effective way to generate natural language, as it cannot capture the discrete semantic information within natural language. Besides, it also greatly increases the cost to generate the text, as it requires to sample for multiple times for denoising. Recent work[1][2] has shown that discrete diffusion may be a good choice, and can outperform continuous diffusion methods. Authors should try its approach on these methods and show the generality of it.\n\n2.the novelty of this work is worth concerning, as the new schedule is similar to the used one in diffusionBERT[1]. Besides, using self-condition is not a novel way, as it has also been widely-used in diffusion models, e.g., Diffusion-NAT[2].\n\n3.the experiments are conducted on some unpopular datasets, but not the commonly-used summarization, dialog and question generation, such as XSum, Persona-chat and Squad. Although DiffuSeq has used these datasets, readers would be more concerned about the performance of all these methods on the popular datasets. I suggest authors to follow the used datasets and experimental settings in Diffusion-NAT[2].\n\n\n[1] DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models\n\n[2] Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation", "questions_to_authors": "Please refer to reasons to reject", "paper_topic_and_main_contributions": "This paper proposes an information entropy aware diffusion model for NAR text generation. It considers the keyinfo-first generation strategy like humans into the existing continuous diffusion models, and revises the schedule of the diffusion model. Experimental results have shown the effectiveness of the proposed approach, on not commonly-used text generation datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[2] Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation"}, "extra_text": {}}, {"review_id": "pFIexNjwUC", "reviewer": "Reviewer_SUds", "summary": "", "questions": "I am not so clear about the difference between your work and other info-based diffusion language modeling methods.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review my paper and provide thoughtful feedback. I appreciate you recognizing the technical soundness and logical flow of the approach. Let us address your questions one by one.\n\n> Q1: Abstract is not well organized.\n\nA1：We will revise the abstract to clearly point out that the notable disparity refers to diffusion models prioritizing the generation of meaningless words, while human linguistic habits prioritize thinking about keywords.\n\n> Q2: Differences between your paper and some others by using info entropy to guide the diffusion process？\n\nA2：Our method places more emphasis on the \"keyinfo-first\" strategy of prioritizing the generation of keywords. To measure whether a word is a keyword, we assume that words with higher information entropy are more critical and important to a sentence. Therefore, we use information entropy as the control. Measuring key information through methods other than info entropy is also feasible. Meanwhile, the models proposed in other works, such as Masked-Diffuse LM[1] and DiffusionBERT[2], still follow an easy-first generation order. On the other hand, the \"keyinfo-first\" strategy we propose can also be combined with other strategies using info entropy to achieve better results. For example, we combined DiffusionBERT and implemented a discrete version of InfoDiffusion to further demonstrate the effectiveness and generalizability of the proposed method. The experimental results are shown in the table below.\n\n| Method                 | PPL$\\downarrow$ | BLEU$\\uparrow$ | Self-BLEU$\\downarrow$ |\n| ---------------------- | --------------- | -------------- | --------------------- |\n| D3PM[3]                | 77.50           | 0.4241         | 0.2288                |\n| Diffusion-LM[4]        | 118.62          | 0.3553         | 0.2668                |\n| DiffusionBERT          | **63.78**       | 0.4358         | 0.2151                |\n| InfoDiffusion-discrete | 68.46           | **0.4529**     | **0.2084**            |\n\nIn addition to the differences mentioned above, InfoDiffusion also outperforms other models using information entropy in terms of performance. For example, we compare the performance of InfoDiffusion with DiffusionBERT on the Quasar-T and QQP datasets as follows.\n\n| Dataset  | Method        | BLEU$\\uparrow$ | Rouge-L$\\uparrow$ |\n| -------- | ------------- | -------------- | ----------------- |\n| Quasar-T | DiffusionBERT | 0.0971         | 0.3420            |\n|          | InfoDiffusion | **0.1924**     | **0.3892**        |\n| QQP      | DiffusionBERT | 0.2420         | 0.5845            |\n|          | InfoDiffusion | **0.2656**     | **0.5928**        |\n\n[1]  [A Cheaper and Better Diffusion Language Model with Soft-Masked Noise](https://arxiv.org/abs/2304.04746)\n\n[2] [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://arxiv.org/abs/2211.15029)\n\n[3] [Structured Denoising Diffusion Models in Discrete State-Spaces](https://arxiv.org/abs/2107.03006)\n\n[4] [Diffusion-LM Improves Controllable Text Generation](https://arxiv.org/abs/2205.14217)\n\n> Q3: If the first keywords are not right, the whole generation will become serious, how to avoid this phenomenon\n\nA3: Please note that we do not use a left-to-right autoregressive decoding strategy. As a non-autoregressive model, keywords are also generated step-by-step through sampling. If keywords are generated incorrectly, the model still has a certain probability of correcting the results in subsequent diffusion sampling steps. For example, in Table 3 of the paper, the model corrects \"patrollingnine\" to the proper keyword \"geologist.\" On the other hand, if this method does not generate keywords well, the original approach of first generating irrelevant words may introduce more noise interference, making it even harder to generate the correct words.\n\n> Q4: What the advantages and disadvantages over the efficient methods, like dictionary-based methods or tf-idf-based methods.\n\nA4：Dictionary-based or tf-idf-based methods, and we even considered using PMI metrics, are just different standards for evaluating the information content of each word in a sentence. As stated in A2, our innovation lies more in discovering the gap and addressing it through the \"keyinfo-first\" strategy. Comparing this paper with information evaluation methods is unnecessary and unreasonable, as they represent work on different levels and in different directions. Additionally, we also tried using the tf-idf metric to measure word information in early experiments. Our method was still effective; the model could still exhibit a generation process prioritizing keywords, and generation quality could still be improved (8% improvement on the QQP dataset).\n\nIn general, we extend our heartfelt gratitude to the reviewer for providing us with such insightful comments.  We will revise the paper according to your recommendations to make it more concise and convincing. We sincerely anticipate the possibility of an improved rating for the paper due to your valuable insights. Thank you again for your time and effort.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. the idea is technique sound.\n2. the whole logic is easy to follow, although the abstract is not so clear.", "reasons_to_reject": "1. The abstract is not well organized, which make me confused, e.g., what's the notable disparity?\n2. What are the differences between your paper and some others that appeared before you by using info entropy to guide the diffusion process?\n3. In principle, the keyword first generation sounds reasonable, but if the first keywords are not right, the whole generation will become serious, how to avoid this phenomenon?\n4. The key contribution is the info-rich word generation, what the advantages and disadvantages over the efficient methods, like dictionary-based methods or tf-idf-based methods.", "questions_to_authors": "I am not so clear about the difference between your work and other info-based diffusion language modeling methods.", "paper_topic_and_main_contributions": "This paper supposes the easy word first generation process affects the generation quality, so they:\n1. proposes an info-rich keywords first generation schedule;\n2.. combine some superior tricks to boost the performance;\n3. The experiment results are good.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "other info-based diffusion language modeling methods."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary:

The paper presents a novel “key-info-first” continuous diffusion model and an information-entropy aware noise schedule for improving open-ended non-autoregressive text generation. The paper is generally well written, and the proposed approach seems sound. Experiments show positive results against selected baselines using automatic metrics like perplexity, BLEU, ROUGE, and BERTScore. No human evaluations were conducted to boost the claims. Additional experiments with discrete diffusions and other datasets were provided during rebuttal period, showing potential for an improved revised paper. 

Strengths:

1. Proposing a new diffusion process that is sensible for text generation by prioritizing keywords first, showing improved performance through automatic metrics. 
2. Proposing an information-entropy aware noise schedule
3. Experiments with automatic metrics, in addition to some qualitative examples, demonstrating the effectiveness of the proposal compared to selected baselines. 


Weaknesses: 

1. The overall presentation and writing should be improved based on reviewers’ comments. 
2. The paper needs to incorporate the additional experiments and analyses provided during the rebuttal period in the final revision.  
3. Missing human evaluation
Decision: Accept-Findings