DiffusionSL: Sequence Labeling via Tag Diffusion Process
Authors: Ziyang Huang
Pengfei Cao
Jun Zhao
Kang Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: Generative Models
Sequence Labeling
Tag Diffusion Process
TL;DR: Tackling Sequence Labeling using a non-autoregressive stepwise generative approach, along with Bit-Tag Converter and Bit Diffusion Transformer.

Abstract: Sequence Labeling (SL) is long-standing in Natural Language Processing (NLP). Traditionally, discriminative models have been widely used to capture the conditional distribution of sequence tags, rather than generative models. In this paper, we present DiffusionSL, a framework that utilizes a conditional discrete diffusion model for generating discrete tag data, resulting in a Tag Diffusion Process. We treat the natural language sequence as the conditional signal and the sequence tags as the generation target, iteratively refining the noisy tags to obtain clean ones. To address the discreteness issue, we propose the Bit-Tag Converter (BTConverter) to model the target in continuous data space. Furthermore, we introduce the Bit Diffusion Transformer (BitDiT) to model the process of noise elimination. Leveraging the powerful iterative refinement capability of the diffusion model, DiffusionSL achieves superior performance against previous state-of-the-art (SOTA) baselines and outperforms gpt-3.5-turbo significantly across multiple benchmark datasets and various tasks.

DiffusionSL: Sequence Labeling via Tag Diffusion Process
Ziyang Huang Pengfei Cao Jun Zhao Kang Liu 2
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China huangziyang2023eia ac.
{pengfei.com jzhao, kliu)@nlpr. ia. cc
Abstract
Most current methods tackled SL in discrimi- native manner (Akhundov et al., 2018), which usu- ally employed language model (Hochreiter and Schmidhuber, 1997; Devlin et al., 2019) to encode the sentence and added a token-level classifier to capture the conditional tag distribution (Zhang and Yang, 2018; Yan et al., 2019; Li et al., 2020; Cui et al., 2021; Wu et al., 2022).
Sequence Labeling (SL) is long-standing in Natural Language Processing (NLP). Tradition- ally, discriminative models have been widely used to capture the conditional distribution of sequence tags, rather than generative models. this paper, we present DiffusionSI. frame- work that utilizes conditional discrete diffu- sion model for generating discrete tag data, re- sulting Tag Diffusion Process. We treat the natural language sequence the condi- tional signal and the sequence tags as the gen- eration target, iteratively refining the noisy tags to obtain clean ones. To address the discrete- ness issue, we propose the Bit-Tap Converter (BTConverter) model the target in continu- ous data space. Furthermore, we introduce the Bit Diffusion Transformer (BitDiT) to model the process of noise elimination. Leveraging the powerful iterative refinement capability of the diffusion model, DiffusionSL achieves su- perior performance against previous slate-of- the-art (SOTA) baselines and outperforms gpt- 3.5-turbo significantly across multiple bench- mark datasets and various tasks! !
Recently, generative pre- pre-trained language mod- els have demonstrated their versatility in various NLP tasks (Raffel al., 2020). In consequence, researchers attempt to formulate SL. as a sequence generation problem (Athiwaratkun et al., 2020; Paolini et al., 2021; Yan et al., 2021; Lu et al., 2022). This paradigm offers the advantage of flexi- ble generation formats (Raman et al., 2022), allow- ing models to be trained to produce output in any desired format using teacher forcing. In these gen- erative approaches, the input sentence is encoded first, and the label sequence is then decoded an autoregressive manner, as depicted Figure (a)). Nevertheless, the intrinsic loken-by-token gener- ation style of the autoregressive model results in inefficient inference, and the difference in condi- tional signal during training and inference leads to the exposure bias problem (Arora et al., 2022).
1 Introduction
Sequence Labeling (SL) is a basic paradigm in the natural language processing (NLP) field, which assigns a predefined label every meaningful unit (e.g.. word or character) in a given sequence (He et al., 2020; Lu et al., 2019; et al., 2021; Liu et al., 2021). Many NI.P tasks belong to this cat- egory, such as Named Entity Recognition (NER) (Zhu and Li, 2022; Liu et al., 2022; Shen et al., 2022; Shen et al. 2023; He and Tang 2022; Zhou
To this end, this paper resorts to the Diffu- sion Probabilistic Model (DPM) (Sohl-Dickstein et al., 2015; Ho al., 2020; Nichol and Dhariwal, 2021; Song and Ermon, 2019; Song et al., 2021b), renowned as one the most powerful generative models in the field of Artificial Intelligence Gener- ated Content (AIGC) (Cao et al., 2023; Wu et al., 2023a), especially in the realm of image synthesis (Liu et al.; Rombach et al., 2022; Hong et al., 2022). DPM gradually introduces noises into clean data through a predefined forward process and subse- quently employs denoising network in reverse process to recover the original data from pure Gaus- sian noise. The generation process operates in par- allel. Compared with an autoregressive decoder,
al. 2022b), Chinese Word Segmentation (CWS) (Fu al., 2020; Huang et al., 2021; Maimaiti et al., 2021), Pfrt-Of-Speech (POS) tagging (Zhou et al., 2022a; Nguyen and Verspoor, 2018).
'Code is available at httpss://wwt www.bgitthu...... com/ f2usson

---

Golden Targel:
Sentence:
EU
rejects
Germon
call
to
boyco
Britih
lamb
[FU I ORG] reject [Gemoon MISC] call baycatt (Briish MISC] lamb
[EU, ORO]
[0erman, MISQ
[Britsh, MISQ
Generated Target with Autoregressive Model:
reverso diffution provess: recovar dota from moise
MSC rejects [German MISC] call boycolt Biritsh MISCI lamb
Golden Torget
S-ORG
S-MISG
O
S-MISC
x
Noay:
S-ORG
S MISC
FORG
O
X
MISC
rofine
x rafine
x
More
Nollyt
S-MISC
HMISC
X;
ENCODER
DECODER
41 EU
MISC
Nose:
Intilo
0
FORG
EMISC
S-PER
FORG
JAMSC
B-ORG
Xy
sample
rojecs German call boycolt British lamb
Gauesion Noise
(a) Autoregressive sequence target generation process
(b) Non-Autoregressive sequence target diffusion process
Figure I: (a) shows traditional autoregressive generative method (Paolini al., 2021) for typical SI. task, Named Entity Recognition (NER). This line of methods generates the target at once and can not refine them. (b) demonstrates our non-autoregressive stepwise generative framework DiffusionSL for NER. inference stage, the reverse diffusion process of DiffusionSL utilizes well-trained denoising network remove the noise step-by-step and recover the clean tags finally, allowing contents (task-specific tags) refinement in this process. Symbols marked yellow background are noisy, whereas symbols without yellow background represent the desired generated ones.
it could naturally solve the exposure bias problem. Furthermore, the diffusion model can generate the targets progressively to refine the contents step-by- step, thereby facilitating the correction of some mistakes during the generation process. com- parison, the conventional autoregressive generative model generates the targets at once. The afore- mentioned advantages contribute to the success of the diffusion model in generating data across vari- ous modalities (Zhang et al., 2023a; Trippe et al., 2023; Li et al., 2022c; Li et al., 2022b). Intuitively, expect that DPM could also present superior performance SL tasks.
forts have addressed this issue (c.g., text symbols (Gong et al., 2023), segmentation maps (Chen et al., 2023b, 2022)). Following them, this paper intro- duces lightweight and flexible module named Bit- Tag Converter (BTConverter) represent the tags as binary bits, which can be viewed as real numbers in continuous space. Additionally, we propose the Bit Diffusion Transformer (BitDiT) to model the noise-removal reverse process, which enjoys faster convergence and stronger performance.
Currently, Shen al. (2023) also introduces a diffusion model into NER, an important SL task. However, their approach only models the diffusson process on entity span boundaries, restricting itself to NER and impeding the potential for extension to other SL. tasks.
Therefore, this paper proposes novel frame- work called Di The input sentence serves as the conditional signal, while the corresponding taskerelated tag sequence is the generation target. We call the entire diffusion process as the Tag Dif- fusion Process. During the training phase, fixed forward process utilized sequentially perturb the tags and reverse diffusion process is learned to eliminate noise based on the noisy tags and the en- coded sentence. In the inference phase, we sample noisy tags from a standard Gaussian distribution and employ the well-trained denoising model to refine these tags into clean ones in an incremental manner (shown in Figure 1(b)).
In summary, our contributions are as follows:
This study proposes a novel framework Diffu- sionSL that addresses the Sequence Labeling (SL) tasks using a non-autoregressive step- wise generative approach. To the best of our knowledge, we are the first to apply the diffu- sion model to SL rather than only NER.
We introduce two key modules, BTConverter and BitDiT, which convert the discrete data into continuous bits and model the iterative denoising process. The first module effec- tively handles the discreteness issue while the second one accelerates the convergence.
However, directly applying a DPM on SL task is challenging. Classical DPM models the image data and Gaussian noise in a continuous space. In- stead, tags in SL are discrete. A few research ef-

---

Tag Diffusion Process shown in Figure 2(a) de- scribes conditional generation process where (4) the conditional signal is input sentence W (Wi,..., w., and the target is the task-specific tags L = (G..... ......,,).. The set of all possible tags is denoted as e.g. (B, M, E, S) for Chinese Word Segmentation. Initially, we construct indices for tags in S and transform them into bits in continu- ous space using BTConverter. The resulting trans- formed clean tag data is referred to xo- Noise is incrementally injected into Xo during the forward diffusion process, while a denoising network is learned to gradually eliminate the noise during the We utilize the DiffusionSL for NER, CWS. and POS tagging. The thorough experiments on several datasets of these typical tasks indi- cate that DiffusionSL can achieve better than previous SOTAs, highlighting the superiority of the proposed Tag Diffusion Process.
Therefore, we train the Mg to predict the jug. Due to x Varaa + /1 aye, we could get bo by predicting it directly or predicting c, Xo indirectly. The variance Eg could be learned or fixed. We keep it fixed in this work. The training objective is the variational lower bound of the negative log- likelihood log pe pe Xo), which can be simplified to predict the clean data Xo or injected noise e.
2 Preliminary
The diffusion model is characterized as a latent gen- erative model. It encompasses forward process that involves the gradual introduction of noise to clean data Xg. Additionally, it includes a reverse process that incrementally eliminates the noise, thereby generating data following the original dis- tribution. The forward process ((xe|x; 1) can be mathematically expressed as:
To expedite inference, it is possible to skip cer- tain timesteps in the reverse process. By taking in the more noisy samples at timestep 4, we can generate the less noisy samples at timestep 6;-1:
pg (xx,) ~ (i, E)
(9)
i v 101,1x00
X1, vār,Xx
(10)
T II ((xx/|x11)) [=1
v1 04
(1)
Σ = n
(11)
((x/\x--1)~ ~ ~ 31x1-1,
(2)
where &g represents the predicted original data by a learned network. 1) is usually set to O, thus result- ing in a deterministic generation process which is called DDIM sampling (Song et al., 2021a).
Here, V represents the Gaussian distribution and 3, represents the noise introduced at timestep t. The magnitude of the noise is governed by a noise schedule 3 - {(3,)(=1- Generally, the noise in- tensity usually follows either a lincar (Ho et al., 2020) or cosine (Nichol and Dhariwal, 2021) func- tion. In this study, we adopt the linear noise sched- ule. By employing the notations o = 1 3e and ā; II'= O'g. we can deduce an equation to di- rectly incorporate noise into Xo, yielding the noisy data any given timestep:
3 Methodology
The details of DiffusionSL are illustrated in Figure 2. We will provide a comprehensive explanation of it in this section. Firstly, we formally describe the overall procedure of the Tag Diffusion Process employed by DiffusionSL. Secondly, we present detailed description of two components we pro- posed to help modeling the Tag Diffusion Process: BTConverter and BitDiT. Lastly, we summarize the training and inference algorithms of DiffusionSL.
(xx(xx0) ~ NN (1- (I)))
(3)
Additionally, the reverse process g(x;-1| Xx) al- lows us to reconstruct samples from pure Gaussian noise, which is not tractable directly. We can use neural network pe to estimate it:
3.1 Tag Diffusion Process
po(x)r)) p(xx)) px(x: 1xx))
po(x:- 1|x+) ~ (Ho (xt,,) t), Ex(xx,t) t))
(5)
The conditional distribution of x; given x; in reverse process could be obtained analytically as follows when Xo is available:
qx Xt, xo)
(ir(Xx, xo), B.I)
(6)
Va-18
Vor(1 σi
1)
3, 1 at
Xo
+
- Or Xt
3
1 - 01-1
Br
(8) reverse diffusion process, ultimately generating the desired tag data.

---

(a) Tag Diffusion Process
X7
X
X--1
X
S-MISC
S-ORG
1ORG
0
0
S-MISC
Pe (xx))
sample
S-PER
0
(xp1xx111
HORG
HWSSC
I-MIS
0
O
Gaussian Noise
label acc:
label
acc:
S-MISC
aOC:
B-RGG
50%
more
- -N Σ)
decade
A Σ
S-ORG [I]
0001
EU
ORG
Xg bds. (x W.t)
(D)
0000
rojocts
rejects
tag2bit
S-MISC [5)
0101
Gorman
:Genmam .........d
MISC
10)
0000
call
call
BTConverter
ENCODER
DECODER
(0)
bit2tag
0000
boycolt
boycolt
S-MISC (5)
0101
British
MISC
10)
0000
lamb
lamb
(b) Bit- Converter
(c) Bit Diffusion Transformer
The overall picture of our framework DiffusionSI. (a) depicts the Tag Diffusion Process that eliminates the noise in the tag data in progressive manner, which transforms the noisy tags sampled from a Gaussian distribution into clean tags. In the actual scenario, the tags in this diffusion process are in bit representation, we convert them into discrete tag representation for clarity in illustrating the denoising process. (b) demonstrates the Bit-Tag Converter (BTConverter) which enables flexible conversion between the discrete tags and the continuous bits. (c) presents the Bit Diffusion Transformer (BitDiT), the denoising neural network.
Figure 2:
Specifically, we follow the Equation (3) to per- turb the data for the forward diffusion process. More specifically, noisy tags {x, at each noise level could be obtained by the following equation:
By iteratively performing À denoising steps, the desired target tags are acquired.
3.2 Bit-Tag Converter
The diffusion model is unable to generate the dis- crete data directly. Thus, we propose Bit-Tag Con- verter (BTCCnverter) to transform the discrete se- quence tags into the bits during training (tag2bit) t) and convert the bits back to the discrete tags dur- ing decoding (bi t2tat).. The functionality of the BTConverter is illustrated in Figure 2(b). We con- struct an index for every tag in S. If the number of elements in the tag set is (S], we use [log, [S]] bits to represent tags. We assign 0 ~ S 1 to every tag in S sequentially, and the bit representation of the i-th tag is its corresponding binary bits (i.e., (101)2 represents the 5-th tag), which could be de- noted as tag2bi t(i). After converting the tags into bits, every bit is taken an independent real num- ber. Then, they are shifted and scaled the range of [-b, b], where represents a parrmmterr (13) named signal- (SNR) (Xu et al., 2023; Ji
(xx: VaxXo (1 ag)e
(12)
During the reverse diffusion process, we sample an incomplete timestep sequence of length À to gen- erate the target tags. We train a denoising network, denoted as bdg. which takes in the noisier tags Xt, at the timestep tü along with the input sentence W and then outputs the predicted An- Here, bd is the abbreviation for BitDiT. Further details on this network will be provided Section 3.3. Sub- sequently, we can obtain the less noisy tags X following the Equation (9), (10) and (11):
X
W. 4)+
04,1 o Xt: v Ct ( W,
VI 6t,

---

Algorithm 1 Training for DiffusionSL
Algorithm 2 Inference for DiffusionSL
input_ids, attention_mask: [bsz, len] # gold_seq_tagst [bsz, len]
# input_ids, attention mask: [bsz, len] encode condi tional signal with PLM hid encoder (inputttin attention_mask)
encode conditional signal with PLM hid (iccoerrinnut_Liie attention_mask)
begin with isotropic Gaussian bits Normal 1)
convert tags into continuous bits # [bsz, seq_len, num_bits]
denoise the data iteratively range(T):
bag_bits taglbit(gold__ seq_sse______ seq_tags) tag_bits (tag_bits
1) SNR
bits_0_pred bd(bits, t, hid) bits ddin_sample(bits...
t Uniform(1, T)
epsilon Normal 1)
tags bit2tag(bits)
crpt_bits sart(alpha_bar((t) & tag_bits sqrt((1 alpha_bar(t))) epsilon
return tags
pred_bits bd(corrupt_bits, t, hid)
loss lpe_e_ssspred_bit bitsbits)
Condition Embedder: It encodes the timestep t using a learned lookup table and fuses with the cached H to get the conditional signal cN allowing the decoder to be aware of input and time:
return loss
et al., 2023). After the inference with the reverse diffusion process, we use bi taglla)(bils) to convert bits back into discrete tags, which then decoded based on the schema of the downstream task. Fur- ther details of the tag2bit and bi bitt functions are provided in Appendix A.
C₁ Cllnear(Concate((( Embeds(()) (16)
Pre Layer: This layer consists multi-layer perceptron (MLP) that projects the bit represen- tations to a higher dimension for modeling more fine-grained interactions between tags.
3.3 Denoising Network: Bit Diffusion Transformer
Generator: This component consists of N our designed Transformer blocks. It generates clean tags based on the conditional signal C₁-
To eliminate noise during the reverse diffusion pro- cess, we employ a denoising network based on the Transformer architecture (Vaswani et al., 2017). This network, named Bit Diffusion Transformer (BitDiT), takes the bit representation of the noisy tags Xt, the input sentence W, and the timestep inputs and generates the predicted clean tags ān- The architecture of BitDiT illustrated in Figure 2(c). It consists a BERT-style (Devlin et al., 2019) encoder that represents the sentence in hidden space and a decoder that generates the targets in parallel. The encoder and decoder are parameterized by 0 and respectively. The overall functionality of the denoising network could be represented follows:
Post Layer: This layer performs layer normal- ization to normalize the representations and then applies another MLP to project them back to the original dimension.
The generation process can be summarized as:
X Post (Generator(Pre(xx)) c,))
(17)
where Pre and Post represents the Pre Layer and the Post Layer. Instead of using cross-attention to incorporate the conditional signal in the Trans- former blocks of Generator, layer normalization (Perez et al., 2018; Brock et al.; Li al., 2022a: Peebles and Xie, 2022), which pre- diets the element-wise affine parameters based on the condition C₁ to substitute the naive layer nor- malization (Ba et al., 2016) used in the original Transformer (Vaswani et al., 2017). The layer nor- malization used in the Post Layer has also been modificd to this accordingly. We also learn ad- ditional scale factor for this Transformer before each skip connection (He al., 2016) to control the amount of newly learned information added. We refer the architecture diagram and calculation details this Transformer block in Appendix B.
resort adaptive
X (X,, W.t)
(14)
In more detail, we first obtain the encoding vec- tors of the input sentence and cache them for all inference timesteps:
H Encodery W)
(15)
Next, we use the decoder to generate the clean tags. The decoder comprises four components:

---

MSRA
Resume
Conlios
Methods
P
R
R
R
discriminative
BiLSTM-agger
90.74
86,96
88.81
93.66
93.31
93.48
Zhang and Yang (2018)
93.57
92.79
93.18
94.81
94.11
94.46
Yan et al. (2019)
92.74
95.00
91.45
Li al. (2020)
94.12
93.45
Wu et al, (2022)
94.92
94.19
94.55
95.63
95.52
95.58
generaaive
Athiwaratkun al. (2020)*
92.00
Puolini al. (2021)"
(2022)'
92.17
DiffusionNER
95.71
94.11
94,91
92.99
92.56
92.78
DifTusionNTER suuiduation
(95.64)
(93.97)
(94.80)
(96.64)
(91 (48
(94.18)
(92.35)
(92.76)
(92.55)
8pi-3.5-turbof (ane-shot)
48.03
57.47
52.33
64.74
28.16
39.25
34.67
61.75
44.41
DiffusionSL (Ours)
95.69
95.28
95.49
96.58
96.22
96.40
93.15
92.26
92.70
Table 1: Results of NER. demonstrates the results the autoregressive generative methods, which exhibit inferior performance compared the non-autoragressive ones. The results of DiffusionNER (Shen al., 2023) on Resume dataset were not reported in the original paper, so we reproduce and test on it, along with the other two datasets. A represents the results reported in the original DiffusionNER paper while the refers to our reproduced results. $ is the results obtained by prompting gpt-3.5-turbo with demonstration example.
3.4 Training and Inference
capture the conditional tag distribution and the lat- ter generates the desired outputs at the decoder end. For NER baselines, the works of Huang et al. (2015a), Zhang and Yang (2018), Yan et al. (2019), Li et al. (2020) and Wu et al. (2022) are discrimina- tive while the works of Athiwaratkun al. (2020), Paolini al. (2021), Lu al. (2022) and Shen al. (2023) are generative. CWS (Yang al., 2017: Ma et al., 2018; Yang et al., 2019; Tian et al., 2020) and POS (Diao et al., 2020; Meng et al., 2019) base- lines are all discriminative ones. We also compare one of the most powerful Large Language Mod- els (LLMs), gpt-3.5-turbo (OpenAI, 2023). More details of baselines deferred to the Appendix E.
For training, the Mean Squared Error Loss (MSE Loss) regress the predicted bits:
MSE Rxx W. w.,.cl/x bdo (xx, W, 1 (18)
For inference, we sample sequence of tags in bit representation with the same length as the in- put sentence from a standard Gaussian distribution. Then, we conduct À denoising steps to get the de- sired clean bits. The discrete tags are acquired by the bit t2tag function of the BTConverter. The pseudo-code of training and inference is provided in Algorithm 1 and Algorithm/ 2, respectively.
4 Experiments
4.2 Overall Results
DiffusionSL. achieves better performance var- ious datasets across different tasks. Promising results validate the effectiveness of DiffusionSL. over the previous baselines, including both dis- criminative and generative methods. NER results in Table show that DiffusionSL achieves 95.49, 96.40, and 92.70 F1 scores on MSRA, Resume, and Conl103 datasets. Meanwhile, for CWS, Dif- fusionSL achieves 98.33, 96.54, 97.46 Fl scores on MSRA, PKU, and CTB6 respectively according to Table 2. For POS tagging, our method achieves 97.18 F1 score on CTB5, with improvement of 0.36, which is demonstrated in Table The strong performance proves this non-autoregressive step- wise generative approach can generate task sseciicccific tags effectively and thus tackle the corresponding
4.1 Experimental Settings
We conduct experiments to evaluate the proposed DiffusionSL framework seven different datasets. We choose MSRA (Levow, 2006), Resume (Zhang and Yang, 2018), and Conl103 (Tjong Kim Sang and De Meulder, 2003) for NER, MSRA, PKU (Emerson, 2005), and CTB6 (XUE et al., 2005) for CWS, and CTB5 (XUE et al., 2005) for POS tagging. The detailed statistics of these datasets are provided in Appendix C. We provide the detailed hyper-parametter settings in Appendix D.
We compare the previous models for NER, CWS. and POS tagging to test the effectiveness of the Dif- fusionSL. Compared methods can be divided into discriminative and generative ones. The former

---

CWS
MSRA
PKU
CTB6
Bit
Fmheddingeons
Fmheddinggss
Ma et al. (2018)
98.10
96.10
96.70
NER-Resume
96.40
50.07
96.28
Yang al. (2019)
97.80
95.90
96.30
POS-CTR5
97.18
96.72
96.86
Tian et al. (2020)
98. 16
96.47
97.13
Cui et al. (2021)*
98.31
96.51
97.39
Table 4: Ablation study of tag representation on Resume NER dataset and CTB5 POS tagging dataset. #bits is the bit representation dimension.
DiffusionSII(Ours)
98.33
96.54
97.46
POS lagging
CTB5
50
100
Diao et al. (2020)
96.64
Meng et al. (2019)
96.61
NER-Resume 96.37
96.40
96.40
96.40
Cui et al. (2021)²
96.82
POS-CTB5 97.18
97.18
97.19
97.19
DiffusionSL (Ours)
97.18
Table 5: Ablation study of denoising sampling steps on Resume NER dataset and CTB5 POS tagging dataset.
Table 2: Results of CWS and POS tagging. Y denotes the results reproduced by us.
F1 performance during training stage
Shot
-Cross Attn -BiLDIT eCros Attn Max ABHDT Max
100
96.58
FI
41.78
44.49
46.73 39.25
90
95.34
Table 3: Gpt-3.5-turbo experiments results Resume dataset. Shot is the number of demonstration example used in prompting. Due to the budget of OpenAl api, we only test one dataset.
80
70
60
50
40
30
problems effciently.
20
Compared to DiffusionNER, which is our con- current work that applies the diffusion model on NER, we surpass it on all the datasets, illustrat- ing the advantages of Tag Diffusion Process over the entity span boundary diffusion process. In our view, the generation format (e.g., span boundary) is unnatural, and it separates the span localization and the entity label classification to some extent. Hence the diffusion model not model the entity label information well. Besides, this generation format necessitates extra heuristic post- post-procsssing operations. As a result, it limits this framework to the NER task and results in sub-optimal perfor- mance. Our DiffusionSL can not only handle NER but also all other SL tasks, which enjoys stronger performance and more flexibility at the same time. We also compare the gpt-3.5-turbo (OpenAI, 2023), which one the most powerful LLMs among OpenAl gpt-series, on NER datasets. We prompt the LLM with one demonstration example (one-shot) for three NER datasets. The experiment of adding more demonstrations is also conducted on Resume dataset. Our experimental results in Table 3 test that the performance will not increase infinitely with the increase of demonstrations. We find that LLM still could not catch up with the task- specific supervised small models. Thus exploring the specific algorithms for sequence labeling is still meaningful. The corresponding prompt details are
10
opoch
Figure 3: The convergence process of Cross-Attn and BiIDiT. Here, *-Max denotes the best performance of method # The best performance of BitDiT differs from the result reported in Table 1, as the latter presents the test Fl score corresponding to the highest dev Fl.
shown in Appendix
5 Ablation Study and Analysis
5.1 Effects of BTConverter
We conduct experiments to compare with the most intuitive and straightforward alternative, embed- ding method, which creates a learnable embedding table for each discrete symbol in S. Results are shown in Table 4. During training, a prototype embedding is learned for each tag. In inference phase, the embeddings generated from the diffu- sion model are decoded into discrete tags based on their distance to different prototypes. For fair comparison, firstly set the dimension of the embedding table the same the bit representa- tion dimension. Resume has 13 tags and CTB5 has 103 tags in S. Their corresponding bit dimen- sion 4 and 7. The embedding representation exhibits inferior performance compared to the bit

---

representation. Even if scale the dimension of embedding representation 768, still can not out- perform bit representation and requires additional [] x htog trainable parameters. Additionally, the decoding method requires calculating the distance from each tag embedding to each prototype, which is more complex than the bit2tag function in BT- Converter that only operates once for every tag during decoding. These results demonstrate the lightweight and flexible nature of BTConverter.
sentence at the final step. Then, we can decode the desired entities 1 (CPC Central Com- mitteeess) NT), (+ 1- (the 11th National Congress of the Chinese Zhigong Party), NT) by the semantics these tags. To the best of our knowledge, we are the first to introduce this iterative denoising process and shift the traditional classification paradigm to the non-autoregressive generation paradigm for sequence labeling.
Related Work
5.2 Effects of BitDiT
7.1 Sequence Labeling
BitDiT incorporates conditional signals through adaptive layer normalization, whereas the standard Transformer decoder block and the DiffusionNER utilize an additional cross-attention layer to percep- tualize the conditional information. We refer to this method Cross-Atte (Vaswani et al., 2017; Shen et al., 2023). Figure 3 shows the convergence process of BitDiT and Cross-Attn the Reusume dataset. From the result, we discover that Cross- Attn exhibits slower convergence speed and inferior final performance compared to BitDiT. This find- ing underscores the significance of our proposed BitDiT architecture.
Sequence Labeling (He et al., 2020) assigns a unique label with special semantics to each unit in sentence. Named Entity Recognition (Huang et al., 2015b), Part-of-speech tagging (Todi et al., 2018), Chinese Word Segmentation (Shi al., 2017), Text Chunking (Zhai et al., 2017), 7), Slot Filling (Zhang et al., 2019), Semantic Role Labeling (Daza and Frank, 2018) and many other NLP tasks could be tackled using sequence labeling. Traditional ma- chine learning methods (Baum and Petrie, 1966; Lafferty et al., 2001) and deep-leamning based meth- ods (Huang et al., 2015b; Devlin et al., 2019) are proposed address it. Recently, knowledge- enhanced pretrained encoders (Diao et al., 2020; Liu et al., 2021; Jiang et al., 2022) boost this field significantly.
5.3 Effects of Denoising Steps
We validate the impact of denoising steps on perfor- mance, while the corresponding results are shown in Table 5, We find that the performance improves with the increase of the steps until reaching the bot- tleneck. Owing to the iterative refinement nature of the diffusion model, we can increase the sampling steps to boost the performance (Chen et al. 2022, Shen et al. 2023). But the performance will not improve infinitely with the increase of the steps. So we choose 10 as the defaulting steps for balance between the performance and the speed.
7.2 Diffusion Model
Diffusion model derived from the image gen- eration field (Sohl-Dickstein al., 2015). The strong performance the diffusion-based gener- ative model (Ho et al. 2020, Ramesh et al. 2021) significantly boosted the development of the im- age field. Thus, lots of eyes from other fields are attracted by the diffusion model, e.g., speech synthesis (Zhang et al., 2023a), protein design (Trippe et al., 2023), time series forecasting (Li et al., 2022c), natural language generation (Li al., 2022b). For NLP, the discreteness nature of lan- guage symbol poses several challenges. Nonethe- less, many techniques are proposed to boost the development of this field (Hoogeboom et al., 2021; Li et al., 2022b; Yuan et al., 2022; Lin et al., 2023; He et al., 2023; Zhou et al., 2023). We provide a more detailed survey about applying the diffusion model to the natural language field in appendix G. Currently, diffusion model dominats the field of generative AI and there are so many works chasing to improve the diffusion model from different per-
6 Case Study
We visually show the denoising process details by one Chinese NER case. In Figure 4, we input the sentence + + + 2 1- jnj (The CPC Central Committee's s congratulatory mes- sage to the 1 11th National Congress of the Chinese Zhigong Party) as condition signal, then we gen- erate the corresponding tags in ten decoding steps by iterative refinement. We also show the label ac- curacy of tags in the bottom for every step, which increases with the number of reverse steps. Finally, we get the exactly correct NER tags of the input

---

denaising process
R
MMG
646
M40
MAG
MWG
PWS
MMM
ANT
640
KAB
5N0
MAP
148
6:8
A44
A-R
440
648
646
648
648
0663
6333
6473
Figure 4: Visulization example for Chinese NER.
means The CPC
Central Committee's 's congratulatory message to the 1ith National Congress of the Chinese Zhigong Party.
spectives, e.g., likelihood estimation (Kingma et al., 2021), efficient sampling (Song al., 2021a), con- dition guided diffusion (Ho and Salimans, 2022).
sampling steps, it is still slower than the discrimi- native models. Secondly, BitDiT incorporates an extra candom-initialin BERT-style tagger models, which needs more com- putation memory and is harder to train, we must search the hyperparameters thoroughly to find an optimal result. Thirdly, the initial noise is sam- pled from Gaussian distribution. thus bringing randomness in the Tag Diffusion Process.
decoder compared the
8 Conclusion
In this paper, we cast the Sequence Labeling (SL) task conditional generation problem. Further- more, introduce novel framework DiffusionSL. model SL using non-autoregrressive stepwise generation approach. Meanwhile, we propose BT- Converter handle the discreteness problem and BitDiT to model the progressive neise-removal pro- cess. Experimental results indicate that our model performs better on NER, CWS, and POS tagging compared to previous strong baselines, thus reveal- ing the superiority of DiffusionSL. Our work is pioneer research cast Natural Language Under- standing task as a non-autoregressive correspond- ing generation task. We hope DissusionSL could used more tasks that could be formulated as sequence labeling and leave this as the future work, e.g. semantic role labeling, text chunking.
Acknowledgement
This work was supported by National Key R&D Program of China (No. 2022ZD01 18501) and the National Natural Science Foundation of China (No. U1936207).. This work was supported by the Strategic Priority Research Program Chinese Academy of Sciences NN..XDA27020100, Youth Innovation Promotion Association CAS, and Yun- nan Provincial Major Science and Technology Spe- cial Plan Projects (No.00200AD080004))
References
Limitations
Adnan Akhundov, Dietrich Trautmann, and Georg Groh. 2018. Sequence labeling: A practical approach. arXiv prepriat 1088v=1808.03922
There are several disadvantages that hard to avoid for DiffusionSL.. Firstly, more sampling steps result in a slowerer sampling speed. Though we choose ddim as sampling method to decrease the
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Cheung. 2022. Why exposure bias matters:

---

An imitation learning perspective of error accumu- lation in language generation. In Findings of the A Association for Computational Linguistics: ACL 2022, pages 700-710, Dublin, Ireland, Association for Computational Linguistics.
Angel Daza and Anette Frank. 2018. sequence-to- sequence model for semantic role labeling. In Pro- ceedings of the Third Workshop on Representation Learning for NLP, pages 207-216, Melbourne, Aus- tralia. Association for Computational Linguistics.
Ben Athiwaratkun, Cicero Nogueira dos Santos, Jason Krone, and Bing Xiang. 2020. Augmented natu- ral language for generative sequence labeling. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 375-3--3503 385, Online. Association for Computa- tional Linguisties.
Jacob Devlin. Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17993.
Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. 2020. ZEN: Pre-training Chinese text encoder enhanced by n-gram representations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4729-4740, Online. Association for Computational Linguisties.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Sander Dieleman, Laurent Sartran, Arman Roshan- nai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. 2022. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089.
Nikita Balagansky and Daniil Gavrilov. 2023, Democ- ratized diffusion language model. arXiv preprint arXiv:2305. 10818.
Leonard Baum and Ted Petrie. 1966. Statistical inference for probabilistic functions of finite state markov chains. The annals of mathematical statis- tics, 37(6))1554 1563.
Thomas Emerson. 2005. The second international Chi- nese word segmentation bakcoff. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural im- age synthesis. In International Conference on Learn- ing Representations.
Jinlan Fu, Pengfei Liu, Qi Zhang, and Xuanjing Huang. 2020. RethinkCWS: Is Chinese word segmentation a solved task? In Proceedings of the 2020 Conference Empirical Methods in Natural Language Process- ing (EMNLP), pages 5676-5686. Online. Association for Computational Linguistics.
Yihan Cao, Siyu Li. Yixin Liu, Zhiling Yan. Yutong Dai, Philip S Yu, and Lichao Sun. 2023. A comprehensive survey of ai-generated content (aige): A history of generative ai from gan to chatgpt. arXiv preprint arXiv:2303.04226.
Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. 2022. Difformer: Empowering diffusion model on embedding space for text generation. arXiv preprint arXiv:22 221.22221999112
Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, and Diyi Yang. 2023a. A cheaper and better diffusion lan- guage model with soft-masked noise. arXiv preprint arXiv:2304.04746.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2023. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations.
Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J Fleet. 2022. A generalist framework for panoptic segmentation of images and videos. arXiv preprint arXiv:2210.06366.
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. 2022. Ssd-Im: Semi-auttoeerressvve simplex-based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210. 17432.
Ting Chen, Ruixiang ZHANG, and Geoffrey Hinton. 2023b. Analog bits: Generating discrete data us- ing diffusion models with self-conditioning, In The Eleventh International Conference on Learning Rep- resentations.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In 2016 IEEE Conference on Computer Viston and Pattern Recognition (CVPR), pages 770-778.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese BERT. IEEE ACM Trans, Audio Speech Lang, Process., 29:3504-3514.
Yuxin He and Buzhou Tang. 2022. SetGNER: Gen- eral named entity recognition as entity set generation.

---

In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 3074-3085, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Peijie Jiang, Dingkun Long, Yanzhao Zhang, Pengjun Xie, Meishan Zhang, and Min Zhang. 2022. Unsu- pervised boundary- language model pretrain- ing for Chinese sequence labeling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 526-537, Abu Dhabi, United Arah Emirates. Association for Com- putational Linguistics.
Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang. Xuanjing Huang. and Xipeng Qiu. 2023. Dif- RUR: Improving generative masked language models with diffusion models. In Proceedings of the 61st Annual Meeting of the Association for Compu- lational Linguisticy (Volume 1; Long Pupers), pages 4521-4534, Toronto, Canada. Association for Com- putational Linguistics.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. 2021. On density estimation with dif- fusion models. In Advances in Neural Information Processing Systems.
Zhiyong He, Zanbo Wang, Wei Wei, Shanshan Feng, Xianling Mao, and Sheng Jiang. 2020. A survey on recent advances in sequence labeling from deep (tl::2011.0672..
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling se- quence data. In Proceedings of the Eighteenth In- ternational Conference on Machine Learning, ICML "01, pages 282-289,, San Francisco, CA, USA. Mor- gan Kaufmann Publishers Inc.
learning models. arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models. In Advances Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Gina-Anne Levow. 2006. The third international Chi- nese language processing bakeoff: Word segmenta- tion and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop Chinese Language Processing, pages 108-117, Sydney, Australia. Asso- ciation for Computational Linguistics.
Jonathan Ho and Tim Salimans. 2022. Classifier- free diffusion guidance.
arXiv preprint
arXiv:2207. 12598.
Jingye Li. Hao Fei. Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng. Donghong and Fei Li. 2022a. Unified named entity recognition as word- word rela- tion classification. In Thirty-Sixxhh AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fowe Fourth Conference on Innovative Applications of Artificial Intelligence, TAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pages 10965-10973, AAAI Press.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 98):1735- 1780.
Susung Hong, Gyuseong Lee, Wooseok Jang, and Se- ungryong Kim. 2022. Improving sample quality of diffusion models using self-attention guidance. arXiv preprint 1XX::22000000333
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. 2021. Argmax flows and multinomial diffusion: Learning categorical dis- tributions. In Advances in Neural Information Pro- cessing Systems, volume 34, pages 12454-12465. Curran Associates, Inc.
Junyi Li, Wayne Xin Zhao, Jianyun Nie, and Ji rong Wen. 2023. Glyphdiffusion: Text generation as im- age generation. arXiv preprint arXiv:2304.125/1
Xiang Lisa Li. John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. 2022b. Diffusion- LM improves controllable text generation. In Ad- vances in Neural Information Processing Systems.
Kaiyu Huang, Jumpeng Liu, Degen Huang. Deyi Xiong, Zhuang Liu, and Jinsong Su. 2021. Enhancing Chi- nese word segmentation via pseudo labels for prac- ticability. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages Online. Association for Computational Linguistics.
Xiaonan Li, Yunfan Shao, Tianxiang Sun, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2021. Accel- erating BERT inference for sequence labeling via early-exit. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 189-199, Online. Association for Computa- tional Linguistics.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015a. Bidi- rectional LSTM-CRF models for sequence tagging. CoRR, abs/ Na01111999..111
Zhiheng Huang, Wei Xu, and Kai Yu. 2015b. Bidirec- tional lstm-crf models for sequence tagging. arXiv preprint
Xiaonan Li, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. FLAT: Chinese NER using flatelattice transformer. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages (836-6842, Online, Association for Computa- tional Linguistics.
Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. 2023. Ddp: Diffusion model for dense visual prediction. arXiv preprint urXiv:2303.17552

---

Yan Li. Xinjiang Lu, Yaqing Wang, and Dejing Dou. 2022c. Generative time series forecasting with dif- Fusion, denoise, and disentanglement. Advances in Neural Information Processing Systems, 35:23009- 23022.
Empirical Methods in Natural Language Processing, pages 9902-4908, Brussels, Belgium. Association for Computational Linguistics.
Mieradilijiang Maimaiti, Yang Liu, Yuanhang Zheng, Gang Chen, Kaiyu Huang. Zhang, Huanbo Luan, and Maosong Sun. 2021. Segment, mask, and pre- diet: Augmenting Chinese word segmentation with self-supervision. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing. pages
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. 2023, Text generation with diffusion language mod- els: A pre-training approach with continuous para- graph denoise. In International Conference Ma- chine Learning, pages 21051-21064. PMLR.
Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on mani- folds. International Conference on Learning Rep- resentations.
Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. Advances in Neural Infor- mation Processing Systems, 32,
Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell, and Mrinmaya Sachan. 2022. Autore- gressive structured prediction with language models. In Findings of the Association for Computational Linguistics: EMNLP 2022. pages 993-1005, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Dat Quoe Nguyen and Karin Verspoor. 2018. An im- proved neural network model for joint pos tagging and dependency parsing. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 81-91.
Wei Liu, Xiyan Fu, Yue Zhang, and Wenming Xiao. 2021. Lexicon enhanced Chinese sequence labeling using BERT adapter. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers). pages 5847-5858, Online, Association for Computational L Linguistics.
Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8162-8171. PMLR.
OpenAI. 2023. Gpt-3.5.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma. Alessandro Achille, RISHITA ANUBHAL, Ci- cero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation be- tween augmented natural languages. In International Conference on Learning Representations.
Tlya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Weinberger. 2022. Latent diffusion for language generation. arXiv preprint arXiv:2212.09462
William Peebles and Saining Xie. 2022. Scalable dif- fusion models with transformers. arXiv preprint arX1v:2212.09748.
Peng Lu, Ting Bai, and Philippe Langlais. 2019. SC- LSTM: Learning task-speeific representations in multi-task learning for sequence labeling. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pages 2396-2406, Min- neapolis, Minnesota. Association for Computational Linguistics.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Du- moulin, and Aaron C. Courville. 2018. Film: Visual reasoning with general conditioning layer. In AAAI.
Lihua Qian, Mingxuan Wang, Yang Liu, and Hao Zhou. 2022. Diff-glat Diffusion glancing transformer for parallel sequence to sequence learning. arXiv preprint arXiv:2212.1000 102.4000..
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han. Le Sun, and Hua Wu. 2022. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5755-5772, Dublin, Ireland. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:67.
Karthik Raman, Iftekhar Naim. Jiecao Chen, Kazuma Hashimoto, Kiran Yalasangi, and Krishna Srinivasan. 2022. Transforming sequence tagging into Seq2Seq task. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing,
Ji Ma, Kuzman Ganchev, and David Weiss. 2018. State-of-the-art Chinese word segmentation with Bi- LSTMs. In Proceedings of the 2018 Conference on

---

pages 11856-11874 Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma. Abhishek Kumar, Stefano Ermon, and Ben Poole. 202 lb. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations. ICLR 2021, Viriual Event, Austria, May 3-7, 2021. OpenRe- view.net.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πya Sutskever. 2021. Zero-shot text-to-image gen- eration. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831, PMLR.
Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl. Nikolay Savinov, Sander Dieleman, Laurent Sifre, et al. 2022. Self-conditioned embedding diffusion for text generation. arXiv preprint arXiv:221 1. 1.04236.
Machel Reid, Vincent Josua Hellendoorn, and Graham Neubig. 2023. DiffusER:: Diffusion via edil-based reconstruction. In The Eleventh International Con- ference on Learning Representations.
Zecheng Tang. Pinzheng Wang, Keyan Zhou, Juntao Li, Ziqiang Cao, and Min Zhang. 2023. Can diffusion model achieve better performance in text generation bridging the gap between training and inference In Findings of the Association for Computational Linguistics: ACL 2023, pages 11359-11388 1386, Toronto, Canada. Association for Computational Linguistics.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. 2022. High- resolution image symthesis with latent diffusion mod- els. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695.
Yuanhe Tian, Yan Song, Fei Xia, Tong Zhang, and Yong- gang Wang. 2020. Improving Chinese word segmen- tation with wordhood memory networks. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8274-8285, On- line. Association for Computational Linguisties.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Diffusion- NER: Boundary diffusion for named entity recogni- tion. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 3875-3890, Toronto, Canada, Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: iiggugggindeeeedentt named entity recognition. In Proceedings of the Seventh Conference on Natural Longuage Learning at HILT-NAACI. 2003, pages 142- 147,
Yongliang Shen, Xiaobin Wang, Zeqi Tan, Guangwei Xu, Pengjun Xie, Fei Huang, Weiming Lu, and Yuet- ing Zhuang. 2022. Parallel instance query network for named entity recognition. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 947-961, Dublin, Ireland. Association for Computa- tional Linguistics.
Ketan Kumar Todi. Pruthwik Mishra, and Dipti Misra Sharma. 2018. Building a kannada pos tagger using machine learning and neural network models. arXiv preprint arXiv: 03175.
Xuewen Shi, Heyan Huang. Ping Jian, Yuhang Guo, Xi- aochi Wei, and Yi-Kun Tang. 2017. Neural chinese word segmentation as sequence to sequence transla- tion. In Communications in Computer and Informa- tion Science, pages 91-103. Springer Singapore.
Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. 2023. Diffusion probabílistic modeling of protein backbones in 3d for the motif-scaffolding problem. In The Eleventh International Conference on Learning Representations.
Jascha Sohl-Dickstein, Fric A. Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2015. Deep un- supervised learning using nonequilibrium thermody- namics. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Work- shop and Conference Proceedings, pages 2256-2265. JMLR.org.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc
Jiaming Song. Chenlin Meng, and Stefano Ermon. 2021a. l. Denoising diffusion implicit models. In Inter- national Conference on Learning Representations.
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Hong Lin. 2023a. Ai-generated content (aige): A survey, arXiv preprint 2rXiv:2304.06632.
Yang Song and Stefano Ermon. 2019. Generative mod- cling by estimating gradients of the data distribution. In Advances in Neural Information Processing Sys- tems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 11895 1 11907.
Shuang Wu, Xiaoning Song. Zhenhua Feng, and Xiao- Jun Wu. 2022. Nilat: Non-ffat-lattice transformer for chinese named entity recognition. arXiv preprint mrXiv:2005.08833
Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tar Zheng, Juntao Li, Zhongyu

---

Wei, Jian Guo, et al. 2023b. Ar-diffusion: Auto- regressive diffusion model for text generation. arXiv preprint 20..2.200000111 09575.
Chenshuang Zhang, Chaoning Zhang, Sheng Zheng, Mengchun Zhang, Maryam Qamar, Sung-Ho Ho Bae, and In So Kweon, 2023a. A survey on audio diffusion models: Text to speech synthesis and enhancement in generative ai. arXiv preprint 2rXiv:2303.1..3 13336. 2.
Mengmeng Xu, Mattia Soldan, Jialin Gao, Shuming Liu, Juan-Manuel Férez-Rúa, and Bernard Ghanem. 2023. Boundary-denoising for video activity localization. arXiv preprint arXiv:2304.0293:.
Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip Yu. 2019. Joint slot filling and intent detection via capsule neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computa- rional Linguistics, pages 5259-5267, Florence, Italy, Association for Computational Linguistics.
NAIWEN XUE, FEI XIA. FU-DONG CHIOU, and MARTA PALMER. 2005. The penn chinese tree-
bank: Phrase structure annotation of large corpus.
Natural Language Engineering. 11(2):207-233
Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019. Tener: adapting transformer encoder for named entity recognition. arXiv preprint
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023b. DiffuSum: Generation enhanced extractive summa- rization with diffusion. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pages 50099-13100 Toronto, Canada, Association for Computational Linguistics.
arXiv:1911.04477
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages (508-582,, Online. Association for Computational I Linguistics.
Yue Zhang and Jie Yang. 2018. Chinese NER using lattice LSTM. In Proceedings of the 56th Anmual Meeting of the Association for Computational Lin- guisties (Volume I: Long Pupers), pages 1554-1564, Melbourne, Australia. Association for Computational Linguistics.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. 2023. reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05733
Jie Yang, Yue Zhang, and Fei Dong. 2017. Neural word segmentation with rich pretraining. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 839- 849, Vancouver, Canada. Association for Computational Linguistics.
Houquan Zhou, Yang Li, Zhenghua Li, and Min Zhang, 2022a. Bridging pre-trained language models and hand-crafted features for unsupervised POS tagging. In Findings of the Association for Computational Linguistics: ACL 2022. pages 3276-3290, Dublin, Ireland. Association for Computational Linguisties.
Jie Yang, Yue Zhang, and Shuailong Liang. 2019. Sub- word encoding in lattice LSTM for Chinese word seg- mentation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages Minneapolis, Minnesota. Association for Computational Linguistics.
Kun Zhou, Yifan Li, Wayne Xin Zhao, and Ji- Ji-Rong Wen. 2023. Diffusion nat: seff-prompting discrete diffu- sion for non-autoregressive text generation. arXiv preprint arXiv:2305.0400 040044
Renjie Zhou, Zhongyi Xie, Jian Wan, Jilin Zhang, Yong Liao, and Qiang Liu. 2022b. Attention and edge- label guided graph convolutional networks for named entity recognition. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6499-65 10, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. 2023a. Dinoiser: Diffused con- ditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.10022
Junjie Ye, Xuanting Chen, Nuo Xu, Can 7u, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023b. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv prepriat arxiv:2303. 10420.
Enwei Zhu and Jinpeng Li. 2022. Boundary smooth- ing for named entity recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 7096-7108, Dublin, Ireland, Association for Computational Linguistics.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2022. Seqdiffuseq: Text dif- fusion with encoder-decoder transformers. arXiv preprint arXiv:2212. 10322.
Feifei Zhai, Saloni Potdar, Bing Xiang, and Bowen Zhou. 2017. Neural models for sequence chunking. In Proceedings of the AAAI conference on artificial intelligence, volume 31.

---

A int2bit and bit2int
Algorithm convert tags into bits
Scale
def (xg num_bits):
FFN
mask torch.arange arange 1, -1,
-,, dtype=torch.inte int32)
Y2. B2
(dsqueeze(dimm=1)
Scale Shift
bits ((x mask) 0). float()
bits bits *
LoyerNorm
return bits
Scale
Algorithm 4 convert bits back into tags
MHSA
def bit2taggx,, num_bits):
(x .in(()
YI, B1
mask torch.arange(num.bii 1, -1,
Scale Shift
-1, dtype=torch. int32)
tags (x mask) sum(dim=-1)
LoyerNorm
Linear
return tags
bits
condition embedding
B Transformer Block of BitDiT Decoder
Figure 5: Transformer black details of the BitDiT De- coder. MHSA represents Multi-Hes Self Seff-Attetion and FFN represents Feedforward Network. 3 and are the learned affine parameters for adaptive layer normaliza- tion, the scale factor before each skip connection.
We show the architecture details of the Trans- former block used in BitDiT decoder in Figure 5. The noisy tag bits x; goes through two sub- blocks. The first block consists of Adaptive Layer Normalization (ALN) layer, a Multi-Head Self-Attention (MHSA) layer and scaled skip- connection. The second basically the same but the MSHA layer is substituted by Feedforad Net- work (FFN) layer. The scale factor before the skip connection is learned based on the conditional em- bedding C₁. The calculation procudure could be summarized follows:
C Statistics of datasets
Task
Dataset
Train
Dev
Test
MSRA
41728
4636
4365
NER
Resume
3821
463
477
Conl103
14041
3250
3453
MSRA
78231
8693
3985
X; = Q1 MHSA(ALN, (x:), C.) + x, (19)
CWS
PKU
17150
1906
1944
CTB6
23458
2079
2796
X; = 02 FFN(ALN, (x.), C+) Xt (20)
01,2 = Linear),2(c:)
(21)
POS
CTB5
18078
350
348
Table 7: The statistics of the datasets. For CWS task, whose dev set is not provided, we randomly select 10% of the original training instances
where a₁ and a2 are scale factors. The ALN learns the scale and shift B3 parameters based on the c₁:
the dev set.
ALN(Xi, c.) 7 LX( +
(22)
D Hyper-parameters settings
[x())
LN(x))
(23) For conditional encoder, we choose chinese-bert-wm--ett for chinese datasets and bert-large-cased for English datasctes. We take AdamW(Loshchilor and Hutter, 2019)
Var x +
7 Linears
(24)
8 Linears(c:)
(25)
hhtpps://huggingaacee
where e is for avoiding division-by-zene error and numerical stability.
chinese-bert-wwr-es.
httpr//aaac.eeeeetraare----e hnttps://hugiingface

---

optimizer. We the peak learning rate as le-5, with 1000 wa steps and liner decay scheduler. The timestep for training is set to 1000 and the ddim sampling step set to 10 for efficiency. The SNR scale is set to 0.1. We use the B-M-E-S schema instead of the B-I-O schema.
gpt-3.5-turbo (OpenAl, 2023) is one of the most powerful LLMs. We its one-shot performance to generate the sequence labeling targets, and the corresponding prompts are in Table 6.
E.2 CWS
E Baselines
Ma et al. (2018) uses a simple unigram and bigram-based Bi-LSTM to extract the entities.
E.1 NER
BiLSTM- (Huang et al., 2015b) uses BiLSTM to encode the sentence and a CRF layer to decode the NER tags.
Yang et al. (2019) improves the Lattice LSTM by integrating the subword embedding.
WMSEGGTian et al., 2020) introduces the memory mechanism to incorporate the word- hood information to enhance the segmentation model.
TENER (Yan al., 2019) designs specific attention mechanism and improves the perfor- mance of the Transformer for NER.
FLAT (Li al., 2020) introduces the character-word lattice into the Transformer architecture to enhance the NER performance.
Cui et al. (2021) enhances the BERT encoder by masking the whole word.
NFLAT (Wu et al., 2022) proposes Inter- Former to apply lexical enhancement and re- moves the word-character and word-word at- tention interaction for memory efficiency.
E.3 POS tagging
For POS tagging, most works focus on adding more features to enhance the word representation ability of the text encoder.
Athiwaratkun al. (2020) uses a generative model for joint sequence labeling and se- quence classification task. It can tackle many sequence labeling at the same time and per- forms well in different scenarios.
ZEN (Diao et al., 2020) incorporates the n- gram information to enhance the BERT.
Glyce (Meng et al., 2019) incorporates the visual information enhance the BERT.
Paolini al. (2021) proposes an augmented natural languages and then cast many se- quence labeling tasks as a translation be- tween original language sequence and the aug- mented ones.
Cui et al. (2021) enhances the BERT encoder by masking the whole word.
F LLM experiments
We show the prompt we used in the LLM exper- iments in Table 6. For one-shot experiments, we only add one example following the task definition. For few-shot experiments tested on Resume, we add more demonstration examples following the task definition. For fair comparison in few-shot experiments, examples used in the 15-shot incorpo- rate the ones used in 10-shot, and so on.
UIE (Lu et al., 2022) utilze the proposed struc- ture extraction language to unify various infor- mation extraction tasks and uses unsupervised corpora to pretrain, thus resulting in better information extraction ability.
DiffusionNER (Shen et al., 2023) models the entity as a labeled span (left boundary, right boundary, entity label) and cast the NER as boundary-denoising process. It generates the entities from the noisy spans using diffusion model. DiffusionNER sets the boundaries as the generation objective, which can only be used for NER. Conversely, DiffusionSL. can be used for all sequence labeling tasks, which enjoys more versatility.
G Detailed Related Work of Diffusion Model with Language
We provide complete and detailed review of pre- vious work about applying the diffusion model language for the potential audiences to get familiar with this promising field. Most existing literature focuses on NLG instead of NLU.

---

Multinomial-Dfffusinn (Hoogeboom al.. 2021) defines a multinomial diffusion process with categorical data to generate text or a seg- mentation map.
among words, distraction resulted by adding standard noise.
Lovelace et al. (2022) demonstrates that the la- tent space of pretrained language model could be used to learn a diffusion model in which the latent representations could be decoded into natural language.
D3PM (Austin al., 2021) also defines the diffusion model in discrete space and the newly proposed discrete corruption process improves the performance.
SeqDiffSeq (Yuan et al., 2022) uses self- condition technique and a newly proposed adaptive noise schedule for sequence-to sequence diffusion language model based on Transformer.
DIffusionLM (Li al., 2022b) firstly pro- poses to embed the text into continuous space to circumvent the discreteness problem and round the final denoised vectors back into words.
Diff-Glat (Qian al., 2022) proposes modal- ity diffusion process and residual glancing sampling to boost the performance of Non- Autoregressive parallel text generation.
DiffluEEQ (Gong et al., 2023) firstly employs the diffusion model for seq2sesq text genera- tion setting. And novelly proposes to con- catenate and embed the source and target text and only add noise to the target text embed- ding while keeping the source text embedding as clean.
GENIE (Lin al., 2023) pretrains novel Transformer-based encoder-decoder diffussin language model on a large-scale corpus with a novel pretraining technique named continuous paragraph denoise.
SSD-LM (Han et al., 2022) iteratively gen- erates text blocks in semi-autoregreesivee manner and conducts the diffusion process on the natural vocabulary space, which bal- ances the advantage of Autoregressive and Non-Autoregressive model and allows conve- nient guidance incorporation.
DiffusER (Reid et al., 2023) applies an edit- based generative model based diffusion de- noising process to revise the already existing text, making gradual text refinement probable.
RDMs (Zheng et al., 2023) proposes a novel family discrete diffusion model with route- and-denoise decoding process.
SED (Strudel et al., 2022) casts the discrete language symbols into continuous embed- dings and incorporates the self-condition trick into the backward denoising process.
Dinoiser (Ye al., 2023a) manipulates the noise in the training and inference stage avoid the pitfall of discreteness the embed- ding space and boost the impact of source sentences for conditional sequence learning.
CDCD (Dieleman et al., 2022) uses the score- matching framework solve several language modeling tasks in continuous time and input space.
-ifffu-Di LM (Chen et al., 2023a) uses linguistic masking strategy to perturb the clean sentence which enables an easy-first- generation backward process.
DiflusionBERT (He et al., 2023) uses the dif- fusion model to improve the masked language model, which combines the advantage of two denoising models. New proposed noise sched- ule and time embedding injection methods are applied it.
RenderDiffusion (Li et al., 2023) transforms the discrete language symbol generation prob- lem into the glyph image generation problem, thus casting the less-studied text-to-text dif- fusion model as a well-studied text-to-image model.
(Gao al., 2022) proposes chor loss function, layer norm module over the embeddings, and noise factor that con- trols the added noise to tackle the problems incorporating the collapse the denoising ob- jective, the imbalanced norm of embedding
Difformer
DiffusionSum (Zhang et al., 2023b) tackles the extractive summary problems by directly

---

generating the desired summary sentence em- bedding to match the corresponding natural sentences the original document.
Tang et al. (2023) proposes two methods named Distance Penalty and Adaptive De- cay Sampling to bridge the gap between the training and (nference exposure bias problem in the traditional NLG setting) of the diffusion language model.
Diffusion-NAT (Zhou et al., 2023) unifies the BART and discrete diffusion model and pro- poses self- pelf-prrop technique for text-to- text generation.
AR-Diffusion (Wu et al., 2023b) combines the Autoregressive language model and the Von-Autoregressive diffusion language model to boost the text generation performance and speed, due the left-to-right sequential nature of language.
DDLM (Balagansky and Gavrilov, 2023) re- produces the CDCD-based (Dieleman al., 2022) LM and publicly releases the corre- sponding training code and checkpoint, then firstly tests the performance of downstream tasks, making it convenient for other re- searchers to explore this field.

---

dataset
prompt
MSRA
#. X XHA.4
# 7 X JJ #4: H. HM F
(X4:
# ....
* [tex7)
#
Resume
A PMIMember) B
# :
*1 RM X& 0 #
A TD I PMIMember, T PMP, 1
: ....
# {text}
Conl103
Please list all Organization, Person, Location, and Miscellaneous Entity in the given text, output using the format as "Entity: Organization: None Person: None Location: Wordl, Word2 Miscellancous: Word3"
Text: The European Commission said on Thursday it disagreed with German advice to consumers shun British lamb until scientists determine whether mad cow disease can be transmitted to sheep.
Entity: Organization: European Commission Person: None Location: None
Miscellaneous: German, British
Text:
Entity: } ....
Text: [text]
Entity:
Table 6: The prompt used in LLM experiments. Following et al. (2023b), we use task definition followed by demonstration examples the prompt. * # J.. and TTxx:: Fintity:] ... adding demonstrations. We refer to text as the input sentence.


Reviews:
[{"review_id": "BXndWwT8Ts", "reviewer": "Reviewer_zKAW", "summary": "", "questions": "- One trivial question: What's the advantage of generative modeling over discriminative modeling on the sequence labeling problem?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to evaluate our paper and giving your careful and insightful comments and suggestions. We have carefully considered the points raised in the reviews and would like to address the concerns and comments raised.\n\n* **Response to `Reasons To Reject`:** \n  * We regret the oversight in not citing [1]. In line 110~113, we discuss and cite two existing representative diffusion models that convert the discrete data into continuous data for tackling the discreteness problem. One is learning a embedding for each discrete class and the other is the bit-representation ([1] is the first to propose this line of work and [2] is another application work) method. Based on the bit-representation method, **our novelty and contribution are modeling the sequence labeling problem as a conditional discrete tag generation problem and using a non-autoregressive step-wise model with self-correction ability for discrete data generation.** What's more,  introducing the bit-representation for discrete label to connect the sequence label/tag data and Gaussian diffusion model (in continuous data space), and using a specific devised bit diffusion Transformer (as you mentioned in review and we introduce it in section 3.3) instead of the vanilla Transformer or UNet in [1] as the generation backbone to utilize and harness the strong generation ability of diffusion model are model-component-level contribution. \n  * It is noteworthy that we cite [2] in line 112, which is a work using [1] directly to solve the panoptic segmentation problem and even enjoying the same first-author with [1]. So please believe that we are unintentional to ommit the citation. We originally cite both [1] and [2] in the earliest draft and the frequent draft-editing by different authors might lead to the missing problem. And we will add the citation back in the revised version. \n  * We provide detailed and strong ablation studies to support the advantage of  component design (BTConverter and BitDiT) over others, e.g., lightweight, flexible for BTConverter and faster convergence and better performance for BitDiT. We will provide more motivation discussion in the introduction section to clarify the necessity that we design these components in the revised version. From our perspective, sequence labeling is almost one of the most popular paradigm to tackle various NLP tasks, so introducing a completely different modeling style (non-autoregressive generation with self-correction) for sequence labeling brings more research opportunities for the future work, and thus it does not lack novelty. \n  * Based on the above points, we sincerely hope you could consider raising the scores. We expect more discussion about the details in DiffusionSL.\n\n* **Response to Question:** In general, we can train generative model to output in any specific formats by teacher forcing. Generative methods enjoy the properties of flexibility and generality (as mentioned in [3])  to tackle various tasks against discriminative ones. What's more, many generative baselines demonstrate strong performance. We believe the potential of generative models for sequence labeling has not been fully tapped. So we introduce another line of generative methods into sequence labeling. The non-autoregressive manner enables the diffusion model to self-correct the generation results in the denoising path, which is a advantage the discriminative methods do not have. Moreover, the DiffusionSL could model the interactions between the tags. Conversely, the discriminative methods just independently predict the tag for each token and do not explicitly model the relations between the tags.\n\n[1] Chen T, Zhang R, Hinton G. Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning\n\n[2] Chen, T., Li, L., Saxena, S., Hinton, G., & Fleet, D. J. (2022). A generalist framework for panoptic segmentation of images and videos.\n\n[3] Raman, Karthik, et al. Transforming sequence tagging into a seq2seq task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow. The problem is well-formulated.\n- The idea is interesting and the design is reasonable to apply diffusion models to the sequence labeling problem.\n- Experiments are sufficient, and ablation studies are conducted to check the effectiveness of the proposed method.", "reasons_to_reject": "- **The proposed approach (i.e., Bit Diffusion Transformer) is similar to [1]**, but the authors did not include any discussion or citation. Since [1] share a very similar design to the proposed one. Therefore, this would result in the novelty problem directly. Furthermore, **the authors did not cite it**.\n\n[1] Chen T, Zhang R, Hinton G. Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning[C]//The Eleventh International Conference on Learning Representations. 2022.", "questions_to_authors": "- One trivial question: What's the advantage of generative modeling over discriminative modeling on the sequence labeling problem?", "paper_topic_and_main_contributions": "Paper Topic: Sequence Labeling\n\nMain Contribution: The paper formulates the Sequence Labeling task as a conditional generation problem, where diffusion modeling is adopted. To overcome the discreteness problem, the paper introduces bit-format inputs with Bit-Tag Converter and a Bit Diffusion Transformer. Experimental results demonstrate the validity of the proposed approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Chen T, Zhang R, Hinton G. Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning[C]//The Eleventh International Conference on Learning Representations. 2022."}, "extra_text": {}}, {"review_id": "bN6m0d4Vo2", "reviewer": "Reviewer_kB8h", "summary": "", "questions": "1. Is it possible to do any nested NER tasks in the proposed approach? Is it possible to see results on ACE04 and ACE05? \n2. I think it will be interesting to see the performance in any few-shot setting with diffusion approach.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to evaluate our paper and giving your careful and insightful comments and suggestions. We have carefully considered the reviews and would like to address the concerns and comments raised.\n\n* **Response to `Reasons To Reject`1:** \n  * DiffusionSL is not an incremental work over DiffusionNER. There are many difference: DiffusionSL focuses on sequence labeling instead of only NER, which is modeled as a conditional discrete tag data generation problem in our framework. We conduct experiments on POS tagging and Chinese Word Segmentation datasets to test the effectiveness on different sequence labeling tasks other than NER. DiffuisonSL could be extended to other sequence labeling tasks easily, such as slot filling. We leave this as future work. DiffusionSL utilizes the bit-representation as an explicit solution to discreteness problem, which could be beneficial and insightful for more NLP tasks (Most NLP tasks are inherently discrete tasks). DiffusionSL uses an specific Transformer instead of the vanilla Transformer, which shows faster convergence (Fig 3).\n  * Yes, DiffusionSL does not directly focuses on span level informaiton. However, we do not think it is the weakness of DiffusionSL. We think the generation format of DiffusionNER (e.g., span boundary) is unnatural, and it separates the span localization and the entity label classification to some extent. Thus the diffusion model (novel and effective part of DiffusionNER) can not directly model the entity label information (the generation target x_0 is just boundary,  and DiffusionNER uses another MLP Classifier to incorporate the span label information) well. What's more, this generation format necessitates extra heuristic post-propocessing operations. Though DiffusionNER considers the span-level information, we think this modeling style also incorporates the above shortcomings. And the experiment results demonstrate the superiority of DiffusionSL over DiffusionNER in flat NER datasets.\n* **Response to `Reasons To Reject`2:** For span level information, we do not explicitly model it in the DiffusionSL. But it could handle span-level tasks well. Sequence labeling is inherently such method to convert the span-level information and token-level information back and forth equvalently. The span-level and token-level information conversion is a bijection. \n* **Response to `Reasons To Reject`3:** DiffusionSL could not be directly extended to nested NER and discontinuous NER (DiffusionNER is also not suitable for discontinuous NER) now. The intuitive way to extend the DiffusionSL to nested NER is to extend the BIO format to BILOU hypergraph-based labeling schema. With more novelty, we could extend the 1D labeling schema (BIO format) to 2D labeling schema (like BiaffineNER[1] or W2NER[2]) for nested NER or even discontinuous NER. We think your suggestion is well, and we leave this idea as a future work to explore.\n* **Response to Question1:** As we mentioned in response to reject3, the current version of DiffusionSl could not be extended to nested NER. We leave this as future work.\n* **Response to Question2:** Most of the NER studies focuses on fully-supervised setting, including the DiffusionNER. For fair comparison with previous methods, we also adopt the fully-supervised setting. However, it is a good research direction to explore the few-shot performance of diffusion model for NLU tasks. We randomly sample 5/10/20 examples from Resume (NER dataset) for training and provide the corresponding experimental results. Thanks for your insightful comments.\n\n| Number of Samples |   P   | R     | F1    |\n| ----------------- | :---: | ----- | ----- |\n| 5                 | 29.35 | 10.96 | 15.96 |\n| 10                | 62.64 | 50.15 | 55.71 |\n| 20                | 80.61 | 71.83 | 75.97 |\n\nThanks for taking time to read these. We sincerely hope you could consider raising the scores. We are also expecting more discussions about DiffusionSL.\n\n[1]Yu, Juntao, Bernd Bohnet, and Massimo Poesio. \"Named entity recognition as dependency parsing. \"ACL2020\n\n[2]Li, Jingye, et al. \"Unified named entity recognition as word-word relation classification.\"AAAI2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method shows some performance gains over other methods in this domain. \n2. Besides diffusion modeling like previous works, the authors also tries to address the quantization of tokens.", "reasons_to_reject": "1. This work iteratively improves over DiffusionNER both using similar techniques. However, DiffusionNER also focuses on span level information which DiffusionSL does not directly address. I am not sure whether that is a strength or weakness for DiffusionNER vs. DiffusionSL. \n\n2. I am also not sure how span level information can be handled in current framework of DiffusionSL. \n\n3. Related to 2, I think nested NER tasks are missing in the evaluation. \n\n4. (After rebuttal) Seems like DiffusionSL is less generalizable than DiffusionNER being not applicable to nested NER which objectively is a more difficult task.", "questions_to_authors": "1. Is it possible to do any nested NER tasks in the proposed approach? Is it possible to see results on ACE04 and ACE05? \n2. I think it will be interesting to see the performance in any few-shot setting with diffusion approach.", "paper_topic_and_main_contributions": "This paper discusses a framework called DiffusionSL that uses Diffusion models for sequence labeling tasks. Moreover to incorporate the discrete nature of NL tokens, they introdcued Bit-Tag converter (for quantization) and Bit Diffusion Transformer (for noise removal).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wVKO3sZzUQ", "reviewer": "Reviewer_ckkr", "summary": "", "questions": "A) How does the initial sampling of random tags impact performance? Have you tried to do some experiments to assess that?\n\nB) Does the number of possible tags affect the effectiveness of the model? In all the selected tasks the tag vocabulary is small, I am curious if you have tried to look into this factor.\n\nC) Are the results reported the average of more runs?\n\nD) What are the computational performance of the method? Something is mentioned in the limitations section, a brief discussion or table in the main text would be interesting.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to evaluate our paper and giving your careful and insightful comments and suggestions. Moreover, thanks for your affirmation to our paper. We have carefully considered the reviews and would like to address the concerns and comments raised.\n\n* **Response to `Reasons To Reject`:** Thanks for bringing up this problem. We provide more details in the appendix C, D, and E. We will add more experimental setup details in the revised version to make our paper clearer.\n\n* **Response to Qeustion A:** We use torch.randn_like() as the sampling method. And we conduct experiments using different random seeds to report the averge results. We find the performance is robust agsinst the randomness of sampling. We will add more detailed descriptions in the revised version.\n* **Response to Question B:** Yes, we have considered this problem. And we conduct experiments to test the effectiveness against the nubmer of tags. In Table 4, we try to test two methods converting the discrete tags into continuous representation. The first one is using bit representation (BTConverter in the paper), and the second one is learning a embedding for each tag. NER-Resume has 13 tags (dimension is 4, 2^4=16) and CTB5 has 103 tags (dimension is 7, 2^7=128). For bit-representation, it shows strong performance for small and big vocabulary. For embedding-representation of same dimension with bit-representation, it shows that the performance of small vocabulary (13 tags, dimension is 4) does not converge. Increasing the dimension of embedding-representation to 768 also dose not increse the performance dractically. So we acclaim the superiority of our bit-representation method.\n\n* **Response to Question C:** Yes, we report the averge runs. The performance is robust.\n* **Response to Question D:** Thanks for bringing up this idea. We would like to add some discussion in the revised version. For prior experiments, the training and inference speed of DiffusionSL is faster than DiffusionNER at least. \n* **Response to Typos Grammar Style And Presentation Improvements:** Thanks for evaluating our paper carefully. We would add more experiments setup details, add simple BERT baseline, try to describe the pseudocode in tradiitonal style instead of pytorch-style, and fix the typos in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a new framework that tackles sequence labeling tasks using a transformer with the diffusion process instead of the standard autoregressive approach.\n\n- Strong experimental setup with good results and comparisons\n\n- Paper is clear and well written", "reasons_to_reject": "- Experimental setup is described a bit too briefly and some small details might be missing, however it can be easy to fix.", "questions_to_authors": "A) How does the initial sampling of random tags impact performance? Have you tried to do some experiments to assess that?\n\nB) Does the number of possible tags affect the effectiveness of the model? In all the selected tasks the tag vocabulary is small, I am curious if you have tried to look into this factor.\n\nC) Are the results reported the average of more runs?\n\nD) What are the computational performance of the method? Something is mentioned in the limitations section, a brief discussion or table in the main text would be interesting.", "paper_topic_and_main_contributions": "The paper tackles sequence labeling tasks by using a diffusion approach. The authors design a diffusion transformer that models the diffusion process and a Bit-Tag converter to handles the discrete tags in continuous space. The approach is evaluated on three tasks with good results.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Experimental setup description should be expanded a bit, describing for example what are the performance metrics which are not mentioned \n\n- I would suggest the authors consider including the results of a simple vanilla baseline like BERT.\n\n- It would be better if the algorithms shown would be in pseudocode\n\n- There are some typos here and there (e.g DiffusionSL is misspelled at the end of the conclusions)"}}]

Author rebuttal:

Meta-review: The paper frames the Sequence Labeling task as a conditional generation challenge, utilizing diffusion modeling. In addressing the issue of discreteness, the paper presents a solution involving bit-format inputs using the Bit-Tag Converter and a Bit Diffusion Transformer. It omits a pertinent citation, namely "Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning." A more comprehensive discourse and experimental evaluation comparing the proposed approach with prior systems, including the absent reference and DiffusionNER, is warranted
Decision: Accept-Findings