Debiased and Denoised Entity Recognition from Distant Supervision
Authors: Haobo Wang
Yiwen Dong
Ruixuan Xiao
Fei Huang
Gang Chen
Junbo Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: Distant Supervision; Named Entity-Recognition; Biased Learning
TL;DR: Combating biases and noise in distantly-supervised NER task

Abstract: While distant supervision has been extensively explored and exploited in NLP tasks like named entity recognition, a major obstacle stems from the inevitable noisy distant labels tagged unsupervisedly. A few past works approach this problem by adopting a self-training framework with a sample-selection mechanism. In this work, we innovatively identify two types of biases that were omitted by prior work, and these biases lead to inferior performance of the distant-supervised NER setup. First, we characterize the noise concealed in the distant labels as highly structural rather than fully randomized. Second, the self-training framework would ubiquitously introduce an inherent bias that causes erroneous behavior in both sample selection and eventually prediction. To cope with these problems, we propose a novel self-training framework, dubbed DesERT. This framework augments the conventional NER predicative pathway to a dual form that effectively adapts the sample-selection process to conform to its innate distributional-bias structure. The other crucial component of DesERT composes a debiased module aiming to enhance the token representations, hence the quality of the pseudo-labels. Extensive experiments are conducted to validate the DesERT. The results show that our framework establishes a new state-of-art performance, it achieves a +2.22% average F1 score improvement on five standardized benchmarking datasets. Lastly, DesERT demonstrates its effectiveness under a new DSNER benchmark where additional distant supervision comes from the ChatGPT model.

Debiased and Denoised Entity Recognition from Distant Supervision
Haobo Wang'*, Yiwen Dong' Ruixuan Xiao', Fei Huang', Gang Chen', Junbo Zhao!
ZZejiang University, Hangzhou, China
2 Allibbba Group, Hangzhou, China
(wanghaobo, dyw424, xiaoruixuan, eg, eg. j.zhao) @ @zju.edu.cn, feirhuang @mmil..com
Abstract
While distant supervision has been extensively explored and exploited in NLP tasks like named entity recognition, major obstacle stems from the inevitable noisy distant labels tagged unsupervisedly. few past works approach this problem by adopting a self-training framework with sample-selection mechanism. In this work, we innovatively identify two types of biases that were omitted by prior work, and these biases lead to inferior performance of the distant-supervised NER setup. First, we characterize the noise concealed in the distant labels as highly structural rather than fully randomized. Second, the self-training framework would ubiquitously introduce an inherent bias that causes erroneous behavior both sample selection and eventually prediction. To cope with these problems, we propose a novel self-training framework, dubbed DesERT. This framework augments the conventional NER predicative pathway to dual form that effectively adapts the sample-selection process to conform its innate distributional -iia structure. The other crucial component of DesERT composes debiased module aiming to enhance the token representations, hence the quality the pseudo-labels Extensive experiments
conducted to validate the DesERT. The results show that our framework establishes a new state-of-art performance, achieves a +2.22% average F1 score improvement five standardized benchmarking datasets. Lastly, DesERT demonstrates its effectiveness under new DSNER benchmark where additional distant supervision comes from the ChatGPT model.
Introduction
As an iconic and pivotal task, named entity recognition (NER) [ has gained wide attention from the natural language processing community. While there have been several successes towards solving this task 22 good quality and large quantity. When it comes to specialized industrial deployment, this leads us to expensive data annotation and curation costs which may hinder the development of NER.
5) most of these solutions demand decent datasets for training that enjoys both
Distant supervision been a number of attempts to combine the additional labels obtained from distant supervision with the original unlabeled data, and then establish selection- self-training framework [9] to handle noise among distant labels. The core of these methods sample reliable clean tokens and cope with potentially noisy tokens involved by the imperfect pseudo-label prediction.
S is a promising direction to alleviate this problem. Indeed, there have
On one hand, characterize that the label distribution for distant supervision is rather skewed. Notably, as shown in Figure 1 (a), we found that true non-entity tokens are far more than the remaining
"qqual contribution.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

100000
210
100000
1e0002
140000
Double-hae head selected
True labels
True labees
35 9962
15000
120000
332
702
100000
30000
50000
30000
Number
00000
10494
24b49
46000
2220
3000
20000
20G
Distant labels
Distant labels
(a) Confusion matrix
(b) True entity confusion matrix
(c) Token class distribution
Figure I: (a) Confusion matrix of true labels and distant labels on the whole CoNLLO3 dataset. Class index O denotes non-entity and denotes all entity types, more than 169. Ik non-entity tokens correctly labeled and dominate true entity tokens. (b) The confusion matrix displays noise among true entity-type labels. (c) The real token class distribution on the CoNLL03 dataset and tokens selected at the fifth epoch by a basic self-training framework with a single-head/dooubl--een pathway. It can be shown that double-head selects more tokens than single-head on the minority entity classes such MISC and LOO..
tokens, quantity-wise Meanwhile, there exists severe class imbalance in true entity tokens with fine-grained noise in Figure (b). Ignoring these phenomena by prior sampling-bas based methods is potentially performance-limited. The reason is that the dominant clean non-entity tokens can be easily distinguished and excessively picked up, while clean entity tokens are largely overlooked, especially on label-deficient entities like MISC and LOC: see Figure (c). Whereas, the clean entity tokens of real interest since NER attempts to distinguish the fine- -ine-grained token types. Unfortunately, there has not been any prior work that discusses or addresses this property specifically.
On the other hand, in spite of that self-training scheme applied with distant supervision is proven applicable [10 9 11, it could still be unreliable mostly due to the deviation between the pseudo- labels and unobservable true labels. In other words, the self-training scheme is inherently prone to assign erroneous pseudo-labels, which can be aggravated due to label noise and finally cause error accumulation and conversely pollutes the selection. We postulate that both biases ought to be properly resolved in order to fully unlock the utility boost of the distant supervision technique for NER.
To this end, we propose DesERT, a holistic framework that resolves the aforementioned problems and achieves statc-of-the-ard DSNER performance. The main idea behind the DesERT is to decouple the initial task/data into entity vs. non-entity and fine- -ee-graided entity type sub-parts, then respectively to split again by selecting and splitting out clean- and noisy-token sets which are processed by different learning paradigms, forming a task/data taxonomy. We further introduce debiased self-training on noisy sets to reduce erroneous pseudo-labels. Empirical findings successfully justify the validity of our approach, where on multiple standardized NER benchmarks we establish a new set of of the-art results. For example, DesERT achieves state-of-art performance, especially an improvement of 3.26% % on the CoNLLO3 dataset and 4.26% % on the Webpage dataset. Finally, we establish a new benchmark for DSNER where the supervision signals are generated via the large language model ChatGPT and DesERT still holds its superiority.
2 Related Works
Distantly-Supervive NER. There are two typical forms of noise in the labeling of distantly- supervised NER (DSNER) tasks: false negative tokens wrongly tagged as non-entities and inaccurate entity types. A series of works mainly focus on processing false negative tokens. A popular line of works adopts PU Learning [12 I3 techniques to resolve DSNER [4] which sees entity tokens as Positive (P) data and non-entity tokens as Unlabeled (U) data, yet ignore the noise in fine-grained entities. Other methods consider both types of noise, usually inspired by traditional NLL approaches. One of them based on clean token selection. Lots of works adopt the model's
prediction consistency or confidence as the selection criterion [6 I [0 18
9 II 19 20]. Besides,
Huang et al. (2T] propose a novel Logit Maximum Difference (LMD) score. HGL 22 introduces hypergeometric distribution to estimate noisy sample size. Additionally, some studies 23 24 also
employ reinforcement learning agents to filter noisy samples. A recent work (25) studies a specific
DSNER setup that collects large set of weakly labeled data and small set of clean training data. The

---

most related work to ours is [26] which also investigates the bias issues in DSNER, but it concentrates on structural causal biases caused by the dictionaries. In contrast, our work focuses on unexpected selection and confirmation biases in popular selection and self-training DSNER frameworks.
Self-training. Self-training 27 28 29 is a popular semi-supervised learning technique that trains
on labeled data but assigns pscudo-labels to unlabeled data for further training [30]. This is also adopted by existing DSNER approaches. For example, BOND [TO] uses pseudo-labels from the teacher model to train the student model. SCDL [9] jointly trains two teacher-student networks to mutually provide pseudo-labels. RoSTER []] ensembles different models' predictions as pseudo- labels. However, some research has pointed out that self- training exists an inherent confirmation bias to assign erroneous pseudo-labels [31 32]. Due to the existence label noise, the DSNER task can face an amplified self-training bias, which, however, has never been touched on in prior studies.
3 Notation and Preliminary
Named Entity Recognition. Assume receive NER training dataset D = {(Em Fm) }m) Jim
with . tuples, where each tuple consists of a sentence x = [zi T2, ..., Enl] with n tokens and a label sequence y [/1, 1/2 .... Unl. Here each entity label ys indicates token 2; is non-entity or belongs to a specific entity type. Yi E T = {0, 1, 1,... K}. where 1 ~ K denote entity types and denotes non-entity. Our goal is to train a NER model f parameterized by 0 to be able to receive testing sentence and predict the entity labels for its tokens. For fully supervised NER, this goal can be achieved by minimizing the empirical standard cross-entropy loss as follows,
Lce Lee (c = log (2 (z:
where Ji (a; ()) the model's predicted probability of i-th token x; belonging to class y; sentence In practice, the NER model is typically composed of pre-trained language model encoder & for extracting token representation and a classification head h, i.c., f(; 0) = a ((x).
Distantly-Supervised NER. This paper considers the distantly-superviee NER problem, which effectively reduces labeling costs. Accordingly, the true label sequences (ym J are replaced by their distantly-annotated counterparts (ym I.M"' m==1- Despite the promise, the distant labels inevitably
contain large portions of label noise due the limited coverage of entity mentions in the knowledge follow previous works 10 9 and
base and the labeling ambiguity. To cope with this problem, introduce the selection and self-training framework to mitigate the label noise, At each epoch, we first perform token selection based on the specific criterion that separates the tokens in each sentence æ into a set of potential clean tokens x' and noisy tokens x", and thus, a robust NER model can be trained on those clean tokens. To further exploit the remaining noisy tokens, regard them as unlabeled tokens and perform semi-supervised training.
Lale C1 ((x:0)) + Lo f()":0)
where C,. and Lu are classification losses on the two sets, g is a pseudo-label generated via previous models. In the sequel, we will specify our selection criterion and self-training algorithms in detail.
4 Proposed Method: DesERT
In theory, distant supervision can provide an arbitrarily large dataset for NER with its scale upper- bounded by all available sources. In this task of distantly-uperrvseed NER, we denote the initial set extracted by distant supervision by D. Of course, directly using D to train a NER model by supervised learning is problematic, due to the noise, uncertainty, or mismatches hidden in D. In this section, we provide a detailed description of our framework. The resulting architecture would effectively decompose 'D into task-data taxonomy, with each component from this taxonomy manifesting an individual training paradigm. This decomposition stemmed from the two bias problems we mentioned previously. The pseudo-code of DesERT is summarized in Appendix D
3

---

Eaiiy takes
Nan-cutity loken
Fiuo-grazzed cuilyy
Teat tokes
BCE Loss
Wet loss
Ri Loss
WiCE Loss
Joint Prediction
Clear Token Selection
Tencber
Stadeut
Stadeal
hb
Rg
Ap
Nec
hp
hg
Backbone
Backbone
Backbone
D3
De
D3
(a) Train
(b) Test
Figure 2: (a): Illustration of decoupled learning paradigm and debiased self-training. (b): Two classification heads are trained independently but make joint predictions when testing. # denotes unselected noisy or invalid tokens.
4.1 Decoupled Learning for Distant Supervision
To begin with, we start by introducing the top-tier decomposition which forms the first- and high-level of the taxonomy.
The Bias Problem. Let recall the observation from Figure Extending from it, we draw the following conclusion:
in Figure (a), we plot the confusion matrix of the true labels against distant labels without distinguishing between fine-grained types. It effectively indicates that the much dominant number of non-entity tokens against the true entity tokens (about ratio 170:23);
in Figure (b), we see the noisy distribution of the fine fine-grained entity-type labels is very much biased and imbalanced. In spite of that the prior work generally omits this problem, we believe this is vital that the neglect of would cause overly high probability of non-entity tokens being picked during the selection process of DSNER. Therefore, only a few entity tokens are selected as clean data for normal NER training:
Figure (c) further offers supporting evidence. It shows that the previous DSNER frameworks, built with a single-head pathway hardly draft the clean and minority entity tokens. In particular, samples 436/4.593 (9.49%) MISC tokens and (almost) none I-LOC tokens which are of more value in solving NER.
To sum up, we believe this loss of a few entity tokens heing selected - caused by the data bias - is genuinely huge waste and may have significantly hindered the performance gain. While the prior works generally overlook this problem, in this article, we intend to study it thoroughly and give a solution to resolve it.
Decompusition of the NER Task. To overcome the aforementioned data bias, we decompose the dataset 'D into sub-parts, D 'D U 'D. where the subscripts of b and indicate binary- and clsity-clasiiiication respectively. In effect, we proactively decompose the NER task of single-level prediction into hierarchical dual that the binary prediction targets entity versus non-entity while the counterpart task is to predict the specific entity category. Notably, the skewed data distribution on D also decomposed into two parts on D, and D., which be alleviated by separate processing and will not interfere with each other. More specifically, to construct this level of taxonomy, the label set y D, is binarized. In the meantime, Ye e D. removes the non- non-entity tag (as invalid) and only reserves entity type labels.
Double-head Pathway. In what follows, to facilitate the training process upon the decomposition, we correspondingly augment a separate pathway to the original single prediction pathway, written as h = (ha, he), where the subscripts follow the same naming rule as the previous paragraph. The end of the two pathways are both fully connected layers with different entry numbers: for the h& and K for be. The two pathways shared the same encoder, é, introduce in Section 3 A more

---

detailed illustration is provided in Figure 2 (a). We leave the further textual description of the model architecture and task-data taxonomy details Appendix A
Joint Prediction. One last piece of utilizing the dual pathway in the framework the inference procedure. Since either pathway is unable to generate complete NER prediction independently, we combine them in a jointly predictive fashion. As shown in Figure (b)) we form prediction hierarchy as follows. Given any token 2; E the binary pathway hi offers probability of that token being a part of an entity denoted as p¹. On the other hand, the entity pathway generates a vector of specific entity probability pf having K entries. Then the double-head pathway generates a joint probability p, [1 - pl, - P. p.]. that 1 - p is the probability of being non-entity. By then, the model can predict each token's specific type, i.e., whether a non-entity or fine- fine-grained entity type.
4.2 Clean Token Selection
Following the previous decomposition, the next step we propose is to follow a general noisy learning setup to select the clean set from the noisy dataset. In particular, upon either task/data component in the first level taxonomy, it can be further decomposed down to be a union of the clean and noisy set. Different learning schemes are then applied to these two sets of data and enacted concurrently. In this section, we discuss how the sample selection is devised.
We take the binarized dataset/task describe the selection mechanism. In hindsight, we assume that the tokens ought to be put into the clean pool when the model's prediction is relatively certain and matches the provided counterpart label by distant supervision. Formally, given any token 2; drawn from any sentence, denote its distant label as y². During training, we take the model obtained at the current step and feed the input through it attaining a current prediction of as dubbed as of- Hereby, we can write down the primary selection criterion:
D'. = [(.,,
D.) A (max(p), 1- -.))>)))))
Noted, we use Dj, to denote the clean set, as superscript implies "labeled" since the clean tokens are primarily utilized as if they were gold. We simply adopt max(p 1 - pp) as proximal form of predictive uncertainty on the given sample where r is a hyperparameter for the threshold. In most of our experiments, we fix it 0.95 without further tuning. Further, the left-out samples where the token samples do not meet this criterion are gathered and then form Dÿ such that D, D` U DDg. Here the superscript means "unlabeled" as we remove the tagged distant supervision labels on them and essentially treat these samples as unlabeled ones. For the finer-graines dataset De, we adopt the same strategy, as D.- = D' U D..
Notably, the decomposition of the task/data taxonomy is grounded at the token level; namely, the same sentence could contain clean tokens and noisy tokens belonging to different sets for training respectively. Further, the assignment of the taxonomy is dynamic and altered on the fly. This means that at every round of training, DesERT performs token selection based on an inference path conducted by the model trained at the current cycle.
4.3 Debiased Self-Training
In the previous step, we resplit the datasets and process on the clean token sets Dj, and Di, leaving unselected noisy token sets Dj and De remained be tackled. Since labels for Dj and Du are likely to be noisy, we remove them and tag these tokens with pseudo-labels via a self-training framework. Notably, the pseudo-labels can be obtained from various sources and we choose a teacher-student architecture that is the same previous work [10]. Despite that self-training has been proven effective repeatedly [10 9 [] we argue that on this occasion it might still be unreliable due to another source of inherent bias.
In practice, the training on the clean and noisy set alternates. In hindsight, the supervised learning on the selected clean set naturally clusters the clean tokens tightly to their respective class centers. By contrast, the noise tokens receive dynamically changing labels, and likely that the labels are wrong. This, in theory, may cause the corresponding representation to deviate from the class centers and even oscillate by big magnitude due its instability assignment during training. Given the imbalance prediction nature of the DSNER task, we empirically find that this kind of bias is
5

---

trended to aggravate if no other mechanism is imposed. Further, if we let this bias be untangled, the clean token selection we introduced previously might be severely biased and polluted.
Therefore, to overcome this issue, we employ a worst-case cross-entropy (WCE) loss objective [2 In nutshell, the WCE loss adversarially optimizes the unlabeled token representation by matching the worst classifier on labeled data. We call classifier the worst if it can correctly classify all the labeled data, but is very close to the edge of clusters, resulting adversarially train the unlabeled tokens to be correctly classified by this worst classifier, their features can be tight least one class center. Therefore, the decision hyperplane can pushed to a lower entropy region, which is known to be beneficial for semi- sami--spervieed learning [33]. By regularizing noisy tokens to approach the class centers rather than the decision boundary. the classifier can also generate more stable and higher-quality pseudo-labels. For better understanding, we elaborate our theoretical insights of this debiased learning procedure in Appendix
minimal margin.
Nevertheless, in practice, it is impossible to obtain the truly worst classifier. Hence, proximally, we estimate a possible worst classifier bho such that it distinguishes the selected tokens correctly while making mistakes on (i)-the unchosen tokens and (ii)-the produced pseudo-labels at previous cycles, as much as possible. Formally, we denote f h o and calculate the worst head by,
h arg max v. Lu (V", f (x")) LL (x*))
where the encoder @ is frozen at this step. y = -f (x")) is the pseudo-label that serves as a proxy to estimate the classification error.
Then, we adversarially optimize the encoder be correctly classified by the worst-case classifier. The formulation Lwee is given by:
Lweee (+) = Lu (h" - L (0) 1 (x'))
Noted that we fix the worst-case head The and only update the encoder representation. Moreover, since the binary head regards all fine -ra-grai types as an integrated class, optimizing WCE on may cause a collapse of the representation and then, hinder the model of distinguishing different fine-grained entity types. Therefore, we calculate WCE loss Ce_wee on the entity-type recognition pathway. Empirically, the noisy token representation gradually approaches to corresponding class center. Consequently, the decision hyperplane is more likely to lie lower entropy region, thereby mitigating the impact of self-training bias.
4.4 Practical Implementation
Dual Co-guessing Mechanism. It has been widely verified that larger labeled set can greatly boost sunevsupee training. However, our selection procedure, the size of the clean set is largely restricted by the number of originally clean samples owing to our label-matching constraint. Fortunately, many tokens, while originally noisy, can be accurately predicted by the model as SSL training proceeds. To this end, we develop co-guessing mechanism that collects well-predicted tokens as a pseudo proxy to enlarge the clean sample set.
Specifically, we train two peer networks having the same architecture. which we denote as f(; 01) and f(x; f2). Given those easy-to-learn noisy tokens, we postulate that different networks can generate consistent and high-confidence predictions. This gives rise to the following criterion,
{ax 06 (T!
TT! = o!(2) A (max(p;") (1) T) A (max (p. (p)))
T)}
(2) is same. When it comes to specific sub-datasets D, and D... the predicted labels should be converted to respective counterparts, and probability is generated by the corresponding pathway.
(1) and (22 are predicted labels from two models, p!d) voftmax(fs(e (z))) and p y3
Training loss.
The overall training loss is defined as follows:
L cle Le + Le weee
Here w: > 0 is a weighting factor. La cts and dt are the classification losses on the two predictive heads respectively. Le we_ is the WCE loss the entity pathway.

---

Table 1: Main results five benchmark datasets measured by Precision (P), Recall (R), and F1 scores. We highlight the best overall performance for distant supervision in bold.
CONLLO3
OntoNotec5.0
Webpuge
Wikignndd
Twitter
Methods
Fulty-supervised methods
BILSTM-CC
91.35
91.66
91.21
85,99
86.36
86.17
50.07
54.76
52.34
55.40
54.30
54,90
60.01
46.16
52.18
ReBERTa-buse
89.14
91.10
90.11
84.59
87.88
86.20
66.29
79.73
72.39
85.33
87.56
86.43
51.76
52.63
52.19
Distantly-xppervisee methods
AutoNER
75.21
60:40
67.00
64.63
69:95
67.18
48.82
54.23
51.39
43.54
52.35
47.54
43.26
18.69
26.10
LRNT
79.91
61.87
69.74
67.36
68.02
67.69
46.70
48.83
47.74
45.60
46.84
46.21
46.94
15.98
23.84
Co Co-teaching+
86.04
68.74
76.42
66.63
69.32
67.95
61.65
55.41
58.36
55.23
49.26
52.08
51.67
42.66
46.73
JoCoR
83.65
69:69
76.04
66.74
68.74
67.73
62.14
58.78
60.42
51.48
51.23
51.35
49.40
45.59
47.42
NegSampling
80.17
77.72
78.93
64.59
72.39
68.26
70.16
58.78
63.97
49.49
55.35
52.26
50.25
44.95
47.45
BOND
82.05
80.92
81.48
67.14
69.61
68.35
67.37
64.19
65.74
53.44
68.58
60.07
53.16
43.76
48.01
SCDL
87.96
79.82
83.69
67.49
69.77
68.61
68.71
68.24
68.47
62.25
66.12
64.13
59.87
44.57
51.09
Ours
86.23
87.28
86.75
66.38
72.08
69.11
71.52
72.97
72.24
60.77
68.10
64.23
56.44
48.38
52.10
(uret(inettne))
86.41
87.49
86.95
66.63
71.92
69.17
72.48
72.97
72.73
62.87
69.42
65.99
57.65
47.80
52.26
Post-hoc Entity Pathway Finetuning. In previous steps, we train the double-head pathway sep- arately both pathways can be unreliable due to the label noise. Over the course of training, we observe the binary pathway learns well thanks abundant training samples. On the contrary, the entity pathway still faces rather challenging condition caused by inefficient samples, label noise, and more categories. To relieve this problem, we propose to post-hoe finetune the entity pathway by constructing a more refined dataset via the binary pathway. In specific, after the training procedure, we freeze the binary pathway hG and collect new clean entity dataset by,
Dr_ = (Z,, 0f u(( 1) A I(f of
That is, while still anticipating selecting clean samples, we remove those tokens predicted as non- entity by has which produces a more refined clean sample set than before. Lastly, we finetune the entity pathway with this clean dataset for a few epochs to boost the performance.
5
Experiments
In this section, we present our main experimental results to show the effectiveness of our method. More empirical setups and results are reported in Appendix
5.1 Setup
Dataset. We evaluate our framework five widely-used named entity recognition benchmark datasets in the English language: (1) CoNLLO3 [4] is collected CoNL. 2003 Shared Task, consisting of 1,393 English news articles and four entity types; (2) OntoNotes5.0 [35] contains 1.6 million words annotated by 18 entity types from multiple domains, including broadcast conversation, Web data, and P2.5 data; (3) Webpage (36] collects 783 entities belonging the four types like CoNLLO3, which are from 20 webpages including personal, academic, and computer-science conference homepages. (4) Wikigold (37] comprise 39,007 Wikipedia article tokens from 2008 English dump with four CoNLLO3 named entity tags: (5) Twitter (38] is an open- doen-nomain NER dataset coming from the WNUT 2016 NER shared task, which consists of 2,400 tweets (a total 34k tokens) with 10 entity tags. To obtain distant supervision scheme, we follow previous works [10) to re-annotate all the training corpus by matching entity types in external knowledge bases including Wikidata corpus and gazetteers extracted from multiple online sources.
Baselines. We compare the proposed framework with a total of seven current distantty-suueerviee methods: (1) AutoNER 7 trains a fuzzy LSTM-CRE network to handle tokens with multiple possible labels, using a Tie or Break scheme; (2) LRNT [17] employs Paritial-CRF for sequence labeling and training with high-quality sentences given by high annotation confidence and coverage: (3) Co-teaching+ [39] is a robust sample selection-based algorithm, which only updates the dual networks by prediction disagreement data; (4) JoCoR 40 adopts dual networks to calculate
7

---

89
72
88
71
87
eogre
80
86
82
84
81
68
single-head
single-head
sinle-heed double-head
67
single-heed
couble- head
double.hes
double-heae
87
10 20 30 Epoch
35
45 50
10 15 25
40 50 Epoch
35
10
20
35 Epoch
45 50
5 10 15
25 30 Epoch
40
(a) PER-FI
(b) LOC-FI
(C) ORG-FI
(d) MISC-FI
Figure 3: The F1-scores of DesERT with/without double-head pathway four entity types, which have 11. 1..././3/10.00/44 tokens respectively (from left to right).
joint loss with so-regularization; (5) NegSampling [4] handles incomplete annotations by negative sampling; (6) BOND [TO] is a self-training paradigm that selects reliable clean tokens for training by teacher-student network with early stopping: (7) SCDL. [9] jointly trains two teacher-student networks to iteratively perform label refinery a mutually beneficial manner. Besides, we select two fully-supervised methods trained with clean annotations to show the performance upper bound: (1)
BiLSTM-CNN-CRE (BILSTM-CC) (3] adopts bi-directional LSTM with character-level CNN to generate token embeddings, followed by CRF layer to predict the most likely label sequence; (2) RoBERTa- (5] adopts a pre-trained RoBERTa-bass model as the encoder.
Implementation details. We adopt two pre-trained language models RoBERTa base and Dis- úlRoBERRa as the backbone. Inspired by [10], we further maintain teacher network that is an exponential moving average of the main model. Thereafter, the data split, pseudo-labeling, loss calcu- lation, and co-guessing operators are done by the prediction of the teacher network; see Appendix C.4 for detailed descriptions. We follow [0 [ to tune most hyperparameters. Specifically, we train the networks for 50 epochs with a few epochs of warm-up. followed by 2 epochs of finetuning. The training batch size is set as 16 on four datasets, except 32 on OntoNotes5.00 The learning rate is 1e-5 2e-5) for these datasets respectively. The confidence threshold parameter t is tuned by 0.9 for Wikigold while 0.95 for others. The co-guessing is performed from the k-th epoch, which
fixed as (1e-5, 25-5 1e-5
set to {6, 40, 35, 30, 30) respectively. For finetuning, the learning rate is one-tenth of the original one. Once the training ends, we choose the first student model for predictions the final results. We run all experiments for three random trials and record the mean results.
5.2 Main Results
We measure the performance of all methods by Precision (P), Recall (R), and Fl scores. As shown in Table [ our proposed method DesERT achieves state-of saaee--ffthe-atte ttt-hherrr art performance compared with other baselines on five benchmark datasets. Specifically, on the CoNLLO3 and Webpage datasets, DesERT outperforms the best baseline methods by 3.26% and 4.26% in terms of F1 score. The performance even close the best fully-supervised baseline (i.e. RoBERTa-base) on Webpage and Twitter, which suggests better adaptation to distant supervision. In addition, DesERT displays significantly higher recall on almost all datasets, demonstrating a strong ability to identify entities from text. also maintains moderate precision as to attain extraordinary F1 scores. Finally, the model after post-hoc finetuning shows obviously better performance. It demonstrates that a well-trained binary head can help the fine-grained head better distinguish entity types, proving the superiority of the task decomposition formula.
5.3 Ablation Study
To explore the effectiveness of each component in DesERT, we conduct an ablation study on the CoNLLO3 dataset without the finetuning stage. As shown in Table test the following cases: (1) remove the WCE loss; (2) do not perform dual networks co-guessing; (3) replace the double-head pathway with single-head pathway.
can be seen from the ablations that the removal of any component leads to performance degradation, while the model can casily obtain the best Fl score under the combined effects of three components.

---

87
7000
labels
86
86
Matched he sel
Rarmbee of Feemdee
Co- Do-guessing sol
84
5000
F1 score
85
4000
82
5000
80
Precision
78
83
w/ WCE
Recall
w/o WCE
1000
76
F1
82
5 10 15 20 25 30 35 40 45 50 Epoch
0
84.00
B-ORG
B.PER
B-MISC
IPER
MMIS
HLC
0.70 0.75 0.80 0.85 0.90 0.95 1.00
(a) Analysis on WCE
(b) Analysis on Co-Guessing
(c) Analysis on
Figure 4: (a) The Fl curves of DesERT with/without WCE loss. (b) The distribution of the true labels and selected labels with/without co ro-guessing. (c) The parameter study of different confidence thresholds on the CoNLL03 dataset.
Specifically, discarding the co-guessing mechanism will cause the maximum performance descent of 3.31% % on F1 scores, which indicates the benefits of using co-guessing to correct distant labels for training. As the core mechanism of DesERT, the double-head pathway can also improve the Fl score by 2.21% % compared with only a single-head pathway, indicating that the idea of task-data decomposition is more suitable for distant supervision. Finally, WCE loss provides an effective performance gain debiased add-on. We additionally note that Precision and Recall metries also optimal, which proves the model can accurately separate entity and non-entity tokens as well as identify relatively complete entities.
5.4 Further Analysis
Efficacy of decoupled learning. To validate the efficacy of decoupled learning, further test the Fl score of each entity type in two cases on the CoNLLO3. First, by comparing the F1 scores of four entity classes shown in Figure 3 it can be seen that the conventional single-head model demon- strates inferior results on all entity types. More importantly, we found that the two models show rel- atively close behavior in identifying entity classes with sample-rich tokens like PER. In contrast, the performance gap in minority categories is large. For example, the Fl score on the MISC entities of the double-head pathway is 7.11% higher than that of the single-head pathway (Figure (d)), and the gap for LOC entities is 2.91% (Figure (b)). Therefore, the design of a double-head pathway or the idea of task decomposition can well adapt to DSNER and demonstrate stronger entity recognition capability for minority entity classes.
Table 2: Ablation study on CoNLL03 dataset.
CoNLL03
Ablations
P
R
F1
Ours*
86.23
87.28
86.75
w/o WCE
85.04
85.98
85.50
w/o co-guessing
82.34
84.57
83.44
w/o double-head
84.18
84.90
84.54
Efficacy of debiased self-training. We study the effectiveness of WCE loss for the DesERT framework. We test the cases with and without adopting WCE loss on the CoNLLO3 dataset, and the results are shown in Figure It can be found that WCE loss more likely to serve as an effective add-on in DesERT. It should be noted that WCE displays high- tigh-peeformance growth in early epochs, which alleviates the model bias towards unpurified labels in the early stage.
Efficacy of co-guessing. Besides, we explore the ability of the co-guessing mechanism to select extra tokens. In Figure 4 (b), compare the selected token distribution with and without the co-guessing operation. Without co- ro-guessing, a huge gap occurs between the selected tokens per class and the real distribution, e.g., more than 3,000 tokens are not selected on the already scarce class MISC. The reason is that we enforce the clean tokens to have matched prediction on given labels. This severely impairs the recognition performance of minority entities. With our co-guessing mechanism, we observe the clean sample set is significantly enlarged and the distribution of selected entity label distribution also approaches that true labels. Combined with the fact in Table 9 that co-guessing does indeed improve performance, the effectiveness of co- ro-guessing is clearly validated.

---

Table Performance DesERT with different sources of distant labels on CoNLL03.
Supervision
Unsupervised
ChatGPT Labels
KB Labels
Hybrid Labels
Model
ChatGPT ChatGPT-A
SCDL
DesERT
SCDL
DesERT
SCDL*
DesERT*
Precision
68.95
79.11
68.39
81,91
87.96
86.23
83.87
87.24
Recall
64.16
63.13
72.74
77.38
79.82
87.28
85.50
88.93
F1
66.47
70.22
70.50
79.58
83.69
86.75
84.67
88.08
Efficacy threshold parameter 7. We study the effect of the important hyperparameter r used in clean sample selection and dual networks co guessinsng the CoNLLO3 dataset. Specifically, vary the value of in the range 0.7 to 0.99, and other parameters are kept as default. Then we validate DesERT's performance by applying different values, shown in Figure 4 (c). We can see that the model's performance is relatively insensitive in the r varies from 0.95 to 0.99, but drops a lot when the value is lower than 0.95. And DesERT achieves the best results with r fixed 0.95 the CoNLLO3. In practice, we suggest setting a high confidence threshold 7.
5.5 Distant Supervision from Large Language Models
Recently, large language models (LLMs), including GPT-3 42 ChatGPT, and GPT-4 have largely revolutionized the NL. landscape. Thanks to their emerging abilities like in-context learning (ICL) 43 and thain-of-though 44. LLMs demonstrate remarkable zero-shot learning performance in wide range of downstream NLP tasks. Despite the promise, some recent studies [45) have shown that LLMs are still legs behind the fine-tuned small language models in many NLP applications including NER. To deal with this problem, we extend the DSNER formulation and design a novel in-context learning algorithm that exploits self- eeffraenerteed text-tag pairs to generate distant labels. Moreover, we modify our original algorithms to fully use hybrid labels including ChatGPT-gennrraee labels and original knowledge-bass baadde--bba--ggerrteedd
generated labels (KB labels). We refer the readers to Appendix C.I for detailed description of our prompt design, data generation process, and methodology.
Empirically, provide series of experiments to verify the feasibility of utilizing ChatGPT for distant supervision. In particular, we compare DesERT with three algorithms: (1) ChatGPT: we employ the ChatGPT model produce zero-shot tagging employ ChatGPT to generate a set of text-tag pairs and use it for few-shot ICL on testing data; (3) SCDL: the most competitive baseline in our main Table. Table lists our full results, have the following conclusions: (1) ChatGPT does indeed demonstrate inferior results even compared with distandy--uupervised SLMs; (2) simply training current perffrmnnnn DSNER methods on ChatGPT supervision can be inferior since they are particularly designed for KB labels; (3) ChatGPT can be complementary source of distant supervision to existing KB labels, and our modified DesERT (dubbed DesERT")) still obtains best performance when handling hybrid labels. Overall, we firmly believe that DSNER remains a pivotal research pursuit in the LLMs era. Looking forward, we envision LLMs serving as a novel source of distant supervision, facilitating the distillation of distantly-superrisee SLMs, which has enhanced accuracy and computational efficiency.
testing data; (2) ChatGPT-A: we
6 Conclusion
In this paper, we propose a novel framework DesERT for NER under distant supervision. We first find that existing self-training methods suffer from two types of biases, i.c., skewed data distribution in distant supervision and an inherent bias to assign erroneous pscudo-labels. To mitigate both biases, we propose novel task/data taxonomy decompose the original task, alongside debiased self-training to improve the quality of pseudo-labels Extensive experiments on other standard benchmarks clearly supplement new ChatGPT-basse DSNER
validate the effectiveness of DesERT. Furthermore, benchmark and show that DesERT still obtains the best performance. We hope our work can draw more attention from the community the bias issues in the histantl--uppervieed NER tasks.
hhttss//openai.coo//blgghhggt
10

---

Acknowledgments and Disclosure of Funding
This work is majorly supported by the National Key Research and Development Program of China (No. 2022YFB3304100 and in part by the NSFC under Grants (No. 62206247). Junbo Zhao also thanks the sponsorship by CAAI-Huawei Open Fund and the Fundamental Research Funds for the Central Universities (No. 226-2022-00028).
References
[1] Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. A survey on deep learning for named entity recognition. IEEE Trans. Knowl. Data Eng., 34(1):50-70, 2022.
[2] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270, San Diego, California, June 2016. Association for Computational Linguistics.
[3] Xuezhe Ma and Eduard Hovy. End-to-end sequence labeling via bi-directional LSTM-CNNS: CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguis- ties (Volume 1: Long Pupers), pages 1064-1074, Berlin, Germany, August 2016. Association for Computational Linguistics.
[4] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-41 Minneapolis, Minnesota, June 2019, Association for Computational Linguistics.
[5] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907, 1692, 2019.
[6] Yaosheng Yang. Wenliang Chen, Zhenghua Li, Zhengqiu He, and Min Zhang. Distantly supervised NER with partial amnotation learning and reinforcement learning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2169, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics.
[7] Jingbo Shang. Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. Learning named entity tagger using domain-specific dictionary. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2054-2064, Brussels, Belgium, October-November 2018. Association for Computational Linguisties.
[8] Stephen Mayhew, Snigdha Chaturvedi, Chen-Tse Tsai, and Dan Roth. Named entity recognition with partially annotated training data. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 645-655, Hong Kong, China, November 2019. Association for Computational Linguistics.
[9] Xinghua Zhang, Bowen Yu, Tingwen Liu, Zhenyu Zhang, Jiawei Sheng, Xue Mengge, and Hongbo Xu. Improving denoising learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 5518-1529, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties.
named entity recognition with self-collaborative
[10] Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang BOND: bert-assisted open-domain named entity recognition with distant supervision. In SIGKDDD, pages 1054-1064. ACM, 2020.
[11] Yu Meng, Yunyi Zhang, Jiaxin Huang, Xuan Wang. Yu Zhang, Heng Ji, and Jiawei Han. Distantly-supervised named entity recognition with noise-robust learning and language model augmented self-training In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 10367-10378, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
11

---

[12] Ryuichi Kiryo, Gang Niu, Marthinus Christoffel du Plessis, and Masashi Sugiyama. Positive- unlabeled learning with non-negative risk estimator. In NeurIPS, pages 1675-1685, 2017.
[13] Yunrui Zhao, Qianqian Xu, Yangbangyan Jiang, Peisong Wen, and Qingming Huang. Dist-pu: Positive-unlabeled learning from a label distribution perspective. In CVPR, pages 14441-14450. IEEE, 2022.
[14] Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuanjing Huang. Distantly supervised named entity recognition using Meeting of the Association for Computational Linguistics, pages 2409-2419, Florence, Italy, July 2019. Association for Computational Linguistics.
learning. In Proceedings of the 57th Annual
[15] Kang Zhou, Yuepei Li, and Qi Li. Distantly supervised named entity recognition via confidence- based multi-class positive and unlabeled learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1198-221, 1, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[16] Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. Cross Training named entity tagger from imperfect annotations. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLA) pages 5154-5163, Hong Kong, China, November 2019. Association for Computational Linguistics.
[17] Yixin Cao, Zikun Hu, Tat-seng Chua, Zhiyuan Liu, and Heng Ji. Low-resource name tagging learned with weakly labeled data. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing JEMNLPPIJCNLP), pages 261-270, Hong Kong, China, November 2019. Association for Computational Linguistics.
[18] Wenxuan Zhou and Muhao Chen. Learning from noisy labels for entity-centric information extraction. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 5381-5392, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[19] Weiran Pan, Wei Wei, and Feida Zhu. Automatic noisy label correction for fine-grained entity typing. In IJCAI, pages 4317-4323, ijcai.org, 2022.
[20] Xiaoye Qu, Jun Zeng, Daizong Liu, Zhefeng Wang, Baoxing Huai, and Pan Zhou. Distantly- supervised named entity recognition with adaptive teacher learning and fine-grained student ensemble. In AAAI, pages 13501-13509. AAAI Press, 2023.
[21] Xiusheng Huang, Yubo Chen, Shun Wu, Jun Zhao, Yuantao Xie, and Weijian Sun. Named entity recognition via noise aware training mechanism with data filter. In Findings of the Association for Computational Linguistics: ACI/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 4791-4803. Association for Computational Linguistics, 2021.
[22] Wenkai Zhang, Hongyu Lin, Xianpei Han, Le Sun, Huidan Liu, Zhicheng Wei, and Nicholas Jing Yuan. Denoising distantly supervised named entity recognition via a hypergeometric probabilis- tic model. In AAAI, pages 14481-14488. AAAI Press, 2021.
[23] Farhad Nooralahzadeh, Jan Tore Lonning, and Lilja Øvrelid. Reinforcement- denoising of distantly supervised NER with partial annotation. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP, Deepl @ EMNLP-IJJCNLP 2019, Hong Kong, China, November 3, 2019, pages 225-2222255 25-333..44 Association for Computational Linguistics, 2019.
[24] Daoyuan Chen, Yaliang Li. Kai Lei, and Ying Shen. Relabel the noise: Joint extraction of entities and relations via cooperative multiagents. In ACL, pages 5940-5950. Association for Computational Linguistics, 2020.
[25] Haoming Jiang, Danqing Zhang, Tianyu Cao, Bing Yin, and Tuo Zhao. Named entity recognition with small strongly labeled and large weakly labeled data, ACIAJJNLLP, pages 1775-1789. Association for Computational Linguistics, 2021.
12

---

[26] Wenkai Zhang, Hongyu Lin, Xianpei Han, and Le Sun. De-biasing distantly supervised named entity recognition via causal intervention. In ACL/JJCNLP, pages 4803-4813. Association for Computational Linguistics, 2021.
[27] Dong- Hayn-Hyunuu Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. JCML, volume 3, page 896, 2013.
[28] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florins Balcan, and Hsuan- Tien Lin, editors, NeurIPS, 2020.
[29] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi sumi--speriised learning with curriculum pseudo labeling. In NeurlPS, pages 18408-18419, 2021.
[30] Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In CVPR, pages 10684-10695. Computer Vision Foundation / IEEE, 2020.
[31] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-supervised learning. In IJCNN, pages 1-8. IEEE, 2020.
[32] Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self-training for semi-supervised learning. In Advances in Neural Information Processing Systems, 2022.
[33] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NeurlPS, pages 529-536, 2004.
[34] Erik F. Tjong Kim Sang and Fien De Mculder. Introduction to the CoNLL-2003 shared task: iennuugg--independent named entity recognition. In Proceedings of the Seventh Conference Natural Language Learning at HLT-NAACL. 2003, pages 142-147, 2003.
[35] Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, al. Ontonotes release 5.0 Idc201 3t19. Linguistic Data Consortium, Philadelphia, PA, 23, 2013.
[36] Lev Ratinov and Dan Roth. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147-155, Boulder, Colorado, June 2009, Association for Computational Linguistics.
[37] Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R. Curran. Named entity recognition in Wikipedia. In Proceedings of the 2009 Workshop The People' Web Meets NLP: Collaboratively Constructed Semantic Resources (People's 's Web), pages 10-18, Suntec, Singapore, August 2009. Association for Computational Linguisties.
[38] Fréderic Godin, Baptist Vandersmissen, Wesley De Neve, and Rik Van Walle. Multimedia lab @ ACL. WNUT NER shared task: Named entity recognition for Twitter microposts using distributed word representations. In Proceedings of the Workshop on Noisy User- sher-generated Text, pages 146-153, Beijing, China, July 2015. Association for Computational Linguistics.
[39] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W. Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, volume 97 of Proceedings of Machine Learning Research, pages 7164-7173. PMLR, 2019.
[40] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularivation. In CVPR, pages 13723-13732. Computer Vision Foundation/ / IEEE, 2020.
13

---

[41] Yangming Li, Lemao Liu, and Shuming Shi. Empirical analysis of unlabeled entity problem in named entity recognition. In ICLR. OpenReview.net, 2021.
[42]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[43] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022.
[44] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning large language models. In NeurlPS, 2022.
[45] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. CoRR, abs/2302.04023, 2023.
[46] Yangming Li, Han Li, Kaisheng Yao, and Xiaolong Li. Handling rare entities for neural sequence labeling. In ACL. Association for Computational Linguistics, 2020.
[47] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of life's code through self-supervised deep learning and high performance computing. CoRR, abs/2007.00625. 2020.
[48] Junyuan Xie, Ross B. Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, volume 48 JMLR Workshop and Conference Proceedings, pages 478 487. JMLR.org, 2016.
[49] Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR. OpenReview.net, 2020.
14

---

A Dataset-Task Taxonomy
A.1 Dataset Taxonomy
We provide Figure 3 to show the dataset taxonomy in the overall training workflow. We first split the initial dataset 2 into two sub-datasets D, and De through decoupled learning to mitigate the distributional bias (first level split). Next, to process respective noise in D2 and De, we conduct selection and self-trainin framework. We divide the two parts into the clean set and noisy set by clean token selection (second level split), then perform standard classification on each clean set. As for noisy token sets, we adopt debiased self-training to yield pseudo-labels for further training.
initial dataset
D
first level split
decoupled learning
binary dataset
entity type dataset
De
second level split
clean token selection
clean token selection
clean token set
noisy token set
clean token set
noisy token set
D
D
D
D
standard classification
debiased self-training
standard classification
debiased self-training
Figure 5: Illustration of dataset taxonomy.
A.2 Task Taxonomy
DesERT modifies the basic NER model architecture with the double-head pathway, yet reserves a shared pre-trained language model encoder such as RoBERTa-base denoted by Φ. Given any sentence x with its binarilized labels yd and entity type labels y", (z. y²) E Db and (z.y") E D z is first fed to the PLM encoder á and we take the last hidden layer output of o as finally embeddings 6(z). Then the double-head pathway hy and he take ((x) as input to yicld respective predictions. The binary pathway h, generates the probablity of being entity tokens, p" sigmoid(h, o ((x)) = Ip?....,p%:. p%).
then take pd > 0.5 as predicted binary labels g². While the entity pathway he offers fine-grained entity type probablities Pd = softmax(h, (h. ((x)) = [pf (pf:-..,PPll where each p has K entries. We take arg max as predicted entity type labels ÿ and note that non-entity tokens are tagged with invalid labels. Finally, any standard classification loss can be calculated on the double-head pathway. We also refer the reader to Figure 2 for visualized illustration.
B Theoretical Insights of Debiased Semi-supervised Learning
Though there is no available theory on why the worst cross-entropy (WCE) [32 works, we would like to provide the following (relatively) theoretical insights that may help the readers to perceive our approach better.
Notably, the self-training bias is mainly caused by noisy tokens approaching the decision boundaries, whose pseudo-labels keep changing. To this end, we optimize WCE to learn compact token clusters for reducing wrong pseudo-label assignments. To see this, we start from the following simplified example to show that WCE does indeed concentrate the token representation.
Assumption. Consider binary classification problem, as the simplest form of DSNER. Denote the input variable by X and the output variable by Y from binary labels -1). The labeled data D
15

---

are sampled from X|Y +1 (B( r)) and X|Y = =-1~ =-1~~ (U,(vr)) r)). Here ((u, r) denotes spherical ball with center and radius r. denotes uniform distributions. The unlabeled sample DD are all from Y = +1 but uniformly distributed inside the B((u + v)/2,r'). Three balls have no intersection, Finally, we assume the maximum margin classifier used, which is a hard proxy the cross-entropy loss.
Derivation Sketch. At first glance, it is obvious that the optimal classifier on D' is ʃ = (u + v) which misclassifies half of the examples in D The worst classifier, however, amounts to be f = -u + (u/ - r' which perfectly classifies D' but possesses the most side-way decision boundary. Next, we optimize the feature extractor & to match the worst decision boundary by ming Lu (y f)). This is to say, with ideally known labels, the unlabeled ball Bi(u + v)/2, r') converges f = (u + /2 -2--1 r',r'). With full samples X|Y = +1 getting closer, the classifier achieves better generalization with compact clusters and low-entropy decision boundaries.
In practice, since the true labels unknown, data are assigned true labels. So, the representation will be partially concentrated.
pseudo-labels as proxy since most unlabeled
Empirical Covariance. We conduct experiments on CoNLLO3 to show the covariance of data to their class centers. When DesERT is run without WCE, the average covariance amongst classes is 0.0085, With WCE, the average covariance amongst classes becomes 0.0056. Thereafter, we can conclude that WCE indeed concentrates the tokens and mitigates the self-training bias.
C Additional Experimental Setups and Results
In what follows, we show more experimental details and results. In section C.1 we report more empirical results, including an interesting series of experiments where additional distant supervision comes from large language models like ChatGPT. In section C.3) we provide more details experiments and implementation.
C.1 Results with Additional Distant Supervision from ChatGPT
Recently, large language models (LLMss), including GPT-3 42). ChatGPT, and GPT-44 have largely revolutionized the NLP landscape. Thanks to their emerging abilities like in-context learning (ICL) 43 and chain-of-thought [44), LLMs demonstrate remarkable zero-shot learning performance in a wide range of downstream NLP tasks. Despite the promise, some recent studies 45 have shown that LLMs are still legs behind the fine-tuned small language models in many NLP applications.
Motivated by this, we conduct experiments to show the zero-shot performance of ChatGPT on the NER problem. In Table we observe that ChatGPT does indeed demonstrate inferior results even compared with distantly-supervian
SL.Ms. Therefore, question arises: how can L.MMs better support NER with minimal human annotation?? Among the numerous potential solutions, we propose natural extension of the conventional distantly-superviste NER problem. This extension considers LLM's predictions on the training set as additional distant supervision. To achieve this, we present a new tagging scheme for NER and modify algorithm to accommodate multiple sources of distant supervision. Subsequently, we provide a detailed elaboration on the aforementioned aspects.
C.1.1 Tagging by Auto-Demonstration
Instead of directly performing zero-shot testing, we introduce a new DSNER paradigm that generates distant labels by LLMs for unsupervised training data to improve the SLMS' performance. To generate distant labels, a straightforward solution to send the raw training texts to the L.MMs and ask them to output all the entities along with their types. However, to achieve satisfactory NER performance. To mitigate this problem. we design novel in-context learning algorithm that exploits self- self-generated text-tag pairs to guide the tagging process.
find such naive strategy fails
Automatic Text- Tag Pair Generation. Our ultimate goal is to perform few-shot in-context learning that better guides the LLMs to locate the entities and output their types. To achieve this,
hhapss//openai.ccomm/bb//hggggg
16

---

Prompted Input: are required produce 100 English sentences annotations the task of name ed entity recognition on the CONLL dataset. These samples will used as prompt examples the CPT model. There are four kinds ertity types:
'PER' for person, 'LOC' for location, 'ORG' organization, and 'MISC' for other miscellaneous specific entilies (e sspeccccc events, nalionalities, products, or works of art.). + Entity Typea Def.
The task is to classify the nemed entities in the given sentence as as one the mentioned entity types. For exampla, 10 of the unlabeled samples CONLL dataset are as follows:
[EU rejects German call to boycott British lamb J...
Unsupery rvi sed
Demo
Now you are required to mimic the style of these samples and produce 100 English labeled examples on CONLL dataset. Please ensure thet only the mentioned entities are included. Do not generate samples that are too similar. It is important to maximize both the diversity of samples and the accuracy of labeling. The oulput formal should be in this form, each sample is oulpul in single line.
[eexmmpee 1) (entity content 1, entity type 1); (entity concent 2, entity type 2)). (entity content 1, entity type Iy
1
Oulpul Fornal Ling
LLMs: [Jack decided to move to Paris for his new job] (Jack, PER), (Paris, LOC)[The World Health
Organization has declared global pandemnc.]]
f Cenerated Samples
Figure 6: An example prompt for automatic demonstration data generation.
we may assume a set of demonstration text-tag pair samples are available. Besides, this set ought to be diverse and representative enough to guide in-context learning. However, the training samples are unsupervised and cannot be utilized directly. To address this problem, we propose to generate sentences by ChatGPT itself, while ensuring the diversity of the generated results to cover all entity types. Additionally, we randomly retrieve unlabeled samples from the training set and employ ChatGPT automatically generate a comprehensive set of sentences, including their corresponding entity tags. An example prompt is shown in Figure 6
Few-Shot In-Context Learning for NER Tagging. After that, we ask the ChatGPT to tag the whole training set by using its self-generate demonstrations. However, directly feeding all the generated samples for ICL may exceed contextual limits and incur high computational costs. To remedy this problem, before feeding the true query sentence, we retrieve the top-k generated samples by the cosine similarity. Empirically, we exploit the BERT embedding for similarity calculation, which performs generally better than ChatGPT embedding. Finally, we instruct ChatGPT to output the NER tags for the entire unsupervised training set. An example prompt shown Figure 7
C.1.2 Modification of DesERT for Multi-Source Distant Labels
One may directly use the ChatGPT labels to train the DSNER models. However, the original knowledge base-driven distant supervision is also free lunch for DSNER. and can be further incorporated. Notably, such a hybrid distant label from multiple sources problem has never been touched in the NER community. Fortunately, labels with only a few modifications. Assume we are given knowledge base-driven labels yku (KB Labels) and ChatGPT Labels Yog for a token I, where slightly abuse the subscript to distinguish these two labels. To warm up the model, we calculate a mean soft label by.
DesERT algorithm can well address such hybrid
y mean (OneHot(y))) + Oy)
Therefore, the model fits equal confidence on these two labels when token receives two disagreed labels. After that, we develop modified selection protocol. Take the binary head as an example, we receive a set of transformed distant labels 4 selection by,
(Vh Je.y for each token and then perform the token
DJ (a, Y I[
Ab) A (max(p", -pp)=== > >))
In other words, we regard either distant label as a candidate of the ground truth. Once there is one label in this label set that exhibits high confidence, we regard it clean token. Finally, run the
17

---

Prompted Input: are helpful assistant for the task of ramed entity recognition a sentense. You reply with brief, to-the- no-the-point answers with elaboration as truthfully as possible. There are four known entity types: 'PER' which refers to person, 'LOC' whicn refers location, 'ORG' which refers organization 'MISC' which refers other specific miscellaneous entities (e.g. specific events, nationalities, products, or works art.). When the name of the country is encountered, it can recognized as a LOC (location) entity.
User: Given a sentence: 'Marcos Ondruska (outh Africa) vs. Felix Mantilla ( Spain y. Which entities may be included the given sertence, please answer in a single line with the form (entityl content, entityl type)Xentity2 content, entity2 type)Xentityy content, entity8 type).... Please only outpu: the token that you are very sure to be entity. If there is no entity in the given sentence, reply '(no entity):.
Assistant: (Marcos Ondruska, PER)xSouth Africa, LOC); Felix Mantilla, PER); LOC)
T
pemons-ration
User: Given a sentence: (Query Sentence] W'ichhh entities may be included in the given sentence, please answer in a single Line with the form (entityl content, entityl type)/entttt2 content, entity2 (ent)iy) content, entity3 type).... Pleasse only output the token that you are very sure to be entity. If there is no entity in the given sentence, reply "nn entity):.
Query
LLMs: (entityl content, entityl lype)(entity2 content, entity2 type) content, entity3 type)....
Answer to Query
An example prompt for few-shot in-context tagging for a query sentence. The demonstra- tions are automatically generated and selected.
Figure 7:
DesERT algorithm without any further modification. Notably, these can be easily extended to more sources of distant supervision, e-g., when there more than one L.LM available.
C.1.3 Experimental Results with ChatGPT Supervision
Our experiments are conducted the CONLLO3 dataset. Specifically, we gen- erate a total of 100 automatically generated samples with the help of 10 demonstration training samples. When tagging training data, we select the 10 most similar gen- erated samples for ICL. In Figure 8 we plot the confusion matrix of the ChatGPT labels. It can be observed that ChatGPT supervision demonstrates similar trends to KB labels and still biased. Nevertheless, ChatGPT labels have three main character- isties: (1) they classify far more non-entity
30000
5064
LA2
240
50
1800
185363
300000
1983
4013
1500
025
2000
Chet labels
Chat lobels
Figure 8: Left: Confusion matrix of true labels and
ChatGPT labels CoNLLO3. Right: The confusion
matrix displays noise among true entity-type labels in ChatGPT labels.
tokens as an entity; (2) they produce more balanced clean tokens; (3) the confusion patterns on fine-grained entity types are different than KB labels.
In Table 4 we report the results of DesERT when it faces different sources of distant supervision signals. In particular, we compare DesERT with three baselines: (1) ChatGPT: we employ the ChatGPT model to produce zero-shot tagging on testing data; (2) ChatGPT-A: we employ ChatGPT to generate set of text-lag pairs and use it for few-shot ICL on testing data; (3) SCDL: the most competitive baseline in our main Table. From Table
we have the following observation:
ChatGPT-A is much better than vanilla ChatGPT, verifying the superiority of our automatic demonstration process. But, ChatGPT and ChatGPT-A underperform other DSNER algo- rithms on the testing set. Though ChatGPT is wonderful seneral-purpose LLM, we may draw the same conclusion as [45) that fine-tuned SLMs still play an important role in NLP.
Given ChatGPT labels, both SCDL and DesERT underperform their counterparts when supervised by KB labels. We postulate that current DSNER algorithms are particularly designed for KB- based supervision and thus can not fully handle such new sources of labels.
18

---

Table Performance DesERT with different sources distant labels on CoNLL03.
Supervision
Unsupervised
ChatGPT Labels
KB Labels
Hybrid Labels
Model
ChatGPT
ChatGPT-A
SCDL
DesERT
SCDL
DesERT
SCDL*
DesERT*
Precision
68.95
79.11
68.39
81.91
87.96
86.23
83.87
87.24
Recall
64.16
63.13
72.74
77.38
79.82
87.28
85.50
88.93
F1
66.47
70.22
70.50
79.58
83.69
86.75
84.67
88.08
Our proposed DesERT algorithm consistently outperforms baselines on different supervised signals. In particular, when trained with hybrid labels, the modified DesERT (DesERT*) improves the KB label-trained counterpart by +..3 score. It suggests that distant supervision from LLMs does indeed brings helpful information for the DSNER task.
In summary, our work makes the first attempt employ LLMs generate distant supervision. Moreover, our DesERT algorithm can be easily extended to learn from multi-source distant labels and demonstrates improved performance.
C.2 More Experimental Results
Table 5: The sensitivity of without fine-tuning.
Datasets
k =15
20
25
30
35
40
CoNLL03
86.72
86.52
86.78
86.82
86.48
86.18
Twitter
52.22
52.15
52.21
52.10
52.44
51.54
C.2.1
Sensitivity of k:
When tuning the co-guessing epoch k, we find that setting A: = 6 has led to good performance for CoNLLO3, For other datasets, we found that it is proper to tune among {30,333 35, 40). In Table
5
we additionally present the parameter sensitivity analysis of k. It can be observed that DesERT is not very sensitive to the value.
CoNLL03
OutoNotes5.0
Webpage
Wikigold
Twitter
Ablations
w/o Binary WCE
86.23
87.28
86.75
66.38
72.08
69.11
71.52
72.97
72.24
60.77
68.10
64.23
56.44
48.38
52.10
w/ Binary WCE
84.29
86.38
85.32
66.43
72.14
69.17
70.78
73,65
72.18
59.64
67.34
63.25
56.32
48.06
51.86
Table 6: Performance of DesERT with binary WCE on five benchmark datasets measured by Precision (P), Recall (R), and Fl scores.
C.2.2 The Effect of Debiased Training the Binary Head
In this section, further investigate whether the binary head can benefit from the debiased self- training procedure. In particular, we optimize the worst-case cross-entropy loss La wee on the binary head pathway well. The final objective is,
L -= + Lo_wee Le_cls
Le_wee
In Table 6 we can observe that training with binary WCE generally lowers the Fl scores on benchmark datasets. We speculate that the reason the binary classification head does not have adequate categorical information for training the representation. Moreover, if we apply WCE on the binary head, tokens from different fine- ane-grained entities might be condensed into a large cluster. This may hinder the model of distinguishing different fine-grained entity types.
C.2.3 The Standard Deviation of Main Results
Our implementation and configurations closely adhere to the codes of existing algorithms 19. Consequently, our baseline results align with those presented in the respective source papers. In
19

---

Table 7: The standard deviation of DesERT on benchmark datasets.
Methods
CoNLL03
OntoNotes5.0
Webpage
Wikigold
Twitter
DesERT
0.22
0.43
0.40
0.46
0.32
DesERT (finetune)
0.17
0.20
0.22
0.30
0.17
Table 8: The standard deviation of DesERT on benchmark datasets.
Methods
SNIPS (20% noise)
SNIPS (30% noise)
ATIS (20% noise)
ATIS (30% noise)
DesERT
94.29
93.51
95.15
94.73
RoBERTa
85.03
77.09
90.89
90.05
order to maintain uniformity in formatting, we have refrained from reporting the variance, due to the absence of such information in the cited references. In Table 7 we present the standard deviation values corresponding to initial results. It can be shown that variances not substantial. In view of the notable performance gap (average +2222 F1), we believe that our algorithm has shown statistically significant improvement.
C.2.4 Experiments on Other Sequential Labeling Tasks
In our main content, we discussed the superiority of DesERT on the DSNER task. However, our method is actually generic framework and can be applied other sequential labeling tasks. To see this, we follow [46] on the sequence labeling benchmarks SNIPS and ATIS in a quest to make the comparison fair. We followed the literature noisy label learning and injected 20% and 30% synthetic noise. The results in Fl are shown in Table It says that DesERT performs way better than its counterparts.
Further, we conducted a simple experiment on the bioinformatics dataset FLIP that predicts the second structures of the proteins [47] We injected 20% noise
Table 9: Valid accuracy on the FLIP dataset.
Methods
Accuracy
into synthetic datasets. Owing the time limits, adapted the decoupled training and debiased SSL modules the benchmark from ProtTrans. In partic- ular, we used 30-layered ProtBERT model as the backbone and only fine-tuned the top layers due limited computation sources. We found that ProtBERT is quite robust, and thus, we only finetune its top 5 layers. However. DesERT still shows some benefits improving the robustness. These results prove the superiority of DesERT in handling reakly-supervisee sequential tagging tasks.
ProtBERT (Supervised)
80.83
ProtBERT (20% noise)
69.03
DesERT (20% noise)
70.38
C.3 More Implementation Details
In this section, provide experimental details for better understanding of training process and also to ensure the reproducibility.
C.3.1 Computation Resources
All experiments are conducted on workstation with NVIDIA RTX A6000 GPUs. It takes about {8, 72, 0.5, 1} hours for training on five benchmarking datasets (ordered as in Table with one single GPU. We adopt the Huggingface Transformer library the RoBERTa-base (125M parameters) and DistilRoBERTa-bor base (66M parameters) models: https//huughngiiie.ccctmmmmomom We run all experiments three times and report the mean results.
C.3.2 Datasets Details
The data statistics of five benchmarking NER datasets are shown in Table 10
20

---

Table 10: The statistics of five datasets, show the number of entity types and the number of sentences in the TrainDev/Test set.
Dataset
Types
Train
Dev
Test
CONLLO3
14,041
3,250
3,453
OntoNotes5.0
18
115,812
15,680
12,217
Webpage
4
385
99
135
Wikigold
4
1,142
280
274
Twitter
10
2,393
999
3,844
C.4 More Implementation Details
Tagging scheme for NER. As for the tagging scheme, follow the classic BIO format. To be specific. the first token of an entity mentioned with type X is tagged as B-X while the remaining tokens inside that entity are labeled as I-X, and the non-entity tokens are annotated as O. Such scheme is more difficult than simple IO format, especially for distant supervision.
Clean token selection for the binary pathway. In general, the precision of entity labels is relatively high, e.g., about 97.96 97.96%(23 (23, 649/24, 141) ) in the CoNLL03 dataset. That indicates us most distantly- labeled entity tokens are real entities if omit fine-grained entity types. Therefore, when performing clean token selection on the binary pathway, it only selects non-entity tokens by the matched and high-confidence strategy while including all tokens labeled entities.
Soft label for the entity pathway. When training on the selected clean token set, the binary pathway regards distant labels true labels. However, discard the distant hard labels but adopt the teacher model's output logits to derive soft labels (48] for the entity pathway, given by:
p².s// E,Pis Vh Ep, E.P.s.)
where P..j softmax(Ji., 0,)), is the probability of i-th token belonging to class j in sentence a, then calculate Bullback-Leibler divergence loss. Because soft labels usually preserve sufficient information and encourage more balanced assignment of target labels.
The implementation of feacher-student newtork. When splitting the clean token set and the noisy token set, we let the teacher model select respective clean tokens to train the student model. Specifically, the teacher model's double-head pathway filters reliable clean tokens independently following the previous defined criterion. Then the double-heat of the student model is trained with selected clean tokens and corresponding labels. The teacher model parameters are periodically updated by the student model with EMA, given by:
8g = at; + (1-a))
where a is a positive constant and is empirically fixed as 0.99 for Webpage/Wikigold and 0.995 for the remaining datasets. Finally, to train the entity pathway, adopt KL divergence loss the student model's output logits and corresponding soft labels from the teacher model's prediction. The formulation is:
es ((x 0.)) = ΣΣ úi log (zs 0.) [ey(:;; log(Js;
Then, the student model's 's entity pathway is trained to approximate the soft labels. For the binary pathway, calculate standard binary cross-entropy loss, which is given by:
et (J", .0)) - log 8² (1 ÿ²) log(1 (0)
where J is the given distant label and g² is generated by student model's 's binary pathway.
D Pseudo-Code of DesERT
We describe the overall training pipeline of DesERT in Algorithm I
21

---

Algorithm Training workflow of DesERT
Input: Training data D (zi1, with distant labels; sets of teacher-student networks,
011. 0s1 and 012, 022; 0
1* Selection and self-training "/
2: while Ti do
3:
Get a batch B = { ki D;;
4: if < l then
5:
Bg U B, + B; //ecoupled datasets
B. [ _. + Sel(f(On), BB);
6:
7:
B [BB. BB..1) Sel(f(0.1), B3);
8:
B. =18% =B Sel(f(0r2), Bo);
2
9:
B =
+ Sel(f(012), Bb);
Update 0.1 with {B B. by minimizing L:
11:
Update 0,2 with (B, B. by minimizing L;
else
13:
Xn {<:)))
14:
BB U B. &- (0e))( f0=2),
15:
Update 0,1,0, 0,2 with (Br, Be) by minimizing C
end if
17:
O²1 EMA(a, O41), 02 + EMA(a, 0:2)
18:
I 1+1+1
19:
end while
1t Post-hoc entity pathway finetuning */
20:
while t Ti + T3 do
21:
Get batch = (x,, W= C D;
22:
NB - ((..)) yn + (9.) p
B EntitySel( (f XB, BB):
24:
B. + EntitySel( f (0(2) XB, DB);
25:
Finetuning 0.1 with B
26: Finetuning 0.2 with B
27:
On1 4 EMA(a, 01), 022 + EMA(a, 0:2)
+
29:
end while
E
Limitations
While DesERT has been proven to be effective for distant supervision, it is still subject to certain limitations. First, our WCE loss estimated from pseudo-labels instead of the real ones. While we empirically find our WCE loss works well, its performance is theoretically restricted. One potential solution is to estimate a small validation set to remedy this problem, but we leave it as our future work. Second, while the imbalance between entity labels is located in Figure [ our framework does not particularly integrate special components to explicitly overcome this problem. We believe it not hard draw inspiration from the recent achievement in long-tailed learning to further improve the NER performance. Lastly, since DesERT ensembles two sets of teacher-student networks as previous works did [9 (9), should train peer-student models simultaneously and utilize the predictions from dual-teacher models iteratively, thus resulting in relatively higher training costs. We hope future efforts made alleviate the cost of network ensembling.
Ethics Statement
While distant supervision deemed a cheap way to collect and curate training data, the off-the-shelf and external knowledge bases steering the autonomous annotation procedure may include bias and
22

---

unfairness. Indeed, if one trains the model by these biased labels, it may unplcasantly yield unfair and biased predictions on the basis of characteristics like race, gender, disabilities, LGBTQ or political orientation. Therefore, when deploying our DesERT framework, it is recommended to equip with some auxiliary tools for labeling censorship so as improve overall fairness and ethical standards. Grounded on this, we would suggest regarding our DesERT framework as an auxiliary weakly-supervised annotation tool for assisting human annotations.
23


Reviews:
[{"review_id": "UvqgW9FDlT", "reviewer": "Reviewer_WUVn", "summary": "The paper introduced DesERT, a new method that improves the distant supervision training framework for DSNER (distant supervision NER) using two highly effective components: 1. To solve the skewed label distribution (entity vs. non-entity tokens) in DSNER, DesERT proposes to decompose learning into two parts: a `binary` setup where the model learns to distinguish entities from non entities, and an `inter-entity` setup where the task is to predict the specific entity category. 2. To improve pseudo-label quality and alleviate the harmful effects of confirmation bias, DesERT proposes to use a worst-case cross-entropy (WCE) loss during training. The proposed approach is tested thoroughly on 5 datasets where it obtains consistent improvements over baselines.", "questions": "Please see the questions from the `Weaknesses` section. Additionally:\n\n- You mentioned that splitting into clean/noisy is done on the fly. I would be curious to see how frequently it happens that at a later iteration a token that was clean is not clean anymore.\n\n- It seems that post-hoc tuning is not that effective  on most datasets and it may be significant only on Wikigold? Do you have some insights on this? What’s special about Wikigold? \n\n- You mentioned that only 2 epochs are used for post-hoc tuning. How was this value chosen? Does the performance degrade if more training is performed here?", "limitations": "I do not see a specific discussion of limitations. I suggest adding such a discussion. Additionally, I don't see any potential negative societal impact of this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The presentation of the paper is clear and concise. The method is easy to understand and the delivery of the paper is smooth.\n\n- The approach is extremely well-motivated; each proposed component has detailed reasoning into why this component was chosen and what problem it solves. For instance, through analysis, the authors state that true non-entity tokens are (unsurprisingly) far more frequent than entity tokens. Due to this imbalance, they notice that these non-entity tokens are picked up much more frequently and that entity tokens (which are the main focus of NER) are largely overlooked. To address this drawback, they propose to split the task in two subtasks: `entity vs. non-entity prediction` and `inter-entity prediction`, which is intuitive and well-motivated.\n\n- The experiments are comprehensive. The method is compared with numerous baselines and 5 datasets.\n\n- The analyses of the method are thorough as well. The authors analyze the benefits of each component through an ablation study and also provide theoretical insights into the WCE loss.", "weaknesses": "- The results come without a sense of variance. Are the performance improvements significant? What is the standard deviation of your approach compared to the baselines?\n\n- The paper presents detailed information about the hyperparameters used. It seems that the values of these parameters are quite specific. For instance, the co-guessing is performed starting from epochs 6,40,35,30,30 on the five datasets.  Can you elaborate more on how all these hyperparameters were chosen and the search space used? My main concern is that DesERT is extremely thoroughly tuned while the baselines are not. Can you detail the tuning of the baselines as well? \n\nThe previous concerns make me uncertain if the reported improvements come from excessive tuning or if they are significant at all. I am willing to adjust my score if this concern/these questions are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments and suggestions! We are happy you enjoyed the paper. Below are our detailed responses:\n\n**Q1. The results come without a sense of variance. Are the performance improvements significant? What is the standard deviation of your approach compared to the baselines?**\n\nA.  Sorry for missing the variance. Our implementation and configurations closely adhere to the codes of SCDL [9] and BOND [10]. Consequently, our baseline results align with those presented in the respective source papers. In order to maintain uniformity in formatting, we have refrained from reporting the variance, due to the absence of such information in the cited references. Below, we present the standard deviation values corresponding to our initial results:\n\n| Method          | CoNLL03 | OntoNotes5.0 | Webpage | Wikigold | Twitter |\n| --------------- | :-----: | :----------: | :-----: | :------: | :-----: |\n| DesERT          |  0.22   |     0.43     |  0.40   |   0.46   |  0.32   |\n| DesERT-Finetune |  0.17   |     0.20     |  0.22   |   0.30   |  0.17   |\n\nIt can be shown that our variances are not substantial. In view of the notable performance gap (average +2.22 F1), we believe that our algorithm has shown statistically significant improvement. We'll add this to our revision. \n\n\n\n**Q2. The paper presents detailed information about the hyperparameters used. It seems that the values of these parameters are quite specific. For instance, the co-guessing is performed starting from epochs 6,40,35,30,30 on the five datasets. Can you elaborate more on how all these hyperparameters were chosen and the search space used? My main concern is that DesERT is extremely thoroughly tuned while the baselines are not. Can you detail the tuning of the baselines as well?**\n\n A. Actually, we didn't spend much time tuning our DesERT since it is based on SCDL. Notably, our novel components only bring three extra parameters $\\tau$, the number of fine-tuning epochs, and the start epoch for co-guessing $k$, which is tuned via a validation set. Other parameters, including learning rate, batch size, and teacher network updating parameters all follow SCDL, i.e. these seemingly 'specific values'. We believe these parameters have been well-tuned by the authors of SCDL [9] and thus, didn't modify them.\n\nWe have to acknowledge that our approach to selecting the co-guessing epoch could be perceived as somewhat arbitrary, lending an unconventional aspect to these values. For CoNLL03, we tested $k=6$ and find the performance has been good enough. For other datasets, we found that it is proper to tune it by $k\\in\\\\{30,35,40\\\\}$. Below we show the ablation on $k$ (feel free to reproduce these results by using our source code). It can be observed that DesERT is not very sensitive to $k$.\n\n| $k$  | CoNLL03 | Twitter |\n| :--: | :-----: | :-----: |\n|  15  |  86.72  |  52.22  |\n|  20  |  86.52  |  52.15  |\n|  25  |  86.78  |  52.21  |\n|  30  |  86.82  |  52.10  |\n|  35  |  86.48  |  52.44  |\n|  40  |  86.18  |  51.54  |\n\n\n\n**Q3. The previous concerns make me uncertain if the reported improvements come from excessive tuning or if they are significant at all. I am willing to adjust my score if this concern/these questions are addressed.**\n\nA. We hope the previous discussion has adequately addressed your concerns. In fact, our core components like decoupled training and debiased SSL are general-purpose components for sequential prediction tasks; see our responses to *Reviewers#WYfp/Nukd*. However, it has been an unspoken rule in the community that one paper has to be SOTA to be accepted (while not NeurIPS's policy). Therefore, we chose the recent SOTA algorithm SCDL as our base implementation. With guaranteed fair comparisons, we believe we've shown the benefit of our framework in improving DSNER.\n\n\n\n**Q5. You mentioned that splitting into clean/noisy is done on the fly. I would be curious to see how frequently it happens that at a later iteration a token that was clean is not clean anymore.**\n\n A.  We found that only a small proportion of clean samples will be moved to the noisy set at later iterations. We may refer the reviewer for our submitted response PDF for the plotted confusion matrices across the epochs.  \n\n\n\n**Q6. It seems that post-hoc tuning is not that effective on most datasets and it may be significant only on Wikigold? Do you have some insights on this? What’s special about Wikigold?**\n\n A. We found that the reason is that Wikigold contains more false entity tokens than other datasets and thus the filtering ability of binary head helps it improve a lot. Particularly, Wikigold has 697 false entity tokens out of 2975 distantly labeled entities, whereas CoNLL03 records a count of 492/23649 in the same regard.\n\n\n\n**Q7. You mentioned that only 2 epochs are used for post-hoc tuning. How was this value chosen? Does the performance degrade if more training is performed here?**\n\n A. This is tuned via the validation set. Yes, we find that too many epochs for fine-tuning may result in slight performance degradation on some datasets, which we suppose may be caused by the overfitting of excessive training.\n\n\n\n**Q8. I do not see a specific discussion of limitations. I suggest adding such a discussion. Additionally, I don't see any potential negative societal impact of this work.**\n\nA.   We provided our limitations and societal impact sections in Appendix E&F. 😊"}, {"review_id": "N6m95cpQGM", "reviewer": "Reviewer_WYfp", "summary": "This paper presents a new method named DesERT for distantly supervised named entity recognition (NER), which starts with analyzing the noise of and biases in distant supervision which is used in self-training frameworks: 1) skewed data distribution, 2) inherent error patterns in pseudo-labels. They decompose the original NER task to a task-data taxonomy, with each component from this taxonomy manifesting an individual training paradigm. On several NER datasets, the proposed method outperforms many previous baseline methods.  ", "questions": "- See weakness. ", "limitations": "In the appendix. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is well motivated based on solid analysis in self-training and distant supervision.\n- The evaluation is done on five common NER datasets and they show great performance on all of them, better than recent existing methods. \n", "weaknesses": "- The presentation can be further improved with more illustrative examples and figures. The current presentation does not clearly introduce the biases and how the proposed method works.\n- The paper lacks a discussion about how recent LLMs could solve the (distantly supervised) NER problem. With the advanced zero/few-shot generalization ability of modern LLMs, do you still even need such a complex framework for NER? If it is still needed, in which extent? These are quite important messages to the audience, but none of them are covered. I saw some content in appendix about ChatGPT, but not sure about the concrete conclusions that you want to tell the readers. \n- The paper is rather limited to a special setting and task: distant supervised NER. To gain more interests of NeurIPS readers, I would suggest the authors generalize the paper to a broader context: sequence tagging, and do more experiments in bioinformatics tasks, considering DNA as sequences.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your insightful comments and suggestions! Our detailed responses are as follows.\n\n**Q1. The presentation can be further improved with more illustrative examples and figures.**\n\nA. Thanks for your suggestions! We provided our visualized illustration of Dataset Taxonomy in Appendix A and the pseudo-code in Appendix D. We'll carefully revise our manuscript and provide more illustrative examples and figures in our revised version!\n\n**Q2. With the advanced zero/few-shot generalization ability of modern LLMs, do you still even need such a complex framework for NER? If it is still needed, in which extent?**\n\nA. In our original submission, we provided a thorough discussion on how recent LLMs can help resolve the DSNER problem without human supervision. Below, we summarize some takeaways:\n\n1. **The zero-shot performance of ChatGPT is far from resolving the NER task.** As shown in Table 3 (Appendix), zero-shot ChatGPT exhibits 66.47 F1-score on  CoNLL03, which legs even behind the conventional distantly-supervised NER methods like our proposed DesERT. Notably, this is also supported by recent works [C1], which find that ChatGPT significantly underperformed the fine-tuned small language models in many NLP applications.\n2. **Self-generated demonstration can improve the ChatGPT on NER.** The performance of ChatGPT on unsupervised NER can be alleviated by an in-context learning algorithm. Specifically, we generate demonstration sentences by ChatGPT-given entity labels. Such a strategy can improve the F1-score to 70.22 without any human supervision.\n3. **ChatGPT prediction can serve as a new source of distant supervision.** Though the ChatGPT prediction is not satisfactory for practical employment, we find it is a good source of distant supervision for distilling a small language model. As shown in Table 3 (Appendix), DesERT obtains 79.53 F1 on such ChatGPT distant labels. Moreover, we also tested a hybrid supervision paradigm that mixes ChatGPT supervision and conventional knowledge-base supervision. We find that DesERT is able to achieve better performance (88.08 F1) than merely using knowledge base supervision (86.75 F1). \n\nIn our response PDF submission, we expound upon our findings by presenting further experimental outcomes from the Wikigold dataset, which substantiate our assertions. As a result, **we firmly believe that DSNER remains a pivotal research pursuit in the LLM era**. Looking forward, we envision LLMs serving as a novel source of distant supervision, facilitating the distillation of distantly-supervised SLMs, which has enhanced accuracy and computational efficiency. We will move part of the discussions and add more supporting results to our main content.\n\n*[C1] Yejin Bang, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interaeictivity. CoRR, abs/2302.04023, 2023*\n\n**Q4. Generalize the paper to a broader context: sequence tagging, and do more experiments in bioinformatics tasks.**\n\nA. We originally tend to focus on the setting of distantly-supervised NER, which is a practical and fundamental task in NLP and benefits many downstream applications (e.g., entity linking and relation extraction). But, the core debiasing components of DesERT can be generalized to more sequential prediction tasks, even beyond NLP tasks (e.g. action recognition for video, speech recognition). \n\nTo see this, we closely follow [C2] on the sequence labeling benchmarks SNIPS and ATIS in a quest to make the comparison fair. We followed the literature on noisy label learning and injected 20%/30% synthetic noise. The results in F1 are shown below. It says that DesERT performs way better than its counterparts, effectively proving its superiority in handling such weakly-supervised sequential tagging tasks. \n\n| Methods | SNIPS (20% noise) | SNIPS (30% noise) | ATIS (20% noise) | ATIS (30% noise) |\n| ------- | --------- | --------- | -------- | -------- |\n| DesERT  | 94.29  | 93.51| 95.15| 94.73 |\n| RoBERTa | 85.03 | 77.09| 90.89| 90.05  |\n\nTo concur with the request made by the reviewer, despite within the short period of time of this week to prepare the rebuttal and lacking of expertise, we supplement another set of the experiments from another domain --- bioinformatics --- that is completely unrelated to the NER. We conducted a simple experiment on the bioinformatics dataset FLIP that predicts the second structures of the proteins [C3]. We injected 20% noise into synthetic datasets. Owing to the time limits, we adapted the decoupled training and debiased SSL modules to the benchmark from ProtTrans. In particular, we use a 30-layered ProtBERT model as the backbone and only fine-tuned the top 5 layers due to limited computation sources. We found that the ProtBERT is quite robust since we only fine-tune 5 layers. But, DesERT still shows some benefits to improve the robustness. \n\n| Methods               | FLIP dataset (Valid Acc) |\n| --------------------- | ------------------------ |\n| ProtBERT (Supervised) | 80.83|\n| ProtBERT (20% noise)  | 69.03|\n| DesERT (20% noise)  | 70.38|\n\nWe hope these new results can alleviate the reviewer's concern about the ability of DesERT for a broader context of problem setups. We'll incorporate it into our revision.\n\nFurther, we want to provide a further defense to our paper.  As we dive into the literature, despite the criticism from the reviewer, there has been a number of papers --- dedicated to the DSNER problem --- that manage to be published in top-tiered AI conferences. Posited by many of these papers, DSNER often acts as an iconic, profound, and representative task for the track of noisy-labeled NLP which extends to a variety of sequence labeling tasks. \n\n*[C2] Li, et al. Handling rare entities for neural sequence labeling[C]//ACL. 2020: 6441-6451.*\n\n*[C3] Ahmed, et al. ProtTrans: Towards Cracking the Language of Lifes Code Through Self-Supervised Deep Learning and High Performance Computing[J]//TPAMI. 2021*"}, {"review_id": "ZEfjTjKWq7", "reviewer": "Reviewer_Nukd", "summary": "The paper studies the distantly-supervised named entity recognition (NER) problem and proposes a framework called DesERT. Motivated by the bias patterns in the distant labeling noise, DesERT conducts clean token selection to split the training set into a clean set and an unlabeled set and then trains the encoder model via debiased self-training with a worst-case cross-entropy (WCE) loss. The experiments are conducted on five standard NER benchmarks and DesERT demonstrates moderate improvements over baselines.", "questions": "RoSTER is discussed in the related work but not included in the baseline comparisons. Is there a particular reason for that?", "limitations": "The authors discussed limitations in the Appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Originality: The overall method design follows the common setup in distantly-supervised NER, by first selecting clean data and then conducting self-training, and is not novel. However, there are some details, including the analysis of the label noise distribution and the method for debiased self-training, that are new.\n* Quality: The methods are well-motivated by the challenges in distantly-supervised NER and are reasonable designs.\n* Clarity: The paper is clearly-written overall.\n* Significance: The analyses and method proposed are specific to the distantly-supervised NER setting. In addition, the empirical performance of the method is not significantly better than the baselines. Therefore, the scope of the paper and its impact is quite narrowed to be considered for being published at NeurIPS. I feel that the paper could be more suitable for an NLP venue.", "weaknesses": "* Problem scope and method design: The distantly-supervised NER setting is a fine research setup, but if the proposed method is very specific and applicable only to such one setup (as in this paper), it still feels quite narrowed especially for an ML conference like NeurIPS. The impact and scope of the paper could be enriched if the method is applicable to more types of tasks where label noise is present (this will have to be shown via empirical results, not just conceptually).\n* The empirical advantage of the proposed method over baselines is not very significant: In most cases, DesERT is only better than the previous best baseline by ~1 F1 point, with the most significant improvements on CoNLL03/Webpage. But it's possible that this is because the authors use either CoNLL03 or Webpage for their method development.\n* Comparisons with LLMs: I understand that LLMs are not typically used in IE tasks since they are not very parameter-efficient for them. However, given the current rapid development in LLMs, it'd have been nice to include some discussions and empirical results of using LLMs for NER tasks. I noticed that some results are in the Appendix, but are not in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your comments and suggestions. Below we address the feedback and comments in detail:\n\n**Q1. Problem scope and method design.**\n\nA. We originally tend to focus on the setting of distantly-supervised NER, which is a practical and fundamental task in NLP and benefits many downstream applications (e.g., entity linking and relation extraction). But, the core debiasing components of DesERT can be generalized to more sequential prediction tasks, even beyond NLP tasks (e.g. action recognition for video, speech recognition). \n\nTo see this, we closely follow [C1] on the sequence labeling benchmarks SNIPS and ATIS in a quest to make the comparison fair. We followed the literature on noisy label learning and injected 20%/30% synthetic noise. The results in F1 are shown below. It says that DesERT performs way better than its counterparts, effectively proving its superiority in handling such weakly-supervised sequential tagging tasks. \n\n| Methods | SNIPS (20% noise) | SNIPS (30% noise) | ATIS (20% noise) | ATIS (30% noise) |\n| ------- | --------------- | --------------- | -------- | -------- |\n| DesERT  | 94.29           | 93.51           | 95.15    | 94.73    |\n| RoBERTa | 85.03           | 77.09           | 90.89    | 90.05    |\n\nMoreover, We conducted a simple experiment on the bioinformatics tasks for predicting the second structures of the Proteins and we refer the reviewer for our responses to *Reviewer#WYfp* for details. \n\nWe hope these new results can alleviate the reviewer's concern about the ability of DesERT for a broader context of problem setups. And we'll incorporate all-the-above into our next revision.\n\nFurther, we want to provide a further defense to our paper.  As we dive into the literature, despite the criticism from the reviewer, there has been a number of papers --- dedicated to the DSNER problem --- that manage to be published in top-tiered AI conferences. Posited by many of these papers, DSNER often acts as an iconic, profound, and representative task for the track of noisy-labeled NLP which extends to a variety of sequence labeling tasks. \n\n*[C1] Li, et al. Handling rare entities for neural sequence labeling[C]//ACL. 2020: 6441-6451.*\n\n**Q2. The empirical advantage of the proposed method.**\n\nA. To recap a bit, despite what was mentioned by the reviewer, our approach achieves quite notable performance gain on CoNLL03 (+3.26 F1) and Webpage (+4.26 F1).\n\nWe further want to refer the reviewer to our response to Q1, where our approach performs significantly better than the rivals, on another sequence labeling task (we did this experiment this week as per the request from the reviewers, so we didn’t perform any dense tuning ;) )\n\nAt last, despite that our approach did achieve SOTA performances on various benchmarks of DSNER, perhaps we humbly ask the reviewer not to attribute heavily to the empirical numbers of a method. This is because wholeheartedly, we believe that a method up to SOTA is not the sole aspect of a NeurIPS submission.\n\n**Q3. Comparisons with LLMs.**\n\n In our original submission, we provided a thorough discussion on how recent LLMs can help resolve the DSNER problem without human supervision. \n\nBelow, we summarize some takeaways:\n\n1. **The zero-shot performance of ChatGPT is far from resolving the NER task.** As shown in Table 3 (Appendix), zero-shot ChatGPT exhibits 66.47 F1-score on  CoNLL03, which legs even behind the conventional distantly-supervised NER methods like our proposed DesERT. Notably, this is also supported by recent works [C2], which find that ChatGPT significantly underperformed the fine-tuned small language models in many NLP applications.\n2. **Self-generated demonstration can improve the ChatGPT on NER.** The performance of ChatGPT on unsupervised NER can be alleviated by an in-context learning algorithm. Specifically, we generate demonstration sentences by ChatGPT-given entity labels. Such a strategy can improve the F1-score to 70.22 without any human supervision.\n3. **ChatGPT prediction can serve as a new source of distant supervision.** Though the ChatGPT prediction is not satisfactory for practical employment, we find it is a good source of distant supervision for distilling a small language model. As shown in Table 3 (Appendix), DesERT obtains 79.53 F1 on such ChatGPT distant labels. Moreover, we also tested a hybrid supervision paradigm that mixes ChatGPT supervision and conventional knowledge-base supervision. We find that DesERT is able to achieve better performance (88.08 F1) than merely using knowledge base supervision (86.75 F1). \n\nIn our response PDF submission, we expound upon our findings by presenting further experimental outcomes from the Wikigold dataset, which substantiate our assertions. As a result, **we firmly believe that DSNER remains a pivotal research pursuit in the LLM era**. Looking forward, we envision LLMs serving as a novel source of distant supervision, facilitating the distillation of distantly-supervised SLMs, which has enhanced accuracy and computational efficiency. We will move part of the discussions and add more supporting results to our main content.\n\n*[C2] Yejin Bang, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interaeictivity. CoRR, abs/2302.04023, 2023.225*\n\n**Q4. RoSTER is discussed in the related work but not included in the baseline comparisons. Is there a particular reason for that?**\n\nA. We apologize for this confusion. We closely followed the implementation and configurations of SCDL and BOND. Thus, all the baseline performances are from the corresponding papers (SCDL's baseline results copied BOND). In RoSTER, a different tagging scheme of the IO format (i.e., only distinguishing whether a token is part of an entity or not) is employed instead of the BIO scheme adopted by the other baselines. As a result, RoSTER is not included in the baseline comparisons to ensure the comparisons are fair."}, {"review_id": "ahhfhaEHY2", "reviewer": "Reviewer_NjrY", "summary": "The paper proposes a novel self-training framework, DesERT, for distantly-supervised named entity recognition (NER) that addresses two types of biases omitted by prior work: structural noise in distant labels and the inherent bias introduced by the self-training framework. The paper shows that the proposed framework establishes a new state-of-the-art performance, with an improvement of +2.22% average F1 score on five standardized benchmarking datasets.", "questions": "1.\tIs your backbone encoder (RoBERTa-base and DistilRoBERTa) consistent with the baseline method? Furthermore, are they comparable in parameter scale?\n2.\tI don’t understand the difference between the two peer networks in dual co-guessing mechanism. Are they only different with other in parameter initialization?\n", "limitations": "see weakness", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper identifies the structural noise in the distant supervision for NER. Specifically, the imbalance lies (1) between the distribution of different types of entity and (2) between named entity and non-entity. \n2.\tThe paper proposes a new framework for self-training when the pseudo label is noisy and sometimes unreliable. \n3.\tIn experiments, the proposed method accomplishes new SOTA on five benchmarks of NER.\n", "weaknesses": "1.\tThe framework seems a simple mixture of existing techniques and their application on the NER task. The two observed omitted bias of previous works resembles the data bias the training bias proposed in [1]. The worse-case cross entropy loss also inherits [1] without much task-specific adaptation. The teacher network and its exponential moving average seem from [2]. I suggest the authors clearly claim where their novelty lies in this work.\n2.\tThe writing is not very easy to follow. A high-level algorithm exhibiting the working flow of the proposed approach is suggested in the main document. In the current state, you just list all the techniques in each paragraph, and it is a little bit hard to get across the inner relationship. For example, there is a teacher network and a student network in Figure2 but the teacher network is not mentioned until implementation details.\n3.\tThe double-head pathway requires an additional head and thus may cause extra inference latency. Analysis and discussion on this should be included thus we can see whether the inference speed declines obviously or not.\n[1] Debiased Self-Training for Semi-Supervised Learning\n[2]Improving Distantly-Supervised Named Entity Recognition with Self-Collaborative Denoising Learning\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your insightful comments and suggestions! Our detailed responses are as follows.\n\n**Q1. The framework seems a simple mixture of existing techniques and their application on the NER task. The two observed omitted bias of previous works resembles the data bias the training bias proposed in [1]. The worse-case cross entropy loss also inherits [1] without much task-specific adaptation. The teacher network and its exponential moving average seem from [2]. I suggest the authors clearly claim where their novelty lies in this work.**\n\n A. We would like to defend the novelty of our work since the purpose of our task-splitting mechanism is not to simply and blindly propose a new architectural approach. \n\nRather, behind it, it manifests a significant issue of selection bias in the DSNER, that commonly exists but has not been uncovered before in the NER literature.\n\nOur approach is partially developed toward resolving this issue and delivered as a holistic DSNER system via decoupled learning and debiased training that establishes new SOTA performances.\n\nIndeed, we think that a simple and blindly-designed extension of existing techniques --- with no rigorous empirical findings and support --- wouldn’t have gotten us here.\n\nWe sincerely hope that the reviewer could reassess our paper following the evidence chain from empirical insights on the bias (Figure 1), holistic debiased DSNER framework (Section 4), to superior performance (Section 5). \n\n**Compared with [1] (WCE):** \n\nTechnically, we made the first attempt to apply the WCE loss in the distant supervision regime where the labeled/unlabeled set can be dynamic. In contrast, the original paper [Chen et al., 2022] studies the conventional semi-supervised learning problem with fixed datasets. \n\nFrom the theoretical standpoint, we also provide new theoretical insights in our Appendix to show that WCE does indeed shrinks the variances, which is blank in the literature. \n\n**Claims on the teacher network technique [2]**:\n\nSorry for confusing the reviewer. Actually, we didn't claim that the teacher-student network is our contribution since it originates from the SCDL framework. We simply put it in the Appendix to ensure the integrity of our work.\n\n**Q2. A high-level algorithm exhibiting the working flow of the proposed approach is suggested in the main document. There is a teacher network and a student network in Figure 2 but the teacher network is not mentioned until implementation details.**\n\n A. Thanks for the suggestion. We've provided the pseudo-code of DesERT in Appendix D. In fact, our core components like decoupled training, debiased SSL, and co-guessing are general-purpose components and can be attached to any existing selection&self-training-based DSNER algorithms. To obtain SOTA performance, we chose the recent SOTA algorithm SCDL as our base implementation, which introduces the teacher-student network and some other tricks. However, we wouldn't like to claim such components as our core novelty and thus, only report it in our Appendix. We'll polish our manuscript in the final version. \n\n**Q3. The inference speed declines of double head.** \n\n A. The double-head module is attached to the top of the backbone, and thus will not cause much extra inference latency. We conducted experiments to show the running time and the results are shown below,\n\n| Method      | Inference Testing Time (Seconds/Epoch) | Back Propagation Training Time (Seconds/Epoch) |\n| ----------- | -------------------------------------- | ---------------------------------------------- |\n| Single Head | 12.19                                  | 74.76                                          |\n| Double Head | 14.58                                  | 79.67                                          |\n\nIt can be shown that our double-head pathway brings negligible extra computation costs. \n\n**Q4. Is your backbone encoder (RoBERTa-base and DistilRoBERTa) consistent with the baseline method? Furthermore, are they comparable in parameter scale?**\n\n A. As we also stated in our responses to *Reviewer#WUVn*, our implementation and configurations directly follow SCDL [9] and BOND [10]. Therefore, our method has the same backbone, teacher-student architecture, and even mostly common parameters (the same learning rate, batch size, teacher updating coefficients, and so on). Therefore, the size of the parameters is also consistent with the baselines. \n\n**Q5. I don’t understand the difference between the two peer networks in dual co-guessing mechanism. Are they only different with other in parameter initialization?**\n\n A. In the beginning, they have different initial weights. As the training proceeds, the two peer networks may tend to fit different noisy patterns from data, especially when the noise is not well rectified. Therefore, an ensemble of label guessing can bring new opportunities for filtering clean samples and improving their robustness. "}, {"review_id": "0ENPCy2zGf", "reviewer": "Reviewer_qqbH", "summary": "The paper presents DesERT, a novel self-training framework for distant-supervised named-entity recognition (NER). It addresses the challenges of noisy distant labels and introduces two types of biases that impact performance. DesERT incorporates a dual-form self-training approach and a debiased module to improve sample selection and token representations. Experimental results demonstrate significant performance gains, surpassing existing methods on multiple benchmark datasets. \n\n\n", "questions": "The experiments of ChatGPT are very interesting and I would recommend the authors to put it in the main content instead of appendix.\n\n", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The problem of open-domain distantly supervised NER is important.\n\n2. The paper is well-written.\n\n3. The motivation of the paper is good. ", "weaknesses": "1. Some important related works are missing, namely:\n\n[1] Zhang, Wenkai, et al. \"De-biasing Distantly Supervised Named Entity Recognition via Causal Intervention.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.\n\n[2] Qu, Xiaoye, et al. \"Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble.\" arXiv preprint arXiv:2212.06522 (2022).\n\n[3] Jiang, Haoming, et al. \"Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data.\" Annual Meeting of the Association for Computational Linguistics. 2021.\n\nSpecifically, [1] shares a very similar motivation with this work for *debiasing* distantly supervised NER. Please discuss the key difference between these baselines.\n\n2. The method proposed in 4.3 is a direct application of [Chen et al., 2022], and the dual co-guessing framework is also a direct extension in [Liang et al., 2020] by adding two model branches. Overall, the authors need to justify the novelty of the proposed method. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your valuable comments and suggestions! Our detailed responses are as follows.\n\n**Q1. The method proposed in 4.3 is a direct application of [Chen et al., 2022], and the dual co-guessing framework is also a direct extension in [Liang et al., 2020] by adding two model branches. Overall, the authors need to justify the novelty of the proposed method.**\n\nA. We would like to defend the novelty of our work since the purpose of our task-splitting mechanism is not to simply and blindly propose a new architectural approach. \n\nRather, behind it, it manifests a significant issue of selection bias in the DSNER, that commonly exists but has not been uncovered before in the NER literature.\n\nOur approach is partially developed toward resolving this issue and delivered as a holistic DSNER system via decoupled learning and debiased training that establishes new SOTA performances.\n\nIndeed, we think that a simple and blindly-designed extension of existing techniques --- with no rigorous empirical findings and support --- wouldn’t have gotten us here.\n\nWe sincerely hope that the reviewer could reassess our paper following the evidence chain from empirical insights on bias (Figure 1), holistic debiased DSNER framework (Section 4), to superior performance (Section 5). \n\n**Compared with [Chen et al., 2022]:** \n\nTechnically, we made the first attempt to apply the WCE loss in the distant supervision regime where the labeled/unlabeled set can be dynamic. In contrast, the original paper [Chen et al., 2022] studies the conventional semi-supervised learning problem with fixed datasets. \n\nFrom the theoretical standpoint, we also provide new theoretical insights in our Appendix to show that WCE does indeed shrinks the variances, which is blank in the literature. \n\n**Compared with [Liang et al., 2020]**:\n\nIndeed, the high confidence selection is popular in weakly-supervised learning, e.g. FixMatch in semi-supervised learning. Our method extends it with an ensemble technique that further improves robustness. But, we have to say that this is not our core contribution. Instead, our work focuses on bringing new insights and findings on the bias problem in DSNER. \n\n\n\n**Q2. Some important related works are missing.**\n\nA. Thanks for your suggestions. We'll add the discussion on these related works in our revised version. We'd like to clarify that these works are quite different from ours. \n\n1. First, [1] is fundamentally different from our work. \n\n    (i)-**Different Bias Issues**: [1] particularly studies two **structural causal biases** caused by the dictionaries, including *inter-dictionary-bias* that overlooks out-of-dictionary entities and *intra-dictionary bias* that different dictionaries can cause different model behaviors. In contrast, our paper studies the selection biases in the current selection&self-training schema. \n\n    (ii)-**Different Technical Solutions**: [1] further introduce a structural causal model (SCM) for DSNER that mainly Backdoor Adjustment and Causal Invariance Regularizer, both come from a causal perspective. Instead, our method still follows the sample selection and self-training framework.\n\n    (iii)-**Performance Gaps**: [1] largely underperforms our DesERT on empirical performance, e.g. 81.54 vs 86.95 (ours) F1 scores on CoNLL03. \n\n2. We felt sorry for missing [2] because this AAAI'23 paper is officially published near the NeurIPS submission deadline. Technically, [2] and our paper shares similar selection and self-training frameworks. But, the core idea of our paper is rather different from [2] since our main novelty lies in the identification of two important biases in DSNER for the first time. Moreover, our DesERT algorithm enjoys better performance, e.g. 85.59 vs 86.95 (ours) F1 scores on CoNLL03 and 70.55 vs 72.73 (ours) F1 scores on Webpage.\n\n3. [3] studies a different DSNER setup, where a large set of weak labels and a small set of clean training data. From a technical standpoint, [3] adopts a continual pretraining framework that jointly trains with clean and noisy entities.\n\nThe different characteristics are summarized in the following table. We believe our work is quite different from these papers and brings new insights to the community.\n\n| Characteristics             | DesERT                           | [1]                     | [2]                      | [3]                    |\n| --------------------------- | -------------------------------- | ----------------------- | ------------------------ | ---------------------- |\n| Biased Problem              | Selection (Label) Biases | Causal Bias             | N/A                      | N/A                    |\n| Techniques                  | Selection&Self-Training          | Structural Causal Model | Selection&Self-Training  | Continual Pre-training |\n| Require Clean Training Data | No                               | No                      | No                       | Yes                    |\n| Performance                 | Best                             | Low                     | N/A (Different Datasets) | Good                   |\n\n**Q3. The experiments of ChatGPT are very interesting and I would recommend the authors to put it in the main content instead of appendix.**\n\nA. We are glad you enjoyed our new results on ChatGPT-based DSNER. In our revised version, we will move part of the discussions and add more supporting results to our main content."}]

Author rebuttal:
We sincerely appreciate all reviewers for their great efforts in providing constructive and valuable feedback. We are pleased that the reviewers find our work **well-written** (**R1,R3**), **theoretically solid** (**R1,R2**), and **experimentally comprehensive** (**R1,R2,R4,R5**). Moreover, we are more than encouraged that all the reviewers agree our work is **well motivated that brings some new findings on biases to improve the DSNER problem** (**R1,R2,R3,R5**). 

We are also happy that our experiments (in the Appendix) on recent LLMs like ChatGPT raises the interest of the reviewers (**R2,R3,R5**). 

We have addressed the reviewers’ comments and concerns in individual responses to each reviewer. We also submitted a response PDF that shows:

[1] Figure 1: (R1Q5) Confusion matrices of data split across the epochs

[2] Figure 2: (R1Q6) Confusion matrix for Wikigold

[3] Table 1: (R2Q2&R3Q3) Additional Results with ChatGPT Supervision

(As abbreviations, we refer to reviewers **WUVn** as R1, **WYfp** as R2, **Nukd** as R3, **NjrY** as R4 and **qqbH** as R5 respectively.)

Meta-review: The paper introduces a novel self-training framework called DesERT to address the challenges of distant supervision in NLP tasks, particularly named-entity recognition (NER). It identifies two types of biases in distant labels that previous work overlooked, leading to suboptimal performance in distant-supervised NER. DesERT adapts the sample-selection process to the structural bias in distant labels and incorporates a debiased module to enhance token representations, improving the quality of pseudo-labels. Extensive experiments demonstrate that DesERT achieves a significant +2.22 average F1 score improvement on five benchmark datasets, establishing a new state-of-the-art performance in distant-supervised NER. Reviewers generally find the work well-written, well motivated, and with sound experiments; and thus support the acceptance of the paper.
Decision: Accept (poster)