SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\{(x _ {i}, \hat {y} _ {i} ^ {(1)}) | \mathbb {I} (\hat {y} _ {i} ^ {(1)} = \hat {y} _ {i} ^ {(2)}) \wedge (\max (\pmb {p} _ {i} ^ {(1)}) > \tau) \wedge (\max (\pmb {p} _ {i} ^ {(2)}) > \tau) \}
$$
$\hat{y}_i^{(1)}$  and  $\hat{y}_i^{(2)}$  are predicted labels from two models,  $\pmb{p}_i^{(1)} = \mathrm{softmax}(f_i(\pmb{x};\theta_1))$  and  $\pmb{p}_i^{(2)}$  is same. When it comes to specific sub-datasets  $\mathcal{D}_b$  and  $\mathcal{D}_e$ , the predicted labels should be converted to respective counterparts, and probability is generated by the corresponding pathway.
Training loss. The overall training loss is defined as follows:
$$
\mathcal {L} = \mathcal {L} _ {b _ {-} c l s} + \mathcal {L} _ {e _ {-} c l s} + w * \mathcal {L} _ {e _ {-} w c e}
$$
Here  $w > 0$  is a weighting factor.  $\mathcal{L}_{b\_cls}$  and  $\mathcal{L}_{e\_cls}$  are the classification losses on the two predictive heads respectively.  $\mathcal{L}_{e\_wce}$  is the WCE loss on the entity pathway.
Table 1: Main results on five benchmark datasets measured by Precision (P), Recall (R), and F1 scores. We highlight the best overall performance for distant supervision in bold.
Methods	CoNLL03	OntoNotes5.0	Webpage	Wikigold	Twitter
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Fully-supervised methods
BiLSTM-CC	91.35	91.06	91.21	85.99	86.36	86.17	50.07	54.76	52.34	55.40	54.30	54.90	60.01	46.16	52.18
RoBERTa-base	89.14	91.10	90.11	84.59	87.88	86.20	66.29	79.73	72.39	85.33	87.56	86.43	51.76	52.63	52.19
Distantly-supervised methods
AutoNER	75.21	60.40	67.00	64.63	69.95	67.18	48.82	54.23	51.39	43.54	52.35	47.54	43.26	18.69	26.10
LRNT	79.91	61.87	69.74	67.36	68.02	67.69	46.70	48.83	47.74	45.60	46.84	46.21	46.94	15.98	23.84
Co-teaching+	86.04	68.74	76.42	66.63	69.32	67.95	61.65	55.41	58.36	55.23	49.26	52.08	51.67	42.66	46.73
JoCoR	83.65	69.69	76.04	66.74	68.74	67.73	62.14	58.78	60.42	51.48	51.23	51.35	49.40	45.59	47.42
NegSampling	80.17	77.72	78.93	64.59	72.39	68.26	70.16	58.78	63.97	49.49	55.35	52.26	50.25	44.95	47.45
BOND	82.05	80.92	81.48	67.14	69.61	68.35	67.37	64.19	65.74	53.44	68.58	60.07	53.16	43.76	48.01
SCDL	87.96	79.82	83.69	67.49	69.77	68.61	68.71	68.24	68.47	62.25	66.12	64.13	59.87	44.57	51.09
Ours*	86.23	87.28	86.75	66.38	72.08	69.11	71.52	72.97	72.24	60.77	68.10	64.23	56.44	48.38	52.10
Ours*(finetune)	86.41	87.49	86.95	66.63	71.92	69.17	72.48	72.97	72.73	62.87	69.42	65.99	57.65	47.80	52.26
Post-hoc Entity Pathway Finetuning. In previous steps, we train the double-head pathway separately as both pathways can be unreliable due to the label noise. Over the course of training, we observe the binary pathway learns well thanks to abundant training samples. On the contrary, the entity pathway still faces a rather challenging condition caused by inefficient samples, label noise, and more categories. To relieve this problem, we propose to post-hoc finetune the entity pathway by constructing a more refined dataset via the binary pathway. In specific, after the training procedure, we freeze the binary pathway  $h_b$  and collect a new clean entity dataset by,
$$
\mathcal {D} _ {e} ^ {l} = \left\{\left(x _ {i}, \hat {y} _ {i} ^ {e}\right) | \mathbb {I} \left(\hat {y} _ {i} ^ {b} = 1\right) \wedge \mathbb {I} \left(\hat {y} _ {i} ^ {e} = \tilde {y} _ {i} ^ {e}\right) \right\}
$$
That is, while still anticipating selecting clean samples, we remove those tokens predicted as nonentity by  $h_b$ , which produces a more refined clean sample set than before. Lastly, we finetune the entity pathway with this clean dataset for a few epochs to boost the performance.
In this section, we present our main experimental results to show the effectiveness of our method. More empirical setups and results are reported in Appendix C.
Dataset. We evaluate our framework on five widely-used named entity recognition benchmark datasets in the English language: (1) CoNLL03 [34] is collected CoNLL 2003 Shared Task, consisting of 1,393 English news articles and four entity types; (2) OntoNotes5.0 [35] contains 1.6 million words annotated by 18 entity types from multiple domains, including broadcast conversation, Web data, and P2.5 data; (3) Webpage [36] collects 783 entities belonging to the four types like CoNLL03, which are from 20 webpages including personal, academic, and computer-science conference homepages. (4) Wikigold [37] comprise 39,007 Wikipedia article tokens from a 2008 English dump with four CoNLL03 named entity tags; (5) Twitter [38] is an open-domain NER dataset coming from the WNUT 2016 NER shared task, which consists of 2,400 tweets (a total 34k tokens) with 10 entity tags. To obtain a distant supervision scheme, we follow previous works [10] to re- annotate all the training corpus by matching entity types in external knowledge bases including Wikidata corpus and gazetteers extracted from multiple online sources.
Baselines. We compare the proposed framework with a total of seven current distantly-supervised methods: (1) AutoNER [7] trains a fuzzy LSTM-CRF network to handle tokens with multiple possible labels, using a Tie or Break scheme; (2) LRNT [17] employs Partial-CRF for sequence labeling and training with high-quality sentences given by high annotation confidence and coverage; (3) Co-teaching+ [39] is a robust sample selection-based algorithm, which only updates the dual networks by prediction disagreement data; (4) JoCoR [40] adopts dual networks to calculate a
(a) PER-F1
(b) LOC-F1
(c) ORG-F1
Figure 3: The F1-scores of DesERT with/without double-head pathway on four entity types, which have  $11.1\mathrm{k} / 8.3\mathrm{k} / 10.0\mathrm{k} / 4.6\mathrm{k}$  tokens respectively (from left to right).
(d) MISC-F1
joint loss with co-regularization; (5) NegSampling [41] handles incomplete annotations by negative sampling; (6) BOND [10] is a self-training paradigm that selects reliable clean tokens for training by a teacher-student network with early stopping; (7) SCDL [9] jointly trains two teacher-student networks to iteratively perform label refinery in a mutually beneficial manner. Besides, we select two fully-supervised methods trained with clean annotations to show the performance upper bound: (1) BiLSTM-CNN-CRF (BiLSTM-CC) [3] adopts bi-directional LSTM with character-level CNN to generate token embeddings, followed by a CRF layer to predict the most likely label sequence; (2) RoBERTa-base [5] adopts a pre-trained RoBERTa-base model as the encoder.
Implementation details. We adopt two pre-trained language models RoBERTa-base and DistilRoBERTa as the backbone. Inspired by [10], we further maintain a teacher network that is an exponential moving average of the main model. Thereafter, the data split, pseudo-labeling, loss calculation, and co-guessing operators are done by the prediction of the teacher network; see Appendix C.4 for detailed descriptions. We follow [10, 9] to tune most hyperparameters. Specifically, we train the networks for 50 epochs with a few epochs of warm-up, followed by 2 epochs of finetuning. The training batch size is set as 16 on four datasets, except 32 on OntoNotes5.0. The learning rate is fixed as  $\{1e^{-5}, 2e^{-5}, 1e^{-5}, 1e^{-5}, 2e^{-5}\}$  for these datasets respectively. The confidence threshold parameter  $\tau$  is tuned by 0.9 for Wikigold while 0.95 for others. The co-guessing is performed from the  $k$ -th epoch, which we set to  $\{6, 40, 35, 30, 30\}$  respectively. For finetuning, the learning rate is one-tenth of the original one. Once the training ends, we choose the first student model for predictions as the final results. We run all experiments for three random trials and record the mean results.
We measure the performance of all methods by Precision (P), Recall (R), and F1 scores. As shown in Table 1, our proposed method DesERT achieves state-of-the-art performance compared with other baselines on five benchmark datasets. Specifically, on the CoNLL03 and Webpage datasets, DesERT outperforms the best baseline methods by  $3.26\%$  and  $4.26\%$  in terms of F1 score. The performance is even close to the best fully-supervised baseline (i.e. RoBERTa-base) on Webpage and Twitter, which suggests better adaptation to distant supervision. In addition, DesERT displays significantly higher recall on almost all datasets, demonstrating a strong ability to identify entities from text. It also maintains moderate precision so as to attain extraordinary F1 scores. Finally, the model after post-hoc finetuning shows obviously better performance. It demonstrates that a well-trained binary head can help the fine-grained head to better distinguish entity types, proving the superiority of the task decomposition formula.
To explore the effectiveness of each component in DesERT, we conduct an ablation study on the CoNLL03 dataset without the finetuning stage. As shown in Table 9, we test the following cases: (1) remove the WCE loss; (2) do not perform dual networks co-guessing; (3) replace the double-head pathway with a single-head pathway.
It can be seen from the ablations that the removal of any component leads to performance degradation, while the model can easily obtain the best F1 score under the combined effects of three components.
(a) Analysis on WCE
(b) Analysis on Co-Guessing
Figure 4: (a) The F1 curves of DesERT with/without WCE loss. (b) The distribution of the true labels and selected labels with/without co-guessing. (c) The parameter study of different confidence thresholds  $\tau$  on the CoNLL03 dataset.
(c) Analysis on  $\tau$
Specifically, discarding the co-guessing mechanism will cause the maximum performance descent of  $3.31\%$  on F1 scores, which indicates the benefits of using co-guessing to correct distant labels for training. As the core mechanism of DesERT, the double-head pathway can also improve the F1 score by  $2.21\%$  compared with only a single-head pathway, indicating that the idea of task-data decomposition is more suitable for distant supervision. Finally, WCE loss provides an effective performance gain as a debiased add-on. We additionally note that Precision and Recall metrics are also optimal, which proves the model can accurately separate entity and non-entity tokens as well as identify relatively complete entities.
5.4 Further Analysis
Efficacy of decoupled learning. To validate the efficacy of decoupled learning, we further test the F1 score of each entity type in two cases on the CoNLL03. First, by comparing the F1 scores of four entity classes shown in Figure 3, it can be seen that the conventional single-head model demonstrates inferior results on all entity types. More importantly, we found that the two models show relatively close behavior in identifying entity classes with sample-rich tokens like PER. In contrast, the performance gap in minority categories is large.
For example, the F1 score on the MISC entities of the double-head pathway is  $7.11\%$  higher than that of the single-head pathway (Figure 3 (d)), and the gap for LOC entities is  $2.91\%$  (Figure 3 (b)). Therefore, the design of a double-head pathway or the idea of task decomposition can well adapt to DSNER and demonstrate stronger entity recognition capability for minority entity classes.
Efficacy of debiased self-training. We study the effectiveness of WCE loss for the DesERT framework. We test the cases with and without adopting WCE loss on the CoNLL03 dataset, and the results are shown in Figure 4 (a). It can be found that WCE loss is more likely to serve as an effective add-on in DesERT. It should be noted that WCE displays high-performance growth in early epochs, which alleviates the model bias towards unpurified labels in the early stage.
Efficacy of co-guessing. Besides, we explore the ability of the co-guessing mechanism to select extra tokens. In Figure 4 (b), we compare the selected token distribution with and without the co-guessing operation. Without co-guessing, a huge gap occurs between the selected tokens per class and the real distribution, e.g., more than 3,000 tokens are not selected on the already scarce class MISC. The reason is that we enforce the clean tokens to have a matched prediction on given labels. This severely impairs the recognition performance of minority entities. With our co-guessing mechanism, we observe the clean sample set is significantly enlarged and the distribution of selected entity label distribution also approaches that of true labels. Combined with the fact in Table 9 that co-guessing does indeed improve performance, the effectiveness of co-guessing is clearly validated.
Table 2: Ablation study on CoNLL03 dataset.
Ablations	CoNLL03
P	R	F1
Ours*	86.23	87.28	86.75
w/o WCE	85.04	85.98	85.50
w/o co-guessing	82.34	84.57	83.44
w/o double-head	84.18	84.90	84.54
Table 3: Performance of DesERT with different sources of distant labels on CoNLL03.
Supervision	Unsupervised	ChatGPT Labels	KB Labels	Hybrid Labels
Model	ChatGPT	ChatGPT-A	SCDL	DesERT	SCDL	DesERT	SCDL*	DesERT*
Precision	68.95	79.11	68.39	81.91	87.96	86.23	83.87	87.24
Recall	64.16	63.13	72.74	77.38	79.82	87.28	85.50	88.93
F1	66.47	70.22	70.50	79.58	83.69	86.75	84.67	88.08
Efficacy of threshold parameter  $\tau$ . We study the effect of the important hyperparameter  $\tau$  used in clean sample selection and dual networks co-guessing on the CoNLL03 dataset. Specifically, we vary the value of  $\tau$  in the range 0.7 to 0.99, and other parameters are kept as default. Then we validate DesERT's performance by applying different  $\tau$  values, as shown in Figure 4 (c). We can see that the model's performance is relatively insensitive in the  $\tau$  varies from 0.95 to 0.99, but drops a lot when the  $\tau$  value is lower than 0.95. And DesERT achieves the best results with  $\tau$  fixed at 0.95 on the CoNLL03. In practice, we suggest setting a high confidence threshold  $\tau$ .
5.5 Distant Supervision from Large Language Models
Recently, large language models (LLMs), including GPT-3 [42], ChatGPT, and  $\mathrm{GPT - 4^3}$  have largely revolutionized the NLP landscape. Thanks to their emerging abilities like in-context learning (ICL) [43] and chain-of-thought [44], LLMs demonstrate remarkable zero-shot learning performance in a wide range of downstream NLP tasks. Despite the promise, some recent studies [45] have shown that LLMs are still legs behind the fine-tuned small language models in many NLP applications including NER. To deal with this problem, we extend the DSNER formulation and design a novel in-context learning algorithm that exploits self-generated text-tag pairs to generate distant labels. Moreover, we modify our original algorithms to fully use hybrid labels including ChatGPT-generated labels and original knowledge-base-generated labels (KB labels). We refer the readers to Appendix C.1 for a detailed description of our prompt design, data generation process, and methodology.
Empirically, we provide a series of experiments to verify the feasibility of utilizing ChatGPT for distant supervision. In particular, we compare DesERT with three algorithms: (1) ChatGPT: we employ the ChatGPT model to produce zero-shot tagging on testing data; (2) ChatGPT-A: we employ ChatGPT to generate a set of text-tag pairs and use it for few-shot ICL on testing data; (3) SCDL: the most competitive baseline in our main Table. Table 4 lists our full results, we have the following conclusions: (1) ChatGPT does indeed demonstrate inferior results even compared with distantly-supervised SLMs; (2) simply training current best-performing DSNER methods on ChatGPT supervision can be inferior since they are particularly designed for KB labels; (3) ChatGPT can be a complementary source of distant supervision to existing KB labels, and our modified DesERT (dubbed DesERT*) still obtains best performance when handling hybrid labels. Overall, we firmly believe that DSNER remains a pivotal research pursuit in the LLMs era. Looking forward, we envision LLMs serving as a novel source of distant supervision, facilitating the distillation of distantly-supervised SLMs, which has enhanced accuracy and computational efficiency.
In this paper, we propose a novel framework DesERT for NER under distant supervision. We first find that existing self-training methods suffer from two types of biases, i.e., a skewed data distribution in distant supervision and an inherent bias to assign erroneous pseudo-labels. To mitigate both biases, we propose a novel task/data taxonomy to decompose the original task, alongside a debiased self-training to improve the quality of pseudo-labels. Extensive experiments on other standard benchmarks clearly validate the effectiveness of DesERT. Furthermore, we supplement a new ChatGPT-based DSNER benchmark and show that DesERT still obtains the best performance. We hope our work can draw more attention from the community to the bias issues in the distantly-supervised NER tasks.
3https://openai.com/blog/chatgpt
This work is majorly supported by the National Key Research and Development Program of China (No. 2022YFB3304100), and in part by the NSFC under Grants (No. 62206247). Junbo Zhao also thanks the sponsorship by CAAI-Huawei Open Fund and the Fundamental Research Funds for the Central Universities (No. 226-2022-00028).
[41] Yangming Li, Lemao Liu, and Shuming Shi. Empirical analysis of unlabeled entity problem in named entity recognition. In ICLR. OpenReview.net, 2021.
[42] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[43] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022.
[44] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
[45] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. CoRR, abs/2302.04023, 2023.
[46] Yangming Li, Han Li, Kaisheng Yao, and Xiaolong Li. Handling rare entities for neural sequence labeling. In ACL. Association for Computational Linguistics, 2020.
[47] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Protrans: Towards cracking the language of life's code through self-supervised deep learning and high performance computing. CoRR, abs/2007.06225, 2020.
[48] Junyuan Xie, Ross B. Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, volume 48 of JMLR Workshop and Conference Proceedings, pages 478-487. JMLR.org, 2016.
[49] Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR. OpenReview.net, 2020.
A Dataset-Task Taxonomy
A.1 Dataset Taxonomy
We provide Figure 5 to show the dataset taxonomy in the overall training workflow. We first split the initial dataset  $\mathcal{D}$  into two sub-datasets  $\mathcal{D}_b$  and  $\mathcal{D}_e$  through decoupled learning to mitigate the distributional bias (first level split). Next, to process respective noise in  $\mathcal{D}_b$  and  $\mathcal{D}_e$ , we conduct a selection and self-training framework. We divide the two parts into the clean set and noisy set by clean token selection (second level split), then perform a standard classification on each clean set. As for noisy token sets, we adopt debiased self-training to yield pseudo-labels for further training.
Figure 5: Illustration of dataset taxonomy.
A.2 Task Taxonomy
DesERT modifies the basic NER model architecture with the double-head pathway, yet reserves a shared pre-trained language model encoder such as RoBERTa-base denoted by  $\phi$ . Given any sentence  $\mathbf{x}$  with its binarized labels  $\mathbf{y}^b$  and entity type labels  $\mathbf{y}^e$ ,  $(\mathbf{x},\mathbf{y}^b)\in \mathcal{D}^b$  and  $(\mathbf{x},\mathbf{y}^e)\in \mathcal{D}^e$ .  $\mathbf{x}$  is first fed to the PLM encoder  $\phi$  and we take the last hidden layer output of  $\phi$  as finally embeddings  $\phi (\mathbf{x})$ . Then the double-head pathway  $h_b$  and  $h_e$  take  $\phi (\mathbf{x})$  as input to yield respective predictions. The binary pathway  $h_b$  generates the probability of being entity tokens,  $\pmb{p}^{b} = \mathrm{sigmoid}(h_{b}\circ \phi (\pmb {x})) = [p_{1}^{b},\dots,p_{n}^{b}]$ , then take  $\pmb{p}^b >0.5$  as predicted binary labels  $\hat{\pmb{y}}^b$ . While the entity pathway  $h_e$  offers fine-grained entity type probabilities  $\pmb{p}^{e} = \mathrm{softmax}(h_{e}\circ \phi (\pmb {x})) = [\pmb{p}_{1}^{e},\dots,\pmb{p}_{n}^{e}]$ , where each  $\pmb{p}_i^e$  has  $K$  entries. We take  $\arg \max (\pmb {p}^e)$  as predicted entity type labels  $\hat{\pmb{y}}^e$ , and note that non-entity tokens are tagged with invalid labels. Finally, any standard classification loss can be calculated on the double-head pathway. We also refer the reader to Figure 2 for visualized illustration.
B Theoretical Insights of Debiased Semi-supervised Learning
Though there is no available theory on why the worst cross-entropy (WCE) [32] works, we would like to provide the following (relatively) theoretical insights that may help the readers to perceive our approach better.
Notably, the self-training bias is mainly caused by noisy tokens approaching the decision boundaries, whose pseudo-labels keep changing. To this end, we optimize WCE to learn compact token clusters for reducing wrong pseudo-label assignments. To see this, we start from the following simplified example to show that WCE does indeed concentrate the token representation.
Assumption. Consider a binary classification problem, as the simplest form of DSNER. Denote the input variable by  $X$  and the output variable by  $Y$  from binary labels  $\{+1, -1\}$ . The labeled data  $\mathcal{D}^l$
are sampled from  $X|Y = +1 \sim U(\mathcal{B}(u,r))$  and  $X|Y = -1 \sim U(\mathcal{B}(v,r))$ . Here  $\mathcal{B}(u,r)$  denotes a spherical ball with center  $u$  and radius  $r$ .  $U$  denotes uniform distributions. The unlabeled sample  $\mathcal{D}^u$  are all from  $Y = +1$  but uniformly distributed inside the  $\mathcal{B}((u + v) / 2, r')$ . Three balls have no intersection. Finally, we assume the maximum margin classifier is used, which is a hard proxy of the cross-entropy loss.
Derivation Sketch. At first glance, it is obvious that the optimal classifier on  $D^l$  is  $f = (u + v) / 2$  which misclassifies half of the examples in  $D^u$ . The worst classifier, however, amounts to be  $f^w = (u + v) / 2 - r'$  which perfectly classifies  $D^l$  but possesses the most side-way decision boundary. Next, we optimize the feature extractor  $\phi$  to match the worst decision boundary by  $\min_{\phi} L_U(y, f^w)$ . This is to say, with ideally known labels, the unlabeled ball  $\mathcal{B}((u + v) / 2, r')$  converges to  $\mathcal{B}(f = (u + v) / 2 - r', r')$ . With full samples  $X|Y = +1$  getting closer, the classifier achieves better generalization with compact clusters and low-entropy decision boundaries.
In practice, since the true labels are unknown, we use pseudo-labels as a proxy since most unlabeled data are assigned true labels. So, the representation will be partially concentrated.