SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Empirical Covariance. We conduct experiments on CoNLL03 to show the covariance of data to their class centers. When DesERT is run without WCE, the average covariance amongst classes is 0.0085. With WCE, the average covariance amongst classes becomes 0.0056. Thereafter, we can conclude that WCE indeed concentrates the tokens and mitigates the self-training bias.
C Additional Experimental Setups and Results
In what follows, we show more experimental details and results. In section C.1, we report more empirical results, including an interesting series of experiments where additional distant supervision comes from large language models like ChatGPT. In section C.3, we provide more details on our experiments and implementation.
C.1 Results with Additional Distant Supervision from ChatGPT
Recently, large language models (LLMs), including GPT-3 [42], ChatGPT, and  $\mathrm{GPT - 4^4}$ , have largely revolutionized the NLP landscape. Thanks to their emerging abilities like in-context learning (ICL) [43] and chain-of-thought [44], LLMs demonstrate remarkable zero-shot learning performance in a wide range of downstream NLP tasks. Despite the promise, some recent studies [45] have shown that LLMs are still legs behind the fine-tuned small language models in many NLP applications.
Motivated by this, we conduct experiments to show the zero-shot performance of ChatGPT on the NER problem. In Table 4, we observe that ChatGPT does indeed demonstrate inferior results even compared with distantly-supervised SLMs. Therefore, a question arises: how can LLMs better support NER with minimal human annotation? Among the numerous potential solutions, we propose a natural extension of the conventional distantly-supervised NER problem. This extension considers LLM's predictions on the training set as additional distant supervision. To achieve this, we present a new tagging scheme for NER and modify our algorithm to accommodate multiple sources of distant supervision. Subsequently, we provide a detailed elaboration on the aforementioned aspects.
C.1.1 Tagging by Auto-Demonstration
Instead of directly performing zero-shot testing, we introduce a new DSNER paradigm that generates distant labels by LLMs for unsupervised training data to improve the SLMs' performance. To generate distant labels, a straightforward solution is to send the raw training texts to the LLMs and ask them to output all the entities along with their types. However, we find such a naive strategy fails to achieve satisfactory NER performance. To mitigate this problem, we design a novel in-context learning algorithm that exploits self-generated text-tag pairs to guide the tagging process.
Automatic Text-Tag Pair Generation. Our ultimate goal is to perform few-shot in-context learning that better guides the LLMs to locate the entities and output their types. To achieve this,
4https://openai.com/blog/chatgpt
Prompted Input: You are required to produce 100 English sentences with annotations for the task of named entity recognition on the CONLL dataset. These samples will be used as prompt examples for the GPT model. There are four kinds of entity types:
'PER' for person, 'LOC' for location, 'ORG' for organization, and 'MISC' for other miscellaneous specific entities (e.g.,specific events, nationalities, products, or works of art..). Entity Types Def
The task is to classify the named entities in the given sentence as as one of the mentioned entity types. For example, 10 of the unlabeled samples in CONLL dataset are as follows:
[EU rejects German call to boycott British lamb.].
Unsupervised Demo
Now you are required to mimic the style of these samples and produce 100 English labeled examples on CONLL dataset. Please ensure that only the mentioned entities are included. Do not generate samples that are too similar. It is important to maximize both the diversity of samples and the accuracy of labeling. The output format should be in this form, each sample is output in a single line:
[(example 1) | (entity content 1, entity type 1); (entity content 2, entity type 2)], (entity content 1, entity type 1); ...
LLMs: [Jack decided to move to Paris for his new job] I (Jack, PER), (Paris, LOC)[The World Health
Organization has declared a global pandemic.] I ...
Generated Samples
Figure 6: An example prompt for automatic demonstration data generation.
we may assume a set of demonstration text-tag pair samples are available. Besides, this set ought to be diverse and representative enough to guide in-context learning. However, the training samples are unsupervised and cannot be utilized directly. To address this problem, we propose to generate sentences by ChatGPT itself, while ensuring the diversity of the generated results to cover all entity types. Additionally, we randomly retrieve unlabeled samples from the training set and employ ChatGPT to automatically generate a comprehensive set of sentences, including their corresponding entity tags. An example prompt is shown in Figure 6.
Few-Shot In-Context Learning for NER Tagging. After that, we ask the ChatGPT to tag the whole training set by using its self-generated demonstrations. However, directly feeding all the generated samples for ICL may exceed contextual limits and incur high computational costs. To remedy this problem, before feeding the true query sentence, we retrieve the top- $k$  generated samples by the cosine similarity. Empirically, we exploit the BERT embedding for similarity calculation, which performs generally better than ChatGPT embedding. Finally, we instruct ChatGPT to output the NER tags for the entire unsupervised training set. An example prompt is shown in Figure 7.
C.1.2 Modification of DesERT for Multi-Source Distant Labels
One may directly use the ChatGPT labels to train the DSNER models. However, the original knowledge base-driven distant supervision is also a free lunch for DSNER. and can be further incorporated. Notably, such a hybrid distant label from multiple sources problem has never been touched in the NER community. Fortunately, our DesERT algorithm can well address such hybrid labels with only a few modifications. Assume we are given knowledge base-driven labels  $y_{kb}$  (KB Labels) and ChatGPT Labels  $y_{cg}$  for a token  $x$ , where we slightly abuse the subscript to distinguish these two labels. To warm up the model, we calculate a mean soft label by,
$$
\hat {y} ^ {\text {m e a n}} = \left(\operatorname {O n e H o t} \left(y ^ {k b}\right) + \operatorname {O n e H o t} \left(y ^ {c g}\right)\right) / 2
$$
Therefore, the model fits equal confidence on these two labels when a token receives two disagreed labels. After that, we develop a modified selection protocol. Take the binary head as an example, we receive a set of transformed distant labels  $\tilde{Y}^b = \{\tilde{y}_{kb}^b,\tilde{y}_{cg}^b\}$  for each token and then perform the token selection by,
$$
\mathcal {D} _ {b} ^ {l} = \left\{\left(x, \tilde {Y} ^ {b}\right) \mid \mathbb {I} \left(\hat {y} ^ {b} \in \tilde {Y} ^ {b}\right) \wedge \left(\max  \left(p ^ {b}, 1 - p ^ {b}\right) > \tau\right) \right\}
$$
In other words, we regard either distant label as a candidate of the ground truth. Once there is one label in this label set that exhibits high confidence, we regard it as a clean token. Finally, we run the
Prompted Input: You are a helpful assistant for the task of named entity recognition in a sentence. You reply with brief, to-the-point answers with no elaboration as truthfully as possible. There are four known entity types: 'PER' which refers to person, 'LOC' which refers to location, 'ORG' which refers to organization, 'MISC' which refers to other specific miscellaneous entities (e.g., specific events, nationalities, products, or works of art.). When the name of the country is encountered, it can be recognized as a LOC (location) entity.
User: Given a sentence: 'Marcos Ondruska ( South Africa ) vs. Felix Mantilla ( Spain )'. Which entities may be included in the given sentence, please answer in a single line with the form (entity1 content, entity1 type);(entity2 content, entity2 type);(entity3 content, entity3 type);... . Please only output the token that you are very sure to be entity. If there is no entity in the given sentence, reply '(no entity) Assistant: (Marcos Ondruska, PER);(South Africa, LOC);( Felix Mantilla, PER);(Spain, LOC)
LLMs: (entity1 content, entity1 type);(entity2 content, entity2 type);(entity3 content, entity3 type);...
Figure 7: An example prompt for few-shot in-context tagging for a query sentence. The demonstrations are automatically generated and selected.
DesERT algorithm without any further modification. Notably, these can be easily extended to more sources of distant supervision, e.g., when there is more than one LLM available.
C.1.3 Experimental Results with ChatGPT Supervision
Our experiments are conducted on the CoNLL03 dataset. Specifically, we generate a total of 100 automatically generated samples with the help of 10 demonstration training samples. When tagging training data, we select the 10 most similar generated samples for ICL. In Figure 8, we plot the confusion matrix of the ChatGPT labels. It can be observed that ChatGPT
supervision demonstrates similar trends to KB labels and is still biased. Nevertheless, ChatGPT labels have three main characteristics: (1) they classify far more non-entity
tokens as an entity; (2) they produce more balanced clean tokens; (3) the confusion patterns on fine-grained entity types are different than KB labels.
Figure 8: Left: Confusion matrix of true labels and ChatGPT labels on CoNLL03. Right: The confusion matrix displays noise among true entity-type labels in ChatGPT labels.
In Table 4, we report the results of DesERT when it faces different sources of distant supervision signals. In particular, we compare DesERT with three baselines: (1) ChatGPT: we employ the ChatGPT model to produce zero-shot tagging on testing data; (2) ChatGPT-A: we employ ChatGPT to generate a set of text-tag pairs and use it for few-shot ICL on testing data; (3) SCDL: the most competitive baseline in our main Table. From Table 4, we have the following observation:
- ChatGPT-A is much better than vanilla ChatGPT, verifying the superiority of our automatic demonstration process. But, ChatGPT and ChatGPT-A underperform other DSNER algorithms on the testing set. Though ChatGPT is a wonderful general-purpose LLM, we may draw the same conclusion as [45] that fine-tuned SLMs still play an important role in NLP.
- Given ChatGPT labels, both SCDL and DesERT underperform their counterparts when supervised by KB labels. We postulate that current DSNER algorithms are particularly designed for KB-based supervision and thus can not fully handle such new sources of labels.
Table 4: Performance of DesERT with different sources of distant labels on CoNLL03.
Supervision	Unsupervised	ChatGPT Labels	KB Labels	Hybrid Labels
Model	ChatGPT	ChatGPT-A	SCDL	DesERT	SCDL	DesERT	SCDL*	DesERT*
Precision	68.95	79.11	68.39	81.91	87.96	86.23	83.87	87.24
Recall	64.16	63.13	72.74	77.38	79.82	87.28	85.50	88.93
F1	66.47	70.22	70.50	79.58	83.69	86.75	84.67	88.08
- Our proposed DesERT algorithm consistently outperforms baselines on different supervised signals. In particular, when trained with hybrid labels, the modified DesERT (DesERT*) improves the KB label-trained counterpart by  $+1.33$  F1 score. It suggests that distant supervision from LLMs does indeed brings helpful information for the DSNER task.
In summary, our work makes the first attempt to employ LLMs to generate distant supervision. Moreover, our DesERT algorithm can be easily extended to learn from multi-source distant labels and demonstrates improved performance.
C.2 More Experimental Results
Table 5: The sensitivity of  $k$  without fine-tuning.
Datasets	k=15	20	25	30	35	40
CoNLL03	86.72	86.52	86.78	86.82	86.48	86.18
Twitter	52.22	52.15	52.21	52.10	52.44	51.54
C.2.1 Sensitivity of  $k$
When tuning the co-guessing epoch  $k$ , we find that setting  $k = 6$  has led to good performance for CoNLL03. For other datasets, we found that it is proper to tune it among  $k \in \{30, 35, 40\}$ . In Table 5, we additionally present the parameter sensitivity analysis of  $k$ . It can be observed that DesERT is not very sensitive to the  $k$  value.
Ablations	CoNLL03	OntoNotes5.0	Webpage	Wikigold	Twitter
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
w/o Binary WCE	86.23	87.28	86.75	66.38	72.08	69.11	71.52	72.97	72.24	60.77	68.10	64.23	56.44	48.38	52.10
w/ Binary WCE	84.29	86.38	85.32	66.43	72.14	69.17	70.78	73.65	72.18	59.64	67.34	63.25	56.32	48.06	51.86
Table 6: Performance of DesERT with binary WCE on five benchmark datasets measured by Precision (P), Recall (R), and F1 scores.
C.2.2 The Effect of Debiased Training on the Binary Head
In this section, we further investigate whether the binary head can benefit from the debiased self-training procedure. In particular, we optimize the worst-case cross-entropy loss  $\mathcal{L}_{b\_wce}$  on the binary head pathway as well. The final objective is,
$$
\mathcal {L} = \mathcal {L} _ {b _ {-} c l s} + w * \mathcal {L} _ {b _ {-} w c e} + \mathcal {L} _ {e _ {-} c l s} + w * \mathcal {L} _ {e _ {-} w c e}
$$
In Table 6, we can observe that training with binary WCE generally lowers the F1 scores on benchmark datasets. We speculate that the reason is the binary classification head does not have adequate categorical information for training the representation. Moreover, if we apply WCE on the binary head, tokens from different fine-grained entities might be condensed into a large cluster. This may hinder the model of distinguishing different fine-grained entity types.
C.2.3 The Standard Deviation of Main Results
Our implementation and configurations closely adhere to the codes of existing algorithms [9]. Consequently, our baseline results align with those presented in the respective source papers. In
Table 7: The standard deviation of DesERT on benchmark datasets.
Methods	CoNLL03	OntoNotes5.0	Webpage	Wikigold	Twitter
DesERT	0.22	0.43	0.40	0.46	0.32
DesERT (finetune)	0.17	0.20	0.22	0.30	0.17
Table 8: The standard deviation of DesERT on benchmark datasets.
Methods	SNIPS (20% noise)	SNIPS (30% noise)	ATIS (20% noise)	ATIS (30% noise)
DesERT	94.29	93.51	95.15	94.73
RoBERTa	85.03	77.09	90.89	90.05
order to maintain uniformity in formatting, we have refrained from reporting the variance, due to the absence of such information in the cited references. In Table 7, we present the standard deviation values corresponding to our initial results. It can be shown that our variances are not substantial. In view of the notable performance gap (average  $+2.22$  F1), we believe that our algorithm has shown statistically significant improvement.
C.2.4 Experiments on Other Sequential Labeling Tasks
In our main content, we discussed the superiority of DesERT on the DSNER task. However, our method is actually a generic framework and can be applied to other sequential labeling tasks. To see this, we follow [46] on the sequence labeling benchmarks SNIPS and ATIS in a quest to make the comparison fair. We followed the literature on noisy label learning and injected  $20\%$  and  $30\%$  synthetic noise. The results in F1 are shown in Table 8. It says that DesERT performs way better than its counterparts.
Further, we conducted a simple experiment on the bioinformatics dataset FLIP that predicts the second structures of the proteins [47]. We injected  $20\%$  noise into synthetic datasets. Owing to the time limits, we adapted the decoupled training and debiased SSL modules to the benchmark from ProtTrans. In particular, we used a 30-layered ProtBERT model as the backbone and only fine-tuned the top 5 layers due to
Table 9: Valid accuracy on the FLIP dataset.
Methods	Accuracy
ProtBERT (Supervised)	80.83
ProtBERT (20% noise)	69.03
DesERT (20% noise)	70.38
limited computation sources. We found that ProtBERT is quite robust, and thus, we only finetune its top 5 layers. However, DesERT still shows some benefits in improving the robustness. These results prove the superiority of DesERT in handling weakly-supervised sequential tagging tasks.
C.3 More Implementation Details
In this section, we provide more experimental details for a better understanding of our training process and also to ensure the reproducibility.
C.3.1 Computation Resources
All experiments are conducted on a workstation with 8 NVIDIA RTX A6000 GPUs. It takes about  $\{8,72,0.5,1,2\}$  hours for training on five benchmarking datasets (ordered as in Table 1) with one single GPU. We adopt the Huggingface Transformer library for the RoBERTa-base (125M parameters) and DistilRoBERTa-base (66M parameters) models: https://huggingface.co/transformers/. We run all the experiments three times and report the mean results.
C.3.2 Datasets Details
The data statistics of five benchmarking NER datasets are shown in Table 10.
Table 10: The statistics of five datasets, show the number of entity types and the number of sentences in the Train/Dev/Test set.
Dataset	Types	Train	Dev	Test
CoNLL03	4	14,041	3,250	3,453
OntoNotes5.0	18	115,812	15,680	12,217
Webpage	4	385	99	135
Wikigold	4	1,142	280	274
Twitter	10	2,393	999	3,844
C.4 More Implementation Details
Tagging scheme for NER. As for the tagging scheme, we follow the classic BIO format. To be specific, the first token of an entity mentioned with type X is tagged as B-X while the remaining tokens inside that entity are labeled as I-X, and the non-entity tokens are annotated as 0. Such a scheme is more difficult than a simple IO format, especially for distant supervision.
Clean token selection for the binary pathway. In general, the precision of entity labels is relatively high, e.g., about  $97.96\%$  (23, 649/24, 141) in the CoNLL03 dataset. That indicates us most distantly-labeled entity tokens are real entities if omit fine-grained entity types. Therefore, when performing clean token selection on the binary pathway, it only selects non-entity tokens by the matched and high-confidence strategy while including all tokens labeled as entities.
Soft label for the entity pathway. When training on the selected clean token set, the binary pathway regards distant labels as true labels. However, we discard the distant hard labels but adopt the teacher model's output logits to derive soft labels [48] for the entity pathway, given by:
$$
\hat {y} _ {i, j} ^ {s} = \frac {\pmb {p} _ {i , j} ^ {2} / \sum_ {i} \pmb {p} _ {i , j}}{\sum_ {j ^ {\prime}} (\pmb {p} _ {i , j} ^ {2} / \sum_ {i} \pmb {p} _ {i , j ^ {\prime}})}
$$
where  $\pmb{p}_{i,j} = \mathrm{softmax}(f_{i,j}(\pmb{x};\theta_t))$ , is the probability of  $i$ -th token belonging to class  $j$  in sentence  $\pmb{x}$ , then calculate a Kullback-Leibler divergence loss. Because soft labels usually preserve sufficient information and encourage a more balanced assignment of target labels.
The implementation of teacher-student newtork. When splitting the clean token set and the noisy token set, we let the teacher model select respective clean tokens to train the student model. Specifically, the teacher model's double-head pathway filters reliable clean tokens independently following the previous defined criterion. Then the double-head of the student model is trained with selected clean tokens and corresponding labels. The teacher model parameters are periodically updated by the student model with EMA, given by:
$$
\theta_ {t} = \alpha \theta_ {t} + (1 - \alpha) \theta_ {s}
$$
where  $\alpha$  is a positive constant and is empirically fixed as 0.99 for Webpage/Wikigold and 0.995 for the remaining datasets. Finally, to train the entity pathway, we adopt a KL divergence loss on the student model's output logits and corresponding soft labels from the teacher model's prediction. The formulation is:
$$
\mathcal {L} _ {e \_ c l s} (\hat {\boldsymbol {y}} ^ {s}, f (\boldsymbol {x}; \theta_ {s})) = \sum_ {i} \sum_ {j} - \hat {y} _ {i j} ^ {s} \log f _ {i j} (\boldsymbol {x}; \theta_ {s}) + f _ {i j} (\boldsymbol {x}; \theta_ {s}) \log (f _ {i j} (\boldsymbol {x}; \theta_ {s}))
$$
Then, the student model's entity pathway is trained to approximate the soft labels. For the binary pathway, we calculate a standard binary cross-entropy loss, which is given by:
$$
\mathcal {L} _ {b \_ c l s} (\tilde {y} ^ {b}, \hat {y} ^ {b}) = - \tilde {y} ^ {b} \log \hat {y} ^ {b} - (1 - \tilde {y} ^ {b}) \log (1 - \hat {y} ^ {b})
$$
where  $\tilde{y}^b$  is the given distant label and  $\hat{y}^b$  is generated by student model's binary pathway.
D Pseudo-Code of DesERT
We describe the overall training pipeline of DesERT in Algorithm 1.
Algorithm 1 Training workflow of DesERT
Input: Training data  $\mathcal{D} = \{(\pmb {x}_i,\tilde{\pmb{y}}_i)\}_{i = 1}^M$  with distant labels; two sets of teacher-student networks,  $\theta_{t1},\theta_{s1}$  and  $\theta_{t2},\theta_{s2}$  .   
1:  $t\gets 0$  /\*Selection and self-training \*/   
2: while  $t <   T_{1}$  do   
3: Get a batch  $\mathcal{B} = \{(\pmb {x}_i,\tilde{\pmb{y}}_i)\}_{i = 1}^B\subset \mathcal{D};$    
4: if  $t <   k$  then   
5:  $\mathcal{B}_b\cup \mathcal{B}_e\leftarrow \mathcal{B}$  //decoupled datasets   
6:  $\overline{\mathcal{B}}_b^1 = \{\mathcal{B}_b^{l,1},\mathcal{B}_b^{u,1}\} \leftarrow \mathrm{Sel}(f(\theta_{t1}),\mathcal{B}_b);$    
7:  $\overline{\mathcal{B}}_e^1 = \{\mathcal{B}_e^{l,1},\mathcal{B}_e^{u,1}\} \leftarrow \mathrm{Sel}(f(\theta_{t1}),\mathcal{B}_b);$    
8:  $\overline{\mathcal{B}}_b^2 = \{\mathcal{B}_b^{l,2},\mathcal{B}_b^{u,2}\} \leftarrow \mathrm{Sel}(f(\theta_{t2}),\mathcal{B}_b);$    
9:  $\overline{\mathcal{B}}_e^2 = \{\mathcal{B}_e^{l,2},\mathcal{B}_e^{u,2}\} \leftarrow \mathrm{Sel}(f(\theta_{t2}),\mathcal{B}_b);$    
10: Update  $\theta_{s1}$  with  $\{\overline{\mathcal{B}}_b^1,\overline{\mathcal{B}}_e^1\}$  by minimizing L;   
11: Update  $\theta_{s2}$  with  $\{\overline{\mathcal{B}}_b^2,\overline{\mathcal{B}}_e^2\}$  by minimizing L;   
12: else   
13:  $\mathcal{X}_B\gets \{(x_i)\}_{i = 1}^B$    
14:  $\overline{\mathcal{B}}_b\cup \overline{\mathcal{B}}_e\gets \mathrm{Guess}(f(\theta_{t1}),f(\theta_{t2}),\mathcal{X}_B);$    
15: Update  $\theta_{s1},\theta_{s2}$  with  $\{\overline{\mathcal{B}}_b,\overline{\mathcal{B}}_e\}$  by minimizing L   
16: end if   
17:  $\theta_{t1}\gets \mathrm{EMA}(\alpha ,\theta_{t1}),\theta_{t2}\gets \mathrm{EMA}(\alpha ,\theta_{t2})$    
18:  $t\gets t + 1$    
19: end while /\* Post-hoc entity pathway finetuning \*/   
20: while  $t <   T_1 + T_2$  do   
21: Get a batch  $\mathcal{B} = \{(x_i,\tilde{y}_i)\}_{i = 1}^B\subset \mathcal{D};$    
22:  $\chi_B\gets \{(x_i)\}_{i = 1}^B,\tilde{\gamma}_B\gets \{(\tilde{\pmb{y}}_i)\}_{i = 1}^B$    
23:  $\overline{\mathcal{B}}_e^1\gets$  EntitySel(f(0t1),X,B,Y);   
24:  $\overline{\mathcal{B}}_e^2\gets$  EntitySel(f(0t2),X,B,Y);   
25: Finetuning  $\theta_{s1}$  with  $\overline{\mathcal{B}}_e^1$    
26: Finetuning  $\theta_{s2}$  with  $\overline{\mathcal{B}}_e^2$    
27:  $\theta_{t1}\gets \mathrm{EMA}(\alpha ,\theta_{t1}),\theta_{t2}\gets \mathrm{EMA}(\alpha ,\theta_{t2})$    
28:  $t\gets t + 1$    
29: end while
While DesERT has been proven to be effective for distant supervision, it is still subject to certain limitations. First, our WCE loss is estimated from pseudo-labels instead of the real ones. While we empirically find our WCE loss works well, its performance is theoretically restricted. One potential solution is to estimate a small validation set to remedy this problem, but we leave it as our future work. Second, while the imbalance between entity labels is located in Figure 1, our framework does not particularly integrate special components to explicitly overcome this problem. We believe it is not hard to draw inspiration from the recent achievement in long-tailed learning to further improve the NER performance. Lastly, since DesERT ensembles two sets of teacher-student networks as previous works did [9, 49], we should train peer-student models simultaneously and utilize the predictions from dual-teacher models iteratively, thus resulting in relatively higher training costs. We hope future efforts are made to alleviate the cost of network ensembling.
F Ethics Statement
While distant supervision is deemed a cheap way to collect and curate training data, the off-the-shelf and external knowledge bases steering the autonomous annotation procedure may include bias and