SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haobo Wang $^{1*}$ , Yiwen Dong $^{1*}$ , Ruixuan Xiao $^{1}$ , Fei Huang $^{2}$ , Gang Chen $^{1}$ , Junbo Zhao $^{1\dagger}$
<sup>1</sup>Zhejiang University, Hangzhou, China
2Alibaba Group, Hangzhou, China
{wanghaobo, dyw424, miaoruixuan, cg, j.zhao}@zju.edu.cn, feirhuang@gmail.com
While distant supervision has been extensively explored and exploited in NLP tasks like named entity recognition, a major obstacle stems from the inevitable noisy distant labels tagged unsupervisedly. A few past works approach this problem by adopting a self-training framework with a sample-selection mechanism. In this work, we innovatively identify two types of biases that were omitted by prior work, and these biases lead to inferior performance of the distant-supervised NER setup. First, we characterize the noise concealed in the distant labels as highly structural rather than fully randomized. Second, the self-training framework would ubiquitously introduce an inherent bias that causes erroneous behavior in both sample selection and eventually prediction. To cope with these problems, we propose a novel self-training framework, dubbed DesERT. This framework augments the conventional NER predicative pathway to a dual form that effectively adapts the sample-selection process to conform to its innate distributional-bias structure. The other crucial component of DesERT composes a debiased module aiming to enhance the token representations, hence the quality of the pseudo-labels. Extensive experiments are conducted to validate the DesERT. The results show that our framework establishes a new state-of-art performance, it achieves a  $+2.22\%$  average F1 score improvement on five standardized benchmarking datasets. Lastly, DesERT demonstrates its effectiveness under a new DSNER benchmark where additional distant supervision comes from the ChatGPT model.
As an iconic and pivotal task, named entity recognition (NER) [1] has gained wide attention from the natural language processing community. While there have been several successes towards solving this task [2, 3, 4, 5], most of these solutions demand decent datasets for training that enjoys both good quality and large quantity. When it comes to specialized industrial deployment, this leads us to expensive data annotation and curation costs which may hinder the development of NER.
Distant supervision [6, 7, 8] is a promising direction to alleviate this problem. Indeed, there have been a number of attempts to combine the additional labels obtained from distant supervision with the original unlabeled data, and then establish a selection-based self-training framework [9] to handle noise among distant labels. The core of these methods is to sample reliable clean tokens and cope with potentially noisy tokens involved by the imperfect pseudo-label prediction.
On one hand, we characterize that the label distribution for distant supervision is rather skewed. Notably, as shown in Figure 1 (a), we found that true non-entity tokens are far more than the remaining
(a) Confusion matrix
(b) True entity confusion matrix
Figure 1: (a) Confusion matrix of true labels and distant labels on the whole CoNLL03 dataset. Class index 0 denotes non-entity and 1 denotes all entity types, more than 169.1k non-entity tokens are correctly labeled and dominate true entity tokens. (b) The confusion matrix displays noise among true entity-type labels. (c) The real token class distribution on the CoNLL03 dataset and tokens selected at the fifth epoch by a basic self-training framework with a single-head/double-head pathway. It can be shown that double-head selects more tokens than single-head on the minority entity classes such as MISC and LOC.
(c) Token class distribution
tokens, quantity-wise. Meanwhile, there exists a severe class imbalance in true entity tokens with fine-grained noise in Figure 1 (b). Ignoring these phenomena by prior sampling-based methods is potentially performance-limited. The reason is that the dominant clean non-entity tokens can be easily distinguished and excessively picked up, while clean entity tokens are largely overlooked, especially on label-deficient entities like MISC and LOC; see Figure 1 (c). Whereas, the clean entity tokens are of real interest since NER attempts to distinguish the fine-grained token types. Unfortunately, there has not been any prior work that discusses or addresses this property specifically.
On the other hand, in spite of that self-training scheme applied with distant supervision is proven applicable [10, 9, 11], it could still be unreliable mostly due to the deviation between the pseudolabels and unobservable true labels. In other words, the self-training scheme is inherently prone to assign erroneous pseudo-labels, which can be aggravated due to label noise and finally cause error accumulation and conversely pollutes the selection. We postulate that both biases ought to be properly resolved in order to fully unlock the utility boost of the distant supervision technique for NER.
To this end, we propose DesERT, a holistic framework that resolves the aforementioned problems and achieves state-of-the-art DSNER performance. The main idea behind the DesERT is to decouple the initial task/data into entity vs. non-entity and fine-grained entity type sub-parts, then respectively to split again by selecting and splitting out clean- and noisy-token sets which are processed by different learning paradigms, forming a task/data taxonomy. We further introduce debiased self-training on noisy sets to reduce erroneous pseudo-labels. Empirical findings successfully justify the validity of our approach, where on multiple standardized NER benchmarks we establish a new set of state-of-the-art results. For example, DesERT achieves state-of-art performance, especially an improvement of  $3.26\%$  on the CoNLL03 dataset and  $4.26\%$  on the Webpage dataset. Finally, we establish a new benchmark for DSNER where the supervision signals are generated via the large language model ChatGPT and DesERT still holds its superiority.
Distantly-Supervised NER. There are two typical forms of noise in the labeling of distantly-supervised NER (DSNER) tasks: false negative tokens wrongly tagged as non-entities and inaccurate entity types. A series of works mainly focus on processing false negative tokens. A popular line of works adopts PU Learning [12, 13] techniques to resolve DSNER [14, 15], which sees entity tokens as Positive (P) data and non-entity tokens as Unlabeled (U) data, yet ignore the noise in fine-grained entities. Other methods consider both types of noise, usually inspired by traditional NLL approaches. One of them is based on clean token selection. Lots of works adopt the model's prediction consistency or confidence as the selection criterion [16, 17, 10, 18, 9, 11, 19, 20]. Besides, Huang et al. [21] propose a novel Logit Maximum Difference (LMD) score. HGL [22] introduces hypergeometric distribution to estimate noisy sample size. Additionally, some studies [23, 24] also employ reinforcement learning agents to filter noisy samples. A recent work [25] studies a specific DSNER setup that collects a large set of weakly labeled data and a small set of clean training data. The
most related work to ours is [26] which also investigates the bias issues in DSNER, but it concentrates on structural causal biases caused by the dictionaries. In contrast, our work focuses on unexpected selection and confirmation biases in popular selection and self-training DSNER frameworks.
Self-training. Self-training [27, 28, 29] is a popular semi-supervised learning technique that trains on labeled data but assigns pseudo-labels to unlabeled data for further training [30]. This is also adopted by existing DSNER approaches. For example, BOND [10] uses pseudo-labels from the teacher model to train the student model. SCDL [9] jointly trains two teacher-student networks to mutually provide pseudo-labels. RoSTER [11] ensembles different models' predictions as pseudolabels. However, some research has pointed out that self-training exists an inherent confirmation bias to assign erroneous pseudo-labels [31, 32]. Due to the existence of label noise, the DSNER task can face an amplified self-training bias, which, however, has never been touched on in prior studies.
3 Notation and Preliminary
Named Entity Recognition. Assume we receive a NER training dataset  $\mathcal{D} = \{(x_m, y_m)\}_{m=1}^M$  with  $M$  tuples, where each tuple consists of a sentence  $\pmb{x} = [x_1, x_2, \dots, x_n]$  with  $n$  tokens and a label sequence  $\pmb{y} = [y_1, y_2, \dots, y_n]$ . Here each entity label  $y_i$  indicates a token  $x_i$  is a non-entity or belongs to a specific entity type.  $y_i \in \mathcal{T} = \{0, 1, \dots, K\}$ , where  $1 \sim K$  denote entity types and  $0$  denotes non-entity. Our goal is to train a NER model  $f$  parameterized by  $\theta$  to be able to receive a testing sentence and predict the entity labels for its tokens. For fully supervised NER, this goal can be achieved by minimizing the empirical standard cross-entropy loss as follows,
$$
\mathcal {L} _ {c e} (\boldsymbol {y}, f (\boldsymbol {x}; \theta)) = - \frac {1}{n} \sum_ {i = 1} ^ {n} \log \operatorname {s o f t m a x} \left(f _ {i, y _ {i}} (\boldsymbol {x}; \theta)\right)
$$
where  $f_{i,y_i}(\pmb{x};\theta)$  is the model's predicted probability of  $i$ -th token  $x_i$  belonging to class  $y_i$  in sentence  $\pmb{x}$ . In practice, the NER model  $f$  is typically composed of a pre-trained language model encoder  $\phi$  for extracting token representation and a classification head  $h$ , i.e.,  $f(\pmb{x};\theta) = h \circ \phi(\pmb{x})$ .
Distantly-Supervised NER. This paper considers the distantly-supervised NER problem, which effectively reduces labeling costs. Accordingly, the true label sequences  $\{\pmb{y}_m\}_{m=1}^M$  are replaced by their distantly-annotated counterparts  $\{\tilde{\pmb{y}}_m\}_{m=1}^M$ . Despite the promise, the distant labels inevitably contain large portions of label noise due to the limited coverage of entity mentions in the knowledge base and the labeling ambiguity. To cope with this problem, we follow previous works [10, 9] and introduce the selection and self-training framework to mitigate the label noise. At each epoch, we first perform token selection based on the specific criterion that separates the tokens in each sentence  $\pmb{x}$  into a set of potential clean tokens  $\pmb{x}^l$  and noisy tokens  $\pmb{x}^u$ , and thus, a robust NER model can be trained on those clean tokens. To further exploit the remaining noisy tokens, we regard them as unlabeled tokens and perform semi-supervised training,
$$
\mathcal {L} _ {c l s} = \mathcal {L} _ {L} (\tilde {\boldsymbol {y}} ^ {l}, f (\boldsymbol {x} ^ {l}; \theta)) + \mathcal {L} _ {U} (\tilde {\boldsymbol {y}} ^ {u}, f (\boldsymbol {x} ^ {u}; \theta))
$$
where  $\mathcal{L}_L$  and  $\mathcal{L}_U$  are classification losses on the two sets.  $\hat{\pmb{y}}^u$  is a pseudo-label generated via previous models. In the sequel, we will specify our selection criterion and self-training algorithms in detail.
4 Proposed Method: DesERT
In theory, distant supervision can provide an arbitrarily large dataset for NER with its scale upper-bounded by all available sources. In this task of distantly-supervised NER, we denote the initial set extracted by distant supervision by  $\mathcal{D}$ . Of course, directly using  $\mathcal{D}$  to train a NER model by supervised learning is problematic, due to the noise, uncertainty, or mismatches hidden in  $\mathcal{D}$ . In this section, we provide a detailed description of our framework. The resulting architecture would effectively decompose  $\mathcal{D}$  into a task-data taxonomy, with each component from this taxonomy manifesting an individual training paradigm. This decomposition is stemmed from the two bias problems we mentioned previously. The pseudo-code of DesERT is summarized in Appendix D.
Figure 2: (a): Illustration of decoupled learning paradigm and debiased self-training. (b): Two classification heads are trained independently but make joint predictions when testing. # denotes unselected noisy or invalid tokens.
4.1 Decoupled Learning for Distant Supervision
To begin with, we start by introducing the top-tier decomposition which forms the first- and high-level of the taxonomy.
The Bias Problem. Let us recall the observation from Figure 1. Extending from it, we draw the following conclusion:
- in Figure 1 (a), we plot the confusion matrix of the true labels against distant labels without distinguishing between fine-grained types. It effectively indicates that the much dominant number of non-entity tokens against the true entity tokens (about ratio 170:23);
- in Figure 1 (b), we see the noisy distribution of the fine-grained entity-type labels is very much biased and imbalanced. In spite of that the prior work generally omits this problem, we believe this is so vital that the neglect of it would cause an overly high probability of non-entity tokens being picked during the selection process of DSNER. Therefore, only a few entity tokens are selected as clean data for normal NER training;
- Figure 1 (c) further offers supporting evidence. It shows that the previous DSNER frameworks, built with a single-head pathway hardly draft the clean and minority entity tokens. In particular, it samples 436/4,593 (9.49%) MISC tokens and (almost) none I-LOC tokens which are of more value in solving NER.
To sum up, we believe this loss of a few entity tokens being selected — caused by the data bias — is genuinely a huge waste and may have significantly hindered the performance gain. While the prior works generally overlook this problem, in this article, we intend to study it thoroughly and give a solution to resolve it.
Decomposition of the NER Task. To overcome the aforementioned data bias, we decompose the dataset  $\mathcal{D}$  into two sub-parts,  $\mathcal{D} = \mathcal{D}_b \cup \mathcal{D}_e$  where the subscripts of  $b$  and  $e$  indicate binary- and entity-classification respectively. In effect, we proactively decompose the NER task of single-level prediction into a hierarchical dual that the binary prediction targets entity versus non-entity while the counterpart task is to predict the specific entity category. Notably, the skewed data distribution on  $\mathcal{D}$  is also decomposed into two parts on  $\mathcal{D}_b$  and  $\mathcal{D}_e$ , which can be alleviated by separate processing and will not interfere with each other. More specifically, to construct this level of taxonomy, the label set  $\tilde{Y}^b \in \mathcal{D}_b$  is binarized. In the meantime,  $\tilde{Y}^e \in \mathcal{D}_e$  removes the non-entity tag (as invalid) and only reserves entity type labels.
Double-head Pathway. In what follows, to facilitate the training process upon the decomposition, we correspondingly augment a separate pathway to the original single prediction pathway, written as  $h = \langle h_b, h_e \rangle$ , where the subscripts follow the same naming rule as the previous paragraph. The end of the two pathways are both fully connected layers with different entry numbers: 1 for the  $h_b$  and  $K$  for  $h_e$ . The two pathways shared the same encoder,  $\phi$ , as we introduce in Section 3. A more
detailed illustration is provided in Figure 2 (a). We leave the further textual description of the model architecture and task-data taxonomy details in Appendix A.
Joint Prediction. One last piece of utilizing the dual pathway in the framework is the inference procedure. Since either pathway is unable to generate a complete NER prediction independently, we combine them in a jointly predictive fashion. As shown in Figure 2 (b), we form a prediction hierarchy as follows. Given any token  $x_{i} \in \mathbf{x}$ , the binary pathway  $h_{b}$  offers a probability of that token being a part of an entity denoted as  $p_{i}^{b}$ . On the other hand, the entity pathway generates a vector of specific entity probability  $\pmb{p}_{i}^{e}$  having  $K$  entries. Then the double-head pathway generates a joint probability  $\pmb{p}_{i} = [1 - p_{i}^{b}, p_{i}^{b} * \pmb{p}_{i}^{e}]$ , that  $1 - p_{i}^{b}$  is the probability of being an non-entity. By then, the model can predict each token's specific type, i.e., whether a non-entity or fine-grained entity type.
4.2 Clean Token Selection
Following the previous decomposition, the next step we propose is to follow a general noisy-label learning setup — to select the clean set from the noisy dataset. In particular, upon either task/data component in the first level taxonomy, it can be further decomposed down to be a union of the clean and noisy set. Different learning schemes are then applied to these two sets of data and enacted concurrently. In this section, we discuss how the sample selection is devised.
We take the binarized dataset/task to describe the selection mechanism. In hindsight, we assume that the tokens ought to be put into the clean pool when the model's prediction is relatively certain and it matches the provided counterpart label by distant supervision. Formally, given any token  $x_{i}$  drawn from any sentence, denote its distant label as  $\tilde{y}_i^b$ . During training, we take the model obtained at the current step and feed the input through it attaining a current prediction of  $x_{i}$ , dubbed as  $\hat{y}_i^b$ . Hereby, we can write down the primary selection criterion:
$$
\mathcal {D} _ {b} ^ {l} = \{(x _ {i}, \tilde {y} _ {i} ^ {b}) | \mathbb {I} (\hat {y} _ {i} ^ {b} = \tilde {y} _ {i} ^ {b}) \wedge (\max (p _ {i} ^ {b}, 1 - p _ {i} ^ {b}) > \tau) \}
$$
Noted, we use  $\mathcal{D}_b^l$  to denote the clean set, as superscript  $l$  implies "labeled" since the clean tokens are primarily utilized as if they were gold. We simply adopt  $\max(p_i^b, 1 - p_i^b)$  as a proximal form of predictive uncertainty on the given sample where  $\tau$  is a hyperparameter for the threshold. In most of our experiments, we fix it to 0.95 without further tuning. Further, the left-out samples — where the token samples do not meet this criterion — are gathered and then form  $\mathcal{D}_b^u$  such that  $\mathcal{D}_b = \mathcal{D}_b^l \cup \mathcal{D}_b^u$ . Here the superscript  $u$  means "unlabeled" as we remove the tagged distant supervision labels on them and essentially treat these samples as unlabeled ones. For the finer-grained dataset  $\mathcal{D}_e$ , we adopt the same strategy, as  $\mathcal{D}_e = \mathcal{D}_e^l \cup \mathcal{D}_e^u$ .
Notably, the decomposition of the task/data taxonomy is grounded at the token level; namely, the same sentence could contain clean tokens and noisy tokens belonging to different sets for training respectively. Further, the assignment of the taxonomy is dynamic and altered on the fly. This means that at every round of training, DesERT performs token selection based on an inference path conducted by the model trained at the current cycle.
4.3 Debiased Self-Training
In the previous step, we resplit the datasets and process on the clean token sets  $\mathcal{D}_b^l$  and  $\mathcal{D}_e^l$ , leaving unselected noisy token sets  $\mathcal{D}_b^u$  and  $\mathcal{D}_e^u$  remained to be tackled. Since labels for  $\mathcal{D}_b^u$  and  $\mathcal{D}_e^u$  are likely to be noisy, we remove them and tag these tokens with pseudo-labels via a self-training framework. Notably, the pseudo-labels can be obtained from various sources and we choose a teacher-student architecture that is the same as previous work [10]. Despite that self-training has been proven effective repeatedly [10, 9, 11], we argue that on this occasion it might still be unreliable due to another source of inherent bias.
In practice, the training on the clean and noisy set alternates. In hindsight, the supervised learning on the selected clean set naturally clusters the clean tokens tightly to their respective class centers. By contrast, the noise tokens receive dynamically changing labels, and it is likely that the labels are wrong. This, in theory, may cause the corresponding representation to deviate from the class centers and even oscillate by big magnitude due to its instability assignment during training. Given the imbalance prediction nature of the DSNER task, we empirically find that this kind of bias is
trended to aggravate if no other mechanism is imposed. Further, if we let this bias be untangled, the clean token selection we introduced previously might be severely biased and polluted.
Therefore, to overcome this issue, we employ a worst-case cross-entropy (WCE) loss objective [32]. In a nutshell, the WCE loss adversarially optimizes the unlabeled token representation by matching the worst classifier on labeled data. We call a classifier the worst if it can correctly classify all the labeled data, but is very close to the edge of clusters, resulting in a minimal margin. If we adversarially train the unlabeled tokens to be correctly classified by this worst classifier, their features can be tight to at least one class center. Therefore, the decision hyperplane can be pushed to a lower entropy region, which is known to be beneficial for semi-supervised learning [33]. By regularizing noisy tokens to approach the class centers rather than the decision boundary, the classifier can also generate more stable and higher-quality pseudo-labels. For better understanding, we elaborate our theoretical insights of this debiased learning procedure in Appendix B.
Nevertheless, in practice, it is impossible to obtain the truly worst classifier. Hence, proximally, we estimate a possible worst classifier  $h_w$  such that it distinguishes the selected tokens correctly while making mistakes on (i)-the unchosen tokens and (ii)-the produced pseudo-labels at previous cycles, as much as possible. Formally, we denote  $f^h = h \circ \phi$  and calculate the worst head by,
$$
h _ {w} = \arg \max  _ {h ^ {\prime}} \mathcal {L} _ {U} \left(\hat {\boldsymbol {y}} ^ {u}, f ^ {h ^ {\prime}} \left(\boldsymbol {x} ^ {u}\right)\right) - \mathcal {L} _ {L} \left(\tilde {\boldsymbol {y}} ^ {l}, f ^ {h ^ {\prime}} \left(\boldsymbol {x} ^ {l}\right)\right)
$$
where the encoder  $\phi$  is frozen at this step.  $\hat{\pmb{y}}^u = f^h (\pmb{x}^u)$  is the pseudo-label that serves as a proxy to estimate the classification error.
Then, we adversarially optimize the encoder  $\phi$  to be correctly classified by the worst-case classifier. The formulation  $\mathcal{L}_{wce}$  is given by:
$$
\mathcal {L} _ {w c e} (\phi) = \mathcal {L} _ {U} (\hat {\boldsymbol {y}} ^ {u}, f ^ {h _ {w}} (\boldsymbol {x} ^ {u})) - \mathcal {L} _ {L} (\tilde {\boldsymbol {y}} ^ {l}, f ^ {h _ {w}} (\boldsymbol {x} ^ {l}))
$$
Noted that we fix the worst-case head  $h_w$  and only update the encoder representation. Moreover, since the binary head regards all fine-grained types as an integrated class, optimizing WCE on it may cause a collapse of the representation and then, hinder the model of distinguishing different fine-grained entity types. Therefore, we calculate WCE loss  $\mathcal{L}_{e\_wce}$  on the entity-type recognition pathway. Empirically, the noisy token representation gradually approaches to corresponding class center. Consequently, the decision hyperplane is more likely to lie in a lower entropy region, thereby mitigating the impact of self-training bias.
4.4 Practical Implementation
Dual Co-guessing Mechanism. It has been widely verified that a larger labeled set can greatly boost semi-supervised training. However, in our selection procedure, the size of the clean set is largely restricted by the number of originally clean samples owing to our label-matching constraint. Fortunately, many tokens, while originally noisy, can be accurately predicted by the model as SSL training proceeds. To this end, we develop a co-guessing mechanism that collects well-predicted tokens as a pseudo proxy to enlarge the clean sample set.
Specifically, we train two peer networks having the same architecture, which we denote as  $f(\pmb{x}; \theta_1)$  and  $f(\pmb{x}; \theta_2)$ . Given those easy-to-learn noisy tokens, we postulate that different networks can generate consistent and high-confidence predictions. This gives rise to the following criterion,