SlowGuess/NAACL_2022_OCR huggingface dataset:
Huaiyuan Ying $^{1*}$  Shengxuan Luo $^{2,3*}$  Tiantian Dang $^{4}$  Sheng Yu $^{2,3}$
$^{1}$ Department of Mathematical Sciences, Tsinghua University
$^{2}$ Center for Statistical Science, Tsinghua University
$^{3}$ Department of Industrial Engineering, Tsinghua University
$^{4}$ Department of Automation, Tsinghua University
{yinghy,luosx18,dtt19}@mails.tsinghua.edu.cn
Distantly-supervised named entity recognition (NER) locates and classifies entities using only knowledge bases and unlabeled corpus to mitigate the reliance on human-annotated labels. The distantly annotated data suffer from the noise in labels, and previous works on DSNER have proved the importance of pre-refining distant labels with hand-crafted rules and extra existing semantic information. In this work, we explore the way to directly learn the distant label refinement knowledge by imitating annotations of different qualities and comparing these annotations in contrastive learning frameworks. the proposed distant label refinement model can give modified suggestions on distant data without additional supervised labels, and thus reduces the requirement on the quality of the knowledge bases. We perform extensive experiments and observe that recent and state-of-the-art DSNER methods gain evident benefits with our method.
Named entity recognition (NER) refers to the sequence tagging task of detecting the interested entities in unstructured texts and classifying them into predefined categories. NER serves as a foundation part of information extraction in natural language processing (NLP) with applications in many downstream tasks such as question answering (Khalid et al., 2008; Jin et al., 2021), knowledge graph construction (Jia et al., 2018; Zhao et al., 2018), and dialog systems (Bowden et al., 2018). Supervised NER models have been developing rapidly in recent years and have achieved enormous success (Huang et al., 2015; Wang et al., 2020). However, acquiring abundant high-quality human annotations where every word within a sentence should be labeled can be very expensive and limits the application of NER models in many domains.
To alleviate the reliance on human annotations, a practical approach is to introduce distant supervision (Mintz et al., 2009) to automatically generate labeled data by matching entities in easily-obtained knowledge bases. Meanwhile, after years of development, there are many open access knowledge bases or dictionaries such as WikiData<sup>1</sup> and YAGO<sup>2</sup> in the general domain and UMLS (Lindberg et al., 1993) and MeSH<sup>3</sup> in the biomedical domain, which makes it possible to annotate large scale training data for NER models automatically. However, distant annotation suffers from two issues: incomplete annotation and noisy annotation. The knowledge bases with limited coverage of entities usually label only part of the entities in text, and the remaining entities are incorrectly labeled as background, denoted as incomplete annotations. The noisy annotation occurs when an entity with more than one word and the knowledge bases only contain a sub-sequence of the entity, resulting in partial annotation and sometimes wrong labeling of the entity type.
The neural network-based NER model has a strong ability in fitting the noise in training data, resulting in poor performance with distant labels. Some previous works focusing on distantly-supervised named entity recognition (DSNER) attempt to mitigate the two issues by applying techniques including: (1) entity selection (Yang et al., 2018; Zhang et al., 2021) or noisy entity removal (Onoe and Durrett, 2019); (2) label smoothing (Yang et al., 2018; Shang et al., 2018), (3) iteration and early stopping Liang et al. (2020); (4) PU-learning (Meng et al., 2021).
In addition, many works have found that pre
1 https://dumps.wikimedia.org/wiki/data wiki/entities/
2 https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga/yago/
3 https://www.nlm.nih.gov/mesh/download_dmesh.html
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2656 - 2666
refining the distant labels by artificial rules (Shang et al., 2018; Zhang et al., 2021; Liang et al., 2020; Meng et al., 2021) and extra semantic information (Zhang et al., 2021; Liang et al., 2020) effectively improves the performance of DSNER models. Inspired by this, we propose a framework to train an automatic distant label refinement model. Specifically, we generate annotations of different qualities by adding noises to weaken the annotations or enhancing the annotations with semantic parsing (Chen and Manning, 2014). Then we apply contrastive learning to guide a sequence scoring model to learn which annotations are better based on the sentence given two different annotations. Finally, the sequence scoring model can make a refinement suggestion on each token in arbitrary sentences and corresponding annotations. Therefore, it is model-agnostic, which can be stably and effectively used in data preprocessing for all DSNER models and consequently improves the performance of the DSNER models.
Due to the complex nomenclature (Néveol et al., 2018) and massive amounts of terminology in the biomedical domain, incomplete and noisy annotations are more evident in the distantly annotated corpus. Therefore, we focus on distant data refinement for DSNER in this work. Experiments show that the framework is good at amending the noisy or incomplete entities in distant data and significantly increases recall and F1 scores for DSNER models. The proposed method is named CReDEL, for Contrastive Refinement of Distant Entity Labels. The source code of our model is publicly available at https://github.com/yinghy18/CReDEL.
- We propose an automatic label refinement method to mitigate the issues in distant NER data. The CReDEL is model-agnostic and consequently improves the performance of all DSNER models.
- We introduce a contrastive learning technique combined with a novel contrastive sample generation module. Trained on automatically annotated enormous corpus with knowledge bases, it empowers the proposed scoring model to evaluate annotation qualities.
- We conducted experiments on BC5CDR and NCBI-Disease to verify the effectiveness of
CReDEL with classical NER and state-of-the-art DSNER methods. We show that our model brings consistent improvement for these methods.
Distantly-Supervised Named Entity Recognition
Compared to fully supervised NER, DSNER gets rid of human annotations and uses knowledge bases or dictionaries to annotate the corpus automatically. Some DSNER works adopt entity selection (Yang et al., 2018; Zhang et al., 2021) or noisy entity removal (Onoe and Durrett, 2019) strategies, while some works design new components to handle multiple possible labels (Yang et al., 2018; Shang et al., 2018). Besides, Liang et al. (2020) applies early stopping to prevent fitting the noise and iteratively self-trains the model to recognize more entities. Peng et al. (2019) formulates DSNER as a positive-unlabeled learning problem. Meng et al. (2021) uses a noise-robust loss and a noisy label removal module and uses a self-training method to improve the generalization ability.
The works mentioned above (Shang et al., 2018; Zhang et al., 2021; Liang et al., 2020; Meng et al., 2021) also demonstrate that the pre-refinement of distant labels significantly improves the performance of DSNER methods. Before applying the DSNER models, these works refine the distant labels by tailoring corpus-aware dictionary (Shang et al., 2018), extending entity boundary by a distant phrase mining model (Shang et al., 2018; Zhang et al., 2021), annotating potential entities via POS tagging, and hand-crafted annotation rules (Liang et al., 2020; Meng et al., 2021). In this work, we propose a model to learn the refinement knowledge of distant NER annotations, which consequently has a lower requirement of the dictionary, corrects labels automatically, and facilitates DSNER models.
Intuitively, the contrastive model is trained via comparing between input data (Le-Khac et al., 2020), in which way the embeddings learn to put together similar samples and push away different ones. The core of contrastive learning is thus generating positive and negative sample pairs (Kalantidis et al., 2020). Previous works in NLP fields provide various ways of designing positive-negative sample
pairs. Yuan et al. (2020) uses CODER to learn term representations by maximizing similarities between positive term-term pairs, where the word "positive" indicates that the terms are synonyms in the UMLS. Gao et al. (2021) constructs positive sentence embedding pairs using the embeddings of the same sentence passed through different dropouts in BERT in their unsupervised version. Contrastive learning has also been applied to the NER task. Lin et al. (2020) figures out the triggers for entity recognition by generating negative samples after randomly mixing triggers and sentences. Das et al. (2021) also utilizes contrastive learning to optimize distributional divergence and improve few-shot NER performance.
The design of contrastive loss always attempts teaching the model to minimize the distance within clusters despite the difference in forms, such as triplet margin loss (Balntas et al., 2016), contrastive loss (Chopra et al., 2005) and probabilistic NCE-based loss (Ma and Collins, 2018). We also mix hard negative samples in this work and transform the typical margin loss for CReDEL.
In this section, we detailedly describe the whole pipeline of CReDEL (Figure 1). Firstly, a knowledge base is used to obtain the distantly labeled tagging sequence. Then we construct sample pairs on a large corpus and train a scoring model via contrastive learning. We also introduce a module to generate high-quality positive samples by modifying the entity boundary. For the purpose of improving the quality of distant labeled training data, we apply the scoring model to modify datasets used by existing DSNER models, leading to their better performances.
3.1 Distant Labels Generation
For a given knowledge base and corpus, the distantly labeled data always refer to the corpus tagged by matching all entities in the knowledge base following the previous works (Peng et al., 2019; Zhang et al., 2021). The matching algorithm is maximum matching which greedily searches the longest string in the knowledge base. We adopt the "BIO" tagging scheme in this work to represent if a token is at the beginning (B) or inside (I) of a matched entity or does not belong (O) to any entity.
3.2 Contrastive Model Training
Given a sentence consisting of  $m$  words  $X = (x_{1},\dots ,x_{m})$  and two "BIO" tag sequences  $Y_{P},Y_{N}$  of  $X$ , the scoring model takes the triple  $(X,Y_P,Y_N)$  as input and learns to assign a better score to the tag sequence of higher quality,  $Y_{P}$ , and a worse score for the other one  $Y_{N}$ . This naturally comes down to contrastive learning, and the tag pair  $(Y_P,Y_N)$  are sampled as follows:
Negative samples generation For one sentence  $X$ , we imitate the distant annotations to generate two pairs of positive-negative tag sequences: (1) Positive sample  $Y_{1,P}$  is the original distant tag sequence described in Section 3.1, while negative sample  $Y_{1,N}$  is obtained by randomly subtracting one entity from  $Y_{1,P}$ . This is the negative tags created by imitating incomplete annotations. (2) Positive sample  $Y_{2,P}$  is the previous negative one,  $Y_{1,N}$ . The negative sample  $Y_{2,N}$  is produced by changing  $Y_{2,P}$  to a different tag sequence within "BIO" tagging scheme with probability  $p$  in each position. This pair contains the incomplete annotation and its inferior version by imitating noisy annotations. To avoid the model from remembering our dictionary tagging, the pair  $(Y_{1,P}, Y_{1,N})$  is only used in development set.
As the scoring model trained through contrastive learning can be insensitive to entity boundary with the aforementioned triples, we also generate another training tag pair using  $\mathrm{Parser}^4$ . In this triple, the negative sample  $\mathbf{Y}_{3,N}$  is the distant tag sequence  $\mathbf{Y}_{1,P}$  and the positive sample  $\mathbf{Y}_{3,P}$  is the parse-enhanced version of  $\mathbf{Y}_{3,N}$ , which can be obtained from the rules in appendix A.2.
These three cases are hard negative samples and are shown in 1. However, the model requires more easy samples to learn some basic rules of the scoring task, so three other kinds of easy samples are mixed with existing hard samples. Specifically, a certain percentage of existing negative samples will be changed into (1) a random permutation of the distant tag sequence  $\mathbf{Y}_{1,P}$ , (2) a sequence of the same length containing only tag "B", (3) a shift of the  $\mathbf{Y}_{1,P}$  by two tokens, For example, the first tag of  $\mathbf{Y}_{1,P}$  will be the third in the negative tag sequence, and the second tag will become the fourth in the negative one.
Training Procedure For one triple  $(X,Y_P,Y_N)$
4In this paper, we use the Stanford Parser(?).
Figure 1: The framework of CReDEL. In training, CReDEL takes a sentence and a generated sample tag pair of distant labels as input, and uses contrastive loss to update model parameters. In inference, CReDEL masks tokens according to the distance between the probability score of the input sentence and distant labels. We use the same sentence as an example in both training and inference for brevity. The refined distant data is then used to train the DSNER models. "CHEM" and "DISE" refer to chemical and disease.
the architecture of the scoring model is as follows:
$$
\boldsymbol {h} = B E R T (\boldsymbol {X}),
$$
$$
\boldsymbol {a} = \left[ \boldsymbol {s} _ {O}, \boldsymbol {s} _ {I}, \boldsymbol {s} _ {B} \right] = \text {s o f t m a x} (\boldsymbol {h} \boldsymbol {W}). \tag {1}
$$
The sentence  $X$  is passed through the BERT language model and turned into the hidden representation  $h$ . Then a linear layer with output dimension three acts on  $h$  to get the probability score for "B", "I" and "O" in each position respectively, denoted by  $a$ , which is a matrix of  $m$  rows and 3 columns.
Bounded Cross-Entropy Distance We expect the output score sequence  $\pmb{a}$  close to the one-hot encoded positive tag sequence  $\pmb{p}$  and away from the encoded negative tag sequence  $\pmb{n}$ . This is where contrastive learning applies. The "close" and "away" are defined through a distance. We first utilize cross-entropy (CE) loss function value to describe this distance, but the model collapses after enlarging the numerical gap through multiplying this distance by a constant only. To avoid this, a parameter  $\varepsilon$  is introduced and gives an upper bound to the distance. Finally, our loss function is the
margin loss with distance, written as:
$$
\mathcal {L} = \max  (d (\boldsymbol {a}, \boldsymbol {p}) - d (\boldsymbol {a}, \boldsymbol {n}) + m a r g i n, 0), \tag {2}
$$
where  $d$  is the introduced distance with upper bound:
$$
d (x, y) = - \log \frac {\exp (- \mathrm {C E} (x , y)) + \varepsilon}{1 + \varepsilon}. \varepsilon > 0. \tag {3}
$$
3.3 Distantly Label Improving
CReDEL will not directly predict a tag sequence, as we find the scoring model cannot effectively discriminate "B" from "I" and a single "I" may appear in the tags without a leading "B". Alternatively, we use the output score defined below to refine distantly labeled data.
During inference, CReDEL takes a sentence and distant tag sequence as input and outputs the token-level distance  $d(\mathbf{a}, \mathbf{Y})$  of classification probability score matrix  $\mathbf{a}$  and the one-hot encoded distant tag sequence as in 3.2. Then, it evaluates each tag by setting a threshold and masking all positions where the distance is larger than this threshold. Finally the masked sequence  $\mathbf{Y}_{mask}$  obtained from the original distant tag sequence  $\mathbf{Y}$  is fed to NER or DSNER models. Here mask means that this tag in this position will not contribute to the calculating of loss in DSNER models.
Since specific NER datasets usually contain several types of entities while the proposed model considers all entities, we add an entity typing module to classify the types of entities extracted from the contrastive scoring model. Then we only mask the tokens predicted to be the desired type in datasets. The entity typing module consists of a classical BERT-based classification model which takes an entity and its surrounding words as input to predict the entity type. The architecture of the entity typing module is described in Appendix A.3.
Our knowledge base for distant annotation is selected from UMLS by retaining the entities from specific high-quality sources and applying basic cleaning procedures (see Appendix A.1). For simplicity, the knowledge base is still denoted as UMLS in the follow-up sections.
We select 1 million sentences extracted from PubMed abstracts and distantly label them with
UMLS for the training of CReDEL. The size of this distant data is much larger than biomedical NER datasets and provides more semantic knowledge for the contrastive scoring model.
The public datasets we used to test the improvements are listed below:
BC5CDR (Li et al., 2015) is a biomedical Chemical Disease Relation dataset, which is also widely used in biomedical Named Entity Recognition. It consists of 1,500 PubMed articles with 4,409 annotated chemicals, 5,818 diseases entities. The training, development, and test set has 500 articles respectively. Since the types of entities in BC5CDR are "Disease" or "Chemical", We only use the terms of these two types in knowledge bases to label the training set.
NCBI-Disease (Doogan et al., 2014) is a dataset focusing on disease entities. The corpus texts are made of 793 abstracts and 6,881 annotated entities. We use the raw texts of the training set, consisting of 593 abstracts, and test on its test set consisting of 100 human-annotated abstracts. We only regard the "Disease" entity type in this corpus.
For both BC5CDR and NCBI-Disease, we adopt two sources of knowledge bases, the UMLS in Section 3.1 and the core dictionaries containing domain-specific terms from AutoNER (Shang et al., 2018), to annotate the two datasets. The four distantly annotated datasets are denoted as BC5CDRUMLS, NCBI-UMLS, BC5CDR-AutoNER, and NCBI-AutoNER, respectively.
The proposed method pre-refines the training data for DSNER models. Therefore, we select three recent DSNER models to train on the distant data or the refined distant data and compare their performance. Besides, we select classical fully-supervised methods for better illustration and dictionary match as a baseline:
Dictionary Match is the distant annotation method in Section 3.1. For better adaptation to specific datasets, we only keep the entities of corresponding types. Fully-supervised Methods. We select two fully-supervised NER models for comparison: (1) BiLSTM-CRF (Huang et al., 2015) is the classical RNN-based NER model. It is trained with the distant annotated data or the data refined by CReDEL without language model pre-training. (2)
RoBERTa (Liu et al., 2019) is a pre-trained language model, and we fine-tune the RoBERTa on the
same data as we use in BiLSTM-CRF. Distantly-supervised Methods. Three recent or the state- or-art DSNER methods are applied to the distant annotated data or the data refined by CReDEL, including: (1) AutoNER (Shang et al., 2018) is a DSNER model with a "Tie or Break" tagging scheme containing entity span detection module and type classification module. (2) BOND (Liang et al., 2020) fine-tunes a RoBERTa on distant data with early stopping, then it iteratively expands the dictionary and self-trains the model. (3) RoSTER (Meng et al., 2021) uses the noise-tolerant mean absolute error loss with the self-training method and augmented sequences generated by the pre-trained language model without fine-tuning.
The contrastive scoring model uses the BERT pretrained model with dimension 768 followed by a three-class classification layer. In the loss function, The hyperparameter  $\varepsilon$  is set to 0.2 and the margin equals 0.3. The training adopts a warming up procedure with the learning rate initialized as  $8 \times 10^{-6}$  and reaching its peak  $3 \times 10^{-5}$  at warming step 1000. We use a batch size of 16 sentence with a max length of 256. The training will take about one day on RTX 2080 Ti.
For the compared methods, we preserve their default or recommended parameters unchanged most of the time. The ensemble model number is set to 1 in RoSTER as our computation resources are limited. And the full dictionary of AutoNER containing high-quality phrases without type information is abandoned because our dictionary does not have a counterpart to compare with.
5.1 NER Performance Comparison
The datasets improved by the CReDEL with Parser tend to produce more complete term phrases after training, which can be seen in the case study in Section 5.2. However, the case study also shows that the complete term may be away from the test annotations which are part of a long entity. The completeness cannot be captured with traditional exact match F1-score, where predicted entities contribute to the true positive only if the left and right boundary and the entity type all match the test annotations. Motivated by the Boundary IoU (intersection over union) metric (Cheng et al., 2021), we promote another precision, recall, and F1 com
Methods	BC5CDR-UMLS	BC5CDR-AutoNER	NCBI-UMLS	NCBI-AutoNER
Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Matching	0.658	0.613	0.635	0.917	0.578	0.709	0.879	0.334	0.484	0.844	0.544	0.662
Original Distant
BiLSTM-CRF	0.638	0.480	0.547	0.918	0.506	0.652	0.811	0.401	0.537	0.88	0.276	0.420
AutoNER	0.641	0.516	0.572	0.880	0.560	0.685	0.803	0.485	0.605	0.863	0.269	0.410
RoBERTa	0.657	0.554	0.601	0.888	0.619	0.729	0.819	0.451	0.581	0.882	0.328	0.479
BOND-stage2	0.603	0.658	0.629	0.896	0.612	0.727	0.819	0.437	0.570	0.877	0.352	0.503
RoSTER	0.645	0.701	0.672	0.825	0.713	0.765	0.913	0.545	0.683	0.873	0.397	0.546
Refined w/o Parser
BiLSTM-CRF	0.672	0.492	0.568	0.846	0.550	0.667	0.847	0.553	0.669	0.827	0.439	0.574
AutoNER	0.660	0.557	0.604	0.802	0.639	0.711	0.800	0.588	0.678	0.775	0.376	0.506
RoBERTa	0.657	0.643	0.650	0.807	0.686	0.741	0.834	0.610	0.704	0.839	0.476	0.607
BOND-stage2	0.631	0.720	0.673	0.819	0.677	0.742	0.888	0.644	0.746	0.846	0.514	0.640
RoSTER	0.648	0.787	0.711	0.761	0.780	0.770	0.748	0.734	0.741	0.828	0.666	0.738
Refined
BiLSTM-CRF	0.617	0.623	0.620	0.844	0.577	0.686	0.856	0.516	0.644	0.808	0.445	0.575
AutoNER	0.634	0.608	0.621	0.793	0.697	0.742	0.814	0.584	0.680	0.739	0.394	0.514
RoBERTa	0.629	0.712	0.668	0.789	0.747	0.768	0.814	0.602	0.692	0.822	0.485	0.610
BOND-stage2	0.570	0.802	0.666	0.689	0.836	0.756	0.843	0.637	0.726	0.800	0.557	0.657
RoSTER	0.619	0.849	0.716	0.717	0.868	0.786	0.714	0.709	0.711	0.781	0.624	0.694
Table 1: The Boundary Intersection metric and corresponding precision, recall and F1 score of the distantly supervised methods on datasets BC5CDR and NCBI-Disease annotated by our UMLS dictionary and AutoNER dictionary. Boldface numbers indicate that this experiment with the left-side model achieves the best performance on one specific dataset among the three data conditions.
puting method, called BI (Boundary Intersection) score, by considering intersection as follows:
$$
\text {P r e c .} = \frac {\left| P _ {e} \cap G _ {e} \right|}{\left| P _ {e} \right|},
$$
$$
\operatorname {R e c.} = \frac {\left| P _ {e} \cap G _ {e} \right|}{\left| G _ {e} \right|}, \tag {4}
$$
$$
F 1 = \frac {2 \times \text {P r e c .} \times \text {R e c .}}{\text {P r e c .} + \text {R e c .}}.
$$