SlowGuess/NAACL_2022_OCR huggingface dataset:
Here  $P_{e}$  and  $G_{e}$  are the set of predicted label positions and true label positions within some entity. Intuitively speaking, the new F1 is token-level F1 after excluding the condition that both prediction and ground truth are 'O'. This metric can describe the actual performance of the methods on data improved by CReDEL with parser in the aspect of improving term completeness. Moreover, it is suitable for real-world settings like extracting terms. The new F1-score, precision, and recall on all datasets and methods are demonstrated in Table 1. The "Original Distant", "Refined w/o Parser" and "Refined" refer to the distant data, the data refined by CReDEL trained without parse-enhanced samples, and the data refined by CReDEL, respectively.
We should compare the scores on the same dataset by the same DSNER model trained with original distant data and the refined data. Under the BI score metric, Table 1 tells us that the data modified by the CReDEL with Parser achieves an almost universal improvement in F1-score compared
to original data. In BC5CDR-UMLS, BC5CDR-AutoNER, and NCBI-AutoNER it also beats the CReDEL without Parser. The promotion of F1 is mainly obtained through relatively more considerable progress in recall score and a slight change in precision score. This can prove the superiority of our method.
We also produce the exact match F1-scores in Table 2 denoted as the precision*, recall*, and F1-score* on the same datasets. In general, both data modified by CReDEL with and without Parser achieve higher recall scores on all the datasets and all the methods, and the one modified by CReDEL with Parser further increases the recall on much more than half the experiments. Meanwhile, the precision scores are boosted on about half of the experiments, especially on the two datasets tagged by the UMLS subset. Consequently, over  $\frac{3}{4}$  of the data reach a higher F1-score among the experiments in Table 2 and not in a trade-off manner.
It is worth noticing that the gain in F1-score differs in range, and we claim that for a dictionary not dataset-specific, such as our dictionary, which can be generally applied, the improvement will be rather noticeable. On the other hand, the construction of AutoNER dictionaries includes more handcrafted cleaning rules which reduce the noise in annotations. They only consist of about 1k terms rather different from general domain dictionaries
Methods	BC5CDR-UMLS	BC5CDR-AutoNER	NCBI-UMLS	NCBI-AutoNER
Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Matching	0.593	0.393	0.473	0.859	0.482	0.617	0.636	0.227	0.335	0.606	0.447	0.514
Original Distant
BiLSTM-CRF	0.568	0.431	0.490	0.836	0.524	0.644	0.621	0.225	0.330	0.613	0.473	0.534
AutoNER	0.558	0.469	0.509	0.798	0.586	0.675	0.585	0.230	0.330	0.595	0.510	0.549
RoBERTa	0.597	0.545	0.570	0.793	0.665	0.723	0.617	0.309	0.412	0.600	0.509	0.551
BOND-stage2	0.565	0.603	0.583	0.788	0.666	0.721	0.623	0.293	0.398	0.596	0.505	0.546
RoSTER	0.596	0.628	0.612	0.733	0.726	0.729	0.586	0.326	0.419	0.649	0.551	0.596
Refined w/o Parser
BiLSTM-CRF	0.572	0.488	0.527	0.787	0.551	0.648	0.604	0.311	0.411	0.614	0.541	0.575
AutoNER	0.547	0.537	0.542	0.711	0.640	0.673	0.507	0.288	0.367	0.525	0.526	0.525
RoBERTa	0.568	0.662	0.612	0.738	0.696	0.716	0.637	0.502	0.561	0.567	0.623	0.594
BOND-stage2	0.585	0.644	0.613	0.743	0.692	0.717	0.639	0.470	0.542	0.614	0.620	0.617
RoSTER	0.553	0.725	0.627	0.674	0.774	0.720	0.605	0.602	0.603	0.484	0.673	0.563
Refined
BiLSTM-CRF	0.537	0.564	0.550	0.759	0.541	0.632	0.634	0.333	0.437	0.637	0.516	0.570
AutoNER	0.544	0.532	0.538	0.671	0.659	0.665	0.503	0.328	0.397	0.553	0.530	0.541
RoBERTa	0.559	0.697	0.621	0.695	0.745	0.720	0.581	0.469	0.519	0.548	0.610	0.577
BOND-stage2	0.575	0.693	0.628	0.682	0.751	0.715	0.600	0.489	0.538	0.591	0.636	0.613
RoSTER	0.569	0.749	0.647	0.652	0.806	0.721	0.582	0.570	0.576	0.455	0.699	0.542
Table 2: The exact match precision*, recall* and F1-score* of the distantly supervised methods on the same datasets and tagging dictionaries. Boldface numbers indicate that this experiment with the left-side model achieves the best performance on one specific dataset among the three data conditions.
BC5CDR-UMLS	NCBI-UMLS
Positive	20,520	18,380
Noisy	11,719	10,199
Mask	28,305	24,652
Masked Noisy	6,290	5,345
FP	4,379	4,285
Masked FP	957	945
FN	6,627	5,264
Masked FN	4,975	4,085
All	148,721	124,250
Table 3: The statistics of tokens in test data by comparing the human annotation, distant annotation, and the refinement results of CReDEL.
and thus have a higher precision score.
5.2 Efficacy of Distant Label Refinement
This section conducts experiments to explain what CReDEL does when noisy annotations exist in distant labels. Specifically, we define the tokens whose labels are different in distant data and golden test data as noisy tokens. The tokens of "O" labels in test data refer to the negative tokens while the remaining tokens refer to the positive tokens. Then we count the positive tokens, noisy tokens, masked, and noise tokens masked by CReDEL. Besides, we also count the false-positive (FP) tokens and false-negative (FN) tokens as well as the masked FP and FN tokens.
For NCBI-Disease, there are 124,250 tokens in all sentences in the test set, while 18,380 tokens are positive, 10,199 are noisy. CReDEL masks 24,652
tokens, and 5,345 of them are noisy. The accuracy of correct masks is much higher than random masking, indicating that CReDEL is conducive to reducing noise. The results in BC5CDR are similar. In addition, CReDEL is better at handling the FN tokens than FP tokens since CReDEL masks  $77.6\%$  of FN tokens in NCBI-Disease, and only  $22.1\%$  FN tokens are masked. The above observations also explain the increase in recall for the DSNER model trained with the refined data.
For better understanding, the case study is shown in Table 4. In the first example, the phrase "ecopic intracranial retinoblastoma" appears in the golden annotation as a disease entity, while the distant annotation makes an incomplete annotation. CReDEL disagrees with the distant annotation and suggests masking this phrase correctly and masking "child with" redundantly. Correspondingly, the BOND trained with the refined data correctly predicts "ectopic intracranial retinoblastoma" and the BOND trained by original data makes mistakes. In the second example, the golden and distant annotation both label "levodopa" as a chemical entity and "dyskinesia" as a disease entity, while CReDEL masks all tokens in "levodopa-induced dyskinesia". The BOND trained with the refined data predicts the disease entity "levodopa-induced dyskinesia", which is better than the prediction of BOND trained with distant data. This case fits our parse-enhanced entity boundary modifying strategy, and both CRe
PMID:9400934 from NCBI-Disease
Golden	The RB1 gene mutation in a child with [ectopic intracranial retinoblastoma]DISEASE.
Distant	The RB1 gene mutation in a child with ectopic intracranial retinoblastoma.
CReDEL	The RB1 gene mutation in a childMASK withMASK ectopicMASK intracranialMASK retinoblastomAMASK.
BOND (Distant)	The RB1 gene mutation in a child with ectopic intracranial retinoblastoma.
BOND (CReDEL)	The RB1 gene mutation in a child with [ectopic intracranial retinoblastoma]DISEASE.
PMID:23952588 from BC5CDR
Golden	Risk factors and predictors of [levodopa]CHEMICAL-induced [dyskinesia]DISEASE among multiethnic Malaysians with [Parkinson's disease]DISEASE.
Distant	Risk factors and predictors of [levodopa]CHEMICAL-induced [dyskinesia]DISEASE among multiethnic Malaysians with [Parkinson's disease]DISEASE.
CReDEL	Risk factors and predictors of levodopaMASK-MASK-inducedMASK dyskinesiaMASK among multiethnicMASK MalaysiansMASK with [Parkinson's disease]DISEASE.
BOND (Distant)	Risk factors and predictors of levodopa-induced [dyskinesia]DISEASE among multiethnic Malaysians with [Parkinson's disease]DISEASE.
BOND (CReDEL)	Risk factors and predictors of [levodopa-induced dyskinesia]DISEASE among multiethnic Malaysians with [Parkinson's disease]DISEASE.
Table 4: Case study in NCBI-Disease and BC5CDR. The dashed line splits the table into data of golden, distant, and refined and prediction of BOND trained with distant data and data refined by CReDEL.
Methods	Prec.	Rec.	F1
BOND (CReDEL)	0.575	0.693	0.628
w/o typing	0.549	0.635	0.589
w. (Y1,P,Y1,N)	0.560	0.690	0.618
w. easy only	0.561	0.597	0.578
w. hard only	*	*	*
BOND w. parse after	0.550	0.385	0.453
RoSTER (CReDEL)	0.569	0.749	0.647
w/o typing	0.533	0.727	0.615
w. (Y1,P,Y1,N)	0.570	0.722	0.637
w. easy only	*	*	*
w. hard only	*	*	*
RoSTER w. parse after	0.403	0.379	0.391
Table 5: Here * represents model collapsing with strange predictions or not converging after ablation. The exact match F1-score of the CReDEL, the CReDEL w/o the entity typing module, the CReDEL trained using negative sample case  $(Y_{1,P}, Y_{1,N})$  and easy samples, the CReDEL trained using easy samples/hard samples only, and the DSNER method with parse improvement after it. The results are computed on the dataset BC5-UMLS by methods BOND and RoSTER.
DEL and the DSNER model trained with refined data prefer to predict complete entities.
To verify the effectiveness of each design of CReDEL, we conduct the ablation studies. We mainly discuss three parts, the first of which is the effectiveness of parse and it has been fully exploited in the results part. The second ablation is CReDEL w/o typing, that is, without the final entity typing module. The other part of ablation considers the selection of training sample cases. We train CReDEL (1) w.  $(Y_{1,P}, Y_{1,N})$ , using only sample pair  $(Y_{1,P}, Y_{1,N})$  in Section 3.2 and easy samples; (2)
w. easy only, as no hard samples are included; and (3) w. hard only, where only easy samples are used. The results are shown in Table 5. The model trained without easy samples simply cannot converge. Its predicted phrases are meaningless and in chaos. The data modified by CReDEL with only easy samples lead the RoSTER method to collapse with zeros or NaN in output probabilities and achieve low scores with BOND method. Other ablations also result in a drop in the F1 score, revealing the contributions of their corresponding part in the model design.
Finally we discuss the condition where the parser does not act on the training set of CReDEL but on the output of DSNER models without CReDEL, denoted as Method w. parse after. As an automatic refinement model, we should outperform the baseline refinement with Parser. We apply the same parse-enhancing rule and find the F1-score is far behind CReDEL as demonstrated in Table 5. Even adopting the new BI metric, this model also performs badly with 0.538 and 0.489 BI F1-score on method BOND and RoSTER respectively.
These drops can be explained. Without entity type classification, the model will mask much more phrases that are valid but beyond the dataset annotation types. The BOND or RoSTER models potentially learn to pay attention to these phrases, resulting in a decline in precision. The model trained with  $(Y_{1,P}, Y_{1,N})$ , on the other hand, tends to fit the positive samples  $Y_{1,P}$  which are fixed by dictionary annotations. The BERT model may recognize these words in some batches, and the trained CReDEL can lose generalization abilities. If using
Parser after DSNER methods, it is only a rigid tool rather than a model exploiting semantic information. The CReDEL with Parser takes effect because it is combined with our other designs.
This paper proposes a novel approach to automatically learn the refinement knowledge of distantly annotated NER labels and modify the distant labels to enhance DSNER models. The proposed method consists of a contrastive samples generation module, a contrastive training procedure, and a distantly label improving strategy. Experiments demonstrate that our method consistently and significantly improves DSNER and NER models on distantly annotated NER data, and it can be stably applied to all the datasets and methods.
Upon the completion of this paper, we would like to thank Zheng Yuan and Hongyi Yuan for their helpful comments and suggestions. We also gratefully acknowledge the support of Jun Xia and his providing of raw data.
Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk. 2016. Learning local feature descriptors with triplets and shallow convolutional neural networks. In *Bmvc*, volume 1, page 3.
Kevin Bowden, Jiaqi Wu, Shereen Oraby, Anita Misra, and Marilyn Walker. 2018. Slugnerds: A named entity recognition tool for open domain dialogue systems. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 740-750.
Bowen Cheng, Ross Girshick, Piotr Dólár, Alexander C Berg, and Alexander Kirillov. 2021. Boundary iou: Improving object-centric image segmentation evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15334-15342.
Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 1, pages 539-546. IEEE.
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J Passonneau, and Rui Zhang. 2021. Container: Few-shot named entity recognition via contrastive learning. arXiv preprint arXiv:2109.07589.
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1-10.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Yan Jia, Yulu Qi, Huaijun Shang, Rong Jiang, and Aiping Li. 2018. A practical approach to constructing a knowledge graph for cybersecurity. Engineering, 4(1):53-60.
Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. 2021. Biomedical question answering: A comprehensive review. arXiv preprint arXiv:2102.05281.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. 2020. Hard negative mixing for contrastive learning. arXiv preprint arXiv:2010.01028.
Mahboob Alam Khalid, Valentin Jijkoun, and Maarten De Rijke. 2008. The impact of named entity normalization on information retrieval for question answering. In European Conference on Information Retrieval, pages 705-710. Springer.
Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. 2020. Contrastive representation learning: A framework and review. IEEE Access.
Jiao Li, Yueping Sun, R Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2015. Annotating chemicals, diseases, and their interactions in biomedical literature. In Proceedings of the fifth BioCreative challenge evaluation workshop, pages 173-182. The Fifth BioCreative Organizing Committee.
Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020. Bond: Bert-assisted open-domain named entity recognition with distant supervision. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1054-1064.
Bill Yuchen Lin, Dong-Ho Lee, Ming Shen, Ryan Moreno, Xiao Huang, Prashant Shiralkar, and Xiang Ren. 2020. Triggerner: Learning with entity triggers as explanations for named entity recognition. arXiv preprint arXiv:2004.07493.
Donald AB Lindberg, Betsy L Humphreys, and Alexa T McCray. 1993. The unified medical language system. Yearbook of Medical Informatics, 2(01):41-51.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Zhuang Ma and Michael Collins. 2018. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv:1809.01812.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Xuan Wang, Yu Zhang, Heng Ji, and Jiawei Han. 2021. Distantly-supervised named entity recognition with noise-robust learning and language model augmented self-training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10367-10378.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2, ACL '09, page 1003-1011, USA. Association for Computational Linguistics.
Aurelie Néveol, Antonio Jimeno Yepes, L Neves, and Karin Verspoor. 2018. Parallel corpora for the biomedical domain. In International Conference on Language Resources and Evaluation.
Yasumasa Onoe and Greg Durrett. 2019. Learning to denoise distantly-labeled data for entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuan-Jing Huang. 2019. Distantly supervised named entity recognition using positive-unlabeled learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2409-2419.
Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, pages 2054-2064. Association for Computational Linguistics.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2020. Automated concatenation of embeddings for structured prediction. arXiv preprint arXiv:2010.05006.
Yaosheng Yang, Wenliang Chen, Zhenghua Li, Zhengqiu He, and Min Zhang. 2018. Distantly supervised ner with partial annotation learning and reinforcement learning. In Proceedings of the 27th
International Conference on Computational Linguistics, pages 2159-2169.
Zheng Yuan, Zhengyun Zhao, Haixia Sun, Jiao Li, Fei Wang, and Sheng Yu. 2020. Coder: Knowledge infused cross-lingual medical term embedding for term normalization. arXiv preprint arXiv:2011.02947.
Wenkai Zhang, Hongyu Lin, Xianpei Han, Le Sun, Huidan Liu, Zhicheng Wei, and Nicholas Yuan. 2021. Denoising distantly supervised named entity recognition via a hypergeometric probabilistic model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14481-14488.
Zhanfang Zhao, Sung-Kook Han, and In-Mi So. 2018. Architecture of knowledge graph construction techniques. International Journal of Pure and Applied Mathematics, 118(19):1869-1883.
A.1 Processing of UMLS
The Unified Medical Language System (UMLS) (Lindberg et al., 1993) is a large-scale resource containing over 4 million unique medical concepts. The restricted set of source ontologies in UMLS include "CPT", "MEDLINEPLUS", "RXNORM", "SNOMED-CT" and so on (Table 6). After that, we apply necessary cleanings such as removing ambiguous or illegal words and abbreviations that are prone to mismatch.
Source	Name or Explanation
CPT	Current Procedural Terminology
HPO	Human Phenotype Ontology
MEDLINEPLUS	Certified patient-oriented web-content
MSH	Medical Subject Headings
MTH	UMLS Metathesaurus Names
NCI	National Cancer Institute Thesaurus
RXNORM	NLM's Nomenclature for Clinical Drugs for Humans
SNOMEDCT	US edn. of the Systematized Nomenclature of Medicine-Clinical Terms
Table 6: The restricted set of source ontologies.
A.2 Parse-based Entity Boundary Modifying
The parser will give a syntax tree of a sentence, with the tokens in the sentence as leaves and syntax labels as other nodes. We set four rules for the parse modifying: (1) If one token is within an entity, we find its parent in the syntax tree generated by the Stanford parser and tag the whole new noun phrase as an entity in the "BIO" tagging scheme if the parent label is "Noun Phrase". (2) If the new entity in (1) starts with a comparative form
of an adjective, cardinal number, conjunction, or pronoun, then we repeatedly delete the first token until it does not belong to one of those labels. (3) If the extension of the original entity encounters conjunction or some specific punctuation, then we reserve the original entity. (4) If none of the tokens in an original entity is a noun word and none of the parent labels is "Noun Phrase", then we delete the entity from the original tag sequence.
A.3 Entity Typing Module
In the entity typing module, we train a BERT-base classifier with the distant data labeled by the same knowledge base. The classifier takes an entity and its surrounding words as input. Then it predicts the entity type of the input entity. In training, the entity type is from the knowledge base, and the model updates with cross-entropy loss.