SlowGuess/ACL_2020_OCR huggingface dataset:
The graph-based model achieves highest precision, but with substantially lower recall, therefore obtaining lowest  $F_{1}$  scores. In contrast, our model improves recall over flat and BIO extension models as well as previously reported results, without sacrificing precision. This results in more balanced precision and recall. Improved recall is especially encouraging for our motivating pharmacovigilance and medical record summarization applications, where recall is at least as important as precision.
Effectiveness on recognizing discontinuous mentions Recall that only  $10\%$  of mentions in these three data sets are discontinuous. To evaluate the effectiveness of our proposed model on recognizing discontinuous mentions, we follow the evaluation approach in (Muis and Lu, 2016) where we construct a subset of test set where only sentences with at least one discontinuous mention are included (Left part of Table 3). We also report the evaluation results when only discontinuous mentions are considered (Right part of Table 3). Note that sentences in the former setting usually contain continuous mentions as well, including those involved in overlapping structure (e.g., 'muscle pain' in the sentence 'muscle pain and fatigue'). Therefore, the flat model, which cannot predict any discontinuous mentions, still achieves  $38\% F_1$  on average when evaluated on these sentences with at least one discontinuous mention, but  $0\% F_1$  when evaluated on discontinuous mentions only.
Our model again achieves the highest  $F_{1}$  and recall in all three data sets under both settings. The comparison between these two evaluation results also shows the necessity of comprehensive evaluation settings. The BIO E. model outperforms the graph-based model in terms of  $F_{1}$  score on CADEC, when evaluated on sentences with discontinuous mentions. However, it achieves only  $1.8F_{1}$  when evaluated on discontinuous mentions only. The main reason is that most of discontinuous mentions in CADEC are involved in overlapping
5865
Model	CADEC	ShARe 13	ShARe 14
P	R	F	P	R	F	P	R	F
(Metke-Jimenez and Karimi, 2016)	64.4	56.5	60.2	-	-	-	-	-	-
(Tang et al., 2018)	67.8	64.9	66.3	-	-	-	-	-	-
(Tang et al., 2013b)	-	-	-	80.0	70.6	75.0	-	-	-
Flat	65.3	58.5	61.8	78.5	66.6	72.0	76.2	76.7	76.5
BIO Extension	68.7	66.1	67.4	77.0	72.9	74.9	74.9	78.5	76.6
Graph	72.1	48.4	58.0	83.9	60.4	70.3	79.1	70.7	74.7
Ours	68.9	69.0	69.0	80.5	75.0	77.7	78.1	81.2	79.6
Table 2: Evaluation results on the whole test set in terms of precision, recall and  $F_{1}$  score. The original ShARe 14 task focuses on template filling of disorder attributes: that is, given a disorder mention, recognize the attribute from its context. In this work, we use its mention annotations and frame the task as a discontinuous NER task.
Sentences with discontinuous mentions	Discontinuous mentions only
CADEC	ShARE 13	ShARE 14	CADEC	ShARE 13	ShARE 14
Model	P	R	F	P	R	F	P	R	F	P	R	F	P	R	F	P	R	F
Flat	50.2	36.7	42.4	43.5	28.1	34.2	41.5	31.9	36.0	0	0	0	0	0	0	0	0	0
BIO E.	63.8	52.0	57.3	51.8	39.5	44.8	37.5	38.4	37.9	5.8	1.0	1.8	39.7	12.3	18.8	8.8	4.5	6.0
Graph	69.5	43.2	53.3	82.3	47.4	60.2	60.0	52.8	56.2	60.8	14.8	23.9	78.4	36.6	50.0	42.7	39.5	41.1
Ours	66.5	64.3	65.4	70.5	56.8	62.9	61.9	64.5	63.1	41.2	35.1	37.9	78.5	39.4	52.5	56.1	43.8	49.2
Table 3: Evaluation results on sentences that contain at least one discontinuous mention (left part) and on discontinuous mentions only (right part).
structure (88%, cf. Table 1), and the BIO E. model is better than the graph-based model at recognizing these continuous mentions. On ShARE 13 and 14, where the portion of discontinuous mentions involved in overlapping is much less than on CADEC, the graph-based model clearly outperforms BIO E. model in both evaluation settings.
We start our analysis from characterizing discontinuous mentions from the three data sets. Then we measure the behaviors of our model and two discontinuous NER models on the development sets based on characteristics identified and attempt to draw conclusions from these measurements.
7.1 Characteristics of Discontinuous Mentions
Recall that discontinuous mentions usually represent compositional concepts that consist of multiple components. Therefore, discontinuous mentions are usually longer than continuous mentions (Table 1). In addition, intervals between components make the total length of span involved even longer. Previous work shows that flat NER performance degrades when applied on long mentions (Augenstein et al., 2017; Xu et al., 2017).
Another characteristic of discontinuous men
tions is that they usually overlap (cf. § 1). From this perspective, we can categorize discontinuous mentions into four categories:
- No overlap: in such cases, the discontinuous mention can be intervened by severity indicators (e.g., 'is mildly' in sentence 'left atrium is mildly dilated'), preposition (e.g., 'on my' in sentence "...rough on my stomach...') and so on. This category accounts for half of discontinuous mentions in the ShARE data sets but only  $12\%$  in CADEC (Table 1).
- Left overlap: the discontinuous mention shares one component with other mentions, and the shared component is at the beginning of the discontinuous mention. This is usually accompanied with coordination structure (e.g., the shared component 'muscle' in 'muscle pain and fatigue'). Conjunctions (e.g., 'and', 'or') are clear indicators of the coordination structure. However, clinical notes are usually written by practitioners under time pressure. They often use commas or slashes rather than conjunctions. This category accounts for more than half of discontinuous mentions in CADEC and one third in ShARE.
- Right overlap: similar to left overlap, although the shared component is at the end. For ex
5866
(a) CADEC
(b) ShARe 13
(c) ShARe 14
(d) CADEC
(e) ShARe 13
(f) ShARe 14
Figure 4: The impact of mention length and interval length on recall. Mentions with interval length of zero are continuous mentions. Numbers in parentheses are the number of gold mentions.
ample, 'hip/leg/foot pain' contains three mentions that share 'pain'.
- Multi-overlap: the discontinuous mention shares multiple components with the others, which usually forms crossing compositions. For example, the sentence 'Joint and Muscle Pain / Stiffness' contains four mentions: 'Joint Pain', 'Joint Stiffness', 'Muscle Stiffness' and 'Muscle Pain', where each discontinuous mention share two components with the others.
7.2 Impact of Overlapping Structure
Previous study shows that the intervals between components can be problematic for coordination boundary detection (Ficler and Goldberg, 2016). Conversely, we want to observe whether the overlapping structure may help or hinder discontinuous entity recognition. We categorize discontinuous mentions into different subsets, described in § 7.1, and measure the effectiveness of different discontinuous NER models on each category.
From Table 4, we find that our model achieves better results on discontinuous mentions belonging to 'No overlap' category on ShARE 13 and 14, and 'Left overlap' category on CADEC and ShARE 14. Note that 'No overlap' category accounts for half of discontinuous mentions in ShARE 13 and 14, whereas 'Left overlap' accounts for half in CADEC (Table 1). Graph-based model achieves better results on 'Right overlap' category. On the 'Multi-Overlap' category, no models is effective, which
emphasizes the challenges of dealing with this syntactic phenomena. We note, however, the portion of discontinuous mentions belonging to this category is very small in all three data sets.
Although our model achieves better results on 'No overlap' category on ShARE 13 and 14, it does not predict correctly any discontinuous mention belonging to this category on CADEC. The ineffectiveness of our model, as well as other discontinuous NER models, on CADEC 'No overlap' category can be attributed to two reasons: 1) the number of discontinuous mentions belonging to this category in CADEC is small (around  $12\%$ ), rending the learning process more difficult. 2) the gold annotations belonging to this category are inconsistent from a linguistic perspective. For example, severity indicators are annotated as the interval of the discontinuous mention sometimes, but not often. Note that this may be reasonable from a medical perspective, as some symptoms are roughly grouped together no matter their severity, whereas some symptoms are linked to different concepts based on their severity.
7.3 Impact of Mention and Interval Length
We conduct experiments to measure the ability of different models on recalling mentions of different lengths, and to observe the impact of interval lengths. We found that the recall of all models decreases with the increase of mention length in general (Figure 4 (a - c)), which is similar to previous observations in the literature on flat men
5867
CADEC	ShARE 13	ShARE 14
Model	#	F	#	F	#	F
No 0	BIO E.	9	0.0	41	7.5	39	0.0
Graph	9	0.0	32.1	45.2
Ours	0.0	36.1	57.1
Left 0	BIO E.	54	6.0	11	25.0	30	15.7
Graph	9.2	45.5	37.7
Ours	28.6	33.3	49.2
Right 0	BIO E.	16	0.0	19	0.0	5	0.0
Graph	45.2	21.4	0.0
Ours	29.3	13.3	0.0
Multi 0	BIO E.	15	0.0	0	-	6	0.0
Graph	0.0	-	0.0
Ours	0.0	-	0.0
Table 4: Evaluation results on different categories of discontinuous mentions.  $\#$  columns show the number of gold discontinuous mentions in development set of each category.  $①$  : overlap.
tions. However, the impact of interval length is not straightforward. Mentions with very short interval lengths are as difficult as those with very long interval lengths to be recognized (Figure 4 (d - f)). On CADEC, discontinuous mentions with interval length of 2 are easiest to be recognized (Figure 4 (d)), whereas those with interval length of 3 are easiest on ShARE 13 and 14. We hypothesize this also relates to annotation inconsistency, because very short intervals may be overlooked by annotators.
In terms of model comparison, our model achieves highest recall in most settings. This demonstrates our model is effective to recognize both continuous and discontinuous mentions with various lengths. In contrast, the BIO E. model is only strong at recalling continuous mentions (outperforming the graph-based model), but fails on discontinuous mentions (interval lengths  $>0$ ).
7.4 Example Predictions
We find that previous models often fail to identify discontinuous mentions that involve long and overlapping spans. For example, the sentence 'Severe joint pain in the shoulders and knees.' contains two mentions: 'Severe joint pain in the shoulders' and 'Severe joint pain in the knees'. Graph-based model does not identify any mention from this sentence, resulting in a low recall. The BIO extension model predicts most of these tags (8 out of 9) correctly, but fails to decode into correct mentions (predict 'Severe joint pain in the', resulting in a false positive, while it misses 'Severe joint pain in the shoulders'). In contrast, our model correctly identifies both of these two mentions.
No model can fully recognize mentions which form crossing compositions. For example, the sentence 'Joint and Muscle Pain / Stiffness' contains four mentions: 'Joint Pain', 'Joint Stiffness', 'Muscle Stiffness' and 'Muscle Pain', all of which share multiple components with the others. Our model correctly predicts 'Joint Pain' and 'Muscle Pain', but it mistakenly predicts 'Stiffness' itself as a mention.
8 Summary
We propose a simple, effective transition-based model that can recognize discontinuous mentions without sacrificing the accuracy on continuous mentions. We evaluate our model on three biomedical data sets with a substantial number of discontinuous mentions. Comparing against two existing discontinuous NER models, our model is more effective, especially in terms of recall.
We would like to thank Danielle Mowery for helping us to obtain the ShARE data sets. We also thank anonymous reviewers for their insightful comments. Xiang Dai is supported by Sydney University's Engineering and Information Technologies Research Scholarship as well as CSIRO's Data61 top up scholarship.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In COLING, pages 1638-1649, Santa Fe, New Mexico.
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications. In SemEval, pages 546-555, Vancouver, Canada.
Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In EMNLP, pages 740-750, Doha, Qatar.
Ronald Cornet and Nicolette de Keizer. 2008. Forty years of SNOMED: a literature review. In *BMC Med Inform Decis Mak*, volume 8, page S2.
Xiang Dai. 2018. Recognizing complex entity mentions: A review and future directions. In ACL@SRW, pages 37-44, Melbourne, Australia.
Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2019. Using similarity measures to select pretraining data for ner. In *NAACL*, pages 1460–1470, Minneapolis, Minnesota.
5868
Xiang Dai, Sarvnaz Karimi, and Cecile Paris. 2017. Medication and adverse event extraction from noisy text. In ALTA, pages 79-87, Brisbane, Australia.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*, pages 4171–4186, Minneapolis, Minnesota.
Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In ACL-IJCNLP, pages 334-343, Beijing, China.
Joshua C Feblowitz, Adam Wright, Hardeep Singh, Lipika Samal, and Dean F Sittig. 2011. Summarization of clinical information: a conceptual model. *J Biomed Inform*, 44(4):688-699.
Jessica Ficler and Yoav Goldberg. 2016. A neural network for coordination boundary prediction. In EMNLP, pages 23-32, Austin, Texas.
Jenny Rose Finkel and Christopher Manning. 2009. Nested named entity recognition. In EMNLP, pages 141-150, Singapore.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In ICASSP, pages 6645-6649.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. MIMIC-III, A freely accessible critical care database. Sci. Data, 3:160035.
Sarvnaz Karimi, Alejandro Metke-Jimenez, Madonna Kemp, and Chen Wang. 2015a. CADEC: A corpus of adverse drug event annotations. J Biomed Inform, 55:73-81.
Sarvnaz Karimi, Chen Wang, Alejandro Metke-Jimenez, Raj Gaire, and Cecile Paris. 2015b. Text and data mining techniques in adverse drug reaction detection. ACM Comput. Surv., 47(4):56:1-56:39.
Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In *NAACL*, pages 861-871, New Orleans, Louisiana.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL, pages 260-270, San Diego, California.
Robert Leaman, Ritu Khare, and Zhiyong Lu. 2015. Challenges in clinical natural language processing for automated disorder normalization. J Biomed Inform, 57:28-37.
Robert Leaman, Laura Wojtulewicz, Ryan Sullivan, Annie Skariah, Jian Yang, and Graciela Gonzalez. 2010. Towards internet-age pharmacovigilance: Extracting adverse drug reactions from user posts in health-related social networks. In BioNLP, pages 117-125, Uppsala, Sweden.
Yinxia Lou, Yue Zhang, Tao Qian, Fei Li, Shufeng Xiong, and Donghong Ji. 2017. A transition-based joint model for disease named entity recognition and normalization. Bioinformatics, 33(15):2363-2371.
Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In EMNLP, pages 857-867, Lisbon, Portugal.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP, pages 1412-1421, Lisbon, Portugal.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In ACL, pages 1064-1074, Berlin, Germany.
Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Flexible text segmentation with structured multilabel classification. In EMNLP, pages 987-994, Vancouver, British Columbia, Canada.
Alejandro Metke-Jimenez and Sarvnaz Karimi. 2016. Concept identification and normalisation for adverse drug event discovery in medical forums. In BM-DID@ISWC, Kobe, Japan.
Danielle L Mowery, Sumithra Velupillai, Brett R South, Lee Christensen, David Martinez, Liadh Kelly, Lorraine Goeuriot, Noemie Elhadad, Sameer Pradhan, Guergana Savova, and Wendy W Chapman. 2014. Task 2: ShARE/CLEF ehealth evaluation lab 2014. In CLEF.
Aldrian Obaja Muis and Wei Lu. 2016. Learning to recognize discontinuous entities. In EMNLP, pages 75-84, Austin, Texas.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In *NAACL*, pages 2227-2237, New Orleans, Louisiana.
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In RepL4NLP@ACL, pages 7-14, Florence, Italy.
Sameer Pradhan, Noemie Elhadad, Brett R South, David Martinez, Lee M Christensen, Amy Vogel, Hanna Suominen, Wendy W Chapman, and Guergana K Savova. 2013. Task 1: ShARE/CLEF ehealth evaluation lab 2013. In CLEF.
Nicky Ringland, Xiang Dai, Ben Hachey, Sarvnaz Karimi, Cecile Paris, and James R. Curran. 2019. NNE: A dataset for nested named entity recognition
5869
in english newswire. In ACL, pages 5176-5181, Florence, Italy.
Abeed Sarker, Rachel Ginn, Azadeh Nikfarjam, Karen O'Connor, Karen Smith, Swetha Jayaraman, Tejaswi Upadhaya, and Graciela Gonzalez. 2015. Utilizing social media data for pharmacovigilance: a review. J Biomed Inform, 54:202-212.
Wei Shen, Jianyong Wang, and Jiawei Han. 2015. Entity linking with a knowledge base: Issues, techniques, and solutions. IEEE Trans. Knowl. Data Eng., 27(2):443-460.
Mohammad Golam Sohrab and Makoto Miwa. 2018. Deep exhaustive model for nested named entity recognition. In EMNLP, pages 2843-2849, Brussels, Belgium.
Gabriel Stanovsky, Daniel Gruhl, and Pablo Mendes. 2017. Recognizing mentions of adverse drug reaction in social media using knowledge-infused recurrent models. In EACL, pages 142-151, Valencia, Spain.
Pontus Stenetorp, Sampo Pyysalo, Goran Topic, Tomoko Ohta, Sophia Ananiadou, and Jun'ichi Tsujii. 2012. brat: a web-based tool for NLP-assisted text annotation. In EACL, pages 102-107, Avignon, France.
Buzhou Tang, Hongxin Cao, Yonghui Wu, Min Jiang, and Hua Xu. 2013a. Recognizing clinical entities in hospital discharge summaries using structural support vector machines with word representation features. In BMC Med Inform Decis Mak, volume 13, page S1.
Buzhou Tang, Jianglu Hu, Xiaolong Wang, and Qingcai Chen. 2018. Recognizing continuous and discontinuous adverse drug reaction mentions from social media using LSTM-CRF. Wirel Commun Mob Com, 2018.
Buzhou Tang, Yonghui Wu, Min Jiang, and Joshua C. Denny. 2013b. Recognizing and encoding disorder concepts in clinical text using machine learning and vector space model. In CLEF.
Bailin Wang and Wei Lu. 2019. Combining spans into entities: A neural two-stage approach for recognizing discontinuous entities. In EMNLP, pages 6215-6223, Hong Kong, China.
Bailin Wang, Wei Lu, Yu Wang, and Hongxia Jin. 2018a. A neural transition-based model for nested mention recognition. In EMNLP, pages 1011-1017, Brussels, Belgium.
Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, and Hongfang Liu. 2018b. Clinical information extraction applications: A literature review. J Biomed Inform, 77:34-49.
Taro Watanabe and Eiichiro Sumita. 2015. Transition-based neural constituent parsing. In ACL-IJCNLP, pages 1169-1179, Beijing, China.
Mingbin Xu, Hui Jiang, and Sedtawut Watcharawittayakul. 2017. A local detection approach for named entity recognition and mention detection. In ACL, pages 1237-1247, Vancouver, Canada.