HCU at the NTCIR-12 MedNLPDoc Task
Ayumu Hiramae

Hidetsugu Nanba

Toshiyuki Takezawa

Hiroshima City University
hiramae@ls.info.hiroshima-cu.ac.jp

Hiroshima City University
nanba@hiroshima-cu.ac.jp

Hiroshima City University
takezawa@hiroshima-cu.ac.jp

ABSTRACT
Our team, HCU, participated in the Phenotyping Task. We tackled
the task by using machine learning techniques and edit distance by
DP matching. This paper outlines the methods we used to obtain
the result evaluated by the task organizer.

Keywords

Table 1 Examples of words that frequently appear in diseasedatabases
Databaseconditional random fields (CRF), edit distance, DP matching.

ICD-101. INTRODUCTION

MEDISExamples of words,

We have studied automatic construction of an ontology by
extracting technical terms from various texts, such as researchpatents [1] and cooking recipes [2]. Our goal in NTCIR-12 [3]was to evaluate our method for technical term extraction.
This paper is organized as follows. Section 2 explains the methodfor extracting diseases from a given medical record and forassigning ICD-10 codes. To investigate the effectiveness of ourmethod, we conducted experiments, as reported in Section 3. We
describe related work in Section 4, and Section 5 presents ourconclusions.

2. OUR METHOD
Our method consists of the following two steps.
1.

Extraction of diseases from a given medical record.

2.

Assignment of ICD-10 codes to diseases extracted in Step 1.

In the following sections, we describe these steps.

2.1 Extraction of Diseases from a Given
Medical Record
We formalized the task as a sequential labeling problem, andemployed conditional random field (CRF) as a machine learning
method using the following features; (we use MeCab as aJapanese morphological analysis tool to identify the part ofspeech):,,Frequency ineach database,5,181(skin),(foot),(hand),(knee),(abdomen)

1,228

2.2 Assignment of ICD-10 Codes
We calculated the edit distance [4] between a disease nameextracted in Step 1 and each disease name in ICD-10, andassigned the ICD code having the highest similarity.

3. EXPERIMENTS
3.1 Extraction of Diseases from a Given
Medical Record
We evaluated our method using NTCIR-10 MedNLP [4] andNTCIR-11 MedNLP-2 [5]. The datasets of MedNLP andMedNLP-2 consist of 45 and 81 records, respectively. We
conducted 5-fold cross validation. We evaluated our method using
precision, recall, and F-measure. Table 2 shows the experimentalresults. As can be seen, our method obtained high precision scoresin both datasets.
Table 2 Experimental results for the extraction of diseases
precision

recallF-measure

MedNLP

0.815

0.701

0.754

MedNLP-2

0.919

0.854

0.885

la word;lits part of speech;

3.2 Assignment of ICD-10 Codeslwhether the word frequently appears in disease names inICD-101;

We submitted one system result. The experimental results at threelevels, SURE, MAJOR, and POSSIBLE, are shown in Tables 3, 4,and 5, respectively.

lwhether the word frequently appears in disease names inthe MEDIS database2.

Table 3 Official evaluation result (SURE)

Examples of words that appear in ICD-10 and MEDIS are shownin Table 1.

Precision
0.209

Recall
0.364

F-measure
0.266

Table 4 Official evaluation result (MAJOR)
Precision
0.361
1

http://www.dis.h.u-tokyo.ac.jp/byomei/icd10/2

http://www2.medis.or.jp/master/syoken/

Recall
0.363

F-measure
0.362

Table 5 Official evaluation result (POSSIBLE)
Precision
0.420

Recall0.230

F-measure
0.297

The experimental results showed that our method obtained low
precision scores, which indicate that edit distance was noteffective. In future, we will examine other similarity methods. We
will also examine a method that takes account of words that occur
in the context of diseases.

3.3 Discussion
There are two typical errors in our system, as described inSections 3.3.1 and 3.3.2.

3.3.1 Main topics in medical records
In the following sentence, our system assigned a code “C187” to“S(sigmoid colon cancer).” However, our systemshould not assign a code to this string, because this medical record
was written in 2005, and most of this record focuses on theoperations in 2005. The following sentence is only the explanation
for the patient’s past disease, which is not the main topic of thisrecord.
SAs another approach for resolving the same problem, we willcollect synonyms of diseases by using a statistical machinetranslation (SMT) technique. Zhou et al. [7] obtained synonyms
automatically using the SMT technique. If translations of twoterms X and Y are the same, then the terms X and Y areconsidered paraphrases. Based on this idea, they automaticallyobtained synonyms from a translation model, which was createdfrom pairs of English and Chinese sentences, and they used it forevaluation of text summarization.
In other Wikification research, McNamee et al. [8] proposed amultilingual named-entity linking method, which links concepts in
multiple languages onto English Wikipedia pages. In the future,following McNamee et al., we will conduct multilingual entity
linking.

5. CONCLUSION
We developed a CRF-based and edit-distance-based system, andsubmitted the result to the Phenotyping Task. Our method
obtained F-scores of 0.266, 0.362, and 0.297 at SURE, MAJOR,and MAJOR levels, respectively. We will incorporate ideas from
Wikification and SMT technique to overcome the MedNLP task.

6. REFERENCES[1](In April, 1999, Hartmann’s operation for sigmoid coloncancer.)

Therefore, we need to detect the main topic (disease or operation)in each record, and assign the corresponding code.

[2]

3.3.2 Mismatch between automatically extracted
disease names and index terms in ICD-10
In the following sentence, our system identified “(tumor)” asa disease, and assigned a code “D489”. However, in this case, theICD-10 code “D430” for “(frontal lobe tumor)”should be assigned.

(The tumor is found in the right frontal lobe by head CT.)
ICD-10 codes differ according to the body site in which thesymptom occurs. Therefore, we need to target nouns that occur inthe context of diseases because they are related phrases such as abody parts.

4. RELATED WORK
Wikification is a related task, in which the procedure of consistsof the following two steps: (1) identification of concepts and (2)the linking of concepts to the corresponding Wikipedia pages. In
the MedNLP task, these steps correspond to (1)’ extraction ofdiseases from a medical record and (2)’ assigning the ICD-10
codes. Illinois Wikifier3[6] used Integer Linear Programming(ILP) for Wikification. The system maximizes the objectivefunction assigning concepts to Wikipedia titles with relations
extracted from both a document and an external knowledge base.
If we consider a medical record and the external medical resourcein Section 2.1 as an input document and Knowledge Base used inWikifier, we can apply this Wikifier approach to our task; resolve
the problem mentioned in Section 3.3.2.
3

https://cogcomp.cs.illinois.edu/page/software_view/Wikifier

[3]
[4]

[5]
[6]
[7]

[8]

Nanba, H., Mayumi, S., and Takezawa, T. (2011) AutomaticConstruction of a Bilingual Thesaurus using Citation
Analysis. In Proceedings of the 4th International CIKM
Workshop on Patent Information Retrieval (PaIR'11), pp.2530.
Nanba, H., Doi, Y., Takezawa, T., Sumiya, K., and Tsujita,
M. (2014) Construction of a Cooking Ontology fromCooking Recipes and Patents. In Proceedings of Workshop
on Smart Technology for Cooking and Eating Activities:
CEA2014.
Aramaki, E., Morita, M., Kano, Y., and Ohkuma, T., (2016)
Overview of the NTCIR-12 MedNLPDoc Task. In
Proceeding of 12th NTCIR Workshop Meeting.
Morita, M., Kono, Y., Ohkuma, T., Miyabe, M., andAramaki, E. (2013) Overview of the NTCIR-10 MedNLP
task. In Proceeding of 10th NTCIR Workshop Meeting,pp.696-701.
Aramaki, E., Morita, M., Kano, Y., and Ohkuma, T., (2014).
Overview of the NTCIR-11 MedNLP-2 Task. In Proceedingof 11th NTCIR Workshop Meeting, pp.147-154.
Cheng, X., and Roth, D. (2013) Relational inference forwikification. In Proceedings of EMNLP.
Zhou, L., Lin, C-Y., Munteanu, D.S, and Hovy, E. (2006)
ParaEval: Using Paraphrases to Evaluate Summaries
Automatically. In proceeding of HLT-NAACL2006,pp.447-454.
McNamee, P., Mayfield, J., Lawrie, D., Oard, D. W., andDoermann, D. S. (2011) Cross-Language Entity Linking.
In Proceedings of IJCNLP pp.255-263.