NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

NAISTSOC at the NTCIR-16 Real-MedNLP Task
Tomohiro Nishiyama

Mihiro Nishidani

Aki Ando

Nara Institute of Science andTechnology
Japannishiyama.tomohiro.ns5@is.naist.jp

Nara Institute of Science andTechnology
Japannishidani.mihironl5@is.naist.jp

Nara Institute of Science andTechnology
Japanando.aki.ab9@is.naist.jp

Shuntaro Yada

Shoko Wakamiya

Eiji Aramaki

Nara Institute of Science andTechnology
Japans-yada@is.naist.jp

Nara Institute of Science andTechnologyJapanwakamiya@is.naist.jp

Nara Institute of Science andTechnologyJapanaramaki@is.naist.jp

ABSTRACT1) and guideline learning (subtask 2). The task is classified
by the amount of training data (small data or no data)
Applications (Subtask 3) : This challenge was designed fromthe practical viewpoints. For case reports, the organizers designed an information extraction task for adverse drug event(ADE). The ADE task has been challenged through workshops (n2c2 2009 etc.). For radiography reports, organizers
designed case identification task, which is to detect the reports originated from the same patient.

This paper describes how we tackled the Medical Natural Language
Processing for Real-MedNLP task as participants of NTCIR16. We
utilized BERT model for solving this task. We found that BERT
model we trained is the best results of subtask 1 with joint-F1-score.

KEYWORDS
Medical Natural Language Processing, Named Entity Recognition,
Case Reports, Radiography Reports, Adverse Drug Event, Case
Identification

Further details of this task can be found in the NTCIR Real-MedNLP
website1 and the NTCIR-16 Real-MedNLP task overview paper[5].
We challenged all subtasks for Japanese corpora (Subtask1-CRJA, Subtask1-RR-JA, Subtask2-CR-JA, Subtask2-RR-JA, Subtask3CR-JA (ADE), and Subtask3-RR-JA (CI)) by constructing BERTbased models. The results for some parts of subtask 1 and subtask
3 were superior to those of the other teams.
Our policy for this challenge is to investigate the feasibility ofthe most standard approaches. Thus, the fundamental design ofeach system is based on the standard approach, which does not
utilize any extra resources.
In terms of pre-trained models, several domain specific pretrained models are already developed, such as BioBERT[4] andclinical BERT[2] in English, UTH-BERT [3] in Japanese. Althoughthese domain-specific models are expected to yield better results inthese tasks, in this paper, we use BERT, a more basic, non-domainspecific pre-training model, to evaluate the model as a baseline.

TEAM NAME
NAISTSOCSUBTASKS
Subtask1-CR-JA
Subtask1-RR-JA
Subtask2-CR-JA
Subtask2-RR-JA
Subtask3-CR-JA (ADE)
Subtask3-RR-JA (CI)1

INTRODUCTION

In recent years, medical records have increasingly been converted
from paper to electronic format, increasing the importance of information processing technology in the medical field. However,privacy-free medical text data is still scarce in non-English speaking countries such as Japan and China.
In this Real-MedNLP task, organizers are restructuring the scheme
toward the ultimate goal (so-called medical AI task) of promoting
and supporting the development of practical tools and systems forthe medical industry to assist medical decisions and treatment byphysicians and co-medicals.
This task provides two core resources; (1) Case-Report corpus(shortly MedTxt-CR) and (2) Radiology-Report corpus (shortlyMedTxt-RR). The challenges of this task are two folds as follows.

2 METHODS
2.1 Subtask 1
BERT is a Transformers-based large-scale language model thatemerged in 2019[1]. It has been very successful in achieving highaccuracy on tasks in a variety of domains by fine-tuning the pretrained model. We approached this task by employing a simple
method that utilizes BERT.
We used the BERT pre-training model ‘cl-tohoku/bert-base-japanesewhole-word-masking,’2 The tokenizer is Mecab3 . Fine tuning was
performed with a batch size of 32 and epochs of 20 (Subtask 1). The
number of epochs was decided by preliminary experiments that weFew-resource NER (Subtasks 1 & 2) : Participants extract important information from the real medical texts. This challenge is classified into two ways: just 100 training (subtask

1 https://sociocom.naist.jp/real-mednlp/
2 https://huggingface.co/cl-tohoku/bert-base-japanese-whole-word-masking
3 https://taku910.github.io/mecab/

330

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Figure 1: Subtask 3: ADE challenge for MedTxt-CR

Figure 2: Subtask 3: CI challenge for MedTxt-RR

confirmed the validation loss reached a plateau. For training, thelabel of each token is trained so that loss from the correct label issmall. For prediction, given a token series 𝑋 and a label series 𝑦, 𝑦𝑖
corresponds to the label of token 𝑋𝑖 taking label 𝑦𝑖 among 𝑘 labels.
ÍThe probability is given by 𝑃 (𝑦𝑖 |𝑋𝑖 ) = 𝑆𝑖 (𝑋𝑖 , 𝑦𝑖 )/ 𝑆 (𝑋𝑖 𝑘, 𝑦𝑖 𝑘). Toobtain the series with the highest probability to be the series (𝑋, 𝑦),
the optimal series 𝑦∗ was obtained from 𝑦∗ = arg max log 𝑃 (𝑦|𝑋 ).
Since we considered to use all the information in the training
set, we predicted all entities even if the task does not require topredict the specific sorts of tags (<f>, <cc>, etc.). We believe that
this approach contributes to improving precision.

2.2

2.3.2 Subtask 3: CI challenge for MedTxt-RR. We regard the CI
task as a binary classification task for two cases, which classify thetwo cases originate from the same patient or not. To decompose
the CI tasks into micro binary tasks, we create all possible pairs of
two reports.
From 71 articles with 8 cases assigned, 648 pairs of sentences
between the same case and the same number of different cases werecreated, and labels of 0 or 1 were assigned according to whether
the cases were the same or different, and data such as (text1, text2,0) was constructed. There were 1,296 pairs of these sentences, ofwhich 80% were used as training data and the rest as validation data.
For the test data, 250,047 pairs between all articles were created.
The same pre-trained model of subtask 1 was finetuned for this task
determining whether the paired texts were the same. The hyperparameters of the model were max epochs=15, learning rate=1e-5,max length=128, batch size=32, and Adam was used as the optimizer. The pairs in the test data, (text1, text2) and (text2, text1),
were treated as different data, and both pairs were considered thesame case only if the model determined that they were identical.
Therefore, either pair was judged to be a different case if it wasdetermined that they were not identical. Finally, identical case pairs
containing common sentences were grouped together.

Subtask 2

We fine-tuned the same pre-trained model of subtask 1 by using
examples of the annotation guideline as a training data. However,dataset is smaller than that of subtask 1 so that 50 epochs which arelarger than subtask 1 were required before the losses fell sufficiently.
Although we fine-tuned two models in Subtask 1 for MedTxt-CR
and MedTxt-RR respectively, we used the same model to predict
entities in the two test sets in Subtask 2.

2.3

Subtask 3

2.3.1 Subtask 3: ADE challenge for MedTxt-CR. We regard theADE task as a classification task for each drug. Note that we do nothandle the table structure as is. Drug by drug, the system estimates
the ADE level, consisting of 4 categories: 0-Unrelated, 1-Unlikely,
2-Probably, and 3-Definitely.
We fine-tuned the same pre-trained model of subtask 1 using
the ADE data and the case report text before and after the ADE. Alist of each ADE entity (case number, case report text, ADE entity,ADE entity label, and ADE levels) was created as training data. For
the body of the case report, 50 characters before and after the ADE
entity were used. The training data were 117 case reports and thevalidation data were 14 cases were used for validation data. The
hyperparameters of the model were max epochs 10, learning rate
1e-5, max length 128, batch size 32, and optimizer Adam.

3 RESULTS AND DISCUSSIONS
3.1 Subtask 1
Table 1 shows the results of overall performance, Table 2 shows theentity-based performance and Table 3 shows the joint entity-basedperformance.
3.1.1 MedTxt-CR. The results for Subtask 1 on MedTxt-CR-JA
showed a character-based accuracy of 88.18, precision of 61.96, andrecall of 68.91. Although our method was simple, F1-score is 65.25,
which is the best result among the submission results.

331

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 1: Overall performance of subtasks 1 and 2.

Dataset

Subtask

Character-base
AccuracyPrecision

CR-JARR-JACR-JARR-JA1
12
2

88.18
95.13
69.34
89.24

61.96
87.11
20.80
60.50

Entity-base
Recall F1-score
68.91
87.23
31.81
69.87

65.25
87.17
25.12
64.85

Table 2: Entity-based performance (F1-score) of subtasks 1
and 2.

EntitySubtask 1
CR-JA RR-JA

Subtask 2
CR-JA RR-JA

adm-keym-val
t-key
t-testt-val
timex3

61.97
68.99
68.67
57.63
47.91
46.44
52.99
80.09

18.21
24.19
1.41
35.64
18.81
026.79
40.39

98.4
87.76
075.86

Subtask 2
CR-JA RR-JA

add_positived_suspiciousd_negatived_general
m-key_scheduled
m-key_executed
m-key_negatedm-key_other
m-val
t-key
t-test_scheduled
t-test_executed
t-test_negatedt-test_othert-val
timex3_date
timex3_time
timex3_duration
timex3_set
timex3_age
timex3_med
timex3_misc

61.97
29.63
62.85
64.52
59.65
49.68
0
58.72
0
47.17
53.57
47.91
0
45.89
0
52.99
85.64
33.33
55.90
65.12
86.61
64.56
2.08

18.21
0
20.41
0.97
2.32
00
2.44
0
0
32.99
18.81
0
0
0
26.79
32.97
00
30.14
26.36
19.63
089.40
91.14
80.99
78.95
82.55
0000
81.48
00-56.23
83.02
16.14
45.70

62.53
83.14
24.60
45.70

59.21
83.08
19.49
46.68

3.1.2 MedTxt-RR. The MedTxt-RR results for Subtask1 showed acharacter-based accuracy of 95.13, precision of 87.23, and recall of
87.17. The MedTxt-RR results for Subtask2 showed a letter-based
accuracy of 95.13, precision of 87.23, and recall of 87.17.
There are two possible reasons for the higher values than in theMedTxt-CR results. This may be because there were fewer types oftags to predict, and the radiation report had more similar patterns.
The joint based results showed that only timex3_date, the date information, could be predicted, while timex3_duration, the duration
information, and timex3_med, the treatment-related information,
could not be predicted. This is because the number of tags in thetraining set was less than two for each.

Table 3: Joint entity-based performance (F1-score) of subtasks
1 and 2.
Subtask 1
CR-JA RR-JA

86.05
93.39
65.45
80.45

Entity-base (Joint)
Precision Recall F1-score

amount of data. Although the number of d-tags in the training data
was the largest (n=2,348), the range of possible entity was slightly
lower (68.99). This is expected because it was difficult to obtain anexact match of d-tags due to the long span of matchable entities.
In the joint entity-base, the accuracy of the character-base was
86.05, precision was 56.23, and recall was 62.53. Compared to theresults without considering attributes, none of the values decreased
significantly, suggesting that the task is only slightly more difficult
than predicting entities alone.

63.16
63.48047.62

Entity_Attribute

Character-base (Joint)
Accuracy

56.89
056.7850.530054.9000
68.29
00-3.2

Subtask 2

Table 1 shows the results of overall performance, Table 2 shows theentity-based performance and Table 3 shows the joint entity-basedperformance.
3.2.1 MedTxt-CR. In Subtask 2, character-based accuracy was69.34, precision was 20.80, and recall was 31.71. As in Subtask 1,timex3 had the highest result for each entity. The results of m-key
was 1.41, which was very small. The m-val score was also relatively
good at 35.64, although the number of training data was very small(7). This indicates that the learning process of m-val is easier thanothers because the words of m-val are normally expressed as anumerical value.
3.2.2 MedTxt-RR. In Subtask 2, the character-based accuracy, precision, and recall were 80.45, 45.70, and 45.70, respectively. Accuracy
is about 10 points lower, and precision, recall and F1-score are about
40 points lower than the results of Subtask 1, but are higher thanthose of MedTxt-CR. It is surprising that such a high level of learning can be achieved from guideline examples alone.
Considering that the same model was used and the difference
was so large, we can imagine that the nature of the documents isvery different between MedTxt-CR and MedTxt-RR. Notably, theprediction results of the models differ greatly depending on theIn the results for each entity without considering attribute, timex3
had the highest F1-score, 80.09. timex3 has a total of 1353 tags in
the training set, which results for its high value due to the large

332

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 4: Performance of subtask 3 (ADE).
ADE (val=0)

ADE (val=1)

ADE (val=2)

ADE (val=3)

Report-level

Presicion
Recall
F1-score
Precision
Recall
F1-score
Precision
Recall
F1-score
Precision
Recall
F1-score
Precision
Recall
F1-score[4] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan HoSo, and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 4 (Feb. 2020),1234–1240.
[5] Shuntaro Yada, Yuta Nakamura, Shoko Wakamiya, and Eiji Aramaki. 2022. RealMedNLP: Overview of REAL document-based MEDical Natural Language Processing Task. In Proceedings of the 16th NTCIR Conference on Evaluation of Information
Access Technologies, NTCIR-16, National Center of Sciences, Tokyo. National Instituteof Informatics (NII).

95.21
76.04
84.55
000000
6.98
52.94
12.33
12.73
77.78
21.88

types of medical documents, even when the models were trainedusing annotated documents following to the same guidelines.

3.3

Subtask 3: ADE challenge for MedTxt-CR

Compared to the other teams, this was the task with the lowest
value. Especially, less correct answers could be obtained for any
case other than ADE (val =0).

3.4

Subtask 3: CI challenge for MedTxt-RR

The result is 0.5415. Although the team had the best value among
the submitting teams, compared to the results of the other teams in
English, there is room for improvement.

4

CONCLUSION

This paper discusses Team NAIST’s system for NTCIR 16 RealMedNLPtask, composed of three tasks, NER, ADE (a task to extract adverse
drug event), and CI (a task to indetify the radiology repots fromthe same patient). We employed the method using BERT, which isa basic way to solve taskes about natural language. The results of apart of Subtask 1 and Subtask 3 was supirior to the other teams, suggesting that a vanilla BERT is already strong enough, and specified
task oriented tunes are sometimes not efficient.

ACKNOWLEDGEMENTS
This work was supported by JST, AIP Trilateral AI Research, Grant
Number JPMJCR20G9, and JST AIP-PRISM Grant Number JPMJCR18Y1,Japan.

REFERENCES[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding. InProceedings of the 2019 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, Volume 1 (Longand Short Papers). Association for Computational Linguistics, 4171–4186. https://doi.org/10.18653/v1/N19-1423[2] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. ClinicalBERT: Modeling
Clinical Notes and Predicting Hospital Readmission. https://doi.org/10.48550/
ARXIV.1904.05342[3] Yoshimasa Kawazoe, Daisaku Shibata, Emiko Shinohara, Eiji Aramaki, andKazuhiko Ohe. 2021. A clinical specific BERT developed using a huge Japanese clinical text corpus. PLoS One 16, 11 (Nov. 2021), e0259763.

333