NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Real-MedNLP: Overview of REAL document-based MEDical
Natural Language Processing Task
Shuntaro Yada

Yuta Nakamura

Nara Institute of Science and TechnologyJapans-yada@is.naist.jp

The University of TokyoJapan
yutanakamura-tky@umin.ac.jp

Shoko Wakamiya

Eiji Aramaki

Nara Institute of Science and TechnologyJapanwakamiya@is.naist.jp

Nara Institute of Science and TechnologyJapanaramaki@is.naist.jp

ABSTRACTTo resolve this no-resource situation, we conducted a seriesof four previous medical natural language processing (MedNLP)
tasks: MedNLP-1 [13], MedNLP-2 [11], MedNLPDoc [12], and MedWeb [17]. In MedNLP-1, an NTCIR-10 pilot task, we proposed afundamental task, named entity recognition (NER), using dummy
medical records created by medical doctors. In MedNLP-2, as an
NTCIR-11 core task, we performed a term normalization task using
dummy medical records created by medical doctors. In MedNLPDoc, one of the NTCIR-12 core tasks, we designed a complete task
starting from a medical record from a medical textbook to provide aproper disease name represented by the ICD code. In MedWeb, as an
NTCIR-13 core task, a disease tweet classification task was designed
to simulate the use of social media data in medical and healthcare
domains, and dummy Twitter data were created in Japanese andtranslated into English and Chinese. The MedNLP task successfully
produced valuable datasets. However, two problems were identified.
(1) The data were not real clinical texts, but were dummy records
or sample texts from medical textbooks. (2) The dataset was limited
to Japanese, which makes it difficult to compare the results withother English-based shared task results1 .
The pilot task, called real-MedNLP, first introduces real clinical text. Our data consisted of two core resources: (1) Case Report corpus (MedTxt-CR) and (2) the Radiology Report corpus(MedTxt-RR).
In addition, we prepared real data in Japanese and translated
the original reports into English, enabling us to develop the firstbenchmark for multilanguage medical NLP.
Considering the data, we also redesigned the task scheme forour ultimate goal, which promotes systems applicable to hospitals.
The challenges of this task are twofold.

A standard dataset collection is essential for the development of
information science. Particularly in the medical field, in which privacy protection is a critical issue, the importance of the dataset issignificant. To discuss the validness of various methods, we build
the clinical text dataset, Real-MedNLP, for multiple medical tasks.
The goal of Real-MedNLP is threefold: (1) Real datasets: Previous
medical shared tasks, MedNLP, MedNLP2, and MedNLPDoc, werebased on the pseudo dataset, which was built from medical textbooks or dummy clinical texts. This task prepares real radiology
and case reports. (2) Bilingual capability: Both English and Japanesedata are handled. (3) Practicality: Both fundamental (named entity
recognition) and applied practical tasks are handled. This study
introduces the task setting of Real-MedNLP and submitted systems.
The methods mostly share the common paradigm, which is basedon a fundamental language model, such as BERT, aiming to separate
the resource problems. Based on their results, this study discusses
the feasibility of their approaches to bring us the future directionof medical NLP. Note that the Real-MedNLP is a shared task that
handles real Japanese medical texts.

KEYWORDS
Medical Natural Language Processing, Named Entity Recognition,
Case Reports, Radiography Reports, Adverse Drug Event, Case
Identification

SUBTASKS
Subtask1-CR-EN, Subtask1-CR-JA, Subtask1-RR-EN, Subtask1-RRJA, Subtask2-CR-EN, Subtask2-CR-JA, Subtask2-RR-EN, Subtask2RR-JA, Subtask3-CR-EN (ADE), Subtask3-CR-JA (ADE), Subtask3RR-EN (CI), Subtask3-RR-JA (CI)1

Few-resource Named Entity Recognition (Subtasks 1 & 2)
: Participants extract important information from real medical texts. This challenge is categorized into two tasks: exactly
100 training (Subtask 1) and guideline learning (Subtask 2).
The task is classified according to the amount of training
data (small or no data).
Applications (Subtask 3) : This challenge was designed froma practical viewpoint. For case reports, we designed an information extraction task for adverse drug events (ADE).
The ADE task has been challenged through workshops (e.g.,

INTRODUCTION

Recently, an increasing number of medical records have been written in electronic format instead of paper, which leads to a higher
importance of natural language processing (NLP) techniques inmedical fields [2]. Because NLP is a hot topic in computer science,
the number of medical NLP studies is increasing dramatically eachyear. Despite the large number of studies, the amount of privacyfree medical text data is still small in non-English languages such
as Japanese and Chinese.

1 https://n2c2.dbmi.hms.harvard.edu/2022-challenge285

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Figure 2: Sample of the radiology report corpus in Japanese(MedTxt-RR-JA)

X-ray, CT, or MRI scan. A radiology report contains (i) descriptionsof all normal and abnormal findings and (ii) interpretations of thefindings, including disease diagnosis and recommendations for the
next clinical test or treatment. Although most of the radiology AI
research tends to focus only on images because image-based AI
draws much attention, NLP on radiology reports also has potential
for a wide variety of clinical application [14].
MedTxt-RR was created by Nakamura et al. and composed of
135 radiology reports. Figure 2 shows its sample. MedTxt-RR aims
to provide information about the diversity of expressions used bydifferent radiologists to describe the same diagnosis. One of the difficulties in analyzing radiology reports is the variety in writing styles,
but simply collecting radiology reports from medical institutions
cannot provide enough information, because only one report is issued for a single radiological examination in usual clinical practice.
MedTxt-RR-JA was created to overcome this problem by crowdsourcing, for which nine doctors independently wrote radiology
reports for the same series of fifteen lung cancer cases. A total of 135
radiology reports are available in MedTxt-RR-JA. MedTxt-RR-EN
is an English translation of MedTxt-RR-JA by nine translators inwhich different translators translated radiology reports by different
radiologists.

Figure 1: A sample case report corpus in Japanese (MedTxtCR-JA)
n2c2, 2009). For radiographic reports, we designed a caseidentification (CI) task to detect reports originating from thesame patient. This task was proposed for the unique corpus.
These challenges yield promising technologies for the developmentof practical systems to support a wide range of medical services.

2

MATERIALS

Two corpora covering multiple languages were used (see Table 11).
Note that this paper notates the corpus as a combination of thename and its language, such as MedTxt-CR-JA.

2.1

CorpusCase report. A case report is a medical research paper written forthe patient. Case report analysis has two potential advantages:
the case report covers most of the disease timeline or a historyof the target disease, and the number of case reports is greaterthan the other papers because each medical society usually has asubmission truck for case reports. Considering these advantages,
case reports could be a rich source of information. The format of acase report is similar to that of a discharge summary, which is a typeof medical report. Therefore, techniques for case report analysis
can be expanded to analyze discharge summaries.
MedTxt-CR-JA comprises approximately 200 open-access casereports available at CiNii2 (in Japanese). Figure 1 shows its sample.
Because the number of medical societies that produce open-accesspublications is limited, the types of patients and diseases reported
in open-access case reports are highly biased. To reduce the bias
caused by the publication policy of each medical society, we selected
approximately 200 case reports based on the actual frequencies ofpatients and diseases. Approximately 200 reports from the J-Stage
article database have been translated from Japanese (MedTxt-CRJA) to English (MedTxt-CR-EN).

2.2 Subtasks and Annotations
Few-resource NER challenge for MedTxt-CR andfor MedTxt-RR (Subtask1/2-CR/RR)
Because NER is the most fundamental information extraction for
MedNLP, we designed challenges regarding NER for our real clinicalreports, which have only 100-200 reports. This corpus size scaletends to be regarded as “few-resource machine learning,” which isthe de facto standard among any kind of MedNLP in general.
1) Just 100 Training (Subtask1) Use the training split of 100
reports to build a model. This corresponds to the standard
type of few-resource supervised learning.
2) Guideline Learning (Subtask2) For each tag, we give only
a handful of sentence examples. This simulates the training of human annotators, who often learn from annotation
guidelines provided by researchers.
Task challengers can use any other resources outside this project ifthey find them useful for their methods.
We adopted the following types of medical entities [19]:Radiology Report. A radiology report is a type of clinical document(also called a report in this paper) written by a radiologist. Each radiology report discusses a single radiological examination such as an• Diseases and symptoms <d>• Anatomical entities <a>• Features and measurements <f>• Change <c>

2 https://ci.nii.ac.jp/286

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan• Time <timex3>• Test <t-test/key/val>• Medicine <m-key/val>• Remedy <r>• Clinical Context <cc>
Detailed definitions and information on modality are provided in[19]. The two corpora share the same NER tag set. The detailed corpus statistics are presented in the Appendix. We evaluated the following tag sets: <a>, <d>, <t-test>, <timex3>, <m-key>, <m-val>,<t-key>, and <t-val> for MedTxt-CR, and <a, <d>, <t-test>, and<timex3> for MedTxt-RR, because the others are rare in the corpora.

ADE challenge for MedTxt-CR (Subtask3-CR)
This subtask was specifically designed for MedTxt-CR. Given an
input report, the system extracts ADE information from the report. The ADE information consists of two tables: <d>-table fordisease and symptom names and <m-key>-table for medication(drug) names as shown in Figure 3. For each entity in these tables,
four levels of ADE certainty (ADEval)3 are to be given:
3 - Definitely
2 - Probably
1 - Unlikely
0 - Unrelated (no ADE)
For disease names, these values are interpreted as the likelihoodof being an ADE, whereas the likelihood of causing an ADE is theinterpretation of the medication names. Note that these values areannotated regardless of the relationship between the ADE and its
trigger. This is because we could not confidently prove explicit
causal relationships among multiple medications and symptoms,
although some case reports might explicitly argue some causal
effects. Eagerly detecting ADEs, including potential ones, would
contribute to the ideal handling of ADEs (i.e., the recall-oriented
reporting strategy, which is adopted in some countries, such as
Japan and EU nations).
To annotate these labels, we let annotators follow the author’s
perspective on whether drugs and symptoms are related to ADE ornot (i.e., writer’s perception). In other words, the annotators respect
the description by the author if the author argues that a symptom
is likely to be an ADE of a certain drug. However, if the annotators
noticed other possibilities of ADEs that were not explicitly pointed
out in the report, we allowed them to label ADEval ≥ 1 as well(i.e., reader’s perception). In a real-world situation, even when theauthor of a report does not state a phenomenon as an ADE, readersof the report may find that it is actually an ADE if they can infer
such possibilities.

Figure 3: ADE challenge

Figure 4: CI challengegroup the radiology reports to diagnose the same image (Figure 4).
MedTxt-RR was originally created by collecting radiology reports
from multiple radiologists who independently diagnosed the same
CT images, and the correspondence between the radiology reports
and CT images was used as the gold standard label. Thus, there
were no additional annotations for the CI challenge.
A part of MedTxt-RR-JA was already publicly available as a sample before launching Real-MedNLP. Therefore, in the CI challenge,
this sample was used as a training set because anyone can look at
the gold standard for the correspondence of the radiology report
and the CT image. Some of the rest of the radiology reports wereused as a test set.
Participants in the CI challenge were asked to submit a prediction
as a CSV file consisting of two columns, “id” and “case”. The column“id” contains a unique number assigned to each radiology report inadvance. Predictions were made by filling the “case” column withnumerical values so that only the rows for the radiology reports
diagnosing the same CT image had the same numbers. During thetask evaluation, only the correspondence between the values in the“case” column and the CT image was evaluated, and the values wereignored. Any numerical value can be stored in the “case” column
provided the correspondence is correct.
In real-world clinical institutions, only one reading report iscreated for each CT image, and a situation in which multiple reading
reports are created for the same image does not occur. Therefore,
the CI challenge is not a task that directly solves a problem that

CI challenge for MedTxt-RR (Subtask3-RR)
The case identification (CI) challenge is an application task designed
for MedTxt-RR. In the CI challenge, a large number of radiology
reports are provided, and participants are required to find and3 We

adapted the four levels in ADE reporting proposed in [8], which was
based on the FDA’s toxicity grading scale guidance from vaccine trials (September 2007, https://www.fda.gov/regulatory-information/search-fda-guidancedocuments/toxicity-grading-scale-healthy-adult-and-adolescent-volunteersenrolled-preventive-vaccine-clinical). We modified the original possibly level tounlikely by considering NIA adverse event and serious adverse event guidelines (https://www.nia.nih.gov/sites/default/files/2018-09/nia-ae-and-sae-guidelines-2018.pdf).

287

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 1: Number of systems developed by each team

Subtask1-CR-JA
Subtask1-CR-EN
Subtask1-RR-JA
Subtask1-RR-EN
Subtask2-CR-JA
Subtask2-CR-EN
Subtask2-RR-JA
Subtask2-RR-EN
Subtask3-CR-JA (ADE)
Subtask3-CR-EN (ADE)
Subtask3-RR-JA (CI)
Subtask3-RR-EN (CI)

A
2

C

D

E
1

2
2

1

1

1

1

1

F
4
4

G
1

H

I

5
1
3

J
4
4
4
4

1
1
10
1
10

1

2
2
1
1

6

1
1
1

baseline systems for reference to other teams. This approachis based on a simple method without any special techniques.
The model was based on general BERT [4]. For Subtask 2,
they trained the NER model only from sentences that wereavailable in the annotation guidelines.
F Subtask1-CR-EN, Subtask1-CR-JA, Subtask3-CR-EN (ADE),
Subtask3-CR-JA (ADE), Subtask3-RR-EN (CI), Subtask3-RRJA (CI) ; For Subtask 1-CR-EN and JA, they employed two
types of close-multilingual approaches: multilingual BERT(mBERT)6 and XLM-RoBERTa (XLM-R)7 to compare theeffectiveness of the multilingual pre-trained models. Subtask
3 (ADE) regarded ADEval as an additional attribute to namedentities and solved the challenge as NER using mBERT and
XLM-R.
G Subtask1-CR-JA, Subtask1-RR-JA ; BERT [4] + data augmentation (synonym replacement and shuffling within segments).
H Subtask1-CR-EN, Subtask1-RR-EN, Subtask3-CR-EN (ADE) ;For Subtask 1, BERT [4], BioBERT [9], clinical BERT [1],PubMed BERT [6], and entity BERT [10]. They utilized spanbased NER + data augmentation (back translation (via theChinese language) and random feature dropout).
For Subtask 3 (ADE), their approach is a combination ofmulticlass classification and prompt learning. In addition,
they attempted ensembles and data augmentation.
I Subtask1-RR-EN, Subtask2-RR-JA, Subtask3-CR-EN (ADE),
Subtask3-RR-EN (CI) ; They proposed a pipeline approach formultiple NLP modules (MetaMap [3], BERT [4], ScispaCy8 ,and MedLinker (NE linker)).
J Subtask1-CR-JA, Subtask1-RR-JA, Subtask1-CR-EN, Subtask1RR-EN, Subtask3-RR-JA (CI) ; They applied both languages(JA and EN), whose methodologies are completely different
for each language. Subtask 1 in JA is based on general BERT
and data augmentation, plus ensemble. For Subtask 3 in
JA, they used TNM classification estimation9 , which is aninternational standard cancer staging system. Note that they
did not rely on the NE tags.
For English NER, they utilized domain-specific BERTs (BioBERT [4]and ClinicalBERT [1]), and RoBERTa (general domain) [20].

Total
12
15
8
7
2
0
2
1
3
19
4
12

occurs in a clinical institution. The CI challenge, however, canhelp AI systems to understand the clinical content of documents
accurately without being confused by synonyms or paraphrases,
as MedTxt-RR contains radiology reports with almost the same
clinical content but various expressions.

3

METHODSThis section briefly introduces each team and the approach to eachsystem. For more information, refer to the system papers for NTCIR16 Real-MedNLP.
Teams and Systems. In total, nine teams formally submitted their results, which were anonymized by one capital letter. Distinct systems
proposed by a team ‘X’, for example, are denoted in combination
with numbers, such as ‘X1’ and ‘X2’. Table 1 lists the number ofsystems submitted by each team.
A Subtask1-CR-JA, Subtask1-RR-JA, Subtask2-CR-JA, Subtask2RR-JA ; They used Japanese Medical Domain Specific BERT(UTH BERT) [18]. For Subtask 2, Dictionary Manbyo dictionary [7], Hyakuyaku4 , and comeJisyo5 + bootstrap method
are used.
B Subtask1-CR-JA ; However, this team finally withdrew, so
this study did not include this team’s results.
C Subtask1-CR-EN, Subtask3-CR-EN (ADE), Subtask3-RR-EN(CI) ; For the NER challenge, they used BioBERT and data augmentation (label-wise token replacement, synonym replacement, mention replacement, and shuffle within segments).
For the CI challenge, the core technology is as follows: (1)Key feature clustering and (2) document embedding using
sentence BERT [15] and K-means clustering.
For the ADE challenge, Vocabulary adapted the BERT model(VART) with a multi-learning mechanism, where we transformed the ADE challenge into a classification task.
D Subtask3-RR-JA (CI) ; They performed parsing-based namedentity recognition without a dictionary and calculated casesimilarity using the distances of the bag-of-entity vectors.
E Subtask1-CR-JA, Subtask1-RR-JA, Subtask2-CR-JA, Subtask2RR-JA, Subtask3-CR-JA (ADE), Subtask3-RR-JA (CI) ; For NER
challenges (Subtasks 1 and 2), the systems of this team are the

4 EVALUATION METRICS
4.1 Few-resource NER challenge for
MedTxt-CR
We employed a set of standard NER metrics (F-measure) and theirvariations designed by considering the following three factors:
Joint factor : We use three different levels of NE matching.
(1) span (only recognition) : It considers matching only
the recognized span (this level of the NE category). Thislevel disregards the NE labels.
(2) +label (a span and a label joint) : It considers the matching of both the recognized span and its NE category. Thisis the standard NER evaluation method utilized in most
NE-shared tasks (such as CoNLL2003).
6 https://github.com/google-research/bert/blob/master/multilingual.md
7 https://huggingface.co/xlm-roberta-base

4 https://sociocom.naist.jp/hyakuyaku-dic/

8 https://allenai.github.io/scispacy/

5 https://ja.osdn.net/projects/comedic/

9 https://www.uicc.org/resources/tnm

288

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Entity level for each entity, Precision, Recall, and F1-score ofeach ADEval (= 0, 1, 2, 3) are calculated.
Report level binary classification for each report. For eachreport, it was judged whether the report contained ADE
information (POSITIVE REPORT) or not (NEGATIVE REPORT). We regarded a report that contained at least one
entity with ADEval ≥ 1 as a POSITIVE-REPORT; otherwise,
it was a NEGATIVE-REPORT. In this binary classification
scheme, the report-wise precision, recall, and F1-score of the
POSITIVE-REPORT were used in the evaluation.

(3) +label+mod (a span and a label+mod joint) : It considers the matching of the recognized span, its NE category
and attribute, that is, “scheduled,” “executed,” “negated,”or “other” for t-test.
Note that the scores satisfy the following relationship:
span > span+label joint > span+label+mod joint).
Matching policy factor (exact/partial ) : Usually, NE performance is based on the exact span match (exact match). We
consider not only the exact match but also a partial match.
If the system-predicted entities partially match (overlap) the
corresponding gold-standard annotations, they are regarded
as “partially correct”. This is because our corpora included
many complex compound medical terms and phrase-level
entities. For downstream information extraction, partially
identified entities would still be helpful. To calculate the partial match score, we considered the proportion of common
sub-characters between the gold standards and predictedentities. We regarded the number of common sub-charactersdivided by the character-length of the predicted entity as theentity-level partial match precision, whereas we regarded thenumber of common sub-characters divided by the characterlength of the gold standard entity as the entity-level partial match recall. Then we obtained the system-level partial
match precision as the division of the sum of entity-levelpartial match precision values for all predicted entities by
the number of all predicted entities; the system-level partial
match recall is calculated in the same manner.
Frequency factor : We assumed that the rare NEs were difficult to recognize. In particular, Subtask 2 guideline learning
contains many rare NEs. To focus on the performance of suchrare NEs, we designed a novel f-measure that is weighted
by the frequencies of the NE The idea is to penalize the correct guesses of the system if the predicted entity appears
frequently in the training dataset. For each gold-standard entity 𝑖 in the test set, we multiply the entity-level precision andrecall scores by the weight 𝑤𝑖 based on the term frequency 𝑓𝑖of the same entity in the training set, 𝑤𝑖 = 1/(log𝑒 (𝑓𝑖 +1) +1).
The weight 𝑤𝑖 gives 1.0 in 𝑓𝑖 = 0, and it decreases to 0.59
in 𝑓𝑖 = 1, 0.48 in 𝑓𝑖 = 2, and so on. This weighting allowed
us to observe the extent to which the system relied on highfrequency entities in the training phase.

4.4

b = {𝑢b1, ..., 𝑢d𝑈y, 𝑖 ≠ 𝑘 ⇐⇒ 𝑢b𝑖 ≠ 𝑢c𝑘}b | | 𝑢b𝑖 ∈ b|𝑈
𝑈 = {𝑢 1, ..., 𝑢 |𝑈 | | 𝑢 𝑗 ∈ y, 𝑗 ≠ 𝑘 ⇐⇒ 𝑣 𝑗 ≠ 𝑣𝑘 }|𝑈 | is the true number of test cases and |𝑈b | is the number of test
cases assumed by the participant (i.e., the number of clusters createdb|by the participant). Because we do not provide the value of |𝑈 |, |𝑈
may differ among participants. The NMI is then calculated as themutual information (MI) of 𝑈b and 𝑈 normalized by the arithmetic
average of the entropy of 𝑈b and 𝑈 :|𝑈b𝑖 | = |{𝑘 | 1 ≤ 𝑘 ≤ 𝑁 , 𝑦b𝑘 = 𝑢b𝑖 }|,|𝑈 𝑗 | = |{𝑘 | 1 ≤ 𝑘 ≤ 𝑁 , 𝑦𝑘 = 𝑢 𝑗 }|,|𝑈b𝑖 ∩ 𝑈 𝑗 | = |{𝑘 | 1 ≤ 𝑘 ≤ 𝑁 , 𝑦b𝑘 = 𝑢b𝑖 , 𝑦𝑘 = 𝑢 𝑗 }|,b) = −
𝐻 (𝑈

log

𝑁

MI(𝑈b, 𝑈 ) =|𝑈 |Õ|𝑈 𝑗 ||𝑈 𝑗 ||𝑈b𝑖 |, 𝐻 (𝑈 ) = −
log
,
𝑁
𝑁
𝑁
𝑗=1

b | |𝑈 ||𝑈ÕÕ |𝑈b𝑖 ∩ 𝑈 𝑗 |𝑁
𝑖=1 𝑗=1

b, 𝑈 ) =
NMI(𝑈

log

b𝑖 ∩ 𝑈 𝑗 |𝑁 |𝑈,b𝑖 ||𝑈 𝑗 ||𝑈b, 𝑈 )2MI(𝑈b
𝐻 (𝑈 ) + 𝐻 (𝑈 )5 RESULTS AND DISCUSSIONS
5.1 Subtasks 1 and 2

Few-resource NER challenge for
MedTxt-RR

Tables 2, 3, 4, and 5 show results of Subtasks 1 and 2 for MedTxt-CR
and MedTxt-RR, respectively.

The few-resource NER for MedTxt-RR is the same as that for MedTxtCR. However, the entity distribution is significantly different from
MedTxt-CR, in which several entity categories, such as medicationand tests, rarely appear. Therefore, for all tags, the following tag sets
were evaluated for MedTxt-RR: <d>, <a>, <timex3>, and <t-test>.

4.3

b|
|𝑈
Õ
b𝑖 |
|𝑈
𝑖=1

The following tag sets were evaluated for MedTxt-CR: <d>, <a>,<timex3>, <t-test>, <t-key>, <t-val>, <m-key>, and <m-val>.

4.2

CI challenge for MedTxt-RR

Because the CI challenge is a clustering task with gold-standard
labels, we used normalized mutual information (NMI).
Let b
y = (𝑦b1, ..., 𝑦c
𝑁 ) and y = (𝑦 1 , ..., 𝑦 𝑁 ) be the prediction andgold labels for 𝑁 radiology reports in the test set. First, we collectb and 𝑈 :unique labels in by and y to compose two sets, 𝑈

CR V.S. RR. Overall, the systems for the RR corpus performed better
than those for the CR corpus. This result follows the fact thatradiology reports mainly consist of frequent patterns and casereports tend to be linguistically diverse [19]. In contrast, the CR
corpus has a large vocabulary that covers most medical fields.

ADE challenge for MedTxt-CR

JA V.S. EN:. The performances of two languages (JA and EN) arealmost the same. In the CR track (exact; +label; normal), the best
JA and EN systems were 0.6525 (E1) and 0.6337 (H2), respectively

The ADE challenge is an information extraction task. We employed
two levels of evaluation: entity level and report level.

289

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 2: Results of Subtask 1 for MedTxt-CR. Bold font indicates the best score for each evaluation metric.
(a) Results of Subtask1-CR-JA.

System ID
A1
A2
E1
F1
F2
F3
F4
G1
J1
J2
J3
J4

span
normal weighted

Exact match+labelnormal weighted+label+mod
normal weighted

span
normal weighted

Partial match+labelnormal weighted+label+mod
normal weighted

0.63880.63780.69880.60950.64970.58970.61790.6766
0.33610.3676
0.27450.28410.61330.61240.65250.56960.6076
0.55500.58130.61890.30880.35850.2656
0.27730.5921
0.52490.56020.5171
0.54200.25910.30130.22470.23080.85240.85300.86170.82970.81270.84220.82210.83860.74620.71490.68560.67040.78410.78460.77270.72670.72570.75220.74640.73610.55140.63950.58650.59670.69770.65520.65960.68280.68210.44880.5307
0.48200.49630.54330.54250.59950.51120.54450.4987
0.52180.57540.26270.30570.22790.23990.51950.51880.55500.47370.50480.46500.48630.51980.23830.29680.21950.23340.49930.43330.46210.43150.45150.19630.24590.18360.19100.74620.74690.75400.72120.69920.73640.71150.72810.6572
0.63630.61590.60620.68120.68190.66890.62300.61640.64890.63810.62930.4712
0.56330.52040.5346

0.59930.5574
0.55620.58500.57960.3777
0.46250.42320.4403(b) Results of Subtask1-CR-EN.

System ID
C1
C2
F1
F2
F3
F4
H1
H2
H3
H4
H5
J1
J2
J3
J4

span
normal weighted

Exact match+labelnormal weighted+label+mod
normal weighted

span
normal weighted

Partial match+labelnormal weighted+label+mod
normal weighted

0.46010.46970.51040.5292
0.52400.54730.62460.65400.64380.61900.62990.48820.55510.55030.52700.43210.4371
0.4683
0.48600.49180.51450.59800.63370.62310.59330.60330.45560.51970.5116
0.49180.42450.44060.44800.4696
0.54840.58530.57490.54520.55400.29570.33350.32630.30770.63570.64010.80940.80020.81280.7936
0.79380.83890.83000.84230.84530.79920.83080.82610.81590.56480.56550.69990.69900.72390.71420.7372
0.78800.77900.77840.78250.70320.73760.72200.71910.62420.62950.64860.64810.67690.72690.71810.7156
0.71800.44790.47110.46090.4526

0.41170.41980.45010.46670.46340.48390.55130.58130.57190.55160.56200.42740.49250.4846
0.46520.38500.38900.40920.42470.43260.45250.52550.5616
0.55150.52650.5364
0.39650.45890.44780.43170.37010.38430.39380.41270.48090.51810.50800.4831
0.49170.25890.29500.28670.2705(Table 2). In the RR track (exact; +label; normal), the best JA system
was 0.8926 (A2) and the best EN system was 0.8266 (H2) (Table 3).
Although slightly better results are shown in the JA systems, thedifficulties of this task are not language-independent.

0.5802
0.58310.73490.72390.73900.71710.70810.75330.74380.76230.76370.72340.75350.74560.73900.51240.51220.62800.62520.65280.64040.65350.7042
0.69470.70050.70340.63030.66380.64530.64550.55830.5616
0.58400.58080.59940.64880.63940.64350.64440.40050.42280.41110.4046

captured medically important phrases at least partially, even though
the training data were very small.
Zero/few-shot: The general effect of zero/few-shot weighting was
to decrease scores. However, the performance was robust, evenafter weighting. The amount of decrease in weighting was larger
in the RR corpus, which means that the RR-solving systems relied
substantially on high-frequency entities in the training dataset.
This would be due to the fact that frequently used patterns often
constitute radiology reports. The decrease in the weighting was
smaller in the EN corpora, even though they are parallel to the JA
corpora, probably because identical Japanese entities were sometimes translated into different surface phrases in English according
to the local context.

Span V.S. Joint: Label-joint NER is more difficult than span recognition. For example, in the CR track (exact; normal), we can see
a 10 point difference between span = 0.6988, +label = 0.6525, and+label+mod = 0.5921 (see Table 2). Between Span and Joint, theperformance difference is approximately 10 point gaps.
Partial V.S. Exact: The partial scores were at least approximately
10 points larger than the exact scores in general, regardless ofthe corpus (track), language, and subtask. In particular, the spanonly scores that disregard the match of entity classes or modality
values achieve approximately a 0.8 F1-score in the CR corpus andan approximately 0.9 F1-score in the RR corpus; almost all systems

Subtask 1 V.S. Subtask 2: Subtask 1 is a standard NER with a small
corpus. Subtask 2 is our new challenge, relying on only a few examples. Because the number of participants was small in Subtask

290

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 3: Results of Subtask 1 for MedTxt-RR. Bold font indicates the best score for each evaluation metric.
(a) Results of Subtask1-RR-JA.

System ID
A1
A2
E1
G1
J1
J2
J3
J4

span
normal weighted

Exact match+labelnormal weighted+label+mod
normal weighted

span
normal weighted

Partial match+labelnormal weighted+label+mod
normal weighted

0.15280.90190.87040.8932
0.58620.60550.58050.57150.15050.8926
0.84880.87030.58110.60220.57790.56740.80790.42590.43630.42240.42160.98070.97550.9603
0.97350.89430.90420.8996
0.88120.96390.96140.92690.93850.82010.83700.82130.82010.87780.57270.58940.58570.58840.11850.52640.50520.52070.32320.33060.31510.31200.11650.5181
0.48710.49920.31910.32780.31270.3096

0.46740.25500.25720.24800.24770.58230.59000.5824
0.58890.55630.56230.56330.55300.56680.57690.5536
0.55800.49710.50780.50030.5024

0.52810.36930.37810.37640.3803(b) Results of Subtask1-RR-EN.

System ID
H1
H2
H3
J1
J2
J3
J4

span
normal weighted

Exact match+labelnormal weighted+label+mod
normal weighted

span
normal weighted

Partial match+labelnormal weighted+label+mod
normal weighted

0.82960.83020.81400.76960.80680.79620.80000.82600.82660.80610.75920.79970.78770.78950.79190.78740.77190.63500.67070.65320.65450.95670.95690.95880.95130.9532
0.95330.95670.92860.92930.92240.90850.91320.91560.91700.88620.88050.87870.74100.75510.74690.75130.5532
0.5536
0.54300.50490.53600.52650.53320.54960.55000.53580.49570.52990.5192
0.52450.5262
0.52310.51050.41070.44000.42640.43090.66700.66740.6726
0.67190.67070.67030.67510.64020.6406
0.63880.6316
0.63250.6346
0.63810.60950.60590.60500.50620.51630.51040.5165

Table 4: Results of Subtask 2 for MedTxt-CR (Subtask2-CR-JA). Bold font indicates the best score for each evaluation metric.

System ID
A1
E1

span
normal weighted

Exact match+labelnormal weighted+label+mod
normal weighted

span
normal weighted

Partial match+labelnormal weighted+label+mod
normal weighted

0.4212
0.3366

0.37100.25120.37100.19490.74580.67970.61630.45890.6163
0.34640.4146
0.3326

0.36440.24740.36440.19120.73790.67380.60910.45470.6091
0.3424

Table 5: Results of Subtask 2 for MedTxt-RR. Bold font indicates the best score for each evaluation metric.
(a) Results of Subtask2-RR-JA.

System ID
A1
E1

span
normal weighted

Exact match+labelnormal weighted+label+mod
normal weighted

span
normal weighted

Partial match+labelnormal weighted+label+mod
normal weighted

0.66380.65570.64850.62550.51330.46680.9106
0.89610.88430.82890.68640.6094

0.63700.63150.62170.60130.49580.44620.88340.87110.85710.80390.66850.5880(b) Results of Subtask2-RR-EN.

System ID
I1

span
normal weighted

Exact match+labelnormal weighted+label+mod
normal weighted

span
normal weighted

Partial match+labelnormal weighted+label+mod
normal weighted

0.56280.5496

0.50370.88430.82890.75990.55460.54220.49682, precise discussion was difficult. However, we can say that several systems surprisingly demonstrated high performance, such as

0.87260.81790.74950.6485 in the RR track (exact; +label, normal) for system A1 (Table 5). In addition, system A1 achieved a partial match of 0.8843

291

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 6: Results of Subtask 3 for MedTxt-CR. Bold font indicates the best score for each evaluation metric.
(a) Results of Subtask3-CR-JA (ADE).

System ID

P

ADEval=0
R

F

P

ADEval=1
R
F

E1
F1
F2

95.21
95.76
96.05

76.04
97.67
97.00

84.55
96.71
96.52

0.00
0.00
0.00

0.00
0.00
0.00

0.00
0.00
0.00

P

ADEval=3
R

F

P

Report-level
R
F

6.98
12.50
27.59

52.94
11.76
47.06

12.33
12.12
34.78

12.73
37.50
25.00

77.78
66.67
44.44

21.88
48.00
32.00

(b) Results of Subtask3-CR-EN (ADE).

System ID

P

ADEval=0
R

F

P

ADEval=1
R

F

P

ADEval=3
R

F

P

C1
C2
C3
C4
C5
C6
C7
C8
C9
C10
F1
F2
H1
H2
H3
H4
H5
H6
I1

95.70
95.79
95.95
96.05
95.87
96.14
95.67
96.42
96.35
95.87
96.53
95.39
96.57
96.57
96.28
96.41
95.88
95.99
97.02

94.94
97.00
93.52
92.10
95.26
94.47
94.31
97.79
91.79
95.26
96.68
98.10
97.95
97.95
98.10
97.63
99.37
98.26
97.63

95.32
96.39
94.72
94.03
95.56
95.30
94.99
97.10
94.01
95.56
96.61
96.73
97.25
97.25
97.18
97.02
97.60
97.11
97.32

20.00
14.29
6.25
25.00
0.00
25.00
0.00
20.00
0.00
7.14
0.00
0.00
14.29
0.00
0.00
0.00
0.00
33.33
30.00

5.26
5.26
5.26
5.26
0.00
10.53
0.00
5.26
0.00
5.26
96.68
0.00
5.26
0.00
0.00
0.00
0.00
5.26
31.58

8.33
7.69
5.71
8.70
0.00
14.81
0.00
8.33
0.00
6.06
0.00
0.00
7.69
0.00
0.00
0.00
0.00
9.09
30.77

62.50
43.75
28.57
22.22
56.25
50.00
33.33
47.62
23.81
26.92
31.25
40.00
60.00
59.09
60.00
57.14
78.57
55.56
100.00

26.32
36.84
21.05
42.11
47.37
21.05
26.32
52.63
52.63
36.84
52.63
42.11
63.16
68.42
63.16
63.16
57.89
52.63
26.32

37.04
40.00
24.24
29.09
51.43
29.63
29.41
50.00
32.79
31.11
39.22
41.03
61.54
63.41
61.54
60.00
66.67
54.05
41.67

22.22
29.41
19.35
18.92
25.93
21.21
19.35
50.00
18.92
23.08
25.00
40.00
50.00
50.00
50.00
50.00
60.00
50.00
50.00

(partial; +label, normal), which is good enough for several practicalapplications, such as case report search. This challenge reveals thepotential feasibility of NER without using big training data. We
believe that there is much room for future research.

Report-level
R
F
66.67
55.56
66.67
77.78
77.78
77.78
66.67
77.78
77.78
66.67
55.56
44.44
66.67
66.67
55.56
66.67
33.33
44.44
88.89

33.33
38.46
30.00
30.43
38.89
33.33
30.00
60.87
30.43
34.29
34.48
42.11
57.14
57.14
52.63
57.14
42.86
47.06
64.00

H2 system, which is based on back-translation. Consideringthat machine translation performance is already feasible, theback-translation approach is promising.

Approach: The best systems vary according to weighted scores.
However, the differences between the second systems are small.
Among them, we can see three major trends among the best (orsecond system).

For Subtask1-CR-JA, the best performing system was E1, whichapplied a general domain BERT. G1 and J1-4 also used it but adopted
data augmentation as well. Because their performance was lower,
simple rule-based data augmentation may not be suitable for medicaldomain NER. The domain-specific BERT used in A1 and A2 did notcontribute significantly to performance. UTH-BERT was trainedon health records, the writing style of which could be fragmented
and compressed owing to the nature of personal notes. Wikipedia
articles, on which the general Japanese BERT model was trained,
could be more akin to case reports.
However, for the other JA tasks, UTH-BERT performed the best.
Radiology reports (RR), another type of in-hospital note, are written
in a similar manner as health records, which may have resulted inperformance gain. For an extremely few-resource setting, that is,
Subtask 2, domain knowledge seems to matter more.
In the English corpora (EN), H1-3 performed best. These systems
characteristically apply span-based NER methods. This emerging
trend in NER appears promising in a few-resource medical setting.
Interestingly, naive data augmentation reduced performance (C <(1) Language Model Approach: Almost all systems employ a
BERT (or BERT family) language model. These BERTs are
the driving force of the current NLP, and BERT without
any special techniques is strong enough like the E1 system,
which shows good performance in Subtask1-CR-JA. The nextquestion is what kind of BERT is suitable for medical NLP.
The EN trend is a domain-specific BERT, such as BioBERT
and Clinical BERT, which is better than the general BERT
shown in team-H systems for Subtask1-CR/RR-EN. However,in JA, a general BERT shows a sufficiently good performance,
as shown in the team-E systems. In the near future, we willneed a more accurate discussion of this issue.
(2) Data augmentation: Many systems utilize data augmentation
techniques. The remaining issue is the suitable type of augmentation. One of the best Subtask1-CR-EN systems is the292

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 7: Results of Subtask 3 for MedTxt-RR (CI). Bold fontindicates the best score for each evaluation metric.

The report-level ADE performance is inconsistent with the entitiylevel performance of some systems. Thus, a better entity-level system is not necessarily a better report-level system. This result implies that such systems generate entity-level false positives, evenin non-ADE reports. In other words, the systems with smaller gaps
between the entity-level ADEval = 3 scores and report-level scoresbehave coherently in the micro-and macroscopic identification of
ADE.
Although a score gap was found between the EN and JA corpora in both entity- and report-level evaluations, we cannot infer
any innate linguistic difficulty in this task because the JA track
participants were much fewer than the EN track participants.
For the Japanese corpus (JA), the F systems performed better in
ADE signal detection, that is, the F-scores of ADEval=3 and reportlevel. This could be because the F1 and F2 systems incorporated
more contextual information than E1 by solving the task as NER.
For the English corpus (EN), I1 performed well at the reportlevel score. The approach jointly solved this ADE with Subtask
1 RR (NER), which is similar to that of the F1 and F2 systems,
although they did not perform much in the English corpus, unlikein the Japanese corpus. We can note that H1-6 performs nicely ingeneral. They adopted an interesting approach based on promptbased learning, which trained on automatically generated snippets
to explicitly explain which entity in a report was related to an ADE.
A global remark could be how to enhance context information thatspecifically matters in this task.

(a) Results of Subtask3-RR-JA.
System ID

Score (Normalized Mutual Info)

D1
E1
F1
J1
J1*

0.35690.54150.1744
0.4161
0.4622

Revised results were submitted after the deadline of the formal run.

(b) Results of Subtask3-RR-EN.
System ID

Score (Normalized Mutual Info)

C1
C2
C3
C4
C5
C6
C7
C8
C9
C10
F1
I1

0.87240.84630.84680.84680.84680.84680.85810.84680.84680.85760.2172
0.78795.2.2 CI challenge for MedTxt-RR. Table 7 lists the NMI scoresof each system for the MedTxt-RR CI challenge. The C1 system,
which uses heuristics for cancer size matching and Sentence-BERT
encoding [15], achieved the highest performance of all systems.
As shown in Table 8, the C1 system succeeded in grouping radiology reports of cases 4 and 5 into a single cluster, suggesting thatmatching lesion size is helpful in case distinction.
A large discrepancy is observed between the scores of the D1
and I1 systems, although they both used the NER-based approach.
This may reflect the difference in the availability of biomedical
knowledge bases between Japanese and English. While system I1
could use UMLS to normalize biomedical entities, system D1 had
to create bag-of-entity vectors from only the training set, which
probably had difficulty dealing with unseen entities in the test set.
As shown in Table 9, most systems grouped the test cases into
the same number of clusters as the gold standard, although we did
not clarify the true cluster number. In this task, the true cluster
number was easily determined by the test sample size as used by
the C1 system.
The baseline E1 system split the test cases into far more clusters
than the true case numbers, although it achieved the highest NMI
in RR-JN. This is problematic considering real-world applications,because NMI has failed to penalize the inability to recognize radiology report similarity. The bottom row of Table 10 shows theresult of an extreme prediction where all test samples are split into
different clusters of size one, which suggests an improper reward
from the NMI.
Thus, we re-evaluated each system by adding adjusted normalized mutual information (AMI) [16] and Fowlkes-Mallows (FM)scores [5]. Table 10 shows that AMI and FM penalize the splitting

F and J). We might require medical domain-specific data augmentation policies.

5.2

Subtask 3

5.2.1 ADE challenge for MedTxt-CR. Table 6 lists the results of theADE challenge for MedTxt-CR. For entity-level ADE, most systems
achieved high performance with ADEval = 0, which is reasonablebecause the vast majority of entities had zero values in both thetraining and test datasets. However, most systems struggled to identify ADEval = 1 and 3 entities. Especially for ADEval = 1 entities,
most systems were almost unable to find the entities. This result isunderstandable because neither ADEval = 1 nor ADEval = 3 entities appeared frequently in the training dataset. However, all thesystems identified many more entities with ADEval = 3 than with
ADEval = 1. In fact, the average F1 performance is 10 times larger
in ADEval = 3 (i.e., 41.05) than in ADEval = 1 (i.e., 4.87), probably because of the “strength” of the signal; under our annotation
scheme, ADEval = 3 is the marker for an ADE or an ADE trigger
whereas ADEval = 1 is a weak sign of a negative ADE or an ADE
trigger. Linguistic clues may appear more clearly in ADEval = 3
than in ADEval = 1. This result is also preferable because detecting
ADE and ADE trigger signals is more important for real-world
applications such as automated ADE reporting. Note that the test
dataset did not contain any ADEval = 2 entities. Thus, we did notevaluate ADEval scores.

293

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 8: The number of clusters into which each case was split by each system in Subtask 3 for MedTxt-RR (CI).
Cluster number
Case ID

TNM cancer staging

4
5
7
8
10
14
15

T2aN0M0
T2bN0M0
T3N1M0
T3N3M0T4N0M0
T4N3M1aT2N2M1c

RR-JA

RR-EN

Gold standard

D1

E1

F1

J1

J1*

C1

C2

C3

C4

C5

C6

C7

C8

C9

C10

F1

I1

5
6
3
3
3
5
2

6
7
5
8
6
5
5

6
5
5
6
5
5
5

3
4
4
5
4
4
2

4
5
3
4
4
3
2

1
1
2
2
1
2
3

1
1
2
3
1
2
3

1
1
2
3
1
2
3

1
1
2
3
1
2
3

1
1
2
3
1
2
3

1
1
2
3
1
2
3

1
1
2
2
1
2
3

1
1
2
2
2
2
3

1
1
2
2
2
2
3

1
1
2
2
1
2
3

5
5
6
7
6
5
5

2
2
1
1
1
3
4

1
1
1
1
1
1
1

The revised results were submitted after the deadline of the formal run.

Table 9: Cluster sizes created by each system in Subtask 3 for
MedTxt-RR (CI).
System ID

Cluster number

Gold standard

7

9, 9, 9, 9, 9, 9, 9, 9, 9

D1
E1

8
33

F1
J1
J1*

7
7
7

18, 17, 9, 8, 4, 3, 2, 2
19, 5, 4, 3, 2, 2, 2, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
11, 11, 10, 9, 8, 7, 7
16, 10, 10, 7, 7, 7, 6
13, 11, 11, 8, 7, 7, 6

C1
C2
C3
C4
C5
C6
C7
C8
C9
C10
F1
I1

7
7
7
7
7
7
7
7
7
7
9
9

10, 9, 9, 9, 9, 9, 8
11, 9, 9, 9, 9, 9, 7
10, 10, 9, 9, 9, 9, 7
10, 10, 9, 9, 9, 9, 7
10, 10, 9, 9, 9, 9, 7
10, 10, 9, 9, 9, 9, 7
10, 10, 9, 9, 9, 9, 7
10, 9, 9, 9, 9, 9, 8
10, 9, 9, 9, 9, 9, 8
11, 9, 9, 9, 9, 9, 7
12, 10, 8, 8, 7, 6, 6, 5, 1
12, 12, 9, 9, 9, 7, 2, 2, 1

RR-JA

RR-EN

Table 10: Performance of each system of Subtask 3 for
MedTxt-RR (CI) in multiple evaluation metrics. Bold fontindicates the best score for each evaluation metric.

Cluster size
System ID

NMI

AMI

FM

RR-JA

D1
E1
F1
J1
J1*

0.35690.54150.1744
0.4161
0.4622

0.19880.1489-0.01170.28380.34090.26740.18140.11700.30440.3622

RR-EN

C1
C2
C3
C4
C5
C6
C7
C8
C9
C10
F1
I1

0.87250.84630.84680.84680.84680.84680.85810.84680.84680.85760.2172
0.78790.84370.81160.81220.81220.81220.81220.82610.81230.81230.8255-0.00450.73090.84360.81100.81260.81260.81260.81260.81660.81190.81190.81500.10850.69920.6392-4.7901

0.0000Extreme prediction(isolate-all)

The revised results were submitted after the deadline of the formal run.

The revised results were submitted after the deadline of the formal run.
NMI, normalized mutual information; AMI, adjusted normalized mutual information;
FM, Fowlkes-Mallows score.

of clinically similar documents into numerous clusters. This is supported by the scores at the bottom of Table 10. With AMI and FM,
the J1 system achieved the highest scores, suggesting the effectiveness of sentence classification in determining TNM staging even ina limited availability of a knowledge base.
In summary, the CI challenge results show a difference in effective strategy for case clustering between RR-JA and RR-EN. For
RR-EN, embedding distance with the help of a knowledge baseworks well and can be applied to other clinical specialties beyond
lung cancer. For RR-JA, the lack of a knowledge base motivatedparticipants to adopt a more dataset-specified approach, resulting in comparatively lower performance and limited possibility of
application beyond lung cancer.

This study introduced the Real-MedNLP task setting, which is a
medical NLP shared task handling three different tasks (namedentity recognition (NER), case identification (CI), and adverse drug
event extraction (ADE)) in a bilingual language (English and Japanese). The basic approach is twofold: (1) to employ data augmentation and (2) to utilize domain-specific language models such as
BioBERT and ClinicalBERT. These approaches partially solve thelow-resource problem in MedNLP. However, the performance in an
extremely low resource-setting task (Subtask 2 guideline learning)is insufficient. In particular, for newly designed tasks (Subtask 3
CI challenge and ADE challenge), we needed to start by deciding
how to evaluate them and still work for suitable evaluation systems.
Considering that not only our three tasks but also various medical

6

294

CONCLUSIONSNTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japantasks are waiting for NLP solutions, it is important to organize andshare the approach and results across the world. We believe thatour datasets and the results of all participants will contribute to theboost of future research.

ACKNOWLEDGEMENTS
This work was supported by JST, AIP Trilateral AI Research, Grant
Number JPMJCR20G9, and JST AIP-PRISM Grant Number JPMJCR18Y1,
Japan.

REFERENCES[1] Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan
Naumann, and Matthew McDermott. 2019. Publicly Available Clinical BERT
Embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop. Association for Computational Linguistics, Minneapolis, Minnesota, USA,72–78. https://doi.org/10.18653/v1/W19-1909[2] Eiji Aramaki, Shoko Wakamiya, Shuntaro Yada, and Yuta Nakamura. 2022. Natural Language Processing: from Bedside to Everywhere. IMIA Yearbook of Medical
Informatics 2022 (2022), 257–267.
[3] Alan R. Aronson. 2001. Effective Mapping of Biomedical Text to the UMLS
Metathesaurus: The MetaMap Program. (2001). http://skr.nlm.nih.gov/papers/
references/metamap_01AMIA.pdf[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. In
Proceedings of the 2019 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, Volume 1 (Longand Short Papers). Association for Computational Linguistics, 4171–4186. https://doi.org/10.18653/v1/N19-1423[5] E. B. Fowlkes and C. L. Mallows. 1983.
A Method for Comparing Two Hierarchical Clusterings.
J. Amer. Statist. Assoc. 78, 383(1983), 553–569.
https://doi.org/10.1080/01621459.1983.10478008
arXiv:https://www.tandfonline.com/doi/pdf/10.1080/01621459.1983.10478008[6] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong
Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specificlanguage model pretraining for biomedical natural language processing. ACM
Transactions on Computing for Healthcare (HEALTH) 3, 1 (2021), 1–23.
[7] Kaoru Ito, Hiroyuki Nagai, Taro Okahisa, Shoko Wakamiya, Tomohide Iwao, andEiji Aramaki. 2018. J-MeDic: A Japanese Disease Name Dictionary based on
Real Clinical Usage. In Proceedings of the Eleventh International Conference on
Language Resources and Evaluation (LREC 2018). European Language Resources
Association (ELRA), Miyazaki, Japan. https://aclanthology.org/L18-1375[8] Colleen R Kelly, Sachin S Kunde, and Alexander Khoruts. 2014. Guidance on
preparing an investigational new drug application for fecal microbiota transplantation studies. Clinical gastroenterology and hepatology: the official clinicalpractice journal of the American Gastroenterological Association 12, 2 (Feb. 2014),
283–288.
[9] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim,
Chan Ho So, , and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language
representation model for biomedical text mining. Bioinformatics 36, 4 (2020),
1234–1240.
[10] Chen Lin, Timothy Miller, Dmitriy Dligach, Steven Bethard, and Guergana K.
Savova. 2021. EntityBERT: Entity-centric Masking Strategy for Model Pretraining
for the Clinical Domain. In BIONLP.
[11] Mizuki Morita, Yoshinobu Kano, Tomoko Ohkuma, and Eiji Aramaki. 2014.
Overview of the NTCIR-11 MedNLP Task. In Proceedings of the 11th NTCIR
Conference on Evaluation of Information Access Technologies, NTCIR-11, National
Center of Sciences, Tokyo. National Institute of Informatics (NII).
[12] Mizuki Morita, Yoshinobu Kano, Tomoko Ohkuma, and Eiji Aramaki. 2016.
Overview of the NTCIR-12 MedNLPDoc Task. In Proceedings of the 12th NTCIR
Conference on Evaluation of Information Access Technologies, NTCIR-12, National
Center of Sciences, Tokyo. National Institute of Informatics (NII).
[13] Mizuki Morita, Yoshinobu Kano, Tomoko Ohkuma, Mai Miyabe, and Eiji Aramaki.
2013. Overview of the NTCIR-10 MedNLP Task. In Proceedings of the 10th NTCIR
Conference on Evaluation of Information Access Technologies, NTCIR-10, National
Center of Sciences, Tokyo. National Institute of Informatics (NII).
[14] E. Pons, L. M. Braun, M. G. Hunink, and J. A. Kors. 2016. Natural Language
Processing in Radiology: A Systematic Review. Radiology 279, 2 (May 2016),
329–343.
[15] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings
using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical
Methods in Natural Language Processing and the 9th International Joint Conference
on Natural Language Processing (EMNLP-IJCNLP). Association for Computational
Linguistics, Hong Kong, China, 3982–3992. https://doi.org/10.18653/v1/D19-1410[16] Nguyen Xuan Vinh, Julien Epps, and James Bailey. 2009. Information Theoretic
Measures for Clusterings Comparison: Is a Correction for Chance Necessary?.
In Proceedings of the 26th Annual International Conference on Machine Learning(Montreal, Quebec, Canada) (ICML ’09). Association for Computing Machinery,
New York, NY, USA, 1073–1080. https://doi.org/10.1145/1553374.1553511[17] Shoko Wakamiya, Mizuki Morita, Yoshinobu Kano, Tomoko Ohkuma, and Eiji
Aramaki. 2017. Overview of the NTCIR-13 MedWeb Task. In Proceedings of the
13th NTCIR Conference on Evaluation of Information Access Technologies, NTCIR-13,
National Center of Sciences, Tokyo. National Institute of Informatics (NII).
[18] Kawazoe Y, Shibata D, Shinohara E, Aramaki E, and Ohe K. 2021. A clinicalspecific BERT developed using a huge Japanese clinical text corpus. PLoS One 15,
11 (2021), e0259763.
[19] Shuntaro Yada, Ayami Joh, Ribeka Tanaka, Fei Cheng, Eiji Aramaki, and Sadao
Kurohashi. 2020. Towards a Versatile Medical-Annotation Guideline Feasible
Without Heavy Medical Knowledge: Starting From Critical Lung Diseases. In
Proceedings of The 12th Language Resources and Evaluation Conference. European
Language Resources Association, Marseille, France, 4565–4572. https://www.
aclweb.org/anthology/2020.lrec-1.561[20] Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A Robustly Optimized BERT
Pre-training Approach with Post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics. Chinese Information Processing
Society of China, 1218–1227. https://aclanthology.org/2021.ccl-1.108

APPENDIX

295

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 11: NEs of the training set in MedTxt-CR and MedTxt-RR.
Data-set

CR-JA

CR-EN

RR-JA

RR-EN

# of texts# of chars (ave.)

148
84471(570)14840383(272)7216861(234)72
8488(117)

823
2348
1695
80
251
302
387
0
362
7
18
1353
539
53
82
34
189
428
28
344
0
266
27
51
64
0
0
2
0
524
427
638
569
678
266

819
2346
1693
80
251
302
388
0
363
7
18
1353
539
53
82
34
189
428
28
344
0
266
27
51
64
0
0
2
0
524
427
636
569
678
266

464
884
465
191
149
1
26
0
19
2
5
29
26
0
2
0
0
1
0
0
0
0
0
0
0
0
0
0
0
1
0
345
22
2
16

465
883
462
191
148
1
27
0
19
2
6
29
26
0
2
0
0
1
0
0
0
0
0
0
0
0
0
0
0
1
0
340
22
1
15

<a>
<d>

<t-test><timex3><m-key><m-val><t-key><t-val>
<f>
<c>
<r>
<cc>

total
total
"positive""suspicious""negative""general"total"scheduled""executed""negated""other"total"date""time""duration"
"set"
"age"
"med"
"misc"
total"scheduled""executed""negated""other"total"scheduled""executed""negated""other"
total
total
total
total
total
total

CR indicates the case report corpus, and RR represents the radiology report corpus.
Although the corpora included general medical annotations, we only evaluated high-frequency labels(highlighted rows).

296