NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

NICTmed at the NCTIR-16 Real-MedNLP Task
Masao Ideuchi

Masatoshi Tsuchiya

National Institute of Information and Communications
Technology
Japan
FUJITSU LIMITEDJapan
Nara Institute of Science and TechnologyJapanmasao.ideuchi@nict.go.jp

National Institute of Information and Communications
Technology
Japan
Toyohashi University of TechnologyJapantsuchiya@imc.tut.ac.jp

Yiran Wang

Masao Utiyama

National Institute of Information and Communications
TechnologyJapanyiran.wang@nict.go.jp

National Institute of Information and Communications
TechnologyJapanmutiyama@nict.go.jp

ABSTRACT

1This paper describes NICTmed team’s challenge to Subtask1-CR-EN,
Subtask1-CR-JA, Subtask3-CR-EN, Subtask3-CR-JA, Subtask3-RRJA and Subtask3-RR-EN in NTCIR-16 Real-MedNLP. Real-MedNLP
is information extraction task in real medical domain text, andapproximately 100 annotated real clinical reports in both English and Japanese are given to its participants. Subtask1-CR-EN/JA
and Subtask3-CR-EN/JA are the tasks based on case reports (CR),
Subtask1 is few-resource Named Entity Recognition (NER) and
Subtask3 is information extraction for adverse drug event (ADE).
Subtask3-RR-EN/JA are the tasks of case identification (CI) basedon radiology reports (RR). We used multilingual BERT (mBERT)
and XLM-RoBERTa (XLM-R) to compare how effective the multilingual pre-trained models work in specific domain downstreamtasks both English and Japanese. Our experiment used no external data to adjust conditions of English and Japanese experiments.
Our experiment showed that the multilingual pre-training models
achieved level of accuracy in Japanese as in English, and got rank 3
in Entity F1 of all target entities for Subtask1-CR-JA, top rank inReport-level precision and F1 for Subtask3-CR-JA.

Recently, the number of annotated corpora published becomeslarger, but Japanese corpora are not enough compared with English,
and the number of natural language processing researches use
English corpora and pre-training models. The corpora without
annotation in Japanese are available such as Japanese Wikipedia1
and Aozora Bunko2 , and Japanese pre-training models are alsoavailable. However, for specific domains and specific tasks, mostannotated corpora are available only in English, and difficult toprepare data in Japanese.
When trying to solve a task in a specific domain in Japanese, themost common first step is to find reference systems which solve thesimilar tasks in the specific domain using pre-trained models withEnglish corpora. In such cases, reproducing the reference systems
with a multilingual pre-training model and then applying to atarget Japanese corpus is a good practice because the changes fromreference system are reduced and problem identification becomes
easily when the accuracy is not enough. Especially, a small change
in machine learning system may bring a large change[7], therefore,changing and comparing a part of experimental conditions froma reference system help to identify how we can improve accuracy
for practical use.
In Real-MedNLP[8], annotated real clinical reports were provided
in both Japanese and English. With the multilingual pre-trainedmodels become familiar, the corpora available in multiple languages
for the similar tasks are increasing, but annotated corpora in multiple languages for specific domains and tasks are still rare. The
bilingual reports in Real-MedNLP are good examples for examininn
how well multilingual pre-trained models work in a particular domain. In this paper, we used multilingual pre-trained models thatare trained on general corpora and fine-tuned to tasks in a specificdomain then we confirmed the accuracy achieved and the differences in accuracy between English and Japanese. As a result, werevealed that the multilingual pre-trained models can work in a
specific domain both English and Japanese.

KEYWORDS
Medical Natural Language Processing, Named Entity Recognition,
Multilingual Pre-trained Language Model, Adverse Drug Event,Case Identification

TEAM NAME
NICTmedSUBTASKS
Subtask1-CR-EN
Subtask1-CR-JA
Subtask3-CR-EN (ADE)
Subtask3-CR-JA (ADE)
Subtask3-RR-EN (CI)
Subtask3-RR-JA (CI)

INTRODUCTION

1 https://ja.wikipedia.org/
2 https://www.aozora.gr.jp/339

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan3

The tasks for which we submitted results are Subtask1-CR-EN,
Subtask1-CR-JA, Subtask3-CR-EN (ADE), Subtask3-CR-JA (ADE),
Subtask3-RR-EN (CI), and Subtask3-RR-JA (CI). However, we expected Subtask3-RR-EN (CI) and Subtask3-RR-JA (CI) are supervised classification problem at first, but they are clustering problem in fact and no significant results were obtained in those tasks.
Therefore, we report mainly Subtask1-CR-EN, Subtask1-CR-JA,
Subtask3-CR-EN (ADE), and Subtask3-CR-JA (ADE) in this paper.
Section 2 describes the definition of the task, and Section 3 describes the data used in our experiment and how they are processed.
Section 4 describes the configuration of our system, Section 5 describes the details of the official submissions, and concludes with
Section 7.

2

3.1

NER in Subtask1-CR-EN and
Subtask1-CR-JA

An NER task is generally treated as a sequence labeling problem
with 𝐵𝐼𝑂 formed labels same as CoNLL[9]. The 𝐵 label represent
the beginning of an entity, the 𝐼 label represents other tokens inside an entity, and the 𝑂 label represents all other non-entity tokens. If there are multiple types of labels, 𝐵-𝐸𝑛𝑡𝑖𝑡𝑦𝐶𝑎𝑡𝑒𝑔𝑜𝑟𝑦 and 𝐼 𝐸𝑛𝑡𝑖𝑡𝑦𝐶𝑎𝑡𝑒𝑔𝑜𝑟𝑦 are used as NER labels for each type of entities. For
training, the input text is divided into tokens 𝑋 = (𝑥 1, 𝑥 2, 𝑥 3, ...𝑥𝑛 ),and NER labels for each token 𝑌 = (𝑦1, 𝑦2, 𝑦3, ...𝑦𝑛 ) are given. In
predicting, the input text is divided into tokens same as training,then the trained model predicts NER labels for each token.
In Subtask1-CR-EN and Subtask1-CR-JA, we converted text lines
in XML files described in Subsection 3.1 to 𝐵𝐼𝑂 labeled data. We
considered a method to predict 𝐵𝐼𝑂 for each kind of label and amethod to predict all kind of labels in the form of 𝐵-𝐸𝑛𝑡𝑖𝑡𝑦𝐶𝑎𝑡𝑒𝑔𝑜𝑟𝑦
and 𝐼 -𝐸𝑛𝑡𝑖𝑡𝑦𝐶𝑎𝑡𝑒𝑔𝑜𝑟𝑦 as options. There were some labels that could
not be predicted at all since the number of 𝐵 and 𝐼 labels become
1% or less of the total number of tokens. Therefore, we adopted aformat in which all kinds of labels are assigned to one file in theform of 𝐵-𝐸𝑛𝑡𝑖𝑡𝑦𝐶𝑎𝑡𝑒𝑔𝑜𝑟𝑦 and 𝐼 -𝐸𝑛𝑡𝑖𝑡𝑦𝐶𝑎𝑡𝑒𝑔𝑜𝑟𝑦.

2.2

Data Format

The training and test data for CR were provided as XML formatedfiles with multiple reports bundled together for each language anddataset. The XML tags that define the metadata and structure of
documents are on separate lines in a XML file, and each report isseparated by lines of “<article>” and “</article>”. The otherlines of XML files are text lines of reports and the line count of eachlanguage and dataset are shown in Table 1.
In the training data, named entities (NEs) are marked up with
XML tags in text lines, and in the test data, no XML tags exist
in text lines. The English and Japanese XML files are documentlevel parallel corpora, which means each report can be paired withthe translated report but each sentence cannot be paired with thetranslated sentence. In many cases, the long sentences in Japanese
article are divided in English articles and there are some unnatural
NEs because of translation. For example, in some English TimeDate NEs, the words “this” and “study” are included and this isbecause the words “¡ñ” in Japanese articles are translated to “in thisstudy”.

TASK DEFINITION

We formulated Subtask3-CR-EN (ADE) and Subtask3-CR-JA (ADE)
as extended labeled tasks of named entity recognition (NER) from
Subtask1-CR-EN and Subtask1-CR-JA those are given as NER. Subsection 2.1 describes the definition of an NER task in Subtask1,
and Subsection 2.2 describes how we converted given ADE labels
to a NER task in Subtask3. For Subtask3-RR, we vectorized documents using the multilingual pre-trained BERT model then appliedclustering.

2.1

DATA

In Real-MedNLP, two types of reports were distributed, Case Report (CR) and Radiology Report (RR). The use of external data wasallowed, but we didn’t use external data, except pre-trained models,in order to adjust the experimental conditions in English and Japanese. We focus on CR data and explain the data format in Subsection
3.1 and annotation format in Subsection 3.2, and the relationshipwith the ADE training data in Subsection 3.3. For RR, data format
and annotation format are same as CR. We removed tags for vectorization because the multilingual pre-trained model treats plain
text.

3.2

Annotation Format

There are 11 types of XML tags in CR data, and some types havemultiple attributes. There are both essential and optional attributes,
however, some of the optional attributes are assigned almost entirely. Therefore, we transfered the XML tags and attributes into 31
types of NER labels according to the actual assignment conditions.
The NER labels used in experiments and XML tags and attributes
in original data, and the number of labels in the training data areshown in Table 2.
For the sequential labeling problems in Subtask1-CR, the total
number of the NER labels are 63, 𝐵 and 𝐼 NER labels for each NE
and 𝑂 for other tokens, were used.

3.3

Subtask3-CR-EN (ADE) and Subtask3-CR-JA(ADE)

ADE supervised training data

The supervised training data for ADE are given as CSV files withcolumns of “report ID”, “NE class” (medicine or disease), ”text” of
the NE, and “𝐴𝐷𝐸𝑣𝑎𝑙”. Only in Japanese training data, an additional
column that contains “standard name” for NEs if available. 𝐴𝐷𝐸𝑣𝑎𝑙
is a categorical integer values that are on ordinal scale from 0 to
3 that expresses the degree of suspicion of ADEs. 𝐴𝐷𝐸𝑣𝑎𝑙 = 0 indicates “Unrelated”, 𝐴𝐷𝐸𝑣𝑎𝑙 = 1 indicates “Unlikely”, 𝐴𝐷𝐸𝑣𝑎𝑙 = 2
indicates “Probably”, and 𝐴𝐷𝐸𝑣𝑎𝑙 = 3 indicates “Definitely”, respectively. The total count of 𝐴𝐷𝐸𝑣𝑎𝑙 = 𝑁 and count of the reports that
contain 𝐴𝐷𝐸𝑣𝑎𝑙 = 𝑁 on given training data are shown in Table 3.

Subtask3-CR-EN/JA ADE prediction is the task of predicting the𝐴𝐷𝐸𝑣𝑎𝑙 from given “articleID”, “tag”, and “text”. 𝐴𝐷𝐸𝑣𝑎𝑙 is a categorical number and the value is from 0 to 3 where the higher value
corresponds to the higher certainty level. 𝐴𝐷𝐸𝑣𝑎𝑙 is assigned to thelist of NEs, executed medicines and diseases diagnosed as positive,
described in each case report. We treated 𝐴𝐷𝐸𝑣𝑎𝑙 as an additional
attribute of each NE and predicted 𝐴𝐷𝐸𝑣𝑎𝑙 by solving NER taskwith extended labels.

340

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 1: Line Count of CR data
Line category

Line count of CR-EN-training

Line count or CR-JA-training

Line count of CR-EN-test

Line count or CR-JA-test

XML tag lines for document attributes
article tag lines
Text lines of reports

15
298
2,160

15
298
1,908

13
152
1,602

13
152
1,524

Total lines count

2,473

2,221

1,767

1,689

Table 2: NER labels for Subtask1 and XML tags and attributes
in CR training data
XML tags

Attributes

Count (EN)Count (JA)dpositive
suspicious
negativegeneral

1698
80
250
302

1698
80
249
302

Disease-Positive
Disease-Suspicious
Disease-Neagtive
Disease-General

a

-

818

822

Anatomical

f

-

636

637

Feature

c

-

571

571

Changetimex3datetime
duration
set
age
med
misc

537
53
81
34
189
430
28

537
52
82
34
189
428
28

Time-Date
Time-TimeTime-Duration
Time-Set
Time-Age
Time-MedTime-Miscscheduledexecutednegated
other

0
366
7
18

0
366
7
18

Test_Name-Executed
Test_Name-NegatedTest_Name-Other

t-key

-

524

524

Test_Item

t-val

-

428

428

Test_Value

m-keyscheduledexecutednegated
other

0
266
27
51

0
266
27
51

Medicine_Name-Executed
Medicine_Name-Negated
Medicine_Name-Other

m-val

-

64

64

Medicine_Valuerscheduledexecutednegated
other

28
545
30
77

28
544
30
77

Remedy-Scheduled
Remedy-Executed
Remedy-Negated
Remedy-Othercc

scheduledexecutednegated
other

1
243
3
17

1
242
3
17

Clinical_Context-Scheduled
Clinical_Context-Executed
Clinical_Context-NegatedClinical_Context-Othert-testmedicines in ADE supervised training data. We found 𝐴𝐷𝐸𝑣𝑎𝑙 are
given to the diseases and medicines when diseases are diagnosed as
positive and medicines are executed in Subsection 3.2, respectively.
Moreover, even when the same diseases or medicines were written
more equal than two times in a report, the 𝐴𝐷𝐸𝑣𝑎𝑙 appears only
once in the ADE training data. We subcategorized the NER labels of“Disease-Positive” and “Medicine-Executed” in Subtask1 and used
new NER labels shown in Table 4 for Subtask3.

NER labels in Subtask1

Table 4: NER labels detailed in ADE task
NER labels in Subtask1

4

Count (EN)Count (JA)

Report count (EN)

Report count (JA)

0
1
2
3

1,382
63
84
175

1,320
61
79
170

146
21
21
24

146
21
21
24

All

1,704

1,630

147

147

NER labels in Subtask3 (ADE)

Disease-Positive

0
1
2
3

Disease-Positive-0
Disease-Positive-1
Disease-Positive-2
Disease-Positive-3

Medicine-Executed

0
1
2
3

Medicine-Executed-0
Medicine-Executed-1
Medicine-Executed-2
Medicine-Executed-3

For the sequential labeling problems in Subtask3-CR, the total
number of the NER labels are 69, subcategolized 𝐵 and 𝐼 NER labels
for each NE and 𝑂 for other tokens, were used.

SYSTEM DETAILSFor the CR task, we reproduced a NER reference system describedin Subsection 4.1 at first, and then renewed the implementation ona newer library. After that, we prepared the CR data as describedin Subsection 4.2, and fine-tuned with the pre-trained models described in Subsection 4.3. The evaluation results in training data
were described in Subsection 4.4. The submission data format was
XML, we applied post-processing described in Subsection 4.5 to the
NER prediction results. For the RR task, we removed tags from document and vectorized each article by the mBERT model described in
Subsection 4.3, then compressed into 3 dimensions by using t-SNE,
and applied k-means clustering. The k values in the k-means are
determined by silhouette analysis.

Table 3: ADEval counts in training data
ADEval𝐴𝐷𝐸𝑣𝑎𝑙4.1

A preliminary experiment

As a preliminary experiment, we reproduced the NER experiments
using English open medical data and renewed implementation. We
referred the NER implementation and evaluation of BlueBERT[6]that is one of pre-trained BERT[2] model using papers in PubMed3 .
We used BC5CDR[10] data on BLUE benchmark site4 and confirmed NER for the disease and chemical NEs. Two types of data

𝐴𝐷𝐸𝑣𝑎𝑙 is heavily biased to 0, and if 𝐴𝐷𝐸𝑣𝑎𝑙! = 0, it is biased to 3.
In addition, because ADEs were not mentioned in every report, 146
of 147 reports in both English and Japanese include 𝐴𝐷𝐸𝑣𝑎𝑙 = 0, and104 reports only include 𝐴𝐷𝐸𝑣𝑎𝑙 = 0 for all diseases and medicines
in the reports.
To understand data relationships in detail, we compared diseases
and medicines annotated to CR texts and the list of diseases and3 https://pubmed.ncbi.nlm.nih.gov/
4 https://github.com/ncbi-nlp/BLUE_Benchmark

341

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 5: Fine-tuning parameters

format were used in the preliminary experiment, a Pubtator filein which annotations are defined in a tab-delimited span data after the title and abstract text5 , and a TSV file in which the text isdivided into tokens and given NER labels (hereinafter, NER file).
The original implementation was worked with Tensorflow v1, so
we editted to work with SimpleTransformers. As a result of thepreliminary experiment, we couldn’t reproduce the accuracy described in the BlueBERT paper by using both pre-trained BlueBERTin GitHub6 and HuggingFace7 , but the NER on BC5CDR disease
and for chemical are achieved about 80 and 90 in F-measure, respectively using multilingual pre-trained BERT model. These scores
seemed to be reasonable because the models that are pre-trainedwith general corpora and the models that are pre-trained with multilingual corpora are known to be slightly inferior than models
that are pre-trained with specific domain monolingual corpora. We
inherited implementation of this preliminary experiment as a reference system, we switched the data to Real-MedNLP CR text andperformed subsequent experiments.

4.2

Parameters
learning_rate
optimizer
scheduler
warmup_steps
train_batch_size
num_train_epochs

4.4

4e-5
AdamW
linear_schedule_with_warmup
016
10

Evaluation in training data

For submission, we used all training data for fine-tuning, but weused divided training data for evaluation in previous experiments.
We divided the training data into three part, 𝑡𝑟𝑎𝑖𝑛-𝑡𝑟𝑎𝑖𝑛, 𝑡𝑟𝑎𝑖𝑛-𝑣𝑎𝑙𝑖𝑑
and 𝑡𝑟𝑎𝑖𝑛-𝑡𝑒𝑠𝑡, and confirmed the accuracy of NER task. Since thenumber of CR data was so small, 𝑡𝑟𝑎𝑖𝑛-𝑡𝑒𝑠𝑡 were the last 10 reports,
and 𝑡𝑟𝑎𝑖𝑛-𝑣𝑎𝑙𝑖𝑑 were the previous 10 reports, and 𝑡𝑟𝑎𝑖𝑛-𝑡𝑟𝑎𝑖𝑛 wereother reports. The evaluation results based on training data areshown in Table 6.

Data preparation

At first, we converted the CR XML files into Pubtator files by referring to the format and implementation of the preliminary experiment. In CR files, the titles of reports were given as attributesof the <article> tags and not annotated. Therefore, we treated
all text of a report in CR files as abstracts and made Pubtator files
without titles. Then we extracted XML tags and attributes from CR
files, with character indices of the start points and end points ofthe tags as span information.
Next, we converted the Pubtator files to NER files. For the English
text, we treated words as tokens according to the preliminary experiment, and divided some symbols adjacent to words such as periods
and commas as other tokens. For the Japanese text, we separated
text character by character when using mBERT, and tokenized text
using SentencePiece[3] attatched to XLM-R when using XLM-R. In
Subtask1, we assigned all the labels described in Table 2 to eachtoken, and in Subtask3, we also used the labels in Table 4. Somepart in the CR data, the start point of a word didn’t correspondto the start point of an XML tag because spaces between words
sometimes included in XML tags. Therefore, we repaired the headof 𝐼 label to 𝐵 label if the NER labels start from 𝐼 label without 𝐵
label then we obtained the NER training data.

4.3

ValuesTable 6: Evaluation result in training data NER

4.5

Data

Pretrained models

Precision

RecallF1

CR-EN Subtask1 NER

mBERT
XLM-R

45.1
45.0

50.8
49.2

47.8
47.0

CR-JA Subtask1 NER

mBERT
XLM-R

57.4
63.3

65.1
68.1

61.0
65.6

CR-EN Subtask3 NER

mBERT
XLM-R

47.3
44.8

51.6
49.3

49.4
46.9

CR-JA Subtask3 NER

mBERT
XLM-R

57.0
60.9

62.9
67.2

59.8
63.9

Postprocess for submission

We processed NER results for submission. In Subtask1, submission
format was an XML file, and in Subtask3, submission format was a
CSV file, we converted the files because prediction results of NER
were NER files.
For Subtask1, similar to data preparation, there were some NER
results that start from 𝐼 labels, we applied two correction options
for to a series of NEs starting from an 𝐼 label. The first option was
repairing from 𝐼 to 𝐵 label if the label starts from the 𝐼 label. The
second option was exclusion, we converted series of 𝐼 labels withouta 𝐵 label to 𝑂 labels. Then we converted NER files to XML files byadding XML tags to the text using span information. Since it was
not obvious which correction was better, we submitted the resultsof two options.
For Subtask3, since the target NEs to be predicted are given in
CSV, we determined the predicted value of the ADEval by which
NER labels were predicted for the texts of NEs. We merged NER
results with weights if multiple NER labels are predicted in texts in
NEs. When nothing is predicted by NER, the ADEval is determinedby text and weighted score based on the training data.

Pretrained models and fine-tuning
parameters

We used mBERT and XLM-R[1] as multilingual pre-trained models,mBERT was from official web site8 , and XLM-R was xlm-robertabase from HuggingFace9 . The fine-tuning parameters were shownin Table 5.

5 https://www.ncbi.nlm.nih.gov/research/bionlp/APIs/format/
6 https://github.com/ncbi-nlp/bluebert
7 https://huggingface.co/bionlp
8 https://github.com/google-research/bert/blob/master/multilingual.md
9 https://huggingface.co/xlm-roberta-base342

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

5

6.1

OFFICIAL RESULTS

The experimental settings and Entity-F1 scores of the systems thatwere submitted to Subtask1-CR are as shown in Table 7.
Table 7: Subtask1-CR submitted systems’ results
Language

Pretrained models

ID

NE label correction

mBERT

1
2

𝐼 to 𝐵𝐼 to 𝑂

47.09
48.85

XLM-R

3
4

𝐼 to 𝐵𝐼 to 𝑂

49.42
51.70

mBERT

1
2

𝐼 to 𝐵𝐼 to 𝑂

56.90
60.70

XLM-R

3
4

𝐼 to 𝐵𝐼 to 𝑂

55.50
58.13

EnglishJapanese

Comparison with different level labelings

We examined whether the level of NER labels affect the accuracyof predictions. If the prediction accuracy was clearly deteriorated
with detailed NER labels, some additional methods from other viewpoints are necessary, such as increasing the number of cases. We
trained 3 models that are trained from different levels of NER labels,
then compared prediction results of NEs that have at least 5 cases inthe 𝑡𝑟𝑎𝑖𝑛-𝑡𝑒𝑠𝑡. The levels are 1) ADE level that is used for Subtask3CR (see Subsection 3.3), 2) detail level that is used for Subtask1-CR(see Subsection 3.2), and 3) coarse level that is summarized to “XMLtags” in 2. All prediction results are summarized to coarse level forcomparison. The results are shown in Table 10.

F1

Table 10: Evaluation results in coarse level NE labels
NEs
DiseaseAnatomical
FeatureChangeTime
Test Name
Test Item
Test Value
Medicine Name
RemedyClinical Context

The experimental settings and Report-level F1 scores of the systems that were submitted to Subtask3-CR are as shown in Table
8.
Table 8: Subtask3-CR submitted systems’ results
Language

Pretrained models

ID

Report-level F1English

mBERT
XLM-R

1
2

34.48
42.11

Japanese

mBERT
XLM-R

1
2

48.00
32.00

6.2

6

ID

Normalized Mutual Info

EnglishJapanese

mBERT
mBERT

1
1

21.72
17.44

65.4
50.0
31.1
28.6
68.3
79.3
57.1
63.2
30.8
56.8
27.6

65.1
50.4
26.7
28.6
65.0
80.7
41.9
46.2
30.8
59.4
35.7

coarse level
64.3
53.3
27.9
27.8
65.4
75.4
34.1
48.0
0.0
56.8
29.6

Evaluation with other methods

We experimented with NegSampling-NER10 and NER-BET-CRF11
implementations and compared with baselines of SimpleTransformer’s NER implementation. We changed the Japanese tokenization using pre-trained model’s tokenizer. Both methods are basedon BERT, the NegSampling-NER is a span-based NER method[4],and the NER-BET-CRF is a method with a CRF layer. The results
are shown in Table 11.
The implementation of NegSampling-NER and NER-BET-CRF
produced better results than that of SimpleTransformers. In theimplementation of NER-BET-CRF, two results of BERT and BETCRF were produced for comparison, but the accuracy of fine-tuned
BERT was greatly improved from the SimpleTransformers implementation. Although detailed analysis of the model was not carriedout, the NER-BERT-CRF has the the layer called prediction mask. It
seems to be possible to improve the accuracy of SubTask1-CR and
SubTask3-CR by changing the method used for the NER task to amore accurate method.

Table 9: Subtask3-RR submitted systems’ results
Pretrained models

81
38
30
16
56
22
11
7
5
56
12

NE label level
ADE level detail level

From these results, there wer no NEs whose accuracy was significantly deteriorated as the levels of NE labels become more detail.
Therefore, subsequent experiments described in the following subsection were conducted using the detail level NE labels. That levelis same as Subtask1-CR, which is considered easy to compare with
official results.

The experimental settings and normalized mutual info scores ofthe systems that were submitted to Subtask3-RR are as shown in
Table 9.

Languagecount in 𝑡𝑟𝑎𝑖𝑛-𝑡𝑒𝑠𝑡

ADDITIONAL EXPERIMENTS

Additional experiments were performed on Subtask1-CR using the𝑡𝑟𝑎𝑖𝑛-𝑡𝑟𝑎𝑖𝑛, 𝑡𝑟𝑎𝑖𝑛-𝑣𝑎𝑙𝑖𝑑 and 𝑡𝑟𝑎𝑖𝑛-𝑡𝑒𝑠𝑡 data used in Subsection 4.4,
because the results seem to similar to the official results. First, we
checked how much the accuracy changes by detail level of thelabels as described in 6.1. Then we use other methods and data on
Subtask1-CR and compared with results in Table 6 as the baseline,
the results of the experiments are described in Sebsection 6.2 and
6.3.

10 https://github.com/LeePleased/NegSampling-NER
11 https://github.com/Louis-udm/NER-BERT-CRF

343

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 11: Evaluation results with other methods

6.3

MethodLanguage

F1baseline1 (SimpleTransformers NER with mBERT)

EnglishJapanese

47.8
61.0

baseline2 (SimpleTransformers NER with XLM-R)

EnglishJapanese

47.0
65.6

NegSampling-NER

EnglishJapanese

53.3
65.7

NER-BERT-CRF (BERT)

EnglishJapanese

62.6
70.7

NER-BERT-CRF (BERT-CRF)

EnglishJapanese

65.1
72.4

REFERENCES[1] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer,
and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learningat Scale. Association for Computational Linguistics, Online. 8440–8451 pages.
https://doi.org/10.18653/v1/2020.acl-main.747[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding(Proceedings of the 2019 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, Volume 1 (Longand Short Papers)). Association for Computational Linguistics, 4171–4186. https://doi.org/10.18653/v1/N19-1423[3] Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language
independent subword tokenizer and detokenizer for Neural Text Processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (2018), 66–71. https://www.aclweb.org/anthology/
D18-2012[4] Yangming Li, lemao liu, and Shuming Shi. 2021. Empirical Analysis of Unlabeled
Entity Problem in Named Entity Recognition. In International Conference on
Learning Representations. https://openreview.net/forum?id=5jRVa89sZk[5] Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. Contrastive Learningfor Many-to-many Multilingual Neural Machine Translation (Proceedings of
the 59th Annual Meeting of the Association for Computational Linguistics andthe 11th International Joint Conference on Natural Language Processing (Volume
1: Long Papers)). Association for Computational Linguistics, 244–258. https://doi.org/10.18653/v1/2021.acl-long.21[6] Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer Learning in Biomedical
Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. Association for Computational Linguistics, Florence, Italy. 58–65
pages. https://doi.org/10.18653/v1/W19-5006[7] D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar
Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in Machine learning systems. , 2503–2511 pages.
[8] Yada Shuntaro, Nakamura Yuta, Wakamiya Shoko, and Aramaki Eiji. 2022. RealMedNLP: Overview of REAL document-based MEDical Natural Language Processing Task. Proc. of the 16th NTCIR Conference on Evaluation of Information
Access Technologies (2022).
[9] Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 Shared Task:
Language-Independent Named Entity Recognition. https://aclanthology.org/W022024[10] Wonjin Yoon, Chan Ho So, Jinhyuk Lee, and Jaewoo Kang. 2019. CollaboNet:collaboration of deep neural networks for biomedical named entity recognition.
BMC Bioinformatics 20, 10 (2019), 249. https://doi.org/10.1186/s12859-019-2813-6

Evaluation with translated NE data

Replacing some words with another language works well with multilingual model in translation task[5]. Therefore, we replaced thewords of NEs with another language and treat that as additional
training corpus. We aligned sentences of the documents and treatedNEs as translation dictionary if the NE with same tag and attribute
was only one in paired sentences. The experiments with the methods described in Subsection 6.2 and the additional training data areshown in Table 12.
Table 12: Evaluation results with NE translated data
MethodLanguage

F1

diff

SimpleTransformers NER with mBERTEnglishJapanese

51.5
60.0

+4.3
-1.0

SimpleTransformers NER with XLM-R

EnglishJapanese

49.6
66.1

+2.6
+0.5

NegSampling-NER

EnglishJapanese

54.4
65.6

+1.1
-0.1

NER-BERT-CRF (BERT)

EnglishJapanese

64.4
68.9

+1.8
-1.8

NER-BERT-CRF (BERT-CRF)

EnglishJapanese

66.6
70.1

+1.5
-2.3

In English with all methods, F1 scores were improved over 1.0.
In Japanese, the F1 score of XLM-R was slightly improved, but F1
scores of other methods using mBERT were deteriorated. Since
there are various combinations in replacement of other languages,
further verification is necessary.

7

CONCLUSIONS

We confirmed that the accuracy of fine-tuned multilingual pretrained models in Japanese can be higher or at least as high as
those of English with only a small amount of data. In Japanese,characteristic substrings are used in disease names or anatomicalnames, therefore the difficulty of the task was less difficult thanin English. This results suggest that translated texts may makeeasier to solve specific domain tasks. By additional experiments, wefound the accuracy of base NER tasks can be improved with othermethods and translated articles with annotation can be useful foreven monolingual NER task.

344