NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan
DOI: https://doi.org/10.20736/0002001285

NAISTSOCRR at the NTCIR-17 MedNLP-SC Radiology Report
SubtaskTakuya Fukushima

Yuka Otsuki

Shuntaro Yada

Nara Institute of Science andTechnology
Japanfukushima.takuya.fo2@naist.ac.jp

Nara Institute of Science andTechnologyJapanotsuki.yuka.pb0@naist.ac.jp

Nara Institute of Science andTechnologyJapans-yada@naist.ac.jp

Shoko Wakamiya

Eiji Aramaki

Nara Institute of Science andTechnologyJapanwakamiya@naist.ac.jp

Nara Institute of Science andTechnologyJapanaramaki@naist.ac.jp

ABSTRACT

to more advanced ones such as T3N2M1. While TNM staging isessential for treatment and research, it traditionally requires intricate reasoning by medical professionals. An inherent challenge inthis realm is the periodic “stage migration” due to the revision ofcancer staging criteria. The necessity to reclassify under updated
guidelines, especially in large patient databases, underscores thepotential value of efficient NLP systems in medical advancements.
Further details of this task can be found in the NTCIR MedNLP-SC
website1 and the NTCIR-17 MedNLP-SC task overview paper [3].
We implemented three standard methods based on several pretrained language models (PLMs). As the PLMs, we used the representative Japanese models including those trained on medical
datasets. The three proposed methods have distinct approaches tohandling the training objectives. The first method employs whole
TNM label combinations as distinct labels. The second method
undertakes multi-task learning using three separate labels (T, N,
and M) as objective variables. to predict three labels separately as
multi-label classification.The third method trains a separate modelfor each of the three labels T, N, and M. The inherent advantage
of our method’s straightforwardness is its reproducibility. It does
not only demonstrate the potential of NLP in the medical field,
but also paves the way for future studies by providing a replicablefoundation.

This paper describes how we tackled the Medical Natural Language
Processing for Radiology Report TNM staging (RR-TNM) Subtask
as participants of NTCIR17. The RR-TNM Subtask is a MedNLPSC original task to classify radiology reports under multiple criteria. We introduced three different methods based on pre-trained
language models (PLMs), including a medical-specific model. Notably, our combination approach, utilizing JMedRoBERTa (manbyowordpiece) for label T, Tohoku-BERT-v3 for label N, and UTH-BERT
for label M, achieved an accuracy of 0.3704 on the test data. This
performance was the highest among all participants, emphasizing
the effectiveness of our strategy.

KEYWORDS
Medical Natural Language Processing, Case Reports, Radiography
Reports, Case Identification

TEAM NAME
NAISTSOCRR

SUBTASKS
RR-TNM

1

2

INTRODUCTION

MATERIALS AND METHODSThe RR-TNM dataset consists of 253 Japanese radiology reports
originating from the open access image data sets. Thus, it can be
distributed without privacy concerns. These radiology reports were
written by nine certified radiologists, who diagnosed the same open
27 lung cancer cases independently. The 27 cases was split into
the training set (108 documents for 12 cases), validation set (54documents for six cases), and test set (81 documents for nine cases).
We introduce our methods consisting of a baseline and the proposed three PLM-based methods.

Recently, the shift from traditional paper-based medical records
to electronic format has become more pronounced, making theaccessibility and storage of patient data more streamlined. This digital transformation not only modernizes healthcare documentation
but also underscores the ever-growing significance of information
processing techniques in medical fields. The capability to swiftly
and accurately process vast amounts of data can greatly influencediagnosis, treatment decisions, and research outcomes.
In such a situation, we tackled MedNLP-SC Subtask 2: Radiology
Report TNM staging (RR-TNM). This task is to classify radiology
reports based on the TNM system, a globally recognized framework
for categorizing cancer severity and spread. This system consistsof three key dimensions: tumor (T), lymph node (N), and metastasis (M), effectively translating into three simultaneous multiclassclassification tasks. Labels range from early stages like T1N0M0

2.1

Majority baseline

The majority baseline always return the most frequent TNM label(i.e., T1N0M0) as the prediction result. The top 5 frequently labels
1 https://sociocom.naist.jp/mednlp-sc/

163

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan• Criterion: Cross Entropy Loss
• Optimization: AdamWin the train data are as follows: T1N0M0 (15 instances), T2N0M0(13 instances), T4N2M0 (13 instances), T2N2M1 (11 instances) andT3N1M1 (7 instances).

2.2

PLM-based methods

3

We propose three methods based on pre-trained language models(PLMs). We adopted BERT, a Transformers-based large-scale language model [1], because of its representativeness of the PLMs. To
solve RR Subtask, we designed three training objectives:
Unilabel single-model Unilabel single-model fine-tuned bysingle label, which is a joint of three labeled of T, N, and M.
Multilabel single-model Multilabel single-model treating T,
N, and M independently as three distinct labels. We employeda multi-task learning paradigm for fine-tuning the model.
Multilabel multi-models Multilabel multi-models employsa segmented approach by independently fine-tuning three
distinct models, each with T, N, and M as their individual
training objectives.
For these objectives, we fine-tuned the following pre-trained
BERT models:
Tohoku-BERT-v22
The Tohoku-BERT-v2 is the commonly used Japanese pretrained language model. This model was trained on the Japanese version of Wikipedia. The training corpus is sourced fromthe Wikipedia Cirrussearch dump file dated August 31, 2020.
UTH-BERT [2]
The UTH-BERT is a Japanese model trained on medical data.
Specifically, it was pre-trained using diagnostic records.
JMedRoBERTa (sentencepiece) [4]The JMedRoBERTa (sentencepiece) is a Japanese model trainedon medical data, specifically pre-trained using medical articles. It employs the SentencePiece tokenizer, which directly
tokenizes sentences into subwords.
JMedRoBERTa (manbyo-wordpiece) [4]The JMedRoBERTa (manbyo-wordpiece) utilizes the "Manbyo" dictionary, a large-scale disease name dictionary, in
conjunction with Mecab for tokenization. It then employs
the WordPiece method to further tokenize into subwords.
Tohoku-BERT-v33
The Tohoku-BERT-v3 is enhanced versions of the TohokuBERT-v2, having been trained on a more extensive dataset.
These models were trained on both the Japanese segment of
the CC-100 dataset and the Japanese version of Wikipedia.
For the latter, a text corpus was generated from the Wikipedia
Cirrussearch dump file as of January 2, 2023.
Tohoku-BERT-v3-char4
Tohoku-BERT-v3-char is a varidation of Toholu-BERT-v3
that employs character-level tokenization.
We set the hyperparameters for fine tuning to the values below:• Batch size: 16• Learning rate: 2e-5
• Max length: 512• Early stopping rounds: 5

RESULTS

Table 1 shows the validation results of the Unilabel single-model performance. The results demonstrated that JMedRoBERTa (manbyowordpiece) achieved the best performance with macro F1 of 0.1510
and Accuracy of 0.4259.
Table 2 shows the validation results of the Multilabel singlemodel performance. The results demonstrated that JMedRoBERTa(sentencepiece) recorded the best performance with macro F1 of
0.2147 and Accuracy of 0.4630.
Table 3 shows the validation results of the Multilabel multimodels performance. The results demonstrated that JMedRoBERTa(manbyo-wordpiece) recorded the best performance with macro F1of 0.2278 and Accuracy of 0.5000. Finally, combining the prediction
results of the following three models, which had the best prediction
performance for T, N, and M. We obtain macro F1 of 0.2932 andAccuracy of 0.5926.
• T: JMedRoBERTa (manbyo-wordpiece)• N: Tohoku-BERT-v3• M: UTH-BERT
We submitted three prediction results for the test data using
Baseline, JMedRoBERTa (sentencepiece) on the Multilabel singlemodel, and Combination (the best models for T, N, and M) on theMultilabel multi-model. Table 4 shows the accuracy in the test data.
Similar to the results for the validation data, Combination (the bestmodel for T, N, and M) on the Multilabel multi-models was the best
not only among our methods but also against all other participants.

4

DISCUSSION

In order to accurately predict the TNM staging, the prediction performance of T, N, and M all need to be improved. In particular,
compared to M, T and N still have challenges in prediction performance. In this paper, we introduced a PLM-based method, but it isworthwhile to try to take advantage of recent methods based onlarge language models. This is because insufficient number of data
is a problem in the medical field, including the present task. Large
language models such as GPT-4 are expected to be utilized becauseof their high performance in Few-shot learning.
To provide clarity on prediction inaccuracies, we conducted anerror analysis using confusion matrices for each label: T, N, and M.
Figure 1 shows the confusion matrix in the validation data.
From the analysis, the major sources of errors for label T, theprimary source of error was misclassification between classes 1 and
3. Label N mainly struggled with distinguishing between classes
0 and 2 and also between classes 3 and 2. For label M, there was abalanced misclassification between the two classes. Based on theerror analysis, Future efforts should focus on enhancing the model’s
discernment between closely related classes in labels T and N, andaddressing the balanced errors in label M.

2 https://huggingface.co/cl-tohoku/bert-base-japanese-v2
3 https://huggingface.co/cl-tohoku/bert-base-japanese-v3
4 https://huggingface.co/cl-tohoku/bert-base-japanese-char-v3

164

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 1: Validation score of Unilabel single-model
T
Model Name

N

M

All

Macro F1

Accuracy

Macro F1

Accuracy

Macro F1

Accuracy

Macro F1

Accuracy

Baseline (most frequency label)

0.12500.33330.16670.50000.33330.50000.05000.3333

Tohoku-BERT-v2
UTH-BERTJMedRoBERTa (sentencepiece)
JMedRoBERTa (manbyo-wordpiece)
Tohoku-BERT-v3
Tohoku-BERT-v3-char

0.42920.44610.39670.45630.41090.39110.5556
0.61110.40740.57410.51850.51850.37440.42230.45200.42010.38580.41370.70370.77780.70370.75930.70370.75930.65890.6506
0.78750.79050.78750.74440.68520.68520.79630.79630.79630.75930.10930.1307
0.09230.15100.13710.12510.33330.37040.25930.42590.35190.3519* Bold font indicates the best score for each evaluation metric.

Table 2: Validation score of Multilabel single-model
T
Model Name

N

M

All

Macro F1

Accuracy

Macro F1

Accuracy

Macro F1

Accuracy

Macro F1

Accuracy

Baseline (most frequency label)

0.12500.33330.16670.50000.33330.50000.05000.3333

Tohoku-BERT-v2
UTH-BERTJMedRoBERTa (sentencepiece)
JMedRoBERTa (manbyo-wordpiece)
Tohoku-BERT-v3
Tohoku-BERT-v3-char

0.40010.50180.62020.6436
0.38660.36020.44440.48150.59260.68520.38890.46300.3964
0.39330.5566
0.42230.40840.42680.70370.68520.75930.77780.70370.77780.76990.82860.81070.81070.83190.74440.77780.83330.81480.81480.83330.75930.17570.13670.21470.20850.14030.11480.44440.33330.46300.44440.35190.3148* Bold font indicates the best score for each evaluation metric.

Table 3: Validation score of Multilabel multi-models
TModel Name

N

M

All

Macro F1

Accuracy

Macro F1

Accuracy

Macro F1

Accuracy

Macro F1

Accuracy

Baseline (most frequency label)

0.12500.33330.16670.50000.33330.50000.05000.3333

Tohoku-BERT-v2
UTH-BERTJMedRoBERTa (sentencepiece)
JMedRoBERTa (manbyo-wordpiece)
Tohoku-BERT-v3
Tohoku-BERT-v3-char

0.47640.59990.57780.66680.34710.4646

0.48150.57410.59260.74070.37040.48150.62280.56000.69620.67590.70230.43150.81480.74070.77780.79630.83330.79630.80820.92590.88750.81070.81070.86920.81480.92590.88890.81480.81480.87040.12740.15700.18630.22780.05370.16730.37040.4074
0.46300.50000.25930.3704

Combination (best model for T, N, and M)

0.66680.74070.70230.83330.92590.92590.29320.5926* Bold font indicates the best score for each evaluation metric.

5

CONCLUSION

label. This result suggested that the independent label handlingshows the feasibility against the limited data task.

In this paper, we presented the methods of the NAISTSOCRR teamin the NTCIR17 MedNLP-SC Radiology Report (RR) Task. We employed three methods based on pre-trained language models, including those trained on medical data, in addition to a baseline
with most frequent labels. As a result, the best-performing models
differed for each of T, N, and M, and the best predictions for TNM
staging were achieved by combining the best models from each

ACKNOWLEDGEMENTS
This work was supported by JST, AIP Trilateral AI Research, Grant
Number JPMJCR20G9, JST AIP-PRISM Grant Number JPMJCR18Y1,
Japan, as well as ANR grant ANR-20-IADJ-0005-01, France, under
the trilateral ANR-DFG-JST AI call.

165

NTCIR 17 Conference: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, December 12-15, 2023, Tokyo, Japan

Table 4: Accuracy in test data
Method

T

N

M

All

Baseline (most frequency label)
JMedRoBERTa (sentencepiece) on the Multilabel single-model
Combination on the Multilabel multi-models

0.30860.60490.66670.41980.60490.56790.59260.87650.83950.19750.30860.3704

Team A’s best model
Team B’s best model

0.48150.39510.60490.80250.74070.92590.2346
0.2716* Bold font indicates the best score for each evaluation metric.

Figure 1: Confusion matrix in validation data

REFERENCES[3] Yuta Nakamura, Shouhei Hanaoka, Shuntaro Yada, Shoko Wakamiya, and Eiji
Aramaki. 2023. NTCIR-17 MedNLP-SC Radiology Report Subtask Overview:
Dataset and Solutions for Automated Lung Cancer Staging. In Proceedings of the
17th NTCIR Conference on Evaluation of Information Access Technologies. https://doi.org/10.20736/0002001328[4] Kaito Sugimoto, Taichi Iki, Yuki Chida, Teruhito Kanazawa, and Akiko Aizawa.
2023. JMedRoBERTa: a Japanese Pre-trained Language Model on Academic Articles
in Medical Sciences (in Japanese). In Proceedings of the 29th Annual Meeting of the
Association for Natural Language Processing. https://www.anlp.jp/proceedings/
annual_meeting/2023/pdf_dir/P3-1.pdf[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. In
Proceedings of the 2019 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, Volume 1 (Longand Short Papers). Association for Computational Linguistics, 4171–4186. https://doi.org/10.18653/v1/N19-1423[2] Yoshimasa Kawazoe, Daisaku Shibata, Emiko Shinohara, Eiji Aramaki, andKazuhiko Ohe. 2021. A clinical specific BERT developed using a huge Japanese clinical text corpus. PLoS One 16, 11 (Nov. 2021), e0259763.

166