NTCIR-17 kickoff event
Sept 28, 2022

NTCIR-17
Medical Natural Language Processing
for Social media and Clinical texts(MedNLP-SC)
Eiji Aramaki, Ph.D. (NAIST, Japan) Co-chair (general)
Shoko Wakamiya, Ph.D. (NAIST, Japan) Co-chair (general)
Shuntaro Yada, Ph.D. (NAIST, Japan) Co-chair (Subtask SM)
Yuta Nakamura, M.D. (The University of Tokyo, Japan) Co-chair (Subtasks RR)
1

MedNLP series
NameText

NTCIR-10 MedNLP pilot Dummy health records (by hand)

TaskLanguages

NERJA

NTCIR-11 MedNLP-2

Dummy health records (from examinations) NER, NEN

JA

NTCIR-12 MedNLPDoc

Dummy health records (from the textbook) ICD coding

JA

NTCIR-13 MedWeb

Dummy social media texts (created bycrowdsourcing)

Phenotyping

JA
ZH

EN

NTCIR-16 Real-MedNLP Real health records: Case reports andRadiology reports

NER, ADE, CI

JA

EN

NTCIR-17 MedNLP-SC
Social Media (SM)

ADE

JA
FREN
DE

NER, TNM

JA

Dummy social media texts (SM)NTCIR-17 MedNLP-SC Real health records: Radiology reports
Radiology Report (RR) (RR)
Named Entity Recognition (NER)
Named Entity Normalization (NEN)
Adverse Drug Event detection (ADE)

Case Identification (CI) (MedNLP original task)
TNM cancer staging (TNM) (MedNLP original task)
2

MedNLP-SC subtasks
Information extraction from medical reports written byphysicians or social media data
• Subtask for Social Media (SM)

• Adverse drug event detection (ADE)• Text: Social media data (n=20,000 artificial tweets)• Language: Japanese, English, French c , and German• Subtasks for Radiology Report (RR)

• (a) Named Entity Recognition (NER)• (b) TNM staging (TNM)• Text: Radiology reports (n=135 reports or more)• Language: Japanese3

Subtask for Social Media (SM)Social media data (tweets)• Create a Japanese corpus consisting of 20,000 artificial tweets• Out of a maximum of 68,000 artificial tweets (= 68 drugs x 1000 tweets)generated using T5• Each tweet is manually checked and annotated symptom label(s)• Translate the corpus into the other three languages by machinetranslation with manual check• All language shares the same symptom label(s)Samples of symptom labels

LabelJapaneseEnglishFrenchGermanPAIN疼痛

paindouleur

Schmerzen

CONSTIPATION便秘constipation

constipation

Verstopfung

DIARRHEA

下痢

diarrhea

diarrhéeDiarrhöe

NAUSEA

吐き気

nauseanauséesÜbelkeit

VOMITING

嘔吐

vomiting

vomissements

vomissements 4

Subtask for Social Media (SM)

Artificial tweet samples of “gamma globulin”(1/2)• Example 1
検索もせずただひたすら我慢してるだけマシなんだが...。γグロブリン⼤量投与で⼀発逆転できるのかしら、それともガンマグロブリン⼤量投与か......ってなると何回やっても思い出せない(I'm better off not searching for it and just holding on... I wonder if a large dose of
gamma globulin would turn it around, or a large dose of gamma globulin. ...... I
can't remember how many times I've tried.) translated by DeepL• Example 2: No drug name
持病の関係で昨⽇病院⾏ったら⾎液検査の結果が出て「リンパ球減少症」と診断されました\nでも今⽇からまた薬が増えます〜!頑張れ私!!ʼ
(I went to the hospital yesterday because of my chronic illness and the results of myblood test came back and I was diagnosed as "lymphopenia." Hang in there me!)translated by DeepL

5Subtask for Social Media (SM)

Artificial tweet samples of “gamma globulin”(2/2)• Example 3: No ADE

けど、このところ連⽇の⾼熱が続きすぎて⾟いです...今夜も頑張って寝ます!(*^^*)今⽇は病院に⾏って⾎液検査とγグロブリン注射をしてきました。これで少しでも良くなりますように!!お騒がせしましたm(=_=;")早く元気になって欲しいものです(But I'm having a hard time because I've had too many high fevers in a row recently... Iwill try my best to sleep tonight! (*^^*) I went to the hospital today for a blood test
and a gamma globulin shot. I hope this will make me feel a little better! Sorry for thetrouble m(=_=;")I hope she gets well soon!) translated by DeepL• Example 4: Just mentioned<user_name> さん、こんにちは。私も昨⽇2回⽬打ってきましたよ!\n接種後から頭痛と熱っぽさが出ていたのでγグロブリン製剤を飲むように指⽰されたそうです。'(Hello, <user_name>. I got my second shot yesterday too! ¥They told me to take agamma globulin preparation because I had a headache and feverishness since afterthe ⲟn vaccination.’) translated by DeepL

6

Subtask for Social Media (SM)

Adverse Drug Event detection (SM-ADE)• Multi-labeling task to identify a set of symptoms caused by adrug (section)
Given an input sentence in the section of FU5:
“After FU5 starts, I am suffering from dry cough and many mouth ulcers.”
The expected output labels:“dry cough” and “mouth ulcers”• Similar to hashtag recommendation because symptom
label(s) can be represented as hashtags

e.g., “After FU5 starts, I am suffering from dry cough and many mouthulcers. #dry_cough #mouth_ulcers"
7Subtask for Social Media (SM)FAQ for Social Media Subtask(a) The performance of T5 to create tweet-like short messages

Answer: To ensure that the artificial tweets are sufficiently similar toreal ones, we will conduct the Turing Test, which annotators determine
whether a tweet is real or artificial(b) The quality of the artificial tweets

Answer: To keep the quality of our dataset, artificial tweets will bechecked by annotators and modified/removed if not appropriate(c) The appropriateness of applying machine translation

Answer: Due to the cost (both time and money), we will employ machinetranslation(d) Evaluation methods

Answer: We plan to employ F-measure
8Subtasks for Radiology Report (RR)

image: https://radiopaedia.org/cases/cavitating-lung-cancer-1
Report: J-MedStd RR

MedTxt-RR Corpus
A set of radiology reports of the same cancer cases diagnosed bydifferent radiologists independently• The latest version contains 135 texts for 15 cases and 9 radiologists,
some of which are already available at https://sociocom.naist.jp/medtxt/rr/• We are planning to make the dataset larger by recruiting additional cancer
cases and radiologists
Radiologist A’s report

…
…

A…

B
Radiologist

Radiologist B’s report

両肺には軽度の気腫性変化、左肺上葉外側にbullaを認めます。
左肺下葉には内部に空洞を伴う⻑径7.8cm ⼤の不整形腫瘤を認め、左肺下葉に⻑径78mm⼤の腫瘤があり，既知の肺癌と考えます。

同⼀肺葉内には副結節を疑う複数の病変を認めます(cT4)。

左肺下葉中枢側にはsatellite lesionと思われる結節を複数認めます。

腫瘤は胸膜と広範に接しており、胸膜外脂肪層も⼀部途絶が⾒られ、壁側胸膜浸潤を疑います。

胸壁に広く接しており，胸壁脂肪織の消失も認められ，胸壁に浸潤している可能性もあります。

肋⾻への浸潤ははっきりしません。
腫瘤は横隔膜とも⼀部接していますが、浸潤所⾒ははっきりしません。
左肺⾨や気管分岐下や⼤動脈下などの同側の縦隔リンパ節の腫⼤を認め、転移を疑います(cN2)。

9

左肺⾨部，気管分岐下，左下部気管傍リンパ節が腫⼤しており，リンパ節転移と考えます。

肝嚢胞、軽度脾腫を認めます。
その他、撮像内の腹部に特記所⾒ありません。

撮像内で肝，⾻には転移を認めません。副腎は撮像範囲外です。

Subtasks for Radiology Report (RR)(a) Named Entity Recognition (RR-NER)
Guideline Learning (proposed inSubtask 2 of NTCIR-16 Real-MedNLP)• Provide only the guideline text forhuman annotators• 30-40 example sentences annotated• A sample of annotation guideline isavailable athttps://sociocom.naist.jp/real-mednlp• Participants can use any other resources
outside this task if they find them usefulfor their methods● Test <t-test/key/val>● Diseases and symptoms <d>● Medicine <m-key/val>● Anatomical entities <a>• Can we teach a model as if it is a
● Features and measurements <f> ● Remedy <r>human?
● Clinical Context <cc>
● Change <c>● Time <timex3>

10

Subtasks for Radiology Report (RR)(b) TNM staging (RR-TNM)
To evaluate the generalization ability of NLP models whenclassifying radiology reports under multiple criteria (cancerstaging)

According to predefined criteria, participant systems assign four labels• tumor (T) 原発のがんの広がり• lymph node (N) がん細胞のリンパ節への転移の有無と広がり• metastasis (M) 原発から離れた臓器への遠隔転移• clinical stage (cStage)e.g.,• Early cancers will be labeled with small numbers such as T1N0M0, cStage 1• Advanced ones will be labeled with large numbers such as T3N2M1, cStage 411

Subtasks for Radiology Report (RR)

FAQ for Radiology Report Subtasks(a) How to build dataset

Answer: We collaborate with a teleradiology company and use an openaccess radiology case repository. Radiology reports are written byradiologists diagnosing real, public, and anonymized cancer images.

(b) The dataset size

Answer: We are working on adding data, which will make the current
dataset (n=135) larger.

(c) RR-TNM: definition of cancer stage

Answer: We follow the Union for International Cancer Control (UICC)
criteria as clinicians do. We are planning to use the 7th and 8th editions tosimulate revision.
12Collaboration
To develop high-quality multilingual data, we collaborate with researchers
from DFKI (Germany), LISN-CNRS (France), and RIKEN and NII (Japan)
••
•••
•
••
•
•
•
•
•
•
•
•
•
•

Akiko Aizawa, Ph.D. (NII, Japan)
Gabriel Herman Bernardim Andrade (NAIST, Japan)
Cyril Grouin, Ph.D. (Université Paris-Saclay, CNRS, LISN, France)
Shouhei Hanaoka, MD, PhD (The University of Tokyo, Japan)
Thomas Lavergne, Ph.D. (Université Paris-Saclay, CNRS, LISN, France)
Yuji Matsumoto, Ph.D. (RIKEN, Japan)
Faith Wavinya Mutinda (NAIST, Japan)
Aurélie Névéol, Ph.D. (Université Paris-Saclay, CNRS, LISN, France)
Noriaki Nishida, Ph.D. (RIKEN, Japan)
Tomohiro Nishiyama (NAIST, Japan)
Patrick Paroubek, Ph.D. (Université Paris-Saclay, CNRS, LISN, France)
Lisa Raithel (DFKI, Germany)
Roland Roller, Ph.D. (DFKI, Germany)
Hiroki Teranishi, Ph.D. (RIKEN, Japan)
Philippe Thomas, Ph.D. (DFKI, Germany)
Narumi Tokunaga (RIKEN, Japan)
Hui-Syuan Yeh (Université Paris-Saclay, CNRS, LISN, France)
Pierre Zweigenbaum, Ph.D. (Université Paris-Saclay, CNRS, LISN, France)

13

Tentative Schedule• March 2023: Dataset Release• (March-June 2023: Dry Run)
• June 1, 2023: Registration Deadline• July 2023: Formal Run• August 1, 2023: Evaluation Result Release• August 1, 2023: Task overview paper release (draft)• September 1, 2023: Submission due of participant papers (draft)• November 1, 2023: Camera-ready participant paper due• December 2023: NTCIR-17 Conference in NII, Tokyo14