Team Yseop at #SMM4H 2024: Multilingual Pharmacovigilance Named Entity Recognition and Relation Extraction (2024)

Abstract: This paper describes three RoBERTa based systems. The first one recognizes adverse drug events (ADE) in English tweets and links them with MedDRA concepts. It scored F1-norm of 40 for the Task 1. The next one extracts pharmacovigilance related named entities in French and scored a F1 of 0.4132 for the Task 2a. The third system extracts pharmacovigilance related named entities and their relations in Japanese. It obtained a F1 of 0.5827 for the Task 2a and 0.0301 for the Task 2b. The French and Japanese systems are the best performing system for the Task 21.

heading: 1 Introduction
text: As of 2008, the European Commission proposed certain measures2 to protect the public from the harm caused by Adverse Drug Reaction (ADR). Nonetheless, Koyama et al. (2023) observed a global increase in Adverse Drug Event (ADE) related deaths. Thus, pharmacovigilance, i.e. activities involving detection, comprehension, and prevention of adverse effects due to medication is an important subject. With the arrival of internet, the general public started using it to seek and share health related information3 (Gerber and Eiser, 2001). With the help of natural language processing systems, this publicly available data can be analysed to extract information related to side effects. As a result, it can play a key role in strengthening pharmacovigilance reporting systems.
Note: Even though, the “ICH E2A Clinical safety data management: definitions and standards
1All the models will shared on https://huggingface. co/yseop and the code will be available on https://github. com/yseop/YseopLab
2https://ec.europa.eu/commission/presscorner/ detail/cs/MEMO_08_782
3https://web.archive.org/web/20150924101434/ https://www.pushdoctor.co.uk/Resources/ PushDoctor-Health-report.pdf
for expedited reporting”4 distinguishes between ADR and ADE, in this paper we will use the terms interchangeably to refer to the unintended consequences of taking a prescribed medication.
SMM4H-2024: The 9th Social Media Mining for Health Research and Applications Workshop and Shared Tasks — Large Language Models (LLMs) and Generalizability for Social Media NLP (Xu et al., 2024) proposed 7 shared tasks. We participated in two of them:
• Task 1: Extraction and normalization of adverse drug events (ADEs) in English tweets
• Task 2: Cross-Lingual Few-Shot Relation Extraction for Pharmacovigilance in French, German, and Japanese

heading: 2 Task 1 - English
text: In this task, we have to identify ADEs in a short text, called “tweets”, written in English. If ADEs are present, then they have to be mapped to Preferred Terms Id (ptId) from the Medical Dictionary for Regulatory Activities (MedDRA)5.
Due to an error on our part, we did this task with the SMM4H -2023 Task 5 dataset, shared on the task’s Google Groups on 06/02/2024.

heading: 2.1 Dataset
text: The training set had 17385 tweets, out of which only 1239 mentioned any ADE. The tweets and the annotations were provided in two separate files. An example of a tweet:
SMM4H2022uCZV2SRsCe4vzjFm @USER__________ have to go to a doc now to see why i'm still gaining. stupid paxil made me gain like 50 pounds ?? and now i have to lose it
4https://database.ich.org/sites/default/files/ E2A_Guideline.pdf
5https://www.meddra.org/
136
The annotation file had the spans (start and end position in the tweet) and lowest level term6 id (llt) of each ADE mention:
SMM4H2022uCZV2SRsCe4vzjFm ADE 61 68 gaining 10047896
A MedDRA dictionary (llt.asc) containing the mapping between ADE, llt, and preferred term id (ptid) was also provided.
For the tweets in quotes we found that the spans were off by 1, we corrected that for the training.

heading: 2.2 Model
text: We augmented data with English texts from the SM-ADE sub-task (Wakamiya et al., 2023) to train a binary classifier that can distinguish between the tweets having ADE from those without ADE. The resulting classifier could get a F-1 score 0.73 on the validation set. This was less than the F1 achieved by Gupta and Rayar (2023)’s multilingual Bert model with similar dataset. Therefore, we did not use the classifier.
We fine-tuned RoBERTa7 (Liu et al., 2019) for token classification8 using Huggingface (Wolf et al., 2020). We preprocessed the training data by aligning the annotations with tokens. The first token of an ADE entity was labelled B-MISC and remaining tokens were labelled I-MISC. The nonADE tokens in the input text were labelled O. While training, the model was evaluated on the validation set using seqeval’s9 f1_score.
We kept aside 20% of the training data for validation using scikit-learn’s (Pedregosa et al., 2011) stratified train_test_split and fine-tuned the model on a) 80% of the train set, and b) on augmented training data consisting of the 80% of the provided training corpus and the non-ADE English data from the SM-ADE sub-task. There was not much of the difference in the performance (see Figure 1), so we did not submit the model trained on the augmented data.
The ADEs detected by the first model (see Table 1 for the parameters used) were searched in the MedDRA dictionary with the help of SentenceTransformers’10 (Reimers and Gurevych, 2019)
6https://www.meddra.org/how-to-use/basics/ hierarchy
7https://huggingface.co/FacebookAI/ roberta-base
8https://huggingface.co/docs/transformers/en/ tasks/token_classification
9https://github.com/chakki-works/seqeval 10https://www.sbert.net/index.html
multi-qa-mpnet-base-dot-v1 model. The results (sub1.zip) were submitted to the leader board.

heading: 2.3 Test Results
text: The model obtained a F1-norm score of 40.0 compared to 43.9 by the baseline provided by DeepADEMiner (Magge et al., 2021). -Norm metrics are calculated by comparing the normalized ADE, i.e. llt. Table 2 compares the performance of the model with the baseline and the median of all the submissions.

heading: 3 Task 2 - French and Japanese
text: The goals of this task were Named Entity Recognition (NER) and Relation Extraction (RE). The corpus consisted of, mainly, German and Japanese text taken from online forums and Twitter/X. It also had four French documents translated from German texts.

heading: 3.1 Dataset
text: The French dataset had 4 documents and the Japanese had 392. The documents were annotated in the brat standoff format11 (Stenetorp et al., 2011). There are 3 entities and 2 relations:
• Entities:
– DISORDER, a symptom not necessarily related to a drug
– DRUG – FUNCTION, bodily functions
• Relations: 11https://brat.nlplab.org/standoff.html
– CAUSED, the first entity causes the second entity
– TREATMENT_FOR, the first entity remedies the second one
An example of French data with annotation:
Salut <user>, pour moi, ça a commencé à l'âge de <pi>. J'ai suivi une thérapie et une cure pendant deux ans, ... Prends soin de toi. <user>
T4 DRUG 123 130 pilules ... R3 CAUSED Arg1:T10 Arg2:T15 R13 TREATMENT_FOR Arg1:T24 Arg2:T26
An example of Japanese data with annotation:
881844583344201728: 昼間のレクサプロが副作用ひどくて未 だに気持ち悪い
T1 DRUG 22 27 レクサプロ T2 DISORDER 28 31 副作用 T3 DISORDER 38 43 気持ち悪い R1 CAUSED Arg1:T1 Arg2:T2
There were some inconsistencies in the data, for example the Japanese term錠 was not annotated as DRUG in most of the documents. Whereas the French equivalent pilules (see the example above) and other common nouns such as 薬 and 製品 were.
In the Japanese training, the span of certain entities was updated as shown in the Table 3.

heading: 3.2 Task 2a - French Model
text: Since there was not enough training data, we decided to use Mistral-7B-Instruct-v0.112 (Jiang et al., 2023) and DrBERT-CASM213 via the medkit14 library. DrBERT (Labrak et al., 2023) was fine-tuned on CASM2 corpus for NER task to produce DrBERT-CASM2. The CASM2 is a private corpus that contains documents from CAS (Grabar et al., 2018).
The Mistral LLM is prompted with the prompt described in Appendix A and parameters do_sample=False and max_new_tokens=256. If the entities returned by the LLM are in the text they are added to the list of candidates. Then, all the entities extracted by DrBERT-CASM2 are added to the candidates. Lastly, the entities in the candidate list are used to find other substrings in the text.
12https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.1
13https://huggingface.co/camila-ud/ DrBERT-CASM2
14https://medkit.readthedocs.io/en/stable/ index.html

heading: 3.3 Task 2a - Japanese Model
text: Baseline: If we label all occurrences of 副作 用 in the training set as DISORDER; 薬 and 製品 as DRUG, we get a Macro F1 of 0.1658. If we also annotate all consecutive sequence of katakana characters that are not present in the JMdict_e15 (Japanese–Multilingual Dictionary) as DRUG, the Macro F1 becomes 0.2842.
We kept aside the 20% of the provided train dataset as validation set using scikitlearn’s stratified train_test_split. Then on the remaining dataset we fine-tuned daisaku-s/medtxt_ner_roberta16 as token classification task. This model was previously trained on MedTxt-CR dataset (Yada et al., 2022). The seqeval F1 score was better than the baseline and hence it was submitted to the leader-board.

heading: 3.4 Task 2b - Japanese Model
text: The training data contains:
• 390 examples of DRUG causing DISORDER
• 100 examples of DRUG TREATMENT_FOR DISORDER
• 98 examples of DISORDER causing DISORDER
• 20 examples of DRUG causing FUNCTION
• 8 examples of DISORDER causing FUNCTION
• 3 examples of DRUG TREATMENT_FOR FUNCTION
out of 8497 possible relations. From the train set, we created a new corpus for relation classification. Similar to Zhong and Chen (2021), we extracted, for each pair of entities, the text between them (entities included). If there is no relation between the pair, it is annotated as ’O’, otherwise the label in the train set was used. For the example in Figure 2, we take the text span抗 うつ剤に関しては抵抗がありましたが、安 酒を煽るより100倍は建設的な精神状態 and annotate it as CAUSED.
We kept aside the 20% of the new corpus as validation set. Then on the remaining dataset
15http://www.edrdg.org/wiki/index.php/ JMdict-EDICT_Dictionary_Project
16https://huggingface.co/daisaku-s/medtxt_ner_ roberta
we fine-tuned daisaku-s/medtxt_ner_roberta as sequence classification task17.

heading: 3.5 Test Results
text: The results are presented in Table 4. One of the reasons for the bad performance of the Japanese model is tokenization error. For example, in the text ja_twjp_060-080_19 one of the entities is DISORDER 47 50副作用, however at the given span, the daisaku-s/medtxt_ner_roberta tokenizer returnsの副作用 as the single token.

heading: 4 Conclusion
text: In Task 1, despite training on the wrong dataset we managed to be in the top 50 percentile. The difficult part was normalization of the ADE using MedDRA dictionary, as a result F1-Norm was lower than F1NER. For the Task 2, using a model adapted to the clinical domain helped to get the best results. The Task 2b (relation extraction) was challenging, given that the winning team obtained an overall F1 score of 0.0189. For future, We would explore approaches such as GLiNER (Zaratiana et al., 2023) and XMC (D’Oosterlinck et al., 2024) to improve NER in Task 1 and Task 2a.

heading: A Mistral Prompt Template
text: The LLM input used for NER is a few-shot prompt containing two examples. It uses the text from fr_1025_lifeline_v1_FR_1971_1_1647857960 and fr_1069_lifeline_v1_FR_6168_1_1648459053 as examples. The entities and the relations for each example is used as shown below:


 From the medical report in French below, extract all the mentions of entities DRUG, DISORDER and the relationships CAUSED and TREATMENT_FOR in brat format. TEXT 1
 DRUG pilules DISORDER angoisses FUNCTION règles ... DISORDER humeur FUNCTION hormones CAUSED Arg1:pilules Arg2:angoisses ... TREATMENT_FOR Arg1:Insidon Arg2:humeur


 From the medical report in French below, extract all the mentions of entities DRUG, DISORDER and the relationships CAUSED and TREATMENT_FOR in brat format. TEXT 2 
 DRUG mirtazapine DISORDER problèmes de sommeil FUNCTION dors ... CAUSED Arg1:antiémétiques Arg2:somnolence ... TREATMENT\_FOR Arg1:zolpidem Arg2:troubles du sommeil


 From the medical report in French below, extract all the mentions of entities DRUG, DISORDER and the relationships CAUSED and TREATMENT_FOR in brat format. TEXT FROM THE TEST SET 


heading: Future Work
text: For future, We would explore approaches such as GLiNER (Zaratiana et al. , 2023) and XMC (D’Oosterlinck et al. , 2024) to improve NER in Task 1 and Task 2a