HBUT at #SMM4H 2024 Task2: Cross-lingual Few-shot Medical Entity Extraction using a Large Language Model (2024)

Abstract: Named entity recognition (NER) of drug and disorder/body function mentions in web text is challenging in the face of multilingualism, limited data, and poor data quality. Traditional small-scale models struggle to cope with the task. Large language models with conventional prompts also yield poor results. In this paper, we introduce our system, which employs a large language model (LLM) with a novel twostep prompting strategy. Instead of directly extracting the target medical entities, our system firstly extract all entities and then prompt the LLM to extract drug and disorder entities given the all-entity list and original input text as the context. The experimental and test results indicate that this strategy successfully enhanced our system performance, especially for German language. Our code is available on Github 1.

heading: 1 Introduction
text: Discussions on drugs and their adverse drug reactions shared by users in social media, including the efficacy, side effects, and personal treatment journeys serve as valuable references for pharmacovigilance.
We participated in the 9th Social Media Mining for Health Research and Applications Workshop and Shared Tasks (Xu et al., 2024). The task targets both the extraction of drug and disorder/body function mentions (Subtask 2a) and the extraction of relations between those entities (joint Named Entity Detection and Relation Extraction, Subtask 2b). The paper is about the task 2a. Our task was to identify drug and disorder mentions in German, French, and Japanese datasets from X(Twitter) and a German patient forum.
In previous works on similar tasks, people commonly use BERT models (Kenton and Toutanova, 2019). For multilingual tasks, the m-BERT (multilingual BERT) model is often employed (Papadim-
1https://github.com/YinZhangJu/SMM4H24code
itriou et al., 2021). However, in this particular open task, the provided dataset is relatively small, especially in the training set. Therefore, it is challenging to perform pre-training and fine-tuning on the traditional BERT model due to the limited amount of data available. Thus, we opted to use a large language model (LLM) for the task.
Because the training data is also not enough to fine-tune a LLM well, we opted for the promptbased approach. Our system utilizes a two-step prompting strategy. A first step to get a list of all entities, and then a second to further extract drug and disorder entities from the list.

heading: 2 Methodology
text: Due to the small size of the provided dataset, we placed our focus primarily on using large language models (LLMs). We build our system based on a GLM (Du et al., 2022; Zeng et al., 2022) through its online API. It is trained on multilingual dataset and performs almost on a par with ChatGPT and worked well for multilingual tasks in our preliminary experiments.
We utilize few-shot prompt engineering technologies to adapt GLM to the drug and disorder mention extraction task as shown in Figure 1.

heading: 2.1 Prompting Method
text: The task is challenging for the numerous abbreviations and colloquial vocabulary in the dataset, which requires our system to be able to recognize such unofficial representations. In our preliminary experiments, we found that our system failed to work well with conventional prompts that guide the LLM to extract drugs and disorder mentions from the input text.
Thus, we address this issue by a novel strategy. Firstly, instead of instructing the LLM to extract drug/disorder mentions, our system prompts the model to extract all entities, just with additional
58
attention on medicine-related entities. Then our system prompts the model to extract drugs and disorders given both the entity list and the original input as the context. In this way, our system is more robust for informal representations and ill-formed sentences. The detailed settings of our two-step prompting are described in Tables 1 and Tables 2.

heading: 2.2 Preprocessing for Sensitive Words
text: The GLM API raised errors for some samples in the test dataset, with error messages indicating that the input texts contained some sensitive words.
When our system detects such failures, it collects and writes the sample IDs, sample contents, and error messages in an error log. We manually check the error records, identify and remove the sensitive words in the corresponding samples, and then redo generation for them.
This method may produce incomplete and illformed sentences. However, because our two-step design is robust against ill-formed text, the adverse effects on the NER accuracy are relatively small. Replacing sensitive words with alternative words may result in better NER performance. However, we did not have enough time to work out a replacement table for the sensitive words in the shared task. We plan to complete it in future work.

heading: 2.3 The Post-Processing Step
text: We save the output results to a CSV file. Then, to match the submission format, a Python script
is used to remove punctuation and symbols, and convert the results into the BRAT format.

heading: 3 Experiments and Results

heading: 3.1 Experiments on LLM Model Selection
text: Conventional works reported that Qwen-14B (Bai et al., 2023) performs well for factual tasks. Thus, we also conducted experiments with Qwen-14B in comparison to GLM.
We randomly sampled 100 samples from the
outputs generated by Qwen-14B and GLM respectively, and manually checked the correctness. The accuracy scores achieved by the two models in this local experiment is as shown in Table 3. Qwen14B failed to get satisfying results. Qwen-14B exhibited tokenization issues with German, French, and Japanese data, resulting in significant problems
with fabricated entities and incorrect scope. In case of GLM, there are several available versions. We tried GLM-3-turbo and GLM-4. However, we encountered some encoding issues with GLM-4. When answering questions in German and French, we observed that GLM-4 firstly translated the text into English and then provided an English response. This led to inaccuracies in the output of German and French words. In contrast, GLM-3-turbo exhibited minimal issues in this regard. Therefore, we used GLM-3-turbo in our system. We used the default generation parameters. The temperature is set to 0.95, top p is set to 0.7, and max tokens is set to 1024.

heading: 3.2 Experiments on Prompting Methods
text: We conducted experiments on three prompt strategies as follows,
• Single-word Prompting (Single Prompt): In this approach, we directly guided the model to perform named entity recognition on German, French, and Japanese data.
• Detached Step-by-step Prompting (DSBS Prompt): In this approach, we utilized a stepby-step prompting method. In the first step, we initially prompted the LLM model to generate a list of all entities in the input text, instead of just medical-related entities like that in conventional system. In the second step, we let the LLM model to extract drug and disorder entities from the entity list.
• Integrated Step-by-step Prompting (ISBS Prompt): This approach is also a step-by-step prompting method. The first step is as the same as DSBS prompt. But in the second step, we combined the entity list and the original input text in the prompts.
The results are shown in Table 4. Single Prompt uses fewer tokens. However, it exhibited lower precision in capturing phrase scope, with larger discrepancies compared to the gold annotation.
DSBS Prompt resulted in more accurate identification of entity words and achieved higher precision. However, as it detached from the original text, it performed poorly in extracting drugs and disorder mentions from the entity list.
ISBS Prompt led to more accurate identification of entity words, higher precision, and effective extraction of drugs and disorder mentions from the entity list.

heading: 4 Final Results
text: Our final system employed GLM-3-turbo model with ISBS Prompt. The overall results are shown in Table 5. Our system achieved a better precision than the mean precision among all teams (including baseline).
Especially, our method demonstrated excellent performance for the German part. The results for the German part are shown in Table 6. The overall precision, recall and F1 scores are higher than the mean scores among all the participants. However, our system failed to address mentions of function.

heading: 5 Conclusion
text: Due to the small size of our dataset, we found that the performance with conventional small language models (e.g. BERTs) is not able to give satisfying precision. Consequently, we build our system based on GLM-3-Turbo large language model. Besides, we found that conventional straight-forward prompting strategy encountered low precision for this task. To address this issue, we proposed a step-by-step prompting strategy. We firstly prompt to extract a list of all entities. Then we prompt to guide the model to choose drugs and disorder mentions from the list within the context of the corresponding original input text. This prompting approach significantly improved the effectiveness of our system.

heading: A Appendix
text: A.1 Dataset Details The detailed information of the dataset is shown in Table 7.

heading: Future Work
text: We plan to complete it in future work