PolyuCBS at SMM4H 2024: LLM-based Medical Disorder and Adverse Drug Event Detection with Low-rank Adaptation (2024)

Abstract: This is the demonstration of systems and results of our team’s participation in the Social Medical Mining for Health (SMM4H) 2024 Shared Task. Our team participated in two tasks: Task 1 and Task 5. Task 5 requires the detection of tweet sentences that claim children’s medical disorders from certain users. Task 1 needs teams to extract and normalize Adverse Drug Event terms in the tweet sentence. The team selected several Pre-trained Language Models and generative Large Language Models to meet the requirements. Strategies to improve the performance include cloze test, prompt engineering, Low Rank Adaptation etc. The test result of our system has an F1 score of 0.935, Precision of 0.954 and Recall of 0.917 in Task 5 and an overall F1 score of 0.08 in Task 1.

heading: 1 Introduction
text: The rise in people using social media for health information has resulted in a significant increase in health-related data, which allows researchers to harness the information, along with NLP and Machine Learning methods, to contribute to public health (Klein et al., 2024). The 9th Social Media Mining for Health Applications (SMM4H) Shared Tasks, aiming to advance methods that utilise social media data for health research, have a special focus on Large Language Models (LLMs) and Generalizability for Social Media NLP.
There are 7 tasks given in the 9th SMM4H workshop (Xu et al., 2024). Our team focused on Task 1 and Task 5. Task 1 has two sub-tasks, which are (1) detecting Adverse Drug Event terms in tweets and (2) normalizing these colloquial mentions to their standard concept to Preferred Term IDs according to MedDRA. Task 5 focuses on classifying tweets reporting children’s medical disorders. The challenge of this task is differentiating between tweets from pregnancy users who declared that their child has specific conditions with
attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorders (ASD), delayed speech, or asthma, and tweets that just mention the disorder. The main challenges of both tasks are as follows: (1) medical terms conveyed in colloquial language, a common issue in social media data, which might misguide the trained model; (2) How to activate, transfer and utilize the knowledge learnt from the pre-training in Pre-trained Language Models(PLM) and Large Language Models (LLM). To address these issues, we use both PLMs and LLMs as basis and implement parameter-efficient tuning during training. To move a step further, we also conduct prompt engineering along with task decomposition to fully activate the knowledge of the LLM and ensure its understanding of the task, which could bridge the gap between the LLM and the specific downstream task.

heading: 2 Methodology

heading: 2.1 Task 1
text: For Task 1, BERT (Devlin et al., 2019) was utilized as the baseline model to conduct domain continual pre-training (Gu et al., 2021; Peng et al., 2021) and fine-tuning on the training dataset, during which BIO tags were utilized. In the extraction task, the models chosen are BERT and LLaMA-2(Touvron et al., 2023). The normalization task was regarded as a classification problem, the models chosen are SapBERT (Liu et al., 2021) and CODER (Yuan et al., 2022). Fine-tune: For ADE extraction in Task 1, two fine-tuning modes are being experimented with: the first is by extracting the [CLS] in the last layer of the BERT model and utilizing the data tagged by BIO tag set to train the BERT classification model; the second is by treating the extraction task as a cloze test task. Given a label sets Y = {y1, y2, . . . yk}, for each tweet sentence S = {t1, t2, . . . tn} where t is the token and n
74
is the number of tokens in S. For a token tq, we create template P = {p1, p2, . . . pm} where P = S + tq + cloze, eg. A tweet sentence, a token in this sentence, this is __. We mask pi, the ith token in P, into [mask] and construct token-label converter vy to ensure the label y1 has a token set Cy1 = {c1, c2...cn} that mapping token c into label y1. We extract the last layer’s representation in the [mask] position to get the fixed-length labels. For a token tn in a tweet, the possibility of its label equals to y1 is as follows:
Pr (y1 | tq) = expM
( v1 ∣∣∣ P{pi}i )
∑k j=1 exp ( vyj ∣∣∣ P{pi}i ) (1)
P {pi}i means pi in template P were replaced with
[mask]. M ( v1 ∣∣∣ P{pi}i )
represents the probability that a masked token predicted by the model is mapped to the label y1.

heading: 2.2 Task 5
text: In task 5, we first designed the prompt by several prompt engineering methods, then converted the original data into instructions by the designed template. These instructions were fed into LoRa adapters to finetune the LLaMA-2 model costeffectively. Low-Rank Adaptation (LoRA) (Hu et al., 2021) aims to improve the efficiency of fine-tuning large language models by training much smaller lowrank decomposition matrices of certain weights. Consider a weight matrix W0 ∈ Rd×k from the pre-trained model. During training, the original weight matrix W0 remains frozen and does not receive gradient updates. The trainable parameters are the matrices B ∈ Rd×r and A ∈ Rr×k (r ≪ min(d, k)), which represent the low-rank decomposition. The forward pass with LoRA is as follows:
W0 x+∆Wx = W0 x+BAx (2)
Instruction Tuning. The importance of prompt templates has been demonstrated in various information extraction studies (Lu et al., 2021; Bao et al., 2022, 2023), particularly in the context of LLM. Task 5 presents a challenge due to its complex requirements. For example, Task 5 involves four types of disorders and requires that the report be from parents regarding their child. If we provide all the requirements in a single pass, LLMs may not

heading: If the tweets do not meet the two requirements above, the tweet is just merely mention another disorder or the patient is not their child.
text: accurately capture the semantic information. Based on the analysis above, we propose a decomposed prompt approach. As illustrated in Figure 1, we break down the task into independent units representing each requirement and fine-tune the LLM to address them individually. The first requirement guides the LLM to focus on determining the presence of the four specific diseases, while the second requirement ensures that the patients reported are children and that the reports come from their parents.
As shown in the penultimate paragraph of the prompt in Figure 1, the LLM is asked to make the final decision about whether the tweet fulfills the two requirements at the same time and if not, a No answer should be given. The tag <Input Text> will be replaced by the specific tweets before fed into the model. When it comes to the target sequence, they are proposed under a similar motivation to the prompt, intending to have the LLM aware of the reason for giving the choice. If the tweet cannot fulfill the requirements, then the second line of the target sequence should be given; otherwise, the first one.

heading: 3 Experiment Results

heading: 3.1 Task 1
text: In Task 1, tweets in training data were treated as continual pre-training data for the masked language modelling process. For both extraction and normalization tasks with PLMs, we run 30 epochs with a
batch size of 4. In Table 1, Norm task scores on the validation set were calculated based on the assumption that there is no error in the extraction task. For the extraction task, the basic BERT 1 classification method proposed in section 2.1 reached 0.51 in F score, while the cloze test method didn’t get the best performance because of the weaker generation ability of encoder-only models. Therefore, LLaMA-2 was introduced.
On the validation set, the F1 score of the extraction saw an around 0.15 rise with LLaMA-2, but the test result still saw a drastic fall for both models (the Task1 test score with LLaMA-2 was uploaded during post-evaluation).
For LLaMA-2 predictions, except for around 1.4% results are empty, 25.9% of the answer in the test set suffers from hallucination problems. This section adopts hallucination categories of "Intrinsic Hallucination" and "Extrinsic Hallucination" (Zhou et al., 2021). Most hallucinations in Task 1 are intrinsic, taking up about 77.2% of the total hallucination errors.
In Figure 2, we demonstrated three common types of intrinsic hallucinations: Case, Abbreviation, and Tense. Some of the intrinsic hallucination happened due to the colloquial nature of the tweet sentence. There are 22.8% extrinsic hallucinations generated without clear grounding in the input. As shown in Figure 2, the model generated "feel like im going to throw up" which is irrelevant to the original input tweet. Hallucination issues might cause
1Bert-base-uncased,https://huggingface.co/ google-bert/bert-base-uncased
severe consequences when facing health-sensitive data. We used heuristic rules to correct part of the intrinsic hallucinations in this task and got a 0.015 improvement on F1. Another solution could be Retrieval Augmented Generation (Chen et al., 2024) which we will implement in future work.

heading: 3.2 Task 5
text: For our LLM, we employ LLaMA-2-7B2 and LoRA fine-tune the adapter parameters. We tune the parameters of our models by grid searching on the validation dataset. We fine-tune the model with 20 epochs and save the model parameters for inference. The LoRA alpha is set to 128 and the LoRA rank is set to 64.
The model parameters are optimized by Adam (Kingma and Ba, 2015), and the learning rate of fine-tuning is 5e-5. The batch size is set to 4 with a cut-off length of 1024. The LoRA adapter would be merged with the original LLaMA-2-7B parameters and frozen during the inference process. During inference, we do the greedy search. Our experiments are carried out with an Nvidia RTX 4090 GPU. The model finally got an F1 score of 0.935 on the test set.
2LLaMA-2-7B-Chat,https://huggingface.co/ meta-LLaMA/LLaMA-2-7b-chat-hf

heading: 4 Conclusion
text: In conclusion, this paper presents PLM and LLMbased methods for Task 1 and Task 5 in SMM4H 2024. Our team balanced efficiency and efficacy by employing strategies like cloze test shifting, instruction tuning, and low-rank adaptation. Through empirical evaluations, this paper proved that the LLM did surpass certain PLMs in the two tasks, though it suffered from the hallucination issue. More methods need to be explored to ensure solid and reliable results from LLM in tasks of the public health domain.

heading: Acknowledgments
text: This research work is supported by a General Research Fund (GRF) project sponsored by the Research Grants Council, Hong Kong (Project No. 15611021).

heading: Future Work
text: , 2024) which we will implement in future work