TLab at #SMM4H 2024: Retrieval-Augmented Generation for ADE Extraction and Normalization (2024)

Abstract: SMM4H 2024 Task 1 is focused on the identification and standardization of Adverse Drug Events (ADEs) in tweets. We introduce a novel Retrieval-Augmented Generation (RAG) method, leveraging the capabilities of Llama 3, GPT-4, and the SFR-embedding-mistral model, along with few-shot prompting techniques, to map colloquial tweet language to MedDRA Preferred Terms (PTs) without relying on extensive training datasets. Our method achieved competitive performance, with an F1 score of 0.359 in the normalization task and 0.392 in the named entity recognition (NER) task. Notably, our model demonstrated robustness in identifying previously unseen MedDRA PTs (F1=0.363) greatly surpassing the median task score of 0.141 for such terms.

text: Proceedings of The 9th Social Media Mining for Health Applications (#SMM4H) Workshop and Shared Tasks, pages 153–157 August 15, 2024 ©2024 Association for Computational Linguistics
SMM4H 2024 Task 1 is focused on the identification and standardization of Adverse Drug Events (ADEs) in tweets. We introduce a novel Retrieval-Augmented Generation (RAG) method, leveraging the capabilities of Llama 3, GPT-4, and the SFR-embedding-mistral model, along with few-shot prompting techniques, to map colloquial tweet language to MedDRA Preferred Terms (PTs) without relying on extensive training datasets. Our method achieved competitive performance, with an F1 score of 0.359 in the normalization task and 0.392 in the named entity recognition (NER) task. Notably, our model demonstrated robustness in identifying previously unseen MedDRA PTs (F1=0.363) greatly surpassing the median task score of 0.141 for such terms.

heading: 1 Introduction
text: Social media is a potential wealth of information for contemporary public health monitoring, offering real-time insights into the effects of medications as reported by patients (Aichner et al., 2021). X (formerly Twitter), with its continuous flow of user-generated content, is a rich but challenging source for identifying Adverse Drug Events (ADEs), largely due to the informal and diverse language present in tweets (Klein et al., 2023).
The task of extracting standardized ADEs from such unstructured text is complex, as conventional methods typically depend on large training datasets. These not only require heavy computational resources, but also may fail to adapt to the nature of social media language (Yang et al., 2024). Here, we present a Retrieval-Augmented Generation (RAG) methodology to incorporate only relevant examples, side-stepping the need for large-scale data annotation and training processes (Gao et al., 2023).

heading: 2 Methods

heading: 2.1 ADE Tweet Classification
text: In the task of classifying tweets for the presence of ADEs, we used Google’s BERT-large-uncased model (Devlin et al., 2019). This model is a widely recognized transformer-based neural network pretrained on a large corpus of uncased English text. Our choice of BERT-large-uncased was motivated by its proven capability in text classification tasks and its relative efficiency, making it suitable for processing the high volume of data typically found on social media platforms (Huang et al., 2022; Sakhovskiy et al., 2021).
To adapt BERT to our specific classification task, we fine-tuned the model on the provided training dataset, which included annotated tweets with a binary label showing the presence of an ADE. We fine-tuned the model over 3 epochs, a number selected to balance training time and computational resources. Future work could involve a more systematic exploration of this parameter to potentially enhance model performance.

heading: 2.2 Named Entity Recognition (NER)
text: NER is necessary for identifying and extracting specific text spans from the tweets that have been classified as containing at least one ADE. Our approach involves retrieving similar examples from a vector database to enhance the model’s understanding and accuracy. We used the SFR-embeddingmistral model to create embeddings for the tweets in our training dataset. As of the Task 1 submission date, SFR-embedding-mistral leads the Massive Text Embedding Benchmark (MTEB) Leaderboard (Muennighoff et al., 2022). These embeddings capture the semantic similarities between tweets and are stored in a vector database. This storage allows for efficient retrieval of relevant examples.
When processing a new tweet, the same SFRembedding-mistral model projects the tweet into
153
the embedding space. The resulting embedding is then compared against the stored embeddings in the tweet vector database using cosine similarity, defined as
cosine similarity = ∑n
i=1 tji ∗ rki√∑n i=1 t 2 ji √∑n i=1 r 2 ki
for a tweet tj and reference tweet rk, with an embedding dimension of n. The top 10 most similar tweets to tj are retrieved based on these similarity scores. Known as few-shot prompting (LoganIV et al., 2021), these similar examples provide contextual references and demonstrate the correct response format.
We pass this added information to Meta’s Llama 3 7b, and ask it to extract the ADEs from tj . The prompts fed to the model are shown in Table 1. We selected Llama 3 through consideration of the cost barrier when using GPT-4 with large quantities of tokens.

heading: 2.3 Text Normalization
text: Text normalization is the process of transforming text into a standardized and structured form (Aliero et al., 2023). Here, we map the extracted ADE terms to standardized MedDRA PTs. We used a RAG framework to compare the identified terms to MedDRA PTs and select the most appropriate standardized term.
To normalize an extracted ADE term, we generated its embedding using the SFR-embeddingmistral model and calculated cosine similarity scores (1) between this embedding and the stored embeddings of MedDRA PTs. We selected the top 10 most semantically similar MedDRA PTs based on these similarity scores. These similar terms provide a set of potential matches for the extracted ADE term.
With the retrieved MedDRA PTs, we used GPT4(version=gpt-4-0125-preview, temperature=0, max new tokens=256), to process the full tweet along with the potential matches. Here, since we are retrieving individual terms rather than phrases, the cost barrier is less drastic than in the NER step. The prompts for this task are shown in Table 2. This strategy allows GPT-4 to consider both the context of the tweet and the extracted term when selecting the most appropriate MedDRA PT. By leveraging the semantic context provided by the full tweet, GPT-4 can map the colloquial language of the tweet to the standardized medical terminology of MedDRA.
We evaluated our approach using standard metrics of precision (P), recall (R), and F1 score across three different tasks: named entity recognition (NER), normalization (Norm), and normalization on unseen data (Norm-Unseen). In Table 3, we compare our results with the mean and median scores of the task to demonstrate the effectiveness of our approach. Additionally, we include our results on the provided validation (Val) dataset.

heading: 3.1 Performance on Named Entity Recognition (NER) Task
text: For the NER task, our model was responsible for identifying and extracting specific ADE terms from tweets. The F1-NER score for our model was 0.392, compared to the task mean of 0.327 and median of 0.376. Our precision in the NER task was 0.437 while the recall was 0.355.

heading: 3.2 Performance on Normalization Task
text: In the normalization task, our approach mapped colloquial tweet language to standardized MedDRA terminology. Our method achieved an F1-Norm score of 0.359, which compares to the task mean of 0.283 and median of 0.293. The precision and recall scores of our approach were 0.400 and 0.326, respectively.

heading: 3.3 Performance on Normalization Task for Unseen Data
text: The ability to generalize to unseen data is important for the practical application of any model. In evaluating our approach on MedDRA PTs not seen in the training data, we achieved an F1-NormUnseen score of 0.363, compared to the task mean of 0.209 and median of 0.209. This demonstrates our model’s robustness and ability to effectively handle the challenge of normalizing terms that were not present in the training data. The precision and recall for unseen data were 0.360 and 0.365, respectively.

heading: 4 Discussion
text: The results of our study highlight the effectiveness of our RAG approach in the context of the SMM4H 2024 Task 1. Our approach, which integrates the capabilities of Llama 3, GPT-4, and the SFR-embedding-mistral model, has demonstrated a particularly strong performance in handling unseen MedDRA Preferred Terms, an important as-
pect of real-world applicability given the evolving nature of language used in social media. This capability can enhance the timeliness and reliability of ADE detection from social media sources, contributing to faster and more informed public health responses.
For future use, there are certain areas of our pipeline that would benefit from experimental validation. For example, we arbitrarily selected the number 10 for both the ADE extraction and text normalization to provide our models with an adequate amount of information without overloading them. In addition to refining our pipeline, future work may aim to explore additional few-shot learning techniques and retrieval methods to enhance the model’s ability to adapt to the continuously evolving language on social media. If willing to trade cost for a potential performance boost, it would be worthwhile to assess GPT-4 for the NER task. Or, with a large quantity of training data, the system may perform better with a fine-tuned local LLM (such as Llama 3) for both the NER and normalization steps.

heading: Future Work
text: For future use, there are certain areas of our pipeline that would benefit from experimental validation. For example, we arbitrarily selected the number 10 for both the ADE extraction and text normalization to provide our models with an adequate amount of information without overloading them. In addition to refining our pipeline, future work may aim to explore additional few-shot learning techniques and retrieval methods to enhance the model’s ability to adapt to the continuously evolving language on social media. If willing to trade cost for a potential performance boost, it would be worthwhile to assess GPT-4 for the NER task. Or, with a large quantity of training data, the system may perform better with a fine-tuned local LLM (such as Llama 3) for both the NER and normalization steps