ADE Oracle at #SMM4H 2024: A Two-Stage NLP System for Extracting and Normalizing Adverse Drug Events from Tweets (2024)

Abstract: This study describes the approach of Team ADE Oracle for Task 1 of the Social Media Mining for Health Applications (#SMM4H) 2024 shared task. Task 1 challenges participants to detect adverse drug events (ADEs) within English tweets and normalize these mentions against the Medical Dictionary for Regulatory Activities standards. Our approach utilized a two-stage NLP pipeline consisting of a named entity recognition model, retrained to recognize ADEs, followed by vector similarity assessment with a RoBERTa-based model. Despite achieving a relatively high recall of 37.4% in the extraction of ADEs, indicative of effective identification of potential ADEs, our model encountered challenges with precision. We found marked discrepancies between recall and precision between the test set and our validation set, which underscores the need for further efforts to prevent overfitting and enhance the model’s generalization capabilities for practical applications.

heading: 1 Introduction
text: This paper outlines Team ADE Oracle’s participation in the 9th Social Media Mining for Health Research and Applications (#SMM4H) 2024 workshop’s Task 1 (Xu et al., 2024), which involved extracting and normalizing adverse drug events (ADEs) from tweets into Medical Dictionary for Regulatory Activities (MedDRA) terms1. The task complexity increased in 2024 by combining ADE detection with normalization, a challenge heightened by the informal and diverse language used on social media (Xu et al., 2024). Addressing ADEs through social media enhances pharmacovigilance, providing critical data for public health interventions (Huynh et al., 2016; Alimova and Tutubalina, 2019; Vydiswaran et al., 2019; Magge et al., 2021; Liu et al., 2022; Lee et al., 2023). Our approach
1https://www.meddra.org
employed a spaCy-based NLP pipeline, retraining a Named Entity Recognition (NER) module to extract ADEs, and a RoBERTa model for aligning text with MedDRA standards (Weissenbacher et al., 2022), navigating the trade-offs between recall and precision. While our system effectively identified many ADEs, the prevalence of false positives points to a need for further refinement to enhance the accuracy and utility of our methods for public health surveillance.

heading: 2 Dataset
text: This study employed the #SMM4H 2024 Task 1 dataset, comprising 30,949 tweets distributed across 18,185 training, 965 validation, and 11,799 test tweets (Klein et al., 2024; Xu et al., 2024).

heading: 3 System Description
text: Our methodology for the #SMM4H 2024 Task 1 involves a two-stage process: We use an NER package to extract ADEs, followed by the normalization of these entities against the MedDRA using vector similarity techniques (Yazdani et al., 2023a,b).

heading: 3.1 Preprocessing
text: For preprocessing the dataset, we implemented a two-step approach to optimize data for training. Initially, all labeled entities representing ADEs were converted to lowercase to ensure consistency and address case discrepancies between labels and their occurrences in the tweet text. Subsequently, we employed the tokenizer2 from the blank, spaCy "en" model3 to tokenize the text (Dai et al., 2017).

heading: 3.2 NER for ADE Extraction
text: We chose to use the blank spaCy model "en" for training a customized NER model tailored
2https://spacy.io/api/tokenizer 3https://spacy.io/usage/models
117
to the extraction of ADE entities due to its robust handling of English syntax and adaptability to the specialized domain of pharmacovigilance (Dai et al., 2017; Jiang et al., 2022). Specifically, we trained the model’s span categorizer4 component to identify and label ADE spans within tweets effectively. The span categorizer comprises two main components: a suggester function and a labeler model. The suggester function, employing the spacy.ngram_suggester.v1, was selected to propose candidate spans with designated lengths—specifically one to five tokens. These candidates, which may overlap, are presented in a ragged array format comprising two columns that denote the start and end positions of each span. Subsequently, the labeler model evaluates each candidate span, assigning the ADE label as appropriate based on the predictive outcomes.
This model was trained on the 18,185 labeled tweets of the official training set. Optimization was achieved over 49 epochs with a batch size of 8 and a dropout rate of 0.5, selecting the best-performing iteration for our analyses.

heading: 3.3 Vectorization and Normalization
text: For the normalization stage, we employed the base model RoBERTa to vectorize the ADE entities and MedDRA entries (Liu et al., 2019; Gencoglu, 2020; Weissenbacher et al., 2022). We did not further finetune the base RoBERTa model, as our focus was solely on utilizing its semantic representation capabilities. We extracted and vectorized ADE entities from the validation set using our span categorizer, and vectorized the textual descriptions of MedDRA adverse event terms. The MedDRA vectors were stored in a vector database from Facebook’s Faiss library, which is designed for efficient similarity searching of dense vectors at scale (Johnson et al., 2019; Douze et al., 2024). We then iterated through our extracted entities and used Euclidean distance (L2 distance) to identify the closest match between each ADE entity vector and the MedDRA term vectors in the database.

heading: 3.4 Evaluation Metrics
text: The performance of our NER model in identifying ADEs, along with the pipeline’s effectiveness in matching ADEs to MedDRA terms, was evaluated using the official metrics of #SMM4H 2024 Task 1, specifically F1, precision, and recall.
4https://spacy.io/api/spancategorizer

heading: 4 Results
text: Our system consisting of the NER model for ADE extraction and RoBERTa for the normalization task is evaluated in Tables 1 and 2 on the validation set and the official test set, respectively.

heading: 4.1 ADE Extraction
text: Table 2 shows that the ADE extraction model achieved an F1 of 13.2 on the test set, with precision and recall scores of 8.0% and 37.4%, respectively. These results highlight the model’s higher success in recall, indicating its effectiveness in identifying ADE mentions. However, the model’s low precision of 8.0% highlights a significant challenge in specificity. The model’s unexpectedly high recall on the test set compared to the validation set, where recall and precision were more balanced, indicates differences in the distribution and complexity of the validation and test set.

heading: 4.2 ADE Normalization
text: For ADE normalization, the official scores in Table 2 show a precision of 5.0%, a recall of 23.7%, and an F1 of 8.2. Additionally, when evaluating the model’s performance on previously unseen MedDRA IDs, it returned significantly lower metrics (precision: 0.7%, recall: 10.0%, F1: 1.4). This considerable drop suggests challenges in generalizing to new, unseen ADE terms, reflecting potential limitations in the model’s generalizing capability. The results on the validation set were somewhat consistent, with an F1-score of 8.4 and slightly lower precision and recall of 7.5% and 9.4%, respectively.

heading: 4.3 Discussion
text: Our results point to the challenges inherent in biomedical NLP tasks, especially in balancing pre-
cision and recall and generalizing to new data. The low precision observed shows issues with generalizability beyond training data in NER. The results may also reflect the complexities of social media language, complicating ADE detection and normalization.
Moreover, the differences in results between validation and official test data underline the importance of robust cross-validation strategies to mimic real-world performance and prevent overfitting. Further efforts need to focus on integrating domain-specific knowledge bases to heighten normalization accuracy and better manage new ADE identifiers.

heading: 5 Conclusion
text: Our contribution to #SMM4H 2024 Task 1 consists of an NER model retrained to identify ADEs and a similarity-based RoBERTa model to normalize them. The findings from our system underline the challenges and opportunities presented by the use of NLP in detecting and normalizing ADEs from social media. Despite achieving high recall, our model’s low precision highlights a significant challenge in accurately identifying relevant ADEs amid the informal language prevalent on platforms like Twitter. Furthermore, the task has demonstrated that while our current methodology is capable of initial identification, it falls short in scenarios involving generalizing to data different from the training data, which is crucial for practical applications.
For future work, we will investigate enhancing model precision through advanced linguistic analysis, employing models pre-tuned on ADE datasets, fine-tuning RoBERTa for vectorization of ADE entities and MedDRA entries, and incorporating additional ADE data.

heading: Future Work
text: For future work, we will investigate enhancing model precision through advanced linguistic analysis, employing models pre-tuned on ADE datasets, fine-tuning RoBERTa for vectorization of ADE entities and MedDRA entries, and incorporating additional ADE data