CogAI@SMM4H 2024: Leveraging BERT-based Ensemble Models for Classifying Tweets on Developmental Disorders (2024)

Abstract: This paper presents our work for the Task 5 of the Social Media Mining for Health Applications 2024 Shared Task Binary classification of English tweets reporting children’s medical disorders. In this paper, we present and compare multiple approaches for automatically classifying tweets from parents based on whether they mention having a child with attentiondeficit/hyperactivity disorder (ADHD), autism spectrum disorders (ASD), delayed speech, or asthma. We use ensemble of various BERTbased models trained on provided dataset that yields an F1 score of 0.901 on the test data.

heading: 1 Introduction
text: The prevalence of child developmental disorders, including attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorders (ASD), delayed speech, and asthma, presents significant challenges for families and healthcare systems worldwide. Understanding the experiences and needs of parents raising children with these conditions is crucial for informing support interventions and public health policies. In response to this challenge, social media data can be leveraged (Kim et al., 2020) after classifying tweets from parents mentioning if their child has one of these developmental disorders. This task holds significant promise for advancing our understanding of the prevalence and correlates of these conditions, particularly in relation to pregnancy exposures. Some studies have been done previously to assess the relationship between prenatal exposure to various substances during pregnancy to the risk of developing these disorders but many had low statistical power (Castro et al., 2016; Linnet et al., 2003). In this paper, we present a comprehensive analysis of multiple approaches for classifying tweets related to parental experiences with ADHD, ASD, delayed speech, and asthma. Our study focuses on a binary classification task, aiming to discern whether
tweets mention these developmental disorders or not. Through this work, we aim to contribute to the advancement of epidemiologic research (Cortese and Coghill, 2018; Rowland et al., 2002) and provide valuable insights into parental experiences with developmental disorders.
In this paper, we provide a detailed explanation of our system architecture, describe our experiments and share results obtained during this task.

heading: 2 Methods

heading: 2.1 Dataset
text: The dataset provided was divided into three partitions: training, validation, and test sets consisting of 7398, 389, and 1947 tweets respectively. Tweets that describe a child with ADHD, ASD, delayed speech, or asthma are annotated 1, and those that simply mention a disorder are annotated 0.

heading: 2.2 Pre-processing
text: In the preprocessing of the data, two steps were undertaken to ensure the quality & balance of the dataset.

heading: 2.2.1 Data Cleaning
text: Since the twitter data contains large amounts of noise, we applied several data cleaning procedures. This included the removal of URLs, mentions (ex: @USER), and hashtags (ex: #funny). Additionally, handling of emojis was crucial. Emojis were replaced with their corresponding textual representations to remove potentially distracting elements and simplify the text for further analysis.
114

heading: 2.2.2 Data Augmentation
text: The technique used for data augmentation was "gender-based" text augmentation. Within the texts labelled as 1, all instances of the term "son" (and corresponding pronouns "he," "his," and "him") were systematically substituted with "daughter" (and corresponding pronouns "she" and "her").The distribution of true labels increased from 30.82% before augmentation to 42.37%. after augmentation improving the class imbalance situation.
This augmentation technique was straightforward to implement and has shown to improve performance by including additional context.

heading: 3 Experiments and Results
text: During our model exploration, we fine-tuned multiple pre-trained models including MentalBERT (Ji et al., 2022), PsychBERT (Vajre et al., 2021), TwitterBERT (Zhang et al., 2022), and DistilBERT (Sanh et al., 2019). We trained each of them for 200 epochs with a learning rate of 1e-5 and a batch size of 32. Each of these models offers unique advantages that could be beneficial for our purpose.
MentalBERT and PsychBERT are tailored for mental health and biomedical text datasets which could potentially capture features relevant to our task of identifying tweets related to "developmental disorders". TwitterBERT’s training on Twitter data makes it adept at handling the informal language of tweets which can improve classification accuracy
while DistilBERT’s computational efficiency can make it suitable for scalable deployment, an advantage in handling large volumes of Twitter data.
We systematically evaluated the performance of each of these models by adopting an incremental approach. Initially, we trained the models using the raw data without any pre-processing (M1). Subsequently, we performed training after augmenting the data (M2). Finally, we conducted training using cleaned data with data augmentation (M3). This comprehensive approach allowed us to assess the impact of classification performance of each model, providing valuable insights into their robustness and effectiveness for our specific task.
Our final model then was an ensemble of all these methods assigned weights according to their "F1-scores" on validation dataset. We hardcoded weights of 0.6 score to MentalBERT, 0.2 score to TwitterBERT, 0.1 score to each PsychBERT and DistilBERT. This was decided based on their individual performance as can be seen in Table 2.

heading: 4 Conclusion
text: In this task, we experimented with various BERT models and presented the results in Table 2. Table 3 presents our results along with the mean and median results obtained during the challenge, depicting a final F1-score of 0.901. In future, we would like to experiment with more ensemble modelling methods, use better data augmentation techniques and For greater scope of research, a potential direction could involve incorporating multimodal data sources, such as images or audio recordings, to enrich the understanding of parental experiences and enhance the accuracy of classification models.

heading: Future Work
text: In future, we would like to experiment with more ensemble modelling methods, use better data augmentation techniques and For greater scope of research, a potential direction could involve incorporating multimodal data sources, such as images or audio recordings, to enrich the understanding of parental experiences and enhance the accuracy of classification models