LT4SG@SMM4H’24: Tweets Classification for Digital Epidemiology of Childhood Health Outcomes Using Pre-Trained Language Models (2024)

Abstract: This paper presents our approaches for the SMM4H’24 Shared Task 5 on the binary classification of English tweets reporting children’s medical disorders. Our first approach involves fine-tuning a single RoBERTa-large model, while the second approach entails ensembling the results of three fine-tuned BERTweet-large models. We demonstrate that although both approaches exhibit identical performance on validation data, the BERTweet-large ensemble excels on test data. Our best-performing system achieves an F1-score of 0.938 on test data, outperforming the benchmark classifier by 1.18%. Our code is available on Github1.

heading: 1 Introduction & Motivation
text: Chronic childhood disorders like attentiondeficit/hyperactivity disorder (ADHD), autism spectrum disorders (ASD), delayed speech, and asthma significantly impact a child’s development and well-being, often extending into adulthood. Approximately 1 in 6 (17%) children aged 3-17 years in the United States experience a developmental disability, with ADHD, ASD, and others contributing to this statistic (Zablotsky et al., 2019). In previous studies (Guntuku et al., 2019; Hswen et al., 2019; Edo-Osagie et al., 2019), Twitter data have been utilized to identify self-reports of the aforementioned disorders; however, the identification of reports concerning these disorders in users’ children has not been explored. It may be of interest to explore Twitter’s potential in continuing to collect users’ tweets postpartum, enabling the detection of outcomes in childhood.

heading: 2 Task and Data Description

heading: 2.1 Task
text: The SMM4H-2024 workshop and shared tasks have a special focus on Large Language Models
1To access the code, please visit: [GitHub Source Code]
(LLMs) and generalizability for natural language processing (NLP) in social media. We participated in Task 5, which is 'Binary classification of English tweets reporting children’s medical disorders'. The objective is to automatically differentiate tweets from users who have disclosed their pregnancy on Twitter and mention having a child with ADHD, ASD, delayed speech, or asthma (annotated as "1"), from tweets that merely refer to a disorder (annotated as "0").

heading: 2.2 Data
text: There were three different datasets provided: training, validation, and test datasets. The training and validation datasets were labeled while the test dataset was not. All datasets are composed entirely of tweets posted by users who had reported their pregnancy on Twitter, that report having a child with a disorder and tweets that merely mention a disorder. The training, validation, and test sets contain 7398 tweets, 389 tweets, and 1947 tweets, respectively.

heading: 3 Methodology

heading: 3.1 Baseline
text: A benchmark classifier, based on a RoBERTa-large model (Liu et al., 2019), has achieved an F1-score of 0.927 for the 'positive' class (i.e., tweets that report having a child with a disorder) on the test data for Task 5 (Klein et al., 2024).

heading: 3.2 Models Used
text: We investigated three Transformer based models which are BioLinkBERT-large (Yasunaga et al., 2022), RoBERTa-large and BERTweet-large (Nguyen et al., 2020). BioLinkBERT was selected for its specialized understanding of biomedical NLP tasks, RoBERTa for its domain-independent NLP capabilities, and BERTweet for its superior performance in Tweet-specific NLP tasks. We fine-
38
tuned each model with the training dataset and evaluated its performance using the validation dataset.

heading: 3.3 Training Regime
text: Experiments were conducted using Google Colab Pro+ equipped with an NVIDIA A100 Tensor Core GPU boasting 40 gigabytes of available GPU RAM. The Hugging Face Transformers Python library (Wolf et al., 2019) and its Trainer API facilitated training procedures. Each model was trained on the training datasets for 3 iterations and 10 epochs per iteration. We used HuggingFace’s Trainer Class’s default 'AdamW' and 'linear warmup with cosine decay' as the optimizer and scheduler respectively. The maximum sequence length for all models was set to 512. FP-16 mixed precision training was employed to enable larger batch sizes and expedited training. Primary hyperparameters including learning rate, weight decay, and batch size were determined as described in the subsequent section.

heading: 3.4 Hyperparameter Optimization
text: We conducted hyperparameter optimization that relied on HuggingFace’s Trainer API with the Ray Tune backend (Liaw et al., 2018). We utilized Ray Tune’s built-in "BasicVariantGenerator" algorithm2 for hyperparameter search, paired with the FirstIn-First-Out (FIFO) scheduler. Since BasicVariantGenerator has the ability to dynamically generate hyperparameter configurations based on predefined search algorithms (e.g., random search, Bayesian optimization), it enables more efficient exploration of the search space. The hyperparameters optimized using BasicVariantGenerator are presented in Table 1.

heading: 4 Preliminary Experiments
text: Each selected model was trained for 3 iterations, with 10 epochs per iteration. At the end of each epoch, its F1-score was recorded. The F1-score for each model was determined based on its performance with the validation dataset. We saved the
2https://docs.ray.io/en/latest/tune/ api/doc/ray.tune.search.basic_variant. BasicVariantGenerator.html
best-performing epoch (i.e., the best F1-score for the positive class) for each model in each iteration. The results are shown in Table 2.
As shown in Table 2, RoBERTa-large and BERTweet-large perform similarly on the validation dataset, and considerably better than BioLinkBERT-large, even though it has been pretrained on a large corpus of biomedical data. Therefore, we decided to remove BioLinkBERT-large to carry out further experiments for this task (Guo et al., 2020).

heading: 4.1 Ensembling Strategy
text: The issue arises when fine-tuning large-transformer models on small datasets: the classification performance varies significantly with slightly different training data and random seed values, even when using the same hyperparameter values (Dodge et al., 2020). To overcome this high variance and provide more robust predictions, we propose ensembles of multiple fine-tuned RoBERTa-large models and BERTweet-large models separately. We create two separate ensemble models using the best models corresponding to three iterations for each RoBERTa-large and BERTweet-large. All three iterations use the same hyperparameters, and only differ in the initial random seed. A hard majority voting mechanism combines the predictions of these models:
ŷ = argmax c
n∑
i=1
1(ŷi = c) (1)
where 1(·) represents the indicator function, which returns either '1' or '0' for the class label c predicted by the i-th classifier.
As shown in Table 3, the BERTweet-large ensemble performs better than the RoBERTa-large
ensemble. This is fundamentally because its performance variation is less for three iterations, as indicated in Table 2. Another noteworthy observation is that the performance of the BERTweet-large ensemble is identical to that of the best iteration (Table 2, 2nd run) of RoBERTa-large as shown in Table 4. Figure 1 shows the corresponding confusion matrices for both classifiers which are also identical.

heading: 5 Results and Conclusion
text: Since RoBERTa-large best-run and BERTweetlarge ensemble are performing equally well on the validation data, we tested the performance of both classifiers on unseen, unlabeled test data. As shown in Table 5, the BERTweet-large ensemble classifier outperforms the mean and median performance on the test data among all teams’ submissions by a considerable margin, as well as the benchmark classifier by 1.18%. Additionally, we can observe that even though both classifiers perform equally well on validation data, the BERTweet-large ensemble model performs significantly better on test data. One possible reason for this is that different runs of BERTweet-large might excel at capturing different aspects of the data or learning different patterns.
Previously, authors (Klein et al., 2024) have achieved an F1-score of 0.92 using a BERTweetlarge classifier on the same dataset. We believe that our approach achieved better results for several fundamental reasons. First, we performed more thorough hyperparameter tuning compared to the previ-
ous authors. Better-optimized hyperparameters can significantly improve model performance. Second, by creating an ensemble of BERTweet-large models from the three best epochs of three runs, we captured more robust and generalized patterns in the data. Ensemble methods typically outperform individual models because they reduce variance and mitigate the risk of overfitting to specific subsets of data. Third, a likely reason is the diversity in model runs. Each run of our BERTweet-large model may have encountered slightly different initialization and training dynamics (e.g., random seed), resulting in diverse decision boundaries. Combining these diverse models helps in making more accurate predictions by leveraging the strengths of each individual model.
When fine-tuning complex pre-trained language models, one issue on small datasets is the instability of the classification performance. To overcome this, we combined the predictions of multiple BERTweet-large models in an ensemble. By doing so, we achieved significantly better results in terms of F1-score for SMM4H’24 Task 5. For future work, it’s interesting to investigate how the system performance varies when adding more BERTweetlarge iterations (i.e., runs) to the ensemble.

heading: Future Work
text: For future work, it’s interesting to investigate how the system performance varies when adding more BERTweetlarge iterations (i. e. , runs) to the ensemble