Helping LLMs "Know The Drill" in Low-Resource Settings A Study on Social Media Posts (2024)

Abstract: The amount of data to fine-tune LLMs plays a crucial role in the performance of these models in downstream tasks. Consequently, it is not straightforward to deploy these models in lowresource settings. In this work, we investigate two new multi-task learning data augmentation approaches for fine-tuning LLMs when little data is available: "In-domain Augmentation" of the training data and extracting "Drills" as smaller tasks from the target dataset. We evaluate the proposed approaches in three natural language processing settings in the context of SMM4H 2024 competition tasks: multi-class classification, entity recognition, and information extraction. The results show that both techniques improve the performance of the models in all three settings, suggesting a positive impact from the knowledge learned in multi-task training to perform the target task.

heading: 1 Introduction
text: Collecting an adequate amount of data to fine-tune Large Language Models (LLM) in low-resource settings is an expensive practice. To address the lack of enough data in such settings, data augmentation techniques have been commonly used in different natural language processing tasks (Wei and Zou, 2019; Feng et al., 2021) as a low-cost solution. Such techniques focus on generating new examples close to the original samples and data distribution, thus increasing the number of training samples.
More recently, data augmentation has been studied from the perspective of multi-task learning. In Multi-Task Learning Data Augmentation (MTLDA), the dataset of the target task is augmented with the data of other auxiliary tasks that can improve the model performance, despite having different characteristics (Sánchez-Cartagena et al., 2021). The model is then jointly optimized in a multi-task manner on the augmented data, resulting in more robust performance compared to traditional augmentation techniques (Wei et al., 2021).
In this work, we investigate two new MTL-DA techniques to fine-tune LLMs for low-resource tasks: a) In-domain Augmentation where we augment the original target dataset with the data (and tasks) collected from the same source (e.g. Reddit Social Forum); and b) Drills where we extract from the target task a set of smaller tasks and replicate the dataset to cover the corresponding samples for the extracted drills. We study the proposed techniques on three language processing tasks in the context of SMM4H 2024 challenge1: I) Task 3: four-class classification of Reddit social forum posts; II) Task 4: an entity recognition task on Reddit posts to identify two entity types (Ge et al., 2024); III) Task 6: information extraction from tweets and Reddit posts. A complete description of the tasks is presented in §A.
The results indicate the effectiveness of both augmentation techniques by achieving higher F1 scores on the validation sets, compared to target task fine-tuning (note that the competition test sets are not publicly available), as well as scoring two to five points higher than overall scores in SMM4H competition.

heading: 2 Approach

heading: 2.1 Proposed Techniques
text: We experiment with two new MTL-DA techniques to fine-tune LLMs for low-resource settings:
a) In-domain Augmentation We jointly finetune the model on an augmented dataset, consisting of the data of the target task and other tasks collected from the same source. Here, we leverage the fact that Reddit is the common data source across all tasks. We then optimize/evaluate the performance of the model on each target task.
b) Drills We extract smaller tasks for each tar1SMM4H 2024: "The 9th Social Media Mining for Health Research and Applications Workshop and Shared Tasks" (Xu et al.)
17
get task to augment the dataset. Regarding Task 3, we extracted two drills of "sentiment classification: Pos|Neg|Neut." and "post relevance classification: Y|N". As a result, the training set was replicated twice to include the additional training examples. Regarding Task 4, we extracted two binary classification drills as "includes social impact: Y|N" and "includes clinical: Y|N", as well as two sequence extraction drills as "social impact extraction" and "clinical impact extraction". As a result, the training set was replicated four times to include the additional training examples. Concerning Task 6, we extracted one binary classification drill, in addition to the main task, as "Source Platform Classification:Reddit|Twitter", and duplicated the training set.

heading: 2.2 Model
text: We experimented with Mistral (7B) (Jiang et al., 2023), a decoder-only LLM. We applied QLoRA (Dettmers et al., 2024) to fine-tune the model, due to the limited GPU memory. Details of our implementation are provided in §B.

heading: 3 Evaluation
text: Since the test sets of the competition tasks are not publicly available, we analyze the performance of the model and the impact of the MTL-DA techniques on the validation set. We consider the vanilla fine-tuning of the model for the target downstream task as the baseline in our analysis. Furthermore, we compare the performance of the model
with GPT-42 via In-Context Learning (ICL) in {0,5}-shot settings in each task as a low-cost alternative. We then present the scores obtained by the best-performing models and MTL-DA techniques in SMM4H competition, compared to the overall mean and median scores obtained by all participants.

heading: 3.1 Validation Set
text: The performance of Mistral with different combinations of the proposed augmentation techniques on the validation set is presented in Table 1. MTL-DA techniques result in considerable improvements in the model performance for Tasks 3 and 4. However, vanilla fine-tuning of the model achieves the highest score in Task 6, suggesting that the relatively bigger training data for Task 6 is adequate to train the model without additional augmentation techniques. In-domain Augmentation shows promising results in all tasks, suggesting the positive impact of joint multi-task learning. Introducing the Drills improved the model in Tasks 3 and 4 when combined with In-domain Augmentation. Regarding ICL, 5- shot learning achieves the highest recall score in all tasks. However, fine-tuned models manage to obtain higher precision and, accordingly, outperforming F1 scores.

heading: 3.2 Error Analysis
text: To gain better insight into the impact of MTL-DA techniques, we manually controlled the fraction of the validation set in which models with different
2GPT-4-Turbo (gpt-4-0125-preview)
MTL-DA techniques provided different predictions (disagreements). Regarding Task 3 we observed that the model with vanilla fine-tuning on the target task excels at correctly predicting the label for emotionally nuanced samples. Meanwhile, the augmentation techniques further improve the model’s ability in correctly predicting the label for neutral/unrelated samples. In Task 4, we noticed that the model with MTL-DA techniques performs better on samples that include keywords for medical side-effects/symptoms and their associated impacts such as "brain zaps", "difficult urinating" and "hair loss". Regarding Task 6, we observed that the model with augmentation techniques is more effective in identifying explicit age mentions. However, the model with vanilla fine-tuning on the target task shows more robustness in predicting cases lacking exact age mentions, which is frequent in discussions focused on age-related health concerns. More details are presented in §D.

heading: 3.3 SMM4H Test Sets
text: Given that the submissions for the competition are limited to 3, we applied Borda Count to determine the three top models in the validation set. The results of the best-performing models on each task, presented in Table 2, indicate the positive impact of augmentation techniques. Similar to the performance observed on the validation set, the model with In-domain augmentation and with Indomain augmentation + Drills achieved the highest performance in Tasks 3 and 4, respectively, each scoring approximately four points higher than the overall median of the competition. Interestingly, while vanilla fine-tuning achieved the highest performance on the validation set in Task 6, it is out-
performed on the test set by the model with Indomain augmentation + Drills by approximately four points. This result suggests that augmentation techniques can increase the model’s robustness to handle unseen samples.

heading: 4 Conclusion
text: We studied two new Multi-Task Learning Data Augmentation techniques to address the lack of adequate data to fine-tune LLMs in low-resource settings. We evaluated the proposed techniques in three tasks in the context of the SMM4H 2024 competition, and we observed a) the positive impact of augmentation techniques in improving model performance; and b) the potential of in-context learning as a low-cost surrogate to fine-tuning. Nevertheless, besides further ablation studies, there are a few unanswered questions: a) "What is the best set of drills given a task?"; b) "When should a dataset be considered in-domain?"; c) "Would the same task on different domains also contribute positively?"; d) "How does the introduced approach compare to existing data augmentation techniques?"; and e) "Does the replication of the dataset potentially lead to overfitting?". We aim to explore these questions in future work.

heading: Acknowledgement
text: We acknowledge the support of the MUR PNRR project FAIR - Future AI Research (PE00000013) funded by the NextGenerationEU.

heading: A SMM4H 2024 Tasks
text: We evaluate the proposed techniques on three tasks: Task 3: Multi-class classification of effects of outdoor spaces on social anxiety symptoms consisting of categorizing posts on the Reddit platform into four classes positive effect, negative effect, neutral, and unrelated. A total of 3000 annotated posts are provided into sets of 1800, 600, and 600 posts as training, validation, and test sets, respectively. Evaluation: To evaluate the model, we compute the macro-average F1-score over all 4 classes as done in the competition. Task 4: Extraction of the clinical and social impacts of nonmedical substance use from Reddit consisting of an entity recognition tasks on Reddit posts for two entity types clinical impact and social impact. A total of 1380 posts are provided into sets of 843, 259, and 278 posts as training, validation, and test sets respectively. Evaluations: The model is tasked to predict the relevant spans. We automatically post-process the predicted tag for each token in the predicted span for both social and clinical impact. The prediction is then evaluated as a traditional entity tagging task (Note that it is different than the evaluation set-up deployed by SMM4H competition organizers). Task 6: Self-reported exact age classification with cross-platform evaluation in English consists of extracting self-reported exact age from posts on X (Twitter) and Reddit. The training data consists of 8.8k tweets as well as 100k unannotated Reddit posts, while the validation set consists of 2.2k tweets and 1000 Reddit posts. The test data for this task includes 2.2k tweets and 14.4k Reddit posts. Evaluations: To evaluate the model, we compute the macro-average F1-score for the selfreport class (post contains exact age) as done in the Task 6 competition.
B Implementation Details
QLoRA (Dettmers et al., 2024) introduces additional adapter layers that are fine-tuned for the downstream task. We trained the model with two different GPUs on a cloud hosting platform: an A100 (80GB) and an A6000 (48GB). Considering our limitation of GPU power, we applied quantization to the LORA layers. We used 100 warm-up steps and, as suggested in QLora (Dettmers et al., 2024), we kept a ratio of
α = 2 × rank with rank=4,8,32 for 30 epochs. We applied LoRA adapters to every linear layer "q_proj", "k_proj", "v_proj", "o_proj","gate_proj", "up_proj", "down_proj", and "lm_head". We used a decaying learning rate that starts from 2e-4. We experimented with 2 parameters for the max sequence length, 796 and 1496. The best-performing max sequence length was 1496. This is probably due to the fact that Reddit posts’ length required a longer number of tokens. To reduce GPU memory occupation, we halved batch size from two to one and doubled gradient accumulation steps from 16 to 32. To keep the memory footprint small in QLoRA, we enabled model loading in 4 bits and the computation type in bfloat16. To keep the memory footprint of the optimizer small, we used Paged AdamW 8bit (a version of Adam Optimizer that leverages CUDA paging). Each entry is packed with multiple sequences, as in T5 (Raffel et al., 2020). The hyperparameter configuration selected is rank=32, alpha=64 LoRA dropout = 0.1.

heading: C Fine-Tuning Costs
text: Using the A100 GPU, the longest training run took 24 hours on the model that included task drills and tasks for all three SMM4H tasks, summing up to a cost of $76. At the end of the study, the total cost, including all experiments, was $1600. The cost per hour of A100 GPU (80GB) and the A6000 (48GB) was respectively $3.18 and $1.89.

heading: D Error analysis
text: D.1 Task 6
In Task 6, the Target model proved to be more effective in predicting the absence of the exact age in posts. We can observe this especially in discussions focused on health conditions broadly associated with age ranges ("cataracts with age...", "diagnosed in my late 20s...") rather than specific numerical ages. Conversely, the In-domain + drills model better recognized posts where age is explicitly stated ("I’m 28 going on 29...", "I’m 19..."). This indicates that the Target model tends to predict more effectively the lack of exact self-reported age, while the In-domain + drills model is more reliable when exact ages are directly provided within the text.
D.2 Task 3
The target task model performs better in recognizing the emotional nuances of outdoor experiences.
It accurately predicts posts expressing anxiety, reporting social difficulties ("Crushed by SA...") or highlighting positive achievements ("I GOT TO RIDE IN A TESLA...").
In contrast, the In-domain + Drills model demonstrates strength in identifying neutral activities involving outdoor spaces ("Went outside for the first time in awhile...") and posts where the outdoor setting is peripheral to the core discussion ("Social anxiety has ruined my diet..."). This suggests that the In-domain + Drills model is better at recognizing when the impact of outdoor spaces is primarily neutral or indirect.
D.3 Task 4 In Task 4, the in-domain model outperforms the target model in recognizing specialized medical terminology and its associated impacts. It accurately identifies clinically significant terms such as "brain zaps" and "hair loss" (potential side effects), as well as the severity of "cravings for heroin" (addiction). The in-domain model also demonstrates a better grasp of the nuanced social impacts related to health. Although the in-domain model’s higher recall leads to some incorrectly predicted social impacts, in general it achieves higher performance in the identification and classification of clinically relevant keywords.

heading: Future Work
text: We acknowledge the support of the MUR PNRR project FAIR - Future AI Research (PE00000013) funded by the NextGenerationEU