WADER at SemEval-2023 Task 9: A Weak-labelling framework for Data augmentation in tExt Regression Tasks
Year: 2023

Abstract:
Intimacy is an essential element of human relationships and language is a crucial means of conveying it. Textual intimacy analysis can reveal social norms in different contexts and serve as a benchmark for testing computational models’ ability to understand social information. In this paper, we propose a novel weaklabeling strategy for data augmentation in text regression tasks called WADER. WADER uses data augmentation to address the problems of data imbalance and data scarcity and provides a method for data augmentation in cross-lingual, zero-shot tasks. We benchmark the performance of State-of-the-Art pre-trained multilingual language models using WADER and analyze the use of sampling techniques to mitigate bias in data and optimally select augmentation candidates. Our results show that WADER outperforms the baseline model and provides a direction for mitigating data imbalance and scarcity in text regression tasks.

heading: 2 Background

heading: 2.1 Past Work
text: Data imbalance and scarcity are problems that are rampant in real world datasets. The high cost of obtaining large amounts of data, and expert annotations, a wealth of research has been done to support limited data settings. Data augmentation for text is broadly done in two ways, conditional data augmentation which involves data augmentation conditioned by the target label, and unconditional data augmentation which involves working with the corpus features only (Bayer et al., 2021; Liu et al., 2020). Conditional data augmentation is done usually by deep generative models and pretrained language models such as BART (Lewis et al., 2019), CBERT (Wu et al., 2018), GPT2 (Radford et al., 2019). Common ways to perform unconditional data augmentation are lexical substitution and back translation. (Wei and Zou, 2019) introduce several lexical techniques to augment textual data, including synonym replacement, random insertion, random swap and random deletion. However, these methods suffer from lack of sufficient diversity and often produce sentences that are not coherent. Back-translation especially has received widespread attention, because progress in machine translation has made back-translation an efficient way to generate diverse sentences in the dataset without compromise in coherence and semantic quality. Common translation tools used are seq2seq based models, NMT and transformers. Different techniques exist for text classification and NER tasks, but to the best of our knowledge our work is unique in the text regression domain.
Weak supervision of text labeling during data augmentation is an example of Semi-Supervised Learning (SSL) methods. The main idea of these methods is to regularize the learning process by training a network with the given data, using the network to label unlabelled data and finally use
both the true-labeled and weak-labeled data points to train the final model.

heading: 2.2 Task Description
text: SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis (Pei et al., 2022) is a task that deals with detecting intimacy in 10 languages. This task is co-organized by University of Michigan and Snap Inc. Intimacy is a fundamental aspect of human relationships, and studying intimacy in a textual context has many potential applications in the field of computational linguistics. The training data is available in 6 languages: English, Spanish, Italian, Portuguese, French, and Chinese. The evaluation is done on the given training languages, as well as 6 unseen languages: Hindi, Arabic, Dutch and Korean.
The metric of evaluation for the task is Pearsons R. Pearsons R, r is expressed as follows for two variables x and y:
r = ∑ (xi − x̄) (yi − ȳ)√∑
(xi − x̄)2 ∑ (yi − ȳ)2 (1)
The correlation coefficient r ranges from -1 to 1, with an absolute value of 1 indicating a perfect linear relationship between the two variables. In such a case, all data points lie on a line that can be represented by a linear equation. The sign of the correlation coefficient is determined by the regression slope, with a value of +1 indicating that all data points lie on a line where y increases as x increases, and a value of -1 indicating the opposite. A correlation coefficient of 0 implies that there is no linear dependency between the two variables.

heading: 2.3 Data Description
text: The dataset for the task is the MINT- Multilingual INTimacy analysis (Pei et al., 2022) dataset. The training set contains sentences in 6 languages: Chinese, English, French, Portuguese, Spanish and Italian. The dataset has 9491 tweets. Distribution of sentences in different languages is given in Table 1.
Intimacy is annotated using a 5-point Likert scale where 1 indicates Not intimate at all and 5 indicates Very intimate. (Cite) have described the annotation process in detail.
The dataset is highly imbalanced, with majority of the labels in each language belonging to the lower spectrum of the scale as seen in Fig1. Overall,
75% of the samples in the dataset have a label less than or equal to 2.667.
The testing set additionally contains 4 unseen languages, Hindi, Korean, Arabic and Dutch.

heading: 3 Methodology

heading: 3.1 Data Augmentation
text: As noted in section 2.3, the data is highly imbalanced for the given labels. Moreover, since the task has 4 unseen languages, there is an additional need for data augmentation. WADER performs data augmentation using the framework described in Fig 2. The steps followed are described as follows:
Distribution based Sampling: Since the distribution of labels is skewed, and not all labels need augmentation, we perform a distribution based sampling to select candidate tweets for data augmentation. We fix a threshold p, and sample all tweets above the given threshold. We take the value of p as 3.2, and less
Translation: Data is augmented through translation and back translation. The translation scheme
is described in Fig 3. For an unseen language, Lunseen, set of sampled sentences Li∀i ∈ L are taken and translated to the target language, Lunseen. The translated sentences are appended to the set Tunseen.
For a seen language, say Li, the language is translated to all other languages except the language itself, Lk∀k ∈ L, k 6= i. The translated tweets are appended to their respective translated sets Tk, and they are translated back to the source language Li, appended to the translated set Ti.
Our final translated set has 49774 sentences. Label Validation: We train a baseline model by finetuning a pretrained language model on the gold labelled data. This model is then used to infer on the concatenated translated corpus of seen and unseen languages.
Difference Based Sampling: We take the absolute difference between predicted and pre-assigned values (derived label from before a sentence was translated). We use this as a metric for quality of translations and pick appropriate thresholds to se-
lect sentences. Table 2 shows an analysis of the distribution of differences. The mean difference is 0.62, which is a below average translation quality since the resolution of labels is 0.1 in the dataset. However, 75% of sentences have differences ≤ 0.86 For our experiments, which means that coarse grain labels (differences of 1) are correctly assigned in most of the cases.
We define β as the parameter which represents the difference threshold. We pick difference values of β as 0.1, 0.2 and 0.3 in our experiments. Table 2 shows the count of sentences in each of these thresholds.

heading: 3.2 Finetuning Pre-trained Language Models
text: Finetuning pretrained language models has become a popular approach for natural language processing tasks in recent years. Transformer based (Vaswani et al., 2017) Pretrained Language Models such as BERT (Devlin et al., 2018), GPT-2(Radford et al., 2019), and RoBERTa(Liu et al., 2019) are trained on massive amounts of text data, which allows them to capture complex linguistic patterns and structures. Finetuning involves taking a pretrained language model and further training it on a specific downstream task, such as sentiment analysis or question answering. This approach has been shown to achieve state-of-the-art performance on a wide range of natural language processing tasks, with significantly less data and computation needed compared to training a model from scratch. Finetuning also allows for the transfer of knowledge learned from a large, diverse set of data to a smaller, more specific task, making it a powerful technique for natural language processing research.
The pre-training models used in our system include:
XLM RoBERTa: XLM-RoBERTa (Conneau et al., 2019) is a variation of the RoBERTa model that has been designed to handle multilingual natural language processing tasks. This model is pre-trained on a massive dataset of 2.5 terabytes of CommonCrawl data filtered for 100 different languages. By training on such a large and diverse dataset, XLM-RoBERTa is able to capture the linguistic nuances and patterns that are unique to different languages. The architecture of XLMRoBERTa is based on the highly successful BERT model, but with key modifications to hyperparameters such as larger mini-batches and learning rates, allowing it to handle the additional complexity of multilingual data. XLM-RoBERTa has shown impressive results across a range of multilingual natural language processing tasks, demonstrating the power of pre-training on large, diverse datasets for building highly effective models. We use implementation of the XLMR model from HuggingFace .
XLNET: XLNet (Yang et al., 2019) is a stateof-the-art natural language processing model that
{Li}  Lunseen Tunseen= {...}
{Li} = {Li | ∀ i ∈ languages in the corpus}
extends the Transformer-XL architecture and uses an innovative pre-training method. Unlike BERT, which corrupts input with masks and neglects dependencies between masked positions, XLNet is able to learn bidirectional contexts by maximizing the expected likelihood over all possible permutations of the factorization order. This allows XLNet to capture complex linguistic patterns and dependencies in the input sequence. XLNet also integrates ideas from Transformer-XL, which is currently the most advanced autoregressive model in use. With its autoregressive formulation, XLNet is able to overcome the limitations of BERT and achieve even better performance on a range of natural language processing tasks.
For finetuning the pre-trained models, we add a single linear layer on top of the embeddings of the classification token 
 for XLM-RoBERTa and [cls]. Since this is a text regression task and the scores are in a limited, we apply a clamp function as the final activation function which clamps the scores in the range [1, 5]. Fig 4 is a representation of our finetuning procedure.

heading: 3.3 Ensembling
text: We evaluate results on the test set using 6 models, XLM RoBERTa and XLNET trained on augmented sets with difference sampling parameter β = 0.1, 0.2, 0.3.
We choose 6 ensembles. The configurations of ensembles are defined in Table 4.
Ensembling is done by taking the mean prediction of all the ensembled models.

heading: 4 Experimental Setup
text: We use the original test and train set. Further, we take 15% of the train set, sampled randomly from each language as our validation set.
We build our models using open source available implementations of the XLM-RoBERTA and XLNET available on HuggingFace. We use xlnet-base-cased xlm-roberta-base2 and 3. We use Adam (Kingma and Ba, 2014) as our optimiser. The size of the the embeddings are D ∈ 768 and the size of the linear layer is D/2×1. The batch size is taken as 8 and the learning rate is 4e-5. We train the models for 2 epochs. Experiments are performed on Google Colab cloud GPU. Google Translate API has been used to perform translations. These hyperparameters are common for all system settings including our two baselines: 1) XLM RoBERTa finetuned on only Gold data, 2) XLNET finetuned on only Gold data.
Final submission is reported on Ensemble 6, configured as per the desciption in Section 3.3.

heading: 5 Results and Discussion
text: Table 5 represents the scores achieved by our system in different experimental settings. The final submission for the competition is denoted by Ensemble 6. Table 6 shows our rank under different categories of the shared task.
As we can observe from Table 5, WADER seems to improve on existing transformer baselines for all
2https://huggingface.co/ xlm-roberta-base
3https://huggingface.co/ xlnet-base-cased
categories except one where it ties with an ensemble.

heading: 5.1 Comparison of Pre-trained Language Models:
text: We observe a general trend that XLM RoBERTa performs better than XLNET on multilingual baselines in our experiments. This can be demonstrated by the fact that the XLNET Baseline outperforms XLM RoBERTa only on English. For all other languages, there is a significant margin in between performance of XLNET and XLM RoBERTa. For Hindi, and Korean which have non latin characters, performance of XLNET is even worse with a negative R coefficient. which has This demonstrated the importance of multilingual pretraining.

heading: 5.2 Comparison of Difference Sampling Threshold β:
text: While lower values of β(= 0.1) give more accurate labelled sets, we observe that moderate values of β(= 0.1, 0.2) outperform them. This is because, moderate values of β allow for larger sized training corpuses, which would positively effect the performance of the models. Moreover, moderate values β include more number of low quality translations, due to a higher difference. We hypothesize that this would have a regularising effect by providing the model with diversity in the training set, and preventing it from overfitting on the training corpus.

heading: 5.3 Discussion on Performance
text: We rank 32 overall, 34 on seen languages and 29 on unseen languages. The lower performance of our model can be understood by the following factors:
• Translation Quality: The quality of translation is a key driver in WADER’s performance. Lower quality translations would produce augmentations with noisy and unreliable labels. Translation quality is often dependant on the pair of languages in question. For languages such with a non latin script such as Hindi, translations are often of a lower quality which is also reflected in the results.
• Overfitting: By translating the data, while we increase linguistic diversity, most sentences would still be semantically similar, causing the model to overfit. This can further be seen by the fact that settings like 0.2, 0.3-XLM RoBERTA (where we can expect higher diversity from gold sentences due to higher differences) give the best performance for a lot of languages. Similarly, Ensemble 1 which preserves data quality while also reaping the regularising benefit of ensembling performs quite well in the given setting. Another indication of overfitting is the the better rank of our model on unseen languages.
• Word Sensitivity: For a task like Intimacy Detection, specific vocabulary used is key to identify the intimacy level. Translations can lead to replacement of words which do not hold the same degree of influence in accounting for textual intimacy.

heading: 6 Conclusion and Future Work
text: This paper proposes a novel data augmentation framework, WADER, for text regression tasks that use weak-labeling strategies to solve the problems of data imbalance and data scarcity. We also provide a method for data augmentation in cross-lingual, zero-shot tasks. Our approach uses sampling techniques to mitigate bias in data and optimally select augmentation candidates. We benchmarked the performance of State-of-the-Art pre-trained multilingual language models XLM RoBERTa using WADER and achieved promising results. Our findings demonstrate the importance of data augmentation for mitigating data imbalance and scarcity in text regression tasks. This study’s contributions provide a direction for future research in the field of computational linguistic and its applications to social information analysis.

heading: Future Work
text: This study’s contributions provide a direction for future research in the field of computational linguistic and its applications to social information analysis