Data Augmentation through Back-Translation for Stereotypes and Irony Detection
Authors: Tom Bourgeade, Silvia Casola, Adel Mahmoud Wizani, Cristina Bosco (LORIA University of Lorraine, Ludwig-Maximilians-University Munich, Università di Torino, 2024)
Topic: Sentiment, Emotion, Irony, Hate

Abstract
Complex linguistic phenomena such as stereotypes or irony are still challenging to detect, particularly due to the lower availability of annotated data. In this paper, we explore Back-Translation (BT) as a data augmentation method to enhance such datasets by artificially introducing semantics-preserving variations. We investigate French and Italian as source languages on two multilingual datasets annotated for the presence of stereotypes or irony and evaluate French/Italian, English, and Arabic as pivot languages for the BT process. We also investigate cross-translation, i.e., augmenting one language subset of a multilingual dataset with translated instances from the other languages. We conduct an intrinsic evaluation of the quality of back-translated instances, identifying linguistic or translation model-specific errors that may occur with BT. We also perform an extrinsic evaluation of different data augmentation configurations to train a multilingual Transformer-based classifier for stereotype or irony detection on mono-lingual data. Warning: This paper may contain potentially offensive example messages.

Introduction
Equipping systems with linguistics-grounded capabilities can be complex. Despite the advancements by Large Language Models (LLMs), the availability of annotated corpora remains crucial. State-of-the-art systems still exhibit shortcomings, for example, when access to context or pragmatics for giving a true comprehension of the features of the involved phenomena is required [1]. Unfortunately, the development of large datasets annotated for specifically complex phenomena can be very time-consuming. When only small corpora are available, data augmentation techniques can be applied [2, 3]. Given a small set of original sample data, data augmentation artificially generates new instances that are similar and comparable to the existing data and can, therefore, be used to train and test systems with an extended dataset. In this paper, we present experiments for augmenting two small datasets annotated for two diverse, challenging phenomena, namely stereotypes and irony detection. In several works exploring data augmentation, Back-Translation (BT) [4] was shown to be a strong and relatively easy-to-implement baseline [5, 6]. A BT process generally consists of two steps: given one or multiple translation systems, a text in a source language is first translated into a chosen pivot language, and the resulting text is then translated back into the source language. The expected output of the BT process is a text that is similar but not the same as the original input, accounting for the linguistic differences intrinsic to the language pair, but also the idiosyncrasies of the chosen translation model(s). This relies on the fact that translation is only partially deterministic: the expected output should have the same meaning as the input, outputs that morphologically or syntactically differ may be considered as correct translations of the input. In BT, the application of (at least) two translations improves the variability between the input and the output text. The usefulness of a dataset augmented by applying BT depends on the quality of the translated outputs. Outputs too similar to the inputs can cause overfitting when used for training, while with too different outputs, there is a risk of a shift in distribution that is too large, which may negatively impact performance, at least in intradataset evaluations. A compromise between these two alternatives must be found. Therefore, an evaluation of the quality of translations and back-translations is important to assess the benefits. In this paper, we want to investigate the viability of BT as a data augmentation technique for low-resource tasks in various configurations. We use French and Italian as source languages — leveraging two multilingual datasets with subsets for these languages — and various languages as pivots for the BT process (French/Italian, English, and Arabic). We compare BT with an alternative process for data augmentation, specific for multilingual datasets, which we refer to as “cross-translation,” where the data from one language subset is translated and then used as a data augmentation source for another language subset. Our contributions are (1) an intrinsic qualitative human evaluation of translations and back-translations for stereotypes detection and irony detection datasets in various combinations of source and pivot languages, followed by (2) an extrinsic evaluation of machine learning model performance on these datasets, using these various data augmentation sources.

Conclusion
In this work, we have investigated using Back-Translation as a data augmentation technique for challenging low-resource tasks like stereotypes and irony detection, in a multilingual context. Through an intrinsic evaluation of the quality of the augmented instances, we identified modes of failure of Machine Translation, which could negatively impact the data augmentation process. These errors stem from the intrinsic differences between typologies and specific languages or translation model idiosyncrasies themselves potentially learned from methods like BT. Through a preliminary extrinsic evaluation of two multilingual datasets, we found that cross-translation can outperform Back-Translation, allowing us to augment one language subset by leveraging the variety of inputs present in the others. In future work, we aim to expand this study to more numerous and varied source and pivot languages, and different data augmentation configurations, namely, different proportions and selections of injected augmented data. We may also compare Back and Cross-Translation against or alongside other related techniques, such as multitasking learning or Active Learning. We also expect that some improvements can be obtained by mitigating translation failures; this can be done, for example, by leveraging an external LLM to check each step and remove or correct the errors from the final augmented dataset. Finally, it could also be interesting to perform tests with different model types on top of RoBERTa.