YNU-HPCC at WASSA 2023: Using Text-Mixed Data Augmentation for Emotion Classification on Code-Mixed Text Message
Year: 2023

Abstract:
Emotion classification on code-mixed texts has been widely used in real-world applications. In this paper, we build a system that participates in the WASSA 2023 Shared Task 2 for emotion classification on code-mixed text messages from Roman Urdu and English. The main goal of the proposed method is to adopt a text-mixed data augmentation for robust code-mixed text representation. We mix texts with both multilabel (track 1) and multi-class (track 2) annotations in a unified multilingual pre-trained model, i.e., XLM-RoBERTa, for both subtasks. Our results show that the proposed text-mixed method performs competitively, ranking first in both tracks, achieving an average Macro F1 score of 0.9782 on the multi-label track and of 0.9329 on the multi-class track.

heading: 1 Introduction
text: Emotion classification is a fundamental task in natural language processing (NLP). The main purpose is to identify the emotions in a written text that potentially represents the author’s mental state. Compared with single-label emotion classification, multi-label emotion classification is more difficult to determine all possible emotions instead of only one emotion in a given text. Accordingly, multilabel classification has shown wide applications, such as health care and e-learning (Maxwell et al., 2017).
With the rapid growth of the Internet, linguistic code-mixed culture has become one of the most prominent communication approaches. Codemixed texts represent texts written by two or more languages, simultaneously. According to Ameer et al. (2022), more than half of Europeans use codemixed texts in communication. Thus, providing a more accurate judgment about the potential emotional state of such code-mixed texts is essential for various real-world applications, such as author profiling and sentiment analysis (Santosh et al., 2013; Ahmed et al., 2015).
Toward this issue, WASSA 2023 proposes a shared task for emotion classification on codemixed (with Roman Urdu and English) text messages, consisting of two tracks. 1) Track 1: Multilabel Emotion Classification (MLEC). 2) Track 2: Multi-class Emotion Classification (MCEC). Codemixed texts are given for both tracks. The MLEC requires a system to classify such texts as neutral or multi-label (one or more) emotions in given texts while MCEC requires the system for neutral or only one emotion that best presents the mental state of the author, as shown in Figure 1.
We participate in both tracks on the shared tasks and found that the main challenges are twofold: 1) Code-mixed texts consist of bilingual languages in a text; 2) Multiple labels are annotated for a code-mixed text. To address these problems, we provide a system that utilizes a text-mixed data augmentation method to handle two tracks at the same time. Initially, we build a shared pair-mixed corpus in a random combination between MLEC
611
and MCEC training samples. Next, hidden states and annotated labels in a paired sample are individually mixed in an interpolation method by the same weights. Note that, hidden states in paired texts are generated from multilingual pre-trained language models (PLMs) (Qiu et al., 2020), i.e., XLMRoBERTa (Conneau et al., 2019), that can align word representations from multilingual tokens and semantics from multilingual sentences. Finally, the predicted probabilities over multiple emotions from the system are converged with mixed annotated labels in a Kullback-Leibler (KL) (Eguchi and Copas, 2006) divergence loss function. Consequently, the proposed system can be shared for both tracks in one training phase. Extensive experiments are conducted to investigate the effect of the proposed method and the best submissions reveal that our system ranks first in both MLEC and MCEC tracks.
The remainder of this paper is constructed as follows. A line of related works is provided in section 2. A detailed description of the proposed system is introduced in section 3. Experimental results are analyzed in section 4. Finally, conclusions are drawn in section 5.

heading: 2 Related Work
text: Emotion classification is a challenging NLP task that aims to automatically classify text documents into one or more predefined emotion categories. It has long been of interest to researchers in areas such as sentiment analysis, opinion mining, and social media analysis.
Recent studies have explored the use of different deep learning architectures, such as Convolutional Neural Networks (CNNs) (Krizhevsky et al., 2017) and Recurrent Neural Networks (RNNs) (Mikolov et al., 2010), to improve the performance of emo-
tion classification. Other studies have focused on the development of pretraining models such as BERT (Devlin et al., 2019), ALBERT (A Lite BERT) (Lan et al., 2019) and RoBERTa (Liu et al., 2019), which have been shown to achieve stateof-the-art results on various NLP tasks, including emotion classification.
A growing body of research has focused on the use of monolingual and cross-lingual models to improve emotion classification. Monolingual methods are based on training models on large amounts of data from a single language, while cross-lingual models make use of data from multiple languages to learn more robust representations. Cross-lingual models such as XLM (Barbieri et al., 2022) have shown promise in many NLP tasks, including emotion classification.
Due to the scarcity of low-resource data, data augmentation is essential. Mixup (Zhang et al., 2017) is a simple and effective data augmentation method, which can significantly improve the effect in multiple fields such as image, text, speech, recommendation. Different variants of mixing methods have also been designed in the different space, the cutMix (Yun et al., 2019) method takes a different approach, instead of interpolating two samples from a numerical perspective, but from the spatial perspective of the image, it cuts a random rectangular area on one picture to another picture to generate a new picture. Manifold mixup (Verma et al., 2018) extends mixup, and extends the mixing of input data (raw input data) to the output mixing of the intermediate hidden layer.

heading: 3 Methodology
text: In this section, we will describe our system that participated in WASSA 2023 shared task on emotion classification on code-mixed text messages. As
shown in Figure 2, the proposed models consist of four parts, including pair-mixed corpus, sentence encoder, text-mixed interpolation, and classifier. Before introducing the proposed model, we describe the shared task 2 in advance.

heading: 3.1 Emotion Classification on Code-mixed Text
text: Given a text code-mixed text x = [x1,x2, . . . ,xN ] where N represents the text length. Regarding MCEC and MLEC, it requires a system fθ(ŷ|x) to predict ground-truth emotions yc ∈ RC and yl ∈ RC where θ represents the whole parameters in the system and C is the total number of emotional labels.

heading: 3.2 Pair-mixed Corpus
text: In this paper, we propose a unified system for both MCEC and MLEC tasks, simultaneously. Hence, we initially mix both corpora in a random combination. In detail, for each pair of training sample ((xci ,x l j), (y c i ,y l j)) , we random select a codemixed SMS message from MCEC training set Dc and another one from MLEC Dl , where (xci , yci ) ∈ Dc, (xlj ,ylj) ∈ Dl, i ∈ [1 : |Dc|], and j ∈ [1 : |Dl|] Consequently. a mixed corpus Dm = ∑ i,j((xi,xj), (yi,yj)) is generated.

heading: 3.3 Multilingual Text Encoders
text: Due to code-mixed texts comprised bilingual languages, aligning word embedding and semantic cross-lingual sentences or phrases is critical for robust text representations. To this end, we adopt a cross-lingual PLM, XLM-RoBERTa, as the sentence encoder to encode paired texts into hidden spaces, formulated as:
hi,hj = XLM((xi,xj)) , (1)
where(xi, xj) ∼ Dm, hi,hj ∈ RN×d; d is the dimensionality of hidden states.

heading: 3.4 Text-mixed Interpolation
text: To further mix up (xi,yi) and (xj ,xj), a mixed interpolation method is proposed. Based on sentence representation hi and hj , we choose [CLS] (a special token in PLMs) to represent the global sentence representation mixed in hidden spaces (denoted as h̃ ∈ Rd), as well as annotated labels
(denoted as ỹ ∈ RC ), as shown in Figure 2.
h̃ = λhi,[CLS] + (1− λ)hj,[CLS] ỹ = λyi + (1− λ)yj λ∼Beta(α, α) λ = max(λ, 1− λ) , (2)
where λ ∈ [0, 1] is generated from Beta distribution with a hyper-parameter α .

heading: 3.5 Training Objective and Inference Strategy
text: In this section, we introduce the proposed system in training and inference phases. Training objective. To predict emotional probabilities ŷ ∈ RC , we apply a Multi-Layer Perceptron (MLP) to encode mixed textual hidden states h̃ :
ŷ = MLP(h̃)
= Linear(tanh(Dropout(h̃))) , (3)
where Linear(·), tanh(·) and Dropout(·) are fully-connected layer, hyperbolic tangent activation function, and dropout function, respectively. The loss function between predicted probabilities and mixed labels is KL divergence (Eguchi and Copas, 2006) for the system training:
L = KL(f(ŷ|(xi,xj); θ)||ỹ). (4)
Inference strategy. In the inference phase, MCEC and MLEC test datasets are separately fed into the system for individual purposes. Note that there are not mixed procedures in the inference phase. For MCEC task, the argmax function is used to predict one emotion; for MLEC task, a threshold score of 0.5 is used to predict all most possible emotions, formulated as:
ŷc = argmax(softmax(ŷ)) ŷl = threshold(sigmoid(ŷ), 0.5) . (5)

heading: 4 Experimental Results
text: In this section, extensive experiments were conducted for both MCEC and MLEC tracks.

heading: 4.1 Datasets
text: The Internet is the most prominent source in promoting global, linguistic code-mixed culture. In South Asian community and particularly in Pakistan, code-mixed (English and Roman Urdu) text became a preferable script for Facebook comments/posts, tweets, and daily communication using SMS messages. The shared task organizers made available the dataset from (Ameer et al., 2022). Table 1 reported the detailed datasets in statistics.

heading: 4.2 Evaluation Metrics
text: To evaluate the performance of participant systems, the official competition provides Micro F1 (MicF1), Macro F1 (Mac-F1), and Accuracy (Acc) for track 1 and Mac-F1, Mac-Precision (Mac-P), MacRecall (Mac-R), and Acc for track 2.

heading: 4.3 Implementation Details
text: Hyper-parameters. All sentence is tokenized by XLM-RoBERTa based tokenizer with a maximum length of 90. Sentence encoder is twitter-xlm-roberta-base-sentiment PLM with the dimensionality of 768. α in Beta distribution is set as 0.75. Dropout ratio in MLP module is set as 0.2. For optimization, AdmW is adopted with learning rate of 2e-5 and batch size of 64. The code of this paper is availabled at https: //github.com/linsongisgood/wassa2023. Baselines. To investigate the effect of the proposed
method, several baseline models are introduced, including BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) and their variants in cross-lingual versions.

heading: 4.4 Results and Analysis
text: Comparative Dev results were reported in Table 2. Due to PLMs that transfer generic language performance learned from a large pretrained corpus into downstream tasks, several PLMs achieved competitive results on both tracks. It can be observed that, with the cross-linguistic pretraining phase, PLMs such as mBERT (Pires et al., 2019) and XLM-RoBERTa (Conneau et al., 2019) outperformed monolingual PLMs such as BERT and XLM-RoBERTa, respectively. This phenomenon demonstrated that aligning cross-linguistic word representation and semantics is crucial to generate robust representation on code-mixed texts.
Furthermore, we found that the introduction of the text-mixed data augmentation method gained more performance on both tracks. A possible reason may be that the combination of multi-class and multi-label corpora improved the generalization capability of the system. Note that the proposed text-mixed method facilitates the shared system simultaneously performing both tracks during one training phase. Table 3 showed our best submissions with official results and ranks, revealing the effectiveness of the proposed system.

heading: 5 Conclusions
text: In this paper, we proposed described our system submission WASSA 2023 shared task 2 in emotion classification. Our system utilizes the text-mixed method and cross-lingual PLMs for robust representation of code-mixed texts. As a result, our system won the first rank in both tracks.
In the future, we will explore the text-mixed method applied to large amounts of unlabeled code-
mixed texts for better performance.
Limitations
The limitations of this work can be concluded into three points: 1) The data in the test set is relatively small, so it cannot more accurately reflect the effectiveness of the method proposed in this paper. We believe that tuning the model on a larger dataset can help improve the performance of the model. 2) Due to device performance limitations,we did not experiment with larger models. In our experiment,we only tested the method with models like XLM-RoBERTa, mBERT and BERT. Its performance with larger models is not known. 3) We did not perform an extensive hyperparameter search, which might further improve the model’s performance.

heading: Acknowledgements
text: This work was supported by the National Natural Science Foundation of China (NSFC) under Grant Nos. 61966038 and 62266051, and the Yunnan Postdoctoral Science Foundation under Grant Nos. C615300504048. The authors would like to thank the anonymous reviewers for their constructive comments.

heading: Future Work
text: In the future, we will explore the text-mixed method applied to large amounts of unlabeled code-
mixed texts for better performance. Limitations
The limitations of this work can be concluded into three points: 1) The data in the test set is relatively small, so it cannot more accurately reflect the effectiveness of the method proposed in this paper. We believe that tuning the model on a larger dataset can help improve the performance of the model. 2) Due to device performance limitations,we did not experiment with larger models. In our experiment,we only tested the method with models like XLM-RoBERTa, mBERT and BERT. Its performance with larger models is not known. 3) We did not perform an extensive hyperparameter search, which might further improve the model’s performance