Emotion classification on code-mixed text messages via soft prompt tuning
Year: 2023

Abstract:
Emotion classification on code-mixed text messages is challenging due to the multilingual languages and non-literal cues (i.e., emoticons). To solve these problems, we propose an innovative soft prompt tuning method, which is lightweight and effective to release potential abilities of the pre-trained language models and improve the classification results. Firstly, we transform emoticons into textual information to utilize their rich emotional information. Then, variety of innovative templates and verbalizers are applied to promote emotion classification. Extensive experiments show that transforming emoticons and employing prompt tuning both benefit the performance. Finally, as a part of WASSA 2023, we obtain the accuracy of 0.972 in track MLEC and 0.892 in track MCEC, yielding the second place in both two tracks.

heading: 1 Introduction
text: Emotion plays an important role in social relationships, decision making, etc. Emotion analysis aims to enable machines to learn the emotions contained in textual information, such as conversations, ecommerce reviews and personal blogs (Balabantaray et al., 2012). With the surge of social media, emotion classification receives increasing attentions and brings huge commercial and social implications. 1 Text message is a common form of communication that exists in Twitter, YouTube, etc. Some text messages may contain typos, e.g., "habe" (have), and emotional words, e.g., "hahhhhh", while others may include visual cues such as icons, e.g., ":-)", and emoji. These features bring rich emotion information, but are typically not included in the vocabulary of pre-trained model like BERT (Bidirectional Encoder Representation from Transformers), T5 (Text-to-Text Transfer Transformer)(Raffel et al., 2020), etc. In addition, codemixing is a critical challenge in emotion classification, which means that two or more languages
1* These authors contributed equally to this work.
(Yulianti et al., 2021), such as English and Roman Urdu, are contained in a single piece of text. Although code-mixed text is widely used, research on emotion classification for code-mixed corpus is still scarce (Ameer et al., 2022) since it is more difficult to recognizing emotions in code-mixing languages than in a monolingual language.
Fine-tuning is a widely-used method to improve performance of pre-trained language models (PLMs), however, it is memory-consuming and time-consuming. Compared to fine-tuning, prompt learning is more lightweight, thus has received increasing attentions (Liu et al., 2021a). T5 introduces a unified framework that converts any language problem into a text-to-text format. By introducing different prompts, the model can then adapt to different tasks, e.g., language translation and emotion classification. GPT-3 (General Pre-trained Transformer-3) (Brown et al., 2020) proposes context learning, where models can be applied directly in zero-shot or few-shot tasks without further finetuning. Utilizing language prompts helps to probe knowledge in PLMs and obtains better performance (Gu et al., 2021). Prompt learning has been widely applied in tasks of natural language understanding (Zhu et al., 2022; Yeh et al., 2022) and generation (Zheng and Huang, 2021), and even has been adopted in some vision tasks (Kirillov et al., 2023) and multi-modality tasks (Yao et al., 2021).
In this paper, we present our effort in emotion classification on code-mixed text messages. The contributions are summarized as follows:
• A concise emoticon pre-processing method is proposed to transform emoticons into textual information.
• Variety of innovative templates are designed to improve emotion classification on codemixed text messages.
• Specific verbalizers are applied, which
596
achieves efficient emotion classification with low resource consumption.

heading: 2 Related Work

heading: 2.1 Emotion Classification
text: Because of the vagueness of definition and the similarity of some emotions, emotion classification is often seen as a subjective and challenging task.
Gaind et al. (2019) combines two different approaches to extract emotions. The first approach employs several textual features like emoticons, degree words, negations and other grammatical analysis. The second approach adopts algorithms based on machine learning techniques.
Polignano et al. (2019) proposes a classification method based on deep neural networks, which is effective on different emotion classification datasets. They also compare three pre-trained word embeddings for words encoding.
Hasan et al. (2019) uses a dimensional model to define emotion categories. Besides, they propose a soft classification method to measure the probability of assigning messages to each emotion category. In addition, a framework called EmotexStream is developed for real-time emotion tracking.
Ameer et al. (2023) proposes multiple attention mechanisms that reveal the contribution of each word to each emotion. They also investigate the usage of LSTM, the fine-tuning of transformer networks for multi-label emotion classification. Experimental results show that these new transfer learning models are able to outperform current state-ofthe-arts on the SemEval-2018 Task-1C dataset.
Ashraf et al. (2022) creates the first multi-label emotion dataset, consisting of six basic emotions from the Urdu Nastalíq script. In addition, they build a set of baseline classifiers and provide insight into these classifiers. The baseline classifiers include machine learning algorithms (i.e., Random Forest, Decision Tree, Sequential Minimal Optimisation, AdaBoostM1 and Bagging), deep learning algorithms and Baseline Based Transformer (i.e., BERT). They use a combination of text representations, which are stylometric-based features, pretrained word embedding, word-based n-grams, and character-based n-grams.

heading: 2.2 Prompt Learning
text: Prompt learning typically includes the design of prompt templates, the optimization of verbalizers and the selection of the PLM. Prompt templates and
verbalizers are usually produced by expert knowledge or generated automatically using search or optimization methods (Hu et al., 2021; Shin et al., 2020; Gao et al., 2021; Liu et al., 2021a). The selection of PLMs needs to take the model structure( such as auto-regressive model), pre-training dataset into consideration.
Liu et al. (2021a) finds that prompt tuning can be effective among different models and natural language tasks. They propose a universal and simple P-tuning v2 method, which proves that prompt tuning can be comparable to fine-tuning, while only 0.1%-3% of the parameters are fine-tuned.
Hambardzumyan et al. (2021) proposes soft verbalizers for prompt learning. They use a continuous vector for each class and adopt dot multiplication between the output of masked language model and the class vector to generate the probabilities.
Lang et al. (2022) finds that co-training can improve the prompt-based learning using unlabelled data. Specifically, co-training can benefit the original prompt model while learning smaller downstream task-specific models.
Han et al. (2022) proposes a prompt tuning method with rules "PTR", which encodes the prior knowledge of a classification task into a rule. They then design sub-prompts based on the rule to adapt the task. Results show that PTR achieves a good trade-off between effectiveness and efficiency in prompt construction compared to the state-of-arts.
In this paper, we propose innovative prompt tuning techniques and bring valuable attempts for emotion classification on code-mixed text messages.

heading: 3 Methods

heading: 3.1 Emoticon Pre-processing
text: Text messages may include icons and Unicode emoji, which provide rich emotional information and are overlooked by many researchers. We propose a concise method to convert these features into textual form, which consists of icon-emotion mapping and Unicode-short name mapping.
We first statistically gather the icons in the training set and collect more icons from the Internet as a supplement. An icon-emotion mapping is then constructed based on the visual features of icons. For example, ":-)" is replaced by [joy]. It is worth mentioning that some icons can represent different emotions in different contexts. For example, ";-/" can indicate emotions such as disgust, anger and sadness. In this case, ";-/" is replaced by [disgust,
anger, sadness], informing the PLM that this text may contain some of these three feelings. Since only text replacements are performed, the computation cost of this operation is trivial.
For Unicode-short name mapping, we convert the Unicode encoding to a CLDR short name, e.g. replacing U+1F600 with [grinning face].

heading: 3.2 Prompt Tuning
text: According to the experience (Liu et al., 2021b), a single word change in prompts could make a drastic difference. Therefore, we tried a variety of prompts.
Firstly, we design textual prompts following the zero-shot and few-shot approach of GPT-3, as shown in Table 1. Considering the short length of the text messages in CM-MEC-21 corpus, we limit the number of manual prompt samples for the few-shot approach.
Then we design several soft prompts following Zhu et al. (2022). We investigate the effect of soft template with different numbers and distribution of soft tokens. The structure of soft template is shown in Fig 1. Finally, we compare the results of the manual verbalizer and the soft verbalizer (Hambardzumyan et al., 2021).
For the PLMs, we adopt XLM-RoBERTa with different model scales and pre-training sets. In the process of optimizing the templates and verbalizers, we just freeze the parameters of the pre-trained model, as shown in Fig 2.
Ultimately, the equations for result prediction and objective optimization are noted as
p̂ν,τ (x) = softmaxVν (F (Tτ (x))) ,
{υ∗, τ∗} = argmax ν,τ
∑ pν,τ (x) log p̂ν,τ (x),
where τ, ν denote the parameters of the template and verbalizer respectively. T, F, V represent the template, PLM and verbalizer respectively. The x refers to the input text message, and p̂ represents the estimated emotion probability distribution corresponding to the text message while p represents real distribution.

heading: 4 Results and Discussion
text: We adopt CM-MEC-21 corpus (Ameer et al., 2022) in this work, which includes 11650 MCEC messages and 11603 MLEC messages. We combine MCEC and MLEC text messages for both training and evaluating. The accuracy based on the merged dataset for both tracks are summurized in Table 2.

heading: 4.1 Emoticon Pre-processing
text: Firstly, we analyze the effects of emoticon preprocessing. We take XLM-RoBERTa fine-tuned on Roman Urdu corpus2 as the baseline, where soft template B5M6 and manual verbalizer are used but emoticon pre-processing are not employed. As shown in Fig 3, both parts of emoticon pre-processing (i.e., icon-emotion mapping and Unicode-short name mapping) are effective in helping the machine to recognize the emotions.
2https://huggingface.co/Aimlab/ xlm-roberta-roman-urdu-finetuned

heading: 4.2 PLMs Comparison
text: We compare the performance of different PLMs, as shown in Fig 4. We use different XLM-RoBERTa models, where soft template B5M6, soft verbalizer and emoticon pre-processing are adopted. We can see that larger models bring better performance. Meanwhile, the language used during pre-training has a significant impact on the final results.
After about 25 epochs of training, XLMRoBERTa-fine-tuned achieves an accuracy of 0.9, indicating that fine-tuned model can better understand the code-mixed messages. We can draw a conclusion from the experiments that, with the finetuned model, just a small amount of prompt tuning can achieve satisfactory results. By contrast, more prompt tuning is required while employing general pre-trained models.

heading: 4.3 Prompt Tuning
text: The combination of template and verbalizer significantly affects classification performance, as shown in Table 3. Although manual prompts are relatively easier to design, their performances are far inferior to that of soft prompts.
We can see from Table 3 that a well-designed prompt template is a prerequisite for efficient classification. Otherwise, the effect of the prompt will be limited, such as Manual 1 and Manual 2F. Also, increasing soft tokens in the template can improve the result, but only to a limited extent.
Soft verbalizer can universally benefit the classification performance. For example, while employing template of Manual 2, the soft verbalizer brings an accuracy gain of 0.780 compared with the manual verbalizer. Besides, soft verbalizer enables low resource consumption while maintaining satisfactory performance, e.g., just one soft token in the template can achieve an accuracy of 0.9.

heading: 4.4 Limitations
text: While dealing with the MLEC task, we find that most outputs from our method have only one category. Therefore, our method still has room for improvement in handling MLEC problem.

heading: 5 Conclusion
text: In this paper, we propose the innovative prompt tuning for emotion classification on code-mixed text messages. We first transform emoticons into textual information to utilize their rich emotional information. Then, variety of innovative templates and verbalizers are applied to promote emotion classification. The results show that transforming emoticons benefits the final results. More importantly, even without model fine-tuning, optimizing the prompts yields efficient classification. Finally, we obtain the accuracy of 0.972 in track MLEC and 0.892 in track MCEC, yielding the second place in both two tracks. In future work, we will pay more attention to combining model fine-tuning with prompt learning, and further explore the capacity of PLMs on emotion classification.

heading: Future Work
text: In future work, we will pay more attention to combining model fine-tuning with prompt learning, and further explore the capacity of PLMs on emotion classification