(2024)

Abstract: This study describes the model design of the NYCU-NLP system for the EXALT shared task at the WASSA 2024 workshop. We instruction-tune several large language models and then assemble various model combinations as our main system architecture for cross-lingual emotion and trigger detection in tweets. Experimental results showed that our best performing submission is an assembly of the Starling (7B) and Llama 3 (8B) models. Our submission was ranked sixth of 17 participating systems for the emotion detection subtask, and fifth of 7 systems for the binary trigger detection subtask.

text: Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 505–510 August 15, 2024 ©2024 Association for Computational Linguistics
This study describes the model design of the NYCU-NLP system for the EXALT shared task at the WASSA 2024 workshop. We instruction-tune several large language models and then assemble various model combinations as our main system architecture for cross-lingual emotion and trigger detection in tweets. Experimental results showed that our best performing submission is an assembly of the Starling (7B) and Llama 3 (8B) models. Our submission was ranked sixth of 17 participating systems for the emotion detection subtask, and fifth of 7 systems for the binary trigger detection subtask.

heading: 1 Introduction
text: Emotion detection is a well-studied NLP task that aims to automatically identify affective information from texts. The EXALT task organized within the WASSA-2024 workshop focuses on the explainability of cross-lingual emotion detection in tweets. In the emotion detection subtask, the participating system should predict for each tweet an emotion label from 6 possible classes: Love, Joy, Anger, Fear, Sadness, and Neutral. To investigate transferable emotion information across languages, training data is provided in English and evaluation data consists of five different target languages: Dutch, Russian, Spanish, French and English. In the binary trigger detection subtask, participating systems should further identify which words or emoticons can be used to express the emotion.
This paper describes the NYCU-NLP (National Yang Ming Chiao Tung University, Natural Language Processing Lab) system for the EXALT shared task (Maladry et al., 2024). Our solution
explores the use of instruction-tuned LLMs, including Mistral (7B) (Jiang et al., 2023), Starling (7B) (Zhu, 2023) and Llama 3 (8B) (Meta AI, 2024). We then assemble various model combinations as our main system architecture. Experimentally, our best performing submission was an assembly of Starling (7B) and Llama 3 (8B), which was ranked sixth of 17 participating systems for the emotion detection subtask and fifth of 7 systems for the binary trigger detection subtask.
The rest of this paper is organized as follows. Section 2 reviews recently related studies on emotion detection. Section 3 describes the NYCUNLP system for the EXALT shared task. Section 4 presents the results and performance comparisons. Conclusions are finally drawn in Section 5.

heading: 2 Related Work
text: Transformer-based language models have been widely applied to emotion detection. An ensemble of the BERT and ELECTRA models was used to detect emotions (Kane et al., 2022). A knowledgeenriched transformer was designed for emotion detection in textual conversations (Zhong et al., 2019). A topic-driven transformer was proposed to detect emotions within dialogues (Zhu et al., 2021). Two hierarchical transformers were trained to use context-/speaker-sensitive information for emotion detection in conversations (Li et al., 2020). Sentiment-enhanced RoBERTa transformers were used to predict emotion and empathy intensities (Lin et al., 2023). Empirical evaluations also showed that transformer-based models such as BERT and XLNet outperformed conventional neural networks for sentiment intensity prediction (Lee et al., 2022). A transformer-based fusion model was developed to integrate semantic
505
representations at different degrees of linguistic granularity for emotional intensity prediction (Deng et al., 2023).
Recently, Large Language Models (LLM) have been used for emotion detection. Fine-tuned GPT3 models with prompt engineering for zero-shot or few shot learning with ChatGPT and GPT-4 models were evaluated for emotion detection (Nedilko and Chu, 2023). Prompt engineering techniques were applied to a GPT model for emotion detection in a code-switching setting (Nedilko, 2023). Multiple features generated by ChatGPT were integrated for emotion recognition in conversations (Tu et al., 2023). The LLM-GEm system was designed to use GPT 3.5 for empathy prediction (Hasan et al., 2024). The abilities of CPT-4, Llama2-Chat-13B and Aplaca-13B to identify emotion triggers were evaluated, analyzing the importance of trigger words for emotion prediction (Singh et al., 2024). Given the results obtained by most such approaches, we are motivated to explore the application of LLMs to the emotion and trigger detection tasks.

heading: 3 The NYCU-NLP System
text: Figure 1 shows our NYCU-NLP system architecture for the EXALT shared task. We first
instruction-tune LLMs and then assemble finetuned LLMs for cross-lingual emotion and binary trigger detection in tweets.

heading: 3.1 Large Language Models
text: The following LLMs are used to explore the effectiveness of our system architecture.
(1) Mistral (7B) (Jiang et al. 2023) Mistral-7B is an open source LLM under the Apache 2.0 license which leverages Group-Query Attention (CQA) for faster inference and uses Sliding Window Attention (SWA) to handle longer sequences at smaller cost. Mistral-7B claims it outperforms Llama 2 (13B) across all evaluated benchmarks.
(2) Starling (7B) (Zhu et al., 2023) Starling-7B is an open LLM trained by Reinforcement Learning from AI Feedback (RLAIF). A new ranking dataset, called Nectar, was used for the proposed new reward training and policy tuning pipeline. Starling-7B was mainly evaluated based on MT-Bench and AlpacaEval, which are GPT-4-based comparisons.
(3) Llama 3 (8B) (Meta AI, 2024) Llama 3 is Meta’s next generation release of the well-known Llama model. We use a pretrained and instruction-fine-tuned Llama 3 model with 8B parameters.

heading: 3.2 Instruction Fine-tuning
text: We continually fine-tune these three publicly released LLM models. Figure 2 shows the prompts used for instruction fine-tuning (Wei et al., 2022). The system is configured as a master of sentiment analysis for both tasks. For the emotion detection subtask, we ask the LLM to classify the given sentence into six defined emotion categories. For the binary trigger detection subtask, we used the “#” symbol to emphasize words and emoticons that can be used to trigger the emotion.
We also use the Low-Rank Adaption (LoRA) technique (Hu et al., 2021), which freezes the pretrained LLM weights and injects trainable rank decomposition matrices into each layer of the transformer architecture, greatly facilitating the instruction-tuning process for downstream tasks.

heading: 3.3 Assembly Mechanism
text: During the inference phase, we use a voting-based assembly mechanism, which each LLM conducting an independent detection for each testing instance, effectively integrating fine-tuned LLMs to determine the system output by a majority of votes.
For the emotion detection subtask, if a testing instance does not have a major category prediction,
1 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
https://huggingface.co/Nexusflow/Starling-LM-7B-beta
we use the ‘neutral’ emotion category as an alternative option.
For cases without a majority of prediction results in the binary trigger detection subtask, if a word or emoticon is identified by any one of our used models, we directly regard it as a trigger for our system output.

heading: 4 Performance Evaluation

heading: 4.1 Data
text: The datasets were mainly provided by task organizers, including the training data in English, along with development and test data for each of the five target languages. For the emotion detection subtask, there are respectively 5000, 3000 and 2500 instances in the training, development and test sets, and we used these datasets without augmentation. For the binary trigger subtask, we had respectively 3000, 300 and 832 instances for each provided dataset.

heading: 4.2 Settings
text: All pretrained models were downloaded from HuggingFace 1 . We continuously fine-tuned the LLM models using the training datasets only. All experiments were conducted on a server with two Nvidia V100 GPUs (Total 64GB memory). The hyperparameter values for our model implementation were manually optimized on the given development set as follows: epochs 20; batch size 2; optimizer AdamW; learning rate 1e-4; LoRA r 16; LoRA alpha 32; LoRA drop 0.1 and max token length of 20.

heading: 4.3 Results
text: Table 1 shows the submission results on the development set. Among three independent LLMs, Llama 3 (8B) outperformed the others in the terms of F1-scores on both subtasks. Assemble LLMs usually outperformed independent LLMs, except the Mistral may reduce performance. The best performance was achieved by an assembly of Starling (7B) and Llama 3 (8B), so we use this LLM setting either individually or in combination as our final submission for official ranking.
Table 2 shows the submission results on the test set. Independent Llama 3 (8B) outperformed independent Starling (7B) for all evaluation metrics.
https://huggingface.co/meta-llama/Meta-Llama-3-8BInstruct
The assembly of Starling and Llama 3 obtained the best F1 score of 0.5951 for the emotion detection subtask, ranking the sixth of 17 participating systems. In addition, this assembly achieved the best token F1 of 0.5636 for the binary trigger detection subtask, ranking the fifth among all 7 participating systems.

heading: 4.4 Discussion
text: We did not use prompt engineering techniques to configure other prompts due to limited computational resources. Therefore, prompts used for instruction fine-tuning needed to be improved for performance enhancements.
In addition, since the LLMs were pre-trained using multi-lingual data, we do not use any machine translation techniques in the tasks.

heading: 5 Conclusions
text: This study describes the NYCU-NLP system for the EXALT shared task at the WASSA 2024 workshop, including model design and performance evaluation. We instruction-fine-tuned the LLMs to effectively detect cross-lingual
emotions and triggers. Experimental results indicate that our best submission is an assembly of Starling (7B) and Llama 3 (8B) models, achieving a F1 score of 0.5961 for the emotion detection subtask (ranking sixth out of seventeen) and a token F1 of 0.5636 for the binary trigger detection subtask (ranking fifth out of seven).
This pilot study is our first exploration in the cross-lingual emotion and trigger detection task. In future, we will exploit other advanced LLMs to further improve performance.

heading: Acknowledgments
text: This study is partially supported by the National Science and Technology Council, Taiwan, under the grant NSTC 111-2628-E-A49-029-MY3. This work was also financially supported by the Cocreation Platform of the Industry Academia Innovation School, NYCU.
Limitations
This work does not propose a new model to address this shared task. Due to computational resource limitations, experiments were conducted
with basic settings without other advanced explorations to enhance system performance.

heading: Future Work
text: In future, we will exploit other advanced LLMs to further improve performance