DS-Group at SIGHAN-2024 dimABSA Task: Constructing In-context Learning Structure for Dimensional Aspect-Based Sentiment Analysis (2024)

Abstract: Aspect-Based Sentiment Analysis (ABSA) is an important subtask in Natural Language Processing (NLP). More recent research within ABSA have consistently focused on conducting more precise sentiment analysis on aspects, i.e., dimensional Aspect-Based Sentiment Analysis (dimABSA). However, previous approaches have not systematically explored the use of Large Language Models (LLMs) in dimABSA. To fill the gap, we propose a novel In-Context Learning (ICL) structure with a novel aspectaware ICL example selection method, to enhance the performance of LLMs in dimABSA. Experiments show that our proposed ICL structure significantly improves the fine-grained sentiment analysis abilities of LLMs. Our code is publicly available at: https://github.com/ Maydayflower/dimABSA-ICL.

text: Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10), pages 127–132 August 16, 2024 c©2024 Association for Computational Linguistics
DS-Group at SIGHAN-2024 dimABSA Task: Constructing In-context Learning Structure for Dimensional Aspect-Based Sentiment Analysis
Ling-ang Meng, Tianyu Zhao and Dawei Song* School of Computer Science & Technology,
Beijing Institute of Technology, China {ling.ang.meng, tyzhao, dwsong}@bit.edu.cn
Abstract
Aspect-Based Sentiment Analysis (ABSA) is an important subtask in Natural Language Processing (NLP). More recent research within ABSA have consistently focused on conducting more precise sentiment analysis on aspects, i.e., dimensional Aspect-Based Sentiment Analysis (dimABSA). However, previous approaches have not systematically explored the use of Large Language Models (LLMs) in dimABSA. To fill the gap, we propose a novel In-Context Learning (ICL) structure with a novel aspectaware ICL example selection method, to enhance the performance of LLMs in dimABSA. Experiments show that our proposed ICL structure significantly improves the fine-grained sentiment analysis abilities of LLMs. Our code is publicly available at: https://github.com/ Maydayflower/dimABSA-ICL.

heading: 1 Introduction
text: Aspect-based Sentiment Analysis (ABSA) has been a significant research topic in Natural Language Processing (NLP). The goal of ABSA is to identify specific aspects within a sentence and determine the corresponding sentiment polarity (positive, neutral, or negative) for each aspect (Zhang et al., 2023b). This is different from traditional sentiment analysis (SA) that provides an overall sentiment prediction for the sentence. ABSA has been extensively studied, resulting in numerous effective algorithms.
However, human emotions are inherently continuous rather than discrete, involving two finergrained dimensions of sentiment, including valence and arousal Russell (1980). As illustrated in Figure 1, the valence dimension represents the degree of pleasure or displeasure sentiment, while the arousal dimension indicates the intensity of the sentiment. In this two-dimensional space, all emotions can be precisely represented. For instance, an emotion
*Corresponding author.
with a valence of 7 and an arousal of 7 would be closer to delighted, whereas a valence of 1 and an arousal of 9 would signify a very intense negative sentiment. Extending the traditional SA and ABSA to the two-dimensional space of sentiment has led to Dimensional Sentiment Analysis (DSA) and Dimensional Aspect-Based Sentiment Analysis (dimABSA).
Proceedings of the 8th International Joint Conference on Natural Language Processing, Shared Tasks, pages 47–52, Taipei, Taiwan, November 27 – December 1, 2017. c©2017 AFNLP
THU NGN at IJCNLP-2017 Task 2: Dimensional Sentiment Analysis for Chinese Phrases with Deep LSTM
Chuhan Wu1, Fangzhao Wu2, Yongfeng Huang1, Sixing Wu1 and Zhigang Yuan1 1Tsinghua National Laboratory for Information Science and Technology,
Department of Electronic Engineering, Tsinghua University, Beijing 100084, China {wuch15,wu-sx15,yuanzg14}@mails.tsinghua.edu.cn, yfhuang@mail.tsinghua.edu.cn
2Microsoft Research Asia, wufangzhao@gmail.com
Abstract
Predicting valence-arousal ratings for words and phrases is very useful for constructing affective resources for dimensional sentiment analysis. Since the existing valence-arousal resources of Chinese are mainly in word-level and there is a lack of phrase-level ones, the Dimensional Sentiment Analysis for Chinese Phrases (DSAP) task aims to predict the valencearousal ratings for Chinese affective words and phrases automatically. In this task, we propose an approach using a densely connected LSTM network a word features to identify dimensional sentiment on valence and a ousal for words and phrases jointly. We use word embedding as major feature and choose part of speech (POS) and word clusters as additional features to train the dense LSTM network. The evaluation results of our submissions (1st and 2nd in average performance) validate the effectiveness of our system to predict valence and arousal dimensions for Chinese words and phrases.
1 Introduction
Sentiment analysis is an important task in opinion mining for both academic and business use. Traditional sentiment analysis approaches mainly intend to identify the positive or negative sentiment polarities of text. This field has been widely researched and has many effective appro ches based on rules or statistical methods. However, analyzing only th polarit s of sentiments is rough and an’t differ sentiment distinctions in fine-grained. In order to go further in fine-grained sentiment analysis, some approaches were proposed to address this problem in more categories or in real-
value, such as dimensional sentiment analysis. Evaluati sentiment in valence-arousal (VA) space was first proposed by Ressel (1980). As shown in Figure 1, the valence dimension represents the degree of positive or negative sentiment, while the arousal dimension indicates the intensity of sentiment. Based on this two-dimensional representation, any affective state can be represented as a point in th VA coordinate plane by determining the degrees of valence and arou al of given words (Wei et al., 2011; Malandrakis et al., 2011; Yu et al., 2015; Wang et al., 2016) or texts(Kim et al., 2010; Paltoglou et al., 2013).
sary to VA sentiment evaluation. However, there is a lack of these resources especially for Chinese, and it’s usually difficult to construct them manually. Thus in order to get large scale lexicons in a reasonable cost, the objective of the shared task DSAP is to automatically acquire the valencearousal ratings of Chinese affective words and phrases. Some typical approaches to word-level VA rating task are based on statistical observations like linear regression (Wei et al., 2011) and kernel function (Malandrakis et al., 2011). However, these methods deeply rely on the affective lex-
47
As a rece tly em rging yet largely und rinvestigated task, dimABSA aims to c nduct finergrained sentiment analysis by assigning corresponding valence and arousal values to ach aspect in a sentence, as illustrated in Figure 2. Despite various DSA methods have been developed, which are mainly based on lexicons at the word-level or phrase-level, there is a lack of extensive and systematic studies on the aspect-level dimABSA. This paper aims to fill the gap. Inspired by the success of Large Language Models (LLMs) on the aspectlevel sentiment analysis tasks (Wang et al., 2023; Zhang et al., 2023a; Yang et al., 2024), we propose an in-context learning (ICL) framework for dimABSA and evaluates its effectiveness on three mainstream LLMs: qwen-plus (Bai et al., 2023), GPT-3.5 (OpenAI, 2023) and GPT-4 (Achiam et al., 2023). The main contributions of this paper are as follows:
127
(1) This paper is the first to explore the performance of LLMs on the dimABSA task. (2) This paper proposes an ICL framework for the dimABSA task, which is facilitated by a novel sample selection method. Experimental results demonstrate that our method significantly improves the performance of LLMs on the dimABSA task.

heading: 2 Related work

heading: 2.1 Aspect-based Sentiment Analysis
text: Early ABSA research is focused on the assessment of single sentiment elements. However, with advancements in the field, ABSA has evolved to include a growing number of sub-tasks, such as Aspect Sentiment Triplet Extraction (ASTE) (Zhang et al., 2020, 2022) and Aspect Sentiment Quad Prediction (ASQP) (Cai et al., 2021; Mao et al., 2022). A more recently emerged area is Dimensional Aspect-Based Sentiment Analysis (dimABSA), which introduces two scalar dimensions to more accurately describe sentiment.

heading: 2.2 Dimensional Aspect-Based Sentiment Analysis
text: Russell proposed a two-dimensional space for more precise emotion modeling, as illustrated in Figure 1. One dimension describes the intensity ranging from pleasant to unpleasant (i.e., Valence), while the other captures the intensity from calm to excited (i.e., Arousal). Based on this model, human emotional states can be represented in a more accurate manner (Bradley and Lang, 1999; Malandrakis et al., 2011). Researchers have incorporated this two-dimensional VA space into sentiment analysis, leading to the development of Dimensional Sentiment Analysis (DSA).
Existing research on DSA is heavily based on lexicons. In the field of Chinese research, the most commonly used lexicon is the Chinese EmoBank proposed by Lee et al., which includes 5,512 single words, 2,998 multi-word phrases, 2,582 single sentences, and 4,969 multi-sentence texts. Consequently, current DSA methods have primarily focused on the word-level (Wei et al., 2011) and phrase-level (Wu et al., 2017), neglecting highlevel emotional features. Lee et al. proposed the task of dimensional aspect-based sentiment analysis (dimABSA), extending DSA to the aspect-level. Our work primarily focuses on this task.

heading: 2.3 In-Context Learning
text: In recent years, Large Language Models (LLMs) have demonstrated remarkable performance across various NLP downstream tasks, and have shown excellent In-Context Learning (ICL) capabilities. ICL refers to the ability of LLMs to be applied directly to downstream tasks by adding a few examples to the prompt, without the need for parameter updates (Dong et al., 2023). Demonstration designing is a crucial component in constructing an in-context learning structure. Although the capabilities of LLMs in sentiment analysis have been widely studied (Lian et al., 2023; Wang et al., 2023; Yang et al., 2024), there has been no research exploring the impact of ICL on the dimensional ABSA abilities of LLMs. In this paper, we propose an ICL framework, demonstrating through experiments that our ICL framework significantly enhances the sentiment analysis capabilities of LLMs.

heading: 3 Methodology
text: In this section, we introduce the task definition and the components of our proposed ICL structure.

heading: 3.1 Task definition
text: Given a n-word sentence s = {w1, w2, ..., wn}, the output of dimABSA is y = {(A1, v1#a1), (A2, v2#a2), ..., (Ax, vx#ax)}, where Ai denotes the representation of an aspect and x represents the number of aspects in the sentence. vi denotes the valence value of the aspect Ai, ranging from 1 to 9, with 1 representing unpleasant and 9 representing pleasant. ai denotes the arousal value of Ai, also ranging from 1 to 9, with 1 representing calm and 9 representing excited.
Test Sample

heading: 3.2 Semantic Similarity Sample Selector (S4)
text: To select the most helpful samples for the dimABSA task, inspired by (Liu et al., 2021), we choose samples from the training set that are semantically closest to the test samples and use them as examples in the prompt. Given the specific nature of the dimABSA task, we believe that directly computing the semantic similarity of two sentences is inadequate. Instead, the aspect present in the samples should also be considered. The same description can represent different sentiment orientations for different aspects. For instance, the word "sour" typically does not convey negative sentiment when describing a lemon, but when referring to spoiled meat, it strongly indicates a negative sentiment.
Therefore, in our approach, we consider the aspect’s presence in the sentence when calculating similarity, leading to an aspect-aware semantic similarity measure.
First, we use BERT (Devlin et al., 2019) to obtain the representation Ti of the text for calculating semantic similarity. BERT is also used to obtain the representation Ai of the aspect. The process of obtaining Ti and Ai from a sentence Si and an aspect a is as follows:
Ti = BERT (Si),Ai = BERT (a) (1)
Then we use Cosine similarity to calculate the semantic similarity between two sentences Sp and Sq. The formula is as follows:
simt = cosine(Tp, Tq) = Tp · Tq
∥Tp∥ · ∥Tq∥ (2)
Next, we take into consideration the aspects contained in the sentences. Assuming the test sample
and the target sample in the training dataset contain m and n aspects respectively, we will calculate the similarity between each pair of aspects across the test sample and the target sample using Equation 3, and ultimately select the highest similarity value for the final computation.
sima = max i∈{1,2,...,m} j∈{1,2,...,n}
AiAj ∥Ai∥∥Aj∥
(3)
We believe that the presence of semantically similar aspects in two samples indicates that these reviews were likely given in similar contexts to some extent.
The overall aspect-aware similarity between two samples is ultimately computed, as shown in Equation 4, where α0 and α1 are trade-off parameters1.
sim(St, Si) = α0 · simt(Tt, Ti) + α1 · sima(At,Ai) (4)
We select N samples from the training set with the highest aspect-aware similarity scores to the test sample for use in subsequent prompt construction. Considering the impact of context length on the performance of LLMs, we set N to 10 in this paper.

heading: 3.3 In-context Learning Structure
text: The prompt we construct comprises a detailed description of the task, including the meanings of Valence, Arousal and Aspect, the input format, the required processing of the input, and the output format. Additionally, depending on various settings, the prompt may also include different sample examples for the LLMs to learn from. Figure 3 illustrates the prompt construction process of our proposed ICL framework.
1In our experiment, α0 and α1 are both set to 0.5.
Zero-shot setting. To demonstrate the effectiveness of our method, we first test the sentiment analysis capabilities of LLMs in a zero-shot setting. In the zero-shot setting, the prompt does not include additional examples for the LLMs to learn from. The prompt content is: [Please analyze the following sentence: test sample].
Few-shot setting with Random Selection. In NLP downstream tasks, the zero-shot setting often fails to achieve satisfactory results. Consequently, a common approach is to randomly select some examples for prompt construction. However, the samples chosen through this method often lack task representativeness, leading to limited improvements in the capabilities of LLMs.
Few-shot setting with S4. To address the lack of effective ICL frameworks in the dimABSA domain, we have proposed a Semantic Similarity Sample Selector (S4) for sample selection, detailed in Section 3.2. After obtaining samples through the S4, we construct the prompt in the following format: [I now provide these examples for you to learn from: Sample 1, Label 1; Sample 2, Label 2;...; Sample N, Label N. Please learn how to analyze from these examples, and then analyze the following sentence: Test Sample].

heading: 4 Experiments

heading: 4.1 Experimental Settings

heading: 4.1.1 Dataset
text: In this task we use the dataset provided by the organizer which contains 3000 sentences. Each sentence contains one or more aspects, and each of these aspects is annotated with corresponding valence and arousal values from 1-9.

heading: 4.1.2 Evaluation Metrics
text: To compare the sentiment analysis capabilities of different models, we use two metrics, Mean Absolute Error (MAE) and Pearson Correlation Coefficient (PCC), to indicate the performance of different models. The formulas for these two metrics are shown in Equation 5 and Equation 6, where ai ∈ A represents the ground truth value, and pi ∈ P represents the model prediction result. µA and µP denote the arithmetic mean of A and P, respectively. σ denotes the standard deviation.
MAE = 1
n
n∑
i=1
|ai − pi| (5)
The smaller MAE value, the better quality of the model’s predictions.
PCC = 1 n− 1 n∑
i=1
( ai − µA σA )( pi − µP σP ) (6)
A larger PCC value indicates a better quality of the model’s predictions.

heading: 4.2 Main Results
text: The evaluation results of our proposed ICL structure are presented in Table 1. Among all the results, the GPT-4o model utilizing our proposed ICL framework achieved the best performance on the VMAE, V-PCC, and A-PCC metrics. The qwen-plus model with our proposed ICL framework, slightly outperformed GPT-4o on the A-MAE metric. Experimental results show that our proposed method significantly improves the sentiment analysis capability of LLMs.

heading: 5 Conclusions
text: This paper explores the enhancement of LLMs for the dimABSA task through ICL. We have designed a sample selection method called Semantic Similarity Sample Selector (S4) and used it to select samples for prompt construction. Experimental results indicate that our proposed ICL framework significantly improves the performance of LLMs for the dimABSA task.
Limitations
The primary limitation of our proposed approach lies in its reliance on proprietary LLMs, which may pose challenges for reproducibility. To achieve optimal results, we did not conduct experiments on
mainstream open-source LLMs such as LLaMA2 and LLaMA3. However, the experimental results on proprietary LLMs demonstrate that our proposed method is significantly effective. In future work, we plan to extend our experiments to include a broader range of LLMs to develop a more performant and generalized approach.

heading: Acknowledgements
text: This work is funded by the Natural Science Foundation of China (grant no: 62376027) and Beijing Municipal Natural Science Foundation (grant no: 4222036 and IS23061).

heading: Future Work
text: In future work, we plan to extend our experiments to include a broader range of LLMs to develop a more performant and generalized approach