CCIIPLab at SIGHAN-2024 dimABSA Task: Contrastive Learning-Enhanced Span-based Framework for Chinese Dimensional Aspect-Based Sentiment Analysis (2024)

Abstract: This paper describes our system and findings for SIGHAN-2024 Shared Task Chinese Dimensional Aspect-Based Sentiment Analysis (dimABSA). Our team CCIIPLab proposes an Contrastive Learning-Enhanced Span-based (CL-Span) framework to boost the performance of extracting triplets/quadruples and predicting sentiment intensity. We first employ a span-based framework that integrates contextual representations and incorporates rotary position embedding. This approach fully considers the relational information of entire aspect and opinion terms, and enhancing the model’s understanding of the associations between tokens. Additionally, we utilize contrastive learning to predict sentiment intensities in the valence-arousal dimensions with greater precision. To improve the generalization ability of the model, additional datasets are used to assist training. Experiments have validated the effectiveness of our approach. In the official test results, our system ranked 2nd among the three subtasks. Our code is publicly available at https://github.com/tongzeliang/ SIGHAN2024.

heading: 1 Introduction
text: Aspect-Based Sentiment Analysis (ABSA) is an important task in Natural Language Processing (NLP), and is beneficial for many downstream tasks, such as emotional conversation generation (Wei et al., 2019; Liu et al., 2022) and recommendation system (Zhao et al., 2023; Wang et al., 2023). However, previous work has focused primarily on discrete sentiment polarity, with little attention given to the Valence-Arousal (VA) space. This dimensional approach represents affective states as continuous numerical values across multiple dimensions, providing more fine-grained sentiment information.
To address this issue, the SIGHAN-2024 shared task formulates three subtasks that challenge partic-
Corresponding Author
ipants to develop ABSA systems based on dimensional sentiment information (Lee et al., 2024). As Figure 1 shows, the three subtask can be illustrated as follows:
• Subtask 1: Intensity Prediction. Predict the valence-arousal ratings for a given sentence and its specific aspect. • Subtask 2: Triplet Extraction. Extract all sentiment triplets (aspect, opinion, intensity) from a given sentence. • Subtask 3: Quadruple Extraction. Extract all sentiment quadruples (aspect, category, opinion, intensity) from a given sentence.
As an extension of the ABSA task, dimABSA becomes notably challenging due to the following two difficulties: 1) Multiple Aspect-Opinion Pairing. In sentences with multiple aspects and opinions, determining which opinion corresponds
102
to which aspect becomes particularly challenging. To mitigate this problem, several efforts (Wu et al., 2020; Chen et al., 2022b; Liang et al., 2023) have been made, but they are not comprehensive for modeling the relationship between tokens, which is significant in the aspect-opinion pairing process (Chen et al., 2022a). 2) Imbalanced Dataset. As illustrated by Figure 2, the number of samples with neutral sentiment intensity is much greater than those with extreme sentiment intensity for both the valence and arousal dimensions, which leads to biased predictions from the model. Although there are some methods for addressing data imbalance like Re-Sampling (RS) (Zhou et al., 2020; Zhang and Pfister, 2021), most of them improve the performance of sparsely labeled samples at the expense of densely labeled samples (Zhang et al., 2023), leading to suboptimal results and and cannot be seamlessly migrated to dimABSA tasks.
In this paper, we develop a Contrastive Learning-Enhanced Span-based Framework for the dimABSA task to address the aforementioned challenges. Firstly, given the excellent performance of span-based methods in various NLP tasks (Xu et al., 2021), we explicitly generate span representations for all possible aspect and opinion spans. To comprehensively capture the relational information between spans, we integrate the contextual representations and incorporate Rotary Position Embedding (RoPE) (Su et al., 2024), which facilitates improved semantic understanding. Secondly, as self-supervised learning can improve robustness to data imbalance (Li et al., 2022), we employ contrastive learning to optimize feature representations in regression tasks. This approach adjusts the dis-
tance between samples in the embedding space according to their target values and subsequently leverages this feature to predict sentiment intensity.
Extensive experiments show that our method performs well across all three subtasks. On the official leaderboard, the Mean Absolute Error (MAE) for valence and arousal in subtask 1 ranks 2nd and 1st, respectively. The Pearson Correlation Coefficient (PCC) for valence and arousal in subtask 1 both rank 3rd. The F1 scores for triplet and quadruple extraction in Subtasks 2 and 3 both rank 2nd.
The paper is structured as follows: Section 2 provides a concise review of related work. In Section 3, we outline our proposed system. Section 4 covers the experimental details, including the dataset, setup, results, and discussions. Section 5 analyzes the effectiveness of contrastive learning and further examines the performance of our methods in low-resource settings. Finally, Section 6 presents a brief conclusion.

heading: 2 Related Work

heading: 2.1 ABSA Tasks
text: ABSA tasks, which aim to analyze sentiment from a fine-grained perspective, include three fundamental subtasks: Aspect Term Extraction (ATE) (Xu et al., 2018; Ma et al., 2019; Yang et al., 2020), Opinion Term Extraction (OTE) (Wan et al., 2020; Veyseh et al., 2020), and Aspect Sentiment Classification (ASC) (Tian et al., 2021; Wang et al., 2021a; Zhou et al., 2021). In recent years, research has increasingly focused on composite ABSA tasks, which integrate multiple basic tasks. Peng et al. (2020) introduced the Aspect Sentiment Triplet Extraction (ASTE) task, and they proposed a two-stage pipeline model to independently extract aspect-opinion-sentiment triplets. Subsequently, some end-to-end methods (Fei et al., 2021; Liang et al., 2023) were also applied to this task. Following this advancement, Zhang et al. (2021) introduced the Aspect-Sentiment Quad Prediction (ASQP) task, addressing it through the Seq2Seq generative modeling paradigm. However, these works primarily focus on discrete sentiment polarity, making it challenging to perceive subtle sentiment differences when predicting continuous sentiment intensity.

heading: 2.2 Contrastive Learning
text: Contrastive Learning methods learn feature representations by contrasting positive pairs against
negative pairs, and have widely used in many downstream tasks, such as recommendation systems (Zou et al., 2022; Wang et al., 2022), knowledge graphs (Fang et al., 2022; Xu et al., 2023), etc.. Recent research has started to utilize contrastive learning to address the long-tail distribution problem in image classification (Wang et al., 2021b; Xuan and Zhang, 2024), aiming to obtain improved feature representations. This prompts us to utilize contrastive learning in the dimABSA task to tackle the challenge of imbalanced datasets.

heading: 3 Methodology

heading: 3.1 Overview
text: Problem Statement. Let s = {wi}n and A = {aj}m be a sentence and a predefined set of aspects, where n and m represents the length of s and the number of aspects contained in s (A is only provided in subtask 1). The goal of subtask 1 is to predict the sentiment intensity valj , aroj ∈ [1, 9] for each aspect aj ∈ A. The object of subtask 2 and 3 is to extract a set of sentiment triplets T = {(a, o, val-aro)m} |T | m=1 and quadruples Q = {(a, o, c, val-aro)m} |Q| m=1, where a, o and c represent aspect term, opinion term and category. Architecture. As Figure 3 demonstrates, our system contains three components: 1) the AspectOpinion Pairing Module identifies aspect terms and opinion terms from the original sentence, and establishes their relationships to form valid aspectopinion pairs, 2) the Sentiment Scoring Module assesses the sentiment intensity based on the original sentence and the extracted aspect-opinion pairs, 3) the Category Prediction Module conducts category classification utilizing the original sentence and the extracted aspect-opinion pairs. Each module is trained independently, and each subtask is accomplished through the collaboration of different modules. This pipeline structure enhances the flexibility and scalability of the system, allowing different processing steps to be optimized and adjusted independently.

heading: 3.2 Aspect-Opinion Pairing Module
text: This module identifies relevant aspects and their corresponding opinions within the sentence and accurately pairs them. As Figure 4 shows, this foundational step is crucial for subsequent analysis and prediction, ensuring that each aspect is matched with its opinion, forming the basis for further inference.
Sentence Encoder. We use MacBERT (Cui et al., 2021) to generate contextual word representations by,
Ĥ = ĥcls, {ĥj}n, ĥsep = MacBERT({wi}n) (1) where ĥj is the contextual embedding of word wj . We then integrate RoPE into the token representation via an additional multi-head attention layer,
H = MultiHead (Q,K,V) (2)
= ||Zz=1Attention ( RiθW z qĤ, R j θW z kĤ,W z vĤ )
(3)
Z is the number of attention heads, W zq ,W z k and W zv are trainable parameter of the zth head of attention. Note that the rotational position encoding matrix should vary for different positions in the sequence, here we use Riθ and R j θ for simplicity. Aspect and Opinion Extraction. We use SP = {spi,j | 0 ≤ j − i ≤ l} to represent all possible spans in s, where i and j represent the start and end positions in s respectively, and the maximum length of span spi,j is l. We define the representation of span spi,j as,
spi,j = [hi;hj ] (4)
where the semicolon represents concatenation. Next, we employ a fully connected layer to evaluate the validity of each span spi,j , assigning a label distribution ye ∈ {Aspect, Opinion, Invalid},
pAi,j , p O i,j , p IV i,j = softmax ( Wespi,j + be ) (5)
where We, be are trainable parameters. Inspred by Xu et al. (2021), to mitigate the complexity inherent in the subsequent calculation process, we retain a specified proportion of spans for
both the aspect and the opinion candidate set, selecting those with the highest scores as determined by Equation 5. The refined sets of aspects and opinions can be denoted as AT = {. . . , spAi,j , . . .} and OT = {. . . , spOi,j , . . .}, respectively, each comprising nr elements, where r ∈ [0, 1] indicates the proportion of retained elements. Aspect-Opinion Pairing. After acquiring the aspect and opinion candidate sets from the previous stage, we proceed by pairing them in all possible combinations, resulting in the following representation,
fa,b,c,d = [ spAa,b; sp O c,d;Cb,c ] (6)
Cb,c =Max-Pooling ([hb+1 : hc−1]) (7)
Cb,c represents the contextual information of spa,b and spc,d. Subsequently, we employ a fully connected layer to process the representation of each fa,b,c,d. This layer evaluates the validity of each aspect-opinion pair, assigning a label distribution yg ∈ {V alid, Invalid}.
pVa,b,c,d, p IV a,b,c,d = softmax (Wgfa,b,c,d + bg) (8)
where Wg, bg are trainable parameters. Training. The training target is to minimize the cross-entropy loss of the extraction and pairing tasks.
L = αLe + βLg (9)
Le =− ∑
spi,j∈SP logP
( y∗e | pei,j ) (10)
Lg =− ∑
spa,b∈AT,spc,d∈OT logP
( y∗g | pga,b,c,d )
(11)
Here, y∗e and y ∗ g represents the ground-truth label of the extraction and pairing tasks for spi,j and fa,b,c,d, respectively.

heading: 3.3 Sentiment Scoring Module
text: In this section, we employ contrastive learning to enhance aspect-specific representations and predict sentiment intensity for each aspect in the valencearousal space, as illustrated in Figure 5.
Aspect Specific Representation. In this part, we first utilize MacBERT as encoder to generate aspect-specific representation for each aspect:
uj =< {wi}n, [SEP], {aj}t̂, [SEP], {oj}t̃ > (12)
Hj = MacBERT(uj) (13)
where t̂ and t̃ are lengths of the aspect aj and its corresponding opinion oj . Note that in the case of multi-aspect sentences, this module is employed multiple times, with each iteration focusing on one aspect. The aspect-specific feature representations is then obtained by max pooling,
vj = Max-Pooling (Hj) (14)
Contrastive Learning. After generating aspectspecific representations, most prior studies directly employ these representations for downstream tasks. Nonetheless, the performance is constrained by imbalanced datasets, resulting in suboptimal outcomes. To address this limitation, we incorporate contrastive learning to enhance feature optimization. Let {vi}G be defined as the set of all representations within a batch, and G denote the number of these representations, we first translate them for contrastive loss through a MLP combined with ℓ2-normalization,
ui = ℓ2-norm (MLP (vi)) (15)
In the absence of category labels, we establish two thresholds, δ1 and δ2, to facilitate the selection of positive and negative sample pairs respectively,
< i, j >= { + if |y∗i − y∗j | ≤ δ1 − if |y∗i − y∗j | ≥ δ2
(16)
where y∗i , y ∗ j represent the ground-truth of the sentiment intensity. Therefore, through the above rules, we can construct a positive set Pi and a negative set Ni for each representation ui. The contrastive loss is calculated as follows,
LCL = − 1
G
G∑
i=1
∑
uj∈Pi log
esim(ui,u + j )/τ
∑ uk∈Ni e sim(ui,u−k )/τ
(17) Training. The sentiment intensity was calculate by the aspect-specific representations {vi}G through a single linear layer, and the total loss can be calculated as follow,
L =αLR + (1− α)LCL (18)
LR = 1
G
G∑
i=1
||y∗i − fθ (vi) || (19)
where fθ denotes the linear projection. Note that the Sentiment Scoring Module is deployed twice within the system, with two identical components operating in parallel to independently extract valence and arousal features for regression prediction. This design allows each encoder to specialize in a specific emotional dimension, optimizing for the unique characteristics of each dimension and reducing feature interference during the contrastive learning process.

heading: 3.4 Category Prediction Module
text: This part employs the same method as the Sentiment Scoring Module to obtain aspect-specific representations vi. As Figure 6 shows, these representations are subsequently passed through a fully connected layer with a softmax activation function, producing probability distributions across all categories,
pi = softmax (Wpvi + bp) (20)
The training loss is formulated as the cross-entropy loss between the ground-truth and the predicted label distributions for all aspects,
LCE = − 1
N
N∑
i=1
log (y∗i | pi) (21)
where y∗ represent the ground-truth label.

heading: 3.5 Deployment Order
text: Table 1 illustrates the computational sequence of each component in the model across the three subtasks. All three subtasks necessitate an initial phase
of aspect-opinion extraction and pairing. In Subtask 1, the Opinion Retrieval (OR) Module is employed, meaning that during sentiment intensity regression, we retrieve the corresponding opinion extracted in the aspect-opinion pair module for each predefined aspect, as this is a critical feature for both valence and arousal predictions. In cases where extraction or pairing fails, “NULL” is used to fill the missing opinion term.

heading: 4 Experiment

heading: 4.1 Dataset and Setup
text: We evaluate our model on the official dataset of the SIGHAN-2024 shared task (Lee et al., 2024), which uses Simplified Chinese characters. Dataset statistics are shown in Table 2. To enhance the model’s ability to discern subtle sentiment nuances when predicting continuous sentiment intensity, we incorporated the Chinese EmoBank (EB) (Lee et al., 2022) as an auxiliary training resource. We fine-tuned the Sentiment Scoring Module on this supplementary dataset using the methodology outlined in Section 3.3, subsequently employing the fine-tuned parameters to initialize the model for the ensuing task training.
The Aspect-Opinion Pairing Module is trained for 30 epochs with a batch size of 16, and the other modules are trained for 10 epochs with a batch size of 128. AdamW optimizer (Loshchilov and Hutter, 2018) is adopted with a learning rate 2e-5 and weight decay 1e-2 for model training. The two thresholds δ1 and δ2 used in contrastive learning are set to 0.5 and 2 respectively. The maximum span length l is set as 10. We select the best model
weights for testing based on performance on the validation set. MAE and PCC are evaluation metrics for subtask 1, while the F1 score is used as the evaluation metric for subtasks 2 and 3.

heading: 4.2 Baseline
text: Since no existing method is specifically designed for dimABSA, we re-implemented Span-ASTE (Xu et al., 2021) and STAGE (Liang et al., 2023), which are high-performing span-based systems closely related to the task, and used them as our baseline.

heading: 4.3 Main Results
text: Table 3 presents the results of out method in the final test set. Observations are: 1) Our purposed model outperforms the baseline, and achieves relatively good results in the final rankings, with one metric ranking 1st, seven metrics ranking 2nd, and two metrics ranking 3rd. The performance improvement of our model primarily stems from a more powerful pre-training model, richer relational information for aspect-opinion pairing, and more robust feature representation for sentiment scoring. 2) Predicting sentiment intensities in the arousal dimension is significantly more challenging than in the valence dimension. In subtask 1, all models exhibit higher MAE in the arousal dimension compared to the valence dimension. In subtask 2 and 3, the F1 score based on arousal is about 5% lower than the F1 score based on valence. We infer that this complexity arises because predicting the level of arousal requires a comprehensive assessment of the overall context, tone, and other nuanced factors, which introduces corresponding challenges in the data annotation and training process.

heading: 4.4 Ablation Study
text: We also conduct an ablation study to verify the effectiveness of our proposed method. The results are shown in Table 3. Observations are: 1) For the Aspect-Opinion Pairing Module, w/o CR and w/o RoPE mean that we remove the contextual representation and rotational position embedding during the computation. Without the enhancement of relational features between spans and spans, the model’s performance slightly degrades. 2) For the Sentiment Scoring Module, w/o OR indicates that the opinion term has been removed from the input, and w/o CL indicates that the contrastive loss has been omitted during the training process. As
a result, the model’s performance drops dramatically, indicating that the opinion term is crucial for predicting sentiment intensity and that contrastive loss guides the model to obtain a more appropriate feature distribution when the dataset is imbalanced. 3) w/o EB indicates that the additional data from Chinese EmoBank was not used during training, resulting in deteriorated model performance. This verifies that Chinese EmoBank provides valuable supplementary information when the training data is insufficient. In summary, each module of our method significantly contributes to the overall performance on the dimABSA task.

heading: 5 Analysis

heading: 5.1 Effect of Contrastive Learning
text: To further verify the effectiveness of contrastive learning, we visualize the sample features with and without it, as shown in Figure 7. Models without contrastive loss struggle to capture the underlying continuous information in the data, resulting in fragmented and disordered representations. Conversely, features derived through contrastive learning preserve a coherent semantic structure, ensuring that semantically similar target values remain proximate in the feature space. Therefore, we infer that the improvement in effect comes from the neat and sequential feature representation brought by contrastive learning, which makes the feature space more discriminative and has stronger generalization ability in unknown data. At the same time, through contrastive learning, even if there are fewer samples with labels in certain intervals, the model will still learn the feature representa-
tion of these samples because they are frequently used for comparison during training. This method helps to balance the model’s attention to different labels, thereby alleviating the problem of imbalanced datasets.

heading: 5.2 Low-Resource Scenario
text: As a challenging task, dimABSA faces significant issues related to data scarcity. To address this, we investigated the impact of contrastive learning under various training data conditions. As depicted in Figure 8, the model utilizing contrastive learning consistently achieves lower MAE values, especially as the dataset size diminishes. Furthermore, the slower increase in MAE for the contrastive learning model indicates that contrastive learning enhances the model’s robustness and generalization capabilities, allowing it to maintain performance even under low-resource conditions.

heading: 5.3 Case Study
text: Figure 9 presents some case studies of this system, where aspect terms are highlighted in red and opinion terms in blue. Observations are: 1) In cases (a) and (b), the complete system achieved optimal results in the majority of sentiment intensity predictions. Notably, even for test data with sparse training data distribution, such as values like “7.62” and “2.17”, CL-Span consistently outperformed other methods, underscoring its robustness in accurately predicting less frequent valence and arousal values. 2) In case (c), our proposed CLSpan successfully pairs all aspect terms with their corresponding opinion terms. In contrast, SpanASTE fails to recognize the pair (“onion”, “caught
my eye”), and the STAGE model overlooks the pair (“onion”, “wasn’t pungent at all”). We attribute the superior performance of our model to the integration of contextual representations and RoPE, which enhances the semantic understanding and connectivity between aspect and opinion terms.

heading: 6 Conclusion
text: This paper describes our system for the dimABSA task. We develop a Contrastive Learning-Enhanced Span-based Framework, which integrates contextual representations and RoPE into feature representation to enhance semantic understanding. Additionally, we employ contrastive learning to optimize feature representations. Our system demonstrates significant effectiveness, achieving a 2nd place ranking across three subtasks.
Limitations
This section discusses some improvements that can be made in future work. 1) The pipeline model structure used in this study divides the processing steps into independent modules, allowing each
module to be developed, tested, and optimized separately. However, it also introduces the issue of error propagation, where errors in earlier stages can affect subsequent modules. In future work, we will focus on minimizing the impact of error propagation or consider testing an end-to-end model paradigm. 2) In the Sentiment Scoring Module, our system employs two MacBERT encoders to separately extract valence and arousal features for independent regression prediction. This approach reduces feature interference during the contrastive learning process and better captures the unique characteristics of each dimension. However, this results in the parameters of this module doubling to 204M. We will consider other encoding strategies instead of simply deploying two MacBERT separately.

heading: Acknowledgments
text: This work was supported in part by the National Natural Science Foundation of China under Grant No. 62276110, No. 62172039 and in part by the fund of Joint Laboratory of HUST and Pingan Property Casualty Research (HPL). The authors would also like to thank the anonymous reviewers for their comments on improving the quality of this paper.

heading: Future Work
text: Limitations
This section discusses some improvements that can be made in future work. 1) The pipeline model structure used in this study divides the processing steps into independent modules, allowing each
module to be developed, tested, and optimized separately. However, it also introduces the issue of error propagation, where errors in earlier stages can affect subsequent modules. In future work, we will focus on minimizing the impact of error propagation or consider testing an end-to-end model paradigm. 2) In the Sentiment Scoring Module, our system employs two MacBERT encoders to separately extract valence and arousal features for independent regression prediction. This approach reduces feature interference during the contrastive learning process and better captures the unique characteristics of each dimension. However, this results in the parameters of this module doubling to 204M. We will consider other encoding strategies instead of simply deploying two MacBERT separately