Overview of the SIGHAN 2024 Shared Task for Chinese Dimensional Aspect-Based Sentiment Analysis (2024)

Abstract: This paper describes the SIGHAN-2024 shared task for Chinese dimensional aspectbased sentiment analysis (ABSA), including task description, data preparation, performance metrics, and evaluation results. Compared to representing affective states as several discrete classes (i.e., sentiment polarity), the dimensional approach represents affective states as continuous numerical values (called sentiment intensity) in the valence-arousal space, providing more fine-grained affective states. Therefore, we organized a dimensional ABSA (shorted dimABSA) shared task, comprising three subtasks: 1) intensity prediction, 2) triplet extraction, and 3) quadruple extraction, receiving a total of 214 submissions from 61 registered participants during evaluation phase. A total of eleven teams provided selected submissions for each subtask and seven teams submitted technical reports for the subtasks. This shared task demonstrates current NLP techniques for dealing with Chinese dimensional ABSA. All data sets with gold standards and evaluation scripts used in this shared task are publicly available for future research.

text: Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10), pages 165–174 August 16, 2024 c©2024 Association for Computational Linguistics
This paper describes the SIGHAN-2024 shared task for Chinese dimensional aspectbased sentiment analysis (ABSA), including task description, data preparation, performance metrics, and evaluation results. Compared to representing affective states as several discrete classes (i.e., sentiment polarity), the dimensional approach represents affective states as continuous numerical values (called sentiment intensity) in the valence-arousal space, providing more fine-grained affective states. Therefore, we organized a dimensional ABSA (shorted dimABSA) shared task, comprising three subtasks: 1) intensity prediction, 2) triplet extraction, and 3) quadruple extraction, receiving a total of 214 submissions from 61 registered participants during evaluation phase. A total of eleven teams provided selected submissions for each subtask and seven teams submitted technical reports for the subtasks. This shared task demonstrates current NLP techniques for dealing with Chinese dimensional ABSA. All data sets with gold standards and evaluation scripts used in this shared task are publicly available for future research.

heading: 1 Introduction
text: Aspect-Based Sentiment Analysis (ABSA) (Pontiki et al., 2014; 2015; 2016) is a critical NLP research topic that aims to identify the aspects of a given entity and analyze the sentiment polarity associated with each aspect. In recent years, considerable research has been devoted to ABSA,
which can be categorized into different tasks based on the number of sentiment elements to be extracted. For example, the Aspect Sentiment Triplet Extraction (ASTE) task (Yuan et al., 2023; Chen et al., 2021; Mao et al., 2021; Peng et al., 2020; Wu et al., 2020; Xu et al., 2020; Zhang et al., 2020) extracts three elements in a triplet, including aspect/target term, opinion term and sentiment polarity (e.g., positive, neutral, and negative). Furthermore, the Aspect Sentiment Quadruple Prediction (ASQP) task (Cai et al., 2021; Gao et al., 2022; Mao et al., 2022; Peper and Wang, 2022; Zhang et al., 2021; Zhou et al., 2023) extracts the same three elements plus an additional aspect category to construct a quadruple.
However, compared to representing affective states as several discrete classes (i.e., sentiment polarity), the dimensional approach that represents affective states as continuous numerical values (called sentiment intensity) in multiple dimensions such as valence-arousal (VA) space (Russel, 1980), providing more fine-grained emotional information (Lee et al., 2022; Deng et al., 2022; 2023; Yu et al., 2016).
Therefore, we organized a Chinese dimensional ABSA shared task (dimABSA) in the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN 2024), providing finegrained sentiment intensity prediction for each extracted aspect of a restaurant review. We have three subtasks: 1) Intensity Prediction, 2) Triplet Extraction, and 3) Quadruple Extraction. Participants are free to participate in any or all subtasks. Given a sentence with/without aspects, participating systems should be able to extract the
165
sentiment elements with the corresponding valence-arousal rating values.
The rest of this article is organized as follows. Section 2 provides a description of the Chinese dimensional ABSA shared task. Section 3 introduces the evaluation data construction. Section 4 describes the performance metrics. Section 5 compares evaluation results from the various participating teams. Finally, we conclude this paper with findings and offer future research directions in Section 6.

heading: 2 Task Organization
text: This task aims to evaluate the capability of an automatic system for Chinese dimensional ABSA. The four sentiment elements are defined as follows:
• Aspect Term (shorted as A): This denotes an entity indicating the opinion target. If the aspect is omitted without being mentioned clearly, we use “NULL” to represent the term.
• Aspect Category (C) This represents a predefined category for the explicit aspect of the restaurant domain. We use the same categories defined in the SemEval-2016 Restaurant dataset (Pontiki et al., 2016). There are a total of twelve categories; each can be split into an entity and attribute using the symbol “#” as follows: 1) “餐 廳#概 括” / “餐 厅#概
括”(restaurant#general); 2) “餐廳#價格” / “餐厅
#价格” (restaurant#prices); 3) “餐廳#雜項” /
“餐厅#杂项” (restaurant#miscellaneous); 4) “食
物#價格” / 食物#价格 (food#prices); 5) “食物#品
質” / “食物#品质”(food#quality); 6) “食物#份
量 與 款 式” / “食 物#份 量 与 款 式 ”
(food#style&options); 7) “飲料#價格” / “饮料#价
格”(drinks#prices); 8) “飲料#品質” / “饮料#品
质”(drinks#quality); 9) “飲料#份量與款式” /
“饮料#份量与款式”(drinks#style&options); 10)
“氛圍#概括” / “氛围#概括” (ambience#general);

heading: 11) “服 務#概 括” / “服 务#概
text: 括”(services#general); and 12) “地點#概括” /
“地点#概括”(location#general).
• Opinion Term (O) This describes the sentiment words or phrases
towards the aspects.
• Sentiment Intensity (I) This reflects sentiments using continuous realvalued scores in the valence-arousal dimensions. The valence represents the degree of pleasant and unpleasant (i.e., positive and negative) feelings, while the arousal represents the degree of excitement and calm. Both the valence and arousal dimensions use a nine-degree scale. Value 1 on the valence and arousal dimensions respectively denotes extremely high-negative and low-arousal sentiment, while 9 denotes extremely high-positive
and high-arousal sentiment, and 5 denotes a neutral and medium-arousal sentiment. Valence-arousal values are separated by a hashtag (symbol “#”) for a mark.
This dimABSA task can be further divided into
three subtasks described as follows.
• Subtask 1: Intensity Prediction The first subtask focuses on predicting sentiment intensities in the valence-arousal dimensions. Given a sentence and a specific aspect, the system should predict the valence-arousal ratings. The input format consists of ID, sentence, and aspect. The output format consists of the ID and valencearousal predicted values that are separated with a 'space'. The intensity prediction is two real-valued scores rounded to two decimal places and separated by a hashtag, each respectively denoting the valence and arousal rating. Example sentences are presented in Table 1. In Example 1, a given sentence “檸檬醬也不會太油，塔皮對我而言稍
軟” (The lemon curd is not too oily and the tart
crust is a little soft for me.) and two aspects “檸檬
醬” (lemon curd) and “塔皮” (tart crust) as an input,
participating systems are expected to respectively predict valence-arousal ratings such as 5.67#5.50 for “檸檬醬” (lemon curd) and 4.83#5.00 for “塔
皮” (tart crust).
• Subtask 2: Triplet Extraction The second subtask aims to extract sentiment triplets composed of three elements. Given a sentence only, the system should extract all sentiment triplets (aspect, opinion, intensity). The output format consists of the ID and sentiment triplet that are separated with a 'space'. In Example 2, the input sentence is “不僅餐點美味上菜速度
也是飛快耶！！” (The meals were not only
delicious but were also served very quickly!!) and the output contains two tuples: the first triple contains “餐點” (meals) as an aspect term, “美味”
(delicious) as an opinion term, with valencearousal ratings as 6.63#4.63; the second triple consists of “上菜速度” (were served) as an aspect
term and “飛快” (very quickly) as an opinion term,
with valence-arousal ratings as 7.25#6.00.
• Subtask 3: Quadruple Extraction The third subtask aims to extract sentiment quadruples composed of four elements. Given a sentence only, the system should extract all sentiment quadruples (aspect, category, opinion, intensity). The output format consists of the ID and sentiment quadruple that are separated with a
'space'. In Example 3, if the input sentence is “這
碗拉麵超級無敵霹靂難吃” (This bowl of ramen
is terribly unpalatable.), the expected quadruple includes “拉麵” (ramen) denoted as the aspect
which belongs to an aspect category “食物#品質”
(food#quality), along with an opinion term “超級
無敵霹靂難吃” (terribly unpalatable) and a
sentiment intensity value in terms of valencearousal ratings of 2.00#7.88

heading: 3 Data Preparation
text: We first crawled restaurant reviews from Google Reviews and an online bulletin board system PTT. Then, we removed all HTML tags and multimedia material and split the remaining texts into several sentences. Finally, we randomly selected partial sentences to retain content diversity for manual annotation.
The annotation process was conducted in two phases. We first annotated the aspect/category/opinion elements and then V#A element. In the first phase, three graduate students majoring in computer science were trained to annotate the sentences for aspect/category/opinion. One task organizer led a discussion to clarify annotation differences and seek consensus among the annotators. A majority vote mechanism was finally used to resolve any disagreements among the annotators. In the second phase, each sentence along with the annotated aspect/category/opinion was presented to five annotators majoring in Chinese language for V#A rating. Similarly, one task organizer also led a group discussion during annotation. Once the annotation process was finished, a cleanup procedure was performed to remove outlier values which did not fall within 1.5 standard deviations (SD) of the mean. These outliers were then excluded from calculating the average V#A for each instance.
We provided two versions of all datasets with identical content, but one in traditional Chinese characters and the other in simplified Chinese characters. The participating teams could choose their preferred version for the task evaluation. The submitted results were evaluated with the corresponding version of the gold standard and ranked together as the official results.
This shared task is presented as an open test, and participating systems can use other publicly available data, but such data must be specified in the final system description paper. For example, we
also provide the Chinese EmoBank (Lee et al., 2022) as a potentially useful sentiment resource annotated with real-valued scores for both valence and arousal dimensions. This data set features various levels of text granularity including two lexicons called Chinese valence-arousal words (CVAW, 5,512 single words) and Chinese valencearousal phrases (CVAP, 2,998 multi-word phrases), along with two corpora called Chinese valencearousal sentences (CVAS, 2,582 single sentences) and Chinese valence-arousal texts (CVAT, 2,969 multi-sentence texts).
Table 1 presents detailed statistics for the mutually exclusive training, development and test sets, where #Sent, #Char, and #Tuple respectively denote the number of sentences, characters and tuples in the dataset. The training set provided for all three subtasks included 6,050 sentences (85,769 characters), annotated with 8,523 tuples. The development set only includes 100 sentences for output format validation. Two mutually exclusive test sets were prepared for system performance
evaluation, each including 2,000 sentences. One was provided for Subtask 1 and the other was used for Subtasks 2 and 3.
We further analyzed the aspect types in the test set, including #unique and #repeat which respectively denote the number of aspects which occurred only one time or more than one time. For Subtask 1, a total of 2,658 aspects belong to the unique type, without the null and repeat cases. For Subtasks 2 and 3, 1821 aspects (51.1% out of total 3,566) occurred more than one time across all testing sentences. In addition, a very small portion (near 1.5%) of aspects belonged to the null cases. Similarly, we also analyzed the opinion terms, the repeat cases occupied about 8.5% (=303/3566). These findings revealed: 1) the aspect has a centered distribution, reflecting that users’ opinion targets may be similar, and 2) the opinion has a diverse distribution, indicating that different affective words or phrases are used to express a user’s feelings.
Figure 1 shows the scatter plots of valencearousal distributions. They presented similar curves for the training and test sets, indicating that both high-positive and high-negative opinion terms usually have high arousal values. Identical results were obtained from the Chinese EmoBank (Lee et al., 2022).
Figure 2 presents the aspect category distributions. The distributions are imbalanced for both the training and test sets for Subtask 3. This finding is the same as that for the SemEval-2016 Restaurant dataset (Pontiki et al., 2016). The most frequently occurring category was “食 物 # 品
質”(food#quality), followed by “食物#份量與款
式 ”(food#style&options) and “飲 料#品 質”
(drinks#quality). In the training set, these 3 categories accounted for 87.4% of the total, with the remaining 9 categories accounting for 12.6%. In the test set for Subtask 3, these 3 categories accounted for 89.5% of the total, with the other 9 categories accounting for the remaining 10.5%.

heading: 4 Performance Metrics
text: For Subtask 1, the sentiment intensity prediction performance is evaluated by examining the difference between machine-predicted ratings and human-annotated ratings using two metrics: Mean Absolute Error (MAE) and Pearson Correlation Coefficient (PCC), defined as the following equations.
MAE = 1 𝑛 ∑ |𝑎𝑖 − 𝑝𝑖| 𝑛 𝑖=1 (1)
PCC = 1
𝑛−1 ∑ (
𝑎𝑖−𝜇𝐴
𝜎𝐴 )
𝑛 𝑖=1 ( 𝑝𝑖−𝜇𝑃 𝜎𝑃 ) (2)
where 𝑎𝑖 ∈ 𝐴 and 𝑝𝑖 ∈ 𝑃 respectively denote the i-th actual value and predicted value, n is the number of test samples, 𝜇𝐴 and 𝜎𝐴 respectively represent the mean value and the standard deviation of A, while 𝜇𝑃 and 𝜎𝑃 respectively represent the mean value and the standard deviation of P.
Each metric for the valence and arousal dimensions is calculated and ranked independently. The actual and predicted real values should range from 1 to 9, so MAE measures the error rate in a range where the lowest value is 0 and the highest value is 8. A lower MAE indicates more accurate prediction performance. The PCC is a value between −1 and 1 that measures the linear correlation between the actual and predicted values. A lower MAE and a higher PCC indicate more accurate prediction performance.
For Subtasks 2 and 3, we use the F1-score as the
evaluation metric, defined as:
𝐹1 = 2xPxR
P+R (3)
where Precision (P) is defined as the percentage of triplets/quadruples extracted by the system that are correct. Recall (R) is the percentage of triplets/quadruples present in the test set found by the system. The F1-score is the harmonic mean of precision and recall.
Each metric for the valence and arousal dimensions is calculated either independently or in combination. First, the valence and arousal values are rounded to an integer. Next, a triplet/quadruple is regarded as correct if and only if the three/four elements and their combination match those in the gold triplet/quadruple. All metrics range from 0 to 1. A higher Precision, Recall, and F1 score indicate more accurate performance.

heading: 5 Evaluation Results

heading: 5.1 System Summary
text: We received a total of 214 submissions from 61 registered participants during the evaluation phase. A total of eleven teams provided submissions to the
leaderboard for each subtask and seven submitted their task technical papers. HITSZ-HLT (Xu et al., 2024) and CCIIPLab (Tong and Wei, 2024) participated in all three subtasks, ZZ-NLP (Zhu et al., 2024) team took part in two subtasks, and the remaining four teams only joined in one subtask.
Table 3 summarizes the participating systems, including involved subtasks, system architectures and additional data usage. HITSZ-HLT (Xu et al., 2024) integrated a BERT-based pre-trained language model (PLM) (i.e., ERNIE 3.0 (Sun et al., 2021)) and a code-style large language model (LLM) (i.e., deepseek (Guo et al., 2024)) to address this task, demonstrating promising performance in different scenarios. CCIIPLab (Tong and Wei, 2024) proposed a Contrastive Learning-enhanced Span-
Subtask 1: Intensity Prediction
Team Evaluation Metrics Overall
Rank V-MAE V-PCC A-MAE A-PCC
HITSZ-HLT 0.279 (1) 0.933 (1) 0.309 (1) 0.777 (1) 1
CCIIPLab 0.294 (2) 0.916 (3) 0.309 (1) 0.766 (3) 2
YNU-HPCC 0.294 (2) 0.917 (2) 0.318 (3) 0.771 (2) 2
DS-Group 0.460 (4) 0.858 (5) 0.501 (4) 0.490 (4) 4
yangnan 1.032 (5) 0.877 (4) 1.095 (5) 0.097 (5) 5
Table 4: Testing results of Subtask 1. V for valence and A for arousal. The best scores of each metrics are in bold.
Team Subtask Architecture Data
Augmentation ST1 ST2 ST3 PLM LLM
HITSZ-HLT V V V Erine-3.0-xbase-zg deepseek-7B-
instruct-v1.5 -
CCIIPLab V V V MacBERT-base - Chinese EmoBank
YNU-HPCC V BERT-wwm-ext - Merged-Train
DS-Group V - GPT-4o -
TMAK-Plus V - GPT-4o -
ZZU-NLP V V BERT Baichuan2-7B -
JN-NLP V - T5-base -
Table 3: Participating system summary. ST for subtask, PLM for pre-trained language models, and LLM for large language models. Subtask 3: Quadruple Extraction
Team Evaluation Metrics Overall
Rank V-Quad-F1 A-Quad-F1 VA-Quad-F1
HITSZ-HLT 0.567 (1) 0.526 (1) 0.417 (1) 1
CCIIPLab 0.555 (2) 0.507 (2) 0.389 (2) 2
ZZU-NLP 0.522 (3) 0.489 (3) 0.376 (3) 3
SUDA-NLP 0.487 (4) 0.444 (4) 0.336 (4) 4
JN-NLP 0.482 (5) 0.439 (5) 0.331 (5) 5
BIT-NLP 0.470 (6) 0.434 (7) 0.329 (6) 6
USTC-IAT 0.438 (7) 0.437 (6) 0.312 (7) 7
Table 6: Testi g results of Subtask 3. V for valence, A for arousal, VA for valence-arousal, and Quad for quadruple. The best scores of each metric are in bold.
based (CL-Span) framework based on MacBERT (Cui et al., 2021) to improve the performance of tuple extraction and sentiment intensity prediction. The Chinese EmoBank (Lee et al., 2022) was also incorporated as an auxiliary training resource to boost performance. YNU-HPCC (Wang et al., 2024) used a BERT-based encoder to generate aspect-specific representation and train linear predictors to jointly predict valence-arousal ratings. DS-Group (Meng et al., 2024) proposed an aspectaware example selection method for in-context learning based on LLM. TMAK-Plus (Kang et al., 2024) presented a Multi-Agent Collaboration (MAC) model to assemble several GPT-based LLM for the dimensional ABSA task. ZZU-NLP (Zhu et al., 2024) proposed a two-stage contextual learning approach based on the Baichuan2-7B (Yang et al., 2023). JN-NLP (Jiang et al., 2024) used a paraphrase generation paradigm based on the T5 (Raffel et al., 2020) pre-trained model to address the dimABSA task.

heading: 5.2 Official Ranking
text: Tables 4, 5, and 6 respectively show the testing results for each subtask. Each metric in each individual subtask is ranked independently. (*) means the rank for each metric. A system’s overall ranking is computed based on the cumulative rank. The lower the cumulative rank, the better the system performance.
The overall best results came from the HITSZHLT (Xu et al., 2024) team, achieving the best scores in all metrics across three subtasks, followed by the CCIIPLab (Tong and Wei, 2024), ranking second on the leaderboard for each subtask.

heading: 6 Conclusions and Future Work
text: This paper provides an overview of the SIGHAN-2024 dimABSA task for Chinese dimensional aspect-based sentiment analysis, including task descriptions, data preparation, performance metrics and evaluation results. We received a total of 214 submissions from 61 registered participants during the evaluation phase. Among eleven participating teams, seven presented their task technical reports. Regardless of actual performance, all submissions contribute to the development of an effective dimensional ABSA solution, and each task technical paper for this shared task also provides useful insights for further research.
We hope the data sets collected and annotated for this shared task can facilitate and expedite future development of Chinese dimensional ABSA. Therefore, the gold standard test set and evaluation scripts are made publicly available in GitHub repositories at: https://github.com/NYCUNLP/SIGHAN2024-dimABSA
Future directions will focus on the development of Chinese dimensional ABSA models. We plan to build new language resources to develop techniques for the future enrichment of this research topic, especially for reviews in the other domains.

heading: Acknowledgments
text: We thank all the participants for taking part in our shared task. We appreciate Tzu-Mi Lin, Zue-Yu Xu, Jian-Yu Chou, and all annotators for their efforts in data annotations and verifications. This work is partially supported by the National Science and
Technology Council, Taiwan, under the grants NSTC 111-2628-E-A49-029-MY3 and NSTC 113- 2221-E-155-046-MY3. The works described in this paper are supported by the National Natural Science Foundation of China (62076158, 62376143), and CCF-Zhipu AI Large Model Foundation of China (CCF-Zhipu202310).

heading: Future Work
text: We hope the data sets collected and annotated for this shared task can facilitate and expedite future development of Chinese dimensional ABSA. Therefore, the gold standard test set and evaluation scripts are made publicly available in GitHub repositories at: https://github. com/NYCUNLP/SIGHAN2024-dimABSA
Future directions will focus on the development of Chinese dimensional ABSA models. We plan to build new language resources to develop techniques for the future enrichment of this research topic, especially for reviews in the other domains