Is ChatGPT a Good Sentiment Analyzer?
Authors: Zengzhi Wang
Qiming Xie
Yi Feng
Zixiang Ding
Zinong Yang
Rui Xia
Venue: COLM (2024)
Keywords: sentiment analysis, chatgpt, aspect-based sentiment analysis, emotion analysis, large language models
TL;DR: We evaluate the performance of ChatGPT on 7 sentiment analysis tasks (covering 17 benchmark datasets), compare it with fine-tuned small language models (e.g., BERT), and comprehensively analyse the sentiment analysis ability of ChatGPT.

Abstract: Recently, ChatGPT has drawn great attention from both the research community and the public. We are particularly interested in whether it can serve as a universal sentiment analyzer. To this end, in this work, we provide a comprehensive evaluation of ChatGPT on the understanding of \emph{opinions}, \emph{sentiments}, and \emph{emotions} contained in the text. Specifically, we evaluate it in three settings, including \emph{standard} evaluation, \emph{polarity shift} evaluation and \emph{open-domain} evaluation. We conduct an evaluation on 7 representative sentiment analysis tasks covering 17 benchmark datasets and compare ChatGPT with fine-tuned BERT and corresponding state-of-the-art (SOTA) models on them. We also attempt several popular prompting techniques to elicit the ability further. Moreover, we conduct human evaluation and present some qualitative case studies to gain a deep comprehension of its sentiment analysis capabilities.

Published as a conference paper at COLM 2024
Is ChatGPT Good Sentiment Analyzer?
Zengzhi Wang Qiming Xie Yi Feng Zixiang Ding Zinong Yang Rui Xia" School of Computer Science and Engineering, Nanjing University of Science and Technology, China zzwangênjust www.:juuutuuuueeeeemmooo qmxieenjust. edu. cn "xiaênjust.edu https://githube wwt.///////////aaA/TTTTT Sentiment-Evaluttiine
Abstract
Recently, ChatGPT has drawn great attention from both the research com- munity and the public. We are particularly interested in whether can serve as universal sentiment analyzer. To this end, in this work, we provide a comprehensive evaluation of ChatGPT on the understanding of opinions, sentiments, and emotions contained in the text. Specifically, we evaluate it in three settings, including standard evaluation, polarity shift evaluation and open-domain evaluation. We conduct evaluation 7 representative sentiment analysis tasks covering 17 benchmark datasets and compare ChatGPT with fine-tuned BERT and corresponding state-of-the-art (SOTA) models on them. We also attempt several popular prompting techniques elicit the ability further. Moreover, we conduct human evaluation and present some qualitative case studies to gain deep comprehension of its sentiment analysis capabilities.
1 Introduction
Large language models (LLMs) have profoundly affected the whole NLP community with their amazing zero-shot ability on various NLP tasks (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Zhang et al., 2022a, inter alia). As a representative, ChatGPT has appeared out of the blue via interacting with people conversationally. It can con- duct fluent conversations with people, write code as well as poetry, solve mathematical problems (Frieder et al., 2023) and so on, which has attracted widespread public attention.
However, despite its huge success, we still know little about the capability bound- aries, i.e,, where it does well and fails. In this work, we are interested in how Chat- GPT performs on the sentiment analysis tasks, i.e, can undersland the opinions, sentiments, and emotions contained in the text? To answer this question, we conduct comprehensive evaluation on 7 repre- sentative sentiment analysis tasks² and 17 benchmark datasets, which involves three different settings including slandard eval- uation, polarity shift evaluation and open- domain evaluation (refer Figure 1). We compare ChatGPT with fine-tuned small
3 Zero-shot Evaluation
3.1 Setup
3.3 Polarity Shift Evaluation
3.2 Standard Evaluation
8 3.4 Open Evaluation
Hunan Evaluation
5 4 Advanced Prompting Techniques
8 4.2
g 4.1 Few- shot Prompting
Chain-of Chain-of-Thoupe Thought & Self-Consistency
Table 1: The overview of our evaluation.
Corresponding Author
hhttps://chat... openai..ppe.cccmmmmm www.ai..coo///c/oooppppo
2hey are Sentiment Classification (SC), Aspect-Based Sentiment Classification (ABSC), End-to- End Aspect-Based Sentiment Analysis (E2E-ABSA), Comparative Sentences Identification (CSI), Comparative Flement Extraction (CEE), Emotion Cause Extraction (FCF), and Emotion-Cause Pair
Extraction (ECPF).

---

Published as a conference paper at COLM 2024
language models like BERT (Devlin et al., 2019) and corresponding SOTA models (if any) on each task for reference. We also attempt several popular prompting techniques, such as chain-off (CoT) (Wei et al., 2022) and self-comsistency (Wang et al., 2022), to induce the ability of ChatGPT. The main findings of our work are as follows:
ChatGPT demonstrates impressive zero-shot capabilities in sentiment classification tasks such as SC, ABSC and CSI, and can rival fine-tuned BERT, although it still trails behind the domain-specific fully-supervised SOTA models (r>r 5 3.2).
O ChatGPT seems less accurate on sentiment information extraction tasks like E2E- ABSA and CEE. We observe that ChatGPT can often make reasonable predictions but can not strictly match the dataset annotations. Our human evaluation finds that ChatGPT actually performs more desirable, not as poor as metrics indicate. (Ee $3.2)
3 Compared to sully-super strong baselines, ChatGPT demonstrates impressive textual emotion inference ability with significantly higher performance on ECE but lower performance on ECPE (Far $3.)).
When coping with the polarity shift phenomenon (e-g,, negation and speculation), a challenging problem in sentiment analysis, ChatGPT can make more accurate predictions than fine-tuned BERT. (var $3.3)
Compared to training domain-specific models, which typically perform poorly when generalized to unseen domains, ChatGPT demonstrates its powerful open- domain sentiment analysis ability in general, though its performance is quite limited in few specific domains. (ra 33.4)
Few-shot prompting (i.e., equipping with few random examples in the input) can significantly improve performance across tasks and domains, surpassing fine-tuned BERT in some cases, though still inferior to SOTA models (E Š $4.1). Applying CoT to the evaluated tasks does not yield gains but diminishes performance. In contrast, self-consistnncy reliably improves results (ta S 4.2).
In summary, we think that, compared to training specialized sentiment analysis system for each task, domain and dataset, ChatGPT can already serve as universal and well-behaved sentiment analyzer.
2 Background and Related Work
2.1 Large Language Models
With the emergence of GPT-3 (Brown et al., 2020), Large language models (LLMs) were spot- lighted. They typically have lots of model parameters and are trained massive volumes t
of unstructured data at huge computational costs, including but not limited to Gopher (Rae et al., 2021), Megatron-Turing NLG 530B (Smith et al., 2022), LaMDA (Thoppilan et al., 2022), Chinchilla (Hoffmann et al., 2022), PaLM (Chowdhery et al., 2022), OPT (Zhang et al., 2022a), LLaMA (Touvron al., 2023), and GPT-4 (OpenAI, 2023), As a result, given simple task instruction, they are able to adapt directly to new task in a training-free manner. In addition to the task instruction, the predictions will be more accurate and controllable if LLMs could be provided some demonstration examples, an ability known as in-context learning (Brown et al., 2020).
Lately, OpenAl has released ChatGPT, a chatbot fine-tuned from GPT-3.5 via reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022), drawing increasingly great attention. Next, researchers start exploring its abilities and limitations, testing it on various benchmarks (Gilson et al., 2022; Frieder et al., 2023; Guo et al., 2023; Jiao et al., 2023; Zhuo et al., 2023; Zhong et al., 2023; Ye et al., 2023; Laskar et al., 2023). For example, Bang et al. (2023) evaluate the multitask, multilingual, and multimodal aspects of ChatGPT, Wang et al. (2023a) conduct robustness evaluation from the adversarial and out-of-domain perspective, and Borji (2023) summarizes 11 categories of failures towards ChatGPT. Related to our work, Zhong et al. (2023) analyze the language understanding
2

---

Published as a conference paper at COLM 2024
ability of ChatGPT GLUE (Wang et al., 2018). In this work, we especially concentrate analyzing its sentiment analysis ability, aiming to answer the question via a rigorous and comprehensive evaluation, i.e, whether ChatGPT can be good sentiment analyzer.
2.2 Sentiment Analysis
Sentiment analysis seeks identify people's opinions, sentiments, and emotions in the text, such as customer reviews, social media posts, and news articles (Liu et al., 2005; Liu, 2015). As one of the most active fields in Natural Language Processing (NLP), it has made rapid progress with the help of deep learning (Zhang et al., 2018; Yadav & Vishwakarma, 2020).
Among the myriad of tasks associated with sentiment analysis, this paper is primarily concerned with 4 representative task categories, including (sentence-level) sentiment classi- fication (SC), aspect-based sentiment analysis (ABSA), comparative opinion mining (COM), and emotion cause analysis (ECA). For ease of understanding, these tasks next. SC aims to identify the sentiment polarity of a given text, whether it is positive or negative. ABSA is designed and determine the sentiment polarity toward each aspect (Liu, 2012; Pontiki et al., 2014; Zhang et al., 2022b). We mainly focus on aspect-based sentiment classification (ABSC) and End-to-End Aspect Based Sentiment Analysis (E2E-ABSA) among many subtasks in ABSA. COM seeks identify comparative sentences, extract the comparative elements, and obtain the corresponding comparative opinion tuples (Jindal & Liu, 2006; Liu et al., 2021). We mainly concentrate on comparative sentences identification (CSI) and comparative element extraction (CEE), i.e, extracting the tuple of (subject, object, comparative aspect, comparison type). The purpose of ECA is to extract the potential cause clauses given the emotion clause or extract the potential pair of emotion clause and cause clause in the text, which correspond to emotion cause extraction (ECE) (Gui et al., 2016) and emotion cause pair extraction (ECPE) (Xia & Ding, 2019), respectively.
will briefly introduce
mine fine-grained aspect terms in the review
In this paper, we are also concerned with two challenging problems in sentiment analysis: polarity shift and open-domain (Zong et al., 2021). Polarity shift refers to the linguistic phenomenon where the sentiment polarity (positive or negative) of a text shifts over time, context, or with respect to other texts (Li et al., 2010; Xia et al., 2016). Understanding sentiment polarity shift is crucial for building accurate sentiment analysis systems. As another challenging issue, open-domain sentiment analysis aims to understand the general sentiment of text regardless of the domain, whereas existing sentiment analysis systems typically focus on analyzing the sentiment texts related to a particular domain (Cambria et al., 2012; Zhang et al., 2015; Luo et al., 2022). Addressing the above two issues is essential to building robust and effective sentiment analysis systems. In this work, we will examine whether ChatGPT can solve them.
3 Evaluation
In this section, we will first introduce the evaluation setup (6 3.1) followed by standard evaluation (S 3.2), polarity shift evaluation (S 3.3) and open-domain evaluation (6 3.4), as illustrated in Figure 1. As mentioned earlier, the tasks involved in our evaluation are SC, ABSC, E2E-ABSA, CSI, CEE, ECE, and ECPE.
3.1 Setup
Comparison Systems. We compare ChatGPT with the state-of-the-art (SOTA) (if any) models on end-tasks. Since SOTA models typically have some task-specific designs, we also provide the results of a commonly used baseline (e.g., fine-tuned BERT*) on each task for reference. For SC, we adopt the most common practice i.e., using the final hidden representation of the [CLS] token as the sentence embedding and feeding it into linear layer for classification. As for ABSC, we concatenate the review sentence and the aspect term via the special token [SEP] and classify the sentiment polarity based on the final
Alll models use BERT WHRT-base-uncasee version and are coupled with linear layer if necessary.
3

---

Published as a conference paper at COLM 2024
hidden representation of [CLS]. We employ the joint tagging scheme (Li et al., 2019) to perform the E2E-ABSA task. For CSI, we report the performance of Multi-StageeaeRT derived from (Liu et al., 2021) for reference. For CEE, given the complexity of modeling this task, we reformulate it into a text generation task based on T5-Base similar to GAS (Zhang et al., 2021), i.e., predicting the sequences of comparison tuples given the input review. We employ PAE-DGL (Ding et al., 2019) and ECPE-2D (Ding et al., 2020) as comparison models for ECE and ECPE, respectively. Unless otherwise specified, the above baseline models are rerun and repeated three times based on our evaluation settings.
Usage of ChatGPT. We mainly use ChatGPT with specific version of gpt-3.5-turbo-0300 for evaluation in this work, given its lower cost and improved performance (as stated in the OpenAI documentation'). We set the temperature to 0, making the outputs mostly deterministic for the identical inputs. Following Jiao et al. (2023), we ask ChatGPT to generate the task instruction for each task to elicit its ability to the corresponding task. For example, the prompt for E2E-ABSA "Given a review, extract the aspect term(s) and determine their corresponding sentiment polarity. Review: (sentence)" Due to limited space, please refer to Table 9 and Appendix A.2 for complete prompts and prompts details, respectively. We report the zero-shot results of ChatGPT unless otherwise specified. We manually observe and record the predictions as the responses of ChatGI'T do not always follow certain pattern under the zero-shot setting.
Evaluation Metrics. We use accuracy and macro Fl score to evaluate sentiment classification tasks. We employ accuracy as the metric for CSI. For tasks involving elements extraction such as E2E-ABSA and CEE, we employ micro Fl score, i.e., a tuple is regarded as correct if and only if all elements inside it are exactly the same as the corresponding gold label. For ECE and ECPE, we compute the F1 score of cause clauses and emotion-cause clause pairs for evaluation, respectively.
Fine-tuned
Zero-shot
Task
Datasets
#Test
Metric
Baseline
SOTA
ChatGPT
SC
SST-2
872
Acc
95.47*
97.50*
93.12
14-Restaurant
1119
Aoc
83.94 77.85"
75.28* +
89.54 / 84.86 P
83.85 70.57
ABSC
14-Laptop
632
Acc /
73.20
83.70 / 80.13 7
76.42 / 66.79
14-Restaurant
496
F1
77.75 1
78.68 6
69.14
R2F-ABSA
14-Laptop
339
F1
66.05
70.32 6
49.11
CSI
Camera
661
F1
93.04
74.89
CEE
Camera
341
F1
34.41
9.10
BCE
Emotion Cause Dataset
100
F1
69.46 1
74.01
ECPE
Emotion Cause Dataset
100
F1
65.20
52.44
Table 2: Performance comparison among ChatGPT, fine-tuned baselines, and SOTA models on 9 datasets. #Test denotes the number of examples used for evaluation. + denotes the performance of fine-tuned BERT we implement. & and & denote the performance of PAE- DGL (Ding et al., 2019) and ECPE-2D (Ding et al., 2020) obtained by re-running experiments. S denotes the model performance of Multi-StageaERT derived from Liu et al. (2021) while represents the results of our implemented GAS-Extraction-syl. baseline (Zhang et al., 2021). a, B, 7, and 5 denote the results derived from T5-11B (Raffel et al., 2020), DPL (Zhang et al., 2022c), RILGNet (Li et al., 2022) and SyMux (Fei et al., 2022), respectively. The best results are in bold.
3.2 Standard Evaluation
In this part, we evaluate ChatGPT on representative sentiment analysis tasks and report its results on related benchmark datasets.
Datasets. We choose SST-2 (Socher et al., 2013) as the testbed of SC. Since the test set of SST-2 not public, we use its validation set for evaluation. We employ the SemEval

---

Published as a conference paper at COLM 2024
2014-ABSA Challenge Datasets (Pontiki et al., 2014) to evaluate the ability of ChatGPT to ABSA. For CSI and CEE, we employ the Camera dataset (Kessler & Kuhn, 2014; Liu et al., 2021). For ECE and ECPE, we adopt the Emotion Cause Dataset (Gui et al., 2016; Xia & Ding, 2019) and sample 100 examples from this. Except as noted above, we evaluate the remaining datasets on the full test set. The statistics are shown in the third column of Table 2.
The comparison results are shown in Table 2. Overall, ChatGPT demonstrates highly competitive sentiment analysis performance compared with baseline models, albeit often being far inferior to SOTA models. Specifically, we observe that ChatGPT is on par with fine-tuned small language models (i.e., BERT) in sentiment classification tasks, despite being inferior to SOTA models. Secondly, when evaluated on E2E-ABSA, the performance of ChatGPT indeed inferior to fine-tuned BERT, and the performance gap varies across domains. We speculate that the poorer performance on 14-Laptop is due to the presence of more proprietary terms and specific expressions in this domain. Thirdly, for the challenging COM tasks (i.e., CSI and CEE), which typically involve implicit expressions, although achieving reasonable performance on CSI, it exhibits extremely undesirable performance on CEE. These results are far from satisfactory compared with fine-tuned baselines. Finally, ChatGPT exhibits reasonably good emotion analysis ability. We find that ChatGPT can comprehend the given document thoroughly, for instance, being capable of identifying multiple reasons and extracting emotion clauses and cause clauses even when they are distant. We also observe that ChatGPT can make some reasonable predictions, whereas the corresponding annotations are not in the dataset.
Results.
Human Evaluation. In light of the poor performance on certain tasks, we naturally raise a question: are the predictions of Chat- GPT truly unreasonable? To acquire a more profound comprehension of the prediction results from ChatGPT, we conduct human evaluation on E2E-ABSA and CEE owing to their unsatisfactory performance. Upon observation of the predicted results, Chat- GPT has made many plausible predictions. However, these either did not exactly match the ground truth, or there are no corresponding annotations in the dataset, leading to subpar performance on the exact-match evaluation. For ABS,, even though the predictions of ChatGPT are not accurate based on exact- match evaluation, it can still infer some highly reasonable aspect categories for the aspect terms thanks to its text generation paradigm. This also demonstrates its ability to identify implicit expressions to some extent. For instance, given the sentence "Runs real quick." ", the ground truth is "(Runs, positive)" whereas the prediction of ChatGPT is "(Speed, positive)". For CEE, the predictions of ChatGPT express the same meaning as the ground truth but in an inconsistent form. As an example, the meaning expressed by ChatGiT is "The SD800 is better than the SD700.", whereas the ground truth meaning is "The SD700 is worse than the SD800.", where the "SD700" and "SD800" refer to the products being compared. From the perspective of sentiment analysis application, this is equally effective. Therefore, to align the predictions of ChatGPT with the annotation standard of existing datasets, we follow few simple rules for human evaluationS:
Task
Dataset
ChatGPT
+HumanEval
E2E-ABSA
14-Rest.
69.14
83.86
14-lap.
49.11
72.77
CEE
Camera
9.10
51.28
Table 3:
Human evaluation results on E2F- ABSA and CEE tasks.
RF
For any extra generated tuples, if they are reasonable but absent from the annota- tions, we will remove them from the prediction results. Otherwise, we will keep them.
Ear We also consider an aspect-sentiment comparative opinion tuple correct if the boundary of aspect or entity is predicted incorrectly but unambiguously, and the predicted sentiment or preference is also correct.
Eo We also regard prediction that paraphrases the ground truth to be correct, given the text generation paradigm.
Seee Appendix A.4 for examples
5

---

Published as a conference paper at COLM 2024
As shown in Table is surprising but reasonable to observe that the zero-shot zero-shotperformnnc of ChatGPT is boosted by 19% (average) and 42% on E2E-ABSA and CEE, respectively, compared to the original results. These results suggest that ChatGPT's predictions are often reasonable, even though they may not strictly adhere to the dataset annotations, leading to the appearance of poorer performance.
Case Study. We also conduct the qualitative analysis for the predictions of ChatGPT. Due to the limited space, please refer Appendix A.5.
3.3 Polarity Shift Evaluation
Comprehending the phenomenon of polarity shift in sentiment analysis is crucial for devel- oping robust and reliable sentiment analysis systems. In this part, we evaluate the ability of ChatGPT to cope with the polarity shift problem. Specifically, mainly focus the situations of negation and speculation and consider two sentiment classification tasks, SC and ABSC.
Datasets. Since there are few datasets tailored to polarity shift for SC, we derive two subsets from SST-2 validation set using a heuristic rule for the evaluation of negation and speculation, namely SST-2-Negation and SST-2-Speculation. In short, it entails identifying whether sentence contains any negation sentence to the negation evaluation subset if it includes the word "never". More details are provided in Appendix A.3. As for ABSC, we adopt the Negettnn 14-Lap-Negation, 14-Res- (iRes--pecullaion, and 14-Lap-Speculation introduced uced by Moore & Barnes (2021), which are annolated for negation and speculation, respectively. The statistics are shown in Table 10.
speculation words. For instance, we assign
Baseline Details. Generally, we fine-tune BERT on the original training set (e.g., SST-2) and evaluate on polarity-shiftin test sets, e.g,, SST-2-Negation and SST-2-Speculatio.
Results. We conduct experiments on six evaluation datasets, and the comparison results are shown in Ta- ble 4. Compared to fine-tuned BERT, ChatGPT exhibits greater robustness in polarity shift scenarios. Essentially speaking, the polarity shift evaluation we conduct can be characterized as an oul-of-diittrbuution (OOD) evalua- tion scenario. Not surprisingly, we observe that fine-tuned BERT expe- riences varying degrees of perfor- mance degradation across datasets compared to standard evaluation re- sults. In comparison, ChatGPT is more robust, especially on ABSC, where ChatGPT outperforms fine- tuned BERT by 10% in terms of av- erage accuracy and 8% in terms of average F1 score. Furthermore, we also find that the speculation case in polarity shift appears more challenging than the negation case, the results of the former is poorer.
Shifting Type
Fine-tuned
Zero-shot
Task
Dalaset
BERT
ChatGPT
Negation
SST-2-Neg.
90.68
SC
Speculation
SST-2-Spec.
92.05
92.05
70.93
79.66
14-Res-Neg.
Negation
61.90
69.12
60.25
72.73
14-Lap-Neg.
53.97
67.27
ABSC
64.29
77.01
14-Res-Spec.
60.53
68.45
Speculation
40.86
47.47
14-Lap-Spec.
39.40
46.96
Table Performance comparison between ChatGPT and BERT on six datasets when dealing with nega- tion and Speculation linguistic phenomena, mea- sured by accuracy (top) and macro F1 score (bottom). The best results are bold.
Case Study. We conduct qualitative analysis for the predictions of ChatGPT in the case of polarity shift. Refer to Appendix A.6 for details.
3.4 Open Domain Evaluation
Existing systems typically trained on specific domains or datasets, leading to subopti- mal generalization performance when dealing with unseen domains. However, an ideal sentiment analysis system could be applied to data from diverse domains. In this part,

---

Published as a conference paper at COLM 2024
Model
Metric
Rest.
Lap.
Books
Cloth.
Hotel
Device
Service
Twitter
Finance
METS
Ave.
Fine-tuned on the Rest, domini
Acc.
81.11
77.78
57.78
74.44
86.67
86.67
71.11
62.22
75.56
53.33
72.67
BERT
F1
74.99
70.60
41.91
55.00
77.59
85.35
67.91
54.11
62.75
47.06
61.14
Fine-tuned on the ap. domatn
BERT
Acc. 84.44
77.78
57.78
76.67
86.67
86.67
71.11
62.22
74.44
50.00
72.78
F1
78.76
72.84
42.84
56.21
76.94
88.92
67.59
56.16
55.59
37.56
60.78
Fine-tuned on the 9 domaiin each time
BERT
Acc.
80.00
76.67
62.22
76.67
85.56
94.44
81.11
70.00
31.11
38.89
69.67
F1 69.63
59.83
46.11
61.66
75,34
98.11
79.29
67.83
31,58
35.65
39,99
supoisse results
Acc.
81.11
77.78
80.00
87.78
100.00
74.44
62.22
82.22
61.11
77.78
BERT
F1
74.99
72.84
57.17
58.15
77.98
100.00
62.69
60.99
79.07
58.53
67.64
Zero-shot results
ChatGPT
83.33
70.00
96.67
77.67
FI 61.16
53,41
51.25
59.65
83.18
98.89
65,30
64.22
72.35
55.56
66.50
Table 5: Performance comparison between ChatGPT and fine-tuned BERT for ABSC task open-domain evaluation. We also report the domain-specific fully-supervised results (in gray) of BERT for reference. The best results (except for sully--up results) are in bold.
Model
Rest.
Lap.
Books
Cloth.
Hotel
Device
Service
Twitter
Finance
Mets-Cov
Ave.
Fine-timed on the Rest. domain
BERT
76.55
43.57
38.35
29.57
64.07
50.74
27.01
1.67
7.74
3.27
34.25
Fine-tuned the Lap, domain
BERT 55.06
68.02
25.93
26.28
53.21
60.19
27.03
3.43
7.11
5.14
33.14
Fine-funed the out-of-domoins each time
BERT 71.10 59.36
46.64
50.72
74.85
58.87
47.67
42.90
14.21
10.27
47.66
Fully-suprised resadts
BERT 76.55
68.02
61.17
67.97
88.67
75.39
57.83
78.84
79.32
71.71
72.55
Zena-soot resuts
ChaiGPT 72.73
45.45
21.92
25.71
50.60
41.86
45.78
19.18
38.36
3.92
36.55
Human 82.22
64.00
29.41
34.78
62.5
69.23
63.89
52.63
76.92
9.88
54.55
Table 6: Performance comparison between ChatGPT and BERT for E2E-ABSA task on the open-domain evaluation. We report the domain-specific fully-supervised results (in gray) of BERT for reference. We also report the human evaluation results ("+ Human") of ChatGPT for reference. The best results (except for fully-supervised results and human evaluation results) are in bold.
we evaluate the capability of ChatGPT to handle open-domain sentiment analysis tasks (i.e., ABSC and E2E-ABSA).
Datasets. As there is currently no widely used open-domain evaluation dataset, we sample 30 examples from each domain of existing 10 ABSA datasets according to the original data distribution, resulting in a total of 300 samples both for ABSC and E2F--ABSA. The ten datasets involved are Restaurant (Pontiki al., 2014), Laptop (Pontiki al., 2014), De- vice (Hu & Liu, 2004), Service (Toprak et al., 2010), Books, Clothing, Hotel (Luo et al., 2022), Twitter (Dong et al., 2014), Financial News Headlines (Sinha et al., 2022), METS-CoV (Zhou et al., 2022), covering various domains such as restaurant reviews, product reviews, social media, finance, and medicine. Note that Books, Hotel, and Clothing are originally document- level ABSA datasets with hierarchical seily--speet-senttmeet annotations. We randomly sample 30 sentences from each dataset and only use the aspect-sentiment annotations.
Baseline Details. To simulate the open-domain setting, we hold out some datasets, fine- tune BERT on the remaining datasets, and select checkpoints based on the mixture of the corresponding validation sets. Specifically, we set the following settings: (1) single-source: the model is trained on one dataset then evaluated on all datasets. Here, we choose Restaurant and Laptop the testbed; (2) multi-source: the model is trained sequentially nine datasets and then evaluated on the remaining one. Finally, we also fully-supervisedll fine-ture BERT and report the results for reference.
7

---

Published as a conference paper at COLM 2024
Results. In terms of ABSC, ChatGPT demonstrates than BERT despite being fine-tuned on this task. As shown in Table 5, ChatGPT matches or even outperforms multi-domain fine-tuned BERT on 7 out of 10 domains in sentiment classification metrics (accuracy or macro-F1) while surpassing it by 8% in accuracy and 7% in F1 score on average across 10 datasets. It is worth mentioning that ChatGPT even performs comparably to full-supervised BERT, which shows its compelling generalization ability. Interestingly, fine-tuning on multiple domains does not necessarily lead improved performance. For example, we observe that it results in a significant decrease in performance in certain datasets such Finance and METS-Cov. Table shows ChatGPT exhibits moder- ate performance on E2E-ABSA under the exact-match evaluation despite in the zero-shot manner. For example, it even beat BERT models on some domains (e.g., restaurant, service, and finance), which are fine-tuned on the nine domains.
compelling open-domain ability
Despite its success, we can observe that the performance of ChatGPT is quite poor in some domains, especially social media relevant domains (i.e., twitter, finance, METS-Cov), which suggests that improving performance on these domains remains challenging. should be noted that due to the use of exact-match evaluation, the actual results of ChatGPT may not be as poor as they appear. Similarly, through our human evaluation (as introduced in $ 3.2), can observe that ChatGPT has achieved an average performance improvement of 18% across domains, indicating that many predictions are inherently reasonable. Overall, these results can demonstrate decent open-domain capabilities of ChatGPT, albeit with poor results in a few domains.
Case Study. We conduct qualitative analysis through four examples of ChatGPT Books and METS-Cov, corresponding to the books and medicine domain, as shown in Figure 5. We also provided a detailed analysis in Appendix A.7.
Advanced Prompting Techniques
Given that ChatGPT still lags be- hind fine-tuned small language mod- els (e.g, BERT) in some tasks and domains to certain extent, we en- deavor to seek help from some ad- vanced prompting techniques fur- ther elicit the capabilities of ChatGPT. Here, we adopt the ABSA tasks as the testbed.
ABSC on 14Res.
ABSC on 14Lap.
85
80
80
75
75
70
70
65
3
9
27
0
9
27
E2E-ABSA on 14Res.
E2E-ABSA on 14Lap.
4.1 Few-shot Prompting
80
70
F1 (%)
65
We randomly select few examples from the training dataset used for demonstration and concatenate them
75
60
55
70
with the target input to prompt Chat-
GPT, a technique also known as in- context learning (Brown et al., 2020). We conduct few-shot prompting ex- periments on ABSC and ASPE with k (i.e., 1, 3, 9 and 27) examples. To re- duce the variance caused by the sam- pling of demonstration examples, we adopt three random seeds for sam- pling to conduct experiments and re- port the average performance. We compare the resulting performance with supervsd BERT and SOTA.
0
3
9
27
0
1
9
27
Shot
Shot
ChatGPT
Fine-tuned BERT
SOTA
Table Few-shot prompting results on ABSC and E2E-ABSA tasks.
Results. As presented in Figure 7, few-shot prompting can significantly improve the
performance across tasks and datasets, even surpassing fine-tuned BERT in some cases. It
8

---

Published as a conference paper at COLM 2024
improves the classification performance by 7% and 10% Fl score for ABSC 14-Restaurant and 14-Laptop, respectively, with 27 demonstration examples. We can also observe certain improvements ASPE, although the improvement curve is relatively flat. We also provide a case study, as shown in Figure
4.2 Chain-of-Thought and Self-Consistency
Although few-shot prompting clearly improves the performance on ABSC, the performance on E2E- ABSA still lags far behind fine-tuned BERT. We attempt more advanced techniques, ie., manual few-shot few-shott chain- of-thought (CoT) prompting (Wei et al., 2022) and self-consistency (Wang et al., 2022) on this task, to further elicit the ability. More details are provided in the Appendix A.8.
Prompting Methods
14-Res.
14-Lap.
Zero-shot prompting
69.14
49.11
Few-shot prompting (3 shot)
69.70
52.90
Few-shot prompting (9 shot)
70.37
56.43
Few-shot prompting (3 shot) + Col
67.24
46.28
Few-shot prompting (9 shot) + CoT
64.98
50.19
3-shot Self-Consist. (N = 5)
72.51
53.45
3-shot Self-Consist. (N = 10)
72.87
54.22
3-shot Self-Consist.
15)
73.22
55.01
3-shot CoT Self-Consist. (N 5)
69.12
48.73
Results. As shown Table 8, observe that equipping standard few-shot prompting with chain-of- thought does not bring the expected gains, but rather lead to a noticeable drop. This similar phenomenon was also observed in Ye & Durrett (2022) and Wang al. (2022) but contrary to the observations in Zhong et al. (2023). We speculate that this may depend on the evaluation tasks. In contrast, self-consistency clearly improves the perfor- mance of few-shot prompting, regardless of whether CoT is equipped, once again confirming the effectiveness of this technique (albeit at the cost of increased inference complexity). Re- grettably, while effective, it is still inferior to fine-tuned BERT. Future work could explore more efficient prompting methods, such retrieval-based ones (Liu et al., 2022; Shi et al., 2023, inter alia).
3-shot CoT Self-Consist. (N 10)
69.17
49.17
3-shot + CoT Self-Consist. (N 15)
70.39
49.77
Fine-tuned BERT
77.75
66.05
Table Results of advanced prompting techniques on E2E-ABSA. N denotes the number of outputs sampled for the same input in the self-consisenccy technique.
5 Conclusion
In this work, we evaluate ChatGPT on range of test sets and evaluation scenarios and compare its performance to fine-tuned BERT, exploring its capacity boundaries vari- ous sentiment analysis tasks. ChatGPT exhibits magnificent zero-shot sentiment analysis abilities (e-g,, sentiment classification, comparative opinion mining and emotion cause analysis), even matching with fine-tuned BERT and SOTA models trained with labeled data in respective domains at times. Compared to fine-tuned BERT, ChatGPT can handle the polarity shift problem more effectively sentiment analysis and exhibits good performance in open-domain scenarios. In addition, we also explore some popular prompting techniques to further induce the capability of ChatGPT. Through experiments, we validate the effective- ness of them on sentiment analysis tasks and provide our findings. We aspire to galvanize future research through our empirical insights in sentiment analysis, LLMs and beyond.
This work has several limitations: (1) Data Leakage: Difficulty in ensuring ChatGPT hasn't tt seen test data (Dodge et al., 2021; Golchin Surdeanu, 2024; Balloccu et al., 2024; Xu al., 2024; Sainz et al., 2024). (2) Prompt Design: Lack of extensive prompt experimentation assumes users prefer simplicity, yet improvements could showcase ChatGPTTss adaptability. (3) Limited Evaluation: Focus on ChatGPT with limited comparison to other models due to resource constraints, narrowing insights into LLM strengths and weaknesses. Looking ahead, promising directions include (1) New Benchmarks: Developing real-world bench- marks and better evaluation methods to accurately assess LLMs (Zheng et al., 2023; Huang et al., 2024; Jimenez et al., 2024, Xie et al., 2024). (2) Implicit Sentiment Analysis: Enhanciig

---

Published as a conference paper at COLM 2024
understanding and analysis of texts with implicit sentiments through comprehensive bench- marks. (3) Domain-Speciffc Enhancement: Improving ChatGPT's performance in specific domains, such as medicine (Singhal et al., 2023), math (Wang et al., 2023b; Azerbayev et al., 2024), code (Rozière et al., 2023) and low-resource languages (Dou et al., 2024; Nguyen et al., 2024), with targeted training for broader applicabilityy Due to space constraints, detailed discussions on limitations and future work directions are available in Appendix B.
References
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2024. URL
nwt/forumm 1dw.umm299999991555555
Simone Balloccu, Patricia Schmidtová, Mateusz Lango, and Ondrej Dusek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), PP. 67- 93, St. Julian's, s, Malta, March 2024. Association for Computational Linguistics. URL eacl-long. 5.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. Á multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, doi: 1044550//aXXiv2230.00022 URL
and interactivity. CoRR, abs/2302.04003, 2023.
https://doi. org/10. 48550/ aarov.. ptt.v.23..0444 04023.
Ali Borji. A categorical archive of chatgpt failures. CoRR, abs /b3020204,,444 2023. doi: 10.48550/ 20555000/2Xi..2220349444 URL https://doi http::///ooooog////// ////1/0.8555//rxxvvv. www..3302.33999.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https::/proceediiiiii newriie httpii/nnnerrrissscpppp//////22 http://4477766666644444444118383
44f6444 html.
Erik Cambria, Yangqiu Song, Haixun Wang, and Newton Howard.
Semantic multidimen- sional scaling for open-domain sentiment analysis. IEEE intelligent systems,, 29(2):44-5, 2012.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Hen- ryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiri- donov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstemn, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language model- ing with pathways. CoRR, abs/ 2204.02311, 2022. doi: 10.48550/ 2204..22004.2222. URL https://doi. org/10. 48550/ Xiv 2204.
10

---

Published as a conference paper at COLM 2024
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Con- ference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4299-4307, 2017. URL ht / /pwp://e//oocceessseoooe neur ips cc/ wwtpaccp/ppppccccccccccccceeeeet pappp/2p11111111 /2017/22//77 //hash// of wbstecet.bbbrrtttt...
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, Minneapllis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N199 URL https://aclantthloog. hww.tthooooooooooooooooo Oogg/N19-1440
Zixiang Ding, Huihui He, Mengran Zhang, and Rui Xia. From independent prediction to reordered prediction: Integrating relative position and global label information to emotion cause identification. In The Thirty-Third AAAT Conference on Artificial Intelligence, AAAI 2019, The Thirty-Firsl Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawnii, USA, January 27 February 1, 2019, pp. 6343 AAAI Press, 2019. doi: 10.1609 waaai..33i01... 033101133313343
URL https: https://dooi.oor wrgorrg////// 609/aaai. v33101 33016343.
Zixiang Ding, Rui Xia, and Jianfei Yu. ECPE-2D: emotion-cause pair extraction based on joint two-dimensionnal representation, interaction and prediction. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pp. 3161-3170. Association for Computational Linguistics, 2020. doi: 10.18653 /v1 wtt//20.acmmmin..a.. 2888. URL https:://on org/10. 18653/v1/2020 18653//1/2220.aal-maiii acl-main.. 288.
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groen- eveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: case study on the colossal clean crawled corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v 4021.emnlp-man mainn99. URL https: http::::///clanhhooooyy org/2021 momnp--miin..
Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. Adaptive recursive neural network for target-dependent Twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 49-54, Baltimore, Maryland, 2014. Association for Computational Linguistics. doi: 1L.1115v1/P14-201 URL https: https//////alaahlool//o org/p14-22000
Longxu Dou, Qian Liu, Guangtao Zeng. Jia Guo, Jiahui Zhou, Wei Lu, and Min Lin. Sailor: Open language models for south-east asia. CoRR, abs/ 24044003448, 2024. doi: 10.48550/ ARXIV.2404.03608. URL
48554445551i11 2404 03608.
Hao Fei, Fei Li, Chenliang Li, Shengqiong Wu, Jingye Li, and Donghong Ji. Inheriting the wisdom of predecessors: A multiplex cascade framework for unified aspect-based sentiment analysis. In Luc De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 4121-4128. ijcai.org, 2022. doi: 10.24463 11224963/iccii.2222 2022 572. URL https: http://oooo.oo//////// org/10. 24963/
Simon Frieder, Luca Pinchetti, Rhan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. Mathemati- cal capabilities of chatgpt. CoRR, 5bbs/2301, 13867, 2023. doi: 10.48550/ 10..4850//axxv.222255 art.aa231.v..113.53 138677 URL https://doi. wwtp//...550/////// 48550/arXiv. 2301 13867.
11

---

Published as a conference paper at COLM 2024
Aidan Gilson, Conrad Safranek, Thomas Huang, Vimig Socrates, Ling Chi, R. Andrew Taylor, and David Chartash. How does chatgpt perform on the medical licensing ex- ams? the implications of large language models for medical education and knowl- edge assessment. medRxie, 2022. doi: 10.1101 2t..22.......2222..1.. 2322839911 12. 12..2....31....2222391111
URL https:
Shahriar Golchin and Mihai Surdeanu. Time travel in lIms: Tracing data contamina- tion in large language models. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
https://openreviae
Lin Gui, Dongyin Wu, Ruifeng Xu, Qin Lu, and Yu Zhou. Event-driven emotion cause extraction with corpus construction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1639-1649, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653/vil /0/3//11D16111/// URL httts:://cllatthlll. org/D16-1170.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. CoRR, abs/2301 .07597, 2023. doi: (0.448500/arXvv220000 .07597. URL httpttt//doi..raa org/10 48550/arXiv 2301 .07597.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, abs/2203. 15556, 2022. doi: 10.48550/ arXiv.2203.15556 URL https: //doi. http//do..rgg/////// 0rg/10.445500 arXiv.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Won Kim, Ron Kohavi, Johannes Gehrke, and William DuMouchel (eds.), Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, Washington, USA, August 22-25, 2004, pp. 168-177. ACM, 2004. doi: 10.1145/1014052. 1014073. URL https: org/10. 10 11 1145/10111 4052. 1014073.
Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, Yikai Zhang, Yuging Yang, Ting Wu, Binjie Wang, Shichao Sun, Yang Xiao, Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yuxiang Zheng, Shaoting Zhang, Dahua Lin, Yu Qiao, and Pengfei Liu. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. arXiv preprint arXio:2406.1277 127553...1........ 2024. URL https://arxivi org htrrg/abs/2066 12753.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. chatgpt A good translator? A preliminary study. CoRR, abs 2301100757 2023. doi: 10.48550/ arxi.. 2301.08745. URL https::/doi..rg/// 48550/ arriv.. 2301. 08745.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openrev ew. net mettforummidd-TFayyy/eee
Nitin Jindal and Bing Liu. Mining comparative sentences and relations. In Proceedings, The Tiwetty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, July 16-20, 2006, Boston, Massachusetts, USA, pp. 1331-1336. AAAI Press, 2006. URL http: aaai org/Librarr/AAtt/206 aaai06- 06-009000000000 php.
Wiltrud Kessler and Jonas Kuhn. A corpus of comparisons in product reviews. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)) pp. 2242-202 2228, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL http:// www.:/ooreooooooeooo wwwponf.oggg//oooooeoo http:///o/oooooiin//////////////
Paper, pdf..
12

---

Published as a conference paper at COLM 2024
Md. Tahmid Rahman Laskar, M. Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. A systematic study and comprehensive evaluation evaluuaooo of chatgpt benchmark datasets. CoRR, abs/ abs/2305 2305.18486, 2023, doi: 10.48550/ art0///333X0.3 18486. URL https: https://doi.org//
org/10 1/gg/.0.00 488550/arXiv.... 2305. 18486.
Longfeng Li, Haifeng Sun, Qi Qi, Jingyu Wang, Jing Wang, and Jianxin Liao. Modeling aspect correlation for aspect-based sentiment analysis via recurrent inverse learning guidance. In Proceedings of the 29th International Conference on Compulationnl Linguistics, pp. 6887- 6896, Gyeongju, Republic of Korea, 2022. International Committee on Computational Linguistics. URL https: / httts:://allanthllog// org/2022. coling-1 1.599.
Shoushan Li, Sophia Y. M. Lee, Ying Chen, Chu-Ren Huang, and Guodong Zhou. Sentiment classification and polarity shifting. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pp. 645-643, Beijing, China, 2010. Coling 2010 Organizing Committee. URL https: /aclan/hoocl org/C10-1072.
Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam. Exploiting BERT for end-to-end aspect- based sentiment analysis. Proceedings of the 5th Workshop on Noisy User-generated Text (W- NUT 2019), pp. 34-41, Hong Kong, China, November 2019. Association for Computational URL https: ac wcw.anhologgyy....ooooo org/D19 5505.
Linguistics. doi: 10.18653/ 21/D19-5505.
Zhengyan Li, Yicheng Zou, Chong Zhang, Qi Zhang, and Zhongyu Wei. Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, PP. 146,256, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: haacllntholooy.orrgg/oo org/2021 emnlp-main. 22.
URL https:
Bing Liu. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1-167, 2012.
Bing Liu. Sentiment Analysis Mining Opinions, Sentiments, and Emotions. Cambridge Univer- http:://ww.ccmmrr.iggg
sity Press, 2015. ISBN 978-1-10-701789-11 URL //www. cambri org/us/acaaemii
Bing Liu, Minqing Hu, and Junsheng Cheng. Opinion observer: analyzing and comparing opinions on the web. In Allan Ellis and Tatsuya Hagino (eds.), Proceedings of the 14th international conference on World Wide Web, WWW 2005, Chiba, Japan, May 10-14, 2005, pp. 342-351. ACM, 2005. doi: 10.1145/1060445.066777 1060745. 1060797.
URL https://doi. wrw..0..g//........ 1145/
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Inte- gration for Deep Learning Architeclures, pp. 100-114, Dublin, Ireland and Online, 2022. Association for Computational Linguistics. doi: 10.1853//1/2022deeloolll URL https://aclanthologgycoo org/ de
Ziheng Liu, Rui Xia, and Jianfei Yu. Comparative opinion quintuple extraction from product reviews. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3955-3965, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. https://aclantthooogs wrgg wor..//...........mmm... 1annlpmmain.22...... 322.
URL
Yun Luo, Hongüe Cai, Linyi Yang, Yanxia Qin, Rui Xia, and Yue Zhang. Challenges for open-domain targeted sentiment analysis. CoRR, abs 0ass/2204.0688 2022. doi: 10.48550/ arXiv:2204.068993 URL https: //doi. org/10. 48550/ 2204. 06893.
Andrew Moore and Jeremy Barnes. Multi-task learning of negation and speculation for targeted sentiment classification. In Proceedings of the 2021 Conference of the North American Chapter of the Associalion for Computational Linguistics: Human Language Technologies, pp-
13

---

Published as a conference paper at COLM 2024
2838-2866 Online, 2021. Association for Computational Linguisties. doi: 10.18653/v/ ma2lmmacl-main 227. URL httpps:://aaeee h/w.stthoooaaaoay.ooooo.omo.o... otg/2021.
227.
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhigiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Qingyu Tan Jianyu Wang, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. Seallms large language models for southeast asia. 2024. URL https://arxiv http://aaxxvv...g/pooe// /ttgp////3//..o/////// 00738.
OpenAI, GPT-4 technical report. CoRR, 08774. URL https://dooi http///ii.or//////////// 20550/arXiv.2333..00
2023. doi: 10.48550/arXiv.230
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. CoRR, abs 220.00215, 2022. doi: 10.48550/ ar....200.00021 URL https://doi, org/oog////o 10. 48550 220.. 2205.01555
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgion, Ion Androut- sopoulos, and Suresh Manandhar. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (Se- mEval 2014), pp. 27-35. Association for Computational Linguistics, 2014. URL https: //aclanthology hwwplllggy.ogg///// org/514-2004.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, An- tonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esmne Sutherland, Karen Simonyan, Michela Paganini, Lau- rent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pa- jarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/2112.11 22112.11144 14446, 2021. URL https://arxiv wwwpaaawwii..oogbbbaaaaeee/eaaee wrg. 12.1.1111111 11446.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-110:67 2020. URL http: 074.html.
wrrgpppppss///////2// //jmlr.org//
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. CoRR, abs/ 2308 12900 2023. doi: 10.48550 A2055//AXX...23.1222222 //doi. org/r.gg///. 0al.e.48550///ii1 2308 12950.
URL https:
Oscar Sainz, Iker García-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, Wei-Lin Chen, Jenny Chim, Leshem Choshen, Luca D'Amico- Wong, Melissa Dell, Run-Ze Fan, Shahriar Golchin, Yucheng Li, Pengfei Liu, Bhavish Pahwa, Ameya Prabhu, Suryansh Sharma, Emily Silcock, Kateryna Solonko, David Stap, Mihai Surdeanu, Yu-Min Tseng, Vishaal Udandarao, Zengzhi Wang, Ruijie Xu, and
14

---

Published as a conference paper at COLM 2024
arxi::2407221 axXio:240.221530, 2024 URL /ttps:://axxiv.or//abbbss
Jinglin Yang. Data contamination report from the 2024 conda shared task. arxiv preprint 21530.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. TePPLUGGrrttrrevaalaggmeeetttt black-box language models. CoRR, abs abbs/230..11533 236.......,.1 2023. doi: 10.48550 / URL https: //tp:://do.. org/10 /00 48550/arXiv. 2301 12652.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Agüera Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle K. Barral, Dale R. Web- ster, Gregory S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. Towards expert-level medical question answering with large lan- guage models. CoRR, abs/2305.09611 2023. doi: A0L.5550/AAXI.23555000 225.5///5xivv225..... 09617.
URL
https://doi.org:
Ankur Sinha, Satishwar Kedas, Rishu Kumar, and Pekka Malo. Sentfin 1.0: Entity-aware sentiment analysis for financial news. J. Assoc. Inf. Sci. Technol. 73(9):131441335, 2022. doi: 1002/asi. 24634.
246444 URL
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turee NLG 530b, A large-scale generative language model. CoRR, /tbb:/2201.199// 2022. URL https: /tttps://arxv.... org/ abs/2201. .11201.11999
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, 2013. Association for Computational Linguistics. URL https://aclanntooogg.oooo wrtp://-000//11111127/0
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Chinn Chang, Igor Krivokon, Will Rusch, Mare Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prab- hakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman- John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. Lamda: Language models for dialog applications. CoRR, abs/ /tt./2222.08299, 2022. URL https: org/ abs 2201. 08239.
Cigdem Toprak, Niklas Jakob, and Iryna Gurevych. Sentence and expression level anno- tation of opinions in user-generated discourse. In Proceedings of the 48th Annual Meeting of the Association for Compudational Linguistics, pp. 575-584, Uppsala, Sweden, July 2010. /ac.///olaaanntogy...oooo org/P10-1059.
Association for Computational Linguistics. URL
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, CoRR, 2332, 2023. doi: 10.48550/ arXiv. 2397. UtL.tttps:/doi.oom org.10..01000 008550/ariia 2302. 13971.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bow- man. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop Blackhox Analyzing
15

---

Published as a conference paper at COLM 2024
and Interpreting Neural Networks for NLP, Pp. Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 2211653/v1/W18-- URL https://aclanthologg. wwwpe//oooooooooo////////
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxing Jiao, Yue Zhang, and Xing Xie. On the robustness of chatgpt: An adversarial and out-of-distribution perspective. CoRR, Abs/2230.110955, 2023a. doi: 10.48550/ 10.48550/arxiv.23000 23tp URL https: //dpi wtt.ooo.////////.. 12095.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. CoRR, abs/2203.11171, 2022. doi: 10.48550/am 21155///XXvv.23.11177 URL https://doi.org 0trg/10.48550 arXiv.2283.1117
Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative for math: Part mathpile: A billion-token-scale pretraining corpus for math. arXiv preprint ar arXio:2312.17120, 2023b.
URL UtL http://ww...r/ab///22112122
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abbs/220.11190, 2022. URL https://arxivv hargg/bbs/2000 11903.
Rui Xia and Zixiang Ding. Emotion-cause pair extraction: A new task to emotion analysis in texts. In Proceedings of the 57th Annul Meeting of the Association for Computational Linguistics, pp. 1003-1012, Florence, Italy, 2019. Association for Computational Linguistics. URL https:
doi: 10. 1111866//1111111111
org.///oooooro//1111 9099..
Rui Xia, Feng Xu, Jianfei Yu, Yong Qi, and Erik Cambria. Polarity shift detection, elimination and ensemble: three-stage model for document-level sentiment analysis. Information Processing & Management, 52(1):36 2016.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environ- ments. CoRR, htss/2404009722 2024. doi: 10.48550/ ARXIV.2404.0972. URL https: //doi.org/1e, htt.://r//....55////r..... 2404.00 07972.
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakaa in large language models. Xiv preprint arXio:2404.18824, 2024. URL /ttttttt////xiv..oom abs/2404. 18824.
Ashima Yadav and Dinesh Kumar Vishwakarma. Sentiment analysis using deep learn- ing architectures: a review. Artif. Intell. Rev., 53(6):4335-4385, 2020. doi: 10.1007/
510462-019-09794-5 URL https://doi org/10 1007/s18462-019--99793
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. A comprehensive capability analysis of GPT-3 and GPT-3.5 series models. CoRR, 1bs/22003.00420, 2023, doi: arXiv. 2303. 10420.
URL htttps://oll https:///ooooo//1////// wrgg/...48500
Xi Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. In NeurlPS, 2022. URL http::/papers. nips wwtp:pcrcppppccceeeeee hash/c402501
ccppaacpppppppeeriiiiiiiccii wwt.rfflllllpppeee plles//pper/2222) html.
Lei Zhang, Shuai Wang, and Bing Liu. Deep learning for sentiment analysis: A survey. WIREs Data Mining Knowl. Discov, 8(4), 2018, doi: 10.1002 h0000//wdm..5555 1253.. URL https: //doi. orgi0. 1002/widm 122.. 1253.
16

---

Published as a conference paper at COLM 2024
Meishan Zhang, Yue Zhang, and Duy-Tin Vo. Neural networks for open domain targeted sentiment. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Pp. 612-621, Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 10.18653 /10////1/111100/// 1073. URL https: / ac www.aanoloogo..oo
wwapollogooogg/l00 org/D15- 1073.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CoRR, abs/2205.0100 2022a. doi: 10.48550/ arXiv. 2205.01068.
0044555//XXi.2225..000 arXiv.2205.01068 URL https: //doi//oiii// http///oo.ogg////////
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. Towards generative aspect- based sentiment analysis. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pp. 504-510, Association for Computational Linguistics, 2021. doi: 10.18653 http://111221..cssoorttt.te URL https: //doi.org//e/ 0088//1/33000 acl-short.64
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. survey on aspect-based sentiment analysis: tasks, methods, and challenges. IEEE Transactions on Knowledge and Data Engineering, 2022b.
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. Sentiment analysis in the era of large language models: A reality check. CoRR, abs 2505. 2023. doi: 10.48550/ ARXIV.2305.1500 URL https: httt:::/ddoioog/111 48550/arXiv. 2305. 15005.
Yiming Zhang, Min Zhang, Sai Wu, and Junbo Zhao. Towards unifying the label space for aspect- and sentence-based sentiment analysis. In Findings of the Associa- fion for Compulational Linguistics: ACL 2022, pp. 20-30, Dublin, Ireland, 2022c. Asso- ciation for Computational Linguistics. doi: 10.18653/v1 http://112222..ndnn---a00 https: hatpss///claathhooyyyy or2/2022. findings-acl.3.
URL
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gon- zalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Sysiems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember 10 - 16, 2023, 2023. URL http: //apers nips cc/paper /wtps//pppeee//23//3 /2223/hash/ html.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Can chatgpt understand too? A comparative study on chatgpt and fine-tuned BERT. CoRR, abs02202.00199, 2023. doi: 1..45500/aXiv.2202....
URL htttps://doi.rrgg hrt.://ooorrr///...////////./ 2302. 10198.
Peilin Zhou, Zeqiang Wang, Dading Chong, Zhijiang Guo, Yining Hua, Zichang Su, Zhiyang Teng, Jiageng Wu, and Jie Yang. Mets-cov: A dataset of medical entity and targeted sentiment on COVID-19 related tweets. CoRR, abs/ 22777 2022, doi: 10.48550/ arXiv. 2209.13773.
2209.13773.
Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. Exploring Al ethics of chatgpt: A diagnostic analysis. CoRR, abs/ /200..1.07..6. 2023. doi: (0.48550/arXiv/// 12867. URL https://dde org/10, wtt.:/../...///ra......oooo/ 2301. 12867.
48550/arXiv.
Chengqing Zong, Rui Xia, and Jiajun Zhang. Text Data Mining. Springer Singapore, 2021. ISBN 978-981-16--100-- doi: 10.1007 10.1000/9978-9811-1111 99900999991111--00002222 URL https://don org/10. 18077977--811-6--1180
17

---

Published as a conference paper at COLM 2024
A Appendix
A.1 Closely Related Work
Prior to our work, there were some early studies that evaluated ChatGPT's performance on common NLP tasks, but they only covered basic sentiment classification tasks, such as SST-2 (Zhong et al., 2023). Our work, however, is dedicated to more comprehensive and thorough evaluation for ChatGPT within the sentiment analysis field, representing, to our knowledge, the first of its kind in this field. After our work was completed (over a month later), new study emerged, evaluating more tasks like ASTE and ASQP within ABSA and also providing few-shot experimental results and human evaluations (Zhang et al., 2023). However, this study did not evaluate critical issues in sentiment analysis such as sentiment polarity shift and open-domain evaluation, which our work does address. Although different works may have their own focuses, we believe they all contribute significantly to providing more comprehensive evaluations in this field.
A.2 Prompts of ChatGPT
Following Jiao et al. (2023), we ask ChatGPT generate the task instruction for each task to elicit its ability to the corresponding task. Taking the E2E-ABSA task as an example, our query is:
Please give me three concise prompts for eliciting your ability to perform Aspect-Based Sentiment Analysis (i.e., extract the aspect terms and sentiment polarity). There is no need to give examples and do not limit the prompts to specific product or domain.
Then, we examine the generated three prompts on small-scale (e-g,, 50 examples) example set driving from the corresponding training set. We select the best and most reasonable one according to the results.? The final prompts adopted for each task are shown in Table 9. During the evaluation process, we feed prompt and corresponding test example to ChatGPT and obtain a generated response. We manually observe and record the results as the responses do not follow a certain pattern.
A.3 Preparation of Polarity Shift Evaluation Datasets
As previously mentioned, we drive SST-2-Neg and SST-2-Spec from SST-2 by detecting whether a sentence contains any negation or speculation words. The seed words adopted are shown in Table 11. And the statistics of involved datasets are shown in Table 10.
A.4 Examples on Human Evaluation
The exact-match metric has limitations for evaluating generative models like ChatGPT since they can produce reasonable outputs not matched to references. To better characterize ChatGPT's capabilities despite this, we manually refine its outputs before comparing them to those of baselines. We acknowledge this may seem unfair compared unrefined baselines. However, our goal is to account for the limitations of the exact-match, not to boost ChatGPT's results unfairly. To further illustrate the rules we use as more intuitive and easier to understand, we provide some examples from the E2E- 12E-ABSA task, as shown in Table 12.
A.5
Case Study for Standard Evaluation
In this part, we conduct the qualitative analysis on ABSA tasks, COM tasks, and ECA tasks.
When necessary, would make minor adjustments to the prompts.
We observe that different prompts have little effect on the performance. We also conducted three experiment repetitions and found minimal deviation in the results. Considering the cost of API calls, we only the experiment for the final evaluation unless otherwise specified.
18

---

Published as a conference paper at COLM 2024
Task
Prompt
Given this text, what the sentiment conveyed? Is positive or negative? Text:
SC
{sentence)
Sentence: (sentence) What is the sentiment polarity of the aspect (aspect) in this sentence?
ABSC
Given a review, extract the aspect term(s) and determine their corresponding sentiment polarity. Review: (sentence)
E2E-ABSA
Does any comparison of products (including implicit products) exist in the product review: (sentence)? If so, outputs 'TRUE', else outputs 'FALSE'.
CSI
products (inchuding implicit products): {sentence); Extract the subject and object of comparison, tell me which aspect of products is being compared, and tell me if the author of the review thinks the subject is better or worse than similar to or different from the object. In If multiple comparisons exist, output multiple comparisons.
The following product review contains comparison
CEE
Document: (doc) \n Each line in the above document represents clause and the number at the beginning each line indicates the clause ID. Clauses expressing emotions are referred to as "emotion clause" and clauses causing emotions are referred to as "cause clauses". It has been identified that the clause with ID {emo_id), (emotion clause) is emotion clause, and the corresponding emotion keyword is (emotion on Based the above information, complete the following tasks: 1. Describe in one sentence the cause of the emotion clause with ID (emo_id). Based on the result of Task 1, output the ID of the cause clause that best fits the requirements. 3. According to the result of Task 2, match clauses with causality into pairs in the form "(emotion clause ID, cause clause ID)" and output all pairs as a set, such as (3,4), Note: the emotion clause and the cause clause may be the same clause, and only the most obvious pairs need be outputted.
ECE
Document: (doc) \n Each line in the above document represents clause and the number the beginning of each line indicates the clause ID. Clauses expressing emotions are referred to as "emotion clause" and clauses causing emotions are referred to as "cause clauses". Based on the above information, complete the following tasks: 1. Describe the emotions and their corresponding causes contained in the document in one sentence. 2. Output the ID of the emotion clause in task 1, you only need to find the one with the strongest intensity. 3. For each emotion clause in task 2, find the corresponding cause clause and output the cause clause ID, you only need to find the most suitable one. Match clauses with causality into pairs in the form "(emotion clause ID, cause clause ID)" and output all pairs as set, such as (1,2),3AA). Note: the emotion clause and the cause clause may be the same clause, and only the most obvious pairs need to be outputted.
ECPE
Table 9:
The prompts used for prompting ChatGPT for each task. We manually design prompts for emotion cause analysis tasks (i.e., ECE and ECPE) due to the task complexity.
Task
Dataset
#Test
SSI-2-Negation
236
SC
SS1-2- -SS----ppeullttonn
88
44-Res-Negation
1008
1peeustSpo
448
ABSC
14-Lap-Negation
462
14-Lap-Speculation
217
Table 10: The tasks and datasets involved in the polarity-shifting evaluation. #Test denotes the number of examples used for evaluation.
Case Study on ABSA. We conduct the qualitative analysis through two examples. Specif- ically, as shown in Figure 1, we present the results generated by ChatGPT for two test examples under zero-shot and few-shot settings, respectively. Given the example "I did swap out the hard drice for a Samsung 830 SSD which I highly recommend", there are multiple aspect terms with different sentiment polarities in sentence (e-g., the sentiment polarity
19

---

Published as a conference paper at COLM 2024
Shifting Type
Seed Words
n't, no, not, never, neither, nor, unless, but, however, rather than, not yet, not only, nonetheless, despite, although, even though, in spite of, unlikely
Negation
if, would, could, should, seems, might, maybe, whether, unless, even if, if only, can't believe, grant that, guessing, suspect, hope, wish, let's probably
Speculation
Table 11: Seed words used for deriving SST-2-Neg and SST-2-Spec from SST-2.
For any extra generated tuples, if they are reasonable but absent from the annotations, we will remove them from the prediction results. Otherwise, will keep them. Example#1
Rulelt:
Input: super fast has outstanding graphics.
Output:
Aspect term: speed, graphics
Sentiment polarity: positive, positive
Ground Truth: (ggraphics, positive)]
Refined Output:
Aspect term: graphics
Sentiment polarity: positive
Rule#2: We also consider an aspect-sentiment or comparative opinion tuple correct if the boundary of aspect or entity is
predicted incorrectly unambiguously, the predicted sentiment or preference correct.
Example#1
Input:
the hardware problems have been so bad, ca n't wait it completely dies in years, TOPS!
Output:
Aspect term: hardware problems
Sentiment polarity: negative
Ground Truth: (hhardware, negative)]
Refined Output:
Aspect term: hardware
Sentiment polarity: negative
Example#2
Input: And
that comes with an processor definitely speeds things up.
Output:
Aspect term: procesoor
Sentiment polarity: positive
Ground Truth: (65 processor, positive)]
Refined Output:
Aspect term: i5 processor
Sentiment polarity: positive
Rule#3: We also regard prediction that paraphrases the ground truth to be correct, given the text generation paradigm.
Example#1
Input: Shipped very quickly and safely.
Output:
Aspect term: Shipping
Sentiment polarity: Positive
Ground Truth: ((hhipped, positive)]
Refined Output:
Aspect term: Shipped
Sentiment polariiyy Positive
Example#2
Input: Runs real quick.
Output:
Aspect term: Speed/Perforrnee /Performaacce
Sentiment polarity: l'ositive
Ground Truth: [[uus, positive)]
Refined Output:
Aspect term:
Sentiment polarity: Positive
Table 12: Examples on human evaluation. For simplicity, the task instruction is omitted.
of "hard drive" is neutral, and that of "SSD" is positive). We can observe that ChatGPT can not accurately identify the sentiment polarity of "hard drive" under the zero-shot setting. Similarly, in another test example "I can say that I am fully salisfied with the performance that
20

---

Published as a conference paper at COLM 2024
ABSC (Zero-shot)
E2E-ABSA (Zero-shot)
Sentense: did swap the hard drive for Samsung SSD which I highly recommend. What the sentiment
review, extract aspect term(s) and determine their corresponding sentiment polanity, Review: say that I fully satisfied with the performance that the computer has
polarity of the uspect hard drive in this sentence?
supplied
Sentiment polarity: negative
Aspect terms: computer performance Sentiment polarity: positive
Ground Truth: neutral
X
Ground Truth: [[performance, positive]]
X
ABSC (Few-shol)
E2E-ABSA (Few-shot)
The keyboard slick. What the sentiment polarity of the uspect keyboard in this sentence!
Sontence:
review, extract aspect term(s) and determine their corresponding sentiment polanity. Here some examples:
Label: negative
Review: It runs perfectly.
Sentence: Decent wine reasonable prices. What the
Label: [[runs, positivel]
senliment polarity of the aspect wine this sentence??
Review: The service awful
Label: positive
Label: [[scrvicc, nogative]]
Sentence: Amazing Quality! What is the sentiment polarity of the aspect Quality this sentence?
Review: lots of extra space but the keyboard ridiculously small Label: [[space, positive), [keyboard, negative]] Review: I can say that am fully satisfied with the performance that the computer has supplied.
Label: positive
Sentence: did swap out the hard drive for a Samsung 830 SSD which I highly recommend. What the sentiment polarity of the aspect hard drive in this sentence?
Label: neutral
Label: [[performance, positive]]
Ground Truth: neutral
Ground Truth: [[performance, positive]
Case study for ChatGPT on ABSC and E2E-ABSA in zero-shot and few-shot settings. The text in blue, black, green and red denote the given prompts, the examples to be evaluated, the responses of ChatGPT and the ground truths, respectively.
Figure 1:
the computer has supplied.", ", the aspect term extracted by ChatGPT is "computer performance", which does not naturally exist in the sentence, indicating that ChatGPT may generate se- mantically reasonable aspect terms but without being aligned with the annotations in the dataset. However, under the few-shot setting (as introduced in S 4.1), after being equipped with few demonstration examples, both of the above types of errors can be corrected by ChatGPT.
Case Study on COM. We conduct qualitative analysis through two examples of ChatGPT in the case of CSI and CEE tasks, as shown in Figure 2. For the CSI task, it can be seen that ChatGPT is able to accurately identify explicit product comparison sentences. However, when the compared objects are implicit products, ChatGPT often considers the sentence not to be a comparison sentence, such the sentence "However, focus accuracy was not as impres- sive." " ChatGPT assumes that there are no explicitly mentioned products in the comment and therefore determines that it is not a comparison sentence. For CEE task, although ChatGPT is able to correctly identify comparison sentences and extract comparative elements, it tends to exhibit paraphrase phenomena when generating answers. For example, in the example sentence "It seems to get less light to the sensors than my old 4MP A80." " the annotation indicates that the comparison subject is "worse" than the comparison object. However, when replying, ChatGPT expresses it as the comparison object is "better" than the comparison subject. This situation causes alignment issues between the generated answers and the annotations during automatic evaluation.
Case Study on ECA. We also conduct qualitative analysis through two examples of ChatGPT in the case of ECE and ECPE tasks, as shown in Figure 3. It can be seen that ChatGPT can perfectly follow our given instructions to complete the task. Given the emotion of a document, ChatGPT can accurately analyze its corresponding cause, but the clause ID is not output as required (refer to the upper part of Figure 3). For the same document, without providing the emotion clause beforehand, we find that ChatGPT can still correctly identify it, which corresponds to the ECPE task (refer to the bottom part of Figure 3). Although the identified cause clause may not be the most obvious, the prediction is still reasonable.
21

---

Published as a conference paper at COLM 2024
CSI
CSI
Does any comparison of products (including implicit products) product review "As for comparisions with other Canon Powershot cameras, the Powershor SD630 doesn't have a view finder, but has larger LCD monitor in its place." ? so, outputs "TRUE", else outputs 'FALSE"
Does any comparison of products (including implicit products) exist in the product reviow "However. focus accuracy was not as impressive, ? If oulputs "TRUE', else outputs "FAL.SE.
Yes, there comparison products in the product review, The roview compares the Powershot SD630 with other Canon Powershot cameras, Therefore, the output 'TRUE".
@
from the given sentence whether there a compartson of products not, The sentence only mentions the
is
focus accumacy of single product, withour any reference other
products, Therefore, the output would be 'FALSE'.
Ground Truth: TURE
Ground Trath: TURE
X
CEE
CEE
The following product roviow contains comparison of products (incloding implicit products): Selling points- In which cxplain why bought the SD My main and sk skipping the 30D) ) was the full-frame sensor. Extract the subject and object of comparison, compared, and tell me if the author of the review thinks the subject is beteer werse than similar different from object. If multiple comparisons exist, output multiple comparisons.
The following product review contains compurison products (including implicit products)) Il seems get less light to the sensors than my old 4MP Extract subject object comparison. tell me which aspect of products is being compured, and tell me if the author the reviow thinks subject better warse similar to or different from the object, If l'mulliile comparisons exist, output multiple comparisons.
for upgading from 3500
which aspeat products heing
Comparison: 350D and 5D
Yes, companison exists the product review.
Subject: 350D
Subject of comparison: "It" (referring to the product being reviewed)
Object: 5D
Object comparison: old 4MP A80"
Aspoct: Full-frame sensor
Aspoct of products being compared: amount of light reosived by the
Author's opinion: The author thinks that the 5D better than the
350D bocause of the full-frane sensor.
Author's 's opinion: The author thinks that the old 4MP A80 reosived more light the sensoos than the product being reviewed. Therefore, the author thinks that the old product botter than the new one in torms of the amount of light received by the sensors.
Ground Truth: (5D, 350D, full-frame sonsor, belter)
Ground Truth: (li, my old 4MP A80, light to the sensons, worse)
Figure 2: Case study for ChatGIT on CSI and CEE. The text in blue, black, green and red denote the given prompts, the examples to be evaluated, the responses of ChatGPT and the ground truths, respectively.
A.6 Case Study for Polarity Shift Evaluation
We conduct qualitative analysis through four examples of ChatGPT in the case of polarity shift (i.e., negation and speculation), as shown in Figure 4. Observing the two examples of SC in the first row, it becomes apparent that ChatGPT can accurately determine the overall sentiment polarity of sentences accompanied by polarity shift due to the presence of negation and speculation expressions. Regarding the two examples in the second row of the ABSC task, when multiple aspect terms are listed in parallel and involve the linguistic phenomena such as negation and speculation (e-g., "faster and sleeker looking" and "super fast and had outstanding graphics"), ChatGPT struggles to identify their sentiment polarity accurately, leading to incorrect predictions.
A.7 Case Study for Open-Domain Evaluation
We conduct qualitative analysis through four examples of ChatGPT on Books and METS- Cov, corresponding to the books and medicine domain, as shown in Figure 5. Regarding two examples in the first column, ChatGPT has difficulty accurately predicting sentiment in books domain since this domain usually contains unique expressions and plot descriptions that differ from typical product reviews. We also find that ChatGPT often generates reason- able aspect-sentiment pairs in the E2E-ABSA task, such as the "(chapter creations, negative)" (in the bottom left example). However, as we mentioned earlier, they are not originally annotated in the dataset, which is also an important reason affecting the performance of E2E-ABSA. As for the two examples in the second column, when dealing with rare domain like medicine, ChatGPT can accurately determine the sentiment polarity of the given aspect term, but accurately extracting aspect-sentiment pairs remains challenge.
22

---

Published as a conference paper at COLM 2024
BCB
BCB
Document:
1. For Mr Lu, she inmociiatlly helped when she met the fallen old mun Accompaniing the old man and waiting for the behavior of the police
B fi
1 2016
Many resiens pasing peatsed her
The puliez who called out slso toli repurters
Old people who realty need help Otherwise, sccond accideen may occur
7. very mae to he comppeed holping the chierly
Each line the shove document represents clusse and the number the beganiing of each line indicates the clause ID. Classes exgrosing euatiions are setirred as "emotion classe und clauses cassing emotions refereed so "couse clasus" It lox boen identified that the classe with ID 3, "Many resdents passing peaised her' emotion clause, the correspending following
38
JAN 7
1 #
7-5
285 # hPF 1978918 st2
infocmstion, cumplete emotion with 1D 3: Based on the resuit of Task I, cutpur the of the couse clause tha best lis the roqurettnnts. 3. Ancedding to the noult of Tisk match clauses with camolity into pain the form "lemotien classe ID, couse clause IDI and output all paiss a set such as (1.233344 Nose: the emetion clause and the came classe may be same clase, and only the masd alivioss pairs outputted.
"prise"
1951 (0.2),344)) 4)).
S
1 09
00 A.
12. R.
2j de
S
"Mary rosidents passing by prisecal The rownn for this cimotiin chasse that Ms. Lu helps fallen old man. Aocoeding to the result of task I, the caise clanse is numbbered 1. The pair ( (1.3) can be outpur, because Ms. Lu helped the fullen old man and the sumounding sesidants priseal her
18h 22.
Trubh ((5D))
Ground Tnuth: (O.0))
ECPE
ECPE
Deciment:
1. For Ms. La, she immediately helped when she met the fallen old man Assongancing the old man and walling foe the behavice of the pollec Many residents precing by prived her
11: F
E #1
The police who called out also sold reporters
Mis 1 1150
5. Old peuple who fall rcally need help
socident may occur 7. very nic comppted helping the elderty
A
Each in the shouc document reprssntle classe and mamber the beginning of coci line indiconess the classe C Classs expresiing emotions are reformed to an "emotion chass" and classss caneing emotions ane referved "casse clzuses". Based on the above information, complete the following tinkr I. Doserite the cnatiions and their cuncopunding canacs cintaincd in the decument you only need find clesse taak find the conosponling caese clunse and ouiput cause chruse II) you only need to find the most suitable one. March clisses with causality into pains the fomm "Lemotan clause ID, cause clause IDY and ourper all paire as set, such (1 11,21 1.43 Note the emution clease and the cause clause may be the same classe, and only the most abvious pains need be outputzd.
AATARNA 15 69 4 2. MITERO 1E-8 I. 1------00 R9
F 1 2 HEEEER RI MO Whit 7-10
244 ti A.EE 3.3
T FOLLE
Output the the emotin clause strergent intersity
62
1898 7 BEB siply TT
M 1
101S1M B 43 MAR
S
0 50
Emootons are peaise becouse
and her actions.
7 3g
The cantion classe mamberscl 3.
9136 7
pa w
The cause chusse of emetion chusse is that is, Ms. La nocompenied the old man and other polies efficens.
02 22 (332H H033 11 2
4. The matching result is (3.22 ). that is, emotion classe No. 3 caused by cnss clause No. 2.
Grannd Tnnh: (3.1)
Ground Truth: ((3.1)
Figure 3: Case study for ChatGPT on ECE and ECPE in both Chinese (left) and English (right). The text in blue, black, green and red denote the given prompts, the examples to be evaluated, the responses of ChatGPT and the ground truths, respectively.
A.8 Details for Chain-of-Thought and Self-Consistent Prompting
The thain-of-thought method augments each demonstration example in standard few- shot prompting with a chain of reasoning for the associated answer (Wei et al., 2022). We manually write CoT for randomly sampled examples. Self-consistency seeks to sample diverse set of candidate outputs from LLMS and then aggregate the answers via a majority vote. We apply the temperature sampling with T = 0.8 as self-consistency is generally robust to sampling strategies (Wang et al., 2022). For the aggregation of answers, unlike the arithmetic reasoning task that typically has only one certain answer, the E2E-ABSA task evaluate usually contains multiple sspect-sentiment tuples in example. We adopt heuristic approach by counting the frequency of each tuple in N sampled predictions and filtering by setting frequency threshold to obtain the final prediction. We can finely control
23

---

Published as a conference paper at COLM 2024
SC (Negation)
SC (Speculation)
Given this text, what is the sentiment conveyed? Is it posirive or nogative? Text: unfortunatlly, it 's not silly fün unless you enjoy really bed movies.
Given this text, what the sentiment conveyed!' positive negative? Text; it can't decide if wamts to be mystery/ibriller, romance or comedy... -
S
The sentiment conveyed in this text is negative.
S
The sentiment conveyed in the text is negative.
Ground Truth: negative
Ground Truth: negative
ABSC (Negation)
ABSC (Speculation)
Sentence: much faster sleeker looking PUNCT, What is sentiment polarity the aspect looking this sentence??
Sentence:
o'only it were super list and had outstamding graphies PUNCT What the sentiment polarity the aspect graphics in this sentence?
Label: positive
positive
Ground Truth: negative
x
Ground Truth: nogative
X
Figure 4: Case study for ChatGPT on SC and ABSC in case of the linguistic phenomena such as negation and speculation. The text in blue, black, green and red denote the given prompts, the examples be evaluated, the responses of ChatGPT and the ground truths, respectively.
ABSC (Books)
ABSC (METS-CoV)
Sentence: Still, was good finish series finally
COVID today, when the mind control kicks in, but focling pretty good overall What is the sentiment polarity of the aspect Pfizer COVID vaccine this sentence!!
Sentence: received
there excupe the magical world just not much fun to read as had bopod. What the sentiment
polarity the aspect series this sentence?
The sentiment polarity of the aspect series in this sentence is negative.
e
The sentiment polarity of the aspect "Piizer COVID vaccine" in this sentence is pusitive.
Ground Truth: neutral
X
Ground Truth: positive
E2B-ABSA (Books)
E2E-ABSA (METS-Cov)
review, extract aspect temm(ss and determine their corresponding sentiment polarity, Review; I enjoyed the plot,
review, extract aspect term(s) and determine their corresponding sentiment polarity, Review: India plays US friend need, , allows hydroocchoooquine export to fight covid.
however in the beginning it was difficult to follow due to lack
chapter creations..
Aspect term: plot, chapter creations Sentiment polarity:
Aspect term: hydrosychloroqquio export Sontiment polarity: positive
plot: positive chapter creations: negative
Ground Truth: [[beginning, negative], [plot, positive]]
Ground Truth: [[hydroxychlorqquum neutral]]
Figure 5: Case study for ChatGPT on ABSC and E2E-ABSA on books and medicine domains. The text blue, black, green and red denote the given prompts, the examples to be evaluated, the responses of ChatGPT and the ground truths, respectively.
the answer aggregation by setting the threshold. In our experiments, we find that when N 15, a threshold between 7 and 12 performs well.
A.9 Other Evaluation Results
Evaluation on text-davinci-003 Some readers might be curious about the performance of other powerful GPT-3.5 models in comparison ChatGPT. To address this concern, we evaluate the powerful GPT-3.55 model, text-davinci-003, on some benchmarks. We carefully tune the evaluation to be as rigorous and controlled as possible, with temperature 0, top-p of 1, and repeated runs to account for any variability (which is turned out to be negligible). As shown in Table 13, text-davinci-06. achieves overall performance on par with ChatGPT.
24

---

Published as a conference paper at COLM 2024
Task
Dataset
Metric
Baseline
SOTA
ChatGPT
text-devinci-003
SC
SST-2
Acc
95.47
97.50
93.12
90.52
ABSC
14-Rest.
Acc/ / Fl
83,94
75.28
89.54 / 84.86
83.85 / 70.57
82.19 / 71.74
ABSC
14-Lap.
Acc Fl
77.85
73.20
83.70 80.13
76.42 / 66.79
75.11 / 70.63
E2E-ABSA
14-Rest.
F1
77.75
78.68
69.14
65.06
E2E-ABSA
14-Lap.
F1
66.05
70.32
49.11
50.44
Table 13: Performance comparison among ChatGPT, text-davinci-003. fine-tuned base- lines, and SOTA models on datasets. Most results are derived from Table 2.
Re-Rest.
14-Lap.
Model
Implicit-split
All Implicit-split
All
Fine-tuned BERT
65.54
77.16
69.54
73.45
ChatGPT
56.31
69.72
52.68
65.92
text-devinci-003
56.85
71.09
57.17
71.09
Table 14: Evaluation results on implicit sentiment analysis among fine-tuned BERT, ChatGPT and text-davinci-003.
Evaluation on Implicit Sentiment Analysis As an interesting and challenging direction, we also explore the evaluation on implicit sentiment analysis. Following the dataset split of implicit sentiment analysis described in Li et al. (2021), we evaluate ChatGPT on the ABSC task and report BERT results (derived from Li et al. (2021)) as reference. We also evaluate the performance of text-dat text-ddvinci--ee -003. Similarly, we run 3 trials and report the average F1 over the implicit subset and the full ABSC dataset (we find that the variance is small). As shown Table 14, we can observe that these large language models perform poorly on implicit sentiment analysis, although coxt-davinci--ee outperforms ChatGPT, both are weaker than fine-tuned BERT. These results suggest ample opportunities for future research.
B Limitations and Future Work
This work has several limitations as follows: (1) Data leakage. Currently, conducting rigorous evaluations for LLMs is extremely challenging. For example, it is difficult for us to determine whether the test set has been seen during the large-scale unsupervised pre- training, especially for models like ChatGPT, which are completely closed-soures and can only be accessed through APIs (Dodge et al., 2021; Golchin & Surdeanu, 2024; Balloccu et al., 2024; Xu et al., 2024; Sainz et al., 2024, inter alia). Nevertheless, in this work, we still find some deficiencies of ChatGPT, such as its sentiment analysis performance in some domains (e.g., medicine and social media) that leaves much to be desired. (2) Prompt design. We do not conduct extensive prompt engineering, so there are likely better prompts to obtain better performance. Nevertheless, prompt designs when using LLMSs. Therefore, if the ChatGPT can achieve sufficiently robust performance on arbitrary prompts, this would better demonstrate its capability. (3) Limited evaluation. Our evaluation is mainly conducted on ChatGIT, without including other equally powerful models. Although we have also supplemented other evaluation results in Appendix A.9, such as closed-source and we do not have access to APIs, or we do not have enough GPUs to rigorously evaluate their performance due to their huge model parameters. However, as a representative of currently the most powerful models, evaluation on ChatGPT can also enable us to understand what LLMs currently do well and not well, thereby inspiring future research.
believe that ordinary users usually do not do very delicate
-003. Unfortunately, such models are either completely
Beyond this work, we believe some promising future directions could include: (1) New evaluation benchmarks. We need to propose new and comprehensive benchmarks from real-world scenarios (Zheng et al., 2023; Huang et al., 2024; Jimenez et al., 2024; Xie et al., 2024, inter alia). Meanwhile, evaluation methods are also worth paying attention to. Due to the text generation paradigm, commonly used exact-match may not truly characterize the
25

---

Published as a conference paper at COLM 2024
model performance. In this paper, we adopt human evaluation to alleviate this issue. (2) Implicit sentiment analysis. Implicit expression is very common linguistic phenomenon. For example, "I know real Indian food and this wasn't it" does not contain explicit opinion words. Moreover, accurate judgment often requires common sense or domain knowledge. Our experiments also confirm that large language models generally perform poorly implicit sentiment analysis (See Appendix 14 for results). Meanwhile, constructing com- prehensive benchmarks for implicit sentiment analysis could be a promising direction. (3) Enhancing the model capabilities in specific domains, As shown in Table 5 and Table 6, we can see that the performance of ChatGPT is not satisfactory on many domains (such as books and twitter). Therefore, in the future, we could improve the performance on certain domains through domain-specific training, as demonstrated by recent efforts (Singhal al., 2023; Wang et al., 2023b; Azerbayev et al., 2024; Rozière et al., 2023; Dou et al., 2024; Nguyen et al., 2024, inter alia).
26


Reviews:
[{"review_id": "b9lC4i3jke", "reviewer": "Reviewer_gr5x", "summary": "The paper is devoted to testing ChatGPT in various sentiment analysis tasks.Evaluation is done  on 7 sentiment analysis tasks covering 17 benchmark datasets. ChatGPT is compared with fine-tuned BERT and corresponding state-of-the-art approaches. The tasks include:  Sentiment Classification (SC), Aspect-Based Sentiment Classification (ABSC), End-to-End Aspect-Based Sentiment Analysis (E2E-ABSA), Comparative Sentences Identification (CSI), Comparative Element Extraction (CEE), Emotion Cause Extraction (ECE), and Emotion-Cause Pair Extraction (ECPE).", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and efforts in reviewing our paper. Moreover, we sincerely appreciate your recognition of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) ChatGPT is evaluated in diverse sentiment analysis tasks. Interesting conclusions are obtained.\n\n2) Human evaluation is made to analyse low results of ChatGPT in some tasks.\n\n3) Advanced prompting techniques such as few-shot prompting, chain-of-thought promptind are also studied.\n\n4) The detailed description of expriments is well-presented in appendices.", "reasons_to_reject": "1. Among all current exisiting LLM, only ChatGPT is tested.\n2. All results are obtained for English, the conclusions can be quite different for other languages.", "questions_to_authors": "No", "extra_scores": {}}, {"review_id": "HVJY8eFNGi", "reviewer": "Reviewer_wJss", "summary": "This paper investigates ability of ChatGPT for various sentiment analysis tasks. The authors tried zero-shot and few-shot experiments with ChatGPT as well as some specific evaluations for samples with negations. The authors compared ChatGPT with BERT and SOTA baselines. The paper claims that ChatGPT can already serve as a universal and well-behaved sentiment analyzer.\n\nIn my opinion, the paper contributes to the thorough studies for sentiment analysis with the aim to evaluate ability of ChatGPT, while there exists limitations, such as the lack of consideration of other LLMs. In addition, I think the results are not surprising given the strong evidences of recent papers describing the ability of ChatGPT in other text classification and relation extraction tasks. I also would like to say that this paper is more likely about technical report because the analyses and claims are too broad.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We will update the results of RoBERTa and other LLMs in our next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Thorough analyses with various subtasks of sentiment analysis (including E2E-ABSA): The authors provide different corpora and tasks in experiments, including major SST-2, Restaurant, Laptop datasets.\n- The experimental setups and results seem scientifically valid\n- The human analysis and case study are provided", "reasons_to_reject": "- The findings of this paper are specific for ChatGPT, limiting generalizability for other LLMs. Given recent rapid trends in this field, using open-sourced LLMs such as Llama and Mistral would be interesting. \n- Related to above, any findings can be obtained from this paper, but they may not be systematic results due to the limited generalizability.\n- The authors use BERT as a baseline. However, I believe at least RoBERTa is a suitable baseline among fine-tuned small language models. It is also known that RoBERTa is a strong baseline in sentiment analysis tasks [1]. \n- The analyses in this paper are too broad in my opinion to convey the major claim that \"ChatGPT can already serve as a universal and well-behaved sentiment analyzer\".  Also, I think the results are not surprising given the strong evidences of recent papers describing the ability of ChatGPT in other text classification and relation extraction tasks. I also would like to say that this paper is more likely about technical report because the analyses and claims are too broad. \n\n[1] Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa, Junqi Dai, Hang Yan, Tianxiang Sun, Pengfei Liu, Xipeng Qiu, NAACL 2021.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9jg3FJAuSi", "reviewer": "Reviewer_Tg6F", "summary": "The paper is an evaluation of the ability of GPT3.5 in different tasks of sentiment analysis. In particular de evaluation is interesting, and the conclusions related to the self-consistency prompt enhance the classification results. However, I miss a wider evaluation, since the aim of the paper is the evaluaation of the sentiment classification capacity of LLMs, so:\n\n1. Why didn't the authors evaluate at least an additional LLM (Llama or Mistral for instance)?\n2. Regarding the open domain evaluation, why didn't the authors use the SFU Review Corpus [1] that has 8 different opinion domains?\n\nReferences:\n[1] Taboada, M., Anthony, C., & Voll, K. D. (2006, May). Methods for Creating Semantic Orientation Dictionaries. In LREC (pp. 427-432).", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your time and efforts in reviewing our paper and offering insightful comments.\n\nWe actually evaluated the performance of `text-davinci-003` in Tables 13 and 14 (in Appendix). For the language, the benchmark used for ECE and ECPE is in Chinese (refer to Fig. 3 for examples).\n\nWe will update the results of other LLMs and benchmarks in our next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I think that this evaluatino is useful in order to consider LLM as text classification methods.\n- The wide evaluation of different sentiment analysis tasks.\n- The  good results.", "reasons_to_reject": "- The lack of assessing other LLMs.\n- Tha lack of using more datasets, as the SFU Review corpus, and also datasets of microblogging genre.\n- The lack of evaluation in other language than English, for instance the Spanish language.", "questions_to_authors": "Why didn't the authors evaluate at least an additional LLM (Llama or Mistral for instance)?\nWhy didn't the authors use the SFU Review Corpus [1] that has 8 different opinion domains?\nWhy didn't the authors consider other language different than English?\n\nReferences:\n[1] Taboada, M., Anthony, C., & Voll, K. D. (2006, May). Methods for Creating Semantic Orientation Dictionaries. In LREC (pp. 427-432).", "extra_scores": {}}, {"review_id": "y1zTUgwAtQ", "reviewer": "Reviewer_Smbz", "summary": "The authors assess ChatGPT across various test sets and evaluation scenarios in order to delineate its potential in different sentiment analysis tasks. Specifically, they perform an assessment across 7 representative sentiment analysis tasks and 17 benchmark datasets, encompassing three distinct evaluation scenarios: standard evaluation, polarity shift evaluation, and open-domain evaluation. On those tasks, they compare ChatGPT against fine-tuned small language models such as BERT  along with any corresponding state-of-the-art models available for reference on each task. Finally, from these experiments, they conclude that ChatGPT can already function as a versatile and effective sentiment analyzer.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time and effort to review our paper. Your recognition of our work is the greatest affirmation we could receive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- They empirically prove that ChatGPT exhibits remarkable zero-shot sentiment analysis performance, encompassing tasks such as sentiment classification, comparative opinion mining, and emotion cause analysis. In certain instances, it even matches the performance of fine-tuned BERT and state-of-the-art models trained with labeled data in specific domains. \n- Compared to fine-tuned BERT, ChatGPT demonstrates superior efficacy in addressing the polarity shift issue in sentiment analysis and performs admirably in open-domain contexts. \n- They additionally explore popular prompting techniques to augment ChatGPT's capabilities further.\n- The paper is well organized and easy to read.", "reasons_to_reject": "The paper might have considered evaluating additional Large Language Models beyond GPT-3.5; however, I still believe it encompasses a satisfactory amount of work and quality to merit acceptance at COLM.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper offers a comprehensive evaluation of ChatGPT's performance across a wide range of sentiment analysis tasks, including Sentiment Classification (SC), Aspect-Based Sentiment Classification (ABSC), and Emotion-Cause Pair Extraction (ECPE), among others. The experiments are well-detailed and include advanced prompting techniques such as few-shot prompting and chain-of-thought prompting. The inclusion of human evaluation to analyze low-performing tasks adds depth to the analysis. The experimental setups are scientifically valid, and the results are well-documented in the appendices. The study's thoroughness in testing multiple sentiment analysis subtasks using diverse datasets, including major ones like SST-2, Restaurant, and Laptop datasets, provides valuable insights into the capabilities of ChatGPT as a sentiment analyzer.

The study's primary limitation is its exclusive focus on ChatGPT, neglecting other large language models (LLMs) such as Llama and Mistral, which could provide a more comprehensive evaluation of LLM capabilities in sentiment analysis. The analyses are criticized for being too broad, making the claims that ChatGPT can serve as a universal sentiment analyzer seem overstated. Additionally, the paper does not explore results in languages other than English, limiting the generalizability of its conclusions. The use of BERT as a baseline, instead of a more robust model like RoBERTa, further limits the comparative analysis. The absence of evaluations on diverse datasets like the SFU Review Corpus and microblogging genres also restricts the study's applicability. Overall, the paper reads more like a technical report due to its broad and somewhat unsystematic analyses.

[comments from the PCs] As much as possible, please follow up on the AC recommendation to include an open-weights LLM, such as Llama or Mistral.
Decision: Accept