Is ChatGPT a Good Sentiment Analyzer?

Authors: zengzhi  wang
qiming  xie
yi  feng
zixiang  ding
zinong  yang
rui  xia
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 13:47:51

Abstract: Recently, ChatGPT has drawn great attention from both the research community and the public. We are particularly interested in whether it can serve as a universal sentiment analyzer. To this end, in this work, we provide a comprehensive evaluation of ChatGPT on the understanding of \emph{opinions}, \emph{sentiments}, and \emph{emotions} contained in the text. Specifically, we evaluate it in three settings, including \emph{standard} evaluation, \emph{polarity shift} evaluation and \emph{open-domain} evaluation. We conduct an evaluation on 7 representative sentiment analysis tasks covering 17 benchmark datasets and compare ChatGPT with fine-tuned BERT and corresponding state-of-the-art (SOTA) models on them. We also attempt several popular prompting techniques to elicit the ability further. Moreover, we conduct human evaluation and present some qualitative case studies to gain a deep comprehension of its sentiment analysis capabilities.

Main paper: 
 introduction: 
 Large language models (LLMs) have profoundly affected the whole NLP community with their amazing zero-shot ability on various NLP tasks (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Zhang et al., 2022a, inter alia) . As a representative, ChatGPT 1 has appeared out of the blue via interacting with people conversationally. It can conduct fluent conversations with people, write code as well as poetry, solve mathematical problems (Frieder et al., 2023) and so on, which has attracted widespread public attention.Table 1 : The overview of our evaluation.However, despite its huge success, we still know little about the capability boundaries, i.e., where it does well and fails. In this work, we are interested in how Chat-GPT performs on the sentiment analysis tasks, i.e., can it understand the opinions, sentiments, and emotions contained in the text? To answer this question, we conduct a comprehensive evaluation on 7 representative sentiment analysis tasks 2 and 17 benchmark datasets, which involves three different settings including standard evaluation, polarity shift evaluation and opendomain evaluation (refer to Figure 1 ). We compare ChatGPT with fine-tuned small language models like BERT (Devlin et al., 2019) and corresponding SOTA models (if any) on each task for reference. We also attempt several popular prompting techniques, such as chain-of-thought (CoT) (Wei et al., 2022) and self-consistency (Wang et al., 2022) , to induce the ability of ChatGPT. The main findings of our work are as follows:➊ ChatGPT demonstrates impressive zero-shot capabilities in sentiment classification tasks such as SC, ABSC and CSI, and can rival fine-tuned BERT, although it still trails behind the domain-specific fully-supervised SOTA models (☞ § 3.2). ➍ ChatGPT seems less accurate on sentiment information extraction tasks like E2E-ABSA and CEE. We observe that ChatGPT can often make reasonable predictions but can not strictly match the dataset annotations. Our human evaluation finds that ChatGPT actually performs more desirable, not as poor as metrics indicate. (☞ § 3.2) ➌ Compared to fully-supervised strong baselines, ChatGPT demonstrates impressive textual emotion inference ability with significantly higher performance on ECE but lower performance on ECPE (☞ § 3.2). ➎ When coping with the polarity shift phenomenon (e.g., negation and speculation), a challenging problem in sentiment analysis, ChatGPT can make more accurate predictions than fine-tuned BERT. (☞ § 3.3) ➏ Compared to training domain-specific models, which typically perform poorly when generalized to unseen domains, ChatGPT demonstrates its powerful opendomain sentiment analysis ability in general, though its performance is quite limited in a few specific domains. (☞ § 3.4) ➐ Few-shot prompting (i.e., equipping with a few random examples in the input) can significantly improve performance across tasks and domains, surpassing fine-tuned BERT in some cases, though still inferior to SOTA models (☞ § 4.1). Applying CoT to the evaluated tasks does not yield gains but diminishes performance. In contrast, self-consistency reliably improves results (☞ § 4.2).In summary, we think that, compared to training a specialized sentiment analysis system for each task, domain and dataset, ChatGPT can already serve as a universal and well-behaved sentiment analyzer.2 Background and Related Work 
large language models: 
 With the emergence of GPT-3 (Brown et al., 2020) , Large language models (LLMs) were spotlighted. They typically have lots of model parameters and are trained on massive volumes of unstructured data at huge computational costs, including but not limited to Gopher (Rae et al., 2021) , Megatron-Turing NLG 530B (Smith et al., 2022) , LaMDA (Thoppilan et al., 2022) , Chinchilla (Hoffmann et al., 2022) , PaLM (Chowdhery et al., 2022) , OPT (Zhang et al., 2022a) , LLaMA (Touvron et al., 2023) , and GPT-4 (OpenAI, 2023) . As a result, given a simple task instruction, they are able to adapt directly to a new task in a training-free manner. In addition to the task instruction, the predictions will be more accurate and controllable if LLMs could be provided some demonstration examples, an ability known as in-context learning (Brown et al., 2020) .Lately, OpenAI has released ChatGPT, a chatbot fine-tuned from GPT-3.5 via reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022) , drawing increasingly great attention. Next, researchers start exploring its abilities and limitations, testing it on various benchmarks (Gilson et al., 2022; Frieder et al., 2023; Guo et al., 2023; Jiao et al., 2023; Zhuo et al., 2023; Zhong et al., 2023; Ye et al., 2023; Laskar et al., 2023) . For example, Bang et al. (2023) evaluate the multitask, multilingual, and multimodal aspects of ChatGPT, Wang et al. (2023a) conduct a robustness evaluation from the adversarial and out-of-domain perspective, and Borji (2023) summarizes 11 categories of failures towards ChatGPT. Related to our work, Zhong et al. (2023) analyze the language understanding ability of ChatGPT on GLUE (Wang et al., 2018) . In this work, we especially concentrate on analyzing its sentiment analysis ability, aiming to answer the question via a rigorous and comprehensive evaluation, i.e., whether ChatGPT can be a good sentiment analyzer.Sentiment analysis seeks to identify people's opinions, sentiments, and emotions in the text, such as customer reviews, social media posts, and news articles (Liu et al., 2005; Liu, 2015) .As one of the most active fields in Natural Language Processing (NLP), it has made rapid progress with the help of deep learning (Zhang et al., 2018; Yadav & Vishwakarma, 2020) . Among the myriad of tasks associated with sentiment analysis, this paper is primarily concerned with 4 representative task categories, including (sentence-level) sentiment classification (SC), aspect-based sentiment analysis (ABSA), comparative opinion mining (COM), and emotion cause analysis (ECA). For ease of understanding, we will briefly introduce these tasks next. SC aims to identify the sentiment polarity of a given text, whether it is positive or negative. ABSA is designed to mine fine-grained aspect terms in the review and determine the sentiment polarity toward each aspect (Liu, 2012; Pontiki et al., 2014; Zhang et al., 2022b) . We mainly focus on aspect-based sentiment classification (ABSC) and End-to-End Aspect Based Sentiment Analysis (E2E-ABSA) among many subtasks in ABSA. COM seeks to identify comparative sentences, extract the comparative elements, and obtain the corresponding comparative opinion tuples (Jindal & Liu, 2006; Liu et al., 2021) . We mainly concentrate on comparative sentences identification (CSI) and comparative element extraction (CEE), i.e., extracting the tuple of (subject, object, comparative aspect, comparison type). The purpose of ECA is to extract the potential cause clauses given the emotion clause or extract the potential pair of emotion clause and cause clause in the text, which correspond to emotion cause extraction (ECE) (Gui et al., 2016) and emotion cause pair extraction (ECPE) (Xia & Ding, 2019) , respectively.In this paper, we are also concerned with two challenging problems in sentiment analysis:polarity shift and open-domain (Zong et al., 2021) . Polarity shift refers to the linguistic phenomenon where the sentiment polarity (positive or negative) of a text shifts over time, context, or with respect to other texts (Li et al., 2010; Xia et al., 2016) . Understanding sentiment polarity shift is crucial for building accurate sentiment analysis systems. As another challenging issue, open-domain sentiment analysis aims to understand the general sentiment of text regardless of the domain, whereas existing sentiment analysis systems typically focus on analyzing the sentiment of texts related to a particular domain (Cambria et al., 2012; Zhang et al., 2015; Luo et al., 2022) . Addressing the above two issues is essential to building robust and effective sentiment analysis systems. In this work, we will examine whether ChatGPT can solve them. 
evaluation: 
 In this section, we will first introduce the evaluation setup ( § 3.1) followed by standard evaluation ( § 3.2), polarity shift evaluation ( § 3.3) and open-domain evaluation ( § 3.4), as illustrated in Figure 1 . As mentioned earlier, the tasks involved in our evaluation are SC, ABSC, E2E-ABSA, CSI, CEE, ECE, and ECPE.Comparison Systems. We compare ChatGPT with the state-of-the-art (SOTA) (if any) models on end-tasks. Since SOTA models typically have some task-specific designs, we also provide the results of a commonly used baseline (e.g., fine-tuned BERT 3 ) on each task for reference. For SC, we adopt the most common practice, i.e., using the final hidden representation of the [CLS] token as the sentence embedding and feeding it into a linear layer for classification. As for ABSC, we concatenate the review sentence and the aspect term via the special token [SEP] and classify the sentiment polarity based on the final hidden representation of [CLS] . We employ the joint tagging scheme (Li et al., 2019) to perform the E2E-ABSA task. For CSI, we report the performance of Multi-Stage BERT derived from (Liu et al., 2021) for reference. For CEE, given the complexity of modeling this task, we reformulate it into a text generation task based on T5-Base similar to GAS (Zhang et al., 2021) , i.e., predicting the sequences of comparison tuples given the input review. We employ PAE-DGL (Ding et al., 2019) and ECPE-2D (Ding et al., 2020) as comparison models for ECE and ECPE, respectively. Unless otherwise specified, the above baseline models are rerun and repeated three times based on our evaluation settings.We mainly use ChatGPT with a specific version of gpt-3.5-turbo-0301 for evaluation in this work, given its lower cost and improved performance (as stated in the OpenAI documentation4 ). We set the temperature to 0, making the outputs mostly deterministic for the identical inputs. Following Jiao et al. (2023) , we ask ChatGPT to generate the task instruction for each task to elicit its ability to the corresponding task. For example, the prompt for E2E-ABSA is "Given a review, extract the aspect term(s) and determine their corresponding sentiment polarity. Review: {sentence}". Due to limited space, please refer to Table 9 and Appendix A.2 for complete prompts and prompts details, respectively. We report the zero-shot results of ChatGPT unless otherwise specified. We manually observe and record the predictions as the responses of ChatGPT do not always follow a certain pattern under the zero-shot setting.Evaluation Metrics. We use accuracy and macro F1 score to evaluate sentiment classification tasks. We employ accuracy as the metric for CSI. For tasks involving elements extraction such as E2E-ABSA and CEE, we employ micro F1 score, i.e., a tuple is regarded as correct if and only if all elements inside it are exactly the same as the corresponding gold label. For ECE and ECPE, we compute the F1 score of cause clauses and emotion-cause clause pairs for evaluation, respectively. 2021) while ♭ represents the results of our implemented GAS-Extraction-style baseline (Zhang et al., 2021) . α, β, γ, and δ denote the results derived from T5-11B (Raffel et al., 2020) , DPL (Zhang et al., 2022c) , RILGNet (Li et al., 2022) and SyMux (Fei et al., 2022) , respectively. The best results are in bold.In this part, we evaluate ChatGPT on 7 representative sentiment analysis tasks and report its results on related benchmark datasets.Datasets. We choose SST-2 (Socher et al., 2013) as the testbed of SC. Since the test set of SST-2 is not public, we use its validation set for evaluation. We employ the SemEval 2014-ABSA Challenge Datasets (Pontiki et al., 2014) to evaluate the ability of ChatGPT to ABSA. For CSI and CEE, we employ the Camera dataset (Kessler & Kuhn, 2014; Liu et al., 2021) . For ECE and ECPE, we adopt the Emotion Cause Dataset (Gui et al., 2016; Xia & Ding, 2019) and sample 100 examples from this. Except as noted above, we evaluate the remaining datasets on the full test set. The statistics are shown in the third column of Table 2 .Results. The comparison results are shown in Table 2 . Overall, ChatGPT demonstrates highly competitive sentiment analysis performance compared with baseline models, albeit often being far inferior to SOTA models. Specifically, we observe that ChatGPT is on par with fine-tuned small language models (i.e., BERT) in sentiment classification tasks, despite being inferior to SOTA models. Secondly, when evaluated on E2E-ABSA, the performance of ChatGPT is indeed inferior to fine-tuned BERT, and the performance gap varies across domains. We speculate that the poorer performance on 14-Laptop is due to the presence of more proprietary terms and specific expressions in this domain. Thirdly, for the challenging COM tasks (i.e., CSI and CEE), which typically involve implicit expressions, although achieving reasonable performance on CSI, it exhibits extremely undesirable performance on CEE. These results are far from satisfactory compared with fine-tuned baselines. Finally, ChatGPT exhibits reasonably good emotion analysis ability. We find that ChatGPT can comprehend the given document thoroughly, for instance, being capable of identifying multiple reasons and extracting emotion clauses and cause clauses even when they are distant. We also observe that ChatGPT can make some reasonable predictions, whereas the corresponding annotations are not in the dataset. Human Evaluation. In light of the poor performance on certain tasks, we naturally raise a question: are the predictions of Chat-GPT truly unreasonable? To acquire a more profound comprehension of the prediction results from ChatGPT, we conduct a human evaluation on E2E-ABSA and CEE owing to their unsatisfactory performance. Upon observation of the predicted results, Chat-GPT has made many plausible predictions. However, these either did not exactly match the ground truth, or there are no corresponding annotations in the dataset, leading to a subpar performance on the exact-match evaluation. For E2E-ABSA, even though the predictions of ChatGPT are not accurate based on exactmatch evaluation, it can still infer some highly reasonable aspect categories for the aspect terms thanks to its text generation paradigm. This also demonstrates its ability to identify implicit expressions to some extent. For instance, given the sentence "Runs real quick.", the ground truth is "(Runs, positive)" whereas the prediction of ChatGPT is "(Speed, positive)".For CEE, the predictions of ChatGPT express the same meaning as the ground truth but in an inconsistent form. As an example, the meaning expressed by ChatGPT is "The SD800 is better than the SD700.", whereas the ground truth meaning is "The SD700 is worse than the SD800.", where the "SD700" and "SD800" refer to the products being compared. From the perspective of sentiment analysis application, this is equally effective. Therefore, to align the predictions of ChatGPT with the annotation standard of existing datasets, we follow a few simple rules for human evaluation5 :☞ For any extra generated tuples, if they are reasonable but absent from the annotations, we will remove them from the prediction results. Otherwise, we will keep them.☞ We also consider an aspect-sentiment or comparative opinion tuple correct if the boundary of aspect or entity is predicted incorrectly but unambiguously, and the predicted sentiment or preference is also correct.☞ We also regard a prediction that paraphrases the ground truth to be correct, given the text generation paradigm.As shown in Table 3 , it is surprising but reasonable to observe that the zero-shot performance of ChatGPT is boosted by 19% (average) and 42% on E2E-ABSA and CEE, respectively, compared to the original results. These results suggest that ChatGPT's predictions are often reasonable, even though they may not strictly adhere to the dataset annotations, leading to the appearance of poorer performance.Case Study. We also conduct the qualitative analysis for the predictions of ChatGPT. Due to the limited space, please refer to Appendix A.5.Comprehending the phenomenon of polarity shift in sentiment analysis is crucial for developing robust and reliable sentiment analysis systems. In this part, we evaluate the ability of ChatGPT to cope with the polarity shift problem. Specifically, we mainly focus on the situations of negation and speculation and consider two sentiment classification tasks, SC and ABSC.Since there are few datasets tailored to polarity shift for SC, we derive two subsets from SST-2 validation set using a heuristic rule for the evaluation of negation and speculation, namely SST-2-Negation and SST-2-Speculation. In short, it entails identifying whether a sentence contains any negation or speculation words. For instance, we assign a sentence to the negation evaluation subset if it includes the word "never". More details are provided in Appendix A.3. As for ABSC, we adopt the 14-Res-Negation, 14-Lap-Negation, 14-Res-Speculation, and 14-Lap-Speculation introduced by Moore & Barnes (2021), which are annotated for negation and speculation, respectively. The statistics are shown in Table 10 .Baseline Details. Generally, we fine-tune BERT on the original training set (e.g., SST-2) and evaluate on polarity-shifting test sets, e.g., SST-2-Negation and SST-2-Speculation. Results. We conduct experiments on six evaluation datasets, and the comparison results are shown in Table 4. Compared to fine-tuned BERT, ChatGPT exhibits greater robustness in polarity shift scenarios. Essentially speaking, the polarity shift evaluation we conduct can be characterized as an out-of-distribution (OOD) evaluation scenario. Not surprisingly, we observe that fine-tuned BERT experiences varying degrees of performance degradation across datasets compared to standard evaluation results. In comparison, ChatGPT is more robust, especially on ABSC, where ChatGPT outperforms finetuned BERT by 10% in terms of average accuracy and 8% in terms of average F1 score. Furthermore, we also find that the speculation case in polarity shift appears more challenging than the negation case, as the results of the former is poorer.Case Study. We conduct qualitative analysis for the predictions of ChatGPT in the case of polarity shift. Refer to Appendix A.6 for details.Existing systems are typically trained on specific domains or datasets, leading to suboptimal generalization performance when dealing with unseen domains. However, an ideal sentiment analysis system could be applied to data from diverse domains. In this part, We report the domain-specific fully-supervised results (in gray) of BERT for reference. We also report the human evaluation results ("+ Human") of ChatGPT for reference. The best results (except for fully-supervised results and human evaluation results) are in bold.we evaluate the capability of ChatGPT to handle open-domain sentiment analysis tasks (i.e., ABSC and E2E-ABSA).As there is currently no widely used open-domain evaluation dataset, we sample 30 examples from each domain of existing 10 ABSA datasets according to the original data distribution, resulting in a total of 300 samples both for ABSC and E2E-ABSA. The ten datasets involved are Restaurant (Pontiki et al., 2014) , Laptop (Pontiki et al., 2014) , Device (Hu & Liu, 2004) , Service (Toprak et al., 2010) , Books, Clothing, Hotel (Luo et al., 2022) , Twitter (Dong et al., 2014) , Financial News Headlines (Sinha et al., 2022) , METS-CoV (Zhou et al., 2022) , covering various domains such as restaurant reviews, product reviews, social media, finance, and medicine. Note that Books, Hotel, and Clothing are originally documentlevel ABSA datasets with hierarchical entity-aspect-sentiment annotations. We randomly sample 30 sentences from each dataset and only use the aspect-sentiment annotations.Baseline Details.To simulate the open-domain setting, we hold out some datasets, finetune BERT on the remaining datasets, and select checkpoints based on the mixture of the corresponding validation sets. Specifically, we set the following settings: (1) single-source: the model is trained on one dataset then evaluated on all datasets. Here, we choose Restaurant and Laptop as the testbed;(2) multi-source: the model is trained sequentially on nine datasets and then evaluated on the remaining one. Finally, we also fully-supervisedly fine-tune BERT and report the results for reference.Results. In terms of ABSC, ChatGPT demonstrates a more compelling open-domain ability than BERT despite being fine-tuned on this task. As shown in Table 5 , ChatGPT matches or even outperforms multi-domain fine-tuned BERT on 7 out of 10 domains in sentiment classification metrics (accuracy or macro-F1) while surpassing it by 8% in accuracy and 7% in F1 score on average across 10 datasets. It is worth mentioning that ChatGPT even performs comparably to full-supervised BERT, which shows its compelling generalization ability. Interestingly, fine-tuning on multiple domains does not necessarily lead to improved performance. For example, we observe that it results in a significant decrease in performance in certain datasets such as Finance and METS-Cov. Table 6 shows ChatGPT exhibits moderate performance on E2E-ABSA under the exact-match evaluation despite in the zero-shot manner. For example, it even beat BERT models on some domains (e.g., restaurant, service, and finance), which are fine-tuned on the nine domains.Despite its success, we can observe that the performance of ChatGPT is quite poor in some domains, especially social media relevant domains (i.e., twitter, finance, METS-Cov), which suggests that improving performance on these domains remains challenging. It should be noted that due to the use of exact-match evaluation, the actual results of ChatGPT may not be as poor as they appear. Similarly, through our human evaluation (as introduced in § 3.2), we can observe that ChatGPT has achieved an average performance improvement of 18% across domains, indicating that many predictions are inherently reasonable. Overall, these results can demonstrate decent open-domain capabilities of ChatGPT, albeit with poor results in a few domains.Case Study. We conduct qualitative analysis through four examples of ChatGPT on Books and METS-Cov, corresponding to the books and medicine domain, as shown in Figure 5 . We also provided a detailed analysis in Appendix A.7. ChatGPT Fine-tuned BERT SOTA Table 7 : Few-shot prompting results on ABSC and E2E-ABSA tasks. 
advanced prompting techniques: 
 Given that ChatGPT still lags behind fine-tuned small language models (e.g., BERT) in some tasks and domains to a certain extent, we endeavor to seek help from some advanced prompting techniques to further elicit the capabilities of ChatGPT.Here, we adopt the ABSA tasks as the testbed.We randomly select a few examples from the training dataset used for demonstration and concatenate them with the target input to prompt Chat-GPT, a technique also known as incontext learning (Brown et al., 2020) . We conduct few-shot prompting experiments on ABSC and ASPE with k (i.e., 1, 3, 9 and 27) examples. To reduce the variance caused by the sampling of demonstration examples, we adopt three random seeds for sampling to conduct experiments and report the average performance. We compare the resulting performance with fully-supervised BERT and SOTA.As presented in Figure 7 , few-shot prompting can significantly improve the performance across tasks and datasets, even surpassing fine-tuned BERT in some cases. It improves the classification performance by 7% and 10% F1 score for ABSC on 14-Restaurant and 14-Laptop, respectively, with 27 demonstration examples. We can also observe certain improvements on ASPE, although the improvement curve is relatively flat. We also provide a case study, as shown in Figure 1 .Prompting Methods 14-Res. 14-Lap. Although few-shot prompting clearly improves the performance on ABSC, the performance on E2E-ABSA still lags far behind fine-tuned BERT. We attempt more advanced techniques, i.e., manual few-shot chainof-thought (CoT) prompting (Wei et al., 2022) and self-consistency (Wang et al., 2022) on this task, to further elicit the ability. More details are provided in the Appendix A.8.As shown in Table 8 , we observe that equipping standard few-shot prompting with chain-ofthought does not bring the expected gains, but rather lead to a noticeable drop. This similar phenomenon was also observed in Ye & Durrett (2022) and Wang et al. (2022) but contrary to the observations in Zhong et al. (2023) . We speculate that this may depend on the evaluation tasks. In contrast, self-consistency clearly improves the performance of few-shot prompting, regardless of whether CoT is equipped, once again confirming the effectiveness of this technique (albeit at the cost of increased inference complexity). Regrettably, while effective, it is still inferior to fine-tuned BERT. Future work could explore more efficient prompting methods, such as retrieval-based ones (Liu et al., 2022; Shi et al., 2023, inter alia) . 
conclusion: 
 In this work, we evaluate ChatGPT on a range of test sets and evaluation scenarios and compare its performance to fine-tuned BERT, exploring its capacity boundaries in various sentiment analysis tasks. ChatGPT exhibits magnificent zero-shot sentiment analysis abilities (e.g., sentiment classification, comparative opinion mining and emotion cause analysis), even matching with fine-tuned BERT and SOTA models trained with labeled data in respective domains at times. Compared to fine-tuned BERT, ChatGPT can handle the polarity shift problem more effectively in sentiment analysis and exhibits good performance in open-domain scenarios. In addition, we also explore some popular prompting techniques to further induce the capability of ChatGPT. Through experiments, we validate the effectiveness of them on sentiment analysis tasks and provide our findings. We aspire to galvanize future research through our empirical insights in sentiment analysis, LLMs and beyond. This work has several limitations: (1) Data Leakage: Difficulty in ensuring ChatGPT hasn't seen test data (Dodge et al., 2021; Golchin & Surdeanu, 2024; Balloccu et al., 2024; Xu et al., 2024; Sainz et al., 2024) . (2) Prompt Design: Lack of extensive prompt experimentation assumes users prefer simplicity, yet improvements could showcase ChatGPT's adaptability.(3) Limited Evaluation: Focus on ChatGPT with limited comparison to other models due to resource constraints, narrowing insights into LLM strengths and weaknesses. Looking ahead, promising directions include (1) New Benchmarks: Developing real-world benchmarks and better evaluation methods to accurately assess LLMs (Zheng et al., 2023; Huang et al., 2024; Jimenez et al., 2024; Xie et al., 2024) . (2) Implicit Sentiment Analysis: Enhancing understanding and analysis of texts with implicit sentiments through comprehensive benchmarks. (3) Domain-Specific Enhancement: Improving ChatGPT's performance in specific domains, such as medicine (Singhal et al., 2023) , math (Wang et al., 2023b; Azerbayev et al., 2024 ), code (Rozière et al., 2023) and low-resource languages (Dou et al., 2024; Nguyen et al., 2024) , with targeted training for broader applicability. Due to space constraints, detailed discussions on limitations and future work directions are available in Appendix B. 
 
 Appendix: Prior to our work, there were some early studies that evaluated ChatGPT's performance on common NLP tasks, but they only covered basic sentiment classification tasks, such as SST-2 (Zhong et al., 2023) . Our work, however, is dedicated to a more comprehensive and thorough evaluation for ChatGPT within the sentiment analysis field, representing, to our knowledge, the first of its kind in this field. After our work was completed (over a month later), a new study emerged, evaluating more tasks like ASTE and ASQP within ABSA and also providing few-shot experimental results and human evaluations (Zhang et al., 2023) . However, this study did not evaluate critical issues in sentiment analysis such as sentiment polarity shift and open-domain evaluation, which our work does address. Although different works may have their own focuses, we believe they all contribute significantly to providing more comprehensive evaluations in this field.Following Jiao et al. (2023) , we ask ChatGPT to generate the task instruction for each task to elicit its ability to the corresponding task. Taking the E2E-ABSA task as an example, our query is:Please give me three concise prompts for eliciting your ability to perform Aspect-Based Sentiment Analysis (i.e., extract the aspect terms and sentiment polarity). There is no need to give examples and do not limit the prompts to a specific product or domain.Then, we examine the generated three prompts on a small-scale (e.g., 50 examples) example set driving from the corresponding training set. We select the best and most reasonable one6 according to the results. 7 The final prompts adopted for each task are shown in Table 9 . During the evaluation process, we feed a prompt and corresponding test example to ChatGPT and obtain a generated response. We manually observe and record the results as the responses do not follow a certain pattern.As previously mentioned, we drive SST-2-Neg and SST-2-Spec from SST-2 by detecting whether a sentence contains any negation or speculation words. The seed words adopted are shown in Table 11 . And the statistics of involved datasets are shown in Table 10 .The exact-match metric has limitations for evaluating generative models like ChatGPT since they can produce reasonable outputs not matched to references. To better characterize ChatGPT's capabilities despite this, we manually refine its outputs before comparing them to those of baselines. We acknowledge this may seem unfair compared to unrefined baselines. However, our goal is to account for the limitations of the exact-match, not to boost ChatGPT's results unfairly. To further illustrate the rules we use as more intuitive and easier to understand, we provide some examples from the E2E-ABSA task, as shown in Table 12 .In this part, we conduct the qualitative analysis on ABSA tasks, COM tasks, and ECA tasks. Case Study on ABSA. We conduct the qualitative analysis through two examples. Specifically, as shown in Figure 1 , we present the results generated by ChatGPT for two test examples under zero-shot and few-shot settings, respectively. Given the example "I did swap out the hard drive for a Samsung 830 SSD which I highly recommend", there are multiple aspect terms with different sentiment polarities in a sentence (e.g., the sentiment polarityThe following product review contains comparison of products (including implicit products): It seems to get less light to the sensors than my old 4MP A80. Extract the subject and object of comparison, tell me which aspect of products is being compared, and tell me if the author of the review thinks the subject is better or worse than or similar to or different from the object. If multiple comparisons exist, output multiple comparisons.Yes, a comparison exists in the product review. Subject of comparison: "It" (referring to the product being reviewed) Object of comparison: "my old 4MP A80" Aspect of products being compared: amount of light received by the sensors Author's opinion: The author thinks that the old 4MP A80 received more light to the sensors than the product being reviewed. Therefore, the author thinks that the old product is better than the new one in terms of the amount of light received by the sensors.Ground Truth: (It, my old 4MP A80, light to the sensors, worse) ✅The following product review contains comparison of products (including implicit products): Selling points -In which I explain why I bought the 5D My main reason for me upgrading from 350D to 5D ( and skipping the 30D ) was the full-frame sensor. Extract the subject and object of comparison, tell me which aspect of products is being compared, and tell me if the author of the review thinks the subject is better or worse than or similar to or different from the object. If multiple comparisons exist, output multiple comparisons.Comparison: 350D and 5D Subject: 350D Object: 5D Aspect: Full-frame sensor Author's opinion: The author thinks that the 5D is better than the 350D because of the full-frame sensor.Ground Truth: (5D, 350D, full-frame sensor, better) ✅ Does any comparison of products (including implicit products) exist in the product review "As for comparisions with other Canon Powershot cameras, the Powershot SD630 doesn't have a view finder, but has a larger LCD monitor in its place." ? If so, outputs 'TRUE', else outputs 'FALSE'.Yes, there is a comparison of products in the product review. The review compares the Powershot SD630 with other Canon Powershot cameras. Therefore, the output is 'TRUE'.Ground Truth: TURE ✅ Does any comparison of products (including implicit products) exist in the product review "However, focus accuracy was not as impressive." ? If so, outputs 'TRUE', else outputs 'FALSE'.It is not clear from the given sentence whether there is a comparison of products or not. The sentence only mentions the focus accuracy of a single product, without any reference to other products. Therefore, the output would be 'FALSE'. We conduct qualitative analysis through four examples of ChatGPT on Books and METS-Cov, corresponding to the books and medicine domain, as shown in Figure 5 . Regarding two examples in the first column, ChatGPT has difficulty accurately predicting sentiment in books domain since this domain usually contains unique expressions and plot descriptions that differ from typical product reviews. We also find that ChatGPT often generates reasonable aspect-sentiment pairs in the E2E-ABSA task, such as the "(chapter creations, negative)" (in the bottom left example). However, as we mentioned earlier, they are not originally annotated in the dataset, which is also an important reason affecting the performance of E2E-ABSA. As for the two examples in the second column, when dealing with a rare domain like medicine, ChatGPT can accurately determine the sentiment polarity of the given aspect term, but accurately extracting aspect-sentiment pairs remains a challenge. model performance. In this paper, we adopt human evaluation to alleviate this issue. (2) Implicit sentiment analysis. Implicit expression is a very common linguistic phenomenon. For example, "I know real Indian food and this wasn't it" does not contain explicit opinion words. Moreover, accurate judgment often requires common sense or domain knowledge.Our experiments also confirm that large language models generally perform poorly on implicit sentiment analysis (See Appendix 14 for results). Meanwhile, constructing comprehensive benchmarks for implicit sentiment analysis could be a promising direction.(3) Enhancing the model capabilities in specific domains. As shown in Table 5 and Table 6 , we can see that the performance of ChatGPT is not satisfactory on many domains (such as books and twitter). Therefore, in the future, we could improve the performance on certain domains through domain-specific training, as demonstrated by recent efforts (Singhal et al., 2023; Wang et al., 2023b; Azerbayev et al., 2024; Rozière et al., 2023; Dou et al., 2024; Nguyen et al., 2024, inter alia) .All models use BERT-base-uncased version and are coupled with a linear layer if necessary.https://platform.openai.com/docs/models/gpt-3-5See Appendix A.4 for examplesWhen necessary, we would make minor adjustments to the prompts.We observe that different prompts have little effect on the performance. We also conducted three experiment repetitions and found minimal deviation in the results. Considering the cost of API calls, we only run the experiment once for the final evaluation unless otherwise specified. 


Decision: The paper offers a comprehensive evaluation of ChatGPT's performance across a wide range of sentiment analysis tasks, including Sentiment Classification (SC), Aspect-Based Sentiment Classification (ABSC), and Emotion-Cause Pair Extraction (ECPE), among others. The experiments are well-detailed and include advanced prompting techniques such as few-shot prompting and chain-of-thought prompting. The inclusion of human evaluation to analyze low-performing tasks adds depth to the analysis. The experimental setups are scientifically valid, and the results are well-documented in the appendices. The study's thoroughness in testing multiple sentiment analysis subtasks using diverse datasets, including major ones like SST-2, Restaurant, and Laptop datasets, provides valuable insights into the capabilities of ChatGPT as a sentiment analyzer.

The study's primary limitation is its exclusive focus on ChatGPT, neglecting other large language models (LLMs) such as Llama and Mistral, which could provide a more comprehensive evaluation of LLM capabilities in sentiment analysis. The analyses are criticized for being too broad, making the claims that ChatGPT can serve as a universal sentiment analyzer seem overstated. Additionally, the paper does not explore results in languages other than English, limiting the generalizability of its conclusions. The use of BERT as a baseline, instead of a more robust model like RoBERTa, further limits the comparative analysis. The absence of evaluations on diverse datasets like the SFU Review Corpus and microblogging genres also restricts the study's applicability. Overall, the paper reads more like a technical report due to its broad and somewhat unsystematic analyses.

[comments from the PCs] As much as possible, please follow up on the AC recommendation to include an open-weights LLM, such as Llama or Mistral.

Reviews: review: paper_summary: paper_summary: The paper is devoted to testing ChatGPT in various sentiment analysis tasks.Evaluation is done  on 7 sentiment analysis tasks covering 17 benchmark datasets. ChatGPT is compared with fine-tuned BERT and corresponding state-of-the-art approaches. The tasks include:  Sentiment Classification (SC), Aspect-Based Sentiment Classification (ABSC), End-to-End Aspect-Based Sentiment Analysis (E2E-ABSA), Comparative Sentences Identification (CSI), Comparative Element Extraction (CEE), Emotion Cause Extraction (ECE), and Emotion-Cause Pair Extraction (ECPE).
strength_weakness: strength_weakness: 1. Among all current exisiting LLM, only ChatGPT is tested.
2. All results are obtained for English, the conclusions can be quite different for other languages.
questions: questions: No
paper_summary: paper_summary: This paper investigates ability of ChatGPT for various sentiment analysis tasks. The authors tried zero-shot and few-shot experiments with ChatGPT as well as some specific evaluations for samples with negations. The authors compared ChatGPT with BERT and SOTA baselines. The paper claims that ChatGPT can already serve as a universal and well-behaved sentiment analyzer.

In my opinion, the paper contributes to the thorough studies for sentiment analysis with the aim to evaluate ability of ChatGPT, while there exists limitations, such as the lack of consideration of other LLMs. In addition, I think the results are not surprising given the strong evidences of recent papers describing the ability of ChatGPT in other text classification and relation extraction tasks. I also would like to say that this paper is more likely about technical report because the analyses and claims are too broad.
strength_weakness: strength_weakness: - The findings of this paper are specific for ChatGPT, limiting generalizability for other LLMs. Given recent rapid trends in this field, using open-sourced LLMs such as Llama and Mistral would be interesting. 
- Related to above, any findings can be obtained from this paper, but they may not be systematic results due to the limited generalizability.
- The authors use BERT as a baseline. However, I believe at least RoBERTa is a suitable baseline among fine-tuned small language models. It is also known that RoBERTa is a strong baseline in sentiment analysis tasks [1]. 
- The analyses in this paper are too broad in my opinion to convey the major claim that "ChatGPT can already serve as a universal and well-behaved sentiment analyzer".  Also, I think the results are not surprising given the strong evidences of recent papers describing the ability of ChatGPT in other text classification and relation extraction tasks. I also would like to say that this paper is more likely about technical report because the analyses and claims are too broad. 

[1] Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa, Junqi Dai, Hang Yan, Tianxiang Sun, Pengfei Liu, Xipeng Qiu, NAACL 2021.
paper_summary: paper_summary: The paper is an evaluation of the ability of GPT3.5 in different tasks of sentiment analysis. In particular de evaluation is interesting, and the conclusions related to the self-consistency prompt enhance the classification results. However, I miss a wider evaluation, since the aim of the paper is the evaluaation of the sentiment classification capacity of LLMs, so:

1. Why didn't the authors evaluate at least an additional LLM (Llama or Mistral for instance)?
2. Regarding the open domain evaluation, why didn't the authors use the SFU Review Corpus [1] that has 8 different opinion domains?

References:
[1] Taboada, M., Anthony, C., & Voll, K. D. (2006, May). Methods for Creating Semantic Orientation Dictionaries. In LREC (pp. 427-432).
strength_weakness: strength_weakness: - The lack of assessing other LLMs.
- Tha lack of using more datasets, as the SFU Review corpus, and also datasets of microblogging genre.
- The lack of evaluation in other language than English, for instance the Spanish language.
questions: questions: Why didn't the authors evaluate at least an additional LLM (Llama or Mistral for instance)?
Why didn't the authors use the SFU Review Corpus [1] that has 8 different opinion domains?
Why didn't the authors consider other language different than English?

References:
[1] Taboada, M., Anthony, C., & Voll, K. D. (2006, May). Methods for Creating Semantic Orientation Dictionaries. In LREC (pp. 427-432).
paper_summary: paper_summary: The authors assess ChatGPT across various test sets and evaluation scenarios in order to delineate its potential in different sentiment analysis tasks. Specifically, they perform an assessment across 7 representative sentiment analysis tasks and 17 benchmark datasets, encompassing three distinct evaluation scenarios: standard evaluation, polarity shift evaluation, and open-domain evaluation. On those tasks, they compare ChatGPT against fine-tuned small language models such as BERT  along with any corresponding state-of-the-art models available for reference on each task. Finally, from these experiments, they conclude that ChatGPT can already function as a versatile and effective sentiment analyzer.
strength_weakness: strength_weakness: The paper might have considered evaluating additional Large Language Models beyond GPT-3.5; however, I still believe it encompasses a satisfactory amount of work and quality to merit acceptance at COLM.

Comments: title: Thank you for your response
comment: Thanks for the reply.
Thank you. I am happy that the results of RoBERTa and other LLMs will be described in the revised paper. The response does not make my evaluation of this paper change, so I would like to keep the score.