SOUL: Towards Sentiment and Opinion Understanding of Language
Authors: Yue Deng
Wenxuan Zhang
Sinno Jialin Pan
Lidong Bing
Venue: EMNLP 2023 Main (2023)
Keywords: sentiment analysis
sentiment classification
sentiment and opinion understanding
TL;DR: This paper proposes a new task called Sentiment and Opinion Understanding of Language (SOUL).

Abstract: Sentiment analysis is a well-established natural language processing task, with sentiment polarity classification being one of its most popular and representative tasks. However, despite the success of pre-trained language models in this area, they often fall short of capturing the broader complexities of sentiment analysis. To address this issue, we propose a new task called Sentiment and Opinion Understanding of Language (SOUL). SOUL aims to evaluate sentiment understanding through two subtasks: Review Comprehension (RC) and Justification Generation (JG).  RC seeks to validate statements that focus on subjective information based on a review text, while JG requires models to provide explanations for their sentiment predictions. To enable comprehensive evaluation, we annotate a new dataset comprising 15,028 statements from 3,638 reviews. Experimental results indicate that SOUL is a challenging task for both small and large language models, with a performance gap of up to 27\% when compared to human performance. Furthermore, evaluations conducted with both human experts and GPT-4 highlight the limitations of the small language model in generating reasoning-based justifications. These findings underscore the challenging nature of the SOUL task for existing models, emphasizing the need for further advancements in sentiment analysis to address its complexities. The new dataset and code are available at \url{https://github.com/DAMO-NLP-SG/SOUL}.

SOUL: Towards Sentiment and Opinion Understanding of Language
Yue Deng" 1.2 Wenxuan Zhang 1.3 Sinno Jialin Pan
Lidong Bing!. 1.
'DAMO Academy, Alibaba Group, Singapore
NNayyang Technological University, Singapore "The Chinese University of Hong Kong
Huppa Lab, 310023, Hangzhou, China
{yue. deng, saike. zwx, 1. bingg)allllln beaaabbaabinnee inc. sinnopaněcuhk. edu. hk
Abstract
As the raptor graphics weren't amazing enough, award-winning editors continued to use the exact same shot throughout the entre movie. even when the background didn't actually match with the setting of the scene. Wow, what genius. And while the movie full of plat-holes ... wil never forget the brillinnce that is Raptor Planet. Thank you Sci-Fi for another classic.
Review:
Sentiment analysis a well-cstablished natu- ral language processing task, with sentiment polarity classification being one of its most popular and representative tasks. However, de- spite the success of pre- pre-trained language mod- els in this area, they often fall short of cap- turing the broader complexities of sentiment analysis. To address this issue, we propose a new task called Sentiment and Opinion Un- derstanding of Language (SOUL.. SOUL aims to evaluate sentiment understanding through two subtasks: Review Comprehension (RC) and Justification Generation (JG). RC seeks to validate statements that focus on subjective information based on review text, while JG re- quires models to provide explanations for their sentiment predictions. To enable comprehen- sive evaluation, we annotate new dataset com- prising 15,028 statements from 3,638 reviews. Experimental results indicate that SOUL is challenging task for both small and large lan- guage models, with performance gap of up 27% when compared human performance. Furthermore, evaluations conducted with both human experts and GPT-4 highlight the lim- itations of the small language model in gen- erating reasoning- -based justifications. These findings underscore the challenging nature of the SOUL task for existing models, empha- sizing the need for further advancements in sentiment analysis to address its complexities. The new dataset and code are available at https://gite http:://ggttuuu.ccoo///-----/ Com//DAMO-LP-SSG/SOO
Label: negative
Statement: The reviewer liked the raplors because they were terrifying
Label: not-given
Justification: reviewer didn't say whether he liked disiiked the raptors in the movie.
Statement: The reviewer thinks the editors are smart for using same repeatedly.
Label: false
Justification: The reviewer sarcastically praised the editors as
"genius" for using the same shot repeatedly throughout the entire movie, even when it didn't match the scene's setting, indicating that the movie had poor editing.
SOUL
Figure I:
Examples from SOUL. We also show tradi- tional sentiment classification task for comparison.
classifying a given text like customer review to a pre-defined sentiment label, such as positive, neg- ative, or neutral (Pang and Lee, 2005; Maas et al., 201 1; Socher et al., 2013; Zhang et al., 2015). With the advent of pre-trained language models, espe- cially the recent large language models (LLMs), remarkable performance has been achieved on SC which sometimes even surpasses human perfor- mance (Yin et al., 2020; Ke al., 2020; Zhang et al., 2021: Fan al., 2022; Wang et al., 2023; Zhang et al., 2023). This leads to a common be- lief that SC, and sentiment analysis in general, has reached its saturation.
However, SC not equivalent to the broader field of sentiment analysis as it does not require a deep understanding of the underlying sentiments and opinions expressed in the text. To determine the overall sentiment orientation, model can sim- ply rely on superficial textual features, such the presence of specific words or phrases indicating positivity or negativity (Wulczyn et al., 2017: Wang and Culotta, 2020, 2021; 1; Moon et al., 2021; Choi et al., 2022). Therefore, even if a model demon-
1 Introduction
Sentiment analysis, well-established natural lan- guage processing task, aims analyze and under- stand subjective information from text (Liu, 2015). One of its most popular and representative tasks is sentiment classification (SC), which involves
Yue Deng under Joint PhD Program between Alibaba and Nanyang Technological University. Wenxuan Zhang is the corresponding author.

---

strates satisfactory performance in sentiment classi- lication, may not fully capture the subtle nuances of sentiment in languages, such as mixed senti- ments towards different aspects, motivation of the expressed opinions, and possible outcomes of such sentiments, etc. In order to assess whether a model can truly comprehend the sentiment and accurately interpret intricate emotions, is essential to adopt more comprehensive approach that extends beyond merely predicting the polarity of sentiment.
the statement is false and identifies the sarcastic meaning conveyed by the reviews.
To facilitate such an investigation, we carefully annotate a new dataset based on common review corpora. In total, it consists of 15,028 statements across 3,638 reviews. Each statement is also an- notated with label and the corresponding justi- fication. We extensively benchmark SOUL with both small language models (SLMs) trained with the complete training set and also LLMs under the zero-shot setting. Our experimental results indi- cate that SOUL is a challenging task that demands a deep understanding of sentiment, with perfor- mance gap of up to 27% when compared to human performance. In addition, based on comprehen- sive evaluations conducted by both human experts and the GPT-4 model, it has been observed that SLMs have demonstrated proficiency in validating statements but struggle with generating reasoning- based justifications, indicating significant potential for enhancement in their comprehension of senti- ment. In comparison, ChatGPT's strength lies in producing well-reasoned justifications, showcasing its powerful aentiment-unnersttannde ability. How- ever, there is still room for improvement regarding the overall accuracy, originality, and conciseness of ChatGPT'ss responses. Overall, we believe SOUL will advance sentiment analysis and encourage the creation models capable of understanding senti- ments at a human-like llke proficiency.
To this end, we introduce new sentiment analy- sis task, namely Sentiment and Opinion Under- standing of Language (SOUL). Our inspiration comes from reading comprehension tasks, which assess human understanding of a passage by asking to judge the validity of a statement. Similarly, we adopt the form of verifying comprehension state- ments regarding opinionated review text. We also generate justifications for such predictions as means of testing the sentiment understanding ca- pability of models. As shown in Figure 1, given a review text, as well as statements that focus on sub- jective information discussed in the review, SOUL features two novel subtasks: Review Comprehen- sion (RC) and Justification Generation (JG).
Specifically, the RC task aims to determine the given statement is true, false, or not-given based on the review, answering the question of what the sentiment is. While this task still involves classification format, it can cover broad range of sentiment phenomena with the flexibility create statements focusing on diverse subjective aspects of the text. This flcxibility breaks the restriction of SC purely focusing on sentiment polarity and allows for the introduction of more complex sentiment problems. In Figure 1, the reviewer's sentiment towards the raptor graphics lacks specific reasons, making it diffieult for simple pattern matching model to accurately predict the first statement as not-given without contextual understanding. The second statement Figure 1 also presents chal- lenge for models in detecting sareasm. The JG task, on the other hand, seeks to provide an explanation for the rationale behind the model's interpretation of sentiment, answering the question of why the sentiment is predicted. By generating justifica- tions for its predicted label, the model is forced to consider the context and nuances of the input text, rather than relying solely on superficial features such as individual words or phrases. For example, the second justification in Figure explains why
2 SOUL
2.1 Task Formulation
Let t be an opinionated text item (e.g., a prod- uet review); 8 be a textual statement about the subjective information in the text; I E [true, false, not not-given] be the label of s; be the justification for 1; f be a model.
Review Comprehension The objective of RC is to determine the validity of the statement in relation to review t. This involves classifying the statement s as either true, false, or not not-given:
st) +
(1)
To accomplish this task effectively, model must fully comprehend the subjective information pre- sented in both the review and the statement, and subsequently judge the validity.
Justification Generation JG aims to generate predictions and justificatiins j jointly:

---

s(t,s) +1,
(2)
# statements
Split
reviews
True
False
Not-given
Total
The purpose is to enable the model to generate a jus- tification that explains its predicted label, thereby helping us to examine whether the model has truly understood the sentiment.
Train
2,182
3,675
3,000
2.159
8,834
Dev Test
365
617
503
361
1,481
1,091
1,956
1,664
1,093
4,713
Total
3,638
6,248
5,167
3,613
15,028
2.2 Dataset Construction
Table 1: Statistics of SOUL dataset.
Data Collection We utilize review texts from two corpora: Yelp (Zhang et al., 2015) and IMDb (Maas et al., 2011). The Yclp dataset is a collec- tion of business reviews from the Yelp website, while the IMDb corpus consists of movie and TV show reviews from the IMDb website. These two datasets cover various review types and are widely used existing sentiment analysis research, e.g., classifying the sentiment polarity of a given review. Therefore, we also take them as our data source for constructing subjective statements.
to the original labels, aiming to evaluate the quality of the constructed statements. In cases of conflict- ing classifications, an expert annotator is consulted to resolve the discrepancies and assign a final label. In addition, annotators are instructed to categorize statements simple, medium, or hard to deter- mine their difficulty level. Reviews containing only simple statements are excluded maintain an ap- propriate level of challenge.
Dataset Statistics The SOUL dataset comprises 15,028 statements related to 3,638 reviews, result- ing in average of 4.13 statements per review. To create training, development, and test sets, we split the reviews in ratio of 6:1:3, respectively. Detailed statisties can be found in Table 1.
Statement and Justification Annotation Firstly, we instruct annotators to propose several statements focusing on various subjective information given a review. To achieve this goal, we request annotators to focus on multiple crucial sentiment elements, in- cluding the sentiment of opinion, sentiment target, opinion holder, the reason for the opinion, customer intent, etc (Liu, 2015). Annotators are instructed to delve beyond the surface-level content and generate more challenging statements that require a deeper level of sentiment and opinion understanding abil- ity. For instance, simply describing the user does not like the product is discouraged, but statements focusing on mixed sentiments towards various as- pects. or the underlying reasons behind opinions are encouraged. In the meantime, the label of each statement is annotated. Unlike traditional natural language inference (NLI) tasks, the primary ob- jective of statement annotation is to extract and label subjective information rather than establish logical connections or entailment between different texts. Besides, we request annotators to provide justifications for their proposed statements. These justifications provide the rationale behind the state- ment categorization. By treating them as the target in the JG task, we can gain valuable insight into the model's prediction processes and verify whether the model possesses real sentiment understanding ability.
Experiments
3.1 Setup
Models We benchmark SOUL with several widely used Small Language Models with the com- plete training set, including Roberta (Liu et al., 2019). T5 (Raffel et al., 2020), and Flan-T5 (Chung et al., 2022). We adopt the base version for each model type. In addition, we extend our analysis to two representative LLMS from the Flan and GPT model families, namely Flan (13B) (Raf- fel al., 2020) and ChatGPT! respectively. We evaluate these LLMs under a zero-shot setting. To reduce variance, we report the average results with three random seeds. The detailed setup can be found in Appendix A.1.
For the RC task. we report fl scores for each class and the overall accuracy. For the JG task, rics for predictions and justifications j. We mea- sure statement predictions Í using overall accu- racy. For justifications j. we employ commonly used text generation metrics, including BLEU (Pa- pineni et al., 2002), ROUGE(1/2/L) (Lin, 2004),
Evaluation Metries
different evaluation met-
Data Validation and Processing After the initial construction phase, a separate group of annotators classifies each proposed statement without access
We conducted the experiments using the May 24th version of ChatGPT.

---

Models
Fluree
Flaab
not-giive
Accuracy
Models
Ace
Similarity Evaluation
human
99.34
99.22
98.72
99.15
BLEU
ROUGE(1/2LL)
BERTScore
TS
74.50
24.01
50. 10/32/ 79/46.65
50.16/32.79/46665
92.33
Roberta
78.21
76.19
69.29
75.49
Flan-T5
79.32
25.05
51.25/33..7/47.667
92.53
T5 Flan-T5
81.04
78.22
71.23
77.87
Flan-T5xm
79.59
12.80
33.95/18.62/31.00 62/31.02
88.98
82.01
80.27
72.64
79.28
ChaGGT
78.04
11.32
59.64/20.79/33.1
90.16
Plan-TSxx 87.82
82.34
69.81
81.72
ChatGPT
85.41
73.50
29.36
72.09
Table 3: Performance of justification generation task.
Table 2:
Performance of review comprehension task. We report human performance as a reference.
facilitated ChatGPT a more thorough comprehen- sion of the sentiment conveyed, thereby enhanc- ing its performance. However, this may require strong reasoning ability, which is not observed in these SLMs. Therefore, attempting to perform justification generation could result in a decrease in accuracy performance. Regarding similarity eval- uation, it can be inferred that Flan-T5 is capable of generating justifications that closely resemble the annotated justifications, whereas Flan-T5xxI. exhibits the weakest performance in this respect. Nevertheless, the results obtained from the similar- ity evaluation contradict the overall accuracy, indi- cating a need for a more robust evaluation method.
and BERTScore (Zhang et al.. 2020) to calculate their similarity with the annotated justifications.
3.2 Main Results
Review Comprehension The results of the RC task are presented in Table 2. We can make the fol- lowing observations: 1) All models exhibit limited sentiment abiliiy, resulting in performance gap of 17% to 27% compared to human performance. This shows the difficulty of the RC task, and there is still much room for improvement in developing models that can accurately comprehend sentiment and opin- ion. The challenges may arise from the complexity and diversity of statements that incorporate mixed sentiments, underlying reasons of opinions, and other aspects. 2) Among SLMs, Flan-T5 achieves the best performance, surpassing T5 with the same model size by 1.41%, possibly due the effec- tiveness of instruction tuning during its training process. 3) LLMS demonstrate effective zero-shot ability, with Flan-T5xx1. achieving the best results even without any training data. In particular, Chat- GPT appears to have difficulty with the not not-given class, due to its overconfidence to misclassify the not-given class false. This failure shows the challenges posed by SOUL and emphasizes that large model size alone is not sufficient to ensure comprehensive sentiment capabilities.
3.3 Comprehensive Evaluation
There is a variation in accuracy between these two tasks, and there are conflicting evaluations of accu- racy and similarity within the JG task. To perform a thorough analysis, we aim to assess the generated justifications using the following criteria, rated on scale of 1 (poor) to 3 (excellent): 1) Correct: whether it is sensible and logical when compared to the gold label; 2) Align: whether it aligns with its generated label: 3) Relevant: whether it is rel- evant to the statement; 4) Concise: whether it is brief and concise; 5) Original: whether demon- strates innovation and uniqueness. We sample 50 instances and utilize both human evaluators and the GPT-4 model (OpenAI, 2023) for assessment. See Appendix A.2 for the GPT-4 evaluation prompt. The evaluation results are shown in Figure 2. We can see that while SLMs and Flan-T5xxi. have satisfactory performance in the RC task, their jus- tifications in the JG task lack originality, which means that they often rely on copied reviews with- out providing additional insights. This prediction process, without proper reasoning, potentially duces its overall accuracy and creates inconsisten- cies between the two tasks. Conversely, ChatGPT exhibits promising performance across various cri- teria, indicating its robust sentiment understanding capability. Nevertheless, there is still room for im-
Justification Generation We exclude Roberta from the JG task as is a discriminative model and not well-suited for text generation tasks. The results for the JG task are presented in Table 3. We report commonly used text generation metrics similarity evaluation and overall accuracy for reference. When it comes accuracy, it appears that SLMs and Flan-T5xxI. show either minimal improvement or even a negative impact. Instead, ChatGPT stands out with a notable improvement of approximately 6% in validating subjective state- ments, The incorporation of justifications likely

---

GPT-4
13
Human
Model
Training Data
RC Accuracy
correct
Cansert Fantissce
correct
NLI-RoBERTa RoBERTa
MNLI & SNLI
55.02
SOUL 75.49
aligh
original
aligh
original
Table Comparison between SOUL and NI.J tasks.
relevant
cencise
relevant
.coricise
the weakness of SLMs in generating reasoning- based justifications while showcasing ChatGPT's powerful sentiment understanding ability. Never- theless, there is still scope for enhancing the over- all accuracy, originality, and conciseness of Chat- GPT's responses.
Figure 2: Evaluation results for generated justifications by GPT-4 and human evaluators.
provement in terms of overall accuracy, as well as enhancing originality and conciseness in the task. We include examples of justifications gener- ated by these models in Appendix A.3 for detailed illustration. Moreover, the high agreement between human evaluators and GPT-4 suggests that auto- mated evaluation using GPT-4 is a more viable approach than similarity evaluation.
Limitation
The newly proposed dataset SOUL utilizes cus- tomer reviews as the main source of construct- ing subjective statements. However, incorporating more opinionated texts, such as social media posts and dialogues, could potentially enable the assess- ment of models in a wider variety of text types. Also, SOUL currently features two tasks, including review comprehension and justification generation, to evaluate the model's sentiment understanding abilities. More task formats can be designed to comprehensively understand the model's capabili- ties and limitations.
3.4 Comparison with NLI
Furthermore, we conduct an inference on SOUL test set using a widely used NLI model, namely the NLI-RoBERTa model?, trained on the SNLI (Bow- man et al., 2015) and MNLI (Williams et al., 2018) datasets, to demonstrate the focus of SOUL on sub- jective information rather than logical connections. As presented in Table the NLI-RoBERTa model achieves an accuracy of only 55.02%, which is sig- nificantly lower compared to the RoBERTa model trained on the SOUL dataset. This outcome em- phasizes the distinction between the objectives of SOUL and traditional NLI tasks. While they may share some similarities, the primary goal of SOUL is to extract and label subjective information, rather than establishing logical connections entailment between different texts.
Acknowledgements
Y. Deng is supported by Alibaba Group through Alibaba-NTU Singapore Joint Research Institute (JRI), Nanyang Technological University, Singa- pore. Sinno J. Pan thanks for the support from HK Global STEM Professorship and the JC STEM Lab of Machine Learning and Symbolic Reasoning.
References
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large an- notated corpus for learning natural language infer- ence. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 632-642.
4 Conclusion
This paper introduces novel task called Sentiment and Opinion Understanding of Language (SOUL), including two subtasks: review comprehension and justification generation. Our experimental results show that SOUL a challenging task that demands a deep understanding of sentiment, with a perfor- mance gap of up to 27% when compared to human performance. Moreover, evaluations conducted with both human experts and GPT-4 demonstrate
Seungtaek Choi, Myeongho Jeong. Hojae Han, and Seung-won woung-won Hwang. 2022. C2L: causally contrastive learning for robust text classification. In Thirty-Sixth Artificial Intelligence, AAAI 2022, Thirty- Fourth Conference on Innovative Ap- plications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium Educational Advances in Ar- tificial Intelligence, EAAI 2022 Virtual Event, Febru- ary 22 March 1, 2022, pages 10526-10534,
AAAI Conference
httpps: httg//huggggaaaageee ace. www./////occcoooooooeo encoder/ ali-roberta base

---

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts. Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-finetuee language models. CoRR, abs/2210.11411
Bo Pang and Lillian Lee, 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACI. 2005, 43rd Annual Meeting of the Association for Computational Lin- guistics, Proceedings of the Conference, 25-30 June 2005, University of Michigan, USA, pages 115-124.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6-12, 2002, Philadelphia. PA, USA, pages 311-318.
Shuai Fan, Chen Lin, Haonan Li, Zhenghao Lin, Jin- song Su, Hang Zhang, Yeyun Gong, Jian Guo, and Nan Duan. 2022. Sentiment- tence level pre-training for sentiment analysis. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 4984-4994.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
word and sen-
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Y. Ng. and Christopher Potts. 2013, Recursive deep mod- cls for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington. USA, A meeting of SIG- DAT, a Special Interest Group of the ACL, pages 1631-1642.
Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. 2020. Sentilare: Sentiment-awand language representation learning with linguistic knowledge. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6975- 6988.
Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81. 81, Barcelona, Spain. Association for Computational Linguistics.
Zengzhi Wang. Qiming Xie, Zixiang Ding, Yi Feng, and Rui Xia. 2023. Is chatgpt good sentiment analyzer? A preliminary study. CoRR, 044339
Bing Liu. 2015. Sentiment Analysis - Mining Opinions, Sentiments, and Emotions. Cambridge University Press.
Zhao Wang and Aron Culotta. 2020. Identifying spu- rious correlations for robust text classification. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, Online Event, 16-20 Novem- ber 2020, pages 3431-3440
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Zhao Wang and Aron Culotta. 2021. Robustness to spu- rious correlations in text classification via automati- cally generated counterfactuals. In Thirty-Fifh AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial In- telligence, EAAI 2021. Virtual Event, February 2-9, 2021, pages 14024-14031.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 201 I. Learning word vectors for sentiment analysis. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Tech- nologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA, pages 142-150.
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. A coveraee challenge corpus for sentence understanding through inference. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI- HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pages 1112-1122.
Seung Jun Moon. Sangwoo Mo, Kimin Lee, Jaeho Lee, and Jinwoo Shin. 2021. MASKER: masked key- word regularization for reliable text classification. In Thirty-Fifth AAAI Conference on Artificial Intel- ligence, AAAI 2021, Thirty-Third Conference on In- novative Applications of Artificial Intelligence, IAAl 1021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13578-13586.
Ellery Wulczyn, Nithum Thain, and Lueas Dixon, 2017. Ex machina: Personal attacks seen at scale. In Pro- ceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pages 1391-1399.
OpenAl. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

---

Da Yin. Tao Meng, and Kai-Wei Wei Chang. 2020. Sentib- ert: A transferable transformer- based architecture for compositional sentiment semantics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 3695-3706.
2. Alignment (scale of 1-3): Assess whether the justification aligns with the generated label.
3. Relevance (scale of 1-3): Determine whether the justification is relevant to the statement.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
4. Conciseness (scale of 1-3): Evaluate whether the justification is brief and concise, without compromising clarity and accuracy.
5. Originality (scale of 1-3) Assess whether the justification demonstrates innovation and uniqueness, rather than simply copying the review and statement.
Wenxuan Zhang. Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023. Sentiment analysis the era of large language models: A reality check. CoRR, abs/2305.15005
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2021. Towards generative aspect-based sentiment analysis.
Please return your answers as Python dictionary, where the key is the model name, and the value is dictionary containing the aforementioned metrics. Please avoid returning any additional text.
Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, ACU/JCNLP 2021, pages 504-510.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks text clas- sification. In Advances in Neural Information Pro- cessing Systems 28: Annual Conference Neural In- formation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.
A.3 Case Study
Table presents examples of justifications gener- ated by various models. In this particular sample, all models, except Flan-T5xxI.. have made the cor- rect prediction. However, when it comes to justifi- cations, both T5 and Flan-T5 have simply copied text from the review without any reasoning. On the other hand, ChatGPT has demonstrated strong ability to understand sentiment by providing rea- sonable justifications based on the original review text, which led to the correct prediction.
Appendix
A.1 Detailed Setup
We perform a grid search on the development set to find the best hyper-parameters for fine-tuning SLMs. Specifically, we search the learning rate among 1ee-6, 5e-6, 1e-5, 5c-5, le-4), batch size among (2, 4, 8), and number of epochs among (4, 8). For LLMs, we utilize their APIs to perform zero-shot inference.
A.2 GPT-4 Prompt for Evaluation
We adopt the following prompt evaluate the jus- tification generated by different models: Given review and statement,
along with their gold label and gold justification, you will also receive labels and multiple justifications generated by different models. Your task is to rate these generated justification based the following metrics:
1. Correctness (scale 1-3): Evaluate whether the justification is sensible and logical when compared to the gold label.

---

This is only related to the first movie by the name. The plot has nothing to do with the first and the whole movie stinks!!! I have idea what they were thinking but this movie is bad. Avoid this all costs, the first movie in the series is acceptable slasher flick and is the fourth but this one and the 3rd are rubbish!! Statement: This movie worse than the first one.
Review:
Gold label: true
Gold justification: The reviewer said that the first movie is still acceptable, but this one and 3rd are rubbish.
T5
Label: true
Justification: The reviewer said that the plot has nothing to do with the first and the whole movie stinks!!!
Flan-T5
Label: true
Justification: The reviewer said that this one and the 3rd are rubbish!!
Flan-T5xx1.
Label: false
Justification:
The plot has nothing to do with the first and the whole movie stinks!!! have no idea what they were thinking but this movie is so bad. Avoid this at all costs, the first movie in the series is acceptable slasher flick and is the fourth but this one and the 3rd are rubbish!!
ChatGPT
Label: true
Justification: The reviewer explicitly states that "the whole movie stinks!!!" and that "this one and the 3rd are rubbish!!" in comparison the first movie, which they found acceptable slasher flick.
Table 5: Examples of justifications generated by various models, with colored text indicating the overlap with the original review text.


Reviews:
[{"review_id": "5RjLSfHYgm", "reviewer": "Reviewer_5t9T", "summary": "", "questions": "\nQ1: Why you have not indicated the results neither in abstract nor in the conclusion? This should be indicated into the final version of the paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comment and suggestion. We agree that it is important to provide a summary of the key findings in these sections to give readers a clear understanding of the results. In the revised version of the paper, we will ensure that the results are appropriately summarized in both the abstract and conclusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The new task The results show that SOUL requires an in-depth understanding of sentiment. It is a difficult concept but also a promising strategy for sentiment analysis research. According to the results, Flan-T5 outperforms T5 among SLMs by 1.41% with the same model size. When it comes to performance, Flan-T5XXL surpasses ChatGPT while LLMs demonstrate effective zero-shot capability. When it comes to accuracy, ChatGPT performs better, with a notable improvement of roughly 6% in validating subjective statements. The results are encouraging, which led to the paper's acceptance.\n \n", "reasons_to_reject": "N/A", "questions_to_authors": "\nQ1: Why you have not indicated the results neither in abstract nor in the conclusion? This should be indicated into the final version of the paper.", "paper_topic_and_main_contributions": "The paper topic is about Sentiment Analysis, and the authors contribution is two folded: (i) the annotated data set (with 15,028 statements) and (ii) the benchmark of new task SOUL (Sentiment and Opinion Understanding of Language) which provides two subtasks, the Review Comprehension and Justification Generation. The first one aims to validate statements based on a review text, whereas the latter one provides explanations for their sentiment predictions. SOUL uses SLM with complete training, including Roberta, T5 and Flan-T5 and extended their analysis to two representative LLMs, Flan-T5XXL and  ChatGPT. Both of them evaluated under zero-shot settings.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jjC8Tq0A92", "reviewer": "Reviewer_Qqt5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your concern. We want to clarify that the proposed SOUL task differs fundamentally from textual entailment and explainable fact checking in its primary focus, i.e., **SOUL focuses on subjective sentiments and opinions while these existing studies focus on factual veracity**.\n\nMore specifically, existing textual entailment or fact checking studies aims to judge the factual veracity or relation for a given text / text pair. However, the newly proposed SOUL task aims to encapsulate any sentiment-related issue through statement proposition and verification, thus breaking the constraints of traditional multiclass sentiment analysis tasks. Therefore, it requires a deep understanding of subjective information in the text to make correct predictions.\n\nTo illustrate this distinction, we conducted a quick inference on the SOUL dataset using a RoBERTa model trained on the SNLI and MNLI datasets. Its accuracy is 55.02%. The performance is significantly lower compared to the RoBERTa result in Table 2, which is 75.49%. This discrepancy highlights the disparity between SOUL and traditional NLI tasks.\n\nSimilarly, while explainable fact checking aims to establish the truth or falsehood of a factual claim, SOUL's justification generation subtask places greater emphasis on discerning the subtle variations in sentiment.\n\nIn our revised manuscript, we will include a detailed comparison between SOUL and these tasks, highlighting the unique characteristics and contributions of SOUL, which can help to clarify the novelty of our proposed task and its distinction from existing tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It's a nice short paper that clearly explains the (seemingly new) task and the results of the experiments.", "reasons_to_reject": "The authors did not explain how SOUL is different from explainable fact checking/textual entailment and how the statements are different from hypothesis generation for the textual entailment task. Unless they address these questions, I'm not convinced that the task they proposed is that new.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper describes a task of sentiment and opinion understanding of language (SOUL), which consists of two subtasks: review comprehension and justification generation.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1GLMH1L1xF", "reviewer": "Reviewer_TkL7", "summary": "", "questions": "a. No novelty method is proposed to solve SOUL.  The experiments focus on the small language models Roberta and Flan-T5.   What’s the performance of the SOTA MRC  and NLI methods on SOUL task?\n\nb.Error analysis should be provided for figure out the key  challenges for SLM and LLM models in SOUL task.\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback! We now address the concerns as below:\n\n> ***1. No novelty method is proposed***\n\nWhile proposing a novel method is indeed important, we would like to emphasize that the primary focus of our paper is to introduce the SOUL task. Our objective is to emphasize the importance of capturing the complexities of sentiment analysis through this task. We appreciate your input and will consider exploring novel methods in our future work.\n\n> ***2. SOTA NLI and MRC performance***\n\nRegarding the performance of SOTA NLI methods on the SOUL task, we conducted a quick inference using a popular NLI model, specifically the NLI-RoBERTa model trained on the MNLI and SNLI datasets (https://huggingface.co/cross-encoder/nli-roberta-base). We show its performance and the reported result of RoBERTa from Table 2 as below:\n| Model        | Training Data | Review Comprehension Accuracy |\n| ------------ | ------------- | ----------------------------- |\n| NLI-RoBERTa | MNLI & SNLI   | 55.02                         |\n| RoBERTa      | SOUL          | 75.49                         |\n\nThe NLI-RoBERTa model only achieves an accuracy of 55.02%, which is significantly lower compared to RoBERTa model trained on SOUL dataset. This result highlights the distinction between the SOUL and traditional NLI tasks. While they share similarities, the main objective of SOUL is to extract and label subjective information rather than establishing logical connections or entailment between different texts.\n\nRegarding MRC, it differs significantly from the justification generation subtask in SOUL. MRC involves extracting answer spans from a passage based on a question, while justification generation in SOUL requires free-form generation, including common sense knowledge and detailed explanations. Additionally, the criterion of \"originality\", as discussed in section 3.3, is crucial in evaluating justifications in SOUL, emphasizing the need for unique and innovative responses. Therefore, directly adopting an MRC method may not be suitable.\n\n> ***3. Error analysis should be provided***\n\nThank you for your feedback. We recognize the significance of error analysis in bridging the gap in the SOUL task. Following your suggestions, we have enhanced our analysis. Through a thorough error analysis, we have pinpointed two primary causes of mispredictions. Below are representative examples to illustrate these reasons:\n\n**Review**: Nice hotel, nice warm cookies!  I have stayed at this one twice and would recommend to out of towners.\n\n| Statement                                               | Label     | Prediction |\n| ------------------------------------------------------- | --------- | ------- |\n| The reviewer said the service was good.                 | Not-given | True    |\n| The reviewer said he would recommend it to his friends. | False     | True    |\n\n1. **Superficial textual features**: The model tends to make incorrect predictions based on surface-level matching. In both statements, the model mistakenly predicts them as true because it simply match with the positive words in statement 1 and the \"recommend\" word in statement 2.\n2. **Lack of common sense knowledge**: The model sometimes struggles to comprehend certain terms or phrases that require background knowledge or contextual understanding. In the second statement, the model fails to comprehend the term \"out of towners\" and makes an incorrect prediction.\n\nIn our revised version, we will include a detailed error analysis section that delves into the key challenges faced by the SLM and LLM in the SOUL task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper explores a novel task called Sentiment and Opinion Understanding of Language (SOUL), which consists of review comprehension and justification generation. \n\n It is an interesting topic and very useful in sentiment applications. \n\nThe authors provide a valuable SOUL dataset based on common review corpus Yelp and IMDb.\n", "reasons_to_reject": "This paper doesn't provide a novelty method to solve SOUL. \n\nThe experiments focus on the small language models Roberta and Flan-T5.   The SOTA MRC methods should be evaluated.\n\nError analysis should be provided to figure out the gap between the existing models and human. \n \n", "questions_to_authors": "a. No novelty method is proposed to solve SOUL.  The experiments focus on the small language models Roberta and Flan-T5.   What’s the performance of the SOTA MRC  and NLI methods on SOUL task?\n\nb.Error analysis should be provided for figure out the key  challenges for SLM and LLM models in SOUL task.\n", "paper_topic_and_main_contributions": "This paper explores a novel task called sentiment and Opinion Understanding of Language (SOUL), which consists of review comprehension and justification generation.  Review comprehension focuses on verifying comprehension statements regarding an opinionated review text.  Justification generation focuses on generating justifications for such sentiment prediction. The authors annotate a SOUL dataset based on common review corpus Yelp and IMDb.  Experimental results on this dataset demonstrates that SOUL is a very challenging task. \n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Izwy9Eh4b9", "reviewer": "Reviewer_HN7A", "summary": "", "questions": "A. What do you mean by overall accuracy in Justification Generation evaluation?\nB. Can you give examples to the \"not-given\" label?\nC. Why the review has statements such as \"The reviewer doesn't like the movie\" in the example given in the introduction. How is this statement different than just labeling the whole text as negative sentiment?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and detailed feedback and suggestions. We have carefully considered each of your concerns and will make the necessary revisions to improve the clarity and comprehensiveness of our paper. \n\n> ***1. The Justification Generation task results are confusing***\n\nWe apologize for any confusion caused and would like to clarify on the significant improvement of ChatGPT in justification generation. This improvement specifically relates to the accuracy of predicting statements and is compared to the performance achieved in the previous task, which solely predicts the statement without generating justification. Specifically, the accuracy has risen from 72.09% to 78.04%.\n\n> ***2. ChatGPT is capable of producing the reasoning & suggestion on more challenge task***\n\nFigure 2 demonstrates that ChatGPT's justifications currently lack conciseness and originality, highlighting the potential for further improvement in its performance.\n\nWe greatly appreciate your valuable suggestion regarding the task format. However, this is a subset of our proposed task format, as we can still rephrase it into a statement and prompt the model to evaluate its correctness and generate a corresponding justification.\n\n> ***3.  Criteria for a simple statement***\n\nDuring the quality check, statements in SOUL are marked as simple, medium, or hard by annotators to determine their difficulty level. Reviews containing only simple statements are excluded to ensure an appropriate level of challenge.\n\nTo evaluate the agreement between annotators, we employed Cohen's kappa coefficient, which yielded an impressive score of 0.9649. This high level of agreement reinforces the validity and reliability of the data annotation process.\n\n> ***4. Example of \"not-given\" label***\n\nNot-given refers to a statement that cannot be determined from the existing context. In the sample review shown in figure 1, an example of a not-given statement could be \"The reviewer liked the raptors because they were terrifying.\" This statement cannot be concluded from the available information.\n\nOn the other hand, a false statement can be directly inferred based on the context, such as \"The reviewer liked the movie.\" We apologize for not providing an example of a \"not-given\" statement, and will include specific examples to clarify the definition of the \"not-given\" label.\n\n> ***5. how is this \"overall accuracy\" measured.***\n\nThe overall accuracy is the same metric used in the RC task, representing the accuracy of all statements. The term \"overall\" is used to distinguish it from the class-wise F1 score. We apologize for not providing a clear definition of \"overall accuracy\" for the JG task, and we will include a formal definition in the revised version.\n\n> ***6. Human judgement would be a more reliable approach***\n\nWe agree that evaluating the quality of generated justifications by humans is valuable, and we do have listed important criteria and adopted human evaluation, as shown in Section 3.3 and Figure 2. However, human evaluation can be time-consuming, costly, and subjective. Therefore, we adopted GPT-4 and found that it had high agreement with human evaluators, proving it to be a viable alternative. Still, we acknowledge the importance of human evaluation and encourage future users to conduct it, following our outlined criteria in Section 3.3, whenever resources permit.\n\n> ***7. The colors in Figure 2 are very hard to distinguish.***\n\nWe apologize for the inconvenience caused by the figure's colors, and we will use more distinguishable and reader-friendly colors for Figure 2 in the revised version. \n\n> ***Question A. What do you mean by overall accuracy?***\n\nWe have addressed this concern in #5. \n\n> ***Question B. Can you give examples to the \"not-given\" label?***\n\nWe have addressed this concer in #4\n\n> ***Question C. Statement in example***\n\nWhile our annotation process encourages complex subjective statements, there still might be some relatively simple statements included. The example in Figure 1 is a randomly sampled instance and not cherry-picked.\n\nActually, instead of just stating \"The reviewer doesn't like the movie,\" our task format allow us to ask for specific details like \"The reviewer finds the movie boring/terrifying/slow-paced/confusing,\" etc. Although they are all negative sentiments, not all of them may accurately reflect the review and could potentially be incorrect. That's one of our task advantages compared to traditional sentiment analysis tasks.\n\n> ***Typos, grammar***\n\nThank you for pointing out the typographical error. We will correct them in the revised version of the paper. \n\nThanks again for your detailed feedback, we believe that addressing these concerns will greatly enhance the quality and impact of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strengths of the paper are the following:\n\n1. It presents a novel task that addresses the shortcomings of traditional polarity detection. Although sentiment classification benchmarks are significantly high and there are numerous pretrained sentiment detection models that illustrate very high performance, indirect expression of sentiment (e.g. irony) is still not accurately detected. The paper introduces a task that attempts to improve exactly this defect in  the sentiment classification task.\n\n2. The justification task is a very good approach for reliable AI. The authors attempt to have the model not only decide on the sentiment label of the text but also justify why it makes the decision. Although the task seems quite challenging, but further expansion on the idea for sentiment analysis and other common NLP task can be very beneficial in minimizing the black box effect of the DL models.\n\n3. The authors make use of the GPT prompting for evaluating the model justification generation output and they show the soundness of this method by comparison to human evaluators.\n4. The paper is easy to read.", "reasons_to_reject": "There are a number of points that need to be addressed:\n\n1. The Justification Generation task results are confusing. The authors mention that ChatGPT accuracy is significantly  higher but Table 3 does not match their conclusion. Table 3 seems to show different results than the ones explained in section \"Justification Generation\".  \n\n2.  If ChatGPT is capable of producing the reasoning behind any sentiment classification label, then it is a solved problem. The ChatGPT justification in Table 4 is even better than the gold-standard. It would be more challenging to have justification of the sentiment label of the text as a whole rather than just the statement. The authors give a good example at the Introduction where indirect expression of emotion is particularly difficult to detect. Justification for the model's sentiment labeling to such an expression of sentiment would be a very good approach to the task of sentiment analysis.\n3. The annotation schema rejects simple statements. It is not very clear the criteria for a simple statement. Also, an Inter Annotator Agreement would have substantiated the soundness of the data annotation. \n4. A major problem with the annotation is the \"not-given\" label for the statements. There aren't any examples for statements with this label in the paper although there is a large number of statement with the not-given label in the training data. This label in particular can be quite confusing to the model specially with the \"False\" label.\n5. In evaluating the generated text by the model, the authors report \"overall accuracy\" as well as BLEU, ROUGE and BERTScore. It is not clear how is this \"overall accuracy\" measured. \n6. The text generation task is very difficult to evaluate by a non-human metric. It is not a straightforward text generation task, it is a justification of the model's decision. Human judgement would be a more reliable approach than the reported metrics.\n7. The colors in Figure 2 are very hard to distinguish.\n", "questions_to_authors": "A. What do you mean by overall accuracy in Justification Generation evaluation?\nB. Can you give examples to the \"not-given\" label?\nC. Why the review has statements such as \"The reviewer doesn't like the movie\" in the example given in the introduction. How is this statement different than just labeling the whole text as negative sentiment?", "paper_topic_and_main_contributions": "The paper presents a novel sentiment analysis task that goes beyond the traditional sentiment/aspect classification tasks. They aim to have the model assess statements on the sentiment of the text as well as generate justification for its decision on the statement label. They employ  two commonly-used sentiment classification datasets, the Yelp and the IMDP sets. They assign annotators to produce approx. 4 statements for each review along with their labels and justifications. The review dataset is around 36k reviews and the statements amount to 15k. They benchmark the dataset on trained SLMs and zero-shot settings with LLM.\n\nThe conclusion on their experiments is that all models show limited sentiment ability when it comes to justifying the sentiment label, specifically in their ability to generate reasoning behind the sentiment labeling. They also use GPT-4 prompts as an evaluation tool to their Sentiment Justification task. They show that GPT-4 performance in assessing the model's generated justification text is on par with the human evaluation.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please correct grammar in line 071-072\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the SOUL (Sentiment and Opinion Understanding of Language) task, which includes review comprehension and justification generation. It provides a dataset with 15,028 annotated statements and evaluates various models, including SLMs like Roberta, T5, and Flan-T5, as well as LLMs like Flan-T5XXL and ChatGPT, in zero-shot settings. The results highlight the challenge of justifying sentiment labels, with GPT-4 performing similarly to human evaluation in assessing generated justifications.

Reasons to accept:

Novel Task: Introduces the SOUL task that addresses shortcomings in sentiment analysis, focusing on detecting indirect sentiment expressions like irony.
Enhances AI model reliability by requiring justification for sentiment decisions.
Effective Evaluation: Uses GPT prompting for model justification evaluation, demonstrating its validity compared to human evaluation.
The paper is well-written and organised.

Reasons to reject:

Novel Method: The paper lacks a novel method to address the SOUL task. -Limited Model Evaluation: The experiments primarily focus on small language models (Roberta and Flan-T5), while state-of-the-art Machine Reading Comprehension (MRC) methods are not evaluated. -Missing Error Analysis: The paper lacks error analysis to identify gaps between existing models and human performance. -Confusing Results: The results for the Justification Generation task are inconsistent.
Ambiguity in Annotation Criteria and lack of Inter Annotator Agreement for data annotation. -The paper reports an "overall accuracy" metric without explaining how it's measured, and relies on non-human metrics like BLEU, ROUGE, and BERTScore for text generation evaluation, which may not be suitable for this task.
Decision: Accept-Main