<figure><img src="image_1.png" /></figure>

HAL
open science

# Enhancing Granular Sentiment Classification with Chain-of-Thought Prompting in Large Language Models

Vihaan Miriyala, Smrithi Bukkapatnam, Lavanya Prahallad

► To cite this version:

Vihaan Miriyala, Smrithi Bukkapatnam, Lavanya Prahallad. Enhancing Granular Sentiment Classification with Chain-of-Thought Prompting in Large Language Models. 2025. hal-05060150

HAL Id: hal-05060150

https://hal.science/hal-05060150v1

Preprint submitted on 7 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Enhancing Granular Sentiment Classification with
Chain-of-Thought Prompting in Large Language Models

Vihaan Miriyala¹, Smrithi Bukkapatnam², and Lavanya Prahallad³

¹John F Kennedy High school Fremont, USA, vihaannagmiriyala@gmail.com

²Queensland Academy for Science Mathematics and Technology, Australia,
smrithibhushan.b@gmail.com

³Research Spark Hub Inc, lavanya@researchsparkhub.com

April 2025

Abstract

We explore the use of Chain-of-Thought (CoT)
prompting with large language models (LLMs) to
improve the accuracy of granular sentiment catego-
rization in app store reviews. Traditional numeric
and polarity-based ratings often fail to capture the
nuanced sentiment embedded in user feedback. We
evaluated the effectiveness of CoT prompting versus
simple prompting on 2,000 Amazon app reviews by
comparing each method's predictions to human judg-
ments. CoT prompting improved classification accu-
racy from 84% to 93%, highlighting the benefit of ex-
plicit reasoning in enhancing sentiment analysis per-
formance.

1 Introduction

Sentiment analysis is a key task in natural lan-
guage processing (NLP), but traditional methods of-
ten oversimplify human emotions by classifying them
into two or three categories: positive, negative, or
neutral. This is particularly limiting in domains such
as App reviews, where users express multifaceted sen-
timents. For example, a 3-star rating might superfi-
cially suggest an average experience, but the accom-
panying textual review can reveal critical insights into
specific frustrations, unmet expectations, or praises
that influenced the rating. This limitation becomes
especially pronounced when analyzing reviews in the
App Store, such as those of the Amazon app. Cus-
tomer feedback in this context often contains a mix of
sentiments, highlighting issues such as usability chal-
lenges and delivery delays alongside positive experi-
ences with the app's functionality or customer sup-

port. Such multifaceted reviews expose the inability
of traditional methods to effectively disentangle and
categorize granular sentiments, particularly for orga-
nizations that rely on polarity-based or star-rating
scales [1].

Recent advances in prompt-based learning with
large language models (LLMs) have enabled more
sophisticated understanding of text, especially when
paired with structured reasoning techniques such as
Chain-of-Thought (CoT) prompting [2].

In this work, we investigate the application of CoT
prompting for fine-grained sentiment classification,
aiming to bridge the gap between polarity ratings and
the nuanced nature of real-world feedback.

2 Related Work

Several studies have focused on fine-grained senti-
ment analysis in user-generated content. Guzman
and Maalej [3] presented a method to extract App
features and their associated sentiments from mobile
App reviews, allowing developers to prioritize prod-
uct improvements. The SCARE corpus [4] further
provided a richly annotated dataset for sentiment
components, while SenticNet [5] emphasized the role
of commonsense and concept-level knowledge in im-
proving sentiment classification beyond lexical polarity.

With the emergence of large language models
(LLMs), the field has witnessed significant advance-
ments. Brown et al. [6] introduced GPT-3, show-
casing its potential for few-shot and zero-shot learn-
ing across NLP tasks, including sentiment classifica-
tion. Shah et al. [7] evaluated GPT-4's capabilities
in extracting feature-sentiment pairs from App re-
views, outperforming conventional baselines even in
low-resource settings.

Chain-of-Thought (CoT) prompting was introduced by Wei et al. [2] to enhance reasoning in LLMs by breaking down complex problems into intermediate steps. Further, Kojima et al. [8] showed that even zero-shot prompts could elicit coherent reasoning chains without requiring explicit task-specific examples. This technique has since been applied in various NLP tasks, including implicit sentiment recognition [9], and further refined through least-to-most prompting strategies [10] that guide models through stepwise problem decomposition. Peng et al. [9] proposed CoTAM, a method that manipulates sentiment attributes via CoT for few-shot data augmentation, improving classification accuracy in limited-data environments.

Recent studies have also explored applying CoT prompting in multimodal and multilingual sentiment analysis. Zhang et al. [11] proposed PEAR, a multimodal CoT reasoning framework for integrating textual, audio, and visual data to assess sentiment in user-generated content. In another work, Hannani et al. [12] evaluated ChatGPT-4 and machine learning models on Moroccan Arabic reviews, highlighting the challenges and opportunities of using LLMs in low-resource, multi-script settings.

While CoT prompting has shown potential in improving reasoning tasks, its application to granular sentiment tagging in App store reviews remains underexplored. Our work addresses this gap by leveraging CoT prompting with GPT-4 to improve fine-grained sentiment categorization, demonstrating significant gains over standard prompting techniques.

## 3 Dataset

Our dataset comprises 2000 reviews from the Amazon App store. These reviews were selected to represent a diverse range of customer feedback, capturing sentiments ranging from highly positive experiences to critical concerns. Redundant reviews were identified and removed to eliminate noise and ensure the dataset's uniqueness. We annotated 2,000 app reviews using three independent human reviewers, with final labels determined by majority vote.

To overcome the limitations of polarity ratings, we adopted a granular sentiment categorization approach. Sentiments were classified into the following categories:

* Very Negative: Represents strong dissatisfaction, anger, or frustration (e.g., "Terrible app! Crashes every time.").

* Negative: Indicates general dissatisfaction or disappointment (e.g., "The app is slow and buggy.").

* Neutral: Reflects a balanced or ambiguous tone, where the review neither strongly praises nor criticizes (e.g., "The app works okay but could use some improvements.").

* Positive: Highlights general satisfaction with the product or service (e.g., "Good app, works as expected.").

* Very Positive: Represents enthusiastic approval, often with specific praise (e.g., "Amazing app! Easy to use and super convenient.").

## 4 Methodology

We compare two different approaches to sentiment classification using Large Language Models (LLMs): a simple prompt-based method and a structured Chain-of-Thought (CoT) prompting technique.

### 4.1 Simple Prompt-Based Approach

The first approach utilized a direct prompt that asked the model to assign a sentiment score from 1 (Very Negative) to 5 (Very Positive) based on the overall tone of a review. This approach did not provide guidance for reasoning or justification.

> Prompt: "Rate the sentiment of this review on a scale from 1 (Very Negative) to 5 (Very Positive) based on the overall tone."

> Example Review: "I don't recommend this book if you want to learn anything at all about how this technique is actually practiced. The impression one gets from the book is that apparently only Alexander himself and a few other self-interested 'disciples' are intellectually talented and/or physiologically gifted enough to learn this magical technique and teach others, so you are left still wondering at the end of the book what the point was of reading it in the first place."

> Model Output: Sentiment Rating: 3 (Neutral)

The simple prompt approach misclassified the review as neutral. Despite the presence of clearly negative phrases such as "don't recommend," "self-interested disciples," and "still wondering", the model failed to account for the overall tone.
## 4.2 Chain-of-Thought (CoT) Prompting

The simple prompt approach exhibited several key shortcomings. First, it struggled to handle ambiguity, often misinterpreting reviews with mixed or nuanced sentiments. Second, it showed an over-reliance on individual keywords or word counts, which led to sentiment scores that did not accurately reflect the overall meaning of the review. Finally, the approach lacked explainability, as the model failed to provide any reasoning or justification for its sentiment classifications, making it difficult to analyze errors or improve performance systematically.

To overcome these limitations, a Chain-of-Thought (CoT) prompting strategy was introduced. CoT guides the model through explicit reasoning steps, helping it understand not just what is said, but how it is said. As a part of CoT prompt, we incorporated keywords hints as they play a critical role in tagging and categorizing sentiment. Specific words or phrases often indicate the tone of a review:

* Positive keywords: Words such as "easy," "love," "amazing," and "convenient" often signify positive sentiments.

* Negative keywords: Words such as "slow," "frustrating," "unreliable," and "buggy" are commonly associated with negative sentiments.

* Neutral keywords: Neutral reviews tend to lack emotionally charged words, using balanced language like "okay" or "average."

We incorporated keyword hints into the CoT prompt design to guide the model in identifying sentiment-bearing expressions. These patterns refined through manual annotation of the dataset set—played a key role in aligning the model's reasoning with human judgment. By integrating these insights, the structured prompting approach improved the model's ability to handle complex sentiment patterns and justify sentiment ratings based on contextual cues.

Prompt: "Let's do this step by step:

* Step 1: Read the review carefully to understand its content and tone.

* Step 2: Identify specific positive and negative expressions. For example, in: 'I love this product, but it is of no use to me, and the service was terrible,' the phrase 'I love this product' is positive, while 'no use to me' and 'service was terrible' are negative.

* Step 3: Evaluate the dominant tone based on repeated themes or emotional emphasis.

* Step 4: Assign a sentiment rating on a scale from 1 (Very Negative) to 5 (Very Positive).

* Step 5: Justify the rating with a brief explanation."

Example Review (same as above): "I don't recommend this book if you want to learn anything at all about how this technique is actually practiced. The impression one gets from the book is that apparently only Alexander himself and a few other self-interested 'disciples' are intellectually talented and/or physiologically gifted enough to learn this magical technique and teach others, so you are left still wondering at the end of the book what the point was of reading it in the first place."

Model Output:

Main Points Identified:

* Positive: None

* Negative: "don't recommend," "self-interested disciples," "magical technique," "still wondering"

Rating Justification: The review expresses strong dissatisfaction. The reviewer explicitly states they would not recommend the book, describes the content as elitist and vague, and ends with frustration and confusion. No redeeming or positive comments are included.

Final Rating (CoT): 1 (Very Negative)

## 4.3 Model Details

We used GPT-4 via OpenAI's API (March 2024 snapshot) with temperature set to 0.3 for determinism. Each review was evaluated individually using both prompting methods.

# 5 Results

In this section, we compare the performance of Simple Prompting and CoT Prompting for sentiment analysis on user reviews. The analysis was conducted on a dataset of 2,000 reviews, with accuracy as the
primary evaluation metric. We compared the predictions from the GPT-4 model against human judgments.

Simple Prompting achieved an accuracy of 84%, but many of the errors were still due to its failure to handle mixed or ambiguous sentiments effectively. CoT prompting reached a notable 93% accuracy, validating the impact of detailed reasoning steps. The results show that when sentiment evaluation includes interpretive breakdowns, models are better equipped to handle real-world user language complexities. The CoT approach, which incorporates structured reasoning, enabled the model to better interpret and classify nuanced sentiments.

<table><caption>Table 1: Accuracy for each method on 2000 reviews.</caption><thead><tr><td>Method</td><td>Accuracy</td></tr></thead><tbody><tr><td>Simple Prompt</td><td>84%</td></tr><tr><td>CoT Prompt</td><td>93%</td></tr></tbody></table>

## 5.1 Error Analysis

A qualitative error analysis was conducted to understand why each prompting strategy failed in specific scenarios. This was essential to pinpoint not only the types of errors but also the patterns of misclassification that persisted across datasets.

## 5.2 Simple Prompting

Simple Prompting frequently misclassified reviews that contained mixed emotions or required contextual understanding. It often defaulted to the dominant polarity of isolated words, disregarding the reviewer's overall experience.

*Oversimplification of Mixed Sentiments:* Reviews that expressed both positive and negative elements were often misclassified because the model prioritized the sentiment with the higher word count.

* Review: “The scenery was breathtaking, but the guide was rude and unprofessional.”

* Error: Classified as *Positive* due to the strong positive sentiment about the scenery, while ignoring the critical comment about the guide.

*Lack of Contextual Understanding:* Simple Prompting lacked nuance in interpreting the tone of context-dependent expressions like “fine” or “okay”.

* Review: “The app works fine most of the time, but crashes occasionally.”

* Error: Misclassified as *Negative* simply due to the mention of crashes, despite the balanced tone.

## 5.3 CoT Prompting

Despite its significantly improved performance, CoT prompting was not immune to errors. Its reasoning sometimes failed to accurately balance competing sentiments, especially in reviews with complex narratives or ambiguous wording.

*Ambiguity in Sentiment Weighting:* When reviews highlighted both strengths and weaknesses, the CoT approach occasionally overemphasized one aspect.

* Review: “The movie had stunning visuals but poor acting and a confusing plot.”

* Error: Overweighted the visuals, resulting in a *Positive* classification instead of *Neutral* or *Slightly Negative*.

*Short or Vague Reviews:* Minimal feedback in brief reviews often lacked enough detail for effective reasoning, leading to misclassification.

* Review: “It’s okay.”

* Error: Classified without accounting for potential subtle dissatisfaction implied by the brevity.

*Manual Intervention Needed:* Some reviews required human input to clarify nuanced sentiments or correct misinterpretations.

* Review: “The tutorials were detailed but long and repetitive, which made them hard to follow.”

* Error: Initially rated *Neutral*; revised to *Negative* after human review of tone and intent.

We specifically analyzed the subset of errors to determine how CoT prompting compared to simple Prompting, and to identify the characteristics of errors within the CoT method.

## 5.4 Simple Prompting vs CoT

We further analyzed whether the CoT prompting method was able to correctly classify reviews that were misclassified by the simple prompting approach. Using simple prompting, approximately 80% of the misclassified reviews (about 64 out of 80) were correctly predicted by the CoT method. These cases often involved conflicting or layered sentiments that were better interpreted through structured reasoning. However, the remaining 20% of errors persisted even with CoT prompting, typically involving vague expressions or sarcasm that challenged both methods.
# 6 Conclusion

In this work, we demonstrated the effectiveness of Chain-of-Thought (CoT) prompting in improving the accuracy and interpretability of sentiment analysis in user-generated content, specifically App store reviews. By incorporating structured reasoning into the prompting process, CoT enables large language models to better capture nuanced, context-dependent sentiments that simple prompting methods often misclassify.

Our evaluation, conducted on 2,000 Amazon App reviews, showed that CoT prompting significantly outperformed simple prompting, achieving up to 93% accuracy compared to 84%. Notably, CoT was able to resolve 80% of the errors made by the simple prompt, particularly in cases involving conflicting or layered sentiments. The remaining errors primarily stemmed from vague language or extremely short reviews, highlighting ongoing challenges in sentiment disambiguation.

These findings emphasize the importance of reasoning-aware prompt design for tasks involving complex emotional and contextual interpretation. As LLMs continue to evolve, integrating reasoning strategies like CoT can enhance their practical utility in real-world applications such as customer feedback analysis, review summarization, and sentiment-aware recommendation systems.

Future work may explore extending this approach to multilingual and multimodal sentiment classification tasks, or integrating CoT prompting with fine-tuning techniques for further gains in interpretability and performance.

# References

[1] B. Pang and L. Lee, "Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales," in *Proceedings of the 43rd annual meeting of the association for computational linguistics (ACL'05)*, pp. 115–124, 2005.

[2] J. Wei, X. Wang, D. Schuurmans, M. Bosma, V. Zhao, K. Guu, et al., "Chain of thought prompting elicits reasoning in large language models," in *Advances in Neural Information Processing Systems*, vol. 35, pp. 2483–2494, 2022.

[3] E. Guzman and W. Maalej, "How do users like this feature? a fine-grained sentiment analysis of app reviews," in *2015 IEEE 23rd Inter-

national Requirements Engineering Conference (RE), pp. 153–162, IEEE, 2015.

[4] A. Gupta, M. Gupta, V. Singh, and P. Kumaraguru, "Scare: The sentiment corpus of app reviews with fine-grained annotations," in *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)*, 2016.

[5] E. Cambria, S. Poria, D. Hazarika, and K. Kwok, "Senticnet 5: Discovering conceptual primitives for advancing sentiment analysis," *AAAI*, vol. 34, no. 05, pp. 5076–5083, 2020.

[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, et al., "Language models are few-shot learners," in *Advances in Neural Information Processing Systems*, vol. 33, pp. 1877–1901, 2020.

[7] F. A. Shah, A. Sabir, R. Sharma, and D. Pfahl, "How effectively do llms extract feature-sentiment pairs from app reviews?," *arXiv preprint arXiv:2409.07162*, 2024.

[8] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, "Large language models are zero-shot reasoners," in *NeurIPS*, 2022.

[9] L. Peng, Y. Zhang, and J. Shang, "Controllable data augmentation for few-shot text mining with chain-of-thought attribute manipulation," *arXiv preprint arXiv:2307.07099*, 2023.

[10] D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. V. Le, and E. H. Chi, "Least-to-most prompting enables complex reasoning in large language models," *arXiv preprint arXiv:2205.10625*, 2022.

[11] Y. Zhang, R. Chen, and Q. Li, "Multimodal pear chain-of-thought reasoning for multimodal sentiment analysis," *Proceedings of the ACM on Multimedia*, vol. 2024, no. 10, pp. 1–12, 2024.

[12] M. Hannani, A. Soudi, and K. Van Laerhoven, "Evaluating chatgpt-4 and machine learning models for sentiment analysis on a multi-script moroccan arabic corpus," *ResearchGate Preprint*, 2025. Available at: https://www.researchgate.net/publication/390432057.