Text Classification via Large Language Models
Authors: Xiaofei Sun
Xiaoya Li
Jiwei Li
Fei Wu
Shangwei Guo
Tianwei Zhang
Guoyin Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Model
Text Classification
Intermediate Rationale Explanations

Abstract: Despite the remarkable success of large-scale Language Models (LLMs) such as GPT-3, their performances  still significantly underperform 
fine-tuned models  in the task of text classification.This is due to (1) the lack of reasoning ability in addressing complex  linguistic phenomena (e.g., intensification, contrast, irony etc); (2) limited number of tokens allowed in in-context learning. 

In this paper, we introduce \textbf{C}lue \textbf{A}nd \textbf{R}easoning \textbf{P}rompting (CARP). CARP adopts a progressive reasoning strategy tailored to addressing the complex linguistic phenomena involved in text classification: CARP first prompts LLMs to find superficial clues (e.g., keywords, tones, semantic relations, references, etc), based on which  a diagnostic reasoning process is induced for final decisions. To further address the limited-token issue, CARP uses a fine-tuned model on the supervised dataset for $k$NN demonstration search in the in-context learning, allowing the model to take the advantage of both LLM's generalization ability and the task-specific evidence provided by the  full labeled dataset. Remarkably, CARP yields new SOTA performances on 4 out of  5 widely-used text-classification benchmarks,  97.39 (+1.24) on SST-2,  96.40 (+0.72) on AGNews, 98.78 (+0.25) on R8 and 96.95 (+0.6) on R52, and a performance comparable to SOTA on MR (92.39 v.s. 93.3). More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups. Specifically, using 16 examples per class, CARP achieves comparable performances to supervised models with 1,024 examples per class.

Text Classification via Large Language Models
Xiaofei Sun**, Xiaoya Li Jiwei Li Fei Wu Shangwei Guo Tianwei Zhang,, Guoyin Wang*
Abstract
Introduction
Despite the remarkable success of large- scale Language Models (LLMSs) such as GPT-3, their performances still significantly underperform fine-tuned models in the task text classification. This due to (1) the lack of reasoning ability in addressing complex linguistic phenomena (e.g., intensification. contrast, irony etc); (2) limited number of tokens allowed in in-context learning.
Large language models (LLMs) (Radford et al., 2019a; Xue et al., 2020; Zhang et al., 2022a; Rae et al., 2021; Brown et al., 2020; Chowdhery et al., 2022; Ouyang al., 2022; Thoppilan et al., 2022) have shown the ability for in-context learning (ICL). Given few demonstration examples, LLMs are prompted to generate results for a new example, and have achieved performance comparable supervised baselines or even state-of-the-a results in a variety of natural language processing (NLP) tasks such as question answering (Trivedi et al., 2022), natural language inference, (Schick and Schüitze, 2020), named entity recognition (Wang et al., 2023), relation extraction (Wan al., 2023) and information extraction (Han al., 2021).
introduce Clue And Reasoning Prompting (CARP). CARP adopts a progressive reasoning strategy tailored to addressing the complex linguistic phenomena involved in text classification: CARP first prompts LLMs to find superficial clues (e.g.. keywords, tones, semantic relations, references, etc), based on which a diagnostic reasoning process is induced for final decisions. To further address the limited-token issue, CARP uses a fine-tuned model on the supervised dataset for ANN demonstration search in the in-context learning, allowing the model to take the advantage of both LLM's generalization ability and the task-specific evidence provided by the full labeled dataset.
In this paper,
In spite of the success, LLMs with ICL still significantly underperform fine-tuned models text classification. This is due two reasons: (1) Text classification requires models with more powerful reasoning abilities to resolve complex linguistic phenomenon including clause composition (e.g., concession, negation, intensification), irony, etc. Recent efforts improve LLMs' reasoning capabilities (Wei et al., 2022; Kojima al., 2022; Ye and Durrett, 2022: Zhang et al., 2022b) mainly focus on tackling math problems, and thus are not tailored addressing the reasoning process necessary for the multitude of intricate linguistic phenomena in text classification; (2) The number of demonstration examples allowed in in-context learning is limited, e.g., the longest context allowed for GPT-3 is 4,096 subtokens. Therefore, LLMs are only able to take the advantage of small proportion of the training set, performing well below supervised baselines;
Remarkably, CARP yields new SOTA performances on out of widely-used text-classffication benchmarks, 97.39 (+1.24) on SST-2, 96.40 (+0.72) on AGNews, 98.78 (+0.25) R8 and 96.95 (+0.6) R52, and a performance comparable to SOTA on MR (92.39 v.s. 93.3). More importantly, we find that CARP delivers impressive abilities low-resource and domain-adaptation setups: using examples per class, CARP achieves comparable performances to supervised models with 1,024 examples per class. Code is available at CARP! 2.
denotes equal contributions.
In this paper, introduce Clue And Reasoning Prompting (CARP), an extensible, annotation- free and efficient framework for text classification via large language models. To address the
* Zhejiang University. Shannon.Al. Amazon.
Nanyang Technological University. Chongqing University "xiaoya li shannonai.com, ctt.eiooomoooiiisiiiiiee xiaofoi_sun, jiwei li] @zju.ccuccc

---

This is an overall sentiment classifir for movis reviews.
First, list CLUES (le., keywords, phrases, contextual information, semantic relations, semantic meaning. tones, references) that support the sentiment dsteriiaation imput.
Second, deduce diagnostio REASONING process from promises (ie., clues, input that supports sentiment determination [Limit 130).
clues, reasoning input, determine overall SENTIMENT
Positive Negative.
INPUT: the delete
CLUES:: delett
REASONING: The phrrase "delete key" Implies an action of removing something, which could be interpreted as negative sentiment.
SENTIMENT: Nagasize
(a) CARP Zero-shot
This ovenall sentiment classifier for movie reviewa.
First, present CLUES (.e., kaywords, phrases, contectual information, semantic relation, semantic meaning, tones, references) that support the sentiment determination of imput.
Second, deduce diagnostic REASONING process from premises (ia., cluss, inpur] that supports the sentiment determination (Limit the number of words to 130). Third, determine the averall SENTIMENT of INPUT Positive or Negative considering CLUES, the REASONING process and the INPUT.
INPUT: noyoe
shocking history lesson than as drama.
Pesttiv: "more. "shooking," "history lesson, "drama." Negadive: None.
REASONING:
input contains positive such "eistory
"shooking." and "drama," which suggest sentiment positive, Additionally thing.
ganerally
SENTIMENT: Posttive
INPUT preae the delete
CLUES: + Positive: None. - Negatiie: "delete key."
The inpur contains the phrase "delate key." which impiles something is being removed or erased. This generally seen as negative action, suggesting that the sentiment of the Input is negathve.
REASONING:
SENTIMENT: Negative
(b) Few-shot (k=1)
Figure I: Examples of CARP prompts under zero-shots and few-shot settings. Comparisons of different prompts can be found in Appendix H.
reasoning process necessary for handling the linguistic phenomena in text classification, CARP decomposes the reasoning process into three steps, where LLMs are first prompted to find superficial clues (e.g.. keywords, tones, semantic relations, etc) in the given text; next, CARP treats the clues and input as premises and induce a diagnostic reasoning process; and finally determine the final label considering the above two steps. We find this progressive reasoning strategy to be effective in enhancing LLMs ability in language reasoning involved in text classification. Due the limited number of tokens allowed in context, more effective demonstration search is needed. CARP uses a fine-tuned (FT) model on the supervised dataset for KNN demonstration search for ICL. Since the fine-tuned model is trained based on task- specific labels, it guarantees that retrieved samples are close to the input sequence with respect to the task. FT-based demonstration search provides channel to connect LLMs with the full training set, in spite of the limited number of tokens allowed in demonstrations. This strategy lets the model take the advantage of both the L.LMs' generalization abilities and all task specific evidence provided by the training dataset.
performance comparable to SOTA on MR (92.39 v.s. 93.3). More importantly, we find that CARP delivers impressive ability on low-resource and domain adaptation setups with orders of magnitude fewer training examples. Specifically, CARP achieves comparable performances with 16 examples per class to supervised models trained on the full training set containing more than thousand examples per class. This demonstrates the capabilities of CARP in real-world text classification cases where training data is limited.
Related Work
2.1 Large Language Models
Large language models (LLMs) are models that are trained using self-teaching algorithms on large unlabeled corpora. LLMs be broadly divided into three categories based on the model architecture. The first category is the encoder-only model like BERT (Devlin et al., 2018). BERT (300M) (Devlin et al., 2018) and its variants (Liu et al., 2019; Sun et al., 2020; Clark et al., 2020; Feng et al., 2020: Sun et al., 2021) adopt the pre-training then fine-tuning paradigm for NLP tasks: use masked language models the main training objective for pretraining, and fine-tune the pretrained model in the annotated downstream datasets. The second category is the decoder- only models like GPT (Radford et al., 2019a). GPT (Radford et al., 2019a) uses the decoder
Remarkably, CARP yields new SOTA performances on four out of 5 widely-used text-classification benchmarks, 97.39 (+1.24) on SST-2, 96.40 (+0.72) on AGNews, 98.78 (+0.25) on R8 and 96.95 (+0.6) on R52, and

---

of an auto-regressive transformer (Vaswani al., 2017) model for predicting the next token in sequence. GPT (Radford et al., 2019a) and its variants (Dai et al., 2019; Keskar et al., 2019; Radford et al., 2019b; Chowdhery et al., 2022; Zhang et al., 2022a) also follow the pre-training then fine- paradigm. GPT-3 (175B) (Brown et al., 2020) proposes to formalize all NLP tasks generating textual responses condition the given prompt. The third category is the encoder- decoder models like T5 (Raffel et al., 2020). T5 (1IB) (Raffel et al., 2020) and its variants (Lewis et al., 2019; Xue et al., 2020).
reformulate input examples into cloze-style phrases and annotate the unlabeled text. Han et al. (2021) design sub-prompts and applied logic rules compose sub-prompts into final prompts. Liu et al. (2021) retrieve semantically-simila examples to test sample to formulate its corresponding prompt. Shi et al. (2022) retrieve label-words- similar examples as demonstrations in prompts.
Prompt Construction
3.1 Overview
We follow the standard prompt-based in-context learning paradigm. Given input sequence Tinpur { 32, 22,,,,,.. m1). the task of assigning a text-class label an input text is transformed generating a pre-defined textual response y C Verb (c.g., positive, negative, etc) conditioning on the prompt prmmpp using a language model.
2.2 In-context Learning
In-context learning (ICL) generates textual responses (i.e., label words) conditioning on the given prompt (usually) with few annotated examples for downstream tasks. Li and Liang (2021); Zhong et al. (2021); Qin and Eisner (2021) propose to optimize prompts in the continuous space. Rubin et al. (2021); Das et al. (2021); Liu et al. (2021); Su et al. (2022) introduce different strategies for selecting in- context examples. Lampinen et al. (2022) show that explamations of examples in sew-shot prompt lead to performance boost. Marasović et al. (2021) find that GPT-3 outperforms other models by large margin in the explanation generation task. Wei al. (2022) propose chain-of-thought reasoning and utilized <input, chain-of-thought, output> triples as the prompt for LLMs. Wiegreffe et al. (2021) traine a supervised filter to select explanations generated by GPT-3 on the SNLI and CommonsenseQA tasks.
3.2 Prompt Construction
The prompt Eprompt, which is constructed based on æ, consists of the following three components:
(1) Task description adese generally describes the task. For different classification tasks, e..g. sentiment classification, topic classification, etc, descriptions are different. Take the sentiment classification task as example, the task description is given follows: Classify the overall sentiment of the imput as positive or negative
(2) Demonstration consists a sequence annotated examples:
2.3 Text Classification
(( Wdcmo): .... (2 demo F }
Text classification is task that aims to assign predefined labels (e.g., sentiment, topic, etc) to given text. Earlier work decouple the task into two steps: (1) extract features using neural models such as RNNs (Irsoy and Cardie, 2014; Yang et al., 2016; Wang et al., 20 Liu et al., 2016; Xie et al., 2020), CNNs (Kim, 2014; Zhang et al., 2015; Lai et al., 2015; Conneau et al., 2016; Wei and Zou, 2019), GCN (Yao et al., 2019), LLMs (Howard and Ruder, 2018; Sun et al., 2019; Chai al., 2020; Chen al., 2020; Lin al., 2021); and (2) feed extracted features into a classifier (Joulin et al.. 2016; Tang and Surdeanu, 2023) to obtain the final label.
where ad S S k denotes the jihh input sequence and y denotes the text which is transformed from the label, e.g., positive or negative for the binary sentiment classification task. Demonstration serves as two purposes: (1) providing the LLM with evidence to consult on for decision making, which will significantly boost performances; (2) provides output format that LLM's outputs need follow, that the output, which takes the form of natural language, can be further easily transformed to labels. is worth noting that demonstrations are only needed for the few-shot setup, but not for the zero-shot setup.
With LLMs, Schick and Schütze (2020)

---

imput sequence with respect to the task. Using fine- tuned model provides a channel to connect LLMs The prompt 2prompr for a test imput is constructed by concatenating the task description deeses a sequence of demonstrations (a v..mo)....,. (:: )) and the test sequence Ueest which can be given as follows:
Finetumed Model FT for short. The key disadvantage of SimCSE (Gao et al., 2021) and other general semantic encoding models (Reimers and Gurevych, 2019; Seonwoo et al., 2022: Sun al., 2022) is that measures the general semantic similarity but not specifically tailored the text classification task. To resolve this issue, CARP uses the model fine-tuned on the training dataset as the &NN encoder model. Specifically, we first fine- tune Roberta model on the training data. Next use the [CLS] embedding the sentence level representation for KNN search. Since the fine- tuned model is trained based on task-specific labels, it guarantees that retrieved samples are close to the (3) Input Vinput is the test text sequence to classify.
with the full training set, in spite of the limited number of tokens allowed in demonstrations. This strategy lets the model take the advantage of both the LLMs' generalization abilities and all task- specific evidence provided by the training dataset.
(Edeses in; <demo>
An; Frest)
Clues Collecting and Reasoning
3.3 Demonstration Sampling
To enhance the models' reasoning ability in addressing linguistic phenomenon tailored to text classification, we propose progressive reasoning strategy that involves clue collection, reasoning and decision making. This process also mimics how human decisions: where we first collect evidence from the input, separating chaff from wheat; next piece together local evidence form global picture, which leads final decision making. Next we first given an overview of the the clue collecting and reasoning process, and then describe implementation details.
The few-shot setup requires demonstrations sampled from the training set. Strategies that explore include:
Random Sampling straightforward strategy from samplings is to randomly sample k examples from the training set DD Damin for text sequence Bestt-
ANN Sampling The key disadvantage for random sampling is that there is no guarantee that selected samples semantically related to the input sequence. One straightforward alternative is to sample examples that are similar to the test sequence using ANN search (Khandelwal et al., 2020). In this process, the test sequence Diew is first mapped to a vector User using an encoder model f. Then using Urest as the query, we search through the entire training set main to retrieve kt nearest text sequence to get & nearest data examples V {ay, M y;3f=1 as demonstrations. We use the following encoder models to obtain sentence representations and similarity scores:
4.1 Overview
Collecting Clues For test sequence, clues are local fact evidence such as keywords, phrases, contextual information, semantic meaning, semantic relationships, tones, references, etc. The following is an example for clues of an input: Input: Steers turns in a snappy screenplay that curls at the edges; so clever you want to hate Clues: "snappy", "clever", "want to hate it" are clues for determining the sentiment of the input sentence.
SimCSE is contrastive learning model for sentence embeddings(Gao et al., 2021).
Reasoning For reasoning, the LLM is prompted to go beyond superficial keywords mine deeper perspectives, considering language phenomenon such as negation, intensification, irony, etc), and piece together local evidence to form the final decision. The following example shows the reasoning process to decide the sentiment of the above example based on the evidence collected:
The phrase "snappy screenplay" " implies that the screenplay is of a high quality and is well-crafted.
2. The phrase "curls at the edges" implies that the screenplay is cleverly written.
3.
Decision Making Based on the reasoning process, the model makes the decision for the sentiment of the given input:
WWe use S Sup-S MCCSF-RROBEE Taage model as an encoder model.

---

Overall, the clues and reasoning process point to a positive sentiment for the input sentence.
INPUT: <lext> SENTIMENT: <label-word>
The merits for the incorporation of clue finding and reasonings are as follows: (I) it prompts the model to progressively think and make decisions: clue finding focuses more on superficial features such keywords, while reasoning makes deeper justifications based superficial features; (2) clue finding and reasoning serve tunnel to let human intervene: in the few-shot setup, where clues and reasons need be prepared in advance for demonstrations, we can modify them as we see fit.
Reasoning Generation Based on clues generated clues, the input, and the label, we ask LLMs to generate reasoning details²:
Based the input and clues, articulate the diagnostic reasoning process that supports the sentiment determination of the input.
INPUT: <text>
LABEL: <label-word>
CLUES; <clues>
This extremely helpful for trouble shooting in the prompt-construction stage for error corrections; (3) from an interpretation and uncertainty estimation perspective, clues and reasoning in few-shot setups are hlmann-redaale influence functions;
REASONING:
Given the generated clues and reasonings for all training examples, at test time, when K-nearest examples are selected demonstrations, its corresponding clues and reasons are concatenated to the demonstration. In this way, each demonstration example is composed by (text, clues, reasons, gol den label word) pair. Examples for prompts with clues and reasons are shown in Figure 4. In this way, for a test example, by following the format demonstrations, the LLM will first output clues, then reasons, and at last decisions.
4.1.1 zero-shot scenario
the zero-shot setup, as no demonstration allowed, no concrete example for clues and reasons can be provided in the prompt. In this way, we only add requests asking the model to output clues and reasons in the prompt.
4.1.2 few-shot scenario
the few-shot setup we need to prepare clues and reasonings for all examples in the training set in advance as all training examples have chances to be selected as demonstrations given different test inputs. Previous efforts in math problems (Wei et al., 2022; Kojima et al., 2022; Ye and Durrett, 2022: Zhang et al., 2022b) prepare hand-drafted reasoning for few examples, and always use these example as demonstrations. This strategy does not fit for our situation it extremely time- intensive to manually generate clues and reasonings for all training examples. To resolve this issue, we harness LLMs for automatic clue and reasoning generation, where we ask LLMs to generate clues and reasoning based on both the input and its corresponding label.
4.2 Voting
Unlike conventional discriminative models for text classification, which generate deterministic results during inferences, LLMs for in-context learning generative models and generate distinct textual responses with diverse sampling strategies in multiple runs. We consider the following voting strategies in the paper:
Majority Vote: the final result is the most frequent prediction among multiple runs. Weighted Probability Vote: the final result is the one with weighted summed probability from multiple runs.
5 Experiments
Clue Generation For a given training example <lex/> paired with the label word <label-word> (e.g., positive), we ask LLM to generate clues that indicate the label:
In order to evaluate the effectiveness of the proposed method, we conduct experiments on two setups: (1) full training setup, where the model has the access to the full training data; and (2) low- resource setup, where the model can only access partial training dataset. The low-resource setup
List CLUES (i.e., keywords, phrases, contextual information, semantic meaning, semantic relationships, tones, references) that support the sentiment determination of the input (limit to 15 words).
'LLMs often generate long responses, in order to ensemble more demonstrations in prompts, we use "limit to 50 words". After conducting an analysis of the generated responses, we find that LLMSs can explain the reason within limited words.

---

Datowet
Model
2::256
Supervinal Methoals
FT RoBERTs GPT-3 Vanilla
51.52
52:31
$3.89
70,49
90.30
BubEkKl> Large clas at 2019)
DeSEkki ilkc al, 2000
34.25
$5.99
05.31
65.55
93.33
27.26
96.42
96.32
90,19
91.16
95.38
94.99
SST-2
GPT-1 Col
89.06
90.15
90.36
90.19
91.70
90.80
91.62
07.86
91.89
94.68
VI AMD Hereee and Bumon 2014)
GEN-SIL (Tong 20077
Not (Nag 20199
GCN (Lis 2021)
05.95
83.37
MGNess
GPT-3 Vamilla
FT RoRERRS
GPT-3 CRAP
89.17
2187
90.45
89.63
91.07
16.00
90.51
91.77
42005
94,00
90.00
50.15
91.
95.20
7K09
GPT-3 Zhro-shot Col
89.66
91.20
95. 15.28
swattigg
GPT-3 CRAP
91.07
RoBERT
GeT (Raeiraa 2022)
92.11
91.25
90-45
91.34
$9.57
90.99
GPT-2
91.70
91.11
CARP
$3.01
92.80
91.75
01.90
$1.34
91.82
GPT-3
91:81
Fun-ohel Settin (tmta)
91.27
Viailla 2000)
Aandox Sumpler
91.55
99.15
R55
FT RoBEKK GFT-3 Vanilla
GPT-3 Col
89.15
89.46
38.29
90.04
90.02
39.10
90.75
59.18
90:29
93.20
67.19
91.85
94.12
92.06
81.53
CoTl (Kginna 2022) 2022
CARP
$4.56
95.20
05.62
95.18
92-09
90.00
91.03
96.19
99.91
93,03
92.90
03.04
GPT-3 CRAP FT RoBERT GPT-3 Vanilla (PT-3 CGT
51.20
90:82
52.11
91:00
95.85
53.58
94. 94.36
68.25
96.27
88.37
Vasila (limmo 2000
93-M0
95-50
94.56
$3.40
$9.99
9408
MR
86.36
86.04
89.00
85.68
88.99
90.01
90.16
$9,80
90.89
90.18
CUT (ogira atal, 2017))
94:21
16-38
4510
$2.48
93.27
GPT-3 CRAP
86.31
87.19
89.
$0.01
91.20
CARP
05.00
95.35
97.85
$6.73
90.24
Vaaila Buown di 2000
94.01
96.14
95.57
95.79
93.00
94.08
Table 2: Experimental results on low-resource (n
CUT (Kgina 2022)
95.48
54.85
35.59
95.89
93.17
94.00
example per class) settings. We compare fine-tuned RoBERTa-Large with 16-shots GPT-3 setting. For GPT- 3. we use SimCSE (Gao et al., 2021) to retrieve annotated examples from the low-resource train set.
Table Accuracy performances of different settings on benchmarks. We report mean results over 5 runs. The In few-shot
GPT-3 denotes text -exitivvvinc-00 experiments, we sample 16 annotated examples (k=16) for every test instance. indicates existing SOTA results. "WP Vote" denotes weighted probability vote.
FT RoBERTa on Train
FT RoBERTa Yelp Train
SST-2 Test
95.99
88.78
Yelp Test
92. 38
96.04
CARP with SST-2 demon.
CARP with Yelp demon.
better mimics real-world situations where training data is limited. For the full training setup, we follow the standard train/dew/test split. For the low- resource setup, randomly sample n instances per class (n in {16, 128, 256, 512, 1024)) from the benchmark training set. The sampled subset forms a new training set test different models' abilities in the low-resource situations, During experiments, we train modess/demonsttration with the new training set.
SST2 Test
Yelp Test
95.94
96.32
Table Results for Yelp test set when using in In-domain/out-of of-domain ÀNN sampler and demonstrations source. We use FT ANN Sampler to retrieve demonstrations on the corresponding train set.
that mimics the -hain-of-thought strategy and CARP to denote the proposed method.
We conduct experiments on five widely-used datasets, including SST-2 (Socher et al., 2013), R8, R525, AGNews (Zhang et al., 2015) and Movie Review (MR) (Pang and Lee, 2005). More details of the benchmarks and low-resource datasets can be found in Appendix
5.1 Models for Comparison
Supervised models are trained on the trained set naturally constitute baselines to compare with. We use the six models. including RoBERTa-Large, ROBERTa-GGO
DeBERTa, XLNet, GCN-SB, and VLAWE. Details of the models and hyper- parameters are shown in Appendix G.2:
For zero-shot and few-shot experiments, we use InstructGPT-3 (Ouyang et al., 2022) sexxt-dvinci--oe -003 175B) as the backbone.
Due to the input token limitation, we use A - 16 for few-shot setups. Prompts on the five datasets are shown in Appendix H. Model haper-parameters can be found in Table 13 7, We use Vanilla to denote the conventional ICL approach where LLMs are prompted to generate labels, use CoT (Kojima et al., 2022) to denote the baseline
Few-shot Setup For demonstration sample strategies in the few-shot setup, we consider the following strategies for comparison: (more details can be found in Section 3.3):
Random Sampler: randomly samples k
examples.
SimCSE ANN-Sampler: samples &NN based on SiMCSE
"R8 and R52 are original from https: htss://wwww /wwww//wcco cs. umb. sau/-smimaze samimmrog/textttt textminingg /tg///aattss/////////
'During experiments, we find that CARP is robust with different parammets,, Experimental results can be found in Appendix G.3
FT kNN-Sampler: sample &NN using Fine-
Tuned RoBERTa representations.

---

5.2 Results the full training set
of CARP in the low-resource setup. As we anticipated, the kNN search efficiency improved at faster rate as the amount of the training data increases; Enlarging the training dataset increases the chances that the chosen examples will correspond to the input, resulting in improved results. Specifically, using 16 examples per class, CARP achieves comparable performances to supervised models with 1,024 examples per class; using 512 instance per class annotation data, CARP achieves comparable performances to supervised models trained on the full set.
Experimental results are shown in Table 1. As can be seen, performances of few-shot setups consistently outperform zero-shot setups. In terms of sampling strategies in the few-shot setups, we observe that simese KNN-sampler outperform random sampler. illustrating the importance of adding demonstrations that are relevant to the test input in the few-shot setup. We also observe that FT KNN-sampler consistently outperforms simese KNN-sampler. This shows that, the fine- tuned model, which takes the advantage of the full training set, serves as a better retriever for task- specific demonstration retrieval than the general- purposed SimCSE retriever.
5,4 Domain Adaptation
It is unclear whether training models on the specific dataset for retrieving demonstrations is essential. In this subsection, we conduct an analysis using demonstrations from out-of-distribation datasets. We use SST-2 and Yelp, and the task is to determine the positive or negative polarity of the given text. SST-2 and Yelp are from different domains: SST-2 are snippets from Rotten Tomatoes*, whereas Yelp consists of product reviews from the online website. Experimental results are shown in Table 3. SST-2 train & SST-2 test means that demonstrations are from the SST-2 dataset and test is performed on SST-2 dataset; Yelp train & SST-2 test means demonstrations are from Yelp and test is performed on SST-2 dataset. We see a significant decrease (-7.2%, 95. 59.99%% v.s.88.78% in performance when switching SST-2 train to Yelp train using supervised RoBERTa, which illustrates that supervised models are very sensitive to the out-of-distributie data. On the contrary, we only observe slight decrease in performance (-0.5%, 96.80% v.s. 96.29%) when switching SST-2 train to Yelp-2 train on SST-2 test, illustration the greater capabilities of CARP on the domain adaptation situations. This means CARP is very robust when training and test are not from the same domain.
For different reasoning strategies, we first observe that the CoT strategy outperforms the vanilla strategy, which straightforwardly asks LLMs to generate results without further reasoning steps. CARP consistently outperforms CoT across all benchmarks, i.e,, +1.48, +0.97, +2.76, + 3.29, +0.47 respectively on SST-2, AGNews, R8, R52 and MR datasets. This demonstrates the necessity of building models with complex linguistic phenomena involved in text classification, and the effectiveness of CARP in doing this job. Compared with supervised learning baselines, we find that the vanilla model using LLM underperforms supervised baselines, while few- shot CoT is able to obtain slightly worse or comparable results agains supervised baselines. Notably, single CARP outperforms fine-tuned RoBBERTa on all benchmarks. Using WP voting strategies, CARP yields new SOTA performances on four out of the datasets, 97.39 on SST-2 (+1.24), 96.40 (+0.72) on AGNews, 98.78 (+0.25) on R8 and 96.95 (+0.6) on R52, and performance comparable to SOTA on MR (92.39 v.s. 93.3).
5.3 Results on low-resource settings
To estimate low-resource circumstances, we sample n = {16, 128, 256, 512, 1024) instances for each class as low-resource setups. Experimental results are shown in Table 2. As can be seen, when the training set size is extremely small (i.e., 16 or 128 sentences), and the performance of the supervised model is far below CARP, Even with only 16 examples train on, the accuracy of CARP of SST-2 already around 90%, whereas supervised models' performance is similar to random guess. This demonstrates the strong generalization ability
Ablation Studies
6.1 Impact of the number of demonstrations
We explore the effect of the number of demonstrations in prompts using SST-2 Results for CARP using different sampling strategies are shown in Figure 2. As can be seen, performances improve as the number of demonstrations increases, which is in line with our expectation.

---

Strategy
Label Words(+,-)
CARP
Position
One,
Annotation Words
Positive, Negative
96.86
Synonyms Words
Great, Teerible
96.27
06
Text Actuncy
Flipped Words
Negative, Positive
64.63
Random Words
Cr.
95.06
86
Special Tokens
<POS>,
<NEO>
96.65
04
Table 5: Label words and results on the SST-2 dataset with different strategies. "+" represents positive polarity; denotes negative polarity.
50
02
16
20
24
Number Demonsinaions
Ranking
SimCSE
FT
Figure Performances V.S.
the number of
CARP
demonstrations for CARP.
Random
95.39
95.99
High-to-Low
95.22
96.71
Prompts
S8T-2
R8
Low-to-High
96.39
96.80
CARP
96.80
98.29
Table 6: Accuracy scores SST-2 when assembling demonstrations with different ranking strategies.
w/o Text
92.28
94.18
w/o Clue
95.48
95.29
w/o Reason
95.72
97.82
Label
96.53
Table The effect components on SST-2 dataset with different strategies.
Special tokens: tokens that do not have semantic meaning. They are independent of the input and added for certain purpose. e.g..
<els>, <mask>.
6.2
The effeet of components in demonstrations
Results are shown in Table 5. As can be seen, using annotation words as label words achieves the best performances. We also observe significant performance decrease when flipped words are used as label words in demonstrations.
CARP
uses (text, clues, reasons, pairs as
golden label word) demonstrations. In this subsection, we exploit the influence of each component (text, clues, by
reasons, golden label word) removing it from prompts. Experimental results are shown in Table 4. As shown in Table 4, text in demonstrations has the greatest impact, followed by clue, reason and label.
6.4 The effect demonstration order
During experiments, we find that the ranking order of demonstration affect final results. In this subsection, we further investigate the influence of orders of demonstrations. demonstrations investigate include:
Orders the
6.3 The effect of different types of label words
Label words denote words generated by LLMs that indicate the label of the input. We explore the impact of using different kinds of label words:
Random: randomly shuffle retrieved demonstrations.
Low-to-High: demonstrations with lower similarity scores come first. Therefore demonstrations with higher similarity scores are placed closer to the sequence, which is placed at the end of the prompt.
Position index: e.g., one, two, three, etc.
Annotation words: e.g., positive, negative."
Synonyms words: c.g., great, terrible.
Flipped words: words that are contrary to original target meanings. e.g., "positive" to denote the negative polarity, "negative" to denote the positive polarity.
High-to-Low: demonstrations with lower similarity scores are placed closer to the test sequence.
Random words: randomly choose words in the vocabulary.
As shown in Table 6, the performance is sensitive the ordering of the demonstrations. The low- tohhigh ordering achieves the best performance compared to the random and high-to-low ordering.
'GPT-3 generates the same lahel words for binary sentiment classification task.

---

7 Conclusion
Kevin Clark, Minh-Thamg Luong, Quoe Lc, and Christopher D Manning. 2020. Electra: Pre- tre-training text encoders discriminators rather than generators. arXiv preprint arXiv:2003. 10555.
In this paper, we introduce Clue And Reasoning Prompting (CARP) for text classification task. CARP yields new SOTA performances on 4 out 5 widely-used text-clasification benchmarks. More importantly, we find that CARP delivers impressive abilities on low-resource and domain- adaption setups. In the future, we would like to explore CARP on more natural language understanding tasks.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. 2016. Very deep convolutional networks for text classification. arXiv preprint 1rxiv:1606.01781.
Zihang Dai. Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint DrXiv:1901.02
Acknowledgements
Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay-Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. Case-based reasoning for natural language queries over knowledge bases. arXiv preprint WwXiv2214.08776
This work is supported by the National Key R&D Program of China ($Q2022AAA000144)
Limitations
Despite the overall promising results, CARP still faces the following shortcomings: (1) clues that are contributing for making decisions are hard to annotate; (2) LLMs still suffer from the token limit issue.
Jacob Devlin, Ming-Wac Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training
of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.0485.
Zhangyin Feng, Daya Guo. Duyu Tang, Nan Duan, Xiaocheng Feng. Ming Gong, Linjun Shou, Bing Qin. Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08755.
References
Samuel R. Bowmam, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint 104.0881...
Xu Han, Weilin Zhao Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. Ptr: Prompt tuning with rules for text classification. arXiv preprint axXi:2105.11259.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Pengeheng He. Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint 18Xiv:1801.066 166.66.....
Duo Chai, Wei Wu, Qinghong Han, Fei Wu, and Jiwei Li. 2020. Description based text classification with reinforcement learning. In International Conference on Machine Learning. PMLR.
Radu Tudor Tonescu and Andrei M Butnaru. 2019. Vector of locally-aggregateed word embeddings (vlawe): novel document-level representation. arXiv preprint 19Xi::1902.08880
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mixtext: sunguistically-informe informed interpolation hidden space for semi-supervised text classification. arXiv preprint arXiv:2004. 12239.
Ozan Irsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionaliit in language. Advances in neural information processing systems, 27.
Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag tricks for efficient text classification, arXiv preprint

---

Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint urXiv:1909.05858.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin. Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Nearest neighbor machine translation. arXiv preprint
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cx/0506075.
arXiv:2010.00710.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Conference on Empirical Methods in Natural Language P Processing.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying Ims with mixtures of soft prompts. arXiv preprial Mix::2144.00577
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. ArXiv.
Alec Radford, Jeff Wu, Rewon Child, David Luan,
Dario Amodei, and Ilya Sutskever. 2019a. Language models are unsupervised multitask learners.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015, Recurrent convolutional neural networks for text classification. In Proceedings of the AAAI conference on artificial intelligence.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019b. Language models are unsupervised multitask learners. OpenAl blog.
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. 2022. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susannah Young, al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 211.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Ahdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zeulemoyer. 2019. Bart: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 1910. 19910.13400 13461.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to- text transformer. The Journal of Machine Learning Research, 21 21(1):5485-5511.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv prepriat Xiv::1011000/99
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks arXiv prepriat arXiv:1908. 1908.
Yuxiao Lin, Yuxian Meng, Xiaofei Sun, Qinghong Han, Kun Kuang, Jiwei Li, and Fei Wu. 2021. Bertgen: Transductive text classification by combining gen and bert. arXiv preprint arXiv:2105.05722
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning, arXiv preprint ar2.v:2112.00633.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint
Timo Schick and Hinrich Schütze. 2020. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07672
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning, arXiv preprint arxiv:1605.05101.
Yeon Sconwoo, Guoyin Wang. Sajal Choudhary, Changmin Seo, Jiwei Li, Xiang Li, Puyang Xu, Sunghyun Park, and Alice Oh. 2022. Ranking- enhanced unsupervised sentence representation learning. arXiv preprint arXiv:2209.044 043333
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 116.2
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. 2022. Nearest neighbor zero-shot inference. arXiv preprint arXiv:2205. 13792.
Richard Socher, Alex Perelygin, Jean Wu. Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical
Ana Marasovié, Iz Beltagy, Doug Downey, and Matthew E Peters. 2021. Few-shot self- rationalization with natural language prompts. arXiv preprini arXiv:2171.02244

---

methods in natural language processing, pages 1631- 1642.
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang. Ricardo Henao, and Lawrence Carin. 2018. Joint embedding of words and labels for text classification. arXiv preprint 18X.:::805.0477.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang. Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. 2022. Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.0193 01975.
Shuhe Wang. Xiaofei Sun, Xiaoya Li, Rongbin Ouyang. Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304. 10428.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18-20, 2019, Proceedings 18. Springer.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.1
Xiaofei Sun, Yuxian Meng, Xiang Ao, Wu, Tianwei Zhang, Jiwei Li, and Chun Fan. 2022. Sentence similarity based on contexts. Transactions of the Association for Computational Linguistics.
Jason Wei and Kai Zou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11190
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng,
Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI conference on artificial intelligence, volume 34.
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta. Mark Riedl, and Yejin Choi. 2021. Reframing human-ai collaboration for generating free-text explanations. arXiv preprint arXiv:2112.0867..
Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu, and Jiwei Li. 2021. Chinesebert: Chinese pretraining enhanced by glyph and pinyin information. arXiv preprint arXiv:2106. 16038.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. 2020. Unsupervised data augmentation for consistency training. Advances in neural information processing systems.
Jian Tang, Meng Qu, and Qiaozhu Mei. 2015. Pte: Predictive text embedding through large-scale heterogeneous text networks. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1165- 1174.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11 19334.
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32.
Zheng Tang and Mihai Surdeanu. 2023. It takes two flints to make fire: Multitask learning of neural relation and explanation classifiers. Computational Linguistics, 1971)1117-156.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-TT Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, al. 2022. Lamda: Language models for dialog applications. arxiv preprint XxXv:2201.082339
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480-1489.
Harsh Trivedi, Niramjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge- intensive multi-step questions. arXiv preprint arXiv:2212.10500
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370-7377
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Mia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Xi
Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems.
Zhen Wan. Fei Cheng, Zhuoyuan Mao. Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023. Gpt-re: In-context learning for relation extraction using large language models. arXiy preprint arXiv:2305.02110
Fang 7eng, Niannian Chen, Dan Yang, and Zhigang Meng. 2022. Simplifeed-boosting ensemble convolutional network for text classification. Neural Process. Leit., 54(6).

---

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022a. Opt: Open pre-trained transformer language models. arXiv preprini urXiv:2205.0000 0.000..
w/o semantic relationships: semantic relationships refer relations between subjects such as "emotional danger" suggests a romantic and thrilling relationship between Idemoto and Kim that creates a positive sentiment..
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
w/o tones: tones the general mood of the text such as The sentence is expressed in an objective tone.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain thought prompting in large language models. arXiv preprint arXiv:2210.03493.
w/o references: references are mentions of commonsense facts books such as The reference to the popular, comedic character "Ferris Bueller" implies that the kid is seen in a positive light..
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing [mask]: Learning learning recall. arXiv preprint arXiv:2104.022.
Experimental results are shown in Table 8. For R8 and SST-2 datasets, keywords play the key role for GPT predictions.
A Prompts
In Figure 3, present examples of vanilla, CoT (Kojima al., 2022) and the proposed CARP prompts in the zero-shot setting.
C.2 Quality of the reasoning process
In this paper, we use LLMSs to generate rationable explanations instead of human editing. Therefore, the quality of generated reasoning process affects the final results. In this subsection, we sample 500 training (text, clues, reason, label) pairs and evaluate the generated reasoning process from the following perspectives:
In Figure 4, we present examples of vanilla, CoT (Kojima al., 2022) and the proposed CARP prompts in the few-shot (k=1) setting.
B Experimental results
In Table 7, we present the experimental results on text classification subsets.
(1) Reliability: Inspired by the emergent generalization ability of LLMs, we use zero-shot GPT-3 (175B) as the self-critique model to evaluate the quality of generated reasoning processes. To be specific, we ask the GPT-3 to return yes/no if the generated reasoning process supports making decisions for the input text. the GPT-3 returns "yes", denotes that the reasoning process reliable for making decisions. the GPT-3 returns "no", it represents that the reasoning process is not reliable.
More ablation studies
C.1 The influence of clues
Celues includare keywords, phrases, contextual information, semantic meaning, semantic relationships, tones, references that support making decisions. We remove different types of words in clues and evaluate its influence on SST-2 and R8 datasets. Editing prompts achieve this goal. The original prompt for clue collecting is List CLUES (i.e., keywords, phrases, contextual information, semantic meaning. semantic relationships, tones, references) that support the sentiment determination of the input. If we want to remove keywords & phrases, we just remove them from the prompt.
The prompt for SST-2 is shown follows: Is the following REASONING process supporting determinate sentiment label to INPUT? Please answer Yes or No.
INPUT: <text>
REASONING: <reasoning-process
w/o keywords & phrases: keywords and phrases are surface evidence for making decisions such as "like", "hate".
where <text> the text sequence for the data and <reasoning-procees>>>>> generated reasoning process.
w/o contextual information & semantic meaning: contextual information and semantic meaning are meaning in sentences/paragraahss such as The author express his happiness.
(2) Fluency: use LLMs to generate reasoning explanations a reference-ffrn free text generation task. We use perplexity evaluate the generated text. (3) Logic Faithful: previous work often use models, which are trained on natural language

---

This is overall sentiment classifier for movie reviews. Classify the overall SENTIMENT the INPUT as Positive or Negative. INPUT: press the delete key
SENTIMENT: Neutral
X
(a) Vanilla Zero-shot
This is an overall sentiment classifier for movie reviews. Classity the overall SENTIMENT the INPUT as Positive or Negative.
INPUT: press the delete key
step- input contain any sentiment as aither positive or nagattive
sentiment, so not possible classify
X
(b) CoT Zero-shot
This overall sentiment classifier for movie reviews, (i.e., keywords, phrases, sentiment determination of input..
information,
meaning, tones, references) support
Second, deduce the diagnostic REASONING process from premises (i.e,, clues, input) that supports the INPUT sentiment determination (Limit the number of words 130).
clues, reasoning Input, determine
as Nogative.
INPUT: prees the delete key
CLUES: delets
REASONING: The phrase "delete key" implies action removing something, which could be Interpreted as negative sentiment: SENTIMENT: Negative
(c) CARP Zero-shot
Figure 3: Examples zero-shot prompting methods for text classification task: (a) represents for vanilla prompting method; (b) denotes for the Chain-of-Thought (CoT) (Kojima et al., 2022) prompting method; represents for the proposed CARP prompting method.
inference datasets, determine whether the given "hypothesis" logically follows from the "premise". However, lacking annotation datasets, NLI-trained models can not generalize across multiple domains (e.g., opinion, reviews, news). Since then, we use 16-shot ICL with GPT-3 to evaluate whether the generated rationable explanations can be entailed from the input text. If the InstructGPT responds with "entailment", it denotes that the generated reasoning process is logic faithful with the text. Otherwise, represents the reasoning process not faithful to the text. We sample training instances from the SNLI dataset (Bowman et al., 2015) as demonstrations. And prompts are shown as follows:
D Dataset
SST-2 (Socher al., 2013). R8, R52 AGNews (Zhang et al., 2015) and MR (Movic Review) (Pang and Lee, 2005).
SST-2:
The original data in SST-2 are sampled from snippets of Rotten Tomatoes HTML files. We use the same train/dev/test splits with Socher et al. (2013).
R8 and R52: R8 and R5211 two subsections of the Reuters collection. containing and 52 classifications, respectively. The R8 dataset composed of 5,485 documents for training and 2, 189 documents for testing. The R52 dataset is composed of 6,532 training and 2,568 test documents.
Given the premise and hypothesis, please justify whether the HYPOTHESIS can be entailed from the PREMISE. Please return yes no.
PREMISE: <text>
HYPOTHESIS: caaasoning-processs
Evaluation results are shown in Table 9. As can be seen, the reliability percentages for SST- 2 and R5 are higher than 95%. This indicates that it is feasible to use the model-generated reasoning process as part of the prompts to augment ICL performances. The perplexity of generated reasoning text is smaller than 4, which denotes that the generated reasoning text is fluent. And scores logic faithful are larger than 93%, which in line with our expectation that LLMs can generate reasonable explanations.
AGNews: The AG News consists of news articles from the AG's corpus. The dataset contains 30,000 training and 1,900 testing examples for each class.
MR (Movie Review): The MR contains reviews of films for determining whether a sentiment either positive or negative. The corpus has 10,662 reviews. We follow (Tang et al., 2015) and use the same train/test split.

---

Classify the
Negative.
noyce
shocking history
BENTIMENT:
INPUT: press the delete key
BENTIMENT: Neutral X
(a) Vanilla One-shot
This an overall sentiment classifter for movie reviews. Classify the overall SENTIMENT of the INPUT as Positive or Negative.
as shocking history lesson SENTIMENT: Let's think step-by-step step The Input states that the films are "more" something than something else. This implies that the films are at least somowhat the first thing mentioned, this case, the films are more of "shocking history losson" than "drama".. This implias that the sentiment is positive, as the films are more of history lesson than drama. The overall sentiment is Positive.
nayce
drama.
INPUT: press the delete key SENTIMENT: Let's think step step. The Input does not provide any context or opinion, so it impossible to determine the sentiment. Therefons, the overall sentiment is Neutral,
(b) CoT One-shot
sentiment classiffer reviows.
First, CLUES keywords, contextual Information, semantic relations, meaning, references) that support
sentiment determination
Second, deduce diagnostic REASONING
determination (Limit the number process the
Third,
Negative considering CLUES,
noyce films
shocking history lesson than Nugativa: None.
CLUES:
Positive:" "more." "shocking." "bistory lesson"" "drama."
contains positive words such "more," "shocking." "drama." which suggest the sentiment is positive, Additionally, the phrsse "history lesson" implies that film educational and informative, which generally seen as positive thing. Therefore, the sentiment the impul positive.
REASONING:
SENTIMENT: Positive
INPUT: presa the deleta key
CLUES:
Positive: None. Negadive: "delete key."
REASONING: The imput contains the phrase "delete key." which implies that something being removed or erased. This generally seen as
negative action, suggesting that the sentiment of the input is negelive.
SENTIMENT:
Nesathik
CARP One-shot
Figure 4: Examples of few-shot (k=1) prompting methods for the text classification task: (a) represents for the vanilla prompting method; (b) denotes for the Shain-of-Thoughh (CoT) (Kojima et al., 2022) prompting method; (c) represents for the proposed CARP prompting method.
E
Case Analysis
GCN-SB:Zeng et al. (2022) propose a simplified boosting algorithm, which makes CNN learn the samples misclassified by GCN again.
We show generated clues, reasoning explanations examples in Table 12.
VLAWE:: and Butnaru (2019) obtain document embeddings based on aggregating the differences between each codeword vector and each word vector (from the document) associated to the respective codeword.
F Baselines
As mentioned in Section 5.1, we use the following supervised models as baselines. More details of the models follows:
ToBERTa-Large:We fine-tune RoBERTa- Large (Liu et al., 2019) on the training set. RoBERTa-GCN:Lin et al. (2021) constructs
G
Hyper-parameters
heterogemeouus graph networks on top of the RoBERTa-Large (Liu et al., 2019) model. DeBERTa:He et al. (2020) improve RoBERTa by using disentangled attention mechanism and an enhanced mask decoder.
G.1 GPT API Hyper-parameters
Hyper-parameters for GPT-3 are shown Table 13.
G.2 Fine-tuning Hyper-parameters
XLNet: Yang et al. (2019) propose a generalized autoregressive pretraining method that enables learning bidirectional
We fine-tune RoBERTa and RoBERT-GCN 4 NVIDIA 3090 GPUs with FP16. Model hyper-parameterrs are tuned on the validation set. where learning rate (2e-5, 3e-5, 4e-5), batch size {16, 32, 32), a dropout rate of 0.3, a weight decay of 0.01, a warmup proportion of 0.01.
contexts.
"8 and R52 are from ht https:/ /ttps://ae /tww./c......oo..ooo.../eeoooe/w umb. edu/ - www.iiiiinnnnnnneiineeeeeeee datasets/

---

SST-2
AGNews
R52
MR
Average
Supervised Methods
RoBERTa-Larg RoBERTa-GCN
95.99
95.55
97.76
96.42
91.16
95.38
95.80
95.68
98.2
96.1
89.7
95.10
Zero-shof Setting
Vanilla
91.55
90.72
90.19
89.06
88.69
90.04
Zero-shot-CoT
92.11
91.25
90.48
91.24
89.37
90.89
CARP
94.41
93.18
93.29
92.69
90.03
92.72
Few-shot Setting
Random Sampler
Vanilla
91.36
11.48
90.60
90.68
89.15
90.65
Zero-shot-CoT
92.56
92.65
92.49
92.03
89.91
91.93
CARP
94.41
93.18
93.29
92.69
90.03
92.72
SimCSE KNN-Sampler
Vanilla
93.90
93,50
94,36
92.40
89.59
92.75
Zero-shot-CoT
94.21
94.28
95.07
92.98
90.27
93.36
CARP
95.99
95.53
95.31
93.84
90.64
94.26
FT KNN-Sampler
Vanilla
94.01
94.14 14
95.57
95.79
90,90
94.08
Zero-shot-CoT
95.48
94.89
95.59
95.89
90.17
94.40
CARP
96.62
95.97
98.13
96.12
91.86
95.74
Table 7: Accuracy performances different settings on test subsets (results are over 5 rums). GPT-3 denotes text text-davinci-003 In few-shot experiments, sample 16 annotated examples (k=16) per prompt. "MJ Vote" is short for majority vote. "WP Vote" denotes weighted probability vote.
Prompts
SST-2
R8
XST-2
Dallase
sentmmett
Tack
# Labell
Source
review
# Trainn
6,720
+De
R72
1.831
Test
Clues
AGNicw
96.000
24,000
7,600
w/o keywom&&phrasee
96.21
96.91
news
2,568
sentiment
reviews
w/o contextual info.
96.23
wlo semantic relations
96.30
97.38
Table 10: Benchmark Dataset
w/o tones
96.40
97.35
who reference
96.50
97.19
Datawi
Back
# Tabl
Source
# FTaain
#
#Suilied
3STE2
Table 8: Label words and results on the SST-2 dataset with different strategies.
topic
topic
4,541
R52
topic
52
news
5,905
627
1,027
MR
sentiisen
reviews
6,398
710
888
(5)
Fluency(pp))
Logie Faithal((%)
Table 11: Dataset Subsets
SST-2
96.18
95.29 3.29
3.89
88
95.34
94.55
Table Results for evaluating the quality generated reasoning explanation. We sample 500 (text, reason) instances for SST-2 and R8.
method. To exploit the effect of temperature т, 7 from 1.0. Experimental results are shown in Table G.3. We tokenize the response with Tokezzzenizer!! and then count the number of tokens.
G.3 The influence of hyper-parameters
SST-2 Accuracy
We investigate the effect of model hyper- parameters including temperature, frequency penalty. We conduct experiments with Instruct- GPT3 on the SST-2 dataset.
0.0
96.39
0.2
96.48
0.4
96.40
0.6
96.59
0.8
96.68
1.0
96.70
Temperature The temperature controls the generated text variety when another hyper- parameter top,=1. More higher r, more variety is introduced. When т is close to O, the model generates the same result with the greedy decoding
H Prompts
Phttps://platffrre .ttp:///pppat.fomo.peee.... wtt.aa/e..cmmmeeiiiiii.m.

---

INPUT & GOLD LABEL
CLUES
REASONING
johnnic fui ure sure to lind entusiastic
diagnosiic reasoming prucess supporting sentiment determination the input that the phrase 'johnnic to and wai ka fai are sure find an euthusiastic sudience suggests
adventure.
audience among american action
Negative Clucs: sarrow, crossover
abventure buils, but the film
umerican action adventure bulls" received by this specific demographic. However, the subsequent phrase "hot the film's interests he too narmw to implies that film unlikely appeal broader aadienes, siggesting a negarive sentiment. Therefore, the overall sentiment of the input negative.
may attract crossover viewers GOLD LARKL: Negative
naroww
likely
moore sell globe unit muore resources inc sad agreed
INPUT:
Acquisition (agreed, sell, unit, sale, close, board upproval)
The input mentions acquisition agreement between Moore Resources Inc and Globe Inc, and the sale expected to close mid-April, suggesting Acquisitions topic. Minerals Co, silcoo and metals products which ure used chemical, aluminum, iron, rabber, and semiconductor industries, and Router ageney, all of which support Acquistions topic.
co undisclosed cypeus sale expected close mid april subject certain cooditions including hoard approval companies plahe produces metals products used chemical alominum imn nahher semicondactor industries renter Acquisitons
peinciple glohe
Companies (Moore Resources Inc,
Cypnas Minerals
input
Cyprus
Products isilicon, metals, chemical, aluminum, iro, nubber, semicondcterr Reuter agency)
Table 12: Examples of generated clues and reasoning for demonstrations.
Parameter
Value
Engine Name
text-davinci-003
Max Tokens
200
Temperature
0.7
Top
Frequency Penalty
Presence Penalty
0.0
Best Of
Table 13: OpenAl API Hyper-parameters.


Reviews:
[{"review_id": "2oxRdK6IC3", "reviewer": "Reviewer_uxYB", "summary": "", "questions": "1. The experiment chose only one large model, which is text-devinci-003. Considering that ChatGPT has been released for eight months, why wasn't it considered in the experiment? \n\n2. What do you think is the essential difference between your method and CoT?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback. We will answer your question in order below.\n\n**Q1**: The experiment chose only one large model, which is text-devinci-003. Considering that ChatGPT has been released for eight months, why wasn't it considered in the experiment?  \n**A1**: ChatGPT's API was open-access on March 2, 2023, just two months before the EMNLP submission deadline (June 16). At that time, we had already completed all the experiments reported in the paper. Given the constrained timeframe, it was not feasible to reconduct all experiments using ChatGPT. However, we did undertake experiments with ChatGPT on five text classification datasets and plan to report the results in the next version. To answer Q1, we show experimental results (16-shot) of ChatGPT on SST-2 and R8 as follows:\n\n| Dataset             | SST-2 | R8 | \n| ------------------- | :---: | :----: | \n| RoBERTa-Large      | 95.99 | 97.76  |\n| Few-shot (FT-Retriever, k=16) |   |   | \n| Vanilla     | 94.6  | 94.0 | \n| CoT     | 95.7 | 94.8 | \n| CARP    | **96.9** | **95.8** |\n\n**Q2**: What do you think is the essential difference between your method and CoT?  \n**A2**: We would like to clarify the difference between CoT (\"Let’s think step-by-step.”) and the proposed CARP. The primary distinction between the proposed CARP and CoT lies in the problem domain and methodology. CoT emphasizes a step-by-step approach, typically for mathematical problems. In contrast, CARP is designed specifically for text classification, addressing the complexities of linguistic elements such as clause composition (including concession, negation, and intensification), irony, and more. Thus, CARP is better suited for comprehensively handling the nuanced linguistic challenges in text classification.\n\n**Q3**: There is insufficient analysis and experimental support for the two reasons given. At the same time, there is no obvious answer to whether the strong enough reasoning capability proposed by the author is needed in these selected text classification datasets.   \n**A3**: We conducted a quality analysis of the generated reasoning. Due to the page limit, we present the analysis in the appendix. Please refer to Appendix C.2 (line 979- line 1042). We will move this section to the main content in the next version. \n\n**Q4**: The method proposed by the author has limited improvement in performance.  \n**A4**: Performance improvements are remarkable. As mentioned in Section 5, CARP yields new state-of-the-art (SOTA) performances on 4 out of 5 widely-used text-classification benchmarks.\n\n**Q5**: The author mentions that large models perform poorly due to inadequate reasoning capability and performance constraints caused by window length limitations. However, the experimental results show that large models perform well (with an average accuracy of 90% under zero-shot), which makes it confusing.  \n**A5**: Large language models perform poorly on the text classification task. We will use SST-2 as an example to illustrate. A Bi-LSTM neural network with random initialization achieves 87.5 accuracy [1] on SST-2, while text-davinci-003 achieves 91.5. Considering the number of parameters of LLMs and data scales used during training LLMs, the improvement of +3.9 under zero-shot is limited. \n\n[1] Socher et al., Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. EMNLP 2013.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The writing is good and the method is easy to follow.", "reasons_to_reject": "1. The author mentions that large models perform poorly due to inadequate reasoning capability and performance constraints caused by window length limitations. However, the experimental results show that large models perform well (with an average accuracy of 90% under zero-shot) which makes it confusing.\n2. There is insufficient analysis and experimental support for the two reasons given. At the same time, there is no obvious answer to whether the strong enough reasoning capability proposed by the author is needed in these selected text classification datasets.\n3. There is a lack of novelty, and clues reasoning can be considered a natural extension of cot. Knn sampling is also something that has already been proposed in existing work.\n4. The method proposed by the author has limited improvement in performance.", "questions_to_authors": "1. The experiment chose only one large model, which is text-devinci-003. Considering that ChatGPT has been released for eight months, why wasn't it considered in the experiment? \n\n2. What do you think is the essential difference between your method and CoT?", "paper_topic_and_main_contributions": "This paper discusses tasks related to text classification through large models. The author first points out two reasons why large language models (LLMs) perform poorly in text classification tasks: a lack of strong reasoning capability and the performance limitations caused by window length restrictions. Subsequently, the author proposes the Clue And Reasoning Prompting (CARP) method. This method instructs large models on how to use superficial clues to enhance their reasoning ability during demonstrations. The most effective demonstrations are retrieved using kNN demonstration search. The author conducted experiments on five text classification datasets using text-devinci-003, and the results showed that this method performed well.\n\nContributions：\n1. The author outlined the reasons why large models underperform in text classification, and introduced CARP to enhance their performance.\n2. Experiments were conducted on five commonly used text classification datasets.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5grt4Tp9Z6", "reviewer": "Reviewer_9Y4M", "summary": "", "questions": "Did you evaluate this method on some open-source LLMs, like Llama?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank you for the very encouraging comments. \n\n**Q1**: Regarding the missing reference and the \"annotation-free\" claim. \n\n**A1**: We will add the missing reference and fix the \"annotation-free claim\" in the next version. \n\n**Q2**: Did you evaluate this method on some open-source LLMs, like Llama?\n\n**A2**: Thank you for asking. It is a great idea to try CARP on open-source LLMs. We will run the experiments and update the results in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is overall written very well and is easy to follow.\nExperiments are thorough and complete. Using multiple methods, multiple tasks and ablation studies lead the generalizability of the paper's claims.", "reasons_to_reject": "The authors should not claim their framework as \"annotation-free\" while using a kNN trained on supervised dataset.", "questions_to_authors": "Did you evaluate this method on some open-source LLMs, like Llama?", "paper_topic_and_main_contributions": "This paper proposes a framework to use LLMs for text classification task. It first guide the LLMs to extract the key clues from the input and use the clues to get the final decision. To improve the few-shot performance, authors also propose to train kNN to generate the examples. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Extract feature and feed into classifier:\nZheng Tang, Mihai Surdeanu; It Takes Two Flints to Make a Fire: Multitask Learning of Neural Relation and Explanation Classifiers. Computational Linguistics 2023; 49 (1): 117–156. doi: https://doi.org/10.1162/coli_a_00463"}}, {"review_id": "SAJc5m2bYe", "reviewer": "Reviewer_jdhr", "summary": "", "questions": "A. The ablation study shows that it works even without the gold label in the example.  How does the examples help without gold labels?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your very encouraging and insightful comments. \n\n**Q1**: The ablation study shows that it works even without the gold label in the example. How does the examples help without gold labels?\n\n**A1**: Thank you for asking. We use SST-2 as an example to illustrate.\nAs mentioned in Section 4, each demonstration in CARP consists of four parts: text, CLUE, reasoning, and gold label. In Section 6.2, after removing the gold label, the CLUE and reasoning in the demonstration still provide evidence that helps to determine overall sentiment. Therefore, they can still help GPT to determine the overall sentiment of the test input. An example of CARP (1-shot) without gold label is as follows:\n\n\n\n> **<CARP-Task-Description>**     \n**INPUT**: noyce films it more as a shocking history lesson than as drama.  \n**CLUES**: - Positive: \"more,\" \"shocking,\" \"history lesson,\" \"drama.\" - Negative: None.  \n**REASONING**: The input contains positive words such as \"more,\" \"shocking,\" and \"drama,\" which suggest that the sentiment is **positive**. Additionally, the phrase \"history lesson\" implies that the film is educational and informative, which is generally seen as a **positive** thing.     \n  **INPUT**: press the delete key.\n\n**Q2**: Regarding “Typos Grammar Style And Presentation Improvements”\n\n**A2**: Thank you for kind reminder. We will correct typos and enlarge experimental tables in the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Good design of prompt including examples.  The difference from the Chain-of-Thought method is clear.\n* The experimental results show the SotA scores in 4 out of 5 tasks, and they outperformed the supervised fine-tuned results.\n* The proposed method addresses the low-resource settings and domain adaptability.\n* The discussion of alternative labels is interesting.\n", "reasons_to_reject": "* The notion of in-context learning has been already proposed.\n* The document-level classification task is too popular and simple and its application is limited.", "questions_to_authors": "A. The ablation study shows that it works even without the gold label in the example.  How does the examples help without gold labels?\n", "paper_topic_and_main_contributions": "This paper proposes an approach to document-level sentiment classification task based on LLMs and in-context learning, using prompting with examples which enables reasoning to get correct labels.\nThe prompt asks the model to generate clue keywords related to positive or negative sentiments, and reasoning process that support the decision of labels, in addition to the final answer.  Also examples of (input, clues, reasoning and label) are added to the prompt for one-short and few-shot settings which can be automatically generated from the subset of training data.\nThe experimental results show the SotA scores in 4 out of 5 tasks, and they outperformed the supervised fine-tuned results.  Also this paper provides extensive discussion including ablation studies, alternative labels and consideration of low-resource scenario.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Figure 2 should be placed on the top of the column.\n* Numbers in Table 1 and 2 are too small.  Since they are key results they should be displayed with larger letters.\n* CRAP->CARP in Table 2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores text classification in GPT-3, and introduces a method (CARP) to help models use superficial clues for better classification performance. The paper reports new SOTA performances in 4 of 5 chosen text-classification benchmarks.

The reviewers agree that the paper is clear and well-written, generally find the methods and experiments solid, and appreciate the demonstrated performance improvement.

The reviewers also raise concerns about the strength of the motivations and impact of the paper, pointing out that baseline model performance is already high on these tasks, and the improvements are fairly incremental. The reviewers would also prefer to see stronger support for the claims made about the reasons for poor model performance in these domains, (alongside stronger motivation and justification for the claim that this performance needs improvement in the first place).
Decision: Accept-Findings