Task-Level Thinking Steps Help Large Language Models for Challenging Classification Task
Authors: chunhui du
Jidong Tian
Haoran Liao
Jindou Chen
Hao HE
Yaohui Jin
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Prompt Engineering
Text Classification
Chain of Thought

Abstract: Large language models (LLMs) have shown incredible performance on many tasks such as dialogue generation, commonsense reasoning and question answering. In-context learning (ICL) is an important paradigm for adapting LLMs to the downstream tasks by prompting few demonstrations. However, the distribution of demonstrations can severely affect the performance, especially for challenging classification tasks. In this paper, we propose the concept of task-level thinking steps that can eliminate bias introduced by demonstrations. Further, to help LLMs distinguish confusing classes, we design a progressive revision framework, which can improve the thinking steps by correcting hard demonstrations. Experimental results prove the superiority of our proposed method, achieving best performance on three kinds of challenging classification tasks in the zero-shot and few-shot settings. Besides, with task-level thinking steps, automatically generated chain-of-thoughts (CoTs) bring more competitive performance.

Automatic Task-Level Thinking Steps Help Large Language Models for Challenging Classification Task
Chunhui Du, Jidong Tian, Haoran Liao, Jindou Chen, Hao He, Yaohui Jin MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University {chunhui18, frank92, liaohaoran, goldenbean, hehao, jinyh]@sjtu edu. cn
Abstract
Revis Framewors
Hard Demondrations
Large language models (LLMs) have shown incredible performance on many tasks such as dialogue generation, commonsense reasoning and question answering. In-context learning (ICL) is an important paradigm for adapting LLMs to the downstream tasks by prompting few demonstrations. However, the distribution demonstrations can severely affect the per- formance, especially for challenging classifi- cation tasks. In this paper, we propose the concept task-level thinking steps that can eliminate bias introduced demonstrations. Further, to help L.MSs distinguish confusing classes, we design a progressive revision frame- work, which can improve the thinking steps by correcting hard demonstrations. Experimental results prove the superiority of our proposed method, achieving best performance on three kinds of challenging classification tasks zero-shot and few-shot settings. Besides, with task-level thinking steps, automatically gener- ated thain-of-thoughee (CoTs) bring more com- petitive performance.
Task Descrption
Description (Damenddndurr))
Inpuls
imputa
Agants
Let's think stop by stap
Lat' think by stop.
LLM
LLM
Automatic: Task-Lene evel Thinding Stops
Chiputte
Cuigust
(a) Zero shot-hhhtooo CoT Prompt (b) Prompt with Automatic Level Thinking Stops
Figure I: Zero-shot-CoT Prompt (Kojima et al., 2022) vs. Prompt with Automatic Task-Level Thinking Steps
"Let's first understand the problem, devise a plan, and solve the problem step by step." to address the issue of missing reasoning steps and "extract relevant variables and their corresponding numer- als" to address the issue of calculation errors. In- spired by this, we propose the concept of task-level thinking steps for challenging classification tasks as shown in Figure 1(b), which helps LLMs clar- ify tasks and remind them of overlooked issues. To this end, we design progressive revision frame- work, which automatically generates thinking steps by LLM agents and then progressively revise the steps by correcting hard demonstrations.
1 Introduction
Large language models (LLMs) have shown in- credible performance on many tasks such di- alogue generation, commonsense reasoning, and question answering (Dong et al., 2022). With the new paradigm of in-context learning (ICL) (Brown al., 2020; Pan al., 2023), LLMs directly adapt to downstream tasks without updating parameters by prompting with few demonstrations. Wei et al. (2022) found that elaborating the reasoning steps in demonstrations can significantly stimulate the com- plex reasoning ability of LLMs, which is called manual saan-of-thoughh (CoT). Zero-shot-CoT" (Kojima al., 2022) enables LLMs generate thinking steps automatically using the simple but efficient prompt "Let's think step by step.", as shown in Figure ((a). More recently, plan-and- solve prompt (Wang et al., 2023) extends with
For classification tasks, it is widely investigated that the demonstrations distribution, including con- tent, label, and even the order, can severely affect the performance (Lu et al., 2022a; Turpin et al.. 2023; Zhao et al., 2021). Therefore, many works fo- cused on selecting high-quality and representative demonstrations, such as selecting similar demon- strations (Liu et al., 2021) or maintaining the class balance between demonstrations (Lu et al., 2022b). However, the best distribution depends on the task, and bias is inevitable. Our proposed task-level thinking steps are helpful in reducing the bias in- troduced by demonstrations. Table shows an ex- ample of the stance detection task. Zero-shot-CoT

---

Task description:
Perform stance detection. classes: (A)none (B)against (c)favor Input: For domain abortion, You can't an't give the "death penalty" to something not living sentient or autonomous.
hard demonstrations, In the framework, as shown in Figure 2, we set up two LLM agents (Andreas, 2022) with character settings and historical memo- ries, the teacher agent and the student agent. For each hard demonstration, the student agent gener- ates outputs based on the task-level thinking steps generated by the teacher agent. If the prediction is correct, move to the next demonstration. If the prediction is wrong, the teacher agent first analyses the error cause and then tries to revise the task- level thinking steps. For the example in Figure 2, class (B) and class (D) tend to be confusing, After the teacher agent's analysis, an explicit reminder can be inserted in a suitable place in the revised task-level thinking steps. Then, the student agent generates outputs again. The iteration may be re- peated several times until the correct prediction the maximum number of times allowed reached. In this paper, we consider three classification tasks: multifaceted analysis of subjective text re- quires first locating aspects, then identifying opin- ions, and finally analyzing polarity; fine-grained text classification requires a deep distinction be- tween numerous and similar classes; domain- specific classification requires the understanding of domain-related questions, choices, and demon- strations. We conduct experiments on zero-shot, few-shot, and few-shot-CoT settings, specifically. Experimental results demonstrate the effectiveness of our proposed method by outperforning the com- petitive baseline models. Further ablation analyses and robustness experiments reveal the benefits and effects of each module.
No Demonstrations
Output of zero-shot-CoT prompt
The stance in this context is (C) favor, as the statement suggests that the speaker is in favor of abortion rights. The argument is that fetus is not a living, sentient, or autonomous being and cannot be subject to the death penalty. / Biased Demonstrations: in.ut:[...] answer:(B) in.ut:[...J answer:(B) input:[...J aB)
Output of zero-shot-CoT prompt
First, the statement is discussing abortion. Next, the statement is arguing against the idea of giv- ing the "death penalty" something that is living, sentient, or autonomous. This suggests a stance against abortion. X
Output with Lask-Level thinking steps 1. Identify the domain abortion discussed the statement. 2. Look for keywords or phrases the speaker is against the idea of considering a fetus living, sentient, or autonomous being. 3. Determine the stance of the statement. The statement is against the idea of considering fetus as a living, sentient, or autonomous heing, which suggests that the speaker is in favor of abortion. 4. Provide the answer to the stance detection task which is (C) in favor of abortion. v
Table 1: Tero-shot-Coo CoT prompt is affected by the biased demonstrations while prompt with task-level thinking steps outputs correctly. Omitted demonstrations are shown in Appendix A.
Our contributions are threefold: 1) We pro- pose the novel concept of task-level thinking steps. which can climinate bias introduced by demonstra- tions 2) We design a progressive revision frame- work, which purposefully improves the thinking steps based on feedback from hard demonstrations for LLMs. 3) Our proposed method surpasses com- petitive baselines in the zero-shot, few-shot, and few-shot-CoT settings.
predicts correctly with no demonstrations. How- ever, the output is severely affected by "against" from the biased demonstrations. The automatically generated task-level thinking steps include: iden- tifying the domain; finding the viewpoint; deter- mining the stance with respect to the domain; and matching the choices. With the help of these steps, LLMs perceive the implicit meaning "a fetus is not a living, sentient, or autonomous being" and make the correct prediction "in favor of abortion".
2 Methods
2.1 Task Definition
We consider general classification task, where T denotes the task description, text input, and y E is the corresponding la- bel. [ = is the number of classes. Dtrain { y1), ..., (IM. y denotes the train dataset and Diest = ((11.))) (æ yN)) denotes the
X is the
In addition debiasing, we would like task-level thinking steps to clarify some confusing classes. To this end, we design a progressive revision framee- work to generate the thinking steps by correcting

---

Initial
Initial task-level thinking steps:
What's the relatinn between <81>
1. Idenlily the two medical terms problems mentioned in the context, marked <e1> and <e2>.
<e2>? Classes:
(B) <e1> worsens <e2>
2. Determine the relationship between <et> and <p2> based the context.
(D) 1> is administered for <e2> demonstrations:
3. Choose the answer choice that best describes the relationship between <e1> and <e2> from the given options.
[20, 30 ... (#c-1, 9c-11
Please generate thinking steps the task.
Hard Demonstration
Therefore, the answer (D)
X
Student
Teacher
#yy continued
Please analyze the outputs from student following your thinking for
low amounts of urine <le2>
Revised thinking steps:
throughout admission, despile
Steps
<e1> aggressive diuresls <la1> git
ineffective
worsens the <e2> medical problem,
consider class
(D)
Please revise the thinking steps lo clarify.
think following
thinking staps.
Therefore, the (B)
Hard Demonstration
Figure 2: Progressive revision framework. There are two LLM agents: teacher and student, Rectangles represent the prompts agents, outputs the teacher agent and outputs of the student agent.
test dataset. An LLM with parameters performs zero-shot learning by conditioning on the task de- scription T and the test input Diest- The likeli- hood of candidate answer y could be represented by scoring function
observations. We set up two LLM agents. The teacher agent is responsible for generating the task- level thinking steps from hard demonstrations with labels. In addition, the teacher agent needs lyze the errors in the student agent's answers and revise the task-level thinking steps. The student agent is responsible for giving answers and expla- nations each presentation with the task-level thinking steps provided by the teacher agent.
P(u[z) fo(
When sampling demonstrations Deemo from Dtrain the likelihood for few-shot learning is:
2.3 Progressive Revision Framework
x) = (T, Ddemo: x))
Given hard demonstrations Deemo the teacher agent first generates initial thinking steps So. then progressively revise it based on the outputs from the student agent shown in Figure 2.
The final predicted label is the candidate an- swer with the highest probability:
i = arg max P(y (tewt) yey
(3)
Generate initial task-level thinking steps. The teacher agent uses hard demonstrations gener- ate initial task-level thinking steps. The prompt is Pleease generate generic thinking steps for the task." Taking the medical relation extraction task as an example, here are the generated initial task- level thinking steps 1. Identify the two medi- cal terms problems mentioned in the context, marked as <el> and <e2>. 2. Determine the lationship between <el> and <e2> based on the context. 3. Choose the answer choice that best de- scribes the relationship between <el> and <e2> from the given options.".
Only hard demonstrations can generate valuable task-level thinking steps. In this paper, we tra- verse Dtrain and select one that the LLMs can not predict correctly for each class to construct Daemo { Bo), --, (#C-1, as shown in Figure 2.
2.2 LLM Agents
LLMs can serve as models of agents as suggested (Andreas, 2022), which output responses based on initial setting, historical memories, and current

---

We can see that the teacher agent understands the task description and gives three reasonable steps: identifying two entities, determining the relation, and matching the classes. However, they are also preliminary and abbreviated.
2.4 Automatic Ohain-of-Thhughhe
Another advantage of our proposed task-level think- ing steps is that is very helpful to generate reliable CoTs for demonstrations automatically. Manual CoTs are labor intensive, and the human annota- tions do necessarily match LLMs. Zero-shot- CoT, while avoiding human engineering, may pro- duce unfaithful explanations especially for hard demonstrations.
Generate outputs. For the i-th hard demonstration Tú, the student agent receives the task-level think- ing steps S; and generates outputs Rg The prompt is "Thinking steps: (S;). Ler's think following the thinking steps." " As shown in Figure 2, the correct answer is "(B) <el> worsens <e2>" but the stu- dent agent outputs "The answer (D)" and sends it to the teacher agent.
3
Experiments
3.1 Datsets
Improve task-Level thinking steps. The teacher agent analyzes the reason for the error with the prompt "Iput: (x,). Outputs: (R.). Please an- alyze the outputs from the student following your thinking steps.". Then the teacher agent revises the task-level thinking steps with the prompt "Please revise the thinking steps to clarify.".
We choose three kinds of challenging classification tasks. The statistics and descriptions of all tasks are shown in Table 7 and Table in Appendix B. Multifaceted analysis of subjective text is the task that involves different aspects of subjective human feeling reflected in the text. The challenge is recognizing and understanding broader range of human emotional states rather than identifying common positive or negative feelings. Following Zhang et al. (2023a), we use Stance 16 (Mohammad et al., 2016) for stance detection and Emotion20 (Barbicri et al., 2020) for emotion recogmition.
We find that the revised thinking steps do not usu- ally change much but rather extend more reminders. The main extension in this example is If the treat- ment is ineffective or worsens the medical problem. consider choice (B) rather than choice (D).". It can be seen that the student agent only focuses on the causal relationship between the treatment and the medical problem while does not pay attention to the effects. fact, this input places more emphasis on the effect of the treatment on the discase than on how the treatment should be taken for the disease. Generate outputs with revised thinking steps. With the revised thinking steps S,, the student agent predicts correctly. Then, the iteration proceeds to the next hard demonstration. Otherwise, the out- put is sent to the teacher again until the correct prediction or the maximum attempts are reached, Experiments show that even if the hard demon- stration was not successfully corrected, the revised thinking step was still useful.
Fine-grained text classification is the task of cate- gorizing text or documents into predefined topics or categories. We carefully select 20news dataset. The challenge lies in numerous and similar classes. Domain-specific classification adopts the medical relation classification task, which refers to identify- ing the relationship between pairs of entities within medical texts. The challenge is the need for domain knowledge, and differences between relations are subtle. i262 ( (Uzuner al., 2011) collects clinical records and used classify relations between medical problems, medical tests, and treatments. ChemProt (Krallinger et al., 2017) is used to de- termine how chemical acts on the protein, such as activation and inhibition. DDI (Herrero-Zazz et al., 2013) is used to determine the interaction relation between two drugs, such pharmacokinetics and pharmacodynamics.
Checking mechanism. We find that LLMs, espe- cially gpt-4, are effective in correcting hard sam- ples by revising task-level thinking steps. However, this may be achieved by extending shortcuts rather than trustworthy analysis. For example, "If you are not sure, choose (B).". Therefore, we set up checking mechanism. The teacher agent tries generate multiple candidate thinking steps that can successfully correct x;- The student agent tests performance on { I and keeps the best one.
3.2 Experimental Settings
We constructed experiments to evaluate the effec- tiveness of thinking steps. C' hard demonstrations sampled for each task to generate thinking steps. The LLM for the teacher agent is gpt-4, the LLM for the student agent and the LLM for running
httt:://qwone.com//jaaaett http://www...mmmiiaaa///////e//e

---

Methods
1tance
Emotion20
20news
i2b2
ChemProt
DDI
Avg.
Default
41.49
70.02
45.99
14.89
41.97
49.17
43.92
Auto
47.62
72.81
39.97
29.63
54.47
36.22
46.78
Zero-shot
ITS
41.66
68.96
63.30
43.82
50.91
39.57
51.37
PRTS
52.38
75.76
67.20
54.99
68.33
57.36
62.67
Default
50.04
71.58
56.29
35.54
43.03
46.13
50.43
Auto
51.09
75.28
41.76
42.62
65.73
49.29
54.30
Few-shot
ITS
36.06
74.01
53.56
40.48
49.01
50.15
50.54
PRTS
56.35
77.82
66.58
52.30
69.67
62.45
64.19
Auto
37.17
72.72
42.02
40.94
67.40
50.73
51.83
Few-shot-CoT
ITS
40.39
75.21
58.71
60.50
76.72
63.23
62.46
PRTS
60.30
78.92
72.35
67.45
79.20
76.67
72.47
Table 2: The overall performance PRTS and baselines on six classification tasks. Bold denotes the best corresponding settings specifically.
experiments are both gpt-3.5-turbo. We set a tem- perature of 0 for all experiments. For each hard demonstration, the maximum number of revision attempts is 5. For the checking mechanism, the number of candidates is 3. We report the Fl-score the evaluation metric.
both 20news and i2b2 have many similar classes, and the thinking steps include analysis and com- parison between classes, which may help in the fine-grained classification of LLMs. With progres- sive revision, PRTS performs best on all tasks.
For the few-shot setting, PRTS outperforms Auto by an average of 9.89%. is worth noting that ITS has worse performance than Auto. One pos- sible reason is that initial thinking steps may be contradictory ICL for LLMs. However, we will show that significant improvements can be obtained by explaining these demonstrations with thinking steps.
The followings methods serve as baselines: De- fault generates outputs directly and Auto gener- ates outputs with the prompt "Let's think step by step" (Kojima et al.. 2022). Our methods gener- ate outputs with Initial Thinking Steps (ITS) to validate the effectiveness of thinking steps and Pro- gressively Revised Thinking Steps (PRTS) to vali- date the effectiveness of framework. For few-shot, prompt demonstrations are the same hard demon- strations to avoid unfair comparisons. In Section 4.2, we construct experiments with other prompt demonstrations. For few-shot-CoT, Auto, ITS and PRTS can automatically explain the prompt demon- strations without human efforts following (Wang et al., 2023; Shum et al., 2023), Detailed prompt templates are shown in Table 9 in Appendix B.
For few-shot-CoT,, PRTS outperforms Auto by 20.64% and ITS by 10.01% on average. For Auto, 1tance and 20news have worse performance compared with the few-shot setting. One possible reason is that LLMs may produce reluctant expla- nations and hallucinations for hard demonstrations. Table 4 shows the automatically generated CoT by Auto, ITS, and PRTS: For Auto, it can be seen that the LLM can not understand the stance detection task. Therefore, the LLM fails to locate the domain word "atheism" and mistakenly believes the answer is favor". However, faced with the golden label "(B) against", the LLM can only generate the reluctant and absurd explanation: "against (B) is the closest to the opposite of favor". With this wrong CoT demonstration, a worse performance than zero-shot and few-shot would make sense. For ITS. LLM successfully locates the domain word "atheism" and seems explain the golden label correctly. However, L.LM only states that "against is the idea of freedom from religion" but does not explicitly state that atheism is equivalent to free-
4 Results
4.1
Main Results
The experimental results are shown Table 2. All results are averaged over three runs. Overall, PRTS performs best on all tasks, which outperforms Auto by 18.75% for zero-shot, 9.89% for few-shot, and 20.64% for few-shot-CoTT on average.
For the zero-shot setting, IT'S has slightly better performance than Default and Auto average, but which one is better depends on the task. Specifi- cally, ITS outperforms Auto by 23.33% for 20news and by 14.19% for i2b2. One possible reason is that

---

Methods
1
20news
i2b2
redking Cars
Affected cboses
Similar 51.82
36.63
47.86
Stance16
262
Similar+PRTS
53.97
59.20
52.91
30-
40-
Unbalanced
41.78
21.98
12.93
30-
PRRS
52.38
65.09
42.36
20-
Easy 53.09
51.20
27.90
Easy+PRTS 51.79
67.33
52.98
20
(8)
040
(5)
Table 3: PRTS consistently improves performance for various sampling methods of prompt demonstrations.
chosse
cboses
Figure 3: The number of corrected samples when cor- recting hard demonstrations each class.
dom from religion. This may lead LLMs to ignore domain words and only decide on the sentence. For PRTS, the CoT is more consistent and com- plete and always revolves around the domain word "atheism". COT explicitly explains that "a stance against atheism" is because "people should not be free from religion".
PM'S mr dhoce mastarssm
- PRTS wel: shecking mediensm
Stance16
1262
52
54-
50-
52-
48
48.
45
4.2 Ablation Studies
Irdel
181
IntUtAi 3)
(b)
Analysis demonstrations distribution. In this section, we explore the effects of various sampling methods of prompt demonstrations. For the few-shot setting, Similar (Liu et al., 2021) dynamically selects C' most similar for each sam- ple. Unbalanced selects prompt demonstrations in the same class, which is considered biased (Zhao et al., 2021). Easy is the same our hard demonstration selection, which selects one that LLMs predict correctly for each class. Results are reported in Table for 1tancel6, 20news, and 12b2. We can find that which sampling method is better depends on the task, but PRTS consistently improves by 9.93% for Similar, 27.71% for Unbalanced, and 13.3% for Easy, respectively. Especially compared with Unbalanced performing poorly on all tasks, Unbalanced+PRTS remains very robust. This shows that PRTS can elim- inate bias introduced by demonstrations effectively.
Figure 4: Performance changes compared PRTS checking mechanism with PRTS w/o checking mecha- nism in progressive revising thinking steps.
3. The class of the hard demonstration being corrected is named revising class while the remain- ing classes {1,..., affected classes. It is always helpful for revising class and may also have positive effect affected classes.
1, + 1,... C} are named
In addition, we further improve thinking steps through the checking mechanism when iterating, i.e., progressive revising thinking steps. As shown in Figure 4, for PRTS w/o checking mechanism. huge performance degradation occurs for class (B) in Stance 6 and class (D) in i2b2. For compar- ison, PRTS w/ checking mechanism almost im- proves progressively in each round and outper- forms the PRTS w/o checking mechanism 5.29% for Stancel6 and 3.07% for i2b2.
Analysis of the progressive revision framework. Our key idea is revising the task-level thinking steps by correcting the hard demonstrations, thus enhancing the ability to distinguish confusing classes. Specifically, we first correct the hard demonstrations and then further improve thinking steps through the Checking mechanism.
4.3 Robustness Experiments
Effects of class names. For classification tasks, class names can seriously affect the performance of LLMs. For 20news and DDI, the default classes are ambitious, and we have manually annotated them as manual classes. In this section, explore the effeets of different class names. More details are shown in Table 8 in Appendix B.
While correcting the hard demonstrations of class c, is also beneficial to all samples. We validate this idea on two datasets for zero-shot setting: Stancele 16 and i2h2 as shown in Figure
We experimented on zero-shot, few-shot, and

---

Input:
For domain atheism, The framers of our Constitution meant we were have freedom of religion, not freedom from religion. Billy Graham quote SemST
classes: (A) none (B) against (C) favor answer: (B)
Auto
The statement suggests a pro-religion stance, implying that the person making the statement is in favor of religion. Therefore, the stance detection task "favor" (C). However, the prompt specifically asks for stance detection from the given choices, and "favor" is not one of the choices. Among the given choices, "against" (B) is the closest to the opposite of "favor"
ITS
1. The context is about atheism and a quote by Billy Graham 2. 3... 4. The stance expressed is against the idea of freedom from religion. 5. Therefore, the appropriate answer is (B).
PRTS
1. Read and understand the context: ... 2. Identify the domain: The domain is atheism. 3. Determine the stance: ... ...This implies stance against atheism, as it suggests that people should not be free from religion. 4. Consider the broader implications.. 5. Choose the choice (B)
Table Case study for automatically generated COT. Auto CoT reluctant and absurd. ITS CoT does not explicitly state the relation between stance "atheism" and "freedom from religion". PRTS CoT is complete and always revolves around the domain word "atheism".
"int is an abbreviation for interaction except pharmacokinetics and pharmacodynamics"
70
Effects of LLMs. We find that only gpt-4 helps the teacher agent to revise thinking steps progressively. text-davinci-o is not a dialogue model and can not even produce reasonable initial thinking steps. gpt-3.5-turbo enables interaction between the stu- dent and the teacher, but it is difficult to generate valuable thinking steps correct the hard demon- strations. For the hard demonstrators covered in this paper, gpt-4 can almost revise thinking steps to correct them within five attempts.
63-
See44ti-CoT awr0-2tt
See-4201
Figure 5: Performance on zero-shot, few-shot and few- Chot-CoT settings for Auto and PRTS. The dashed lines indicate default classes and the solid lines indicate man- ual classes.
Therefore, we keep the previous obtained think- ing steps and rerun the experiment on text-davinci- 003 and gpt-4 as shown in Table 5. Overall, the classification performance of text-davinci-003, gpt- 3.5-turbo, and gpt-4 improves sequentially for both Auto and PRTS. Interestingly, the perfor- mance improvements of the fine-grained classi- fication task, 20news (39.97% -> 70.03%) and i2b2 (29.63% + 58.33%). are huge from gp13,5- turbo to gpr-4 for Auto. It is not clear whether this is because gpt-4 was pretrained on these datasets or its instruction tuning is similar to thinking steps. Besides, all three LLMs perform poorly on Stance 16 for Auto, while PRTS can get huge boost on gpt-4. This suggests that PRTS can accu- rately analyze and disambiguate the task for ambi- tious tasks, then provide explicit thinking steps.
few-shot-Co Th for Auto and PRTS. To our surprise, the default classes performed much better than the manual classes, especially for Auto on 20news, as shown in Figure 5(a). The reason may be that 20news is a pre-trained corpus for LLMs and is therefore more compatible with the default classes. Nevertheless, PRTS has similar performance with manual classes and default classes.
For DDI, the performance on the default classes is particularly worse than on manual classes for Auto as shown in Figure 5(b). We find that LLMs can correctly recognize and interpret the medical terms such as "pharmacokinetics" and "pharmacodynamies" with the strong capability on medical problems (Nori et al., 2023). The poor performance is because default class "int" was mistakenly thought to be a useless and distracting choice. However, PRTS achieves similar performance to manual classes because the level thinking steps explicitly state that
4.4 Cost Analysis
In this section, we analyze the additional cost of task-level thinking steps includes the generation cost based on hard demonstrations and prompt

---

LLMs
Stancel6
Emotion20
20news
i2b2
ChemProt
DDI
Avg.
Auto
44.44
65.29
32.45
12.81
40.69
25.20
36.81
lext-davinci-003
PRTS
45.35
70.91
57.55
54.64
39.94
41.96
51.72
Auto
47.91
75.28
39.97
29.63
54.47
36.22
47.25
8pt-3.5-turbo
PRTS
52.38
77.82
67.20
54.99
68.33
57.36
63.01
Auto
41.81
78.64
70.03
58.33
60.71
48.01
59.58
gpt-4
PRTS
66.76
80.20
78.33
66.59
68.33
76.54
72.79
Table 5: Zero-shot performance of PRTS and Auto for different LLMs.
methods
generation tokens
prompt tokens
general dataset
Default
T Tinput X Niest (Tinpat Teask) X Ntest (Ténput + taass)) x Ntest
Auto
PRTS
T Toppat H C' + 2R baas + 2mmppat) x C
i2b2 dataset
Default
T Timput X Niest
Auto
Timput X Niest
PRTS
168T 168Tinput
5T imput X Ntest
Table 6: Cost comparison for different methods.
cost for test inputs.
marised in the Table 6. Taking i2h2 dataset as an example, 3, C = 8, = 4, and Thask is approximately 2Tmpptt For practical applications, Ntest is much larger than 168, so generation to- kens can be ignored. The task-level thinking steps introduced in the inference phase may need to be improved in the future.
Generation cost includes the selection of hard demonstrations and iterative refinement. Assume that the average tokens of train/test inputs Tinput (outputs with simple answers can be ig- nored). It takes an average traversal of H train in- puts select hard demonstrations for each of the C classes. Thus, the selection cost is imput x x C. Assume that the average tokens of task-level think- ing steps is Trask. In each iteration of the refine- ment phase. the teacher agent needs to predict each hard demonstration with cost Trask imput based task-level thinking. There is degree of simplification, but roughly the input tokens are all Tinpatt + Tiask and the output tokens are all Trask- The student agent needs to update task- level thinking steps based on test input with cost Trask + ipput- There is a degree of simplification, but roughly the input tokens are all T Vipput + Tiask and the output tokens are all Trask. Then the refine- ment cost is 2R x (Tusk + mmpp) C, where R the number of iterations for each hard demon- stration.
Related Work
5.1 In-context Learning
It is widely investigated that severe bias exists in the output of LLMMs for different demonstrations (Lu et al., 2022a; Turpin et al., 2023). Zhao et al. (2021) proposed calibration method that fits the calibration parameters for uniform content-free in- put "N/A" is uniform across choices. More studies try to select specific demonstrations based on dif- ferent philosophies to reduce bias, such as similar- ity (Liu et al., 2021) selecting similar examples to facilitate analogical learning, high uncertainty to improve inference efficiency (Diao et al., 2023) se- lection. Min et al. (2022) showed that LLMs could largely ignore the mapping between inputs and la- bels, while Webson and Pavlick (2022) showed that L.MSs perform well on the NI. dataset with correct demonstrations even the instruction irrelevant or misleading.
In the inference phase, the cost of zero-shot prompt (Default) is T Timpat, zero-shot prompt with "Let's think step by step" (Auto) is imput + Trosk- and prompt with level thinking steps (PRTS) is T Tinput 2Ttask-
The total cost with Ntest test inputs sum-

---

5.2 Chain-of-Thought
lenging. Therefore, we proposed the concept of task-level thinking steps and verified the robustness for biased demonstrations. To further enhance the classification capabilities of LLMs, we designed a progressive revision framework, which purpose- fully improves the thinking steps based on feedback from hard demonstrations. Experimental results proved the superiority of our proposed method. Be- sides, with task-level thinking steps, we found that the automatically generated CoTs are more reason- able and effective.
Recent works have shown that LLMs can output better responses for ICL with high-quality expla- nation (Lampinen et al., 2022). Manual chain-of- thought (Wei et al., 2022) could prompt LLMs to generate step-by-step solutions, which leads sub- stantial improvements on many reasoning-intena tasks. However, the performance improvement de- pends on manual efforts. The zero-shot CoT (Ko- jima et al., 2022) enables LLMs to automatically generate explanations. For unlabelled demonstra- tions, auto-CoT (Zhang et al., 2023b) partitions demonstrations into a few clusters, selects rep- resentative demonstration from each cluster, and generates explanations using zero-shot COT. For labeled demonstrations, Shum et al. (2023) opti- mizes a set of latent variables select the most helpful and suitable demonstrations with correct ceff-generated answers. Except for the first-try CoT, Zheng et al. (2023) used previous outputs as hints to progressively guide toward the correct answers. Madaan et al. (2023) provided multi aulti-aspect feed- back on the previous output and refined into new output. Our framework is also iterative, but it works on the task level and is based on feedback from hard demonstrations.
Limitations
Task coverage. We have proposed three challeng- ing classification tasks: multifaceted analysis of subjective text, fine-grained text classification, and medical relation classification. Although there are many nontrivial findings, they are limited to traditional classification tasks. Future work would extend a wider range of classification tasks, such as natural language inference and multiple choice.
Experimental analysis and explanation. Al- though many ablation experiments, robustness experiments and case studies have been conducted, there are still some aspects that have not been discussed. As an example, we adopt default revising order for each task. While the checking mechanism reduces the negative impact of revi- sion, there may still be other unforeseen challenges.
5.3 Prompt Engineering
Prompt offers natural and intuitive interface for humans interact with language models. For parameter-accessibee models, gradient-based meth- ods are widely adopted (Shin et al., 2020; Qin and Eisner, 2021; Lester et al., 2021). In the era of LLMs, searching directly in the natural language hypothesis is more popular. Honovich et al. (2022) found LLMs can generate the task instruction with several demonstrations. Zhou al. (2022) auto- matically generates many instructions for the given demonstrations and selects the one with the max- imum score function. The concept of task-level thinking steps is also part of prompt engineer- ing, from simple "Let's think step by step" (Ko- jima et al., 2022) to well-designed plan-and-solve prompt (Wang et al., 2023).
Reliability. Although our progressive revision framework has been validated several chal- lenging datasets, LLMs remain sensitive small changes the thinking steps. Future work would improve the reliability of the thinking steps by refining hard demonstrations selection, thinking steps revision and checking mechanisms.
Unlabeled training sets. The absence of mas- sively labeled training sets is indeed a practical problem. However, we think there are still feasible solutions to obtain hard demonstrations. For ex- ample, measure uncertainty by running zero-shot prompts multiple times. If LLMSs have the same outputs multiple times for the same train input, it means that the LLMs have high confidence. Ei- ther completely correct or completely wrong due bias. On the contrary, if L.L.Ms have very different outputs for the same input, it means that LLMs hesitate between certain classes and this train input
6 Conclusion
Although LLMs have shown incredible perfor- mance on many tasks, we pointed out that many classification tasks, such as multifaceted analysis of subjective text, fine-grained classification, and domain-specifie classification tasks are still chal-

---

can be considered hard demonstration. Annotate them and continue to use the proposed method in this paper.
Martin Krallinger, Obdulia Rabal, Saber Akhondi. and et al. 2017. Overview of the biocreative vi chemical-protein interaction track. In Proceedings of the sixth BioCreative challenge evaluation workshop, pages 141-146.
Ethics Statement
Andrew Lampinen. Ishita Dasgupta. Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. 2022. Can language models learn from explanations in con- text? In Findings of the Association for Compula- tional Linguistics: EMNLP 2022, pages 537-563,
Acknowledgements
This work was supported by the National Key Research and Development Program of China (2018YFC08304000 and the Shanghai Sci- ence and Technology Innovation Action Plan
Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
(20511102600).
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.
The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 3045-3059.
References
Jacob Andreas. 2022. Language models as agent mod- els. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5769-5779, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804.
Francesco Barbieri, Jose
Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020. pages 1644-1650, Online. Association for Computational Linguistics.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022a. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086 8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 137777711
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022b. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Shizhe Diao, Pengeheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models. arXiv preprint arXiv:2302.1 12246.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self- self-fee arXiv preprint arXiv:2303.1765:
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. survey for in-context learning. arXiv preprint arXiv:2301.00234
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-1 1164,, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
María Herrero-Zaud Zazo, Isabel Segura-Bedmar, Paloma Martinez, and Thierry Declerck. 2013. The DDI corpus: An annotated corpus with pharmacological substances and drug-drug interactions. 46(5):914- 920.
Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. 2022. Instruction induction: From few examples natural language task descriptions. arXiv preprint arXiv:2205. 10782.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. 2016 task Detecting stance tweets. In Proceedings of the 10th International Workshop Semantic Evaluation (SemEval-2016) pages 31-
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in Neural Information Processing Systems.
41, San Diego California. Association for Computa- tional Linguistics.

---

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabili- ties of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303. 13375.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023b. Automatic chain of thought prompt- ing in large language models. In The Eleventh In- ternational Conference on Learning Representations (ICLR 2023).
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What in-context learning" learns" in-context: Disentangling task recognition and task learning. arXiv preprini arXiv:2305.09731.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In ternational Conference on Machine Learning, pages 12697-12706. PMLR.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023, Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXtv:2304.09797.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with au- tomatically generated prompts. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages -235.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910
KaShun Shum, Shizhe Diao, and Tong Zhang. 2023. Automatic prompt augmentation and selection with chain-of-thought from labeled data. arXiv preprint arXiv:2302.12822
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2023. Language models don't always say what they think: Unfaithful explana- tions in chain-of-thought prompting. arXiv preprint
arXiv:2305.0445 04388.
Özlem Uzuner, Brett R South, Shuying Shen, and Scott I. Vula 2011. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Associ- ation, 18(5):552-556.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka- Wei Lee, and Ee- Ee-Peng Lim. 2023. Plan- and-solve prompting: Improving zero-shot chain-of- thought reasoning by large language models. Pro- ceedings of the 61th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1 Long Papers).
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Tech- nologies. pages 1300-2344 Seattle, United States. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang. Dale Schuurmans. Maarten Bosma, Brian Ichter, Fei Xia. Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing clicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023a. Sentiment analysis in the era of large language models: reality check. arXiv preprint arxiv:230 20to:2305.1005 15005.

---

A Details of Biased Demonstrations
Here are three biased demonstrations omitted in Table 1:
For domain abortion, Our Tour cont' in the @user as we show the humanity of the unborn & inhumanity of abortion!
For domain feminist, Feel things will truly be equal marriage when see jock strap tossed to the "single women' weddings alongside the garter For domain hillary, user Here's to fearless women chasing their goals.
B Details of Classification Tasks
We present the statistics of challenging classifica- tion tasks in Table description in Table 8. prompt template Table 9.

---

Task
Dataset
Train
Dev
Test
Sampled Test Labels
Stancel 16
2620
294
1249
500
3
Multifaceted analysis of subjective text
Emotion20
3257
374
1421
500
4
Topic classification
20news
10.2k
1.1k
7.53k
500
20
i2b2
2808
312
6193
500
8
Medical relation classification
ChemProt
3370
2030
2910
500
5
DDI
3598
400
976
500
4
Table 7: Statistics of the number of samples and labels challenging datasets.
Dataset
Description
Please perform stance detection task.
classes: (A) none (B) against (C) favor
PRTS: 1. Read and understand the context: Carefully read the given statement or
context to fully comprehend its meaning and the topic it addresses.2. Identify the domain: Determine the domain or subject matter the context is related to, such polities, environment, social issues, etc. Make sure the domain matches the one specified in the question or task. 3. Determine the stance: Consider the context's position in relation to the domain and its implications, not just the general topic. Look for keywords, phrases, or sentiments that indicate a position in favor, against, or neutral. 4. Consider the broader implications: Assess how the context's position on the specific issue relates to the broader domain. For example, if the context supports woman's 's right to choose in the domain of abortion, this would imply stance in favor of abortion rights. 5. Choose the most appropriate stance: Based your analysis, select the stance that best represents the position of the context in relation to the specified domain from the given choices (A) none, (B) against, (C) favor. If the context not relevant the domain, choose (A) none.
1
Please perform emotion recognition task. classes: (A) anger (B) joy (C) sadness (D) optimism
PRTS: 1. Read and understand the given context. 2. Identify the emotions expressed in the context. Pay attention to that (B) Joy is transient feeling of happiness and fulfillment (C) while optimism a lasting positive outlook on the future. typically associated with a low mood, a lack of energy, and a tendency to withdraw from social interactions. Consider (C) sadness rather than (D) 3. Match the identified emotions with the given choices. 4. Select the most appropriate choice as the answer.
Emotion20

---

Dataset
Description
Please perform topic classification task from choices.
manual classes:
(A) alt.atheism includes discussions and articles related to atheism, atheistic beliefs, and criticisms of religion (B) comp.gran comp.graphiss covers topics related to computer graphics, including discussions on image processing, rendering, and computer-aided design (CAD) (C) comp.os.ms www.swss...in.ss........... misc focuses on discussions related to the Microsoft Windows operating system, including troubleshooting, software recommendations. and general Windows-related topics (D) wwwpp.yy..bb.p..hardwwaaa revolves around discussions related to IBM-compatible PC hardware, including topies like motherboards, processors, memory, and peripherals (E) ceapp.yy..machhadwee deals with discussions related to Apple Macintosh hardware, including Mac models, peripherals, and troubleshooting (F) comp.windows.x focuses on the X Window System, a widely used Unix-like operating systems, including System configuration, applications, and programming (G) misc.forsale includes postings of items for sale, ranging from electronics to household goods (H) rec.autos covers discussions related to automobiles, including car models, maintenance, buying/selling advice, and automotive technologies (I) rec.motorcycles includes discussions related to motorcycles, covering topics such as different motorcycle models, maintenance, safety, and riding experiences (J) baseball focuses discussions related to baseball, including game analysis, player statistics, team performance, and baseball news (K) rec.sport.oockey covers discussions related to ice hockey, including game analysis, player statistics, team performance, and hockey news (L) sci.crypt revolves around discussions related to cryptography, encryption algorithms,, and cryptographic protocols (M) sci.electronics deals with discussions related to electronics, including electronic circuits, components, and troubleshooting (N) sci.med includes discussions related to medical topics, including diseases, treatments, healthcare practices, and medical research (O) sci.space focuses on discussions related to space exploration, astronomy, and topics related to outer space (P)
20news
christian covers discussions related Christianity, including theological debates, biblical interpretations, and religious practices (Q) talk. dalk..olities..g. revolves around discussions related to gun ownership, gun control policies, and the Second mideast includes discussions related to politics in the Middle East, covering topics such as conflicts, peace negotiations, and regional dynamics (S) halk,politics.miss misc covers discussions related to politics that do not fit into the other specific political categories (T) talk.religion.mise includes discussions related to religion that do not fit into the other specific religious categories
Amendment in the United State (R) talk.politics. politiiss..ideastt
default classes: (A) alt.atheism
(B) comp-gran graphies (C) comp.os. httppossms-windows.iiss..
(D) www.ssys.sc.hharete. hardware (E) comp.sys. eyys.mac.hardwa.
(F) comp. comp.windows.x (G)
forc.forsale (H) rec.autos (I) ret.ootossssles (J) rec.sport.basee basebll (K) rec.sport.hockey
(L) sei.crypt (M) sci.clectronics (N) sci.med (O) sci.space (P) soc. www.riigion.ccrstiie
@ttalkkpoliticss guns (R) talk, woliloooiiiiidbeeeee mideast (S)talk, foalkk.olitics..isse (T) talk.religion.

---

Dataset
Description
What's 's the relation between <c letrraattentt medical test medical problem</el> and <e2>medical problem</e2>?
classes:
(A) Treettmnccc improves <e2>medical problem<</22> (B) <el>Treameet<</l>>
worsens <e2>medical problem</e2>> (C) <el>>rrettent<</l>> causes <e2>medical
problem</e2> (D) <el>Trreament<</l>>>> administered for <e2>medical problem</e2>
(E) <<>>Trcatmentt<<l>>>> is not administered because of <<2>medical problem</e2> (F)
<el>Medical Test</el> reveals <e2>medical problem</e2> (G) <el>Medical Test</el>
conducted to investigate <e2>medical problem</e2> (H) <e <el>Medical problem</el> indicates <e2>medical problem</e2>
PRTS: 1. Read the context carefully and identify the el and e2 elements. 2. Understand the meaning of and e2 in the context. Analyze the relationship between el and e2 based on the context. Keep in mind that the relationship could between medical problems, treatments, or tests. Pay close attention to whether the relationship implies improvement, worsening, causation, or a decision not to administer treatment or test. Also, consider if the treatment is ineffective or has an unintended effect on the medical problem. 4. Compare the relationship with the given choices and select the one that best matches the context. If the relationship is between two medical problems, consider choice (H) possible answer. If the context suggests decision not to administer treatment test, consider choice (E) possible answer. If the treatment is ineffective or worsens the medical problem, consider choice (B) as a possible answer. Be cautious not confuse choice (B) with choice (D) when the treatment is administered but does not have the desired effect on medical problem.
i2b2

---

Dataset
Description
What's the relation between <el>Chemical</el> and <e2>Protein</e2>?
classes:
activity of <e |>>Chmmicak//e>> downregulation, inhibitor, block, decrease activity of <o2>Proocin<<e2>>
<el <el> >Chhmiccak//el>>>>
upregulation,
activate,
promote,
increase
<rotrennin></
<el <ol>Chemiccakk/eD>> is agonist of <<2>Protcin<<e>>
<el> <el>Chemicalk</e>>> is antagonist of <e2>Protein</e2>
<el>Chemical<//b>> is the substrate metabolic of <<2>Protein</e2>
PRTS: 1. Read the context and identify the key terms (el and e2). Understand the ChemProt relationship between el and c2 based on the context. If the relationship not explicitly stated, try to infer from the information provided, including any indirect indications of their interaction. 3. Carefully analyze the inferred relationship and ensure aligns with the context. Pay attention to the specific actions effects of the chemical on the protein, any other relevant information. 4. Match the relationship or inferred relationship with of the given choices (A, B, C, D, E), considering the specific actions effects mentioned in the context and any indirect indications of their interaction. Select the appropriate choice the answer, considering both explicit and inferred information from the context, and double-check to ensure accurately reflects the relationship between and c2.
What's 'ss the relation between <e <el>Drug</el> and <e2>Drug</e2>? manual classes:
Pharmacokinetics mechanism between <el>Drug</el> and <<2>Drug</e2>, the process absorption, distribution, metabolism and excretion of drug. Pharmacodynamica mechanism between <<l>Drug<<el> and <r2>Dru<</22>, i.e., the mechanism action and effect of the drug in the body.
A recommendation or advice regarding a interaction between <el>Drug</el> and <e2>Drug</e2> is given.
An interaction between <el <el>Drug</el> and <e2>Drug</e2> appears without providing any additional information.
Pharmacokinetics, Pharmacodynamics, advise, int PRTS: 1. Carefully read the context sentence and identify the relationship between the two drugs mentioned. Pay close attention the specific details of the interaction, such as metabolism, mechanism of action, recommendations, or lack of additional information. 2. Review the given choices and understand the differences between them. 3. Compare the relationship described in the context sentence with the given choices, focusing on the specific details of the interaction or the absence of such details. If there is recommendation advice regarding interaction between the drugs, consider (C). If the emphasis is the action of the drug in the body, choose (B) Pharmacodynamies: mechanism of action and effect of the drug in the body. (D) Without any additional information, consider (D). 4. Select the appropriate choice based on the context and the definition provided in the choices, ensuring that it aligns with the details mentioned in the context sentence or the lack thereof. Double-check your selection before finalizing your answer, and consider whether the chosen option truly reflects the information provided in the context.
default classes:
DDI
Table 8: Description of tasks.

---

Setting
Default Template
Auto/Random/Similar
Other Templates
Templates
[question]
{question)
(question)
"classes": (classes)
"classes": (classes)
"classes": (classes)
"input": (input)
zero-shot
"input": {input}
"input": (input)
"thinking steps": {think-
"answer":
Let's think step by step.
ing steps)
Let's think step by step.
[question]
(question)
{ (question)
"classes": [classes)
"classes": (classes)
"classes": (classes)
"input": (input)
"input": {imput}
"input": (input)
"answer": [answer]
"answer": (answer)
"answer": (answer)
few-shot
"input": (input)
"input": (input)
"input": (input)
"thinking steps": (think-
"answer":
Let's think step by step.
ing steps)
Let's think step by step.
[question]
(question]
"classes"; [classes)
"classes": (classes)
"input": (input)
"input": (input)
"answer": (answer with CoT)
"answer":
(answer with
few-shot-CoT
CоТ)
"input": (input)
"input": (input)
"thinking steps": [think- ing steps)
Let's think step by step.
Let's think step by step.
Table 9; Prompt templates. Elements in braces {} are replaced with cnestion-speciing values.


Reviews:
[{"review_id": "TCHE2oviHN", "reviewer": "Reviewer_J9Vu", "summary": "", "questions": "Refer to weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer and address all concerns below.\n\n**1. My concern is the cost issue. This iterative framework introduces extremely high token costs, the number of times and tokens to avoid inviting GPT-4 is much more compared to doing ICL directly with GPT-4s, and the ratio of boosts to token growth determines the potential of this approach in practice. And, I also care about the average number of iterations needed.**\n\nThanks for your careful observation, but the cost is inevitable for almost all methods. RL-based methods such as **ReAct** [1] and **Reflexion** [2] require the most cost because of trajectory memory and multiple explorations. **Self-Refine** [3] requires refining each input based on manual feedback. Our method only requires refining one hard demonstration per class on most datasets. Therefore, as many iterations as the number of classes is sufficient to get a satisfying task-level thinking step. No iterative refinement is required for each test input.\n\n**2. From a high-level perspective, the method and reflection are strongly related and some discussion is suggested for the authors.**\n\nThanks for the enlightening insights. Feedback has been the consensus for improving the capabilities of LLMs. However, there are still differences between these approaches. **React** [1] and **Reflexion** [2] excel at multi-step tasks such as sequential decision-making and multi-step reasoning. The key is that the environment and feedback change at each step, fitting well into the reinforcement learning framework. **Self-refine** [3] is good at generative tasks such as dialogue response generation, acronym generation, and sentiment reversal. The key is the richness and plausibility of manually designed feedback. We argue that our contribution is focusing on classification tasks that generative LLMs are not very good at and trying to get LLMs to distinguish between classes through feedback.\n\n**3. This approach requires traversing the training set (147l) to find difficult demonstration, and in the face of a small training set, it may be difficult to find challenging demonstration to generate task-level thinking.**\n\nThanks for your comments. LLMs have achieved performance close to supervised training for simple or publicly exposed tasks. For example, 0.936 accuracy for zero-shot ChatGPT while 0.932 accuracy for fine-tuned T5-large [4]. In this case, hard demonstrations are not easy to find, but applying our method is also unnecessary. We believe that our method is effective for domain-specific classification tasks because of the weaker performance of LLMs (only 0.5025 accuracy for Stance16 [4]).\n\n[1] Yao S, Zhao J, Yu D, et al. React: Synergizing reasoning and acting in language models[. arXiv preprint arXiv:2210.03629, 2022.\n\n[2] Shinn N, Cassano F, Labash B, et al. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv preprint arXiv:2303.11366, 2023.\n\n[3] Madaan, A, Tandon, N, Gupta, P, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.\n\n[4] Zhang W, Deng Yue, Liu B, et al. Sentiment Analysis in the Era of Large Language Models: A Reality Check. arXiv preprint arXiv:2305.15005, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introduces a intuitive technique of task-level thinking steps tailored to classification tasks, advancing prompt engineering for LLMs.\n2. Progressive revision framework purposefully enhances LLM capabilities based on feedback, unlike standard pre-training or fine-tuning.\n4. Ablation studies, robustness tests, and case analyses give useful insights into the framework's capabilities.\n5. Allows generating higher quality chain-of-thought explanations without human involvement.", "reasons_to_reject": "1. My concern is the cost issue. This iterative framework introduces extremely high token costs, the number of times and tokens to avoid inviting GPT-4 is much more compared to doing ICL directly with GPT-4s, and the ratio of boosts to token growth determines the potential of this approach in practice. And, I also care about the average number of iterations needed.\n2. From a high-level perspective, the method and reflection [1] are strongly related and some discussion is suggested for the authors.\n3. This approach requires traversing the training set (147l) to find difficult demonstration, and in the face of a small training set, it may be difficult to find challenging demonstration to generate task-level thinking.\n\n[1] Reflexion: Language Agents with Verbal Reinforcement Learning. 2023.03.", "questions_to_authors": "Refer to weakness.", "paper_topic_and_main_contributions": "This paper proposes using task-level thinking steps and a progressive revision framework to improve the performance of large language models (LLMs) on challenging classification tasks. The key ideas are:\n* Introducing task-level thinking steps to guide the LLM's reasoning process and reduce bias from demonstrations.\n* Progressively revising the thinking steps by having one LLM agent correct another on hard examples, in order to enhance the distinction between confusing classes.\n\nMain contributions:\n1. Proposes the concept of task-level thinking steps to help debias LLM predictions and clarify the reasoning process.\n2. Designs a framework where one LLM agent revises the thinking steps based on feedback from another LLM agent on hard examples.\n3. Shows improved performance over baselines in zero-shot, few-shot, and few-shot-CoT settings on classification tasks requiring multifaceted text analysis, fine-grained distinction, and domain knowledge.\n4. Demonstrates the framework's robustness to biased demonstrations and ability to generate better chain-of-thought explanations.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "o9VG5Obpx8", "reviewer": "Reviewer_ZJta", "summary": "", "questions": "For the \"conversations\" between teacher and student, did the authors put the whole dialog history as part of the prompts for hard demonstrations, or only include the last revised turn of the task-level thinking steps?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer and address all concerns below.\n\n**1. Unlike the simple zero-shot CoT, the new CoT prompt method may not be easy to transfer to more tasks. Since it requires manual design for the prompt words, like \"identify the two medical terms\" , \"identify the domain xxx\" and \"determine the stance of statement\", those template hinders more general-purposed text classification tasks.**\n\nThanks for your comments. In general, \"determine the stance of statement\" etc. are task goals or descriptions, which are usually known a priori [1]. The difficulty lies in what prompts can align the knowledge/bias of  LLMs (introduced by pre-training and RLHF) [2] with task goals. In this paper, we argue that prompts that elucidate the differences between categories learned from hard demonstrations are particularly important for classification tasks.\n\n**2. Lacking stronger baselines, such as ReAct, Reflex, self-critque, self-refine, and AutoGPT. Those are more general design prompt engineering techniques and also can take the demonstration labels as supervised signals to continually revise and review LLM's output, thus are able to generate richer CoT demonstrations for in-context learning. Although I understand some of the baselines are contemporary work, only comparing PRTS with \"let's think step by step\" is too weak.**\n\nThank you very much for the provided relevant work! This issue has been partially discussed in Question2 of the last review. We tried to reproduce the above methods. AutoGPT is a great approach in the unlabelled few-shot setting. **Self-critque** [3] allows LLMs to assist in evaluating complex tasks by fine-tuning simple tasks that humans can evaluate. This is similar to our approach to gaining a deeper understanding of tasks by allowing LLMs to distinguish between choices. However, our approach belongs to prompt engineering and does not require fine-tuning. **Auto** in few-shot-CoT in Table 2 is actually a labeled version of **AutoGPT** [4] but is still inferior to our method. We will complete the related work and provide as many baseline comparisons as possible.\n\n**3. For the \"conversations\" between teacher and student, did the authors put the whole dialog history as part of the prompts for hard demonstrations, or only include the last revised turn of the task-level thinking steps?**\n\nConsidering the computational cost, historical dialogues are not included in the prompt. Also, the initial prompt is always on while the task-level thinking steps are constantly updated.\n\n**4. The authors can elaborate more about why choosing multifaceted subjective text analysis etc, as challenging classification tasks. There are also many other difficult classification tasks like BigBench, MathQA, MSQA, HotPotQA, etc.**\n\nThanks for your advice. The tasks mentioned above are \"multiple choice\" whereas this paper focuses on \"classification\". The difference is that the choices are always the same for all the examples for the classification task. Therefore, our task-level thinking steps can explicitly clarify the difference between the choices and prompt all examples.\n\n**5. Table 1 & 4, what does the different color mean? May need to explain in the captions.**\n\nThanks for your kind advice. Red text indicates incorrect or misleading keywords, while blue text indicates correct or enlightening keywords in the prompt. Additional captions will be added to the revised manuscript.\n\n[1] Zhang W, Deng Yue, Liu B, et al. Sentiment Analysis in the Era of Large Language Models: A Reality Check. arXiv preprint arXiv:2305.15005, 2023\n\n[2] Si C, Friedman D, Joshi J, et al. Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations. ACL, 2023\n\n[3] Saunders, W, Yeh, C, Wu, J, et al. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802.\n\n[4] Zhang, Z, Zhang, A, Li, M, et al. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Propose a set of new prompt templates for text classification tasks. The progressive revision framework uses the data labels as supervision to continually revise and improve the chain-of-thought prompts in an automatic manner. With those generated detailed demonstrations as input, the LLMs can be more robust to biased data samples\n\n2. This work also ensembles a wide range of downstream tasks, including multifaceted analysis of subjective text, fine-grained topic prediction, and vertical domain relation classification, which spans various scenarios. This benchmark could potentially serve as a standard testbed for the evaluation of LLMs or any prompt engineering methods.", "reasons_to_reject": "1. Unlike the simple zero-shot CoT, the new CoT prompt method may not be easy to transfer to more tasks. Since it requires manual design for the prompt words, like \"identify the two medical terms\" , \"identify the domain xxx\" and \"determine the stance of statement\", those template hinders more general-purposed text classification tasks. \n\n2. Lacking stronger baselines, such as ReAct, Reflex, self-critque, self-refine, and AutoGPT. Those are more general design prompt engineering techniques and also can take the demonstration labels as supervised signals to continually revise and review LLM's output, thus are able to generate richer CoT demonstrations for in-context learning. Although I understand some of the baselines are contemporary work, only comparing PRTS with \"let's think step by step\" is too weak.\n\n\n\n\n\n", "questions_to_authors": "For the \"conversations\" between teacher and student, did the authors put the whole dialog history as part of the prompts for hard demonstrations, or only include the last revised turn of the task-level thinking steps?", "paper_topic_and_main_contributions": "This paper proposes a new form of Chain-of-Thought (CoT) prompt design by using GPT-4 to improve the thinking iteratively steps for the demonstration data. Compared with naive zero-shot CoT (i.e., let's think step by step), this method leverages the data labels as a signal to progressively revise the CoT texts so as to cover more nuanced explanations for the hard & biased demonstration data samples. Given the new CoT prompts, the paper shows that text-davinci003, gpt3.5, gpt-4 can all be significantly improved than the baseline on the 7 different classification tasks, such as stance detection and emotion recognition. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Wang X, Wei J, Schuurmans D, et al. Self-consistency improves chain of thought reasoning in language models[J]. arXiv preprint arXiv:2203.11171, 2022.\n2. Saunders W, Yeh C, Wu J, et al. Self-critiquing models for assisting human evaluators[J]. arXiv preprint arXiv:2206.05802, 2022.\n3. Yao S, Zhao J, Yu D, et al. React: Synergizing reasoning and acting in language models[J]. arXiv preprint arXiv:2210.03629, 2022.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "1. The authors can elaborate more about why choosing multifaceted subjective text analysis etc, as challenging classification tasks. There are also many other difficult classification tasks like BigBench, MathQA, MSQA, HotPotQA, etc. \n\n2. Table 1 & 4, what does the different color mean? May need to explain in the captions.\n\n3. Line 195 task-Level ->Task-Level"}}, {"review_id": "Zv1T0wp7xj", "reviewer": "Reviewer_bGCR", "summary": "", "questions": "A Given that the order of demonstration could change the LLM prediction when in context learning, could the order of Hard Demonstration in figure 2 affect the final prediction in your method?\n\nB As an analogy to traditional backdrop finetuning, we could consider task-level thinking steps as the parameter in the student model. How do you know when to stop training and loss coverage in your method?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer and address all concerns below.\n\n**1. Missing comparison with Madaan etal. (2023) to demonstrate the effectiveness of hard demonstrations. Does hard demonstrations method outpeforms all demonstrations method?**\n\nSelf-refine (Madaan et al., 2023) needs to manually design task-specific feedback and refine each test input of the task. Our approach uses simple feedback “*Please revise the thinking steps to clarify.*” and select hard demonstrations to obtain task-level thinking steps. No iterative refinement is required for each test input. Besides, self-refine is better suited for generative tasks (dialogue response generation, acronym generation, sentiment reversal), whereas our approach is better suited for classification tasks.\n\n**2. The task-level thinking steps are updated data by data during PRTS by the teacher. How do you ensure the tokens in task-level thinking steps generated from previous data would not be overwritten by the next data? Given that the order of demonstration could change the LLM prediction when in context learning, could the order of Hard Demonstration in figure 2 affect the final prediction in your method?**\n\nThe order of hard demonstrations does matter for each round of updates. For example, the generated prompt \"*If unsure, choose choice (B)*\" may significantly reduce the accuracy of the other classes. However, such overfitted prompts are usually corrected by later rounds. Alternatively, we employ a checking mechanism that selects the prompt with the least negative impact on the rest of the samples (classes) from among the $K$ candidates. This is discussed from line222 to line232 and Figure 4.\n\n**3. Most results only ran once, please include standard deviation.**\n\nWe have added standard deviation in 3 runs for the zero-shot setting below.\n\n| |Stance16  | Emotion20 | 20news | i2b2 | ChemProt | DDI\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Auto | $47.62\\pm4.01$  | $72.81\\pm2.08$ | $39.97\\pm0.72$ | $29.63\\pm6.59$ | $54.47\\pm6.35$ | $36.22\\pm5.51$ |\n|PRTS|$52.38\\pm6.39$|$75.76\\pm5.75$|$67.20\\pm7.28$|$54.99\\pm4.36$|$68.33\\pm2.36$|$57.36\\pm3.17$|\n\n**4. Missing code to reproduce, and also lack details to differ from previous work. For example, is 1st call in line197 and 2nd call in line 200 implemented as two separate question answer calls to GPT (like step1-feedback and step2-refine in self-refine), or chat call where the 2nd call has access to both human message and AI message of the 1st call**\n\nThe 2nd call accesses to both human message and AI message of the 1st call. We found that the two-stage process of analyzing the student agent's output and then correcting it was more efficient and robust than the single-stage prompt \"*Please analyze the student's output and revise the reflection steps to clarify*\". This is different from the self-refine (Madaan et al., 2023), which iterates several times until the score exceeds the threshold. Our approach does not iterate on this revision. If the prediction is correct, the revision is accepted; otherwise re-analyzed.\n\n**5. As an analogy to traditional backdrop finetuning, we could consider task-level thinking steps as the parameter in the student model. How do you know when to stop training and loss coverage in your method?**\n\nFor the $i$-th update, we use a candidate set of size 3 and choose the one that performs best on $\\\\{x_0, ..., x_{i-1}\\\\}$. This is similar to the greedy selection of the direction with the largest gradient in SGD. Adopting this checking mechanism makes the convergence smoother, as shown in Fig. 4. In our experiments, correcting one hard demonstration of each class provides sufficient improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 The proposed method outperforms let’s think step by step COT and other baselines in all classification proposed.\n\n2 Table 3 shows that PRTS reduced the bias introduced by demonstration significantly. \n\n3 Even though the model is only “trained” on hard demonstrations, however, it does not “overfit” to the corrected classes in hard demonstrations and still knows how to assign to other calsses. This is shown in Figure 4.\n\n", "reasons_to_reject": "1 Missing comparison with Madaan etal. (2023) to demonstrate the effectiveness of hard demonstrations. Does hard demonstrations method outpeforms all demonstrations method?\n\n2 The task-level thinking steps are updated data by data during PRTS by the teacher. How do you ensure the tokens in task-level thinking steps generated from previous data would not be overwritten by the next data? \n\n3 Most results only ran once, please include standard deviation.\n\n4 Missing code to reproduce, and also lack details to differ from previous work. For example, is 1st call in line197 and 2nd call in line 200 implemented as two separate question answer calls to GPT (like step1-feedback and step2-refine in self-refine), or chat call where the 2nd call has access to both [human message and AI message](https://python.langchain.com/docs/modules/model_io/models/chat/) of the 1st call\n", "questions_to_authors": "A Given that the order of demonstration could change the LLM prediction when in context learning, could the order of Hard Demonstration in figure 2 affect the final prediction in your method?\n\nB As an analogy to traditional backdrop finetuning, we could consider task-level thinking steps as the parameter in the student model. How do you know when to stop training and loss coverage in your method?", "paper_topic_and_main_contributions": "This paper tries to solve classification tasks by creating an automatic chain-of-thought (COT) that is task-specific, instead of the general let’s think step by step COT or the data-specific COT that human labels. They named this as task-level thinking steps which reduce the bias introduced by demonstration and help LLMs to clarify tasks. The author also designs a progressive revision framework, which uses a teacher LLM and student LLM to generate and improve the thinking steps based on feedback from hard demonstrations. To demonstrate effectiveness, the author evaluates the proposed method on three kinds of challenging classification tasks: multifaceted analysis of subjective text, fine-grained similar-class text classification, and medical domain-specific classification. They showed that the proposed Progressively Revised Thinking Steps (PRTS) method outperforms competitive baselines in zero-shot, few-shot, and few-shot-CoT settings.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Pryzant et al. Automatic Prompt Optimization with \"Gradient Descent\" and Beam Search (not required to be cited)"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a useful and intuitive concept of task-level thinking steps to help LLMs overcome any bias brought in by demonstration examples in in-context learning setups. It also proposes a framework to revise the thinking steps for confusing classes in classification tasks, by having one LLM correct another on hard examples. It provides comprehensive experiments with several relevant baselines, and demonstrates robustness to biased demonstrations and ability to generate chain-of-thought explanations without human involvement. 

Concerns include the generalisability of the prompt templates (and the manual effort in designing them) to other classification tasks, and the lack of sufficient baselines for more comprehensive comparisons. 

For the authors — please include the cost analysis and a discussion on how to use the framework when the training sets are small in the paper. It would also be useful to open-source the code for future works on this effort.
Decision: Accept-Main