R$^3$ Prompting: Review, Rephrase and Resolve for Chain-of-Thought Reasoning in Large Language Models under Noisy Context
Authors: Qingyuan Tian
Hanlun Zhu
Lei Wang
Yang Li
Yunshi Lan
Venue: EMNLP 2023 Findings (2023)
Keywords: Chain-of-Thought
Large Language Models
Arithmetic Reasoning
Prompt Learning
TL;DR: We propose a novel prompting method for Chain-of-Thought reasoning under noisy context.

Abstract: With the help of Chain-of-Thought (CoT) prompting, Large Language Models (LLMs) have achieved remarkable performance on various reasoning tasks.
However, most of them have been evaluated under noise-free context and the dilemma for LLMs to produce inaccurate results under the noisy context has not been fully investigated.
Existing studies utilize trigger sentences to encourage LLMs to concentrate on the relevant information but the trigger has limited effect on final answer prediction.
Inspired by interactive CoT method, where intermediate reasoning steps are promoted by multiple rounds of interaction between users and LLMs, we propose a novel prompting method, namely R$^3$ prompting, for CoT reasoning under noisy context.
Specifically, R$^3$ prompting interacts with LLMs to perform key sentence extraction, variable declaration and answer prediction, which corresponds to a thought process of reviewing, rephrasing and resolving.
The responses generated at the last interaction will perform as hints to guide toward the responses of the next interaction.
Our experiments show that R$^3$ prompting significantly outperforms existing CoT prompting methods on five reasoning tasks under noisy context.
With GPT-3.5-turbo, we observe 3.7\% accuracy improvement on average on the reasoning tasks under noisy context compared to the most competitive prompting baseline.
More analyses and ablation studies show the robustness and generalization of R$^3$ prompting method in solving reasoning tasks in LLMs under noisy context.

R3 Prompting:
Review Rephrase and Resolve for Chain-of-Thought Reasoning in Large Language Models under Noisy Context
Qingyuan Tian', Hanlun Zhu', Lei Wang', Yang Li3, Yunshi Lan
EEstt China Normal University, Shanghai, China
SSigaapore Management University, Singapore
Alibaba Group, Beijing, China
(qytian, hlzhu | www.ueccnueduu.c. yslan @ dase. dase.ecnu. ecnu. .cn..edu..n
lei. waw.wan.2009 2aw.nn2211@pphcss..ocoo sondes.smu.edu.ss. 1ato0100@@bbbbaaincccoom
Abstract
processing. A series of studies have proved that multi-step reasoning tasks can be easily solved by LLMs via different prompting methods (Kojima et al., 2022; Wei et al., 2023; Wang et al., 2023; Zhang et al., 2022b; Zhou et al., 2022; Zheng et al., 2023; Shi et al., 2023). Shain-of-thougee (CoT), as the representative prompting approach, narrows the gap between human intelligence and machine intelligence by applying the rationales few-shot prompting (Zhou al., 2022). It has achieved im- pressive results on arithmetical reasoning tasks (Ko- jima et al., 2022: Wei et al., 2023; Wang et al., 2023; Zhang et al., 2022b; Zhou et al., 2022; Zheng et al., 2023; Shi et al., 2023).
With the help of Chain-of-Thought (CoT) prompting, Large Language Models (LLMs) have achieved remarkable performance on var- ious reasoning tasks. However, most of them have been evaluated under noise-free context and the dilemma for LLMs to produce inac- curate results under the noisy context has not been fully investigated. Existing studies utilize trigger sentences to encourage LLMs concen- trate on the relevant information but the trigger has limited effect on final answer prediction. In- spired by interactive CoT method, where inter- mediate reasoning steps are promoted by mul- tiple rounds of interaction between users and LLMs, we propose novel prompting method, namely R prompting, for CoT reasoning un- der noisy context. Specifically, R prompting interacts with LLMs to perform key sentence extraction, variable declaration and answer pre- diction, which corresponds thought pro- cess of reviewing, rephrasing and resolving. The responses generated at will perform hints to guide toward the re- sponses of the next interaction. Our exper- iments show that R prompting significantly outperforms existing CoT prompting methods on five reasoning tasks under noisy context. With GPT-3. 5-turbo, we observe 3. 3.7% accu- racy improvement on average on the reasoning tasks under noisy context compared to the most competitive prompting baseline. More analy- ses and ablation studies show the robustness and generalization R prompting method in solving reasoning tasks in LLMs under noisy context.
However, most CoT approaches are investigated under a noisy-free context, where all the informa- tion provided in the problem of interest is relevant to the final answer. While in real-world scenarios, is frequent to encounter problems with irrele- vant information. Shi al. (2023) first defined the reasoning tasks LLMs under noisy context. As shown in Figure 1, the noisy context in the problem of interest includes "Helen baked 12 berry cookies yesterday", which can easily distract LLMs. They further proposed Instructed-Cor prompt, which includes trigger sentence instructing LLMs to ig- nore the irrelevant information given in a question ahead of one-turn response exemplar. Instructed- CoT prompt intuitive to denoise context but it has limited effect on the final answer prediction.
interaction
Inspired by prior studies (Zhou et al., 2022), in- teractive CoT prompting approaches show that con- ducting multi-round of interaction between users and LLMs as hints is efficient in progressively guiding toward the correct answers for complex reasoning tasks (Zheng et al., 2023; Zhou et al., 2022). Least-to-Most (Zhou et al., 2022; Zheng et al., 2023) is CoT prompting method that guides L.Mss to decompose a complex problem into sim- ple -uestuusssons. The prompt first decomposes the problem into subproblems, then the generated inter-
1 Introduction
Recent advances in Large Language Mod- els (LLMs) like GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), OPT (Zhang et al., 2022a), and LLaMa (Touvron et al., 2023) have revolutionized the landscape of natural language
"Corresponding author

---

[ Examplar Problem] Helen baked 19 chocolate cookies and 12 berry cookies yesterday, and she baked 231 raisin cookies and 237 chocolate cookiesthis morning. How many more chocolate cookies than raisincakes did Helen bake?
Noisy Context : Helen baked 12 berry cookies yesterday,
One-turn CoT
Interactive CoT
Least to-most
prompting
Instructed-CoT
Examplar Proalem
Noley Context
Examplar Problem
Noisy Context
Instr
Examplar Problem
Nokey Context
Key Sontences
Exomplar Proplem
Noisy Context
Rationale
A2
Answer Hint
Ratianale
Variable Hint
Rationale
Probiem of Interest + No Context
Prob im of interest Noois Context
Problem of interest Nolsy Context
Problem of Interest Noisy Context
Interaction content In prompt
Other content in prompt
Inrelevant or "trap" information
Interaction with LLMs
Figure I: Comparison between R prompting and existing CoT prompting baseline methods. The exemplar problems are multiple problems we used as exemplars for in-context learning. Rationales are reasoning chains in prompts. The problem interest is the query problem.
mediate answer is appended to the next subproblem until the final answer is returned. PHP (Zheng et al., 2023) is another prompting method with interactive responses that combine the candidate answers and problems for re-evaluation purposes, which allows double-check to the answer prediction. Neverthe- less, these interactive CoT prompting methods are still vulnerable to noisy context.
randomly inserting irrelevant sentences into prob- lem descriptions from MultiArith and SingleEq, respectively. Specifically, follow their template- based method to create templates for noisy context and instantiate with role names in the problems. From the experimental results, we observe that in- teractive CoT has some advantages in solving noisy problems and our R prompting method could out- perform series of CoT prompting methods for rea- soning tasks in LLMs under noisy context, which leads to average improvement of 3.7 absolute points on accuracy. More analyses shed lights on the robustness and generalization of R prompting.
In this paper, we propose new method named R prompting, that includes sequentially interac- tion with LLMs to gradually approach final answers via thought process of Reviewing, Rephras- ing and Resolving. In the review stage, review prompt designed to extract key sentences that are essential conditions required for the final answer prediction. In the rephrase stage, we guide LLMs to reformulate the problem narratives to variables with the hint of extracted key sentences. In the resolve stage, LLMs predict the final answers tak- ing account of the generated variables. All stages are implemented under few-shot setting with show- ing the responses of several exemplar problems. comparison between the existing prompting meth- ods displayed in Figure 1.
2 Related Work
Chain-of-thought prompting. CoT prompt- ing (Wei et al., 2023) is a technique that guides LLMs to conduct multi-step reasoning which can be logically sequential without gradient accumu- lation. CoT has proved to be effective in solving reasoning tasks in few-shot scenarios. There are two major paradigms for COT. One is leverage simple instruction sentence to elicit LLMs do step-by-step thinking before answering a ques- tion (Kojima et al., 2022: Wang et al., 2023) or provide a few manual demonstrations with reason- ing chains to show LLMs how to solve the ex- emplar problems (Wei et al., 2023; Zhang et al., 2022b). This paradigm is designed with one-turn response. Another paradigm of CoT has an inter- active framework (Zheng al., 2023; Zhou et al., 2022; Creswell et al., 2022; Yao al., 2022), which enables automatic multiple rounds of interactions between users and LLMs by appending previously
We evaluate our prompting method on five dif- ferent benchmarks containing misleading and ir- relevant context, that are AddSub (Hosseini et al., 2014), SVAMP (Patel et al., 2021). MultiArith- IC, SingleEq-IC, and GSM-IC (Shi et al., 2023). The last three datasets manually created by in- serting noise to MultiArith (Roy and Roth, 2016), SingleEq (Koncel-Kedzion GSM8K(Shi et al., 2023). We follow Shi et al. (2023) to create MultiArith-IC and SingleFq-IC by
al., 2015) and

---

Review Prompt
Q: Josh had marbles in his collection. Josh's father works hours day. lost 8 marbles found new ones. many more marbles did he find
A: To answer this question, need notice:
LLM
Stage1
H.He lost 8 marbles 2.He found 10 ones
last?
Key Sentences
Stage2
Rephrase Prompt
Josh had marbles his collection. Josh's father works 8 hours day. He lost marbles and found 10 new ones, How many more marbles did he find than those he lost?
A: Number of last marbles Number of found marbles 10.
LLM
Q: Translate owing sentences equation:
He lost marbles 2.Hr found new ones
Variables
Stage3
Resolve Prompt
Josh marbles in his collection. Josh's works hours day. lost marbles and found marbles did he find than those he lost?
Q:
The number of many marbles he found than he lost Number found marbles Number of lost marbles -0-3 The answer is 2.
ones. many
LLM
A: With the Equation Hints: 1.Number of lost marbles = 8, 2.Number of found marbles = 10, we will answer the question.
A running example of the inputs and outputs of R prompting in LLMs each prompting stage. Green: In-topic noisy context. Red: Off-topic noisy context. Blue: Key sentences.
Figure 2:
generated. For example, Least-to-Men Most prompt- ing (Zhou et al., 2022) interacts with LLMs via showing the decomposed sub-questions and cor- responding solutions obtained from the last inter- action, which enables complex reasoning. PHP prompting (Zheng et al., 2023) attaches the can- didate answers generated from the last interaction to ask LLMs to double-check the answers. Our proposed method follows the interactive paradigm but is invested with novel interactive process of thinking.
tence in prompts to decrease the influence from the noisy context to LLMs. Different from the simple instruction for denoising in their prompts. our work focuses on designing a interactive CoT prompt that can steadily solve problems without being misguided by noisy context.
3 R Prompting
Researches on human problem-solving indicate that when solving reasoning task under noisy contexts, the instinct of humanity is to conduct multiple rounds of thinking (Newell et al.. 1959). In this paper, we propose that this process simulated in LLMs by reviewing, rephrasing, and resolving successively. Specifically, we design a prompting method with three-stage prompts: (1) Review prompt: instruct LLMS to read the ques- tion and extract the key sentences from the noisy contexts. (2) Rephrase prompt: with the hint of extracted key sentences, reformulate the narra- tives into abstractive representations with explicit variables. (3) Resolve prompt: with the hint of explicit variables, solve the question and predict the final answers. Figure 2 illustrates the proposed interaction and prompt design for each stage. Next,
Prompting with noisy ground truth. A few stud- ies have investigated the effect of prompts with noisy ground truth, including pairing exemplar problems with incorrect answers (Min al., 2022; Kim et al., 2022) and adding irrelevant or mislead- ing context into exemplars (Webson and Pavlick, 2021; Shi et al., 2023). In particular, Madaan and Yazdanbakhssh (2022) proved that the correctness of math equations in exemplars does not notably influence model's performance, whereas the noise included in the text could hamper the performance dramatically. Similar observation has been dis- cussed in another work (Zhang et al., 2022b). To solve the reasoning task under noisy context, Shi et al. (2023) introduced an intuitive trigger sen-
be
we will introduce the detailed design of prompts

---

each stage.
As shown in Figure 2, "He lost marbles" and "He found 10 new ones" are directly taken as the interactive response of the review prompt. These two sentences are also the essential conditions that we can use to solve the problem.
3.1 Review Prompt
The goal of this stage is to review the input and dis- tinguish the key sentences from the noisy context. Following the prior studies (Shi al.. 2023), we focus on two common categories of noisy informa- tion in contexts: 1) Deceptive in-topic sentences designed to complex the problem, as "Josh had 7 marbles in his collection" shown in Figure 2; 2) Off-topic sentences mixed to distract the solvers, "Josh's father works 8 hours day" shown in Fig- ure 2. To prevent LLMs from being trapped by the noisy information, we provide exemplar problems and their expected responses before the problem of interest in prompts with the following design principles:
So, in the first round of interaction, given the exemplar problems, corresponding responses well as the problem of interest, LLMs will learn to review a problem under noisy context by focusing on the key sentences.
3.2 Rephrase Prompt
In the second round of interaction, we ask LLMs rephrase the narratives of the extracted sentences into abstractive representation, which explicates the variables and corresponding numeral values. To this end, we append the extracted key sentences behind the problem of interest to guide LLMs to translate these sentences into variables.
The exemplar problem should include both categories of the above noisy information, which can clicit comprehensive reviewing ability of LLMs.
We organize the format of response
1. [V,] 2. [V2]
The demonstrations should show the expected key sentences be extracted by L.Mss, which are the essential conditions needed to reason the answers.
where V, denotes the variables corresponding to C,+ In Figure 2, we have "Number of lost marbles = 8" and "Number of found marbles = 10" as output of the rephrase prompt. As opposed to existing zero- shot prompting methods (Wang et al., 2023) where simple trigger sentence is included in the prompt as the instruction, rephrase prompt instructs LLMs by providing well-organized exemplar problems demonstrations.
To apply the first principle, synthesize ex- emplar problems containing diverse noisy contexts. Specifically, we randomly sample several problems from the training data which includes in-topic noisy sentences. Then we write some templates for off- topic sentences which can be instantiated with dif- ferent role names and we randomly insert them into the problems. For example, in Figure 2, have exemplar problems with both types of noise "Josh had marbles in his collection" and "Josh's father works 8 hours a day", which corresponds to in-topic and off- noisy sentences, respectively.
3.3 Resolve Prompt
The third round of interaction to resolve the ques- tion and predict the answer based on the given variables. Similar with the previous prompt, we append the variables behind the original question and request LLMs to provide final answers and we organize the format of response as:
In order to apply the second principle, we orga- nize the format of responses as:
[Question of Interest] [Equations involving V1, V2, ...]. The answer is [A].
To answer this question, we need to notice: 1. [C,] 2. [C,] ...
where the equation shows the reasoning process and [A] denotes the final answer. We extract the arabic numeral as our prediction. In Figure 2, we have "2" as the predicted answer to the question. It is worth noting that even thought we conduct multiple rounds of interaction with LLMs, we in- clude the same exemplar problems with different responses for each stage interaction in R prompt- ing.
where [c,] denotes the i-th key sentence in the problem of interest that LLMs should focus on during reasoning. We arrange them via numerical indexes guide LLMs to explicitly specify their order. Furthermore, ensure the semantic integrity between the key sentences and the original prob- lems, we consider directly extracting textual spans from the original problems as the key sentences.

---

Dutaset
#Sumple
Ave. in-topic
Ave. off-topic
one-turn response prompting method but is de- signed for tasks with noisy context by including a trigger sentence. (2) Least-to-Most (Zhou et al., 2022) and PHP (Shi et al., 2023) are two prompting methods with interactive responses, which lever- age the reasoning paths and candidate answers of LLMs to refine reasoning, respectively, We test their distractibility in the presence of noisy context as illustrated in Figure 1.
GSM-IC
1000
0,5
0.5
MultiArith-IC
600
0.5
0.5
SingEq-IC 508
0.48
0.52
Table 1: Details of constructed datasets. "Ave. in-topic" and "Ave. off-topie" denotes averge number of in-topic sentences and off-topic sentences, respectively.
4 Experimental Setup
4.3 Implementation
4.1 Datasets
To demonstrate the efficacy of R3 prompting method under noisy context, our evaluation is con- ducted on multiple datasets: (1) AddSub (Hos- seini et al., 2014) and (2) SVAMP (Patel et al., 2021) are two challenges of arithmetic reason- ing. where AddSub contains one-unknown arith- metic word problems for up-to-4 grad level students and SVAMP covers addition and subtraction arith- metic word problems. The problems of both these datasets involve noisy context that is designed to confuse problem solvers. These noisy sentences are often in-topic and highly related to the problem but play no role in solving the question. We further include (3) GSM-IC, which is a dataset introduced by Shi et al. (2023) and created for the investigation on the destructibility of LLMs. It consists of sub- set of GSMSK dataset (Cobbe al., 2021) and each problem is interrupted by manually inserting some irrelevant in-topic and off-topic sentences that are generated by templates. (4) MultiArith-IC and (5) EII are two datasets we constructed following the above procedure (Shi et al., 2023) to include more noisy context for each problem. The original problems are extracted from Multi- Arith (Roy and Roth, 2016) and SingleEq (Koncel- Kedziorski et al., 2015), respectively. Since most of them are noisy-free, we create templates for noisy context and instantiate with role names in the prob- lems The statistics of those constructed dataset shown in Table
We employ engines including text-davinci-002. text -daviici-iiie 1003 and 5-TTrbb for ex- periments due to their strong CoT reasoning abil- ities (Brown et al., 2020; Kojima et al., 2022; Ouyang et al., 2022). The main results are obtained via GPT-3. 5-Turbo API. For temperature, the tem- perature of all interactions of comparable prompt- ing methods is set to O except for PHP prompting. Following setup in the original paper (Zheng et al., 2023), the temperature for the second interaction of PHP prompting is set to 0.7 and the others are set to O. Mcanwhile, following Wei et al. (2023), the total number of demonstrations k is set to 8. We select the same problems as exemplars for all the comparable methods to make fair comparison.
Experimental Results
5.1 Main Results
We find that our proposed R prompting enhances LLM's reasoning ability, especially when facing problems under noisy context.
R prompting performs well for CoT reason- ing in LLMs under noisy context. RD prompting consistently outperforms other few-shot prompting methods across all arithmetic reasoning datasets under noisy context. There is minimum improve- ment of 2% in accuracy on AddSub datasct and an overall average increase of 3.3% in accuracy. Among them, it achieves the largest performance gain on SVAMP dataset, where it improves the sec- ond best result 83.6% produced by Auto-CoT to 87.3%. Considering that most of the problems in SVAMP contain in-topic noisy sentences, we be- lieve the improvement from the accurate recog- nition to these sentences. Even though Instructed- CoT is designed with a denoising instruction, it does not provide explicit demonstrations to LLMs
4.2 Baselines
We compare Rd prompting with a set of few-shot CoT prompting baseline methods, which can be cat- egorized into two groups based the interaction with LLMs: (1) Manual-Cof (Zhang et al., 2022b) and Auto-CoT (Zhou al., 2022) are two prompt- ing methods that provide one-turn response in the prompt to show the reasoning process of exemplars to LLMs. Instructed-Com (Shi et al., 2023) is also
'we apply EPT-3.5-Turro-0000 from https://platfomm openai, com/ docs httlo////ools/o/to///eeo/////e

---

Methods
SVAMP
MultiArith-IC
SingleEq-IC
AddSub
GSM-IC
Average
Manual-CoT
79.9
79.5
77.7
85.3
81.0
79.7
One-turn
Auto-CoT
83.6
79.7
77.6
88.0
81.5
82.1
Instructed-Co CoT
81.3
80.1
78.2
87.3
82.0
81.8
Least-to-Most
80.8
77.4
76.2
85.3
81.1
80.2
Interactive
PHP
83.1
80.0
79.0
85.3
85.1
82.5
R Prompting (Ours)
87.3
82.2
81.5
90.0
88.0
85.8
Table 2: Main result on five evaluated datasets, The best and second best results boldfaced and underlined respectively.
such that LLMs struggle learn what kind of infor- mation should be ignored. In contrast, R prompt- ing provides strong supervision on the final answer prediction by showing demonstrations (81.8% vs. 85.8%).
Improvement of R prompting is still signifi- cant with self-consistency. We combine prompt- ing methods with self-consistency strategy (Wang et al., 2022). We sample answers with numbers 5. 10, 20, 40 and draw Figure 3 to show the per- formance change with the increasing number of answers.
The design of interactive prompts important for denoising. Comparing the overall performance one-turn and interactive prompting methods, we notice there is a moderate advantage of interac- tive prompting methods for reasoning under noisy context. We believe this is because interactive CoT prompting methods provide more intensive guidance toward the answers for LLMs. Since the other interactive CoT prompting methods are not designed for the purpose of denoising, their im- provement limited. The three-stage prompts of R prompting are elaborately designed to stimu- late the human ability of thinking and this enables LLMs to learn more comprehensive and efficient problem-solving skill under noisy context.
We discover that the improvement brought by R prompting is not conflict with self-consistency. For example, on AddSub, when the sample number is 5, R$ prompting (90.0%) outperforms Instructed CoT (85.3%) with an improvement of 4.7% abso- lute point. When the sample number increases 40, this gain does not diminish, R prompting still outperforms Instructed CoT by 1.1%. Similar trend occurs on SVAMP dataset. This indicates that the improvement brought by R3 prompting and self- consistency is orthogonal. They play roles on the different aspects of prompting.
Moreover, to demonstrate the generalizability of our method, we also evaluate R prompting on Auit,,rith, SingleEq and GSM8K. The results in- dicate that our method generally shows good results for reasoning in LLMs under noisy context. The complete results are provided in the Appendix A.1.
5.3 Ablation Study
To demonstrate the significance of all stages em- ployed in Rd prompting, display the results of ablation study in Table 3 by testing different com- binations of prompts.
Stages of reviewing, rephrasing and resolving are all important. Combination (1) is similar to traditional one-turn CoT method. Combination (2) and (3) yield average accuracy (84.1% and 82.0%) respectively, which are significantly higher than combination (1). This indicates that both review and rephrase prompts are important. Among them, review prompt provides larger performance gain on average, this indicates that reviewing the prob- lem and identifying the key sentences is important under the noisy context. Overall, including all the prompts achieves the best results, which demon- strates improvement of 6.1 absolute points over combination (1).
5.2 Performance with Diverse Settings
Improvement of 3 prompting is consistent over various engines. We evaluate Manual- COT, Instructed-CoT and R$ prompting with var- ious engines including text-davinci-ee0 and text-davinci-003 and display the results in Ta- ble 4. We observe that R prompting generally outperforms Manual-CoT and Instructed-CoT1 un- der noisy context with various engines. Further- more, the performance R3 prompt can achieve bet- ter results with more powerful engine. On AddSub dataset, few-shot R3 prompting method surpasses the SoTA results (Roy and Roth, 2016) which is ob- tained by training with full data (94.9% - 95.7%).
The effect of review and rephrase prompts vary over different datasets. We notice that combina-

---

(a)AddSub
(b)SVAMP
91.5
88.5
35%
90.0
87.0
accuracy
88.5
85.5
accuracy
81
Instructed CoT
87.0
84.0
78
Manual CoT
82.5
81.0
3 CoT
R CoT
Instructed CoT
Instructed CoT
72
82.5
69
5
2
sample number of enswers
sample number of answers
irrelevant sentences
Figure 3: (a). The results of prompting methods after adding Self- Consistency (SC) on AddSub dataset. (b). The results of prompting methods after adding Self-C Coffssoonsstency (SC) SVAMP dataset.
Figure 4: Accuracy change of various methods with the increasing number of irrelevant sentences AddSub.
Stage Rephrase
AddSub
MultiArith-IC
SingleFq-IC
SVAMP
GSM-IC
Average
Review
Resolve
(1)
x
85.3
79.5
77.8
79,9
81.0
79.7
(2)
88.6
81.0
78.9
84.4
83.0
84.1
(3)
88.3
80.0
79.7
80.7
81.5
82.0
(4)
90.0
82.2
81.5
87.3
88.0
85.8
Table Ablation result of evaluation.
Engine
Method
AddSub
GSM-IC
in datasets AddSub-1, AddSub-2, AddSub-3, and AddSub-4, where "-n" denotes the number of irrel- evant sentences manually inserted in each problem. As shown in Figure 4, R prompting maintains curacy fluctuating around 84% over all the datasets. In contrast, both Instructed-CoT and Manual-CoTT experience decrease in accuracy with the increas- ing amount of irrelevant sentences involved. It proves that R prompting exhibits robust perfor- mance under noisy context while Instructed-Cot and Manual-CoT are vulnerable when facing a large amount of noisy information.
Manual-CoT
84.8
55.8
lext-davinci-0U2
Instructeu-Col
85.8
58.5
Rd prompting
87.0
59.6
Manual-CoT
91.4
72.4
text-davinci-003
Instructed-CoT'
92.0
75.2
Rd prompting
95.7
78.1
Table 4: Performance of prompting methods AddSub and GSM-IC datasets with different engines.
tion (3) shows more observable performance gain over combination (1) on AddSub and SingleEQ-IC datasets than the other datasets. In comparison, combination (2) shows more observable perfor- mance gain AddSub and SVAMP than the other datasets. After investigating the datasets in detail, we find that SingleFQ-IC contains more questions requiring multi-step reasoning, GSM-IC contains more questions with noisy context and AddSub contains both. This highlights an intriguing finding that rephrasing is more crucial for complex reason- ing and reviewing is significant for denoising.
As we discussed before, this study focuses on two categories of noisy context, which are in-topic noise and off-topic noise. To investigate which type of noise R prompting handles better, we collected 200 problems from the AddSub dataset and intro- duce different categories of irrelevant sentences. This form AddSub-in-topic and AddSub-off-topic datasets that contain single type of noisy context. We discovered that Rd achieves 92% accuracy on AddSub-ote offouub-ooff--ppiii topic dataset, while it achieves an curacy of 87.5% AddSub-in-topic dataset. This indicates that compared with off-topic noise, in- topic noise is more difficult be resolved as it provides more deceptive information that can cas- ily "trap" LLMs.
5.4 Effect Different Noisy Context
To explore the effect of R prompting differ- ent numbers and types of noisy sentences, first sample 200 problems from AddSub dataset, then introduce irrelevant sentences to each problem with numbers ranging from 1 to 4. This results

---

Error Type
R3
Original Problem
Calculation error
50%
Mary had dimes and pennies her bank dad borrowed 18 pennies from Mary How many pennies does she have now
False negative in reviewing
4%
False positive in reviewing
36%
Rephrase errors
10%
Manual-CoT Output
Mary had dimes and 38 pennies, so she had total of 21 + 38 59 coins.
Table 5: Error analysis of R prompting.
Her dad borrowed 18 pennies from her, she now has 59 18 41 pennies.
Therefore, Mary has pennies left
5.5 Error Analysis
To better understand how R prompting works un- der noisy context, we present an illustrative exam- ple in Figure 5 that is correctly predicted by R prompting but incorrectly predicted by other meth- ods. As we can see, sentence "21 dimes" is noisy sentence that is related the problem narratives but irreeevan to the final answer prediction. Both Manual-CoT and Instructed- Cor produce wrong rationales that include the noisy sentence but this eventually results in a wrong predicted answer. For R prompting, after reviewing the problems, key sentences are extracted from the problem so that the rephrasing and resolving stages could escape from the disturbance of the noisy context and even- tually output the correct answer. Interestingly, in the rephrase stage, the hint "The number of dimes is irrelevant this question." considered a weak requirement for correct equation. Therefore, the LLMs can directly benefit and obtain a final output which is clear, concise, and free from unnecessary verbosity.
Instructed-Con Output
Mary had 21 dimes and 38 pennies, she had total 59 coins. Her dad borrowed 18 pennies from her, so she now has 59 41 pennies. The answer is 41.
R Prompting Output
We need to notice: 1. "Mary had 38 pennies in her bank.", 2. "Her dad borrowed 18 pennies from Mary.". 3. 3."The number of dimes is irrelevant to this question."
1. "Number of pennies Mary had = 38."
2. "Number of pennies borrowed by dad = 18.
3. "Number of pennies Mary has Number of pennies Mary had Number of pennies borrowed by dad."
Number of pennies Mary has now Number pen nies Mary had Number of pennies borrowed by dad 38 18 20. The answer is 20.
Figure 5: An illustrative problem from AddSub dataset, the result of which is incorrectly predicted by other prompting methods but correctly predicted by R prompting. Incorrect rationales and answers are dis- played color. Correct rationales and answers are displayed in green color. Noisy context is displayed in blue color.
We further sample 50 error cases that R prompt- ing have incorrectly predicted and analyzed them. As shown in Table 5, calculation error is the most common error accounting for around half of the total mistakes. This is caused by the fundamen- tal mechanism of LLMs, where the probabilistic model is utilized to conduct calculation and may not exactly follow the mathematical calculation rules. This can be deemed as the upper bound- ary of solving reasoning tasks in LLMs. Regard- ing the errors caused by problem understanding and reasoning, compared with other baselines like Manual-CoT, R prompting method can already reduce them into a relatively low proportion (70% + 50%). Among them, errors caused by distin- guishing wrong key sentences account for 10% of the total errors, which is significantly lower than Manual-CoT (66% > 40%). We also notice that false positive errors occur more frequently than false negative errors.
Conclusion
In this paper, we proposed R prompting en- able LLMs solve problems under noisy context. involves prompts for three different stages: Re- viewing, Rephrasing, and Resolving. Evaluation on eight datasets across two types of noisy context suggests R prompting sigmificantly outperforms the previous baselines. More analyses and ablation studies show that the performance of R prompt- ing is stable and powerful. Overall, the advantages of R prompting are presented in the following aspects: 1) Achieving substantial performance im- provement on arithmetic reasoning tasks under the interference of noisy context; 2) Exhibiting strong

---

robustness, maintaining stable performance even when facing progressively increasing amounts of noisy context.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14111
Limitations
Antonia Creswell, Murray Shanahan, and Trina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712.
Despite the improvement achieved by R prompt- ing, there are still aspects that can be improved. For instance, greedy decoding may not be the optimal strategy to produce the output answer as the best- first search may not result in the global best result. How to design efficient strategy for sampling is a direction that many researchers are currently investigating (Wang et al., 2022; Weng et al., 2022: Li et al., 2022). In our method, the output in the review stage is currently obtained through greedy decoding. To achieve higher-quality review out- comes, future work could design a strategy, similar to Seff-Consistency, to evaluate and sample the re- sults from the review stage which are necessary and sufficient for problem-solving. Therefore, de- veloping effective sampling strategies based on R$ prompting would be an fascinating direction to ex- plore.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman, 2014, Learning to solve arithmetic word problems with verb categorization. In EMNLP, pages 523-533.
Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo. Sang-Wos Woo Lee. Sang-goo Lee, Kang Min Yoo, and Taeuk Kim. 2022. Ground-truth labels matter; A deeper look into input-label demon- strations. arXiv preprint arXiv: 1205.
Takeshi Kojima, Shixiang Shane Gu. Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint DrXiv:2205.1191
Rik Kedzzio-eeiiiskki.. Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equa- tions. Transuctions of the Association for Computa- tional Linguistics, 3:585-597.
Ethics Statement
This is a study about Arithmetic Reasoning. does not have any data privacy issues; we did not collect any personal information. This is a task that involved no risk as the participants are not exposed to any harmful material or asked to perform any risky tasks.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian GannGuang Lou, and Weizhu Chen. 2022. On the advance of making language models better reasoners. arXiv preprint 2t06:22200.3333
Aman Madaan and Amir Yazdanbakhsh. 2022. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint urXiv:2209.0768. 07686.
A Acknowledgments
Sewon Min, Xinxi Lyu, Ari Holizman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in-context learning work? arXiv preprint a22iv
The authors would like thank the anonymous reviewers for their insightful comments. This work
was supported by Natural Science Foundation of China (Project No. 62206097) and Shanghai Pu- jiang Talent Program (Project No. 22P51403000).
Allen Newell, John C Shaw, and Herbert A Simon. 1959. Report on a general problem solving program. In IFIP congress, volume 256, page 64, Pittsburgh. PA.
References
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1877-1199
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint a1X1v:2103.07191.
Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Subhro Roy and Dan Roth. 2016. Solving gen- eral arithmetic word problems. arXiv preprint 108

---

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023, Large language models can be easily distracted by irrelevant context. arXiv preprint arXiv:2302.00093.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397.
Lei Wang, Wanyu Xu, Yihuai Lan. Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan- and-solve prompting: Improving zero-shot chain-of- thought reasoning by large language models, arXiv preprint wrXh:2335.00099
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Lc, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11111
Albert Webson and Ellie Pavlick. 2021. Do prompt- based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01244:
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thoughh prompting elic- its reasoning in large language models.
Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. 2022. Large language models are reasoners with serfiverifiaation. arXiv preprint 1rX.v.221..09561.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint 10.0322.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022a. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain of thought prompt- ing in large language models. arXiv preprint arXiv:2210.03493.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023. Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXhv:204.099997.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205.1062 10625..

---

A Appendix
A.1
Experiment Result on More Benchmarks
Methods
AddSub
MultiArith
SingleEq
SVAMP
GSMSK
Avg.
Manual-CoT
85.3
97.2
92.9
79.9
78.9
86.8
One-turn
Auto-CoT
88.0
98.2
92.3
83.6
79.9
88.4
Instructed-CoT
87.3
97.2
92.9
81.3
79.1
87.6
Least-to-Most
85.3
89.8
85.0
80.8
79.3
84.0
Interactive
PHP
85.3
98.0
92.9
83.1
84.5
88.8
R$ Prompting (Ours)
90.0
98.2
93.1
87.3
80.1
89.7
Table Accuracy of different methods on benchmarks with the engine of GPT-3.5-turbe
A.2 R Prompting Details
A.2.1 Rd Prompting I: REVIEW (7 Examples)
Q: Zachary did 46 push-ups and 58 crunches in gym class today. David did 38 more push-ups but 62 less crunches than Zachary. In the same gym class, there were 25 students in total. How many more crunches than push-ups did Zachary do?
A:We need notice: L."aachary did 46 push-ups.", 2. 2."Zachary did 58 push-ups.", 3. The number of crunches Zachary did is irrelevant this question.", 4. The number of students is irrelevant this question.".
Q: There are many different books in the 'crazy silly school' series. If you have read 13 of the books and are yet to read 8 books. The author of the series, Mr. Johnson, has written total of 35 books.
How many books are there in the crazy silly school' series?
A: We need to notice: You have read 13 of the books.", 2. "You are yet to read 8 books.", "he sum of read and unread books equal how many books here.". 4. Howw many books did the author write is irrelevant this question.".
Q: Every day Ryan spends hours on learning english and 7 hours on learning chinese. Dur- ing the weekend, Ryan also spends hours on playing video games each day. he learns for 5 days, how many hours does he spend on learning english and chinese in all?
A: We need to notice: 1. "."Eerry day Ryan spends 6 hours on learning english and 7 hours on learning chinese.", 2. "He learns for days.", 3. "How long does Ryan spend on games is irrelevant to this question.".
Q: Matthew had 29 crackers and 30 cakes. The total number of cookies Matthew had was 15. If Matthew gave equal numbers of crackers and cakes his 2 friends. How many cakes did each person eat?
A: We need to notice: 1. ".Mtthhw had 30 cakes.", 2. "atthhw gave equal numbers of crackers and cakes to his 2 friends.". The number of crackers and cookies irrelevant to this question.".
Q: For Gwen's birthday she received dollars from her mom. Her dad gave her 5 more dollars. The total number of guests at Gwen's birthday party was 12. If she spent 4 dollars. How much more money did she receive from her mom than she did from her dad?
A: We need to notice: F.oo Gwen's birthday she received 8 dollars from her mom.". 2. "Her dad gave her 5 more dollars." "The dollars she spent irrelevant to this question." ", 4. "The number of guests is irrelevant to this question.".

---

Q: Jack received 5 emails and letters in the morning, He then received 9 emails and letters in the aftermoon. The total number of text messages Jack received during the day was 24. How many letters did jack receive in the day?
A: We need to notice: 1."Jack received 6 letters in the morning.., ", 2."Jack received 7 letters in the afternoon.", "The number of emails and text messages Jack received is irrelevant to this question.".
Q: Paul had 108 books and 55 pens. After selling some books and pens in garage sale he had 66 books and 59 pens left. Paul also sold 4 tables during the garage sale. How many books did he sell in the garage sale?
A:
We need to notice: "Pal had 108 books.", ", 2. "After selling some books in garage sale he had 66 books left." ", 3. "The number of pens and tables irrelevant this question.".
A.2.2 R Prompting II: REPHRASE (4 Examples)
Jack received emails and letters in the morning. He then received 9 emails and letters in the afternoon. The total number of text messages Jack received during the day was 24. How many letters did jack receive in the day?
Q: Translate following sentences into equation: I.'Jack received 6 letters in the morning." 2. 2.Jack received
7 letters in the afternoon." 3. 'The total number of letters jack received is what need to find'
A: 1. "."ettter received in the morning 6."
2. 2."Letters received in the afterooon 7."
3. Total number of letters = letters received in the morning + letters received in the afternoon = 6 7."
Mary is baking a cake. The recipe calls for 14 cups of flour and 6 cups of sugar. She already put in 5 cups of flour. How many more cups of flour than cups of sugar does she need to add now? Q: Translate following sentences into equation: 1.e recipe calls for 14 cups of flour and 6 cups of sugar." 2. She already put in 5 cups of flour.' 3he difference between the cups of flour and cups of sugar still needed is what need find.'
A: 1. "C"Cp of flour needed 14, Cups of sugar needed 6."
2." "Cups of flour already added 5."
3. "Difference cups of flour still needed cups sugar still needed= cups of flour needed cups of flour added cups of sugar needed."
Every day Ryan spends 6 hours on learning english and 7 hours on learning chinese. During the weekend, Ryan also spends 3 hours on playing video games each day. If he learns for 5 days, how many hours does he spend learning english and chinese in all?
Q: Translate following sentences into equation: 1''Every day Ryan spends 6 hours on learning english and
7 hours on learning chinese." 2. 'He learns for days.'
A: 1. "ime for english in one day 6 hours, Time for chinese in one day 7 hours."
2. The total time of learning chinese and english = (Time for english in one day + Time for chinese in one day) 5."
For Gwen's birthday she received 8 dollars from her mom. Her dad gave her 5 more dollars. If she spent 4 dollars. How much more money did she receive from her mom than she did from her dad?
Q: Translate following sentences into equation: 1. I'Gwen's birthday she received dollars from her mom."
2. Her dad gave her more dollars.'
A: 1. "dollars received from mom =8."
2. "dollars received from dad 5."

---

A.2.3 R Prompting III: RESOLVE (8 Examples)
Q: Zachary did 46 push-ups and 58 crunches in gym class today. David did 38 more push-ups but 62 less crunches than Zachary. In the same gym class, there were 25 students in total. How many more crunches than push-ups did Zachary do?
A: With the Equation Hints: PPush-ups number of Zachary 46.", 'Crunches number of Zachary =5...
"More crunches number = Crunches number of Zachary Push-ups number of Zachary = 58 46.', we will answer the question. Difference in crunches number and Zachary number Crunches number of Zachary Push-ups number of Zachary = 58 46. 58 46 = 12. The answer is 12.
Q: There are many different books in the 'crazy silly school' series. If you have read 13 of the books and are yet read 8 books. The author of the series, Mr. Johnson, has written total of 35 books. How many books are there in the crazy silly school' series?
A: With the Equation Hints: 'books (you have read) number 13, books (yet to read) 8.', 'books number in the series books (you have read) number books (yet to read) = 13 8.', we will answer the question. Books number in the series books (you have read) number books (yet to read) 13 8. 13
+ 8 = 21. The answer is 21.
Q: Every day Ryan spends 6 hours on learning english and 7 hours on learning chinese. Dur- ing the weekend, Ryan also spends 3 hours on playing video games each day. If he learns for 5 days, how many hours does he spend on learning english and chinese in all?
A: With the Equation Hints: 'Time for english in one day 6 hours, Time for chinese in one day hours., 'The total time of learning chinese and english (Time for english in one day + Time for chinese in one day)*5 *5 13 * 5.', we will answer the question. The total time of learning chinese and english (Time for english in one day + Time for chinese in one day) * 13 5. So, 13 # 5 65. The answer is 65.
Q: Mary is baking a cake. The recipe calls for 14 cups of flour and 6 cups of sugar. Mary's 's mother has already made 3 cakes before. She already put in 5 cups of flour. How many more cups of flour than cups of sugar does she need to add now?
A: With the Equation Hints: "Cups of flour needed = 14, Cups of sugar needed 6.', 'Cups of flour already added = 5.,, 'Difference cups of flour needed cups of flour already added cups of sugar needed 14 5 6.', we will answer the question. There 14- 5 66 6= 6- cups of flour than cups of sugar she need to add. The answer is
Q: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?
A: With the Equation Hints: 'Shawn started with 5 toys.', 'Shawn got 2 * 2 4 toys.' we will answer the question. Shawn started with 5 toys. Then Shawn got 2 * 2 = 4 toys. 5 + 4 = 9, The answer is 9.
Q: Jack received 5 emails and letters in the morning. He then received 9 emails and letters in the afternoon. The total number of text messages Jack received during the day was 24. How many letters did jack receive in the day?
A: With the Equation Hints: 'Letters received in the morning = 6.', 'Letters received in the afternoon 7.', 'Total number of letters = letters received in the morning + letters received in the afternoon = 6 + 7.', we will answer the question. Total number of letters = letters received in the morning + letters received in the afternoon = 6 7. So, 6 7 = 13. The answer is 13.
Q: For Gwen's birthday she received dollars from her mom. Her dad gave her 5 more dollars.
If she spent 4 dollars. How much more money did she receive from her mom than she did from her dad? A: With the Equation Hints: 'Dollars received from mom = 8.', 'Dollars received from dad = 5.', we will answer the question. She received 8 5 dollar from her mom than from her dad. The answer is 3.

---

Q: Paul had 108 books and 55 pens. After selling some books and pens in garage sale he had 66 books and 59 pens left. Paul also sold tables during the garage sale. How many book did he sell in the garage sale?
A: With the Equation Hints: "Original number books = 108.", 'Final number of books 66.", 'Books sold = original number of books final number of books = 108 66.', we will answer the question. The num- ber of books he sold original number of books- final number of books 108 66 42. The answer is 42.
A.3 Success Example Outputs by R Prompting on Different Datasets
A.3.1 SVAMP
Question:
Bobby ate 33 pieces of candy. Then he ate more. He also ate 14 pieces of chocolate. How many pieces of candy and chocolate did Bobby eat altogether?
Review stage output:
We need to notice: 1. 1."Bobby ate 33 pieces candy.". He ate 4 more pieces candy.", H. also ate 14 pieces of chocolate.", 4. The total number of pieces of candy and chocolate relevant to this question."
Rephrase stage output:
1. "Peices of candy eaten 33." 2." "Additional pieces of candy eaten 4." "PPiecs of chocolate eaten 14." 4. TToaa number of pieces of candy and chocolate eaten pieces of candy eaten additional pieces of candy eaten + pieces of chocolate eaten."
Resolve stage output:
Total number of pieces of candy and chocolate eaten = pieces of candy eaten + additional pieces of candy eaten + pieces of chocolate caten = 33 4 + 14 51. The answer is 51.
A.3.2 MultiArith-IC
Question:
Wendy's grandmother sold 82 concert tickets. Wendy had 82 files on her computer. Wendy deleted 37 of them from computer and put the rest into folders with 5 files each one. How many folders did Wendy end up with?
Review stage output:
We need to notice: 1. 1."Wendy had 82 files on her computer.", 2. "Wendy deleted 37 of them from her computer.", "he number of concert tickets sold by Wendy's grandmother is irrelevant to this question.", 4. "The number of files in each folder is given as 5."
Rephrase stage output:
1." "..Number of files on Wendy's 's computer = 82." 2. "umber of files deleted from Wendy's computer = 37." 3. "Number of concert tickets sold by Wendy's grandmother = irrelevant." " 4. N.uuub of files each folder = 5." 5. "Number of folders Wendy ended up with = (Number of files on Wendy's computer Number of files deleted from Wendy's computer) Number of files in each folder."
Resolve stage output:
Number of folders Wendy ended up with (Number of files on Wendy's 's computer Number of files deleted from Wendy's computer) Number files in each folder = (82 37) 5 = 9. The answer is 9.

---

A.3.3 SingleEq-IC
Question:
Last year, David bought 10.0 books about science. This year, Vincent bought 10 books about animals, book about outer space, and books about trains. Each book cost $16. How much did Vincent spend on the books, this year?
Review stage output:
We need to notice: 1. 1."Vincent bought 10 books about animals, book about outer space, and 3 books about trains.", 2. "Each book cost S16.", 3. "How much David spent is irrelevant to this question."
Rephrase stage output:
1. "NNubber of books Vincent bought 10 + 1+3 14." 2. "Cost of each book =$16." "otal cost of books Vincent bought = number of books Vincent bought * cost of each book = 14 $16."
Resolve stage output:
Total cost of books Vincent bought = number of books Vincent bought * cost of each book = 14 $16 $224. The answer is $224.
A.3.4 AddSub
Question:
Jason has 18 books and he has read of them Mary has 42 books How many books do they have together?
Review stage output: We need to notice: 1. J."Jason has books and he has read them.". 2."Mary has 42 books.". 3. "The number books they have read together is irrelevant to this question."
Rephrase stage output:
I. "Number of books Jason has = 18, Number of books Jason has read = 9.. 2. "Number of books Mary has = 42." Total number of books they have together Number of books Jason has Number of books Mary has 18 + 42."
Resolve stage output:
Total number of books they have together = Number of books Jason has + Number of books Mary has 18 + 42. So, 18 + 42 = 60. The answer is 60.
A.3.5 GSMIC
Question:
Jackie spends 8 hours working, 3 hours of exercise and spends hours of sleep per day. Jackie's 's sister is 3 years old. How much free time does Jackie have?
Review stage output:
We need to notice: 1."Jackie spends 8 hours working, 3 hours of exereise and spends 8 hours of sleep per day.", 2. "The age of Jackie's sister is irrelevant this question.".
Rephrase stage output:
1. Tooal hours in a day = 24." "Hours spent working = 8, Hours spent exercising = 3, Hours spent sleeping
= =8." "Total hours spent = Hours spent working Hours spent exercising Hours spent sleeping = 8

---

"Free time Total hours in day Total hours spent 24-08434 8)." 2. "The age of Jackie's sister is irrelevant to this question."
Resolve stage output:
Total hours spent Hours spent working + Hours spent exercising + Hours spent sleeping = +8+3 3 + 19. Free time = Total hours in a day Total hours spent = 24 - (8 + +3+ 8) 5. The answer is 5.


Reviews:
[{"review_id": "IXqCaXojYs", "reviewer": "Reviewer_k3pf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer k3pf for positive comments and precious feedback on our work. We response specific comments below.\n\nQ: The contributions of this paper are incremental. This paper targets a narrow area of irrelevant context in reasoning. The proposed approach of incorporating extracting and rephrasing steps is very straightforward and does not offer substantial improvements over existing methods.\n\nA: We thank the reviewer for raising this point. We would like to explain that the main contributions of our work lie in following aspects.\n1. Research of irrelevant context in reasoning is a novel and highly practical endeavor. As we all know, early CoT work has achieved significant success in simple reasoning tasks. However, as shown in the latest research [1], it was discovered that the reasoning ability of LLMs can be easily disrupted by irrelevant information, leading to a significant drop in performance. As demonstrated in Table A1 in the paper's appendix, the existence of such irrelevant information has led to a 13% decrease in the performance of studies like PHP (98% on MultiArith → 85.3% on AddSub). This highlights the necessity of enhancing LLMs ability for noisy context reasoning. However, aside from [1], there is currently limited research exploring this issue further.\nTherefore, our research aims to address the shortcomings of current CoT research, rather than simply transplanting existing approaches into a completely new domain.\n2. Our method may appear simple and straightforward, but it has been carefully designed. By simulating the human reasoning-solving patterns, we have developed this concise yet efficient method. As shown in Figure 1, our method follows an interactive CoT manner but with different prompting strategies. Our goal is to impart general reasoning capabilities to LLMs, not just eliminating interference. As demonstrated in Appendix Table A1, our method outperforms other baseline methods, even in reasoning problems without interference, showcasing its universality.\n3. Our research has shown general improvement on a series of reasoning tasks and yielded insightful conclusions that could inspire future work. As described in Section 5.5, reasoning problems with trap sentences are often more challenging. LLMs frequently struggle to identify these traps during the review stage, leading to errors. Therefore, we suggest future research could explore better strategies for handling trap information.\n\n[1] Shi, Freda, et al. \"Large language models can be easily distracted by irrelevant context.\" International Conference on Machine Learning. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is straightforward and intuitive.", "reasons_to_reject": "The contributions of this paper are incremental. This paper targets a narrow area of irrelevant context in reasoning. The proposed approach of incorporating extracting and rephrasing steps is very straightforward and does not offer substantial improvements over existing methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "LLMs are easily distracted by irrelevant context, leading to lower performance. This paper presents a prompting method for mitigating this issue. The method involves: (1) extracting relevant context; (2) rephrasing the context into hints; (3) getting the answers based on both original problems and hints. Experiments show the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZKWVXp15CR", "reviewer": "Reviewer_56UP", "summary": "", "questions": "Issues with reproducing the example in the error analysis. In the presented error analysis example in Figure 5, reviewing process in $\\mathrm{R^3}$ prompting fails to recognize \"She missed 12 games this year\" as irrelevant, but its corresponding variable doesn't appear in the rephrase stage. As such, I reproduce rephrase process based on the rephrase prompt provided in the Appendix of the paper, LLM models fails to generate outputs as shown in the error analysis. The temperature is set as 0.\n\n---\nReproduced outputs\n\ngpt-3.5-turbo-0301\n>A: 1.\"Number of soccer games attended this year = 11.\"  \n2.\"Number of soccer games missed this year = 12.\"  \n3.\"Number of soccer games attended last year = 13.\"  \n4.\"Number of soccer games planned to attend next year = 15.\"  \nTotal number of soccer games attended = Number of soccer games attended this year + Number of soccer games attended last year + Number of soccer games planned to attend next year = 11 + 13 + 15.\"\n\ntext-davinci-003\n>A: 1.\"Soccer games attended this year = 11.\"  \n2.\"Soccer games missed this year = 12.\"  \n3.\"Soccer games attended last year = 13.\"  \n4.\"Soccer games planned to attend next year = 15.\"  \n5.\"Total soccer games attended = soccer games attended this year + soccer games attended last year + soccer games planned to attend next year - soccer games missed this year.\"\n\ntext-davinci-002\n>A: 1.\"Games this year = 11.\", 2.\"Games missed this year = 12.\", 3.\"Games last year = 13.\", 4.\"Games next year = 15.\"  \n5.\"Total games = games this year + games last year + games next year - games missed this year.\"", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 56UP for positive comments and precious feedback on our work. We response specific comments below.\n\n**Q1**: The paper describes incremental work and the contribution is limited. The core contribution seems to be the fusion of interactive CoT prompting with CoT reasoning in the presence of noisy contexts. Given that both of these domains are already established and naturally aligned, the integration does not come across as particularly novel.\n\n**A1**: We thank the reviewer for raising this point. We would like to explain that the novelty of our work lies in following aspects.\n1. The core contribution is the design of an interactive CoT method that aligns with human reasoning thinking patterns. The essence of nearly every interactive CoT method is to integrate interactive CoT prompts with CoT reasoning, such as PHP and Least-to-most. But why can't other interactive CoT methods effectively distinguish noisy contexts, while our method is highly effective in both noisy and noiseless contexts? We believe the core reason lies in the fact that our interactive mode aligns more with human thinking patterns. Human thinking is a deliberate and thoughtful decision-making system, as system 2 in [1], which allows it to better handle more complex reasoning problems.\n2. Research of irrelevant context in reasoning is a novel and highly practical endeavor. As we all know, early CoT work has achieved significant success in simple reasoning tasks. However, as shown in the last research [2], it was discovered that the reasoning ability of LLMs can be easily disrupted by irrelevant information, leading to a significant drop in performance. As demonstrated in Table A1 in the paper's appendix, the existence of such irrelevant information has led to a 13% decrease in the performance of studies like PHP (98% on multiArith → 85.3% on Addsub). This highlights the necessity of enhancing LLMs ability for noisy context reasoning. However, aside from [2], there is currently limited research exploring this issue further.\n\nNotably, Reviewer 1 acknowledges the novelty of our idea, and Reviewer 3 agrees that our method's superiority for unlocking LLM’s in-context learning ability for reasoning tasks.\n\n[1] Yao, Shunyu, et al. \"Tree of thoughts: Deliberate problem solving with large language models.\" arXiv preprint arXiv:2305.10601 (2023).\n\n[2] Shi, Freda, et al. \"Large language models can be easily distracted by irrelevant context.\" International Conference on Machine Learning. PMLR, 2023.\n\n**Q2**: The paper does not seem to tackle new problems or deliver significant new insights into the CoT reasoning under noisy context.\n\n**A2**: We do believe there is a misunderstanding. This paper focuses on a new defined problem that is improving LLMs under noisy context and ends with some useful conclusions. Specifically, our method can generalize to solve reasoning problems containing noise. As we mentioned in last question, reasoning with irrelevant information represents a novel and more challenging category of tasks. As shown in Table 2 and Table A1, R3 prompting is an efficient and versatile method that outperforms other methods on datasets with and without irrelevant information. Additionally, Figure 4 demonstrates that as irrelevant information interference increases, this method exhibits strong robustness.\n\nFurthermore, our research has yielded some insightful conclusions. As described in Section 5.5, LLMs often struggle to identify these traps during the review stage, leading to errors. Therefore, how to encourage LLMs to recognize traps becomes crucial. This applies not only to reasoning tasks but is equally important in tasks like misinformation detection, question answering, translation, and more.\n\n**Q3**: Didn't use Instructed LTM as a benchmark. \n\n**A3**: We thank the reviewer for raising this point. Actually, we have conducted experiments with Instructed LTM, as shown in the following table. The results indicate that this approach inherits the drawbacks of LTM in simpler reasoning tasks, where deliberate decomposition of problems can lead to a significant performance decline. An intuitive observation is that while LTM achieves perfect performance on GSM8K, its performance drops on AddSub. Instructed LTM follows a similar pattern. Thanks for your suggestions, in our revised version, we will add Instructed LTM into the second section in Table 2 to give a more comprehensive evaluation. But it is worth noting our method still surpasses Instructed LTM on those five datasets.\n\n|                     | SVAMP | MultiArith-IC | SingleEq-IC | AddSub | GSM-IC |\n| :-----------------: | :---: | :-----------: | :---------: | :----: | :----: |\n|   Instructed LTM    | 81.1  |     78.3      |    77.2     |  85.6  |  84.5  |\n| R3 Prompting (Ours) | 87.3  |     82.2      |    81.5     |  90.0  |  88.0  |\n\n**Q4**: Question about the output.\n\n**A4**: We are so sorry if we make any confusion in writing part. Actually, Figure 5 is not a complete output demonstration. And your output is absolutely right, but in the process of figure making, due to space constraints and other reasons, we omitted the phrases \"Number of soccer games missed this year = 12.\" and \"Total number...13 + 15.\"\n\nThe omission of the first sentence was because it constituted a transcription of irrelevant information. Since the transcription results of irrelevant information were not used in the resolve stage, and due to space limitations, we decided to remove it.\n\nThe reason for omitting the second sentence is: After obtaining the initial output from the rephrase stage, we use regular expressions to remove extra sentences and retain only the sentences within double quotation marks. Therefore, to avoid unnecessary ambiguity in the figure, we decided to remove it.\n\nAfter you raised this question, we realized that we chose an inappropriate example for illustration, especially since the first stage fails to recognize noisy context. It also caused confusion for the readers in the figure. We sincerely apologize for that. Here is a better example in which each stage has perfectly achieved its intended goals:\n\n**Target question**：Jason has 18 books and he has read 9 of them . Mary has 42 books . How many books do they have together ?\n\n**Review output**：We need to notice: 1. \"Jason has 18 books.\", 2. \"Mary has 42 books.\", 3. \"The number of books they have read is irrelevant to this question.\"\n\n**Rephrase output**：1.\"Number of books Jason has = 18.\" 2.\"Number of books Mary has = 42.\" 3.\"Total number of books they have together =Number of books Jason has + Number of books Mary has.\"\n\n**Resolve output**：Total number of books they have together = Number of books Jason has + Number of books Mary has = 18 + 42 = 60. The answer is 60.\n\nIn the example above, the first stage correctly identifies \"The number of read book\" as irrelevant information. Secondly, during the rephrase stage, the LLMs directly outputs the reason for irrelevance, which is the correct problem-solving equation. Since this output is used as a hint-prompt input in the third stage, the LLMs can directly benefit and obtain the correct result in the resolve stage. Therefore, the last resolve process is clear, concise, and free from unnecessary verbosity. We will replace Figure 5 with this example in revised versions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- $\\mathrm{R^3}$ prompting is effective, as evidenced by marked improvements when compared to baseline methods.\n- The paper is easy to follow and the experimental results are thoroughly analyzed.", "reasons_to_reject": "- The paper describes incremental work and the contribution is limited. The core contribution seems to be the fusion of interactive CoT prompting with CoT reasoning in the presence of noisy contexts. Given that both of these domains are already established and naturally aligned, the integration does not come across as particularly novel. In addition, the paper does not seem to tackle new problems or deliver significant new insights into the CoT reasoning under noisy context. For a submission to EMNLP, one might expect a deeper exploration.\n- In [1], which serves as the primary reference for this paper, an interactive CoT prompting method (least-to-most prompting) is adopted within the context of CoT reasoning under noise. This method, designated as Instructed LTM, achieves best performance in [1]. However, in this paper, the $\\mathrm{R^3}$ prompting is benchmarked only against a sub-optimal model in [1], the Instructed CoT, without any comparison to the Instructed LTM.\n\n---\n[1] Shi, Freda, et al. \"Large language models can be easily distracted by irrelevant context.\" International Conference on Machine Learning. PMLR, 2023.", "questions_to_authors": "Issues with reproducing the example in the error analysis. In the presented error analysis example in Figure 5, reviewing process in $\\mathrm{R^3}$ prompting fails to recognize \"She missed 12 games this year\" as irrelevant, but its corresponding variable doesn't appear in the rephrase stage. As such, I reproduce rephrase process based on the rephrase prompt provided in the Appendix of the paper, LLM models fails to generate outputs as shown in the error analysis. The temperature is set as 0.\n\n---\nReproduced outputs\n\ngpt-3.5-turbo-0301\n>A: 1.\"Number of soccer games attended this year = 11.\"  \n2.\"Number of soccer games missed this year = 12.\"  \n3.\"Number of soccer games attended last year = 13.\"  \n4.\"Number of soccer games planned to attend next year = 15.\"  \nTotal number of soccer games attended = Number of soccer games attended this year + Number of soccer games attended last year + Number of soccer games planned to attend next year = 11 + 13 + 15.\"\n\ntext-davinci-003\n>A: 1.\"Soccer games attended this year = 11.\"  \n2.\"Soccer games missed this year = 12.\"  \n3.\"Soccer games attended last year = 13.\"  \n4.\"Soccer games planned to attend next year = 15.\"  \n5.\"Total soccer games attended = soccer games attended this year + soccer games attended last year + soccer games planned to attend next year - soccer games missed this year.\"\n\ntext-davinci-002\n>A: 1.\"Games this year = 11.\", 2.\"Games missed this year = 12.\", 3.\"Games last year = 13.\", 4.\"Games next year = 15.\"  \n5.\"Total games = games this year + games last year + games next year - games missed this year.\"", "paper_topic_and_main_contributions": "The paper investigates the problem of CoT prompting under noise context and propose an interactive prompting method termed $\\mathrm{R^3}$ prompting. This method contains three stages: reviewing, rephrasing, and resolving. These processes collaboratively work to extract key sentences, reformulate the problem narratives to variables, and subsequently predict the final answers. Empirical results from five distinct datasets demonstrate the effectiveness of the proposed method.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RAlz1v737u", "reviewer": "Reviewer_FVdJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer FVdJ for positive comments and precious feedback on our work. We response specific comments below.\n\n**Q1**: The paper's evaluation scope is quite narrow. A comprehensive understanding of the method's versatility could be gained by testing it across a wider range of tasks. \n\n**A1**: We thank the reviewer for raising this point. Because our research falls within the CoT category, we followed the established standard settings of this field in experiment section, which is the same setting as [1], [2], [3], [4], [5] and so on. As shown in Table A in the paper's appendix, we initially conducted experiments on five datasets (AddSub, MultiArith, SingleEq, SVAMP, GSM8K) to demonstrate the versatility of our method. Furthermore, as we aimed to showcase the excellent noise resilience of our method in the face of irrelevant context, we followed the approach outlined in [1] to construct datasets (MultiArith-IC, SingleEq-IC, GSMIC). The experimental results for these datasets can be found in Table 2 in Section 5.2 of our paper.\n\nIn summary, the paper's evaluation scope not only follows the experimental setups of related works but also introduces newly constructed tasks involving irrelevant context. And thanks for your suggestions, in our revised version, we will try more tasks like symbolic reasoning, commonsense reasoning and so on.\n\n[1]Shi, Freda, et al. \"Large language models can be easily distracted by irrelevant context.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2]Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in Neural Information Processing Systems 35 (2022): 24824-24837.\n\n[3]Zhang, Zhuosheng, et al. \"Automatic chain of thought prompting in large language models.\" arXiv preprint arXiv:2210.03493 (2022).\n\n[4]Zhou, Denny, et al. \"Least-to-most prompting enables complex reasoning in large language models.\" arXiv preprint arXiv:2205.10625 (2022).\n\n[5]Zheng, Chuanyang, et al. \"Progressive-hint prompting improves reasoning in large language models.\" arXiv preprint arXiv:2304.09797 (2023).\n\n**Q2**: The lack of in-depth ablation studies on each stage of R3 prompting prevents a clear understanding of their individual contributions to the overall performance. \n\n**A2**: We thank the reviewer for raising this point. From Table 4 in Section 5.3, it's evident that each stage of R3 prompting plays an indispensable role. Moreover, the next stage depends to some extent on the previous stage, as average accuracy on combination 1 (79.7%) $\\lt$ combination 2 (84.1%) $\\lt$ combination 4 (85.8%). As for the individual roles of each stage, the review stage is the most crucial for filtering out irrelevant information, as evidenced by accuracy of combination 2 on the four datasets surpassing combination 3 (e.g. On AddSub 88.6% $\\lt$ 88.3%). On the other hand, the rephrase stage is more beneficial for complex reasoning, as it essentially pre-processes information before problem-solving. Because these two points are intuitive and easily explainable, we did not provide overly detailed explanations in the paper. Overall, by combining these three stages, we achieved the best performance.\n\n**Q3**: The paper lacks complex examples and a detailed error analysis, which could offer a more nuanced understanding of the method's capabilities and limitations. \n\n**A3**: We thank the reviewer for raising this point. Regarding our method's capabilities and limitations, we provided detailed explanations of these aspects in Section 5.5 and the Limitations section of the paper. The primary concerns relate to the efficiency and accuracy of the review stage, with more complex examples or detailed error analyses largely associated with the first stage. However, we believe that employing sampling strategies akin to Self-consistency might significantly enhance the efficiency of this method.\n\nWe will consider including more detailed error examples in the appendix in future versions.\n\n**Q4**: There is a conspicuous absence of comparison with related works, particularly those involving interactive prompting. \n\n**A4**: We thank the reviewer for raising this point. The main distinctions between our approach and other interactive prompting methods are as follows:\n1. Different prompt design. Our approach simulates a general human thinking process. As shown in Figure 1, We employed a workflow-based approach to design this human-like prompting method. In contrast, other interactive prompting methods often rely on a multi-turn dialogue approach. For instance, PHP requires multiple iterations to refine the answer until a confidence threshold is met. Similarly, Least-to-most involves a multi-turn question-and-answer process until a final answer is obtained. Therefore, from a design perspective, the differences between these methods are quite distinct.\n2. Different performance. As shown in experiments in Section 5.1 and 5.4, our model is mainly designed for reasoning tasks involving noisy context. Experiment results show the superiority of our method and its robustness as interference increases. Additionally, as mentioned, our method also outperforms PHP and Least-to-most in typical reasoning tasks.\n\n**Q5**: The document could provide more information on how the R3 prompting strategy might be adapted or modified for different types of problems or different models.\n\n**A5**: We thank the reviewer for raising this point. Firstly, our method can be easily applied to other large language models, as demonstrated in Table 4 of the paper. Secondly, our method is a human-like, universal approach, making it highly adaptable to other tasks. It might require some modifications to the prompt descriptions for different stages when applied to other tasks to meet specific requirements, but the functionality of each stage remains unchanged.\n\n**Q6**: The authors could also explore more about how this strategy could be incorporated into existing LLM training strategies to further enhance performance.\n\n**A6**: We thank the reviewer for raising this point. However, for a paper that utilizes prompting learning as its fundamental method, we believe that LLM training strategies do not fall within the scope of discussion. Nevertheless, we are excited to see future work incorporating this strategy into the training of LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1)The paper offers an innovative, multi-stage design with R3 Prompting, which emulates human reasoning processes.\n2)The authors present evidence of their method's superiority over certain Chain-of-Thought (CoT) methods, such as Manual-CoT and Instructed-CoT.\n3)The authors candidly acknowledge the limitations of their methodology, which adds a degree of credibility to their research.", "reasons_to_reject": "1)The paper's evaluation scope is quite narrow. A comprehensive understanding of the method's versatility could be gained by testing it across a wider range of tasks.\n2)The lack of in-depth ablation studies on each stage of R3 prompting prevents a clear understanding of their individual contributions to the overall performance.\n3)The paper lacks complex examples and a detailed error analysis, which could offer a more nuanced understanding of the method's capabilities and limitations.\n4)There is a conspicuous absence of comparison with related works, particularly those involving interactive prompting.\n5)The document could provide more information on how the R3 prompting strategy might be adapted or modified for different types of problems or different models.\n6)The authors could also explore more about how this strategy could be incorporated into existing LLM training strategies to further enhance performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This research paper addresses the problem of reasoning performance in Large Language Models (LLMs) in the presence of noisy or irrelevant information. Noise can significantly degrade the performance of an LLM, leading to suboptimal responses. Current LLMs often fail to effectively extract essential information from the context, especially when the context is polluted with irrelevant or misleading information.\nTo tackle this problem, the authors propose a novel method known as R3 (Review, Rephrase, Resolve) prompting. This methodology enhances the reasoning ability of LLMs, especially in noisy contexts. The authors leverage the Chain-of-Thought (CoT) reasoning approach to implement the R3 prompting strategy.\nThe main contributions of this paper are as follows:\n1)Proposes a novel method (R3 prompting) for improving the reasoning performance of LLMs in noisy contexts.\n2)Demonstrates the application of the R3 prompting strategy across various problem types through several examples.\n3)Shows the effectiveness of the R3 prompting strategy by conducting experiments and comparing the results with other methods. The authors provide empirical evidence that R3 prompting significantly outperforms other methods on various benchmarks.\n4)Broadens the understanding of how to improve the performance of LLMs in the presence of noise, providing a new perspective on LLM reasoning.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a three-stage interactive prompting approach to ensure that LLMs perform well in reasoning tasks in noisy contexts. The proposed method is simple and effective as shown in the experiments on several datasets. The contributions are somewhat slim though, the paper can benefit from having the evaluation across more tasks and having more error analyses.
Decision: Accept-Findings