Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
Authors: Huaixiu Steven Zheng
Swaroop Mishra
Xinyun Chen
Heng-Tze Cheng
Ed H. Chi
Quoc V Le
Denny Zhou
Venue: ICLR 2024 poster (2024)
Keywords: Prompting
Large Language Models
Reasoning
Abstraction
Primary area: generative models
TL;DR: We show that using a simple prompting technique called Step-Back Prompting, LLMs are capable of doing abstractions to derive high-level concepts and first principles from specific examples which helps them in solving complex tasks.

Abstract: We present STEP-BACK PROMPTING, a simple prompting technique that enables LLMs to do abstractions to derive high-level concepts and first principles from instances containing specific details. Using the concepts and principles to guide reasoning, LLMs significantly improve their abilities in following a correct reasoning path towards the solution. We conduct experiments of STEP-BACK PROMPTING with PaLM-2L, GPT-4 and Llama2-70B models, and observe substantial performance gains on various challenging reasoning-intensive tasks including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, STEP-BACK PROMPTING improves PaLM-2L performance on MMLU (Physics and Chemistry) by 7% and 11% respectively, TimeQA by 27%, and MuSiQue by 7%.

Published as a conference paper at ICLR 2024
TAKE A STEP BACK: EVOKING REASONING VIA AB- STRACTION IN LARGE LANGUAGE MODELS
Huaixiu Steven Zheng'
Swaroop Mishra
Xinyun Chen Heng-Tze Cheng
Ed H. Chi Quoe Le
Denny Zhou
Google DeepMind
ABSTRACT
We present STEP-BACK PROMPTING, a simple prompting technique that enables L.MSs do abstractions to derive high-level concepts and first principles from instances containing specific details. Using the concepts and principles to guide rea- soning, LLMs significantly improve their abilities following correct reasoning path towards the solution. We conduct experiments of STEP-BACK PROMPTING with PaLM-2L, GPT-4 and Llama2-70B models, and observe substantial perfor- mance gains on various challenging reasoning-intensivy tasks including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, STEP- BACK PROMPT- ING improves PaL.M2LL performance MMLU (Physies and Chemistry) by 7% and 11% respectively, TimeQA by 27%, and MuSiQue by 7%.
The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise. Edsger W. Dijkstra
INTRODUCTION
The field of natural language processing (NLP) is witnessing ground-breaking revolution because of the Transformer-bor
(Vaswani et al. 2017) large language models (LLMs) (Devlin et al. 2018
Raffel 2020 Brown al. 2020 |020]Annl Anil et al 2023). Scaling up the model size and pre-training
corpus (Hoffmann et al. 2002 C Chowdhery al.) 2022) has brought remarkable improvement in model
capabilities and sample efficiency with insights from scaling law (Kaplan al. 2020 Hoffmann
al. 2022), well emergent abilities (Wei et al. 2022a) such as multi-ster reasoning (Wei et al.
2022b
Zhou et al. 2022) and instruction following Mishra et al. 2022b Wei et al. 2021).
GPT-4 PaLM-2L PaLM-2L CoT PaLM-2L Step- Back Prompling
1.0
0.8
0.6
1:m1
0.2
MMLU Physics
MMLU Chemistry
TimeQA
SituatedCA
MuSiQue
StrategyQA
Figure 1: Strong Performance of STEP-BACE BACK PROMPTING: proposed Abstraction-and -aaadnnnnnnnng scheme leads to a substantial improvement in a wide range of challenging tasks in STEM, Knowledge QA and Multi-Hop Reasoning requiring complex (often multi-hop) reasoning.
"qqual Contribution

---

Published as a conference paper at ICLR 2024
Despite the great advancements, complex multi-step reasoning remains challenging for even the state- oh- LLMs. Lightman et al. (2023) show that
with step-by-step verification is promising remedy to improve the correctness of intermediate reasoning steps. Techniques such as Chain-of-Thooulle (Wei et al.) 2022b) were introduced to produce a coherent series of intermediate reasoning steps to increase the success rate of following the right decoding path. Inspired by the fact that when faced with challenging tasks humans often step back and do abstractions to arrive at high-level principles guide the process, we propose STEP-BACK PROMPTING ground reasoning on abstractions to reduce the chance of making emrors in the intermediate reasoning steps.
Chainstl Thoushh Thought
Step-Back Prompting
temperatur increased a factor of and the volume is increased by a factor of 8, we can rewnts the idcal gas low as: 2P (BV) = nR(2T), where P represents the new pressure.
Stsphack Queation
Stepbook Answer
What are the physics principles behind this question?
Idoal gas law: Pl nRT, whore P is the pressune, the volume. n the number of makes, R the ges constant, and T the tamparature.
Simpitying equation, get 16PV= 2nRT pr (2nRT)/(16V)
Step 2: Reasoning
Fiui Answer
Step Abstraction
Apply Ideal ges
pressure.
Since nu R, and are conslants, we cen wrile this as: Pt (1/8)* (RRTV)
the lamperature increased by factor 2, than T becomes 2T. it the vallume is increased by a factor of then V bacomes SV.
Originnl Cuestion
Sinoa NRTV the original pressure we (11)) This pressure Pi of original pressure
What happerss the pressure, P. of ideal gas the temperature increased factor 2 and the increased by fackor
Subsiiuting these values into the ideal law, we gel: P(BV) mR(2T)
Original Arsonst
Divddigg sides BV, we get nR(2T)
the tomporature is incroased by a factor 2 and the volume is incressed by factor of then the pressure will decrease by a factor of 18.
We can see that the pressure has decreased by a factor 4.
Chair-of: Chais-af-Thought
Step-Back Prompting
... After that, she pursued graduate studies at the Univarsity Caltamia, Bankely, eaming her masterss degree in 1950 and her Ph.D. in 1855.
SSipparkk Duestion
Stopback Answer
Estella Leopold's
Botany, University of Wisconsin, Madison, 1948 Botany, University Caltomia, Borkelay, 1950
educsaionn history?
Considering this timelie, is highly likely that Legpold was allending University Califomia, Berkeley, between August 1954 and Novembar 1854...
Ph.D. Bolany, University. 1955
Step 1: Abstraction
Step 2: Reasoning
Originall Question
Final Answer Sho was enrolled in the Ph.D. program in Botany at Yole from 1951 to 1955. Therefore, Estells Lecpold was most Ioaly artenccing Yalo Universily betweon August 1954 and November 1954.
Original Answer
Estellss Leopold went to which school between Aug 1954 and Nov 19647
Eslells Lespold allended University Wisconsin Madison between August 1954 and Novembar 1854
X
Figure 2: Illustration of STEP-BACK PROMPTING with two steps of Abstraction and Reasoning
guided by concepts and principles. Top: an example of MMLU high-school physics (Hendrycks et al. 2020) where the first principle of Ideal Gas Law is retrieved via abstraction. Bottom: an example from TimeQA (Chen et al.] (02T) where the high-level concept of education history is result of the abstraction. Left: PaL.M21 22 (Anil al. 2023) fails to answer the original question. Chain-of-Thouuht Thought prompting (Wei et al. 2022b; Kojima et al. (022) ran into errors during intermediate reasoning steps (highlighted as red). Right: M1L (Anil al. 2023) successfully answers the question via
STEP-BACK PROMPTING.
Among many of the cognitive skills, abstraction (Lachmy et al. 2022 is ubiquitous to humans' ability to process vast amounts of information and derive general principles. For example, Kepler compressed thousands of measurements into Kepler's three laws of planetary motion, which precisely describe the orbits of planets around the Sun (Russell 1964). In critical decision-making humans find abstraction to be helpful since provides a broader view of the environment. This work explores how I.I.Ms can tackle complex tasks involving many low-level details through a two-step process of iesarraation-and-rraaoo learning prompting them to derive high-level abstractions such concepts and principles for a specific example. The second step is to leverage the reasoning ability to reason on top of the high-level concepts and principles. We use few-shot exemplar demonstrations execute STEP-BACE PRoMPTING on LLMs.
The first step is to show L.LMS how to step back through in-context
We experiment across range of tasks involving domain specific reasoning such as Physics and Chem- istry, snowledge-intensivee question answering requiring factual knowledge, multi-hop commonsense reasoning. We observe significant performance improvements (up 27%) in PaL.- (Anil et al.
2

---

Published as a conference paper at ICLR 2024
2023) demonstrating the efficacy of STEP-BACK PROMPTING in tackling complex tasks, which otherwise challenging due to the amount of details needed for reasoning. Figure showws a summary of all the key results presented in this paper. Some the tasks very challenging: both P and GPT-4 achieve only- ~ 40% accuracy on TimeQA and MuSiQue. Chain-of-Thooue to minor improvement on few tasks, while STEP-BACK PromptinG improves the performance of PaLMM2L across the board: 7% and 11% on MMLU Physics and Chemistry, 27% on TimeQA, and 7% on MuSiQue.
prompting leads
We conduct a variety of analyses and find that STEP-BACK PROMPTING leads to strong performance improvements (up to 36%) over chain-or thought (CoT) prompting (Wei et al. 2022b) and "take-a- deep-breath" breath" (TDB) prompting (ang et al. 2023). We perform qualitative evaluation where we find that Step-Back fixes a large portion of errors of the hase model (up to ~ 40%) while introducing small portion of new errors (max ~ 12%). We also conduct an error analysis and find that majority of the errors made by STEP-BACK PROMPTING is attributed to the intrinsic limitations of reasoning capabilities of LLMs while abstraction skills are relatively easy to demonstrate to LLMs, pointing out the direction for future improvements of methods alike STEP-BACK PROMPTING.
STEP-BACK PROMPTING
STEP- BACK PROMPTING is motivated by the observation that many tasks contain a lot of details, and it is hard for LLMs retrieve relevant facts to tackle the task. As shown in the first example (top) in Figure 2 for a Physics question of "What happens to the pressure, P. of an ideal gas if the temperature increased by a factor 2 and the volume is increased by a factor of 8 ?", the LLM can deviate from the first principle of Ideal Gas Law when reasoning directly on the question. Similarly, question of "Estella Leopold went to which school between Aug 1954 and Nov 1954?" very hard to address directly given the detailed time range constraint. In both cases, asking step-back question helps the model to solve the problem effectively.
We define a step-back question as derived question from the original question at a higher level of abstraction. For instance, instead of directly asking "which school Estella Leopold went to during a specific period", step-back question (Figure bottom) would ask about the "education history", which is a high-level concept encompasses the original question. Answering the step-back question of "Estella Leopold's education history" in this case will provide all the necessary information to reason about "which school Estella Leopold went to during a specific period". The premise is that the step-back question is typically much easier. Grounding the reasoning on top of such abstractions helps to avoid reasoning errors in the intermediate steps such as the example shown in Figure 2 (left) from Chain-of- -hhouhhh. In short, STEP- BACK PROMPTING consists two simple steps:
Abstraction: Instead of addressing the question directly, we first prompt the LLM to ask generic step-back question about a higher-level concept or principle, and retrieve relevant facts about the high-level concept or principle. The step-back question is unique for each task in order to retrieve the most relevant facts.
Reasoning: Grounded on the facts regarding the high-level concept or principle, the LLM can reason about the solution to the original question. term this as Abstraction grouudded Reasoning.
In the following sections, we present empirical study of -TAC-BACK PROMPTING on range of challenging tasks covering STEM, Knowledge QA, and Multi-Hop Reasoning involving complex reasoning.
3 EXPERIMENTAL SETUP
Here we define the tasks and models we experiment with. We also describe our evaluation metric and the baselines we consider.
3.1 TASKS
We experiment with the following diverse tasks: (a) STEM, (b) Knowledge QA, and (c) Multi-Hop Reasoning. We describe below the datasets consider (see Appendix B for more details).
3

---

Published as a conference paper at ICLR 2024
We evaluate MMLU and GSM8K for STEM tasks. MMLU (Hendrycks et al. 2020 contains series of benchmarks across diverse domains to evaluate the model's 's language understanding. We consider the high school physics and chemistry portions of MMLU because of the deep reasoning involved.
STEM:
Knowledge QA: We consider TimeQA (Chen et al., 2021) since it contains complex queries that require challenging time-sensitive knowledge. We also experiment with Situat- edQA (Zhang & Choi, 30 another challenging open-retrieval QA dataset requiring the model to answer questions given temporal geographical contexts.
Multi-Hop Reasoning: We experiment with MuSiQue (Trivedi et al. 2022), a hard multi- hop reasoning dataset created via composable pairs of single-hop questions, and StrategyQA (Geaa al. 202I with open-domain questions that demand some strategy solve.
3.2 MODELS
We use the following state-of-the-art LLMs: instruction-tuned PaLM-21.. (Anil et al. 2023 GPT- 4 (OpenAl 2023). and Llama2-70B (Touvron al. 2023)
3.3 EVALUATION
Conventional evaluation metrics such accuracy, F1 score have limitations specifically for evaluating the generations of state-of-the-art LLMs since these models often generate long-form answers which are hard to capture. We instead conduct evaluation using the PaLM-2L model where we few-shot prompt the model to identify equivalence between target answers and the model predictions. Few-shot examples, prompts and other details used for this evaluation are in Appendix
3.4 BASELINE METHODS
PaLM-2L, PaLM-2L 1-shot: PaLM-2L is either queried directly with the question or has single demonstration exemplar of question-answer included in the prompt.
PaLM-2L COT, PaLM-2L CoT 1-shot: PaLM-2L model is queried with zero-shot CoT prompting (Kojima et al. 2022): "Ler's 's think step by step" is appended to the question. For 1-shot, One demonstration example of question and answer pair is provided in the prompt, where the answer is the style CoT (Wei et al. 20226).
PaLM-22 + TDB: Zero-shot prompting with "Take a deep breath and work on this problem step-by-step." " (Yang al., 2023) prepended to the question.
PaLM-2L RAG: For Sections and 6 we use retrieval-augmented generation (RAG)
where the retrieved passage is used as context by the LLM.
GPT-4 and Llama2-70B: run GPT-4 and Llama2-70BB MMLU tasks for all methods. In addition, we also run GPT-4 on all baselines for all tasks.
We do not use RAG for STEM tasks, because of the inherent reasoning nature of the tasks contrary to the other fact-seeking datasets. All inferences are done using greedy decoding.
STEM
We evaluate STEP-BACH BACK PROMPTING on STEM tasks (Hendrycks al. 2020 gauge the efficacy of our method on reasoning in highly specialized domains. We explain below our experimental setup, result, and analysis of applying STEP-BACH PROMPTING on the MMLU high-school Physics and Chemistry, and GSM8K benchmarks.
4.1 STEP-BACK PROMPTING
Questions in the MMLU benchmarks require deeper reasoning. Furthermore, they also require understanding and application of formulae which often physics and chemistry principles and concepts. In this case, we first demonstrate to the model abstraction skills in the form of concepts and first principles such Newton's 's first law of motion, Doppler effect, and Gibbs free energy etc.
4

---

Published as a conference paper at ICLR 2024
Table 1: Strong performance of STEP-BACK PROMPTING on MMLU tasks across three model
families. CoT: zero-shot Chain of Thought prompting (Kojima al. 2022). TDB: Take Deep Breath prompting (Yang et al. 2023).
Method
MMLU Physies
MMLU Chemistry
PaLM-2L
66.4% (0.8%)
70.9% (0.9%)
PaLM-2L 1-shot
64% (1.6%)
75.6% (0.4%)
PaLM-2L CoT
65% (2%)
75.3% (1.5%)
PaLM-2L + CoT 1-shot
61.5% (1.8%)
76.6% (1%)
PaLM-2L TDB
65.7% (0.7%)
73.8% (1.1%)
PaLM-2L Step-Back (ours)
73.2% % (1.9%)
81.8% (1.4%)
GPT4
69.4% (2.0%)
80.9% (0.7%)
GPT-4 1-shot
78.4% (2.4%)
80.5% (1.6%)
GPT-4 CoT
82.9% (0.5%)
85.3% (1.0%)
GPT-4 + CoT' 1-shot
79.3% (1.0%)
82.8% (0.5%
+
74.4% (4.0%)
81.5% (1.3%)
GPT-4 + Step-Back (ours)
84.5% (1.2%)
85.6% (1.4%)
Llama2-70B
51.9% (3.6%)
55.7% (2.1%)
Llama2-70B 1-shot
57.3% (1.6%)
58.5% (2.5%)
Llama2-708 + CoT
59.3% (2.0%)
64.1% (1.2%)
Llama2-70B + CoT 1-shot
59.6% (2.0%)
68.1% % (1.4%)
Llama2-70B + TDB
60.4% (2.1%)
63.6% (1.9%)
Llama2-70B + Step-Back (ours)
64.8 % (1.5%)
66.7% (1.6%)
The implicit step-back question here "what are the physics or chemistry principles and concepts involved in solving this task?". We provide demonstrations the model to recite the relevant principles for solving the task from its own knowledge (see Appendix for few-shot exemplars).
4.2 RESULTS
Table iillustrates model performance across various setups across three model families:
Accuracy
0.75
PaLM-2L. GPT-4, and Llama2-70B. Average ac- curacy over 5 evaluation runs is reported along with standard deviations (in the parentheses). PaLM-2L baseline performance is 66.4% and 70.9% on Physics and Chemistry, respectively. We find that CoT and TDB zero-shot prompt- ing do not significantly increase model perfor- mance, which could be due to the inherent dif- ficulty and deep reasoning associated with these tasks. M-1. 1-shot and PaI.M-21. CoT 1-shot do not improve against the baseline much either, highlighting the challenge demonstrat- ing the reasoning steps to the model. In con- trast, STEP-BACK PROMPTING significantly im- proves model performance: +7% and +11% compared to PaLM-2L. Similarly. with GPT-
0.74
0.73
0.72
0.71
0.70
2
3
4
Number of Shots
Figure 3: Ablation study of STEP-BACK PROMPT- ING accuracy using PaLM-2L on MMLU high- school Physics against the number of few shot exemplars: robust performance with respect to varying number of shots.
4 and Llama2-70B models, STEP-BACK PROMPTING is very competitive among all the baseline methods we tested, showing that STEP-BACE PROMPTING model-agnostic, We present the results of GSM8K in Appendix A.1
4.3 ABLATION AND ANALYSIS
Few-shot Ablation: First, in Figure 3 we observe that STEP-BACE PROMPTING is robust to the number of few-shot exemplars of (question, principles) pairs used demonstrations. Adding more
5

---

Published as a conference paper at ICLR 2024
Table Strong performance of STEP-BACK PROMPTING on Knowledge QA tasks. CoT: Chain of Thought prompting, TDB: Take a Deep Breath prompting, RAG: aetrival-auugeenee generation. STEP-BACK PROMPTING results in significant performance improvements.
Method
TimeQA
TQA Easy
TQA Hard
SituatedQA
PaLM-2L
41.5%
42.6%
40.4%
54.3% (0.3%)
PaLM-2L 1-shot
40.7%
41.7%
39.1%
51.8% (0.6%)
PaLM-2L + CoT
40.8%
41.8%
39.8%
56.4% (0.2%)
PaLM-2L + CoT 1-shot
38.15 1%
39.3%
36.8%
54% (0.8%)
PaLM-2L TDB
40.9%
42.6%
39.1%
54% (0.5%)
PaLM-2L RAG
57.4%
67.8%
46.8%
59.3% (0.4%)
PaI.M-21 Step-Back (ours)
66%
70.4%
61.6%
57.5% (0.3%)
PaLM-2L + Step-Back + RAG (ours)
68.7% %
75.2%
62.3%
61% (0.4%)
GPT-4
45.6%
48.9%
42.6%
63.2% (0.4%)
demonstration examples beyond a single example does not lead to further improvements. This indicates that the task of retrieving the relevant principles and concepts is relatively easy through in-context learning and a single demonstration suffices. Therefore, we use a single exemplar for few-shot prompting throughout the paper except the ablation studies.
Error Analysis: Comparing the predictions of STEP-BACK PROMPTING to the baseline PaLM-2L model for MMLU high-school Physics: find that STEP-BACK PROMPTING corrects 20.5% errors from the baseline while introducing 11.9% errors.
To further understand where the errors come from in STEP-BACK PROMPTING, we annotate all the wrong predictions of STEP-BAC BACK PROMPTING in the test set, and categorize them into 5 classes (see Appendix T.T for examples in each class):
Principle Error: The error happens at the step of Abstraction, where the first principles generated by models are wrong or incomplete.
Factual Error: There is at least one factual error when the model recites its own factual knowledge
Math Error: There
least one math error in the intermediate steps when math calculations are involved in deriving the final answer.
Context Loss: There is at least one error where the model response loses context from the question, and deviates from addressing the original question
Reasoning Error: We define Reasoning Error as when the model makes least one error in the intermediate Reasoning steps before arriving at the final answer.
five types emrors are happening during the Reasoning step except Principle Error which points to the failure of the Abstraction step. As shown in Figure 4 (right), Principle Error com- prises only a small fraction model makes: more than 90% of the errors hap- pen at the Reasoning step. Among the four er- types during Reasoning, Reasoning Error and Math Error are the major error categories. This corroborates with the finding in the abla- tion study above that very few exemplars are needed to demonstrate to LLMS the Abstrac- tion skill. Reasoning step is still the bottleneck of how well STEP-BACK PROMPTING can per- form tasks such MMLU requiring complex reasoning. For MMLU Physics specifically. the Reasoning and Math skills are critical for soly- ing the problems successfully: even if the first principles are retrieved correctly. deep reasoning and math involved derive correct final answer through typical multi-step reasoning process.
0.55
0.4
the errors the
0.25
0.04
0.07
0.09
0.0
Factual Error
Math Emar
Context Loss
Reasoning Error
Principle Enror
Figure Error Analysis of STEP-BACE PROMPT- ING on MMLU high-school Physics: five classes of errors Step-Bacc Back makes with Reasoning being the dominating class.

---

Published as a conference paper at ICLR 2024
5 KNOWLEDGE QA
We evaluate STEP-BACK PROMPTING on question-answering benchmarks requiring intensive factual knowledge. Knowledge QA has been challenging for LLMs. In this section, we first describe the experimental setup, followed by results and analysis on STEP-BACK PROMPTING.
5.1 STEP-BACK PROMPTING
We evaluate STEP-BACK PROMPTING on TimeQA (Chen al. 2021) and SituatedQA (Zhang Choi 2021) in the Knowledge QA category. We first show the LLMs how to do Abstraction through in-context demonstrations. The step-back question "What was Estella Leopold's s education history" in Figure 2 is generated by the LLM through few-shot demonstrations (see Appendix D.2 for details). Given the knowledge-intenses combination with intensive nature of these queries, we use retrieval augmentation (RAG) in PROMPTING. The step-back question is used to retrieve relevant facts, which work as additional context (see Table 14 for the prompt) to ground the final reasoning step.
5.2 RESULTS
We evaluate the models on the set of TimeQA. As shown in Table 2 the baseline models of GPT-4 and PaLM-21. achieved 45.6% and 41.5%, highlighting the difficulty of the task. Applying either CoT or TDB zero-shot (and one-shot) prompting to the baseline model shows no improvement. In contrast, augmenting the baseline model by regular retrieval augmentation (RAG) improves the accuracy to 57.4%, highlighting the fact-intensive nature the task. The result of Step-Back + RAG shows the effectiveness of going back high-level concept, which enables much more reliable retrieval augmentation: the accuracy on TimeQA achieves a remarkable 68.7%.
Next, segment TimeQA into the Easy and Hard difficulty levels provided in the original dataset. As expected, all methods perform worse on the Hard subset. While RAG can improve the Easy accuracy from 42.6% 67.5 67.8%, the improvement much smaller the Hard accuracy: 40.4% 46.8%. This where STEP-BACE PRoMPTING shines by retrieving facts regarding high-level concepts to ground the final reasoning: Step-Back + RAG further improves the Hard accuracy 62.3%, outperforming GPT-4's 42.6% from GPT-4, We hypothesize that facts regarding the high-level concepts (such as education history) are much more accessible than the low-level details.
On the SituatedQA benchmark, we observe moderate quality gain from 54.3% to our best method of Step-Back RAG (61%) with a small gap GPT-4's 63.2%. Similar to TimeQA, prompting techniques such as CoT and TDB don't 't help significantly for SituatedQA.
Accuracy
0.6
Hard
0.52
0.80
0.45
0.75
0.70
0.2
0.55
2
4
0.02
0.01
Number of Shots
Reasoning Error Scoring Eror
RAG
StepBack
Figure 5: Ablation and enror analysis of STEP-BACK PROMPTING TimeQA. Left: ablation against the number few-shot exemplars. Right: four classes of errors Step- Back makes with Reasoning and RAG being the dominant enror sources.
5.3 ABLATION AND ANALYSIS
We observe in Figure (left) that the performance of STEP-BACE PROMPTING on TimeQA robust to the number of exemplars used in demonstration. highlighting again the sample efficiency of in-context learning Abstraction skills for models like PaLM-
Few-shot Ablation:

---

Published as a conference paper at ICLR 2024
Table 3: Results of STEP-BACK PROMPTING on Multi-Hop Reasoning. CoT: Chain of Thought prompting, TDB: Take Deep Breath prompting, RAG: retrieval augmentation generation. The average accuracy over evaluation runs with the standard deviations included in the parentheses.
Method
MuSiQue
StrategyQA
PaLM-2L
35.5% (3%)
82.8% (0.7%)
PaLM-2L 1-shot
29.0% (0.5%
76.6% (0.5%)
PaLM-2L CoT
38.7% (3.2%)
83.6% (0.4%)
PaLM-2L + CoT 1-shot
38.5% (2.2%)
76.8% (1.4%)
PaLM-2L TDB
39.0% (2.3%)
82.7% (0.9%)
PaLM-2L + RAG
39.6% (2.8%)
84.2% (0.5%)
PaLM-2L Step-Back (ours)
42.6% (3.1%)
82.7% (0.4%)
PaLM-2L + Step-Back RAG (ours)
42.8% (2.0%)
86.4% % (1%)
GPT-4
38.5% (0.2%)
78.3% (1.1%)
Error Analysis: Figure 5 (right) shows the breakdown of all the remaining errors made by STEP-
BACK PROMPTING on TimeQA. Similar to Section 4.3 we categorize the errors into
StepBack: The step-back question generated is not helpful in solving the task.
RAG: RAG fails to retrieve relevant information despite that the step-back question is on target.
Scoring Error: The evaluation by the judge model made mistake.
Reasoning Error: The retrieved context is relevant, but the model still fails to reason through the context arrive at the right answer.
We find that the StepBack rarely fails. In contrast, we find more than half of the errors are due to reasoning errors. Additionally, 45% of errors due failure in retrieving the right information despite that Abstraction provided by step-back makes a much easier task. This reflects the difficulty level of the TimeQA task. Additional error analysis of TimeQA is in Appendix A
MULTI-HOP REASONING
We evaluate STEP-BACK PROMPTING on challenging Multi-Hop reasoning benchmark MuSiQue (Trivedi al. 2022) and StrategyQA (Geva et al. 2021). We follow the same protocol as Section to implement STEP- BACK PROMPTING.
Table shows performance of various baselines on the dev set of MuSiQue and Strategy QA. Baseline performance of PaI.M-2I. and GPT-4 are low (35.5% and 38.5% for Pal.M-2L. and GPT-4 respectively) in MuSiQue since it is a hard multihop reasoning benchmark. In contrast, StrategyQA has stronger baselines (82.8% and 78.3% for PaLM-2L and GPT-4 respectively) probably because it is binary classification task. CoT and TDB improve model performance bit in the case of MuSiQue (~ 3% and 3.5% respectively) which can be attributed to the inherent reasoning nature of this task where these methods are shown be helpful. In the case of StrategyQA, there is no significant performance gain with CoT and TDB which could be due to the high baseline performance in this task, with limited scope for these prompting methods to improve performance. Often, 1-shot performance is significantly lower than their zero-shot methods, which could be attributed to potential example bias (Zhao al. 202T Parmar al.) 2023). RAG improves model performance (~~ 4% and 2% for MuSiQue and Strategy respectively.) STEP-BACK PROMPTING with the power of abstraction produces the best performance of all methods: 42.8% in MuSiQue and 86.4% in QA. significantly outperforming GPT-4 on both tasks. We present detailed error analysis on StrategyQA in Appendix A.3
7 DISCUSSION
Abstraction helps humans to solve complex tasks by removing irrelevant details and distilling high- level concepts and principles to guide the problem-solving process. STEP-BACK PROMPTING breaks
8

---

Published as a conference paper at ICLR 2024
complex tasks such as knowledge-intensiee QA, multi-hop reasoning, and science questions into two separate steps of Abstraction and Reasoning. We demonstrate through empirical experiments that Abstraction is easy skill for the LLMs such as PaLM-2L via sample-efficient in-context learning. Grounding on the high-level concepts and principles, L.MSs can leverage their intrinsic Reasoning capabilities to derive the solution. This reduces the chance of reasoning failures in the intermediate steps and is shown to improve the performance on a wide range of complex reasoning tasks. Despite the success, through error analysis, we find that Reasoning is still one of the hardest skills for LLMs to acquire: it is still the dominant failure mode even after the large reduction of task complexity by STEP-BACK PROMPTING.
Nevertheless, Abstraction is neither necessary nor possible in all scenarios. For instance, the task can be as simple as who was the president of the United States in 2000?, in which case there is no such need to step back and ask high-level question as the answer to such questions is readily available. Questions such as what is the speed of light? point to the first principles themselves. Doing Abstraction in this case would not make difference either.
RELATED WORK
8.1 PROMPTING
Few-shot prompting (Brown et al. 2020 Liu et al. 2023; Mishra et al. 2022a; Wei et al. 2022b) has significantly improved model performance across range of tasks without requiring updating any model parameters. Our work STEP-
PROMPTING is in the same category as the oh thought prompting (Wei et al. 2022b) and scratchpad (Nye et al.) 2021) owing to its simplicity and generic nature. But our approach is focused on the key idea of abstraction which is inspired from the fact that taking a step back often helps humans in performing complex tasks. Our work is also related to the recitation-augmeett explicitly perform step-back and abstraction, with optional use of retrieval augmentation depending on the nature of the task at hand.
language models (Sun et al. 2002);; however in contrast to their work, we
8.2 DECOMPOSITION
Decomposing a task into simpler tasks and solving these tasks to complete the original task has been an effective way (Zhou al. 2022 Patel et al. 2022 Khot et al. 2022] Press et al. 2022) to improve model performance on complex tasks. Several prompting methods have been successful in this regard. Our work STEP-BACK PROMPTING, in contrast, is on making the question more abstract and high-level, which is different from decomposition that is often low-level breakdowns of the original question. For instance, generic question for which employer did Steve Jobs work for in 1990? could be what is the employment history of Steve Jobs? While decomposition would lead to sub-questions such as What was Steve Jobs doing in 1990?, Was Steve Jobs employed in 1990? and If Steve Jobs was employed, who was his employer? Furthermore, abstract questions such as what is the employment history of Steve Jobs? are often generic in nature to have a many-to-one mapping since many questions (e.g. which employer did Steve Jobs work for in 1990? and which employer did Steve Jobs work for in 2000?) can have the same abstract question. This is in contrast to decomposition where there is often one-to one-to-many mapping since there are multiple decomposed sub aub-problems necessary to solve a given question.
CONCLUSION
We introduce STEP-BACK PROMPTING as a simple yet generic method to elicit deep reasoning via abstraction in large language models. Experimentation on LLMs across fact-seeking, commonsense reasoning and domain-speciffe reasoning benchmarks shows that STEP-BACK PROMPTING signifi- cantly improves model performance. We hypothesize that abstraction helps models to hallucinate less and reason better, probably reflecting the true nature of the model which are often hidden while responding to the original question without abstraction. We hope our work will inspire more human-inspired approaches to elicit the hidden potential of large language models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank Andrew Dai, Adams Yu and Hannah Rashkin of Google DeepMind for their insightful feedback on this paper.
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arXiv preprint arXiv:2330. 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
Wenhu Chen, Xinyi Wang, and William Yang Wang. dataset for answering time-sensitive questions. arXiv preprint arXiv:2108.06111 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204002311, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXtv:1810.04805. 2018.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot. Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.0883 2020.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406. 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213, 2022.
Royi Lachmy, Valentina Pyatkin, Avshalom Manevich, and Reut Tsarfaty. Draw me a flower: Processing and grounding abstraction in natural language. Transactions of the Association for Computational Linguistics, 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050. 2023.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
10

---

Published as a conference paper at ICLR 2024
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to gptk's language. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 5899612, 2022a.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470-3487 2022b.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint urXiv:21/2.00111 2021.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Mihir Parmar, Swaroop Mishra, Mor Geva, and Chitta Baral. Don't blame the annotator: Bias already starts in the annotation instructions. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1771-1781, 2023.
Pruthvi Patel, Swaroop Mishra, Mihir Parmar, and Chitta Baral. Is a question decomposition unit all we need? In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp.
2022.
Ofrr Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint 10X0v:2210.03350, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (11):5485-5555 2020.
John L Russell. Kepler's ' laws of planetary motion: 1609-1666. The British journal for the history of science, 2(1):1-24, 1964.
Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmenttè language models. arXiv preprint arXh::2210.01266 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi. Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554 2022.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Iliia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint 1XXiv2200.01160 2021.
11

---

Published as a conference paper at ICLR 2024
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities large language models. arXiv preprint arXhv:2206.07666 2022a.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24833 2022b.
Chengrun Yang. Xuezhi Wang. Yifeng Lu, Hanxiao Liu, Quoc Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.0344 03409, 2023.
Michael Zhang and Eunsol Choi. Situatedqa: Incorporating extra-linguistic contexts into qa. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp- 7371-7387, 2021.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pp. 12697-12706. PMLR, 2021.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 10625, 2022.
12

---

Published as a conference paper at ICLR 2024
Table 4: Step-Batt Back Prompting on GSM8K. COT: zero-shot Chain of Thought prompting (Kojima
et al. 2022). TDB: Take Deep Breath prompting YYngg et al. 2023), DP: Decomposed Prompting (Khot et 2022). The Table reports the average accuracy over 5 evaluation runs, with standard
deviations in the parentheses.
Method
GSM8K
PaLM-2L
75.8% (0.2%)
PaLM-2L 1-shot
84.5% (0.4%)
PaLM-2L + CoT
84.4% (0.2%)
PaLM-2L + CoT 1-shot
81% (0.2%)
PaLM-2L + TDB
82.2% (0.2%)
PaLM-2L + DP
82.2% (0.08%)
PaI.M-21. + Step-Back Back (ours)
84.3% (0.2%)
A GSM8K RESULTS, AND ERROR ANALYSIS
A.1 GSM8K RESULTS
We present in Table TableItth the results of STEP-BACK PROMPTING GSM8K along with other strong baselines from PaLM-2L runs. We observe that STEP-BACK PROMPTING achieved competitive performance together with zero-shot CoT and 1-shot standard prompting. We hypothesize that the simplicity of principles (e.g. addition, subtraction, etc.) in GSM8K makes it not absolutely necessary retrieve the principles first before reasoning. Nonetheless, PROMPTING the most competitive among all the prompting methods we tested, including the "Take Deep Breath" prompting optimized for GSM8K in Yang al. (2023) and Decomposed Prompting in Khot et al. (2022).
still find that STEP-BACE
A.2 TIMEQA ERROR ANALYSIS
We conduct error analysis understand where STEP- BACC PROMPTING fixes the errors the baseline models make. Figure shows that compared to the predictions of baseline PaLM-2L, STEK-BACK PROMPTING can fix 39.9 39.9% of the predictions where the baseline prediction is wrong, while causing 5.6% errors.Furtie Furthermrrr, introduced by STEP-BACE PROMPTING to RAG is still relatively low (6.3%). Together, this shows that the STEP-BACK PROMPTING is helpful most of the time, signifying the need and effectiveness of doing Abstraction before directly addressing the original question.
hep-Back- + RAG fixes 21.6% eтrors coming from RAG. The % of errors
Step Back RAG Wrong
Step Back RAG Wrong
5.8%
Wrong
6.3%
Both Wrong
5%
RAG Whoang
25 8%
21. 8%%
Beseline Wronn
Both Right
Both Right 48 2%
97.
Figure 6: Error Analysis of Step-Back Prompting on TimeQA. Left: Step-Back + RAG vs Baseline predictions. Right: Step-Back RAG vs RAG predictions. Step-Back RAG can fix 39.9% of the predictions where the baseline prediction wrong while causing 5.6% errors. Furthermore. Step-Back RAG fixes 21.6% errors coming from RAG. The % of errors introduced by STEP-BACK PROMPTING to RAG is still relatively low (6.3%).
13

---

Published as a conference paper at ICLR 2024
Table 5: Stats of the evaluation datasets used in this paper.
Domain
Dataset
Split
Number of Examples
STEM
MMLU high-school Physics
Test
151
MMLU high-school Chemistry
Test
203
GSM8K
Test
1319
Knowledge QA
TimeQA
Test
5226
TimeQA Easy
Test
2613
TimeQA Hard
Test
2613
SituatedQA
Test
2901
Multi-hop Reasoning
MuSiQue StrategyQA
Dev
2417
Dev
229
A.3 STRATEGYQA ERROR ANALYSIS
Figure shows the analysis of StrategyQA the predictions of Step-Back RAG against the baseline model and the raw retrieval augmentation variant of Pal MM-21. Compared to the baseline, Step-Back RAG can 15.4% wrong predictions correct predictions, while leading 6.1% eπrors the other way around. Furthermore, Step-Back RAG fixes 12.7% 7%% errors coming from RAG. The errors introduced RAG by Step-Back are just 4.4%.
B DATASET DETAILS
Table shows the split and number of examples used for evaluations TimeQA, StrategyQA, MMLU, and GSM8K.
C EVALUATION DETAILS
C.I FEW-SHOT EXAMPLES FOR EVALUATION WITH PALM-2L
Given the model free-form outputs and the target label, we use one positive and one negative output as few-shot examples to demonstrate to the scoring model how to score the output. Table iillustrates the prompt we used for the scoring model. We parse out the "Yes" or "No" answer from the scoring model output as TRUE FALSE score of the model output.
Bath Wrong
Both Wrong
3. 3.9%
5.7%
Baseline Whong
RAG Wrong
15.4%
12.7%
Step-Back + RAG Wrong 6.1%
StepkBack Wrong
4.4%
Both Right
Both Right
74.6%
Figure 7: Error Analysis of Step-Back Prompting StrategyQA Left: Step-Back + RAG Baseline predictions. Right: Step-Back RAG vs RAG predictions. Step-Back + RAG able to turn 15.4% wrong predictions into correct predictions, while leading 6.1% errors the other way around. Furthermore, Step-Back RAG fixes 12.7% 7% emrors coming from RAG. The errors introduced to RAG by Step-Back just 4.4%.
14

---

Published as a conference paper at ICLR 2024
Are the following two answers to the given question equivalent? Do not consider whether the answers right wrong, but only whether they are equivalent. Directly state "Yes" or "No".
Question: Which title conferred to Anna Muzychuk in 2007? Answer 1: Anna Muzychuk was conferred the title of International Master (IM) in 2007. She earned the title by scoring three norms in rapid chess tournaments.
Answer 2: International Master
Answer (short): International Master Answer (short): International Master Are the two answers equivalent? Yes Question: What state Seattle located in?
Answer 1: Seattle is in Washington State.
Answer 2: The answer is George Washington.
Answer (short): Washington State
Answer (short): George Washington
Are the two answers equivalent? No
Question: < Question
Answer 1:
Model Output>>
Answer 2:
Target Label.>
Table 6: Illustration of few shot evaluation with the PaLM-2L model.
MMLU Physics/Chemistry First-Principle Prompt
You are an expert Physies/Chemistrr. You are given Physics/Chemistry problem. Your task is to extract the Physics/Chemistry concepts and principles involved in solving the problem. Here are few examples:
Question: <Question Example
Principles Involved: <Principles Examplel>
... Question: <Question Example5>
Principles Involved: <Principles Example5>
Question: <Question>>
Principles Involved:
Table 7: Prompt of extracting the underlying principles involved in MMLU physics and chemistry questions.
C.2 HYPER PARAMETERS FOR EVALUATION WITH PALM-2L
We use PaLM-2L the scoring model for evaluation. We experiment with different sampling temperatures, and find that T gives us a highly-accurate evaluation. For example, we sampled 100 test examples and the model predictions, and manually rated the correctness of the model scoring. We found that out of 4 trials, the model scoring agrees with human ratings 97%, 98%, 99% and 99% of the time.
D PROMPTS AND FEW SHOT EXAMPLES
D. STEM
For MMLU high-school Physics and Chemistry, we first prompt the model to generate the first principles behind the question. Using the generated first principles, further prompt the model to generate the final answer through few-shot demonstrations The prompt generating first principles is shown in Table for MMLJU high-school Physics and Chemistry.
15

---

Published as a conference paper at ICLR 2024
MMLU Physics/Chemistry Final Answer Prompt
You are an expert at Physics/Chemistrr. Physics/Chemistry problem and set of principles involved in solving the problem. Solve the problem step by step by following the principles. Here are few examples:
You are given
Question: <Question Examplel>
Principles: <Principles Examplel>
Answer: <Answer Example
Question: <Question Example5>
Principles: <Prrnncipess Example5>
Answer: <Answer Example5>
Question: <Question>>
Principles: <Principles>
Answer:
Table 8: Prompt of querying the model for final answer with first principles behind the question in MMLU high-school Physics and Chemistry.
After extracting the first principles of solving a particular question, formulate the prompt in Table to query the model for the final answer.
Tables TO show demonstration exemplar of Question-Principles--nssse triplets for MMLU high-school Physics and Chemistry, respectively. For GSM8K, given the simplicity of the principles, we directly combine the principles and the solution in the demonstration exemplar. Table II shows the exemplar we used in the paper.
D.2 KNOWLEDGE QA
We use the following prompting in [2ble|12ju demonstrate to the LLM on asking step-back question for TimeQA and SituatedQA including up exemplar demonstrations of pairs of Original Question and Step-back Question.
Table 13 shows 5 exemplars from the Train split of TimeQA and SituatedQA demonstrations of asking step-back questions.
The step-back question is extracted from the model output using the prompt. Using the step-back question, we do retrieval augmentation. Using both the retrieval augmentations from the original question and the step-back question, formulate the final prompt query the model for the final answer, as shown in Table [4
D.3 MULTI-HOP REASONING
For Multi-Hop Reasoning, we use the same prompting template as in Knowledge QA to ask the step-back question, and query for the final answer given the retrieval augmentations. Table shows 5 demonstration exemplars for asking step-back questions from the Train split of MuSiQue and Str.tegyQA.
shows
D.4 BASELINE PROMPTS
For standard zero-shot and few-shot prompting of the baseline model, formulate the prompt using the template in Table T6 with up to exemplars.
D.5 CHAIN OF THOUGHT (COT)
For zero-shot CoT prompting, we simply append Let's think step by step. to the question to query the model.
16

---

Published as a conference paper at ICLR 2024
Question
A spherical conductor carries a net charge. How is this charge distributed on the sphere?
0) The charge is evenly distributed on the surface.
The charge resides on the surface only; the distribution of charge on the surface depends on what other charged objects are near the sphere.
2) The charge moves continually within the sphere.
3) The charge is distributed uniformly throughout the sphere.
Principles
Coulomb's Law: the force between two charged particles is proportional to the product of their charges and inversely proportional the square of the distance between them, F= ql q2 / where F is the electric force, k is a constant, ql and q2 are the charges the particles carry, and is the distance between them.
Answer
Using the Principles of Coulomb's s Law, can solve the problem following: Step 1: Apply Coulomb's ' Law to find out how charges are distributed on the surface.
the case of a spherical conductor, the charges on the surface will repel each other. The further apart the charges are, the less force they will exert each other. Therefore, the charges will distribute themselves evenly on the surface of the sphere, this is the configuration that minimizes the repulsive force between them.
Step 2: Apply Coulomb's Law to find out what happens if there are other charges present.
The distribution of charge on the surface may also be affected by the presence other charged objects near the sphere. For example, a negatively charged object is brought near positively charged sphere, the negative charges the sphere will be repelled and will move to the opposite side of the sphere. This will result in non-uniform distribution of charge the surface of the sphere.
Therefore, the correct answer is 1) The charge resides the surface only; the distribution of charge on the surface depends on what other charged objects are near the sphere.
Table 9: Few-shot demonstration exemplar for MMLU high-school Physics.
17

---

Published as a conference paper at ICLR 2024
Question
sample of an unknown chloride compound was dissolved in water, and then titrated with excess Pb(NO3)2 to create precipitate. After drying, it is determined there are 0.0050 mol of precipitate present. What mass of chloride is present in the original sample?
0) 0.177 1) 0.355 g
2) 0.522
0.710
Principles
Precipitation reactions occur when two soluble salts are mixed and form an insoluble product, called a precipitate. The precipitate can be separated from the solution by filtration centrifugation.
Precipitation reactions:
Molar mass: The molar mass substance is the mass of one mole of that substance. The molar mass is expressed in grams per mole (g/mol).
Limiting reactant: The limiting reactant is the reactant that is completely con- sumed in chemical reaction. The amount of product formed is determined by the amount of limiting reactant.
Answer
Assuming the unknown chloride compound is MCI, where M represents the metal cation, the balanced chemical equation for the precipitation reaction is: Pb(NOa)2 + 2MCl(ug) - (s) + 2M (Oq(aq)
Since Pb(NO3)2 is in excess, MCI is the limiting reactant. The stoichiometry of the reaction indicates that 2 moles of MCI produce 1 mole of PbCl2 precipitate. Therefore, 0.0050 mol of PbCIz corresponds to 0.010 mol of MCI.
The mass of chloride in the original sample can be calculated using the molar of chloride (35.45 g/mol): 0.010 mol CI x 35.45 g/mol 0.355 Cl
The correct answer is 1) 0.355 g-
Table 10: Few-shot demonstration exemplar for MMLU high-school Chemistry.
Question
Bella bought stamps at the post office. Some of the stamps had a snowflake design, some had truck design, and some had snowflake stamps. She bought 9 more truck stamps than snowflake stamps, and fewer rose stamps than truck stamps. How many stamps did Bella buy all? Let us find and apply the math principles to solve the problem step by step:
design. Bella bought
Answer
Step 1. Addition: Calculate the number of truck stamps.
Bella bought 11 snowflake stamps. She bought 9 more truck stamps than snowflake stamps: there are 11 9 20 truck stamps.
Step 2. Subtraction: Calculate the number of rose stamps. Bella bought 13 fewer rose stamps than truck stamps: there are 20 - 13 = 7 rose stamps.
Step 3. Addition: Calculate the total number of stamps in all three colors. Bella bought 11 snowflake stamps, 20 truck stamps, 7 rose stamps: there are 11 20 + 38 stamps in total.
Conclusion: Bella bought 38 stamps in all.
Table 11: Step-Back few-shot demonstration exemplar for GSM8K.
18

---

Published as a conference paper at ICLR 2024
Knowledge QA Step-Back Prompt
You are expert world knowledge. Your task is step back and paraphrase question to more generic step-back question, which is casier to answer. Here are few examples:
Original Question: <Original Question Example1>
Stepback Question: <Stepback Question Example
Original Question: <Original Question Example5>
Stepback Question: <Stepback Question Example5>
Original Question: <Original Question>>
Stepback Question:
Table 12: Prompt of asking step-back question in Knowledge QA tasks.
dataset
Original Question
Step-back Question
TimeQA
Which position did Knox Cunningham hold from May 1955 Apr 1956?
Which positions have Knox Cunning-
ham held in his career?
TimeQA
Who was the spouse of Anna Karina from 1968 to 1974?
Who were the spouses of Anna Karina?
TimeQA
Which team did Thierry Audel play for from 2007 to 2008?
Which teams did Thierry Audel play for in his career?
TimeQA
What was the operator of GCR Class 11E from 1913 to Dec 1922?
What were the operators of GCR Class 11E in history?
TimeQA
Which country did Sokolovsko belong to from 1392 to 1525?
Which countries did Sokolovsko belong to in history?
SituatedQA
when was the last time team from canada won the stanley cup as of 2002 when did england last get to the semi final in a world cup as of 2019
which years did team from canada won the stanley cup as of 2002
SituatedQA
which years did england get to the semi final a world cup as 2019?
SituatedQA what the biggest hotel las vegas nv as of November 28, 1993
what is the size of the hotels in las vegas nv as of November 28, 1993
SituatedQA
who has scored most runs in 120
What are the runs of players in 120 matches as of 2017
matches as of 2017
SituatedQA
who is the highest paid player in the nba
what is the salary of the high paid play- ers the nba this season as 2017
this season as of 2017
Table 13: Few-shot demonstration exemplars for asking step-back questions in TimeQA and Situat- edQA.
19

---

Published as a conference paper at ICLR 2024
Knowledge QA Final-Answer Prompt
You are an expert of world knowledge. am going to ask you a question. Your response should be comprehensive and not contradicted with the following context if they are relevant. Otherwise, ignore them they are not relevant.
Passage from original retrieval augmentation> Passage from step-back retrieval augmentation>>
Original Question:
<OOiiginl Question>>
Answer:
Table 14: Prompt of querying the model for final answer with additional contexts from original and step-back retrieval augmentations TimeQA and SituatedQA
dataset
Original Question
Step-back Question
MuSiQue
at year saw the creation of the region where the county of Hertfordshire is located?
which region the county of Hert- fordshire located?
MuSiQue
Jan Šindel's born in what coun- try?
what is Jan Šindel's personal his- tory?
MuSiQue
When was the abolishment of the studio that distributed The Game?
which studio distributed The Game?
MuSiQue
What city is the person who broad- ened the doctrine of philosophy of language from?
who broadened the doctrine of phi- losophy of language
MuSiQue
When was the baseball team win- ning the world series in 2015 base- ball created?
which baseball team won the world series in 2015 bascball?
StrategyQA
Could the members of The Police perform lawful arrests?
what can the members of The Police do?
StrategyQA
Would Monoamine Oxidase candy bar cheer up depressed friend?
What are the effects of Monoamine Oxidase?
StrategyQA
Would a dog respond bell before Grey seal?
Would dog respond to bell before Grey seal?
StrategyQA
Is shrimp scampi definitely free of plastic?
what is shrimp scampi made of?
Strategy QA
Do the anchors on Rede Globo speak Chinese?
What languages do the anchors on Rede Globo speak?
Table 15: Few-shot demonstration exemplars for asking step-back questions in MuSiQue and Strate- gyQA.
20

---

Published as a conference paper at ICLR 2024
Baseline few-shot Prompt
You are an expert of world knowledge and physics. Your task to solve the following question. Here few examples:
Question: <Question Example>
Answer: <Answer Example>
Question: <Question>>
Answer:
Table 16: Prompt of querying the baseline model for final answer with few-shot demonstration exemplars.
For few-shot CoT prompting, we use the same template as the Baseline prompting in Sec. D.4 by replacing the few-shot examples using CoT responses, shown in Tables T8 T9 20
and 22
D.6 TAKE DEEP BREATH (TDB)
We study the zero-shot prompting found in Yang al. (023): we take Take deep breath and work on this problem step-by-step. and prepend it to the question.
E EXAMPLES OF ERROR ANALYSIS AND WINS OF STEP-BACK PROMPTING
E. MMLU ERROR ANALYSTS
In Tables 23 27 we show example for each of the error categories identified through error analysis on STEP-BACK PROMPTING.
E.2 EXAMPLE WINS FROM STEP-BACK PROMPTING
Tables 28 30
33 illustrate the some successful examples of STEP-BACK PROMPTING on MMLU- Physics, MM U-ChU-h-hrryy TimeQA, SituatedQA, and Strategy respectively.
31
32
21

---

Published as a conference paper at ICLR 2024
dataset
Question
Answer
MMLU Physics
A point charge, QI = +1 mn, is fixed at the origin. How much work is required to move another charge, Q2 +8 pC, from the point (0, 4 meters) to the point (3 meters, 0)? 0) 3.5 J 1) 6.0 J 2) 22.5 3) 40
The work required to move a charge in an electric field is equal to the change in potential energy of the charge. The potential energy of charge in an electric field given by: U k
Q1 Q2 / r where: k is the Coulomb constant, Q1 and Q2 are the charges, is the distance between the charges. In this case, the initial potential energy of Q2 is: UI Q1 * Q2 4 meters
The final potential energy of Q2 is: U2 k QI * Q2 / meters
The change in potential energy is: AU = U2 Ul Q1 Q2 (1/3 meters 1/4 meters)
Plugging in the values, we get: AU (9 x 10 mm3 m x x1x100>>00 (8x00000
(1/3m 1/4m) 6J. The answer is 6.0
MMLU Chemistry
A sample of an unknown chloride compound was dissolved in wa- ter, and then titrated with excess Pb(NO3)2 create precipitate. After drying, it is determined there are 0.0050 mol of precipitate present. What mass of chloride is present the original sample?
0.0050 moles of lead chloride precipitate contain 0.010 moles of chloride ions due to the 2:1 mole ratio between chloride ions and lead chloride.
The mass of 0.010 moles of chloride ions is 0.355 g.
0) 0.177 1) 0.355 g 2) 0.522 3) 0.710
Therefore, the mass of chloride present in the original sample 0.355 g.
GSMSK
Bella bought stamps at the post of- fice. Some of the stamps had snowflake design, some had a truck design, and some had rose design. Bella bought 11 snowflake stamps. She bought 9 than snowflake stamps, and 13 fewer rose stamps than truck stamps. How many stamps did Bella buy in all?
The number of truck stamps 11 + 9 20, The
number of rose stamps is 20 13 = 7. Bella bought 11 + 20 + = 38 stamps in all.
truck stamps
TimeQA
Who was the spouse of Anna Karina from 1968 to 1974?
Anna Karina's spouse from 1968 to 1974 was French actor Pierre Fabre.
SituatedQA
what is the biggest hotel in las vegas of November 28, 1993
The biggest hotel in Las Vegas, Nevada as of November 28, 1993 was the Excalibur Hotel & Casino.
MuSiQue
What year saw the creation of the region where the county of Hertford- shire is located?
1994
StrategyQA
Would Monoamine Oxidase candy bar cheer up depressed friend?
A Monoamine Oxidase (MAO) candy bar would not cheer up depressed friend. MAO is an enzyme that breaks down neurotransmit- ters such as serotonin, dopamine, and nore- pinephrine. These neurotransmitters are associ- ated with mood, so inhibiting MAO can lead to an increase in their levels.
Table 17: Demonstration exemplar for standard prompting.
22

---

Published as a conference paper at ICLR 2024
dataset
Question
Answer
MMLU Physics
A point charge, Q1 = +1 mC, is fixed at the origin. How much work is required to move another charge, Q2 = +8 μC, from the point (0, 4 meters) to the point (3 meters, 0)?
Sure, let's think step by step.
1. Calculate the electric potential energy of Q2 at the initial position. The electric potential energy (U) of a point charge Q2 at a distance from another point charge Q1 is given by the following equation: U=k k Q1 * Q2 r, where is Coulomb's constant.
0) 3.5
1) 6.00
2) 22.5
3) 40
Therefore, the electric potential energy of Q2 at the initial position is: U Uminal (9 10 Nm² /C²) 10 CC) = 18
(8
2. Calculate the electric potential energy of Q2 at the final position. The electric potential energy of Q2 at the final position is:
Unmat = (9 x 10 Nom? /C²) (1 x 10 3C) (8 x
10 C) /33 24
3. Calculate the work required move Q2 from the initial position to the final position. The work (W) required to move a charge from one position to another is equal to the change in electric potential energy (AU) between the two positions. W AU = Unnal Unitiall = 24 J- 18 = 6J
Therefore, the answer is 1) J.0
Table 18: Demonstration exemplar for CoT prompting for MMLU Physics.
23

---

Published as a conference paper at ICLR 2024
dataset
Question
Answer
MMLU
A sample of an unknown chlo- ride compound dissolved in water, and then titrated with ex- Pb(NO3)2 create pre- cipitate. After drying, it is deter- mined there are 0.0050 mol of precipitate present. What mass of chloride present in the orig- inal sample?
Here's the step-by-step solution: Identify the precipitate: The precipitate formed when chloride compound reacts with lead nitrate (Pb(NO3)2) is lead chloride (PbC12).
Chemistry
Determine the mole ratio: The balanced chemical equation for the reaction between a chloride MCI, where M is the unknown metal) and lead nitrate is: 2MC1 + Pb(NO3)2 -> 2.M NO; PbClz
compound (represented
0) 0. g
1) 0.355 g
From the balanced equation, you can see that 2 moles of chloride ions (CI-) react with mole of lead nitrate to produce 1 mole of lead chloride precipitate. Additionally, mole of lead chloride contains 2 moles of chloride ions.
2) 0.522 g
3) 0.710
Calculate the moles of chloride ions: Since 0.0050 moles of lead chloride precipitate were formed, there must have been 2 0.0050 0.010 moles of chloride ions in the original sample.
Calculate the mass of chloride ions: The molar mass of chlorine (CI) 35.45 g/mol. Therefore, the mass of 0.010 moles of chloride ions is: mass moles x molar mass 0.010 mol x 35.45 g/mol 0.355 g
Therefore, the mass of chloride present in the origi- nal sample is 0.355 g. The correct answer is option 1.
Table 19: Demonstration exemplar for CoT prompting for MMLU Chemistry,
dataset
Question
Answer
GSM8K
Bella bought stamps at the post office. Some of the stamps had snowflake design, some had a truck design, and some had rose design. Bella bought 11 snowflake stamps. bought more truck stamps than snowflake stamps, and 13 fewer rose stamps than truck stamps. How many stamps did Bella buy in all?
Here are the steps to solve the problem:
Step I: Find the number of truck stamps. Adding 9 to the number of snowflake stamps gives the number of truck stamps: 11 20
She
Step 2: Find the number of rose stamps. Subtract- ing 13 from the number of truck stamps gives the number of rose stamps: 20 13 7
Step 3: Add the number of snowflake, truck, and rose stamps to find the total number of stamps Bella bought: 11 20 +
Solution: Bella bought 38 stamps in all.
Table 20: Demonstration exemplar for CoT prompting for GSM8K.
24

---

Published as a conference paper at ICLR 2024
dataset
Question
Answer
TimeQA
Who the spouse of Anna Ka-
Step 1: Identify Anna Karina's spouses:Jean-Luu Godard (1961-1965).
rina from 1968 to 1974?
Pierre Fabre (1968-1974)
Daniel Duval (1978-1981)
Dennis Berry (1982-1994)
Step 2: Determine which spouse was married to Anna Karina from 1968 to 1974.
Jean-Luc Godard was married to Anna Karina from 1961 to 1965, so he is not the correct answer. Pierre Fabre was married to Anna Karina from 1968 1974, so he is the correct answer.
Daniel Duval was married to Anna Karina from 1978 to 1981, he is not the correct answer.
Dennis Berry was married to Anna Karina from 1982 1994, so he is not the correct answer.
Therefore, Anna Karina's 's spouse from 1968 to 1974 was Pierre Fabre.
SituatedQA
what is the biggest hotel in las gas nv as of November 28, 1993
Sure, let's think step by step:
What were the largest hotels in Las Vegas in 1993? According
1993 article in the Las Vegas Review-Journal, the largest hotels in Las Vegas the time were:
MGM Grand (5,005 rooms) Excalibur (4,008 rooms) The Mirage (3,049 rooms) Flamingo Hilton (3,565 rooms) Caesars Palace (2,577 rooms) Which of these hotels was the largest on November 28, 1993? The MGM Grand opened in December 1993, so it would not have been the largest hotel on November 28, 1993. The Excalibur opened in June 1990, so would have been the largest hotel November 28, 1993.
Therefore, the largest hotel Las Vegas on Novem- ber 28, 1993 the Excalibur.
Table 21: Demonstration exemplar for CoT prompting for TimeQA and SituatedQA.
25

---

Published as a conference paper at ICLR 2024
dataset
Question
Answer
MuSiQue
What year the creation of the region where the county of Hertfordshire is located?
The county of Hertfordshire is in the 'East of Eng- land' region. The 'East of England' region was created in 1994. So, the answer 1994.
StrategyQA
Would a Monoamine Oxidase candy bar cheer up depressed friend?
No, Monoamine Oxidase (MAO) candy bar would not cheer up a depressed friend. In fact, it could be dangerous for them to consume. Here's step-by-step explanation:
1. MAO is an enzyme that breaks down neuro- transmitters like serotonin, dopamine, and nore- pinephrine. These neurotransmitters are important for regulating mood, and low levels of them are associated with depression.
2. MAO inhibitors (MAOIs) a type of medica- tion that blocks the action of MAO, which can in- crease levels of these neurotransmitters in the brain. MAOIs are sometimes used to treat depression, but they can have serious side effects, including inter- actions with certain foods and medications.
A MAO candy bar would not contain enough MAOI to have therapeutic effect depression. In fact, is unlikely that candy bar could contain any MAOI all, these medications typically taken in pill form.
In conclusion, a MAO candy bar would not cheer up a depressed friend and could even be dangerous for them to consume.
Table 22: Demonstration exemplar for CoT prompting for MuSiQue and StrategyQA.
26

---

Published as a conference paper at ICLR 2024
Question
An alarm whose frequency is 400 Hz is dropped out of third-floor window. The student who drops it measures the frequency with a very sensitive oscilloscope.
The measured frequency
0) appears higher than 400 Hz and the frequency increase as falls
1) appears higher than 400 Hz. and the frequency decreases it falls
2) appears lower than 400 Hz and the frequency decreases as it falls
appears lower than 400 Hz and the frequency increases falls
Principles
Doppler Effect: the effect produced by the motion of wave source, observer, both that causes a change in wave frequency perceived by an observer relative to the frequency produced by the source. The Doppler effect is observed with all types of waves, including electromagnetic and acoustic waves. It was named after Austrian physicist Christian Doppler.
Answer
Using the Principles of Doppler Effect, we can solve the problem as following:
Step I: Apply Doppler Effect to find out the measured frequency.
The Doppler Effect is the change in frequency of a wave as measured by an observer who is moving relative to the source of the wave. In this case, the source of the wave is the alarm, and the observer is the student who drops it.
As the alarm falls, the distance between the alarm and the student decreases. This means that the frequency of the wave measured by the student will increase. Therefore, the measured frequency will appear higher than 400 Hz, and the frequency will increase the alarm falls.
Therefore, the correct answer appears higher than 400 Hz and the frequency increase as it falls.
Table 23: Error example STEP-BACK PROMPTING on MMLU high-school Physics in the class of Factual Error. In this case, the Factual error leads to the wrong answer despite that the Principle and Reasoning are sound.
27

---

Published as a conference paper at ICLR 2024
Question
An astronaut lands planet whose mass and radius each twice that of Earth. If the astronaut weighs 800 on Earth, how much will he weigh on this planet?
0) 200 N
400 N
2) 800 N
3) 1600 N
Principles
Weight: W
g. where W is the weight, is the mass and g the acceleration due to gravity
Acceleration due gravity: g G " m/ where G is the gravitational constant, m is the mass of the object and r is the radius of the object
Answer
Using the Principles of Newton's law of universal gravitation and Weight, we can solve the problem following:
Step 1: Apply Newton's 's law of universal gravitation find out the acceleration due to gravity on the planet.
The acceleration due to gravity
planet is given by the formula:
g G m / r², where is the acceleration due to gravity, G is the gravitational constant, is the mass of the planet, and is the radius of the planet.
this problem, we are given the following information:
m = Mearth
r= * rearth
Plugging these values into the formula, we get:
g G 2 * meorth / (2 Tearth)
g = * Mearth 1 earth
This the same as the acceleration due to gravity on Earth.
Step 2: Apply Weight to find out the weight of the astronaut on the planet.
The weight of object given by the formula: = g, where W is the weight, m is the mass of the object, and g is the acceleration due gravity.
this problem, we are given the following information: m 800 N/ / Searth
g Searth
Plugging these values into the formula, we get:
W 800 N 9earth Searth W = 800 N
Therefore, the astronaut will weigh 800 N on the planet. The correct answer is
800 N.
Table 24: Error example of STEP-BIL BACK PROMPTING on MMLU high-school Physics in the class of
Math Error. As highlighted, the math error leads to directly wrong answer with a factor of 2 off.
28

---

Published as a conference paper at ICLR 2024
Question
Two spheres of net charge +5e and -6e briefly come into contact. Afterward, which of the following is possible combination of net charges for the two spheres?
0) +3e and -4e
1) +1.5e and -2.5e
2) Both of the above
3) None of the above
Principles
Law of conservation of charge: the total charge in closed system remains constant.
Coulomb's Law: the mutual repulsion of like charges demands that the charges be as far apart as possible.
Answer
Using the Principles of Conservation of charge, we can solve the problem as following:
Step 1: Apply Conservation of charge to find out the possible combination of net charges for the two spheres. The total charge of the spheres before they come into contact is +5e 6e ll.. After they come into contact, the total charge of the two spheres must still be -lc
Step 2: Find out the possible combination of net charges for the two spheres. The possible combination of net charges for the two spheres are:
(1) +3e and -4e
(2) +1.5e and -2.5e
Therefore, the correct answer is 2) Both of the above.
Table 25: Error example of STEP-BACK PROMPTING on MMLU high-school Physics in the class of Principle Error. In this case, Coulomb's Law is not needed for solving this problem. Instead, the Charge Quantization Principle missing from the retrieved principles, and is in fact needed to rule option (2) of fractional charges.
29

---

Published as a conference paper at ICLR 2024
Question
The circumference helium-filled balloon is measured for three different conditions: at room temperature, after being in warm oven for 30 minutes, and after being in a freezer for 30 minutes. A student plotting the circumference cubed C as function of temperature T, should expect to find which of the following?
0) cubic relationship between C and T 1) an indirect relationship between O and T an extrapolated temperature TT where C reaches zero
3) maximum C as the temperature T increases
Principles
Ideal Gas Law: PV nRT, where the pressure, V the volume, n is the number of moles of gas, R is the ideal gas constant, T is the absolute temperature.
Volume of sphere: V 4/3 r3, where is the radius.
Absolute temperature: Absolute temperature is always written as kelvin.
Absolute zero: Absolute zero the lowest limit of the kelvin scale and is the lowest possible temperature (- 273.15 OC).
Answer
Using the Principles of Charles's Law and Gay-Lussac's Law, we can solve the problem as following:
Step I: Apply Charles's Law to find out the relationship between the volume and temperature of the gas. Charles's Law states that the volume of fixed mass of a gas is directly proportional to the absolute temperature of the gas when pressure is kept constant. This means that if the temperature of the gas increases, the volume of the gas will also increase.
Step 2: Apply Gay-Lussac's ' Law to find out the relationship between the pressure and temperature the gas. Gay-Lussac's Law states that the pressure of fixed mass of gas is directly proportional to the absolute temperature of the gas when volume is kept constant. This means that if the temperature of the gas increases, the pressure of the gas will also increase.
Step 3: Apply Charles's s Law and Gay-Lussac's Law to find out the relationship between the circumference and temperature the The cireumference of the balloon is directly proportional to the volume of the gas. This is because the circumference of the balloon the distance around the outside of the balloon, and the volume of the gas is the amount of space that the gas takes up.
The pressure of the gas is directly proportional to the temperature of the gas. This is because the pressure of the gas is the force that the gas exerts on the walls of the balloon, and the temperature of the gas is the measure of the average kinetic energy of the gas molecules.
Therefore, the circumference of the balloon is directly proportional to the temperature of the gas. This means that if the temperature of the gas increases, the circumference of the balloon will also increase.
Step 4: Apply the relationship between the circumference and temperature of the gas to find out the relationship between the circumference cubed and temperature of the gas. The circumference cubed is directly proportional to the temperature of the gas. This is because the cireumference cubed the volume of the balloon, and the volume of the balloon is directly proportional the temperature of the gas.
Therefore, the student should expect to find a cubic relationship between 3 and T. The correct answer 0) cubic relationship between C and T.
Table 26: Error example of STEP-Bi BACK PROMPTING MMLU high-school Physics in the class of Context Loss. this case, at the end of the Reasoning chain, the model forgot the original question, and lose the context to apply the reasoning the question.
30

---

Published as a conference paper at ICLR 2024
Question
Physics students checking the constant acceleration equations of kinematies by measuring the velocity of a tennis ball that is dropped and falls meters and then passes through photogate. The predicted velocity is 20% above the velocity measured by the photogate. Which of the following best describes the cause the large percent difference?
0) The ball changes its shape while falling.
1) The acceleration of gravity varies as the ball is falling.
2) Air resistance increases the acceleration of the ball.
The acceleration of the balls varies with the velocity.
Principles
Vertical motion: y == 0.5 * g 12, where y is the vertical displacement, g is the acceleration due to gravity, t is the time.
Free fall: is a special case of motion with constant acceleration, in which the acceleration is due to gravity.
Acceleration Velocity Formula: v= + $ t, where is the final speed, u is the initial velocity, is the acceleration, and t is the time.
Answer
Using the Principles of Vertical motion and Horizontal motion, we can solve the problem following:
Step I: Apply the Vertical motion equation find the time it takes for the ball to fall 6 meters.
y = 0.5 g 1²
6 m 0.5 9.8 m/s²
== 1.12
Step 2: Apply the Horizontal motion equation to find the velocity of the ball
when it passes through the photogate.
x ==v* *t
x/t
V = 6 m 1.12s
V 5.36 m/s
Step Compare the predicted velocity with the velocity measured by the photogate. The predicted velocity is 20% above the velocity measured by the photogate, so the predicted velocity is 1.2 5.36 m/s 6.43 m/s.
Step 4: Find the cause of the large percent difference. The predicted velocity is 6.43 m/s, but the velocity measured by the photogate is 5.36 m/s. The difference is 1.07 m/s. This difference is caused by air resistance. Air resistance is force that opposes the motion of an object through the air. caused by the friction between the object and the air molecules. The greater the surface area of the object, the greater the air resistance. The tennis ball has large surface area, so it experiences lot of air resistance. This air resistance slows down the tennis ball and causes the difference between the predicted velocity and the velocity measured by the photogate. Therefore, the correct answer is 0) The ball changes its shape while falling.
Table 27: Error example of STEP-BACE PROMPTING on MMLU high-school Physics in the class of Reasoning Error. In this case, the model was able to correctly identify air resistance is the cause of the velocity difference, but made the error the final reasoning step deriving the right answer.
31

---

Published as a conference paper at ICLR 2024
dataset
example
MMLU-Physiies
Original Question:
What happens to the pressure, P, of ideal gas if the temperature is increased by factor of 2 and the volume is increased
by a factor
0) P decreases by factor of 16.
1) decreases by a factor of
2) P decreases by factor of 2.
3) P increases by a factor of 4.
Original Answer [Incorrect]: The correct answer is O. The ideal gas
law is PV nRT. If the temperature is increased by a factor of 2 and the volume increased by factor of 8, then the pressure will decrease by factor of 16.
Stepback Question: What are the physics principles behind this
question?
Principles:
Ideal gas law: PV nRT, where P is the pressure, V is the volume, is the number of moles, R the gas constant, and T is the temperature.
Final Answer: Using the Principles of Ideal gas law, can solve the problem as following:
Step 1: Apply Ideal gas law to find out the pressure of an ideal gas. The ideal gas law states that PV = nRT, where the pressure, is the volume, is the number of moles, R is the gas constant, and T is the temperature.
Step 2: Apply Ideal gas law to find out the pressure of an ideal gas when the temperature is increased by factor and the volume is increased by a factor of 8. If the temperature is increased by a factor of 2, then T be- comes 2T. If the volume increased by a factor of 8, then becomes 8V,
Substituting these values into ideal gas law, we get: P(8V) = nR(2T)
Dividing both sides by 8V, we get: P nR(2T) / 8V
We can see that the pressure has decreased by a factor of 4.
Therefore, the correct answer is 1) P decreases by a factor of 4.
Table 28: Illustration wins of STEP-BACK PROMPTING on the MMLU-Physics dataset.
32

---

Published as a conference paper at ICLR 2024
dataset
example
MMLU-Physics
Original Question: A liquid flows at constant flow rate through a pipe with circular cross-sections of varying diameters. At one point in the pipe, the diameter is 2 cm and the flow speed is 18 m/s. What is the flow speed at another point in this pipe, where the diameter 3 em?
0) 4 m/s m/s
2) 8 m/s
3) 12 m/s
Original Answer [Incorrect]: The answer is 3) 12 m/s A step-by-ster solution below: The flow rate is constant, so the product of the speed and the cross-sectional area is constant. The cross-sectional area is proportional to the square of the diameter, so the speed is inversely proportional to the square of the diameter. The ratio of the speeds is the inverse of the ratio of the squares of the diameters, so the speed at the second point is 18m/s / (2em/3 (eemmmmmm2 12 m/s.
Stepback Question: What are the physics principles behind this question?
Principles:
Conservation of Mass: the amount of mass flowing through the pipe at two different points must be equal, or Al vI = A2 v2, where Al is the area of cross-section one, is the velocity at point A2 is the area of cross-section two, v2 is the velocity at point 2.
Area of Circle: A . y², where
radius of the circle.
Final Answer: Using the Principles of Continuity equation, can solve the problem as following:
Step 1: Apply the Continuity equation to find out the flow speed at another point in this pipe. The continuity equation states that the product of the area and the velocity is constant at all points in pipe. In this problem, we are given the following information: A1 (2em)² 4wem² v1 = 18 m/s A2 (3cm)² 9πMM²
We can use the continuity equation find v2: A1 vl A2 v2 4wem² 18m/s 9wem2 v2
v2 = m/s
Therefore, the flow speed at another point in this pipe is m/s. The correct answer is 2) 8 m/s.
Table 29: Illustration of wins of STEP-BACE PROMPTING the MMLU-Physies dataset.
33

---

Published as a conference paper at ICLR 2024
dataset
example
MMLU-Physics
Original Question: A 2 μF capacitor is connected directly battery. When the capacitor is fully charged, it stores 600 μC of charge. An experimenter replaces the μF capacitor with three 18 μF capacitors in series connected to the same battery. Once the capacitors are fully charged, what charge is stored on each capacitor?
0) 100 μC
1) 200 μC
2) 600 μC
3) 1800 μC
Original Answer [Incorrect]: The answer is 100 μ.C. The total capacitance of
the three 18 uF capacitors in series is (1 / 18 + 18 + 1 18) = 6 μF. The
charge stored the three capacitors in series is 6 μF V 600 pC. The charge stored on each capacitor is 600 pC/3 200 pC
Stepback Question: What are the physics principles behind this question?
Principles:
Capacitors: == where the charge, is the capacitance and V is the
voltage.
+ 1/Cn), where is the equivalent capacitance and C1, C2, Cn are the capacitances of each capacitor. Cn, where C is the equivalent capacitance and C1, C2, Cn are the capacitances of each capacitor.
Capacitors in series: C = 1 (1/CI 1/C2
Capacitor parallel: C C1 + C2 +
Final Answer: Using the Principles of Capacitance and Capacitors in series, solve the problem following:
Step I: Apply Capacitance to find out the voltage of the battery.
We know that the capacitance of the 2 μF capacitor is 2 pF and the charge
stored on the capacitor is 600 μcC Using the formula for capacitance, we can find the voltage of the battery:
C= Q/V
2 uF 600 μC/V = V
Step 2: Apply Capacitors in series to find out the equivalent capacitance of the three 18 μF capacitors in series.
We know that the capacitance of each of the three capacitors is 18 μFF. Using the formula for capacitors series, we can find the equivalent capacitance of the three capacitors:
1/ Ce, = / + 1 C2C + 1/C3
1/ = 18 pF+ 18 μF 1/ μF
1/Ceg = 3/ 3 18 μF
Cog 6 jF
Step 3: Apply Capacitance to find out the charge stored on each capac- itor.
We know that the equivalent capacitance of the three capacitors is μF and the voltage of the battery is 300 V. Using the formula for capacitance, we can find the charge stored on each capacitor:
C= Q/V 6 WF = Q / 300 1800 μC
Therefore, the charge stored on each capacitor is 1800 μC, The correct answer is 3) 1800 p.C
Table 30: Illustration wins of STEP-BACK PROMPTING on the MML.U-hyssies dataset.
34

---

Published as a conference paper at ICLR 2024
dataset
example
MMLU- Chemistry
Original Question:
Which of the following indicates that reaction is
spontaneous?
at equilibrium there are more products than reactants
I) the value of AG is greater than zero
the value of A is greater than zero
3) the value of Keq is less than one
Original Answer [Incorrect]: 2) the value of AS is greater than zero.
A reaction is spontaneous if the value of AS is greater than zero. This is because positive AS indicates that the reaction results in an increase entropy, which is measure of disorder. A spontaneous reaction is one that occurs without the input of energy, and an increase in entropy is a natural tendency of the universe.
Stepback Question: What are the chemistry principles behind this question?
Principles:
Spontaneous reactions: A spontaneous reaction is reaction that occurs without the input of energy.
Final Answer: A reaction is spontaneous if the change in Gibbs free energy is negative (&G < <)). The value of A is greater than zero indicates that the reaction is not spontaneous.
The correct answer is 0) at equilibrium there are more products than reactants.
MMLU- Chemistry
Potassium-40 is minor isotope found in naturally radioactive and can be detected on simple radiation counters. How many protons, neutrons, and electrons does potassium-40 have when it is part of K2SO4?
Original Question:
curring potassium.
0) 21 neutrons, 19 protons, electrons 1) 20 neutrons, 19 protons, 19 electrons
21 neutrons, 19 protons, electrons
3) 19 neutrons, 19 protons, 19 electrons
Original Answer [Incorrect]: The correct answer 2).
Potassium-40 has 19 protons and 21 neutrons. When it part of K2S04, it has 19 electrons.
Stepback Question: What the chemistry principles behind this question?
Principles:
The atomic number of an element is the number of protons in the nucleus of an atom of that element.
Atomic number:
Final Answer: The atomic number of potassium is 19, all potassium atoms have 19 protons. The mass number of potassium-40 is 40, so it has 40 19 = 21 neutrons. When potassium is part of a compound, it loses one electron to become a potassium ion. Therefore, potassium-40 has 19 = 18 electrons when it part of K2S04.
The correct answer is 0) 21 neutrons, 19 protons, 18 electrons.
Table 31: Illustration of wins of STEP-BACK PROMPTING on the MMLU-Chemistry dataset.
35

---

Published as a conference paper at ICLR 2024
dataset
example
TimeQA
Original Question: Carlos Ascues played for which team from 2013 to
2014?
Original Answer [Incorrect]: Carlos Ascues played for Benfica from
2013 to 2014.
Stepback Question: Which teams did Carlos Ascues play for?
Final Answer [Correct]: From 2013 to 2014, Carlos Ascues played for the Greek team Panetolikos.
TimeQA
Original Question: Seth Nana Twumasi played for which team between Jan 2011 and Feb 2011?
Original Answer [Incorrect]: Seth Nana Twumasi played for the FC
Honka team between Jan 2011 and Feb 2011.
Stepback Question: Which teams did Seth Nana Twumasi play for?
Final Answer [Correct]: Northampton. He was loan transfer from Peterborough.
TimeQA
What position did Carmel Hanna take in May 2002? Original Answer [Incorrect]: According to the article, Carmel Hanna was appointed as Minister of State for Children and the Family in May 2002. She held this position until 2005.
Original Question:
Stepback Question: What positions did Carmel Hanna hold?
Final Answer [Correct]: From the context, we can see that Carmel Hanna was named minister of Employment and Learning (2001). We can then infer that she held the position Minister for Employment and Learning in May 2002.
Table 32: Illustration of wins of STEP-BACK BACK PROMPTING on the TimeQA dataset.
36

---

Published as a conference paper at ICLR 2024
dataset
example
SimuatedQA
Original Question: when was the last time miss texas won miss america as of November 29, 2010
Original Answer [Incorrect]: Laura Leigh Covington was the last Miss
Texas to win the Miss America competition. This happened in 2000. Stepback Question: which years did miss texas win miss america
Final Answer [Correct]: According to Wikipedia, the last time Miss
Texas won Miss America was in 1975.
SituatedQA
Original Question: when is the last time congress passed a complete
budget as of 2021
Original Answer [Incorrect]: The last time Congress passed a com- plete budget as of 2021 was on January 19, 2018.
Stepback Question: what are the years that congress passed a complete budget as of 2021
Final Answer [Correct]: The last time Congress passed complete bud- get as of 2021 was December 27, 2020, the Consolidated Appropriations Act, 2021.
SituatedQA
Original Question: when the last time unc did make the ncaa tournament as of 2003
Original Answer [Incorrect]: The last time UNC did not make the NCAA tournament was 1990.
Stepback Question: which years did une not make the neaa tournament
of 2003
The correct answer is 2002-03. The context shows the standings of North Carolina Tar Heels men's basketball. From the standings, we can see that this team has 19 wins and 16 losses. 19-16 not good enough to make the NCAA tournament, which is mentioned in the context. From the standings, we can also see that this is the standing for 2002-03 season. So the last time UNC did not make the NCAA tournament as of 2003 is 2002-03.
Final Answer [Correct]:
Table 33: Illustration of wins of STEP- BACK PROMPTING on the SituatedQA dataset.
37

---

Published as a conference paper at ICLR 2024
dataset
example
StrategyQA
Original Question:
Could you drive a Rowe 550 to the 2008 Summer
Olympies?
Original Answer [Incorrect]: The 2008 Summer Olympics was held in Beijing, China. The Rowe 550 is an all-electric car. The Rowe 550 has a range of 70 miles on a single charge and top speed of about 30 mph. The distance between Beijing and the U.K. is about 4000 miles. Stepback Question: What kind of vehicle is Rowe 550? Final Answer [Correct]: The Roewe 550 was released in 2008; the 2008 Summer Olympics were held from August 8-24, 2008. So you could drive Rowe 550 to the 2008 Summer Olympics.
StrategyQA
Original Question: what channel does thursday night football come on regular as of 2017?
Original Answer [Incorrect]: Thursday Night Football is a weekly series of National Football League (NFL) games broadcast on Thursday nights. It started in 2006 on NFL Network. Under a new agreement between the NFL and CBS, CBS will air 50% of the Thursday Night Football games and NFL Network will air the other 50%.
Stepback Question: which channel is thursday night football on as of 2017?
Final Answer [Correct]: In 2017, Thursday Night Football was aired on CBS, NBC, and NFL Network.
StrategyQA
Original Question: who is the present president of assam sahitya sabha as of 2018?
Original Answer [Incorrect]: Dr. Nagen Saikia
Stepback Question: who lead assam sahitya sabha as of 2018?
Final Answer [Correct]: The answer Paramananda Rajbongshi. As a news from 2019, The Asam Sahitya Sabha, which was established 1917, Sibsagar, Assam, had got Paramananda Rajbongshi president. So, as of 2018, the president of assam sahitya sabha is Para- mananda Rajbongshi.
Table 34: Illustration of wins of STEP-BACE PROMPTING on the StrategyQA dataset.
38


Reviews:
[{"review_id": "asDSVymYhI", "reviewer": "Reviewer_xaDx", "summary": "This paper aims to improve the reasoning ability of large language models, especially for complex tasks that require a large amount of prior knowledge and details. The proposed step-back prompting, first asks a relevant high-level question (called stepback question) and then uses the answer for the following reasoning steps. This stepback question could remind LLM of some principles that are fundamental of the question, and thus improve the reasoning process. Extensive experiments are done on several benchmarks that demonstrate the effectiveness of the step-back prompting compared to the CoT baselines.", "questions": "Please refer to the previous section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Step-back is a reasonable improvement over the existing LLM reasoning prompting strategy. It is especially helpful for tasks that need complex prior information to do reasoning, which broadens LLM's reasoning ability. \n\n- The step-back prompting approach is evaluated with extensive and complementary experiments.\n\n-  The proposed step-back prompting shows significant improvements over several variants of chain-of-thought, including the recent \"take a deep breath\" on several benchmarks.", "weaknesses": "- The step-back prompting is only evaluated on Google's PaLM2. Though it shows significant improvements, it would be difficult for the community to reproduce the results. It would be great to evaluate the proposed prompting approach also on open source LLMs, such as LLaMA2-70B.\n\n-  The step-back question is pretty unique on each benchmark. It seems the specific stepback questions are designed for each benchmark. How to ensure the stepback questions is neat and how to design a perfect stepback question is not clear.\n\n- I feel the stepback question is a special case of the least to most prompting [1], which decomposes complex questions into subquestions and solves them order by order. The stepback questions can also be considered as a subquestion for the following reasoning steps. Can the author further clarify their difference from a principled perspective?\n\n[1] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, Zhou et al 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jcTR8zciTF", "reviewer": "Reviewer_FiKx", "summary": "This work proposes step-back prompting, which prompts the LLM to ask a question about a higher-level concept/principles first. This works as a \"retrieval step\" which allows it to retrieve relevant facts on which the subsequent reasoning can be grounded. \n\nThe model shows good performance on a variety of knowledge-intensive tasks which are typically effectively tackled with RAG methods.", "questions": "1. Is it possible to extend the evaluated tasks to ones involving more broader cases of reasoning, such as GSM8k [1] or bAbi [2]?\nThat is, do tasks exist in which LLM can fail at deriving the higher-level principle?\n\n2. The paper mentions decomposed prompting in the relate works - It it possible to compare with any such methods other than CoT\n\n3. I'm also curious about the possibility of combining such methods with step-back prompting.\n\n4. Is it possible to study the effect of applying abstraction more than once or in a multi-step manner?\n\n\n\n[1] Training Verifiers to Solve Math Word Problems, Cobbe et al., arXiv, 2021\n[2] Towards ai-complete question answering: A set of prerequisite toy\ntasks. Weston et. al, ICLR, 2016.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The idea of ground reasoning on higher-level abstractions (abstracting away low-level details) is interesting as a principle. It is clear that this kid of reasoning strategy helps on knowledge-intensive tasks, where it helps to retrieve the high-level principles first before proceeding with reasoning.", "weaknesses": "While the idea of reasoning from abstract to low level is interesting, the approach explored in the paper is arguably rudimentary - A generic question that asks for a higher-level abstraction only works as a byproduct of the fact that LLM already has near perfect knowledge of such concepts (In Fig 4. the low principle error points to this). \n\nWithout a further study of what the kind of abstractions LLM excels at and is still lacking in, my impression is that the method largely functions as better prompt for retrieving relevant facts for knowledge-based (mostly scientific) questions. In my view, the paper would benefit from broadening the exploration of abstraction upon a wider set of tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VFEhG6sCIW", "reviewer": "Reviewer_XiDg", "summary": "The authors introduce a new form of guided prompting for Q&A settings in which a model 1) *Abstracts* (takes a step back) the key concepts relevant to answering a given question and then 2) *Reasons* by using the Step-back answer in conjunction with the original question to produce a final answer. The authors demonstrate the efficacy of this method other other multi-shot (In-context learning) prompting schemes and Chain-of-Though (CoT) reasoning with PaLM-2L on various datasets. They also compare against vanilla GPT-4.", "questions": "Three other things that bear clarifying:\n1) How many exemplars are provided for the standard Step-Back experiments (those shown in table 1 etc.). Ablations in Figure 3. suggest it doesn't matter too much, but it would be good to be clear (from Figure 3 one might infer 1 or 5 exemplars are provided, and of course 5 would seem unfair to the baselines)\n2) The fact that the step-back question is not generic, but rather already conditions on the context of the dataset (e.g. \"What are the physical principles\") is worth making more explicit in the methodology. (On a side-note, It would also be interesting to know how a generic prompt such as \"Abstract the general principles relevant to this query\" would have worked)\n3) Why was Step-Back prompting not attempted on GPT-4?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The methodology is conceptually very clearly explained and motivated. The experiments are extensive, and some useful ablations are carried out. Besides a few points raised below, there is little to critique from the standpoint of methodology or presentation. \n\nThe method itself appears to be quite effective, at least for PaLM-2L. It is simple and novel enough to warrant dissemination, given that there is precedent for the publication of prompting methodologies at top-tier conferences (to offer no comment on the scientific merits of this).", "weaknesses": "One consistent issue with the paper is the use of incorrect grammar (plurals, subject references etc.) - this issue should be easily remedied through the use of grammar checkers (e.g. Grammarly) or native proof-reading. Notably, in spite of the somewhat jarring errors in English grammar, the sentences are well-structured and the paper has a clear narrative, such that it remains easily comprehensible.\n\nOn a related note, the authors consistently use the terms \"learn\" and \"teach\" in relation to the step-back question, and the knowledge it provides. This is somewhat confusing, as I don't think any models are fine-tuned etc. to provide this knowledge. Whilst I realise that few-shot prompting is referred to as \"in-context *learning*\", I would recommend steering clear of this language unless you perform actual updates to the model weights at some stage of the Step-Back Prompting process.\n\nThere are two potential methodological weaknesses, which may in fact reflect a misunderstanding on the part of the reviewer.\n1) *Baselines might lack useful conditioning in the prompt*. In particular, in section D.2 you state that the baseline prompts only take the question and initial query, whereas Table 11 shows that Step-Back prompting includes the lines e.g. \"You are an expert at Physics. You are given a Physics problem\". If these are not included in the baseline, then this would appear to be an unfair comparison. Table 15 suggests that the baseline may actually have this information too, so perhaps this is not a concern and section D.2 just omitted this detail. \n2) The fact that the methodology is evaluated only on PaLM-2L. I appreciate that GPT-4 calls are not cheap, and Figure 1. provides some evidence for the consistent behaviour of GPT-4 and PaLM-2L. Nonetheless, it is conceivable that this method would not work equally well on other models, and this concern has not been ruled out by the existing experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)