Large Language Models as Analogical Reasoners
Authors: Michihiro Yasunaga
Xinyun Chen
Yujia Li
Panupong Pasupat
Jure Leskovec
Percy Liang
Ed H. Chi
Denny Zhou
Venue: ICLR 2024 poster (2024)
Keywords: large language model
prompting
analogical reasoning
reasoning
Primary area: general machine learning (i.e., none of the above)
TL;DR: We introduce Analogical Prompting, a new language model prompting approach that self-generates relevant reasoning exemplars for solving problems. We show that this outperforms previous chain-of-thought prompting methods.

Abstract: Chain-of-thought (CoT) prompting for language models demonstrates impressive performance across reasoning tasks, but typically needs labeled exemplars of the reasoning process. In this work, we introduce a new prompting approach, analogical prompting, designed to automatically guide the reasoning process of large language models. Inspired by analogical reasoning, a cognitive process in which humans draw from relevant past experiences to tackle new problems, our approach prompts language models to self-generate relevant exemplars or knowledge in the context, before proceeding to solve the given problem. This method presents several advantages: it obviates the need for labeling or retrieving exemplars, offering generality and convenience; it can also tailor the generated exemplars and knowledge to each problem, offering adaptability. Experimental results show that our approach outperforms 0-shot CoT and manual few-shot CoT in a variety of reasoning tasks, including math problem solving in GSM8K and MATH, code generation in Codeforces, and other reasoning tasks in BIG-Bench.

Published as a conference paper at ICLR 2024
LARGE LANGUAGE MODELS AS ANALOGICAL REASONERS
Michihiro Yasumaga, 1.2
Xinyun Chen,' Yujia Li,² Panupong Pasupat,' 1
Jure Leskovec, 2
Percy Liang, 2 Ed H. Chi,' 2
Denny Zhou'
"Googee DeepMind 2 Sttnnford University
myasuêcs. .ta.uuesssttaoordrd stanford.edu,
ABSTRACT
Chain-of-thought (CoT) prompting for language models demonstrates impressive performance across reasoning tasks, but typically needs labeled exemplars of the reasoning process. In this work, we introduce a new prompting approach, analogical prompting, designed to automatically guide the reasoning process of large language models. Inspired by analogical reasoning, a cognitive process in which humans draw from relevant past experiences to tackle new problems, our approach prompts language models to self-generate relevant exemplars or knowledge in the context, before proceed- ing to solve the given problem. This method presents several advantages: it obviates the need for labeling or retrieving exemplars, offering generality and convenience; it can also tailor the generated exemplars and knowledge each problem, offering adaptability. Experimental results show that our approach outperforms O-shot CoT and manual few- shot CoT in a variety of reasoning tasks, including math problem solving in GSMSK and MATH, code generation in Codeforces, and other reasoning tasks in BIG-Bench.
INTRODUCTION
Large language models (LLMs) demonstrate strong performance across various tasks (Brown al., 2020; Chowdhery et al., 2022: Liang et al., 2022; Qin et al., 2023). Recently, chain-of-thought (CoT) prompting has demonstrated LLMs' abilities to tackle complex tasks, such as solving math problems, by prompting them to generate intermediate reasoning steps (Wei et al., 2022b; Kojima et al., 2022). For instance, common methods like few-shot CoT (Wei et al. 2022b; Figure I, middle) make LLMs generate reasoning steps by offering few exemplars of ieaatiionrationaalaanseeee triplets; 0-shot CoT (Kojima et al. 2022; Figure 1, left) aims for the same objective by offering instructions like "think step by step." " These studies highlight the importance of devising effective methods to guide LLMs to reason.
However, the existing CoT paradigm faces two key challenges: providing relevant guidance or exemplars of reasoning, and minimizing the need for manual labeling. Specifically, 0-shot CoT offers generic reasoning guidance, which may not suffice for complex tasks like code generation ($6). Few-shot CoT provides more detailed guidance but demands labeled exemplars of the reasoning process, which can be costly to obtain for every task. This raises a research question: can we achieve the best of both worlds and automate the generation of relevant exemplars guide L.MMs' reasoning process?
In this work, we propose analogical prompting, a new prompting approach that automatically guides the reasoning process of LLMs. Our inspiration comes from analogical reasoning in psychology, concept where humans draw from relevant past experiences to tackle new problems (Vosniadou & Ortony, 1989). For instance, when given new math problem (e.g., finding the area of a square given four points in coordinate system; Figure 1), humans often think about "Do I know a related problem?" (Polya, 2004) and recall how they solved related problems in the past (e.g.. finding the area of a square with known side length) to derive insights for solving the new problem. They also recall high-level knowledge, such as the need to find the side length to calculate a square's area. Our idea to prompt L.MM generation to mimic this reasoning process to effectively solve new problems.
Concretely, given a problem to solve, we prompt LLMS to self-generate relevant exemplars in the context, using instructions like "# Recall relevant problems and solutions:... ... and then proceed to solve the original problem (Figure 1, 2). We can also prompt the LLM to generate high-level knowledge that complements specific exemplars, using instructions like "# Provide a tutorial:.... (Figure
3). This turns out to be particularly useful for complex tasks like code generation (see $6). Notably, our method can operate in single prompt, generating knowledge, exemplars, and solution to the initial

---

Published as a conference paper at ICLR 2024
0-shot
Few-shot CoT
Analogical Prompting (Ours)
Model
Model
Model Imput
Q: What is the aroa of the square with the four vertices at 1-2, 2). (2.
Roger tannis balls. He
Q: What is the area of the square with the vortices at (22 2), 12. -2), (:2, -6), and (-8, x2)?
buys more cans tannis halls.
-2). (-2 -5), and (-6, -2/?
Each can has lernis balls. How many tennis balls does he have?
# Instruction:
## Recall relevant examplars: He Solve the inttial probiem:
0-shot CoT
cans of balls each is ballo.
A: Roger started with 5 balls, 2
The answer 5 8
Model Imput
Model Output
Q: What is the area of the squara
# Relevant exemplars:
with the four vertices at (2. Think
Q: What is the area of the aquare with the four vertices at (-2, 2), -2). and (-6, -21?
a: What is the area the square with side longth of 5?
-6), and -2)?
A: The area of square found squaring the lingth of side. So, the area of square 5*2 25.
Solve inttial problem: To find
Generic guidance reasoning
Need labeled exemplars reasoning
length. length leqrt((2 (-20)2 1-2 2)"2 (sar1(32), So, the area of square is (2qrt(22))22
square,
find
Exemplars are automatically generated and tailored to each problem
Existing methods for prompting LLM to reason are either generic (0-shot CoT) or demand laheled exemplars (few-shot CoT). Right: Given problem, our method prompts LLMs to self-generate relevant exemplars before solving the problem. This eliminates the need for labeling and also tailors the exemplars to each individual problem. See Figure for sample prompt where the LLM self- both knowledge and exemplars.
Figure 1:
Overview of our approach, analogical prompting. Left:
problem end-to-end in one pass. The underlying idea here is that modern LLMs have already acquired knowledge of various problems during training. Explicitly prompting them to recall relevant problems and solutions in the context guides LLMs to perform in-context learning to solve new problems.
Our proposed approach offers several advantages. self- self-ggeneates exemplars and obviates the need for manually labeling reasoning exemplars for each task, addressing the challenges faced by O-shot and few-shot COT. Furthermore, the self-generated exemplars are tailored to individual problems, such as 'geometry' or 'probability', rather than generic 'math problems'. This can simplify the complexity associated with recent CoT techniques that retrieve relevant exemplars from external data (Zhang et al., 2022b; Shum et al., 2023).
We evaluate the proposed approach in various reasoning- ceasoning-nttensvve tasks, including mathematical problem solving in GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b), code generation in Codeforces, and other reasoning tasks in BIG-Bench (Srivastava et al., 2022). We use several base LLMs: GPT-3.5, GPT-4 (OpenAT, 2023; Ouyang et al., 2022), and M2 (Anil et al., 2023). Experimental results show that our approach outperforms 0-shot CoT and few-shot CoT across range of tasks and base L.MSs, achieving an average accuracy gain of +4%. Notably, our approach improves performance on datasets involving diverse types of reasoning, such as MATH (including algebra, probability, geometry, etc.) and Codeforces (involving dynamic programming, graph algorithms, etc.). This result suggests the effectiveness of generating tailored exemplars for individual problems to guide the reasoning process of LLMs.
2
RELATED WORKS
2.1 LARGE LANGUAGE MODELS AND PROMPTING
A language model estimates probabilities over text. Recent research has scaled up these models from hundreds of millions (Devlin et al., 2019) to hundreds of billions of parameters (Brown et al., 2020) and expanded training data to include web texts and instruction data (Gao et al., 2020; Ouyang et al., 2022; Chung et al., 2022). These advances have made large language models proficient in various NLP tasks.
LLMS with hundreds of billions of parameters demonstrate in-context learning and few-shot learning abilities (Brown et al., 2020; Liu et al., 2022; Su et al., 2022; Mishra et al., 2022; Wei et al., 2022a; Yasunaga al., 2023; Shi et al., 2023). They use input prompts (instructions a few exemplars) to guide LLMs to generate desired responses for tasks, marking the advent of the prompting era. Our approach harnesses the in-context learning abilities of LLMs to guide their reasoning process using self- eeff-geneateed exemplars.
Closely related to ours are works that perform self generation in LLM prompting (Sun et al., 2022; He et al., 2023; Kim et al., 2022; Li et al., 2022a). For instance, Sun et al. (2022) prompts LLMs to recite relevant facts in context for open-domain question answering. Our idea of self self seff-geneaatin exemplars is related to recitation, but focuses on recalling problem-solving and reasoning processes rather than factual knowledge.
2

---

Published as a conference paper at ICLR 2024
2.2 CHAIN-OF-THOUGHT PROMPTING
Chain-of-thought (CoT; Wei et al. 2022b) is a prompting strategy that guides LLMs to produce intermediate reasoning steps towards final answer, enhancing problem-solving performance. Common instances of CoT include O-shot CoT (Kojima et al., 2022) and few-shot CoT Wei et al. (2022b).
0-shot CoT prompts LLMs with general instruction like "think step by step" produce intermediate rea- soning steps. Few-shot CoT achieves stronger performance by providing multiple exemplars of reasoning process caaetiionraaionllee--e labeled exemplars. Our approach tackles this challenge by prompting LLMs to self welf-generate exemplars.
leveraging LLMs' in-context learning abilities. However, it requires
Within few-shot CoT, the original approach employs fixed set of labeled exemplars for all test problems. Recent work explores retrieval- -ased CoT, which aims to obtain more relevant exemplars from external data for each problem (Zhang et al., 2022b; Shum et al., 2023). While our work shares the goal of providing relevant exemplars, instead of retrieval, we make LLMs self-generate exemplars. Self-generation offers several advantages: it is simpler, as it does not require external data retrieval, and it is more versatile, it can produce not only specific exemplars but also broader insights knowledge that complement them. Empirically, our generation-be based CoT outperforms retrieval-base -aased CoT, especially with larger base LLMs, while retrieval-based CoT excels with smaller base LLMs ($6).
Finally, there are other techniques for enhancing CoT, such as colfsconny (Wang et al., 2022) and least-to-most (Zhou et al., 2022). Our work can complement and integrate with these efforts (Table 8). Please see SA for additional related works.
3 PRELIMINARIES
We focus problem-solving tasks, where the objective is to produce a solution y for given problem statement x, such as mathematical questions or code generation specifications. The solution may include both the intermediate reasoning steps or rationale and the final answer
prompting method a function that maps a problem statement into specific textual input ((x) for an I.M,, which then generates solution (ó()). For instance,
In 0-shot prompting, é directly yields x.
In 0-shot CoT, @ supplements with general instruction, such as "[x] think step by step".
In few-shot CoT, & supplements æ with several labeled exemplars, {() such as "[o,]
[r J [a,]...
[ [rx] [ax] [x:]"
Our aim is to design prompting method o that enhances the accuracy of solutions LLMs generate.
4 APPROACH
We introduce analogical prompting, new prompting approach that automatically provides exemplars to guide LLMs' reasoning process. Inspired by how humans recall relevant past experiences when tackling new problems, approach makes LLMs self-generate relevant exemplars knowledge in context, before proceeding to solve the problem (Figure 1, right). We present two techniques to achieve this: self-ge self-generated exemplars 154.1) and self-generated knowledge exemplars ($4.2).
4.1 SFL.F-EENEATTED EXEMPLARS
Our approach is based the idea that modemn LLMs possess broad range of problem-solving knowledge acquired during training. Explicitly prompting them to recall or generate relevant problems and solutions in context aids LLMss perform in-context leamingg to solve new problems.
Specifically, given a target problem to solve x, our prompt augments with instructions like:
Problem: [x]]
Relevant problems: Recall three relevant and distinct problems. For each problem, describe it and explain the solution.
Solve the initial problem:
3

---

Published as a conference paper at ICLR 2024
For concrete example, see Figure 2. The LLM first generates several (K) relevant exemplars in the form of question-ratinn aationale-answer sequences (*# Relevant problems:" part of the instruction). Then the model proceeds to solve the initial problem, leveraging the recalled exemplars in the context ("# Solve the ini- tial problem: "oblemm" part of the instruction). Note that all these instructions are provided within a single prompt, allowing the L.M generate relevant problems and solution to the initial problem in one continuous pass. Using '#' symbols in the prompt (e.g., *# Relevant Problems') )) helps LLMs structure the response better.
Below are key technical decisions we made:
Generating relevant and diverse exemplars is important: To achieve this, we explicitly include an instruction in the prompt, such "generate problems that are distinct from each other" (e.g.. Figure 2). This step is cnucial as some L.MSs have a tendency to repetitively generate identical problems, which can be misleading when solving the target problem.
Single-pat pass vs, independent exemplar generation: An altemaatve approach is to independently generate exemplars by separately sampling them from the LLM and then re-prompting the LLM with all the exemplars. While this method does work, our current single-pass prompt approach achieves comparable performance and offers greater convenience, eliminating the need for multiple prompts. Consequently, we have chosen to adopt the single-pass method.
The number of exemplars to generate (K): Through experimentation, we have found that generating K= 3 5 exemplars works the best (more details in $6.5).
Our approach offers two advantages. It offers detailed exemplars of reasoning without manual labeling, addressing the challenges in O-shot and few-shot COT. The generated exemplars tailored to individual problems (e.g., 'geometry' or "probability", offering more relevant guidance than traditional few-shot CoT, which uses fixed exemplars (e.g. general math problems; Figure I,, middle).
4.2 SELEEGGNERATED KNOWLEDGE EXEMPLARS
While generating exemplars is useful, in complex tasks like code generation, LLMs may overly rely on the low-level exemplars and fail to generalize when solving the target problems. To address this challenge. we also allow L.Mss to self-generate high-level takeaways that complements the exemplars, which we refer as "knowledge." " Specifically, we enhance the prompt with an additional instruction like the following. For a concrete example, see Figure 3.
Tutorial: Identify core concepts in the problem and provide a tutorial
One technical consideration is whether to generate knowledge before or after exemplars. We found that generating knowledge before exemplars yields superior results (Table 7). By generating knowledge first, ILMS identify the core concepts of the problem. This, in turn, helps LLMs generate exemplars that align more closely in terms of the fundamental problem-solving approaches rather than surface-level lexical similarities. For further discussion, please refer to $6.2.
5 EXPERIMENTAL SETUP
5.1 TASKS
We evaluate the proposed approach in diverse reasoning- -intiing---eesivveeeee tasks, including mathematical problem solving, code generation, and other reasoning tasks like logical and temporal reasoning.
Mathematical problem solving. We use popular benchmarks, GSM8K (Cobbe et al., 2021). comprising elementary math word problems, and MATH (Hendrycks et al., 2021b), consisting of advanced math problems from high school math competitions. For each problem, we obtain an output from LLMs using a temperature of O, and report the accuracy.
Code generation. Code generation involves synthesizing programs to solve algorithmic problems. Competitive programming is especially challenging, requiring reasoning about various algorithms like dynamic programming and graphs (Li et al., 2022b; Kulal et al., 2019; Yasunaga & Liang, 2020).
As a benchmark, we collected competitive programming problems from codeforces.com (details in ŞB). We focus level-A problems published in 2023 to prevent test set contamination (Magar & Schwartz, 2022). Each problem comprises a problem statement, which serves as input to LLMs, and a set of test to assess generated code. The correctness of code is determined by whether it passes all test cases.
4

---

Published as a conference paper at ICLR 2024
Our Prompt: Self- gall--gnerrte Exemplars Your is to tackle natheratical problees. When presented rath problan, recoll relevent probless exaplies, Aftervord, proceed to solve the initial, problen.
Problew:
airline serves dinner all the passengers atrolane. They get their choice of steak fissh Three steek reals and rembers randenly, is the probibility
seals aside for six-morber that both pilots get the fish?
moals distributed
Instructions:
Relevant
Recall three exceples of rath problees
relevant to the initial problex. Your problems should be distinct from other and from the initial, probles (e. g., involving different numbers and nomes). For each problem:
After 'Q: describe the problex
After "A: ", explain the solution and enclose the ultimate ansver in Nbowed().
Initial 0: Copy paste the initial problom here. A: Explain the solution and enclose the uitinate arswer in \boved() here.
Output GPT3.5-turbo
Self-generated Exemplars
Relevant Problems:
o: In a beg, there are red balls and 3 blue balls, If two balls are drawn at randos wi without replacementt what is the prosability that both balls are red?
the total number of bells is 5. the probability drowing red ball the first drow is 5/5. After red bell draw, there balls. therefore, probubility drowing second
balls
probability of both product of the individual probabliities, so the probability draving two red balls (5/8) (4/7)
events haspesing (boxed( (1r3c)))11))
A box contairs 18 red marbles and 5 blue narbles. If three rarbles are drawn at randon without replacenent, what is the probability that all three naroles red!
of marble is the probabtlity of drowing total marbles. Therefore, the probability of draving a secord red warble 8/14, After two red warbles are draw, there are 8 red norble left out of 13 total warbles. Therefore, the probability of drawing a third red morale is 8/13, The probability of all three events happening is the product of the individual probabilities, so the probantl:ty of draving chree red narbles is (10/15) (9/14) (8/13) Vboved 241111))))
the
rarble on first drow 10/15, After
rortlee is
drave, there are 9 narbles left out
bag contairs green morbles, red narbles, and blue morbles. If two rarbles are drown rondor without replacement, what the probability that bath marbles are green?
As The total number of marble is 9. The probability of draving a green warble on the first draw is 4/3, After one green marble is draw,, there are 3 green marbles left out of total rarbles. Therefore, the prebability of drowing second green narble is 2/8. The prosability of both events haspening is the product of the indivioual pratabllities, the probibility of drawing two grean narbles is (4/9) (3/8) Mbovos (x66))
Let's solve the following
problem:
An airline serves a dinner to all the passengers on an airolane. They get their choice of steak or fiish Three steak reals and chree fish seals are set aside for the six rember crew. If the sells are distributed to the crea rembers randorly, y, what is the probsbility that both pilots get the fish?
probability of the first pilot gatting rish 3/6, After rish is given Therefore, probaiility the second getting 2/5. The prabability of events happering is the product the indi vidiallual protabilities, ao the probability that both piloss get the fish is (3/6) + (2/5) - Vooved Trraac(eee 1155))
There
total of
sorbers,
first pilot, there
Figure 2: Actual example of our prompt (top) and LLM output (bottom) for MATH task. Top: Our prompt supplements the problem statement with instructions to generate relevant exemplars and then solve the problem. Bottom: Exemplars generated by GPT3.5-turbo are indeed relevant to the problem, focusing on probability. It then accurately solves the problem. See $D.1 for the complete prompt and output. Using '#' symbols the prompt (e.g., '# Relevant Problems') helps LLMs structure the response better.
In line with existing work on code generation (Li et al., 2022b; Chen et al., 2023), we report the Acc@1 1 and Acc@ 10 metrics. Acc@1 measures whether at least one of the l sampled model outputs is correct. For each problem, we sample 10 outputs from LLMs, using a temperature of 0.7.
Other reasoning tasks. We further evaluate on various reasoning tasks in BIG-Bench (Srivastava al., 2022: Suzgun et al., 2022): word sorting, logical deduction five objects, temporal sequences, reasoning about colored objects, and formal fallacies. These tasks are diverse and may not have dedicated training data, so they align well with our approach of self ceff-generating custom exemplars. For each problem, we obtain an output from L.LMs using a temperature of 0, and report the accuracy.
5.2 MODELS
We experiment with several base LLMs: GPT-3.5-turbo, GPT-4 (OpenAI, 2023; Ouyang et al., 2022) (version 0613, accessed in June Sen-eeetembrr 2023), and PaLM 2-L (Anil et al., 2023).
5.3
METHOPS TO COMPARE
We compare the following prompting methods, including ours.
0-shot and 0-shot CoT. These methods, like ours, do not use labeled exemplars. We aim to show that method offers more tailored guidance for L.MM reasoning and yields superior task performance.
5

---

Published as a conference paper at ICLR 2024
Our Prompt: Self generate Exemplars + Knowledge
is
corpetitive programing aroblers. problees. Then original probl
explain
it previde
Problew:
Find out intagerr exists so that folloving conditiors are ret. n-l, mn1 1, A.k n_(eet) oun, IF there extxt miltiple that satisfy the grven condition, print the lest.
You are gévan sequance a.t. a.2, aun. slament of
a Instraction:
as Algorithes:
rdentify the
algorithex used soloe the problex.
Tutorial:
abgoriches.
Exemple Problens:
Frovide three exarples relevant conpetitive programiing problaes involve these sigorithes, For each problce, describe the problon, esplain the solution detail,
write correct Pythená code.
Python3 code salve original problem:
Explaration of the solution:
Pythord cods to solve the problem:
Outpput by GPT3.5-turbo
Algorithem: The algur ille for sulving this probilen the prefix prosiact alguritles.
Self-generated Knowledge
Tutorial:
The grafix product algoritte is used caloulace
clements in prefix The algorithe works way, right, each iteration, prosuct
initializing variable
iteracing through
elerent
with profix, clenents
variahie the snput array. ......
prefsi the array.
complexity this algortche is o6n), where 15 Self-generated Exemplars
Exieple Problens:
Probllen Givwn en arrey integers, find the product wlll elements wach subwrnay of
Solution: We can use the prefix product stgorithn to solve this problee. Ne instialize a prefix vortable to 1, and ther iterete through the array fron left to right. As each iseration, ve multiply the current elament with the profix,
def product__nf traduct_of__iieee Cffeaaaarrayy,rrr k0:
prefix =1
result []
For 1 in rangeClen(arr)):
prefis - prefis wrr[k]
k-1:
prefix prefix arn[i-ke1]]
Given an array integers, 1ind the lorgest subarray product than equal given target.
Solution: ......
3. Problen: Given on array of integers, find the swallest such that the product of all the clements in the prefix of the orraz is equal to the prosuct of all the clenents in the surtsx or the array.
Solution:
** Python3 code to solve the original problem:
Explanation of the solution:
Initialias prafix and suffix variables to and k to -1. sequence the prefix product algoritte. from right left.
Calcullaee the product of all eleeenss
Therate through the
Updoto the prefix and curfix variables.
grefix
suffis, uslate
indur.
Lnt(Arput())
rarge(t):
n= int(inpez())
are = Hast Crop(tt, (nputO)split(t)))
grefix, suftix 1,
=--n -1
for i in range(n):
pretis - prefis wrr(s]
for 1 In rangein-1, 6, -1):
puffie prefis // wi[[i]
siffis suffix arCi]]
prefix suffix: ==
print(k)
Figure 3: Actual example of our prompt (top) and LLM output (bottom) for the Codeforces task. Top: Our prompt supplements the problem statement with instructions to generate knowledge (e.g.. tutorials on core concepts) and relevant exemplars, followed by solving the original problem. Bottom: The knowledge and exemplars generated by GPT3.5-turbo prefix product algorithm. The final code generated by the LLM effectively applies the algorithm to solve the problem. See $D.3 for the complete prompt and output.
indeed relevant to the problem to solve, focusing on the
Few-shot COT. This is the standard few-shot COT, using fixed set of reasoning exemplars across all test problems within a dataset. For the GSM8K and MATH datasets, as their training sets include solutions labeled with reasoning steps, use K = exemplars from these training sets. For the other datasets, exemplars. We aim to show that our method, which reff-generates
we use = 3 exemplars, match or surpass this baseline, which uses labeled exemplars.
Fhw retrieved COT. Instead of using a fixed set of exemplars, for each test problem, we dynamically retrieve relevant labeled problem-solutinn pairs from the train set for each test problem. Specifically,

---

Published as a conference paper at ICLR 2024
GSMSK Accuracy
MATH Accuracy
Prompting Method
G- lext-davinci-003 PaLM2
GPT3.5-turbo
PaLM2
O-shot
75.0%
14.8%
60.8%
33.0%
27.1%
0-shot Col
75.8%
50.3%
78.2%
33.9%
29,8%
S-shot CoT
76.7%
54.0%
80.7%
34.9%
34.3%
Ours: Self-generated Exemplars
77.8%
61.0%
81.7%
37.3%
34.8%
Table I: Performance on mathematical tasks, GSM8K and MATH. Our prompting method, which self--enerates
exemplars, outperforms baselines such O-shot CoT and few-shot CoT. For text-davinci models, we use an in-context demonstration of how to generate exemplars. For the other models, we do not.
GPT3.5-turo- 16k
GPT4
Prompting Method
Acc I
Acc 10
Acc@! I
Acc@
0-shot
8%
24%
16%
30%
0-shot CoT
9%
27%
16%
29%
3-shot CoT
11%
27%
17%
31%
Ours: Self-generated Exemplars
13%
25%
17%
32%
Ours: Self-generated Knowledge Exemplars
15%
29%
19%
37%
Table 2: Performance on Codeforces code generation task. Our prompting method outperforms baselines such as 0-shot CoT and few-shot CoT. Moreover, self-generating knowledge provides additional gains over self-generating exemplars, demonstrating its usefulness for the challenging Codeforces task.
Word sorting
I Logical deduction five objects
Temporal sequences
Reasoning about colored objects
Formal fallacies
Prompting Method
0-shot 0-shot CoT
66.8%
30.0%
40.4%
50.4%
53.6%
67.6%
35.2%
44.8%
61.6%
55.6%
3-shot CoT
68.4%
36.4%
58.0%
62.0%
55.6%
Ours: Self-generated Exemplars
75.2%
41.6%
57.6%
68.0%
58.8%
Table 3:
Performance BIG-Bench reasoning tasks in accuracy. GPT3.5-turho is used the hase LLM. Across diverse tasks, our method outperforms baselines (0-shot CoT) and competitive with manual 3-shot CoT.
use Bentence-BERRT (Reimers & Gurevych, 2019) to encode each problem statement. For each problem in the test set, retrieve the top K =5 similar problems from the training set based cosine similarity.
Our method. We let LLMs self-generate K = 5 exemplars for GSM8K and K =3 exemplars for MATH and BIG-Bench tasks. For Codeforces, we self-generate both knowledge and K exemplars.
RESULTS
6.1
MAIN RESULTS
Mathematical problem solving. Table presents results for GSM8K and MATH tasks. Our prompting method, which self-generates exemplars, outperforms baselines such 0-shot CoT and few-shot COT. The improvement over few-shot CoT is notable for the MATH task, which involves a range of reasoning types. including algebra, probability, and geometry. This aligns with our approach of crafting tailored exemplars for each problem.
Figure and 2 provide qualitative examples of both examples, the LLM indeed generates relevant exemplars (geometry problems in Figure 1. probability problems in Figure 2), and subsequently produces correct solutions. contrast, in the standard few-shot CoT (Figure 1, middle), the exemplars are math-related (e.g., algebra) but may not always match the test problem (c.g., geometry), as the dataset contains diverse test problems.
outputs generated using our prompt. In
Code generation. Table 2 presents results for Codeforces task. Our prompting method outperforms baselines such as 0-shot CoT and few-shot CoT in both GPT3.5-turbo and GPT4. Moreover, self generatinnn knowledge provides additional performance boost over self-generating exemplars, demonstrating its usefulness for the challenging Codeforces task. With our prompting method, GPT3.5-turbe achieves competitive performance with GPT4, with 15% Acc @1 compared GPT4's 16% Acc@1 1.
Figure (more complete version in $D.3) provides qualitative example of GPT3.5-turbe output generated using prompt. The knowledge and exemplars generated by GPT3..-turbo are indeed relevant to
7

---

Published as a conference paper at ICLR 2024
Prompting Method
(t- scale down) lext-curie- 001
(scale text-davinci-001
lext-davincii
lext Seak--avini---ee 002
D-shor
14%
D-shot CoT
2%
6%
22%
50%
5-shot (fixedy CoT
25
10%
43%
54%
5-shot retrieved CoT
3%
11%
47%
57%
Ours: gelf-geeratee Exemplars
2%
9%
48%
61 %
Table 4: Performance analysis using GSM8K task. Across varied scales strengths of base LLMs (increasing
from left to right), our prompting method outperforms O-shot CoT and standard few-shot CoT with fixed exemplars. Self-generated exemplars vs. retrieved exemplars: our method, with self generated exemplars, performs better with larger-scale LLMs, while few-shot CoT with retrieved exemplars performs better with smaller-scale LLMs.
the problem to solve, focusing on the prefix product algorithm. The final code generated by the L.M effectively applies the algorithm solve the problem. In contrast, in the O-shot Co'T' baseline, the L.LM output does not recall relevant exemplars and fails to employ the prefix product algorithm, resulting in an incorrect solution (SD.3).
BIG-Bench reasoning tasks. Table 3 presents results for BIG-Bench tasks. Our prompting method outperforms baselines like 0-shot CoT, confirming its effectiveness across a wide range of tasks. Our method is also competitive with manual few-shot COT. $D.4 offers GPT3.5-turbo output examples for the deductive reasoning task ("BIG-Bend formal fallacies"). Using our prompting method, the L.LM generates relevant deductive reasoning exemplars. Conversely, O-shot Co'T, with no relevant exemplars, tends to adopt an incorrect approach address the deductive reasoning problem.
6.2 KNOWLEDGE CAN COMPLEMENT EXEMPLARS
Generating knowledge alongside exemplars is particularly useful in Codeforces task (Table 2), where LLMs need to apply nontrivial algorithms for code generation. In our qualitative analysis, we observe two concrete advantages of generating knowledge: (1) knowledge act as high-level takeaways that complement low-level exemplars, which prevents LLMs from overly relying on specific exemplars and helps to generalize new problems: (2) when generating knowledge. LLMs identify the core concepts of the problem and produce exemplars that align more closely in fundamental problem-solving approaches (e.g., the prefix product algorithm in Figure 3), rather than surface-level lexical similarities (e.g., without knowledge, LLMs tend to produce exemplars palindromic sequences).
The performance gains achieved by generating knowledge are less significant in other tasks like GSM8K and BIG-Bench, however, likely because these tasks are less complex.
6.3 GENERATING VS RETRIEVING EXEMPLARS
A key motivation behind our idea of self-generating exemplars is its ability to offer relevant exemplars for problem solving. An alternative approach retrieve relevant exemplars from external data, provided there is a labeled dataset of exemplars (e.g., the training set of GSMSK, which includes solutions labeled with reasoning steps). What trade-offs exist between these two approaches?
The advantage of retrieval lies in its reliability. Exemplars retrieved from a labeled dataset are inherently valid and correct, unlike generated exemplars, which lack this guarantce. Nevertheless, retrieval typically needs labeled exemplars and involves the complexity of an additional retrieval step. The performance may also depend on the specific retriever used.
In contrast, generation is more self-containes and convenient, as it does not rely on external labeled data or retrieval steps. Additionally, generation may yield exemplars better tailored to specific test problems because it can draw upon the entire (pre- pire))training data the LLM has been exposed to. The downside of generation is that may fail to produce valid exemplars the LLMs are weak or have not learned problems related to the ones to be solved.
Table shows empirical results for GSMSK task, comparing our self self-generated exemplars method ("Ours") and the few- CoT method using exemplars retrieved from the GSM8K train set ("5-shot retrieved CoT"). We conducted experiments using base LLMs of various scales, from text-curie-00) to text-davinci-003, where scale broadly indicates the amount of training data and parameter count used by the LLM.
Our method outperforms the retrieved CoT with larger-scale LLMs, such as text-davinci-O0 This is likely because the I.M has effectively learned related tasks during training and can generate useful
8

---

Published as a conference paper at ICLR 2024
exemplars. Conversely, with smaller-scale LLMs, the retrieved CoT performs better, and self- self-gennration fails to produce useful or valid exemplars.
6.4 SCALE OF BASE LLMs: ANALOGICAL PROMPTING EXCELS WITH LARGER MODELS
Table presents the result using varying scales and strengths of base LLMs, ranging from text-curic-001 to text-davinci-001 to text-davinci-002 and text-davinci-003 (more parameters and training data). Our prompting method surpasses vanilla 0-shot and O-shot CoT across all scales. When using smaller-scale LLMs (text-curie-00) and text-davinci-001) few-shot CoT leveraging labeled exemplars exhibits superior performance compared ours. However, as the LLMs are scaled up to text-davinci-002 and text-davinci-003, method outperforms few-shot COT. This is due the LLMs' enhanced ability self-generate more relevant and useful exemplars.
# Exemplars self-generate
6.5 NUMBER OF EXEMPLARS TO GENERATE
GSMSK
MATH
K=1
76.1
34.8
In Table 5, we analyze the effect of varying the number of self-generated exemplars (K) in our approach. When K=1, the LLM underperforms due to excessive reliance on single exemplar generated. When K 3, the LLM demonstrates con- sistent performance. This observation aligns with the findings the standard few-shot in-context learning in LLMs (Brown
K=2
77.0
36.7
K=3
77.5
37.3
K=4
77.3
K=5
77.8
37.1
Table 5: Analyzing the effect of varying the number self-generated exemplars (K) our approach. We assess performance on GSM8K and MATH tasks using GPT3.5- turbo the base LLM.
et al., 2020).
6.6 QUALITATIVE ANALYSIS
We manually analyzed the performance of our prompting approach, based on a random sample of 50 correctly and 50 incorrectly solved problems from GSMSK MATH (50%, 50%).
50 correctly solved problems:
(6/50) Generated exemplars are irrelevant
(9/50) Generated exemplars are relevant but contain incorrect solutions
(35/50) Generated exemplars are relevant and correct
50 incorrectly solved problems:
(10/50) Generated exemplars are irrelevant
(12/50) Generated exemplars relevant but contain incorrect solutions
(28/50) Generated exemplars are relevant and correct, but L.LM fails to solve the new problem:
(12/50) A generalization gap between the exemplars and the new problem
(8/50) Overreliance on specific exemplars, leading misdirection
(8/50) Other issues, such as calculation errors
The generated exemplars were often relevant or correct. common failure occurred when the LLM could not solve the new problem due to generalization gap (c.g.. the problem is harder than the exemplars). This observation motivates future research to generate exemplars that not only possess relevance but also facilitate generalization for solving new problems.
7 CONCLUSION
We introduced analogical prompting, a new language model prompting approach that self-generates relevant reasoning exemplars for solving problems. This approach provides detailed, customized exemplars for individual problems without requiring labeled data, effectively addressing the challenges faced by existing 0-shot CoT and few-shot CoT prompting methods. Experimental results show that approach outperforms 0-shot CoT and few-shot CoT in various reasoning tasks, including math problem solving, code generation, and other logical/temporal reasoning tasks.

---

Published as a conference paper at ICLR 2024
8 LIMITATIONS AND FUTURE RESEARCH
One limitation of our approach is increased inference computation, as our approach generates more tokens than vanilla 0-shot and 0-shot CoT prompting. Compared to few-shot CoT. we use fewer input tokens and more output tokens, as exemplars are counted as input in few-shot CoT and as output in our approach.
Another limitation is that self- generatteninn can fail the LLM lacks sufficient strength or has learned relevant knowledge to the new problems to solve. Conversely, with stronger L.M,, it can draw upon relevant prior knowledge to tackle slightly more complex problems. Therefore, our approach better suited for stronger or larger-scale LLMs.
Finally, it known that LLM performance can be influenced by specific prompt phrases used to query the model (Jiang et al., 2020), and our work is also subject to this prompt sensitivity.
ACKNOWLEDGMENTS
We thank Qian Huang. Chen Liang, Andrew Dai and other colleagues Google DeepMind. Stanford SNAP and P-Lambda groups for valuable feedback on this work.
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arXiv preprint arXiv:2305. 10403, 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.077 2021.
Jiang Bian, Bin Gao, and Tie- Yan Liu. Showledge-powered deep learning for word embedding, In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Purt 14, pp. 132-148. Springer, 2014.
Léon Bottou. From machine learning to machine reasoning: An essay. Machine learning, 94: 1333149,, 2014.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry. Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Eisuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https: wttps////cocceee...ooom. neurips. cttplpp..cc/paaeee cc/paper///ecceceeecceeeeeee. 1 22000/2a2h///// html.
1457c8d6bf Bbb
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprini 2r0ivv2305.1 17/26, 2023.
Jaime Carbonell. Learning by analogy: Formulating and generalizing plans from past experience. In Machine learning, pp. 137-161. Elsevier, 1983.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0334,, 2021. URL
2107.03374
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disen- tangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 2022.
10

---

Published as a conference paper at ICLR 2024
Xinyun Chen, Chang Liu, and Dawn Song. Execution Execution-guided neural program synthesis. In International Conference on Learning Representations, 2018.
Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoe V Le. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference Learning Representations, 2019.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke 7ettlemoyer, et al. Binding language models in symbolic languages. arXiv preprint arXiv:22/0.02877 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Issyashchenko,, Joshua Maynez, Abhishek Rao, Parker Bares,, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan. Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL https: https///axxxvvvrooeeeoo
2204. 02311.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finttun language models. arXiv preprint arXiv:2210.1144 1411, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxivv http:/www.oo//b///111100/ 14168.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computaational Linguistics: Human Language Technologies, Volume / (Long and Short Pupers), pp. 4171-41866 Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.11863/v1/N19--11 1423. URL URLLhttps://aclaatholoo org/N19-1423.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song. Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. arXiv preprint arXiv:2209. 15003, 2022.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.0016: 2019.
Kevin Dunbar. The analogical paradox: Why analogy is so easy in naturalistic settings yet difficult in the psychological laboratory. The analogical mind: Perspectives from cognitive science, pp. 313-334, 2001.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. Scalable multi- hop relational reasoning for knowledge-aware question answering. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv: 00x0022001.0002, 2020. URL https://arxiv http::///aaxiv.org/a/////aae wwt.:o////bb//xii-211ee/00 21010000077.
11

---

Published as a conference paper at ICLR 2024
Dedre Gentner. Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2):55-1700 170. 1983.
Dedre Gentner and Keith Holyoak. Reasoning and learning by analogy: Introduction. psychologist, (2)1);3 1997.
American
Dedre Gentner and Arthur B Markman. Structure mapping in analogy and similarity. American psycholo- gist, 52(1):45, 1997.
Kavi Gupta, Peter Ebert Christensen, Xinyun Chen, and Dawn Song. Synthesize, exccute and debug: Learning to repair for neural program synthesis. Advances in Neural Information Processing Systems, 17695, 2020.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023,
Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujuu Yang, Rui Wang. Zhaopeng Tu, Shuming Shi, and Xing Wang. Exploring human-like translation strategy with large language models. arXiv preprint arXiv:2005.04118 2023.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burs, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint a 20211.
Keith Holyoak. Analogy and relational reasoning. The Oxford handbook of thinking and reasoning, pp. 234-259, 2012.
Xiaoyang Hu, Shane Storks, Richard I. Lewis, and Joyce Chai. In-context analogical reasoning with pre-trained language models. arXiv preprint arXiv:2305.17022 2023.
Ziqi Huang, Hongyuan Zhu, Ying Sun, Dongkyu Choi, Cheston Tan, and Joo-Hwee Lim. A diagmostic study of visual question answering with analogical reasoning. In 2021 IEEE International Conference on Image Processing (ICIP), pp. 2463-2467. IEEE, 2021.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-433 438, 2020.
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maicutic prompting: Logically consistent reasoning with recursive explanations, arXiv preprint
1822 2022.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li. David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-seench-ppeadd predccc Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212. 14024, 2022.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint ar3iv:2303. /749/, 2023.
Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. Self- generated in-context leaming: Leveraging auto-regressive language models a demonstration generator. arXiv preprint arXiv:2206.0888 080822 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Thirty-sixhh Conference on Neural Information Processing Systems (NeurIPS 2022), 2022. URL htRthttss://aaee http:/:://aaaww.o/////aass/////s a225 11916.
12

---

Published as a conference paper at ICLR 2024
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search- based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
Junlong Li, Zhuosheng Zhang, and Hai Zhao. Self- prommprtpiingg large language models for open-domain qa.
arXiv preprint arXiv:22/2.08635, 2022a.
Yujia Li, David Choi, Junyoung Chung. Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling. Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624); 1092-1097, 2022b.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, al. Holistic evaluation language models. arXiv preprint arXiv:2211.091100 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, and Ilya Sutskever. Improving mathematical rea- soning with process supervision. 2023. URL https://openai. com/research/// improving
setth-orocessssuprriisses supervision.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Empirical Methods in Natural Language Processing (EMNLP), 2019.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100-114, 14, 2022. doi: 10. 00.18653w/2022..eeeee-....0 URL https: /atpatoo//ccannthlooygy 002/2022. deelio-1.10.
Inbal Magar and Roy Schwartz. Data contamination: From memorization exploitation. arXiv preprint wrXiv:2203.822// 2022.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-tast generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pp. 3470-3487, 2022. doi: 10. 1/20://1/22222clllooog/
URL https: / //tss//thllaalog .rg/ 2022. acl-long. 244.
Tom M Mitchell, Jaime G Carbonell, Ryszard S Michalski, and Rogers Hall. Analogical reasoning in the context of acquiring problem solving expertise. Machine Learning; Guide to Current Research, pp. 85 88, 1986.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353- 367. PMLR, 2023.
OpenAL. Gpt-4 technical report. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 15:27730-27744, 2022.
Joon Sung Park, Joseph C O''Brient Carrie J Cai, Meredith Ringel Morri,, Percy I. Liang, and Michael Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiw:2304.034442 2023.
George Polya. How to solve it: A new aspect of mathematical method, volume 85, Princeton university press, 2004.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt. Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350. 2022.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint orX.v.230..06477, 2023.
13

---

Published as a conference paper at ICLR 2024
Nils Reimers and Iryna Gurevych. Sentence-BERI BERT: Sentence embeddings using Siamese BERT networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interational Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pp. 3982-3992, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10. 10.11865/1//11- 1410. URL https://aallanthologg - org org/D19-144
Hongyu Ren, Hanjun Dai, Bo Dai, Xinyun Chen, Michihiro Yasunaga, Haitian Sun, Dale Schuurmans, Jure Leskovec, and Denny Zhou. Lego: Latent execution execution-guided reasoning for multi-hop question answering on knowledge graphs. In International Conference on Machine Learning (ICML), 2021.
Timo Schick, Jane Dwivedi-Yo Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.1263: 2023,
KaShun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with chain-of-thought from labeled data. arXiv preprint 2rXiv:2302./2282, 2023.
Aarohi Srivastava, Abhinay Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alnso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:220600611, 2022.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975 2022. URL https: / /awxiv.ivvvvv//////eee/aaaa org/ abep//2/s....///////////// 2209. 01 97575.
Zhiqing Sun, Xuezhi Wang, Yi Tay. Yiming Yang. and Denny Zhou. Recitation augmente language models. arXiv preprint urXiv:2210.0129h 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung. Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint 1rX:v22210099261, 2022.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:18111.00997 2018.
Stella Vosniadou and Andrew Ortony. Similarity and analogical reasoning. Cambridge University Press, 1989.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 1rXiv:2203.1117/ 2022.
Thomas B Ward, Steven M Smith, and Jyotsna Ed Vaid. Creative thought: An investigation of conceptual structures and processes. American Psychological Association, 1997.
Taylor Webb, Keith Holyoak, and Hongjing Lu. Emergent analogical reasoning large language models. Nature Human Behaviour, pp. 1-16, 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. International Conference on Learning Representations, 2022a. URL /ttps://oppnreeview net het/forummidd-gEZrrooo
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, Chain of thought prompting clicits reasoning in large language models. In Thirty-sixhh Conference on Neural Information Processing Systems (NeurIPS 2022), 2022b. URL https: //arxiv...v////// www.ivoooo///oooo abs/ 10001..19100 11903.
Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pp. 11251-11261 PMLR, 2021.
14

---

Published as a conference paper at ICLR 2024
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien- Sheng Wu, Ming Zhong, Pengcheng Yin, Sida Wang, et al. Unifiedskg: Unifying and multi- mulli-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.055 2022.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: (800:099000000 2018.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 2022.
Shunyu Yao, Dian Yu, Jeffiey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 10601, 2023.
Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. In International Conference Machine Learning (ICML., pp. 10799-10808... PMLR, 2020.
Michihiro Yasunaga and Percy Liang. Break-it-fix-it: Unsupervised learning for program repair. In International Conference on Machine Learning, pp. 11941-11952. PMLR, 2021.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
Michihiro Yasunaga, Antoine Bossclut, Hongyu Ren, Xikun Zhang, Christopher Manning, Perey Liang, and Jure Leskovec. Deep bidirectional Neural Information Processing Systems, 35:37309-37322 2022a.
graph pretraining, Advances in
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. LinkBERT: Pretraining language models with document links. Association for Computational Linguistics (ACL), 2022b.
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zeutlemoyer, and Wen-tau Yih. Retrieval- -ugmented multimodal language modeling. In International Conference on Machine Learning (ICML), 2023.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning labeled dataset for complex and cross cross- domain
Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled semantic parsing and text-to-sql task. arXiv preprint arXiv: 1809,08887 2018.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D Goodman, and Nick Haber. Parsel: A unified natural language framework for algorithmic reasoning. arXiv preprint arXiv:2212. 2022.
Hanlin Zhang, Jiani Huang, Ziyang Li, Mayur Naik, and Eric Xing. Improved logical reasoning of language models via differentiable symbolic programming. arXiv preprint arXiv:2305.03744 23000032.. 2023.
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning, and Jure Leskovec. Greaselm: Graph reasoning enhanced language models for question answering, In International Conference Learning Representations (ICLR), 2022a.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.034493 2022b.
Wenting Zhao, Mor Geva, Bill Yuchen Lin, Michihiro Yasunaga, Aman Madaan, and Tao Yu. Complex reasoning in natural language. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pp. 11-20, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653w//2022.aallutttat aclanthology.org hwwpolllooooooog//ggoeooe www. 2023.2222222 tutorralss....
URL https://
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song. Mingjic Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-hased self-verification. arXiv preprint arXiv:2308.07921, 2023.
15

---

Published as a conference paper at ICLR 2024
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint 2wXh:22005.10625, 2022. URL https://arxxv..rr//aat 10625.
A ADDITIONAL RELATED WORKS
A.1 LANGUAGE MODELS AND REASONING
Reasoning involves the application of knowledge to derive solutions for new problems, often through a series of steps. Teaching language models to reason has been long-standing area of research (Bottou, 2014; Zhao et al., 2023; Wei et al., 2022b).
To assess the reasoning capabilities of language models, researchers have created datasets for various tasks that demand reasoning skills. These tasks include multi-step question answering (Yang et al., 2018; Dua et al., 2019; Talmor et al., 2018), mathematical problem-solving (Cobbe et al., 2021; Hendrycks et al., 2021b), and code generation (Yu et al., 2018; Chen et al., 2021; Hendrycks et al., 2021a; Austin et al., 2021). this study, we evaluate our methods using these diverse datasets.
To teach language models to reason effectively, one line of approaches involve training fine-tuning them. This can include using reasoning-intensive data during training (Wu et al., 2021; Yasunaga et al., 2022b; Lightman et al., 2023; Moor al., 2023), retrieving structured knowledge (Lin et al., 2019; Feng et al., 2020; Zhang et al., 2022a; Yasunaga et al., 2021; 2022a; Xie et al., 2022), and incorporating external modules for reasoning such as logic and program execution (Chen et al., 2018; 2019; Yasunaga & Liang, 2020; Gupta et al., 2020; Ren et al., 2021; Zhang et al., 2023).
Recently, with the rise of large language models (LLMs), prompting them to engage in reasoning has proven effective and gained attention. common approach is prompting LLMs to generate intermediate reasoning steps, as demonstrated by the chain-of-thougle method (Wei et al., 2022b; Kojima al., 2022; Zhou et al., 2022; Wang et al., 2022), which assists L.MMs in tackling complex reasoning tasks. Several studies have extended this approach with more structured algorithuss and search methods (Khot et al., 2022; Drozdov et al., 2022: Zelikman et al., 2022; Yao et al., 2023; Press et al., 2022; Khattab et al., 2022; Jung et al., 2022), as well as honger-horizon action and planning (Yao al., 2022; Hao et al., 2023; Park et al., 2023). Another line of work incorporates tools and programs into the prompting process to facilitate reasoning (Chen et al., 2023; 2022: Cai al., 2023; Cheng et al., 2022; Kim al., 2023; Zhou et al., 2023; Schick et al., 2023).
Our work complements these efforts to enhance LLM reasoning and the first to draw inspiration from human analogical reasoning to improve L.M prompting.
A.2 ANALOGICAL REASONING
Analogical reasoning is cognitive process in which humans recall relevant past experiences when facing new challenges (Gentner & Holyoak, 1997; Gentner, 1983; Holyoak, 2012). This phenomenon has been studied extensively in psychology, revealing its significance various cognitive tasks such as problem- solving (Gentner & Markman, 1997) and creativity (Ward et al., 1997). It is rooted in the capacity to identify structural and relational similarities between past and current situations, facilitating knowledge transfer (Dunbar, 2001).
Analogical reasoning has also influenced the development of artificial intelligence and machine learning algorithms (Carbonell, 1983; Mitchell et al., 1986) and has been employed reasoning benchmark for assessing machine learning models (Bian et al., 2014; Huang et al., 2021). A recent work also evaluates the ability of language models to identify analogies (Webb et al., 2023; Hu et al., 2023).
Our work makes a pioneering effort of applying analogical reasoning principles to enhance language model inference.
16

---

Published as a conference paper at ICLR 2024
B CODEFORCES DATA COLLECTION
We scraped data from codeforces.com com, following the procedure in prior works (Li et al., 2022b; Kulal et al., 2019; Yasunaga & Liang, 2021). We use Level-A problems that published between January 2023 and August 2023. Each problem includes the full problem descriptions and test cases accessible on the website. The test cases include the public test cases found on the problem page and hidden test cases made available on the evaluation result pages once contest is finished. Some of the hidden test cases were truncated on the website due excessive input/output length, and we skipped those. We retained problems whose problem descriptions were within the length of 2000 tokens in GPT3.5-turbo, resulting in 50 problems. Because this dataset relatively small, we conduct the evaluation twice and then report the average results.
C ADDITIONAL RESULTS
Prompting Method
GSM8K
MATH
Ours: Non-diverse exemplars
75.9%
35,2% 37.3%
Ours: Diverse exemplars
77.8%
Table Ablation study on effect of encouraging diversity self-generating exemplars ($4.)
Prompting Method
Acc@1
Acc@) 10
Ours: Self-generate knowledge after exemplars
14%
27%
Ours: Self Self-generate knowledge before exemplars
15%
29%
Table 7: Ablation study on Codeforces using GPT3.5-turb. Self-generating knowledge before exemplars performs slightly better than self- self-generating knowledge after exemplars ($4.2).
Prompting Method
GSMBK
MATH
Ours
77.8%
37.3%
Ours Sclf-consistency
85.3%
46.0%
Table 8: Self-consistency can be used to further boost the performance of our analogical prompting approach.
D FULL EXAMPLES
D.I MATH
Our prompt (self-generate exemplars):
Your task is to tackle mathematical problems. When presented with math problem, recall relevant problems examples. Afterward, proceed solve the initial problem.
# Initial Problem:
An airline serves dinner to all the passengers
airplane. They get their choice of steak or fish. Three steak meals three fish meals are set aside for the six-member crew. If the meals are distributed to the crew members randomly, what is the probability that both pilots get the fish?
Instructions:
Make sure to include all of the following points:
## Relevant Problems:
Recall three examples math problems that are relevant initial problem. Note that your problems should be distinct from each other and from the initial problem (e.g., involving different numbers and names). For each problem:
After "Q: describe the problem
17

---

Published as a conference paper at ICLR 2024
After "A: explain the solution and enclose the ultimate answer in boxed{}.
## Solve the Initial Problem:
Say "Let's solve the following math problem. Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in (boxed here.
Output by GPT3.5-turb::
Relevant Problems:
Q:
bag, there are red balls and 3 blue balls. If balls drawn at random without replacement, what the probability that both balls are red?
A: The total number of balls is 8. The probability of drawing red ball the first draw is 5/8. After of 7 total balls. Therefore, the probability of drawing a second red ball is 4/7. The probability of both events happening is the product of the individual probabilities so the probability of drawing two red balls is (5/8) x (4/7) \boxed(\frac (5}(14))
red ball is drawn, there
balls left
Q: box contains 10 red marbles and 5 blue marbles. If three marbles are drawn at random without replacement, what is the probability that all three marbles
red?
A: The total number of marbles is 15. The probability of drawing red marble on the first draw is 10/15. After one red marble is drawn, there are 9 red marbles left out of 14 total marbles. Therefore, the probability of drawing second red marble is 9/14. After two red marbles drawn, there marbles. Therefore, the probability of drawing third red marble 8/13. The probability of all three events happening is the product of the individual probabilities, so the probability of drawing three red marbles is (10/15) (9/14) (8/13) = (baxe(((rra
8 red marbles left out of 13 total
Q: bag contains green marbles, red marbles, and blue marbles. If two marbles are drawn at random without replacement, what is the probability that both marbles are green?
A: The total number of marbles is 9. The probability of drawing green marble on the first draw is 4/9. After one green marble is drawn, there are 3 green marbles left out of 8 total marbles. Therefore, the probability of drawing second green marble is 3/8. The probability of both events happening is the product of the individual probabilities. so the probability of drawing two green marbles is (4/9) (3/8) (16
Let's solve the following math problem:
An airline serves dinner to all the passengers
airplane. They get fish. Three steak meals three fish meals are set aside for the six-member crew. If the meals are distributed to the crew members randomly, what is the probability that both pilots get the fish?
their choice of steak
A: There a total of 6 crew members, so the probability of the first pilot getting the fish first pilot, there are 2 fish meals left out of 5 total meals.
3/6. After one fish meal is given to the
Therefore, the probability of the second pilot getting the fish is 2/5. The probability of both events happening is the product of the individual probabilities. so the probability that both pilots get the fish is (3/6) * (2/5) \boxed(
18

---

Published as a conference paper at ICLR 2024
D.2 GSM8K
Our prompt (self-generate exemplars):
Your task to tackle mathematical problems. When presented with math problem, recall relevant problems as examples. Afterward, proceed to solve the initial problem.
Initial Problem:
Kyle bought last year selling book for $19.50. This is with 25% discount from the original price. What was the original price of the book?
Instructions:
Make sure that your response follows the instructions below.
## Analogous Problems:
Offer five diverse examples of math problems that are relevant
analogous to the initial problem. For each problem, elaborate the solution and conclude with the ultimate answer (enclosed in \boxed()) For each problem:
After "Q: describe the problem
After "A: explain the solution and enclose the ultimate answer in boxed
## Solve the Initial Problem:
Q: Copy and paste the initial problem here.
Explain the solution and enclose the ultimate in \boxed here.
Output by GPT3.5-turb::
Analogous Problems:
Q1: shirt is on sale for 28% off the original price of $48. What the sale price of the shirt?
A: The original price of the shirt is $48. With a 20% discount, the discount amount is 20% of $40, which is 0.20 * $40 $8 Therefore, sale price the shirt (32}s.
$8 $32. The answer $\boxed
Q2: A store is offering 15% discount on all items. If customer buys pair of shoes for $85, what the original price of the shoes?
A: Let assume the original price of the shoes discount, the discount amount is 15% of $x, which is 0.15 $x $0 .5xx. The sale price of the shoes is $85, so we can set up the get 0.85x
$x. With 15%
equation: $x $0.15x $85. Simplifying the equation, find that x $100. Therefore,
$85. Dividing both sides by 0.85, original price the shoes was $1boxed
Q3: A restaurant offers a 10% discount on all meals. If customer pays $45 for meal, what was the original price of the meal?
A: Let's assume the original price of the meal is $x. With a 10% discount the discount amount is 10% of $x, which is 0.10 5x 10x10x. The sale price of the meal is $45, so we can set up the equation: $x s0 get 0.90x $45. Dividing
10x $45. Simplifying the equation, $50. Therefore, the original
both sides by 0.90, we find that price of the meal was $iboxed
04: A laptop is on sale for 30% off the original price of s800. What is the sale price of the laptop? A: The original price discount amount is 38% of $800, which is 0.30 $800 $240. Therefore, the sale price of the laptop is $800 $248 = $560. The answer is (560)5.
the laptop
s800. With discount, the
19

---

Published as conference paper at ICLR 2024
Q5: toy store is offering 12% discount all toys. If customer
buys toy for $22, what was the original price of the toy?
A: Let assume the original price of the toy is $x With a 12% discount, the discount amount is 12% of $x, which is 0.12 * $x $0 12x. The sale price of the toy is $22, so we can set up the equation: $x $0 12x $22. Simplifying the equation, both sides by 0.88, we find that x $25. Therefore, the original price of the toy was (25)5.
get 0.88x $22. Dividing
Solve the Initial Problem:
Q: Kyle bought last year's 's best-selling book for $19.50 50. This is with a 25% discount from the original price. What was the original price of the book?
A: Let's the discount amount is 25% of $x, which sale price of the book is $19.50, so we can set up the equation: $x $0.25x $19.50 50. Simplifying the equation, we get 0.75x $19.50.
the original price of the book is $x. with 25% discount
0.25 5x $8 35.25x. The
Dividing both sides by 0.75, we find that $26. Therefore, the original price of the book (26)S.
Our prompt (self-generate exemplars) with in-context demonstration, used by text-davinci models:
Your task is to solve math problems.
For example,
# Problem:
Q: pet shelter spends $30 grooming for each cat, $60 for each adult dog, and $98 for each puppy prepare them for adoption. If they prepare cats, adult dogs, and 4 puppies for adoption, what is the total cost of getting them ready?
Relevant problems:
Q: a bakery, there a sale of baked goods. Every muffin costs $2, every cookie $1, and a loaf of bread $3. How much you need to pay for five muffins, six cookies, and two loaves of bread?
A: Let's calculate the cost for each type of item:
Five muffins cost: 5 $2 $10.
Six cookies cost:
$6.
Two loaves of bread cost: 2 $3 56.
Now, add up the costs for each item to find the total cost: Total cost $10 (muffins) $6 (cookies) $6 (bread) = $22.
So, for five muffins, need to pay (Nboxed([$22)).
cookies, and two loaves of bread, you would
Q: In garden, there are four different types of fruit trees. The first type of tree has 15 apples, the second type has 10 oranges, the third type has 24 pears, and the fourth type has 30 plums. Sarah decides to pick fruit from these trees for making a fruit salad. She picks 48% of the apples, 60% of the oranges, 75% of the pears, and 5ex of the plums. How many pieces of fruit are in her fruit salad? A: Number
apples Sarah picks:
x 40%
apples.
Number of oranges Sarah picks: 10 x 60% = 6 oranges.
Number of pears Sarah picks: 24 x 75% 18 pears.
Number of plums Sarah picks: 30 x 50% 15 plums.
Now. let's calculate the total number of pieces of fruit in her fruit salad:
Total 6 apples + oranges 18 pears + 15 plums = pieces of fruit.
18 15 45
So, there are (Wboxed((45)) pieces of fruit in Sarah 's fruit salad.
20

---

Published as a conference paper at ICLR 2024
Q: Sarah small jewelry business and sells necklaces for 25% than costs to make them. If she sold 40 necklaces and earned $400, how much does each necklace cost to make?
A: To find out how much each necklace costs to make, we can start by determining the selling price per necklace.
First, need calculate the selling price per necklace. Since Sarah sold necklaces and earned $480, we can use Selling Price per Necklace $400 40 $10
formula:
Now that we know each necklace was sold for $10, we can find out the cost than their cost
make each necklace. Sarah sells them for 25%
Cost to Make each Necklace $10 / (1 0.25) $10 / 1.25 $8
so, each necklace costs [boxed to make.
Q: machine
coffee shop usually brew pot of coffee in 6 minutes. However, this morning malfunction occurred, and it took 10 minutes to brew one pot of coffee. If the malfunction remains unresolved, how much longer will it take to brew 8 pots of coffee than normal day when the machine works fine?
A: On regular day, it would take 8 48 minutes to brew 8 pots of coffee.
With the malfunction, it will take 10 8 80 minutes to brew 8 pots of coffee.
It will take 80 48 ((32) more minutes if the malfunction goes unresolved.
Q: In jar, there were 800 candies. Sarah took 3/8 of the total number of candies, while David ate 2/5 of the remaining candies. What percentage of the original candies remained?
A: Sarah took 3/8 the total candies, which is 3/8 800 300 candies.
After Sarah s maddyttaking,. there were 800 300 500 candies left. David then ate 2/5 of the remaining candies, which is 2/5 * 500 200 candies.
candies left. The percentage of candies that remained out the original amount
After David's candy
there were 500 200
(300 800) 100% = (337.5
Solve the initial problem step step: Q: pet shelter spends $30 on grooming dog, and 590 for each puppy to prepare them for adoption. If they prepare 3 cats, 2 adult dogs, and 4 puppies for adoption, what is the total cost of getting them ready?
each cat, $60 for each adult
A: find the total cost of getting these animals ready for adoption,
'11 multiply the cost of each type of animal by the number of that type being prepared:
Cost for 3 cats
$30 $90
Cost for 2 adult dogs
$60 $128
Cost for 4 puppies 4
$360
Now, add up the costs for each type:
Total cost $90 + $120 + $360 = $570
So, the total cost to get cats, adult dogs, and 4 puppies ready for adoption is (bboxed[[$570)).
Solve the following math problem step by step.
Problem:
Q: Kyle bought last year 's best-selling book for $19.50. 50. This is with a 25% discount from the original price. What was the original price of the book?
Relevant problems:
21

---

Published as a conference paper at ICLR 2024
D.3 CODEFORCES
Our prompt (self-generate exemplars and knowledge):
Your goal to write Python3 code to solve competitive programming problems. Given problem, explain the core concepts in it and provide other relevant problems. Then solve the original problem.
Problem:
You are given sequence a_1, a_2, Aldots, a.n. Each element of is
2. Find out if an integer k exists so that the following conditions are met
1 Aleq k Vleq n-1, and a_l Vedot a_2 Aedot Aldots Vcdot a_k = a_(k+1) \edot a_(k+2 Nedot Aldots \edot a_n, where Vedot denotes product
If there exist multiple k that satisfy the given condition, print the smallest.
Input
Each test contains multiple test cases. The first line contains the number of test cases t (1 Ale t Ale 100). Description of the test cases follows.
contains one integer Aleq Aleq The second line of each test case contains n integers a_1, a_2, Mldots, a_n (1 Aleq a_i Aleq 2).
The first 1ine of each test 1000).
Output
For each test case, if there no such k, print -1. Otherwise, print the smallest possible k.
Instruction:
## Algorithms:
Identify the core concepts or algorithms used to solve the problem.
## Tutorial:
Write a useful tutorial about these algorithms.
## Example Problems:
Provide three examples of relevant competitive programming problems that involve these algorithms. For each problem, describe the prablem, explain the solution in detail, and then write the correct Python3 code.
## Python3 code to solve original problem:
Include the following points in your response: Explanation of the solution:
Python3 code to solve the problem:
Output by GPT3.5-turbo-66k:
## Algori thms:
The core algorithm for solving this problem is the prefix product algorithm.
## Tutorial:
The prefix product algorithm is used to calculate the product of all the elements in prefix of array It is useful algorithm for solving problems that involve calculating product all elements in a subarray.
The algorithm works by initializing variable "prefix" to 1, and then iterating through the array from left to right. At each iteration,
22

---

Published as a conference paper at ICLR 2024
the current element is multiplied with the prefix, and the result is stored in the prefix variable. This way, the prefix variable keeps track of the product of all the elements in the prefix of the array.
Here is the pseudocode for the prefix product algorithm:
prefix
for i in range(n): prefix prefix arr[i]
The time complexity of this algorithm is O(n), where is the size of the input array.
## Example Problems:
1. Problem: Given an array of integers, find the product of all the elements in each subarray of size k.
Solution: We can problem.
the prefix product algorithm to solve this initialize prefix variable to 1, and then iterate right. At each iteration, we
through the array from left multiply the current element with the prefix, and store the result in the prefix variable. If the index i is greater than or equal to k-1, divide the prefix by the element arr[i-k+1], which represents the first element of the subarray of size k starting at index i-k+l. The result is the product of all the elements in the subarray of size k starting at index i-k+1. We can store these results in new array and return it the output.
"'yython
def = len(arr)
k):
prefix = result []
for i in range(n):
prefix prefix * arr[i]
if k-1:
result. append prefix prefix // arr[i-k+1]
return result
2. Problem: Given array of integers, find the longest subarray with given target.
product less than equal product algorithm Solution: can use the prefix
solve this problem. We initialize a prefix variable to 1, and then iterate through the array from left to right. At each iteration, we multiply the current element with the prefix, and store the result the prefix variable. If the prefix becomes greater than the target, we divide the prefix by the first element of the subarray, and move the start pointer of the subarray one step forward. We update the maximum length of the subarray if necessary.
pythhn
= len(arr) def
than_ thargatggettttttttteee (arr, target):
prefix =1 start max_length
for i in range
prefix prefix arr[il
23

---

Published as a conference paper at ICLR 2024
while prefix target and start i: prefix prefix // arr[start]] start +=
max_length max (max_ 1-start+1)
return max_length
3. Problem: Given array of integers, find the smallest k such that the product of all the elements in the prefix of the array is equal the product of all the elements in the suffix of the array
Solution: We can use the prefix product algorithm to solve this
problem. We initialize through the array from left right. At each iteration, we multiply the current element with the prefix, and store the result in the prefix variable. We also initialize suffix variable to and then iterate through the array from right to left. At each iteration,
prefix variable to 1, and then iterate
multiply the current element with the suffix, and store the result the suffix variable. If the prefix equal to the suffix, we return the current index as the smallest k. If no such index is found, we return -1.
ppyyhhnn
def wtw.llest...ithheuuaaaaaaaaapaae
= len(arr) prefix 1 suffix
for in range(n):
prefix prefix * arr[i]
suffix suffix * arr[n-i-1]
if prefix suffix: return i+1
return
## Python3 code to solve the original problem:
Explanation of the solution:
Read the number
test cases.
Iterate through each test case
Read the number of elements in the sequence and the sequence itself.
Initialize prefix and suffix variables to 1, and to -1.
Calculate the product of all the elements in the sequence using the prefix product algorithm.
Iterate through the sequence from right to left.
Update the prefix and suffix variables.
If the prefix is equal to the suffix, update k to the current index
Print the value of k.
"''ythhn
Read the number of test cases intpiuutt
# Process each test case for in (ange(t):
# Read the test input int ())
arr list (map input () split())
Initialize prefix and suffix variables
24

---

Published as a conference paper at ICLR 2024
prefix suffix
k
Calculate prefix product for i in range(n): prefix prefix arr[i]
# Calculate suffix product and find smallest for i in range 8, -1):
prefix prefix // arr[i] suffix suffix arr[i]
if prefix = suffix: ==
Print the result (k)
Below is baseline output generated by GPT3.5-turbo-Iie 16k using vanilla O-shot prompt. This output does not recall any relevant exemplars and fails to employ the prefix product algorithm, resulting in incorrect solution.
def sma smallest_k(t (t, test_cases):
results [] for i in range(t):
[est_cases[i[[]
test_cases [i][1]
k -1
for j in (1nge n):
if a[:j] == a[j:]:
k=j
break
results, append (k)
return results
int(input ())
test_cases []
for in range (t):
int
(map (int, input () split())
test_cases append ([n] a)
output smallest_ke (t, test_cases) for in output: print(o)
D.4 BIG-BENCH: FORMAL FALLACIES
Our prompt (self-generate exemplars):
Your task is to tackle reasoning problems. When presented with problem, recall relevant problems as examples. Afterward, proceed solve the initial problem.
# Initial Problem:
"Is Siri a stepsister of Mary? Is Susan related Kate? In large families, it is sometimes difficult to keep track all one relatives. The following argument seeks to clarify some such relations: To start with, everyone who is a schoolmate of Mildred is daughter of Jeanette, too. Now, being daughter of Jeanette is sufficient for not being cousin of Cristi. Therefore, schoolmate
25

---

Published as a conference paper at ICLR 2024
of Mildred is cousin of Cristi Is the argument, given the explicitly stated premises, deductively valid invalid? Options: valid invalid
Instructions:
Make include all of the following points:
## Relevant Problems:
Recall three examples of problems that are relevant to the initial problem. Note that your problems must be distinct from each other and from the initial problem. For each problem:
After "Q: describe the prablem
After "A: explain the solution and enclose the ultimate answer in A boxed
## Solve the Initial Problem:
Say "Let's solve the following reasoning problem." Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate in \boxed here.
Output by GPT3.5-turo:
Relevant Problems:
Q: In a certain town, all doctors are required to have a medical degree. John is doctor. Therefore, John must have medical degree. Is this argument valid invalid?
A: This argument valid because follows the logical form of categorical syllogism, where the first premise establishes a universal statement and the second premise applies that statement to specific case. Therefore, the conclusion can be logically deduced. \boxed[valid)
Q: All cats have fur. Fluffy is a cat. Therefore, Fluffy has fur. Is this argument valid or invalid?
A: This argument is valid because it follows the logical form of categorical syllogism, where the first premise establishes universal statement and the second premise applies that statement to a specific case. Therefore, the conclusion can be logically deduced. \boxed(valid)
Q: is raining, then the ground wet. The ground wet. Therefore it must be raining. Is this argument valid or invalid? A: This argument is invalid because it commits the fallacy of affirming the consequent. Just because the ground is wet does not necessarily that it raining, as there could be other factors that caused the ground to become wet. \boxed(invalid)
Solve the Initial Problem:
Let's 's solve the following reasoning problem:
Q: "Is Siri a stepsister of Mary? Is Susan related to Kate? In large families, it is sometimes difficult to keep track of all one relatives. The following argument seeks to clarify some such relations:
start with, everyone who a daughter Jeanette, too. Now, being a daughter sufficient for not being a cousin of Cristi. Therefore, no schoolmate of Mildred is cousin of Cristi." Is the argument. given the explicitly stated premises, deductively valid invalid?
schoolmate of Mildred Jeanette is
A: The argument is deductively valid. The premises state that everyone who is a schoolmate of Mildred is a daughter of Jeanette, and being a daughter of Jeanette is sufficient for not being cousin of Cristi. Therefore, it logically follows that schoolmate of Mildred is cousin of Cristi. \boxed[valid)
26

---

Published as a conference paper at ICLR 2024
Below is a baseline output generated by GPT3.5-turbo using O-shot CoT prompt. This output does not recall any relevant exemplars and adopts an incorrect approach address the deductive reasoning problem.
The argument is deductively invalid.
To determine the validity of the argument, we need to analyze the logical structure and see if the conclusion necessarily follows from the premises.
Let 's break down the argument:
1. Everyone who is schoolmate of Mildred is daughter of Jeanette.
2. Being daughter of Jeanette is sufficient for not being cousin of Cristi.
Conclusion: No schoolmate of Mildred is a cousin of Cristi.
The first premise establishes relationship between being a schoolmate Mildred and being a daughter of Jeanette. However, does not provide any information about the relationship between being schoolmate of Mildred and being cousin of Cristi.
The second premise establishes relationship between being daughter Jeanette and not being cousin of Cristi. However, it does not provide any information about the relationship between being schoolmate of Mildred and being cousin of Cristi
Therefore, the conclusion that no schoolmate Mildred a cousin Cristi cannot be logically derived from the given premises. The argument is deductively invalid.
(invalinn
27


Reviews:
[{"review_id": "DmfhHDWi42", "reviewer": "Reviewer_DVcm", "summary": "The authors proposes a new prompting paradigm, which has three phases: \n1. Related knowledge retrieval\n2. Exemplar generation\n3. Answer prompting. \n\nWith this paradigm, the prompting on math and reasoning required tasks has an average accuracy gain of +4%.", "questions": "1. It would be nice to conceptually compare the work against the neural symbolic method [1]. \n\n[1] Zhang, Hanlin, et al. \"Improved logical reasoning of language models via differentiable symbolic programming.\" arXiv preprint arXiv:2305.03742 (2023).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: 3/5 \n\nThis paper aims to solve the problem that the few-shot prompting schema requires manually collected examples via a template-based method. \nAlthough there are quite a few previous works on prompt templates and knowledge retrieval, such as recitation-augmented models, this work focuses more on reasoning-based problem-solving. \n\nQuality: 2.5/5\n\nThis work has performed studies on quite a few benchmarks, including GSM8K and MATH, code generation in Codeforces, and other reasoning tasks in BIG-Bench. However, the experiment setup is a bit weird, as some of the studies include the in-context demonstration of generating examples, some do not; some adopt with knowledge paradigm, and some do not. A minor concern is that this study does not include the GPT-4 performance. \n\nClarity: 3.5/5\n\nOverall the paper is well written and easy to follow. The examples are quite illustrative but may be a bit repetitive, as Figure 3 seems to already include all the information that Figure 2 contains. \n\nSignificance: 3/5\n\nThis method proposes a new prompting schema for leveraging the language model as a knowledge base. However, this exemplar generation procedure does not provide an in-depth guarantee or study on the quality of generated examples.", "weaknesses": "1. The experiment setup could be better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ij8vxvkeY", "reviewer": "Reviewer_ExxM", "summary": "The authors propose the use of analogical prompting to enhance reasoning performance. This method requires LLMs to first self-generate relevant examples or knowledge before attempting to solve a problem. Then, the model provides a response based on the generated concepts.", "questions": "The most important questions are mentioned in the \"Weaknesses\" section. Here is an additional question I am interested in, which may not necessarily be included in this paper.\n\nCan a verification stage be added to filter out incorrect or irrelevant examples to further improve performance?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and easy to understand. And the proposed method is intuitive.\n2. The experiments demonstrate the effectiveness of the proposed methods across diverse datasets, while the ablation study indicates that the method outperforms the baseline approaches.", "weaknesses": "1. Will the language model be easily distracted when the generated examples are irrelevant or incorrect? The experiments in section 6.6 should provide more details. For instance, it should specify how many problems in MATH fail due to incorrect and irrelevant generated examples.\n2. Can current LLMs generate helpful examples for challenging questions? The authors are encouraged to include more examples in the paper.\n3. Can this method be integrated with self-consistency decoding? For instance, could the majority voting result from multiple reasoning chains with the generated knowledge, lead to better outcomes?\n4. The authors are encouraged to include more qualitative examples to compare the behavior between reasoning with a 0-shot prompt and reasoning with generated examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uKtYeLrD1e", "reviewer": "Reviewer_fGoM", "summary": "This paper introduces the analogical prompting method. It's a method similar to CoT, but instead of just plaining \"let's think step by step\", it first prompts the model to automatically generate some knowledge and exemplars for solving tasks. Experiments on mathematical reasoning, code generation, and big-bench confirm its validity.", "questions": "There is error analysis for incorrectly solved tasks, how about correctly solved questions? How many generated exemplars are wrong, but the solution to the new question is correct (i.e., it is known that sometimes LLMs can few-shot generalize from wrong exemplars)?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written, with a detailed depiction of the design principles and implementation details for the proposed analogical prompting method. Experiment results on GSM8K, MATH, codeforces, and a bigbench subset show moderate improvement compared to few-shot CoT, and a noticeable boost compared to other zero-shot prompting methods.\n\n- The analogical prompting method is well-motivated and intuitive enough to follow. Some ablation studies on the scalability, w/o knowledge, and the number of exemplars are good.", "weaknesses": "- Limited technical contribution. Although I think this paper is a good example of how to perform prompting engineering when solving zero-shot mathematical reasoning and code generation tasks, it is still more like a trick to an existing method (analogous to CoT->zero-shot CoT, in this case is retrival few-shot -> self-generated few-shot).\n\n- The codeforces dataset only contains 50 questions; perhaps it is too small to make claims on the improvements (~2% is only one more solved question). Any experiments on larger code generation tasks, e.g., HumanEval?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XwCk9v4V8C", "reviewer": "Reviewer_ey56", "summary": "This work presents a novel prompting method inspired by the concept of analogical reasoning in cognitive science. The method is shown to outperform few-shot chain-of-thought methods, while not requiring any few-shot examples.", "questions": "I am curious to hear the authors thoughts regarding whether the generated exemplars are genuinely retrieved from memory, or are novel problems based on general statistical knowledge. The latter case seems more consistent with the memory mechanisms in LLMs (and with their tendency to generate fabricated but plausible sounding information), but it is not something that has been considered much in the psychology literature on analogical reasoning, and it is somewhat more difficult to understand how it could improve performance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method involves an interesting take on concepts from cognitive science, and yields consistent improvements across reasoning benchmarks.\n- The proposed method is simple to implement, and does not require any task-specific prompts or training examples.\n- A qualitative analysis is performed of the generated exemplars and their relationship to downstream performance.\n- The code dataset is limited to recently published problems, thus addressing test set contamination concerns.", "weaknesses": "- The paper emphasizes the importance of instructing the LLM to generate *distinct* exemplars, but there is no ablation performed for this specific aspect of the method.\n- It seems unlikely that the generated exemplars are literally retrieved from memory in the sense that they are in human reasoning. It seems more likely that these are novel problems generated by the LLM, based on general statistical knowledge. I don't think this really undermines the usefulness of the approach, but it might be worthwhile to briefly discuss this issue.\n- The caption for Table 1 mentions that an in-context demonstration was used for the davinci models, but I couldn't find any explanation of this description (e.g., does this include an in-context training example, or merely a demonstration of the formatting?). \n\nMinor comments:\n- The authors might consider citing work that analyzes the analogical reasoning ability of LLMs [1,2] (though I should note that I don't think this undermines the contribution of the present work). There are also a few references that would be good to include when introducing the general concept of analogical reasoning and its role in the psychology literature [3,4].\n\n[1] Webb, T., Holyoak, K. J., & Lu, H. (2023). Emergent analogical reasoning in large language models. Nature Human Behaviour.\n\n[2] Hu, X., Storks, S., Lewis, R. L., & Chai, J. (2023). In-Context Analogical Reasoning with Pre-Trained Language Models. arXiv preprint arXiv:2305.17626.\n\n[3] Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2), 155-170.\n\n[4] Holyoak, K. J. (2012). Analogy and relational reasoning. The Oxford handbook of thinking and reasoning, 234-259.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)