StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving
Authors: Chang Gao
Haiyun Jiang
Deng Cai
Shuming Shi
Wai Lam
Venue: NeurIPS 2024 poster (2024)
Keywords: chain-of-thought
generalizability
strategy
reasoning
Primary area: natural_language_processing

Abstract: Most existing prompting methods suffer from the issues of generalizability and consistency, as they often rely on instance-specific solutions that may not be applicable to other instances and lack task-level consistency across the selected few-shot examples. To address these limitations, we propose a comprehensive framework, StrategyLLM, allowing LLMs to perform inductive reasoning, deriving general strategies from specific task instances, and deductive reasoning, applying these general strategies to particular task examples, for constructing generalizable and consistent few-shot prompts. It employs four LLM-based agents: strategy generator, executor, optimizer, and evaluator, working together to generate, evaluate, and select promising strategies for a given task. Experimental results demonstrate that StrategyLLM outperforms the competitive baseline CoT-SC that requires human-annotated solutions on 13 datasets across 4 challenging tasks without human involvement, including math reasoning (34.2\% $\rightarrow$ 38.8\%), commonsense reasoning (70.3\% $\rightarrow$ 72.5\%), algorithmic reasoning (73.7\% $\rightarrow$ 85.0\%), and symbolic reasoning (30.0\% $\rightarrow$ 79.2\%). Further analysis reveals that StrategyLLM is applicable to various LLMs and demonstrates advantages across numerous scenarios.

LLM:: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving*
Chang Gao Haiyun Jiang 04 Deng Cai Shuming Shi Wai Lam*
* The Chinese University of Hong Kong Tencent Al Lab
9 School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University (gaochang, wl ala) @se. cuhk edu hk haiyunjiane angnnpee 1p@mma 1 com (jcykca i, shumi ngshi @tencent. com
Abstract
Most existing prompting methods suffer from the issues of generalizability and consistency, as they often rely on instance-specific solutions that may not be appli- cable other instances and lack task-level consistency across the selected few-shot examples. To address these limitations, we propose a comprehensive framework, LLM, allowing LLMs perform inductive reasoning. deriving general strategies from specific task instances, and deductive reasoning, applying these general strategies to particular task examples, for constructing generalizable and consistent few-shot prompts. It employs four LLM-based agents: strategy gener- ator, executor, optimizer, and evaluator, working together to generate, evaluate, and select promising strategies for given task. Experimental results demon- strate that StrategyL.MM outperforms the competitive baseline CoT-SC that requires human-annotated solutions on 13 datasets across 4 challenging tasks without human involvement, including math reasoning (34.2% - 38.5 38.8%), commonsense reasoning (70.3% + 72.5%), algorithmic reasoning (73.7% -> 85.0%), and symbolic reason- ing (30.0% >> 79.2%). Further analysis reveals that StrategyI.LM L.M applicable various LLMs and demonstrates advantages across numerous scenarios.
Introduction
Recent advances in large language models (LLMs) have facilitated the development of prompting techniques 226 43 20 g. particular, chain-of-thought (CoT) prompting methods [43] 41],
which condition L.LMs on few task examples with step-by-step solutions, guide L.MSs to break down complex reasoning processes into simpler steps. These approaches have markedly improved performance compared to standard few-shot prompting across variety of tasks.
Despite their potential, current CoT approaches employing few-shot prompts with instance specific solutions may face challenges in terms of generalizability and consistency. Concerning generaliz- ability, the solution can be highly specific the question each instance, limiting its applicability to other instances. For example, as illustrated in the left part of Figure [ solution for particular system of linear equations with two variables may not provide valuable insights for addressing another system with three variables. Furthermore, the solutions in different instances within the few-shot prompt may exhibit lack of task-level consistency, which complicates the process for L.MSs to
"This research partially supported by the Center for Perceptual and Interactive Intelligence (CPII) Ltd. under Innovation and Technology Commission's InnoHK scheme.
Work done during an internship at Tencent Al Lab. $ Haiyun Jiang is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Tosk
(Strteggy (Gaussian Elimination Mcthod)
1. Write the aupmented matrix of the system liner equations.
Solving syslem of linerr equations
2. Perform row operations transform the matri into upper trinnuder matris. The now openstions include Perform back-subet tution find the solution
Specific Solutions
Questions
Stralegy-bssed Solutions
Solution First, solve for from
f: 1. Augmented matric G -16 (
equation - x)/2.
substhute eepression into second equation
Queation 1:
Row operations: subtracting times
from
x3/2 Solve x c.x -
second row Lg -71-7) New we have an upper trianguarr mathi.
subothull back the first equation: y [3 31/2
3. Back-substtutonn y= 77/-7 -1 x (5 2 13/1 -
Solution 2: Subtractng the second equation from the first equation: y = (8 - 23/3 - 2. Now substitute y im the second equation:
Question 2: [ žy [x-y=2
Solution 2: Augmented
G
2. Row operations: subiracting frrst row from the second row
G [ ). Now we have an upper trianguier matrix
= 2
BBBck y -6/-3 x (3 21/1
Specific Solution
New Queetion
Strstegy-based Solution
How soive?
Applynng strategy
x+y + 2x-1y 4am1 (3x By =
Augmanted matroc....
Row openstions Back bockksubsttution....
Figure 1: Comparison of specific solutions and strategy-based solutions.
develop effective solutions for tackling new instances. As demonstrated in the left part of Figure I] the two specific solutions are based on different approaches: Solution 1 employs expression substi- tution, while Solution 2 utilizes equation subtraction, which may not provide consistent guidance for LLMs to solve new instances. To address these limitations, it is crucial to incorporate effective wroblem-solving strategies and develop consistent strategy-based solutions within few-shot prompts. The right part of Figure [1 presents an effective strategy, i.e., Gaussian Elimination Method, offering generalizable steps applicable to any system of linear equations. By providing this strategy and consistently applying it across various instances in the few-shot prompt, LLMs can be better equipped to generate effective solutions for new task instances.
This paper aims to construct generalizable and consistent strategy- -aaegg besedfewsshor few-shot prompts for various tasks automatically, while being highly cost-efficiett. Our proposed framework. StrategyLLM, draws inspiration from human cognitive processes to derive general problem-solving strategies. This approach enables LLMs to reason inductively, i.e., deriving general strategies from specific task instances, and deductively, i.e., applying general strategies to particular task examples, to formulate prompts. An example of strategy-basee prompts can be seen in Figure The inductive reasoning process enhances generalizability by formulating general problem- solving strategies, while the deductive reasoning process improves consistency by producing consistent solutions using a given strategy. Developing effective problem-solving strategies is crucial to the success of our framework. To achieve this, we design StrategyI.LMM as a multi-agent collaboration framework comprising four LLM-based agtats-ytrategy generator, executor, optimizer, and evaluator, as shown in Figure 2 The strategy generator initially creates a pool of strategies that are exccuted on task examples to assess accuracy, with qualified strategies cached based threshold and further evaluated. Unqualified ones may be optimized and re-evaluated iteratively. Through the collaboration of these intelligent agents, our framework is capable of autonomously generating, evaluating, and selecting effective strategies for various tasks and eliminating the need for human involvement.
Crucially, the strategy- based few-shot prompt generation phase is applied once for given task, after which the learned prompt can be employed for inference on the entire test set. This inference process does not require any additional input beyond the standard few-shot prompting settings. The prompt generation process is highly cost-effective as it necessitates only a few task examples. In particular, Strategy LLM expends less than $0.24 to develop a strategy- based prompt for variety of tasks using the latest version of GPT-3.5-Turbo
We conduct comprehensive evaluations of StrategyLLM on 13 datasets across 4 challenging tasks: math reasoning, commonsense reasoning. algorithmic reasoning, and symbolic reasoning. The experimental results reveal the following key findings: (1) StrategyLLM outperforms competitive baselines on tasks without using any human-annotated reasoning processes; (2) Strategy can be applied to various LLMs and is robust to different groups of task examples; (3) StrategyI.LM can generate generalizable and consistent prompts in cost-effective manner. These findings demonstrate the potential of StrategyLLM an effective, efficient, and reliable problem-solving framework.

---

Qualtted Strloges
divarse prohiema..
Tadk Exampies
Stretey
Stratogy 10
Execition Rasull
Execution Risull 1:
Cache
Top
Strategy Evaluator
Question ..... Answer 1:...
Answer
Strategy Executor
Unqualited Stratogles
Strotogy 2:
Execution Risult
[Cache]
Strategy Optimizer
Strategy Generator
Streteyy
Execution Rasull
Strategics
Updated Strategies
Strategy Identity the information..
Stratogy 2:
Stratagy B:
Strategy
problem by....
Figure 2: Overview of LLM. Initially, the strategy generator creates pool of strategies, which are then applied by the strategy executor to task examples to calculate execution accuracy. Qualified strategies meeting a pre-defined threshold are cached, and necessary, unqualified strategies are optimized and re-evaluated in iterative cycles. Once sufficient number of qualified strategies are obtained or the maximum iteration number is reached, the top strategies are ranked by execution accuracy and evaluated using validation set.
2 StrategyLLM
Our Strategy LLM framework is designed to efficiently create strategy-based few-shot prompts for wide range of tasks. Subsequently, these prompts can be utilized for inference. In this section, will introduce our framework in detail. The inference procedure will be discussed in Section
our framework consists of four key agents: strategy generator, executor, optimizer, and evaluator. The prompts for the strategy generator and executor are presented in Figure 3 and Figure respectively. The prompts of the strategy optimizer are in Appendix C Typically, only a few task examples are used in the collaboration process, making our framework highly efficient.
Overview of StrategyLLM As presented in Figure 2
The collaboration process begins with the strategy generator formulating a pool of strategies based on its understanding of the target task. These strategies then undergo rounds of validation and selection. In the first round, the strategy executor applies each strategy to a set task examples to yield its execution result and compute its execution accuracy. Strategies that meet exceed pre-set threshold of execution accuracy are deemed qualified and are cached with their corresponding execution results and accuracy. If the number of qualified strategies is less than a pre-defined number k, the optimizer refines the unqualified strategies using their execution results. These enhanced strategies are then back the strategy executor for the next iteration. This cycle may repeat until sufficient number of qualified strategies are achieved or the maximum iteration limit is reached. Following this, all cached strategies are ranked based on their execution accuracy, and the top strategies are selected. In the second round, the strategy evaluator constructs strateyybbssd few-shot prompts for each candidate strategy using itself and execution result and assesses all candidate strategies using their corresponding prompts for inference validation set.
Notations We use p, q, st, so, and to denote the prompt, question, strategy. solution, and answer, respectively. During inference, given question q, the language model M (p, q) + (so, a) generates solution so and an answer a for it conditioned on the prompt p. We denote the target task as its definition as d, and the set of task examples as E. Each example in E is (g. a) pair.
Strategy Definition In this paper, a task-solving strategy st is defined as a systematic approach designed for universal application across task examples, comprising a series of subtasks that encode task knowledge address the target task t. It is characterized by the following properties: (1) Task-Level Applicability: The strategy is formulated in a manner that allows for its application across all task instances, ensuring universality and consistency its implementation. Organization: The strategy comprises sequence of subtasks that are organized in logical order to collectively tackle the target task. These subtasks are interconnected and contribute to the overall achievement of the task objective. (3) Task Knowledge Encoding: The strategy encapsulates general
(2) Structured
3

---

Strategy Generator
Task: [task definition]
Some examples of the task are as follows: [task examples)
Let's understand the task and write strategy that consists of sequence of subtasks solve the task. For writing, you must satisfy the following requirements.
Include all necessary subtasks.
All subtasks are easy to solve.
Subtasks are in an appropriate order.
Do not include specific information in the examples.
Make sure the strategy is general and concise.
The result must be numbered list in the following format:
First subtask
2. Second subtask
Figure 3: Prompt of the strategy generator.
Strategy Executor
Task: (task definition]
Example of the task: (example)
Strategy: {strategy]
The strategy consists of sequence of subtasks for solving the task. Please execute the strategy on the provided example. For executing, you need to write a step-by-step solution to the example based on the subtasks. The solution must satisfy the following requirements:
Adjust and execute these subtasks for this example.
Compute as many intermediate results as possible.
The answer obtained from the solution must be the same as the original answer.
The result must be in the following format:
Question: Question the provided example
Solution: Solution obtained based on the subtasks in the strategy
Answer: Answer in the provided example, which must include the string 'The answer
Figure 4: Prompt of the strategy executor.
task knowledge and principles, avoiding any specific details unique to individual task examples. These properties collectively contribute the effectiveness and efficiency of a strategy by promoting consistency, clarity, and informed decision- beciiinn-making in addressing task-level challenges. By embodying these properties, strategy can serve valuable tool to navigate complex tasks and achieve optimal outcomes. An example of the strategy is presented in Figure
Strategy Generator G The strategy generator, represented G (E, d, n) + [sl,)1 aims to generate n diverse strategies for the target task based on a task examples E and the task definition using temperature sampling.

---

Standard Prompl
Strategy based Prompt
Strategy:
Question: Jaeon had 20 kollpops. He gave lolipops.
Identity the given information the word problem
Denny some lolipops. Now Jason
2. Determine the desirnd infor mationaton or what needs to be caltulated
Hew many kolipops did Jeson give to Denny?
3. Identifl the mathemasical operation needed to salve the problom
Answer: The answer B.
4. Calculate the solution using the given information and the approoñitte mathematical operation Verify the solution by checking saliffes conditions
prozlem
ToT Prompt
Question: Jason had 20 tolipopss. He gave Denny some tolipops. New Jason has 12 lolipops.
Question: Jason had 20 lollpaps. He gave
How many Iolipops did Jason give to Denny?
Denny some tolipope. Now Jason has 12 tellipops.
lolipope and has kilipops. 2. Dasired information: Number tolippops Jason geve to Denny.
Given information. Jason
many lollpoops
Denny?
Solution:
kolioops, Then
3. Mathematical operation: Subtraction.
had aftr giving some to Denny. So he gave
4. Calculalion: find the differ belween the inilial and final number lollioops: 20 12
Denny 20- 12-8. 6. Answer: The answer
5. Venficction: Jason had polipops
Danny, he would have pops
Answer: The answer
Figure 5: Comparison of the strategy-basee based, standard, and chain ofainnoo-thoughtt (CoT)
4 prompt.
Strategy Executor X The strategy executor, denoted as X (E, st) - (Rat. eacc,t), writes solutions a of task examples & following the strategy st obtain the execution result Re {(g, 80, a)) } of st. The execution accuracy eacest is calculated as the proportion of examples whose solutions yield correct answers, reflecting the degree of alignment between the strategy and task. Therefore, we select strategies with high execution accuracy as qualified strategies.
Strategy Optimizer O The strategy optimizer, represented as O (E, d, st, Rst) - st optimize the strategy st according to its execution result Rat to obtain the updated strategy st". Firstly, the strategy optimizer O analyzes why some solutions in Rest are not correct and provides suggestions for improving st. Secondly, it modifies sl to obtain st" based on the analysis and suggestions.
Strategy Evaluator E We select top k candidate strategies according to the execution accuracy. However, to ensure efficiency, we use a limited number of task examples for execution, making the execution accuracy not very informative metric for choosing strategies. Therefore, a strategy evaluator to further evaluate the candidate strategies on a validation set V. This process only requires to perform inference once for each candidate strategy and is efficient. The strategy evaluator, denoted as E (sl, Rat, V) + vaccst, computes the validation accuracy vacc, of the strategy sl on V. To achieve this, it constructs the strategy- brased few-shot prompt pst = (sl, Rwt) and conducts inference on V using Pst- An example of strategy-based prompts is presented in Figure The validation aceuracy vaccst is calculated as the percentage of validation examples whose answers are correct, reflecting the effectiveness of st in real-world scenarios. Strategies with high validation accuracy can be used for inference.
introduce
3 Inference
Through collaborative efforts among multiple agents, we have obtained multiple candidate strategies, each with its few-shot prompt and validation accuracy. Depending on the task hand, we can select one or more strategies with high validation accuracy for inference. For simple or specific tasks, single optimal strategy that solves all task examples effectively may exist, making it sufficient to use only one strategy. However, for complex or diverse tasks, it is unlikely to find a strategy with absolute superiority. In such cases, adopting multiple strategies for inference more appropriate, as they may excel in different task examples. To harness the strengths of multiple strategies, we employ two methods. The first method involves taking a majority vote on all answers obtained by multiple strategies, akin to the self-consistency (SC) method [41] The second method requires
LLMs to determine the final answer by considering the solutions derived from multiple strategies in a zero-shot (ZS) manner, making it more proper for complex and diverse tasks. We denote the first and second methods as StrategyLLM-SC and StrategyLLM--ZS respectively. The prompt for the second approach is provided in Appendix D
4 Experiments
4.1 Experimental Setup
Evaluation Tasks and Datasets
We evaluate StrategyLLM on variety of tasks:
5

---

Table 1: Experimental results on the math reasoning task. The numbers in parentheses represent the
relative improvement compared CoT-SC.
Methods
PA
IA
CP
GE
Avg
SP
32.0
50.0
17.5
27.0
20.5
21.0
20.5
26.9
SolutionLLM
58.5
56.5
13.5
33.0
32.0
28.0
19.5
34.4
CoT
57.0
57.5
15.0
33.5
28.0
23.0
20.0
33.4
CoT-SC
59.0
62.0
16.5
34.5
28.0
24.5
15.0
34.2
StrategyLLMM LLM
58.5
57.5
18.0
35.0
29.5
24.5
22.5
35.1
StrategyLLMMSC
60.0
61.5
18.0
38.5
30.5
28.0
24.0
37.2 (+8.8%)
StrategyLLM-ZZ
64.5
65.5
19.0
39.0
32.5
28.5
22.5
38.8 (+13.4%)
Table Experimental results on the commonsense, algorithmic, and symbolic reasoning tasks. The numbers in parentheses represent the relative improvement compared to CoT-SC.
Commonsense
Algorithmic
Symbolic
Methods
StrategyyQ4
DU Avg
K8 Avg
14C-4
LLC-8
M.C- 16
SP
56.5
48.5 52.5
73.3 2.0 37.7
0
0
0
0
SolutionI.LM
59.5
52.0 55.8
74.7 55.3 65.0
81.5
25.5
0
35.7
CoT
64.0
70.5 67.3
67.2 84.0
75.6
68.5
22.0
30.2
CoT-SC
70.0
70.5 70.3
61.3 86.0 73.7
68.0
22.0
30.0
StrategyLLM
67.5
68.5 68.0
80.0 86.7 83.4
98.0
86.5
51.5
78.7
StrategyLLM-SC
71.0
74.0 72.5 (43.1 %)
79.3 90.7 (+15.44)%
98.0
87.0
52,5
79.2 (+164.0%)
Strategyl .M-SS
70.0
72.5 71.3 (+1.4%)
78.7 89.3 (410..14411))
98.0
86.0
44.0
76.0 (+153.3%)
We use the challenging MATH benchmark [6] which comprises problems from mathematics competitions that require than standard K-12 mathematics tools. It consists of seven datasets of different subjects, namely, Algebra (AL), Prealgebra (PA), Intermediate Algebra (IA), Counting and Probability (CP), Number Theory (NT), Geometry (GE), and Precaleulus (PC).
Math Reasoning:
Commonsense Reasoning: We employ StrategyQA 14 and the Date Understanding (DU) task StrategyQA necessitates inferring a multi-hop strategy to answer questions, while the DU task involves deducing a date from given context.
from Big-Bench Hard 38 9
Algorithmic Reasoning: We adopt the Word Sorting (WS) task and the Multi-step Arithmetic (MA) task from Big-Bench Hard 38 9]. The WS task involves sorting a list of words lexicographically, and the MA task requires solving multi-step equations with basic arithmetic operations.
Symbolic Reasoning: We utilize the Last Letter Concatenation (LLC) task from 43 which requires concatenating the last letters of words in a sequence. In the few-shot prompt, the model only sees examples with two words. To evaluate the generalization abilities of different methods, we construct three out-of-distribure test sets (LLC-4, LLC-8, and LLC-16) with 4, 8, and 16 words in a sequence, respectively.
Baselines We conduct experiments in the few-shot setting and compare StrategyLLM with the following baselines:
Standard Prompting (SP): SP is the most direct approach for problem-solving SP, the prompt p contains a set of question-answer pairs without intermediate reasoning steps.
Chain-of-Thoughe (CoT) Prompting 43 CoT incorporates step-by-step solutions for questions in the prompt p elicit the multi-step reasoning capabilities of LLMs, We use few-shot CoT prompts from 4 for StrategyQA, DU, and LLC, and prompts from [38] for WS and MA. For MATH datasets, we create few-shot CoT prompts by randomly sampling 4 examples from each dataset's training set since these datasets contain human-annotated solutions. The CoT prompts for these datasets are Appendix H
Conf-ssssitencyy with CoT (CoT-SC) 41 CoT-SC generates set of solutions using CoT via
temperature sampling obtain multiple answers. Subsequently, it takes a majority vote over these answers to determine the final answer. For experiments, we sample reasoning paths using temperature sampling with temperature of 0.7.

---

Table 3: Experimental results on two math reasoning datasets, namely AL and CP, with different groups of examples.
Methods
AL-dev
AL random
CP-dev
CP-random
SP
36.0
29.1
25.5
26.8+22.5
SolutionLLM
58.0
26.5
31.0
32.2-12.8
CoT
57.5
1..
34.0
33,4=1.2
CoT-SC
59.5
58.3=1.2
31.5
33.0=1.2
Strategy
57.0
54.7=2.5
34.5
35.6+22.3
StrategyLLM-SC
64.0
58.9
36.5
3..4+1.3
LLLSS
62.5
60.8
38.5
3.7
LolutionLLM: We construct this baseline to leverage LLMs to directly write the solution for each example in the few-shot prompts using greedy decoding. without using any strategies. The prompt of SolutionL..M is in Appendix E Since both SolutionLLLM and StrategyLLMM generate prompts using LLMs, we can eliminate the potential effect of human expertise in the comparison, isolating the impact of incorporating effective strategies.
Implementation Details We employ GPT-3.5 (gpt-3.5-turbo- 16krbo--66-0133)) 32 as the language model for our main experiments, serving as the backend for the strategy generator, executor, optimizer, and evaluator. For a fair comparison with baselines such as Co'T, we use the same examples in their few-shot prompts for strategy generation, execution, and optimization. We select the top I or 3 strategies with the highest validation accuracy for inference. This allows us to demonstrate the performance of the optimal strategy and the benefits of using multiple strategies. We adopt greedy decoding for inference. Details of the strategies for each dataset can be found Appendix G The validation set size is 100 for all the datasets. For datasets with over 200 test examples, randomly sample 200 examples for testing to reduce API costs. More details can found in Appendix
4.2 Main Results
Tables [ and present the experimental results of StrategyLLM and several baselines across four reasoning tasks. We have the following observations:
StMategyLLM an effective and efficient framework for problem-solvingg StrategyLLM using multiple strategies, i.e., StrategyLLM-SC and StrategyLLM-2S, outperforms all baselines across the four reasoning tasks. Furthermore, StrategyI.IMM employing the best discovered strategy consistently outperforms CoT and SolutionI.LM Notably, Strategyi.I.M automatically constructs generalizable and consistent few-shot prompts for tackling various tasks without human expertise, while CoT relies on
examples for each task.
Explicitly incorporating effective strategies significantly enhance the complex reasoning and out-of- distribution (OOD) generalization abilities of LLMs. For example, our framework demonstrates more considerable improvements on the MATH benchmark compared to the simpler commonsense reasoning datasets. Furthermore, StrategyLLM, substantially surpasses CoT and SolutionL.LM on the three OOD test sets of the LLC task, showcasing the generalizability of effective strategies.
Adopting multiple strategies brings obvious benefits on complex or diverse tasks. The performance of StrategyLLM significantly improved by using multiple strategies the math and commonsense reasoning tasks. The benefits of leveraging multiple strategies on simpler or more specific tasks, i.e., symbolic and algorithmic reasoning, is less significant. These observations indicate that our framework is capable of creating multiple complementary strategies for diverse complex tasks. Furthermore, LLM-ZS outperforms StrategyLLM-SCC on the math reasoning task, showing that allowing LLMs to determine the answer is more appropriate for intricate tasks.
5 Analysis
Evaluating the robustness of StrategyLLMM We conduct an investigation assess the robustness of our StrategyLMM framework with respect to varying groups of examples. For this purpose,
7

---

Table 4: Experimental results of closed-source models on the CP, StrategyQA, and MA datasets. The numbers parentheses represent the relative improvement compared CoT-SC.
GPT-4
Claude-3 3 Sonnet
Methods
CP
StrategyOR
MA
Avg
CP StrategyOA
MA
Avg
L..
52.0
75.5
96.7 74.7
21.0
73.5
69.3 54.6
CoT
49.5
80.5
92.7 74.2
26.0
69.0
72.7 55.9
CoT-SC
54.5
83.5
94.7 77.6
26.0
75.0
76.7 59.2
StrategyLLM
52.5
81.5
98.7
77.2
28.0
75.0
83.3
62.1
StrategyLLM-S
56.0
83.5
98.7 79.4 (+2.4%)
28.0
77.0
88.0 64.3 (+8.6%)
Table Experimental results open-source models on the CP, StrategyQA, and MA datasets. The numbers in parentheses represent the relative improvement compared to CoT-SC.
Meta-Llama- Bas-llama--------- 3-8B-Instruct
Methods
CP
StrategyOA
MA
Avg
StrategyOA
MA.
Avg
So
20.5
64.0
43.3 42.6
51.5
79.0
72.0 67.5
CoT
44.7
40.6
80.5
81.3 70.1
CoT-SC
19.5
71.0
45.3 45.3
47.0
81.5
82.0 70.2
StrategyL.IM
24.5
74.0
64.7 54.4
82.0
88.0 73.8
StrategyI.IMMe M--C
25.0
74.0
66.0 55.0 (+21.5%)
54.0
83.5
91.3 76.3 (+8.7%)
Mixxrrl--xx7B-Insttuutt
Mixtral-8x22B Mixtral--x222--Immmmete Inntru2ttvttt
Methods
StrategyQA
Avg
StrategyQA
Avg
SolutionLLM
22.5
61.0
34.7 39.4
44.5
72.0
60.7 59.1
CoT
24.5
63.0
59.3 48.9
41.0
72.0
80.0 64.3
CoT-SC
26.5
73.5
62.7 54.2
40.5
75.0
80.7 65.4
LLMM
28.5
73.5
76.0
59.3
44.5
76.5
84.0 68.3
StrategYLLM-SS SC
32.0
75.0
78.0 61.7 (+13.7%)
47.5
77.0
89.3 71.3 (+9.0%
select math reasoning datasets with diverse examples, namely AL and CP, and randomly sample distinct groups of examples from their respective training sets. We then report the mean and standard deviation of the results. Additionally, we employ the validation set identify a group of 4 examples from the training set. Specifically, we use the OpenAl embedding model API (the text-embedding. 3-arg model) map training and validation questions embeddings and subsequently select the 4 training examples with the highest cosine similarities to all validation examples. We designate these groups of examples as AL.-dev and CP-dev, respectively. The results, as presented in Table 3 demonstrate that StrategyIIMM consistently delivers satisfactory performance on both datasets, suggesting that Strategy is robust and reliable framework for problem-solving.
Exploring the universality of StrategyLLM To investigate the universality of our Strategy framework, we apply it to a variety of LLMs to evaluate its effectiveness. For closed-source models, we utilize GPT-4 (gpt-4-0613) 31 and Vlaude-3-Sonner Cclaude-3ssomme-22022222
2 For open- 11. Mixtral-8x7B: [8] We conduct experiments on the CP, Strategy and MA datasets, which represent three distinct reasoning tasks. The results, summarized in Tables 4 5 reveal that integrating effective strategies for constructing generalizable and consistent few- shot prompts yields significant benefits across a range of model capabilities and task complexities, underscoring the framework's universality. Strategy LLM notably enhances performance in open- source models such Meta- (alama---BB-Imn the CP and MA datasets which demand complex reasoning, indicating the effectiveness of our framework in scenarios requiring sophisticated problem-solving These findings further corroborate StrategyLLMM robustness and reliability as a problem-solvine framework.
source models, we employ --ta---la-------mmmmee Instruct-v0.1, and Maxtral-8x22--Instruu -nssstuuttt,t
and Mixtral-8x77 -aattral--x---nstttee
particularly
Comparing reasoning via task-level strategy and instance-specific planning Our framework facilitates generalizable and consistent reasoning by developing task-level strategies. To evaluate

---

Table 6: Comparison of Plan-and-Solvee CoI+Strategy and StrategyLLMM
Methods
CP
StrategyOA
MA
Avg
Plan-and-Solve
26.0
54.0
69.3
49.8
Plan-and-Solve-SC
27.5
64.5
70.0
54.0
CoT++Strategy
30.5
63.0
62.7
52.1
CoI+Strategy-Se
36.5
70.0
70.0
58.8
StrategyLLM
35.0
67.5
86.7
63.1
StrMCegyLLMMSCC
38.5
71.0
90.7
66.7
the necessity of effective task-level strategies, we compare our framework against two baselines: (1) Plan-and-Se and-Solve Prompting 40). which directs LLMs to formulate specific plans for each test instance at inference time and execute these plans to solve the instances; (2) CoT+Strategy. which combines the CoT prompt with instructions that guide LLMs devise a task-solving strategy and apply it to specific test example at inference time. The prompt for CoT+Strategy is detailed in Appendix F
The performance of GPT-3.5 the CP, StrategyQA, and MA datasets, representing three distinet reasoning tasks, is presented in Table 6 Our observations are as follows: (1) StrategyLLMM sig- nificantly outperforms both Plan-and-Solve Prompting and CoT+Strategy across all three datasets. This highlights the superiority of generalizable task-level strategies over instance-specific plans in enhancing performance across various problem-solving contexts. This improvement be attributed to two key factors: (a) our task-level strategies encapsulate essential task-level knowledge, thereby providing professional and high-level guidance; (b) generating high-quality, specific plans for each test example at inference time is inherently challenging, making it difficult to ensure the quality of these plans. (2) Even when explicitly encouraged devise general task-solving strategy in the Ctr+Strat method, the LLM tends to produce strategies that are highly specific to the test example and encode limited task-level knowledge. This underscores the necessity of creating generalizable strategy- -ased few-shot prompts.
Analyzing the cost of strategy -based prompt generation In this analysis, we evaluate the cost of the strategy-based prompt generation process. The process includes the strategy generator, executor, optimizer, and evaluator, each contributing to the overall cost for each reasoning task. Table details the average cost incurred by our StrategyLLM framework in generating a candidate strategy- based prompt, calculated by dividing the total cost of the process by the number of candidate strategies k. The costs are presented terms of input and output tokens and the money associated with using GPT-3.5-Turbo. The results indicate that framework is economically efficient. The average cost for gpt-p-333 3. 5-turbo- - 6k 0613 ranges from $0.33 to $1.12 across the four reasoning tasks. For the latest version of GPT-3.5-Turbe specifically gpt-3. 3. 5-turbo- 0125, the cost considerably lower, ranging from $0.08 to $0.24. Generally, tasks of higher complexity consume more tokens due their inherently longer solutions.
CoT-SC
StrategyLLMM-E SC
Examining results across various difficulty levels The problems in the MATH benchmark are classified by difficulty on scale of to The easiest problems are assigned difficulty level of while the most challenging prob- lems are given a difficulty level of 5. Figure 6 illustrates the performance of CoT-SC and StrategyLLM-SCC on the seven datasets within the MATH benchmark, considering different dif- ficulty levels. It is evident that the enhanced performance of StrategyLLM-SS over CoT-SC stems from its ability to tackle complex problems, underscoring the significance of gen- eralizable strategies in augmenting intricate rea- soning.
60
Perforance (%)
50
(9.81
45.21
30
11.50 13:10
Difficulty Level
Figure 6: Comparison of CoT-SC and StrategyLLM-SC performance on the MATH benchmark across various difficulty levels.
More analysis can be found in Appendix A

---

Table 7: Average cost of prompt generation across four reasoning tasks.
Math
Commonsense A Algorithmic Symbo i
Input Tokens
287.83K
228.67K
107.27K
70,94K
Output Tokens
63.14K
33.15K
32.95K
28.48K
Cost of gpt 3ptt335-turbb-111 5-tu:rbo-66k006.1
$1.12
$0.82
$0.45
$0.33
Cost of gpt-3 5-turbo-0125
$0.24
$0.16
$0.10
$0.08
Related Work
Prompting LLMs for Problem Solving The prominent chain-of-thought (CoT) prompting ap- proach [43] has inspired a variety of prompting methods aimed at enhancing the problem-solving abilities of LLMs. These methods include using programming languages describe the reasoning representing the reasoning process with complex structures such as trees or
process 6
13
28
graphs [6
3 36 47
applying task decomposition (49
19
34 4 implementing self-correction
with automatic feedback (22 29 30 77. and combining different prompting techniques [27] [50].
However, most of these approaches require manual annotation of reasoning processes, limiting their generalizability and flexibility. By comparison, our StrategyLLMM LLM framework can automatically construct strategy-based few-shot prompts for any task, ensuring generalizable and consistent so- lutions following effective strategies. This approach sets it apart from existing automatic prompt construction methods [48] 37 45), which may generate inconsistent solutions within the prompt. The plan-and-solve prompting method [40) aims to address missing-step errors by requesting LLMs to generate plan before solving a specific example zero-shot manner. The plan instance-specific and significantly different from the task-solving strategy which can be applied to all task examples. The learning- toannngtt-plaee to-p approach [15] learns a text plan for each task to assist LLMs in problem-sol solving.. The plan, which is not necessarily strategy, Moreover, it demands a large training and validation set during the learning process, resulting in high costs. In contrast, our framework is efficient and cost-effective.
be any instruction helpful for solving the task.
LLM-based Autonomous Agents
The adoption of autonomous agents driven by LLMs across methodologies for tackling problems, making decisions, and These agents have heen utilized to enhance the reasoning capabilities of
various disciplines is revolutionizing
fostering innovation
39 44
LLMs 42 24 11, contribute to social simulation 33 23 25 21), and advance software development
17 10. In this paper, we employ multiple based agents collaborate in the generation, execution, optimization, and evaluation of problem-solving strategies.
35
7 Discussion
Limitation and Impact The key idea behind StrategyI.IM to harness the knowledge and reasoning capabilities of LLMs to develop and refine task-solving strategies tailored to specific tasks. By utilizing the extensive knowledge embedded these LLMs, which are trained on diverse data sources spanning multiple domains, Strategy is able to generate generalizable strategies that incorporate Jomain-specific expertise. However, the model possesses limited knowledge particular domain, it is unlikely to create effective strategies for that domain. In such cases, merely optimizing the prompt may not significantly improve performance, and domain-specinn continual training may be necessary. As LLMs continue to expand their knowledge bases and enhance their reasoning capabilities, their ability to generate generalizable strategies for diverse tasks is expected to improve, implying the potential of our StrategyI.LM framework.
Conclusion This paper proposes StrategyLLM, harnessing the power of LLMs to construct gener- alizable and consistent few-shot prompts for various tasks efficiently. Our framework's 's effectiveness and reliability are substantiated through extensive evaluations on four challenging tasks: mathematical reasoning, commonsense reasoning. algorithmic reasoning, and symbolic reasoning. Further analysis reveals that our framework exhibits robustness across different task example groups, application to various LLMs, cost-efficiency in prompt generation, and effecctiveness in complex reasoning.
10

---

References
[1] Meta AI. Introducing meta llama 3: The most capable openly available Ilm to date. 2024.
[2] Authropic. Introducing the next generation of claude. 2024.
[3] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682-1769) 2024.
[4] Jiaao Chen, Xiaoman Pan, Dian Yu, Kaiqiang Song, Xiaoyang Wang, Dong Yu, and Jianshu Chen. Skills-in-context prompting: Unlocking compositionality large language models. arXiv preprint arXiv:2308.003300 2023.
[5] Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. Iterative translation refinement with large language models. arXiv preprint arXiv:2306. 0333636666, 2023.
[6] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023.
[7] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference Learning Representations, 2024.
[8] Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309. 15402, 2023.
[9] BIG-Bench collaboration. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
[10] Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. arXiv preprint arXiv:2304.07590 2023.
[11] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305. 14325, 2023.
[12] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-band based prompting for multi-step reasoning. In The Eleventh International Conference Learning Representations, 2023.
[13] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAI:: Program-aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10764-10799. PMLR, 23-29 Jul 2023.
[14] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346 361, 2021.
[15] Yiduo Guo, Yaobo Liang, Chenfei Wu, Wenshan Wu, Dongyan Zhao, and Nan Duan. Learning to plan with natural language. arXiv preprint arXiv:2304.10445 2023.
[16] Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.

---

[17] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang. Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024.
[18] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401. 04008, 2024.
[19] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, 2023.
[20] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[21] Grgur Kovač, Rémy Portelas, Peter Ford Dominey, and Pierre-Yves Oudeyer. The socialai school: Insights from developmental psychology towards artificial socio-cultural agents. arXiv preprint arXiv:2307.0787/, 2023.
[22] Miaoran Li, Baolin Peng, and Zhu Zhang. Self-checker: Plug-and-play modules for fact- checking with large language models. arXiv preprint arXiv:2305. 14623, 2023.
[23] Siyu Li, Jin Yang, and Kui Zhao. Are you in a masquerade? exploring the behavior and impact of large language model driven social bots in online social networks. arXiv preprint arXiv:2307. 10337, 2023.
[24] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang. Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint 2rXiv:2305.19111 2023.
[25] Jiaju Lin, Haoran Zhao, Aochi Zhang, Yiting Wu, Huqiuyue Ping. and Qin Chen. Agentsims: An spen-source sandbox for large language model evaluation. arXiv preprint arXiv:2308. 04026, 2023.
[26] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Comput. Surv., 55(9), jan 2023.
[27] Tengxiao Liu, Qipeng Guo, Yuqing Yang, Xiangkun Hu, Yue Zhang, Xipeng Qiu, and Zheng Zhang. Plan, verify and switch: Integrated reasoning with diverse X-of-thoughts. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 2807-2822, Singapore, December 2023. Association for Computational Linguistics.
[28] Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Bure Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi, editors, Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguisties (Volume I: Long Papers), pages 305-329, Nusa Dua, Bali, November 2023. Association for Computational Linguisties.
Faithful chain-of of-thought reasoning. In Jong C. Park, Yuki Arase,
[29] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann. Sean Welleck, Amir Yazdanbakhsh. and Peter Clark. Self-refine: Iterative refinement with self- feeebake In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[30] Ning Miao, Yee Whye Teh, and Tom Rainforth. Selfcheck: Using LLMS to zero-shot check their own step-by-step reasoning. The Twelfth International Conference on Learning Repre- sentations, 2024.
12

---

[31] OpenAI. Gpt-4 technical report. arXiv preprint arXIi:2303.08774,20000
[32] OpenAl. Introducing chatgpt. 2023.
[33] Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST '23, New York, NY, USA, 2023. Association for Computing Machinery.
[34] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687-5711, Singapore, December 2023, Association for Computational Linguistics.
[35] Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprini arXiv:2307.07924 2023.
[36] Bilgehan Sel. Ahmad Al-Tawaha, Vanshaj Khattar, Lu Wang, Ruoxi Jia, and Ming Jin. Algo- rithm of thoughts: Enhancing exploration of ideas in large language models. arXiv preprint a30iv:2308. 10379, 2023.
[37] Kashun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with chain-of- from labeled data. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational I Linguistics: EMNLP 2023, pages 12113-12139, Singapore, December 2023. Association for Computational Linguisties.
[38] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thouge for Computational Linguistics: ACL 2023, pages 13051, Toronto, Canada, July 2023. Association for Computational Linguistics.
can solve them. In Findings of the Association
[39] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang. Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.0.. 10088.1133. 1432, 2023.
[40] Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee- Peng Lim. Plan-and-solve prompting: Improving zero-shot cfain-of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 209922634, Toronto, Canada, July 2023. Association for Computational Linguisties.
[41] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi. Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Selficoon improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
[42] Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi pulti-persona self- collaboration. arXiv preprint arXiv:2307.05300 2023.
[43] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[44] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang. Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint 233i::300900766. 2023.
[45] Weijia Xu, Andrzej Banburski-Fahey and Nebojsa Jojic. Reprompting: Automated chain-of- thought prompt inference through gibbs sampling. arXiv preprint arXiv:2305.099. 09090,9993..0. 2023.
13

---

[46] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In seirtt- Conference on Neural Information Processing Systems, 2023.
[47] Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large language models. arXiv preprint arXiv:2308 04371, 2023.
[48] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference Learning Representa- lions, 2023.
[49] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023.
[50] Jianpeng Zhou, Wanjun Zhong, Yanlin Wang, and Jiahai Wang. Adaptive-solver framework for dynamic strategy selection in large language model reasoning. arXiv preprint arXiv:3310.01446, 2023.
14

---

A Additional Analysis
Examining the optimization process In our primary experiments, the stralegy generator initially produces 30 strategies, and we aim to obtain 10 qualified strategies for validation. For complex datasets, it difficult to directly acquire 10 qualified strategies without optimization. Table presents the optimization process for three challenging datasets: CP from the mathematical reasoning task, Strategy QA from the commonsense reasoning task, and MA from the algorithmic reasoning task, along with the iteration in which the optimal strategy (i.e., the strategy with the highest validation accuracy) is obtained. qualified strategies and superior strategies, indicating its significance in our framework.
evident that the strategy optimizer plays vital role in obtaining more
Table 8: Total count of qualified strategies achieved in each iteration of the optimization process and the specific iteration when the optimal strategy is attained.
Iteration
CP
StrategyOA
MA
9
2
12
10
9
10
Optimal
1
Analyzing the inference cost In this section, assess the inference associated with the optimal strategy-based prompt generated by our StrategyLLM framework, as well as the inference costs of the SolutionLLMM and CoT baselines. The costs represented by the average input and output tokens required for each test example, which are displayed in Table 9 Generally, LLM consumes
tokens during inference compared CoT and SolutionLLMM, as its prompt encompasses both general strategies and step-by-step solutions adhering to these strategies. In this paper, our primary objective is develop framework capable of generating generalizable and consistent prompts for various tasks without human intervention. To reduce inference costs, we may encourage LLMs create more succinet strategies by imposing additional constraints during the prompt generation process, which will be explored in our future work.
In main experiments, utilize 4 examples in the few-shot prompts of datasets within the MATH benchmark. To compare StrategyLLM with baselines of comparable costs, we construct CoT baselines comprising few-shot examples, with average input and output tokens the datasets of the MATH benchmark amounting to 2697 and 288, respectively. This baseline referred as CoT-8, and its first 4 examples are the same as CoT in Table The comparison results between CoT-8 and Strategy LLM are presented in Table 10 Our framework achieves marked improvement over CoT-8, showcasing its effectiveness.
Examining the impact of prompt consistency In our primary experiments, we have demonstrated that StrategyLLM outperforms inconsistent CoT prompts. To further examine the impact of prompt consistency, we intentionally create inconsistent prompts by employing multiple strategies. For each test example within a specific dataset, we generate an inconsistent few-shot prompt for by randomly selecting examples from different strategy-basee prompts. Specifically, we apply the top strategies randomly and uniformly to the examples the prompt. As result, the prompt fails to
Table Average inference cost of each test example across four reasoning tasks. and O denote the number of input and output tokens, respectively.
Math
Commonsense
Algorithmis
Symbolic
SolutionLLM (# I)
1693
740
373
301
SolutionLLM (# O)
373
90
103
48
CoT (# I) CoT (# O)
1332
487
830
261
304
50
331
65
Strategy (# I)
2649
2139
888
842
Strategy LLM (# O)
511
279
227
244
15

---

Table 10: Experimental results on the math reasoning task. The numbers in parentheses represent the relative improvement compared CoT-8-SC.
Methods
AL
IA
NT
PC
Avg
CoT-8
56.5
59.5
20.5
28.5
27.0
21.0
22.0
33.6
CoT-8-SC
61.0
57.5
17.0
38.5
29.0
23.5
18.0
34.9
StrategyLLM
58.5
57.5
18.0
35.0
29.5
24.5
22.5
35.1
Strategy
60.0
61.5
18.0
38.5
30.5
28.0
24.0
37.2
(+6.5%)
StrategyLLM-ZS
64.5
65.5
19.0
39.0
32.5
28.5
22.5
38.8 (+11.1%)
Table 11: Results of StrMtegyLLM using the best discovered strategy and the method employing inconsistent prompts.
Methods
CP
Strategy0A
MA
Avg
Inconsistent Prompt
29.0
56.5
77.3
54.3
LLM
35.0
67.5
86.7
63.1
offer consistent guidance for LLMs, requiring them to choose the most appropriate strategy for each test example based on their understanding of the strategies and the test example itself. We evaluate the effect of prompt consistency the CP, StrategyQA, and MA datasets. Table 11 presents the comparison between this method and StrategyLLM using the best discovered strategy. It is evident that the approach employing inconsistent prompts performs considerably worse than StrategyLLMM indicating that automatically identifying the most suitable strategy for each test example is quite difficult. Therefore, consistently applying effective strategy various examples within the prompt advantageous.
Assessing the complementarity of strategies To this end, we employ multiple strategies to derive various solutions and ascertain the answer by majority voting. Specifically, we employ the top 1, 3, 5, 7, and 9 strategies for the CP, StrategyQA, and MA datasets. Figure 7 unveils the following observations: (I) StrategyLLM-SC consistently surpasses CoT-SC on all three datasets when employing multiple solutions. suggesting that explicitly introducing effective strategies to obtain solutions is beneficial; (2) Leveraging multiple strategies outperforms the utilization of single strategy across all three datasets. implying that multiple complementary strategies exist in most scenarios; (3) Incorporating additional strategies generally demonstrates advantageous, However, this does not guarantee enhancement in performance, it relies on the effectiveness of the newly introduced strategies and their complementarity with pre-existing strategies.
Upper limit of accuracy with multiple strategies The upper limit of utilizing multiple strategies can be determined by calculating the coverage, which is defined as the percentage of examples that can be accurately solved by at least one strategy. The coverage represents the maximum potential accuracy achievable with multiple strategies. Figure g illustraaees the coverage and accuracy (i.e.,
72
40
70
90
66
35
StrategyLLM SC
B
Strnegy LLM SC QAT.SC
StrangyLL sc
CoT-SC
CaT-SC
64
7
3
9
5
Number of solutions
Number of solutions
Number solutions
(a) CP
(b) StrategyQA
(c) MA
Figure 7: Performance of StrAtegyLLM-SC and CoT-SC on the CP, StrategyQA. and MA datasets.
16

---

85
100
50
90
40
Coverage Accoracy
Covenge
Covargge
Accunacy
Accurayy
3
9
Number of solutions
Number of solutions
Number of solutions
(a) CP
(b) StrategyQA
(c) MA
Figure 8: Coverage and accuracy of StrategyLLM using multiple strategies on the CP, StrategyQA, and MA datasets.
through majority vote) of the StrategyLLM using multiple strategies on the CP, StrategyQA, and MA datasets. We can observe that there is a significant gap between accuracy and coverage, indicating that substantial performance improvements could be realized if the optimal strategy for each test example is selected rather than relying on simple majority vote.
Case study In this paragraph, we present a case study to demonstrate the advantages of Strate- gyLLM over CoT reasoning. We utilize an example with difficulty level of 5 from the CP dataset, as illustrated in Figure 9 which includes the example and solutions provided by both Strategy and COT. While CoT offers flexible reasoning process, it can sometimes lead inconsistencies or omissions in understanding and solving problems. Specifically, in the provided example, CoT mishandles rotations and reflections and fails to provide adequate details how address them. In contrast, StrategyLLM is designed to reduce the likelihood of misinterpretations and enhance reasoning through a structured approach. The generalizable strategy employed by Strategy facilitates comprehensive understanding of the problem, effective utilization of relevant concepts, and systematic breakdown of the task. This structured approach ensures that critical details are not overlooked, thereby leading to the correct solution. This case study clearly highlights the superiority of StrategyLLM handling complex reasoning tasks. Beyond accuracy, the solutions derived from StrategyLLMM are more detailed, interpretable, and accessible, making them particularly suitable for contexts that demand clarity and comprehensibility. such as educational settings.
B Implementation Details
The strategy generator creates 30 strategies using temperature sampling with temperature of The strategy executor, optimizer, and evaluator employ greedy decoding with temperature of The threshold of the execution accuracy for all datasets is 0.75, except for the CP and GE datasets. Their thresholds are set to 0.5 due to the challenging examples in their few-shot prompts. The maximum iteration number for strategy optimization is 3. We retain k 10 candidate strategies for validation, selecting the top or 3 strategies with the highest validation accuracy for inference.
C Prompts of Strategy Optimizer
For optimizing strategy, the strategy optimizer conducts the following tasks: (1) constructing the examination result for each task example by comparing the answer extracted from its execution result and the gold answer, which is demonstrated in Figure 10, (2) analyzing examination results of all provided task examples and providing suggestions for improving the strategy using the prompt in Figure [T] (3) modifying the strategy to obtain an updated strategy based on the feedback, i.e., the analysis and suggestions, leveraging the prompt in Figure [2
17

---

D Prompt of SLM-S
The prompt is shown Figure
E Prompt of SolutionLLM
The prompt is presented in Figure 14
F Prompt of CoT+Strategy
The prompt is demonstrated in Figure 15
G
Strategies and Execution Results
Figures 27 provide the best strategies discovered for various datasets and their execution results on task examples obtained by GPT-3.5. Figures 28 30 present the best strategies discovered for various datasets and their execution results task examples obtained by GPT-4.
H Chain-of-Thought Prompts for MATH Datasets
Figures 39 demonstrate the chain-of-thought prompts for the seven datasets from the MATH benchmark.
18

---

Case Study
How many ways there to arrange beads of distinct colors in 2 grid if reflections and rotations are considered the same? (In other words, two arrangements are considered the same if can rotate and/or reflect one arrangement to get the other.)
Question:
Gold Answer: 180
CoT Solution:
We can consider the grid as a 6-element cycle. There 6! ways to arrange the beads in line, but we must divide by 6 to account for rotations. However, this overcounts the number of arrangements by a factor of 2 because we are also counting reflections. Therefore, the final answer is G 6-2 60.
Answer: The answer is 60.
StrategyLLM Solution:
Understand the problem statement:
We have 6 beads of distinct colors.
We need to arrange these beads in a 2x3 grid.
Reflections and rotations are considered the same, meaning two arrangements are considered the same one can be obtained from the other by rotating or reflecting.
Identify the counting or probability concept: We need to use the concept of combinations to solve this problem.
3. Break down the problem:
We have beads to arrange in a 2x3 grid.
We need to consider the symmetries of the grid eliminate redundant cases.
We need to determine the number of distinct arrangements considering reflections and rotations.
Calculate the required counting probability: To calculate the number of distinct arrangements, we need to count the number of combinations that satisfy the given conditions.
We can use combinations to calculate this.
Let's consider the possible arrangements:
We can choose any of the beads for the first position the grid.
We can choose any of the remaining 5 beads for the second position.
We can choose any of the remaining 4 beads for the third position.
We can choose any of the remaining 3 beads for the fourth position.
We can choose any the remaining 2 beads for fifth position.
We can choose the last remaining bead for the sixth position. Total number of distinct arrangements
x x 2 720.
5. Simplify or reduce the problem:
We need to consider the symmetries of the grid to climinate redundant cases.
Thee 2x3 grid has rotational symmetry of order 2, meaning it can be rotated 180 degrees to look the same.
The 2x3 grid also has reflectional symmetry, meaning it can be reflected horizontally or vertically look the same.
Therefore, we need to divide the total number of distinct arrangements by 2 to account for the rotational symmetry.
We also need to divide the total number of distinct arrangements by 2 to account for the reflectional symmetry.
Calculate the total number of distinct arrangements: Total number of distinct arrangements 720 / 180.
Answer: The answer is 180.
19 Figure 9: Case study.

---

Strategy Optimizer Construction Examination Result
Example: {example}
Execution result obtained by executing the strategy the example: (execution result]
The answer extracted from the execution result is (predicted answer), and the correct answer is [gold answer). They are (different the same).
Figure 10: Construction of the examination result for a task example.
Strategy Optimizer Analysis & Suggestion
Task:
(task definition]
Strategy:
{strategy]
Examination results obtained by executing the strategy on the provided examples of the task and examining the execution results: /cxamination results)
We can see that we do not get the correct answer after executing this strategy on some of the provided examples. Please carefully analyze why the answers extracted from the execution results of these examples are incorrect and provide suggestions for improving the strategy.
Figure 11: Analysis & Suggestion Prompt of the strategy optimizer.
Strategy Optimizer Modification
Task: [task definition]
Some examples of the task are as follows: (task examples)
Original strategy to solve the task:
(original strategy]
Feedback: [feedback]
You need to modify the original strategy based on the feedback to obtain a better strategy.
The newly obtained strategy must be numbered list in the following format:
First subtask
2. Second subtask
Figure 12: Modification Prompt of the strategy optimizer.
20

---

StrategyLLMMZSS
Question:
{question}
Candidate Solutions: (candidate solutions)
Please determine the final answer for the question based the candidate solutions and their corresponding answers. The output should be in the following format: Final Answer: The final answer, which must include the string 'The final answer is
Figure 13: Prompt of StrategyLLMM-SS
LoLutionLLM
Task: (task definition)
Example of the task: (example)
Please write solution to the provided example. The answer ohtained from the solution must be the same as the original answer. The result must be in the following format: Question: Question in the provided example Solution: Solution the question
Answer: Answer in the provided example, which must include the string 'The answer is"
Figure 14: Prompt of Solution
CoT+Strategy
(CoT Prompt)
We show you several task examples of how to generate solutions. Please develop general strategy that consists of a sequence subtasks solve the task and apply the strategy the following question to obtain its solution and answer.
Question: (question)
The result must be in the following format:
Strategy: Strategy employed to solve the task
Question: Provided question
Solution: Solution obtained by applying the strategy to the question
Answer: Answer to the question, which must include the string 'The answer is
Figure 15: Prompt of CoT+Strateg.
21

---

MATH-Algehra
Strategy:
1. Simplify expressions involving radicals by multiplying and combining under single radical if possible.
Rationalize the denominators of fractions with radicals by multiplying the numerator and denominator by the conjugate of the denominator.
Use the quadratic formula factoring find the roots of quadratic equations.
4. Find the coordinates of points of intersection between two curves by solving the system of equations formed by setting the two equations equal each other.
5. Find the product or sum of the roots of a quadratic equation by applying Vieta's formulas.
Executing the strategy on a task example:
Question: The quadratic 8x² + 12r has two real roots. What is the sum of the squares of these roots? Express your answer common fraction in lowest terms.
Solution:
1. Simplify expressions involving radicals by multiplying and combining under single
radical if possible.
There are radicals in the given quadratic equation, simplification is needed.
Rationalize the denominators of fractions with radicals by multiplying the numerator and denominator by the conjugate of the denominator.
There are no fractions with radicals in the given quadratic equation, so no rationalization is needed.
3. Use the quadratic formula or factoring to find the roots of quadratic equations.
The quadratic formula is given by: =b -bt vt=-lue where ax + bz + 0.
For the given quadratic equation 8x + 12x 14, we have b 12, and = 14. Plugging these values into the quadratic formula, we get:
-22 122 (44)
2(8)
-22 448
12.1 592
12-44v37
337
I
Therefore, the two real roots of the quadratic equation are -3+v37 and v37
4. Find the sum of the squares of the roots.
Let r1 and r2 be the roots of the quadratic equation.
The sum of the squares of the roots is given by rf +r2 Plugging in the values the roots, we get:
r + r
33
337
+ 12 r
(37)
/37)2
ri + 2 3--377-37
9+6
/37-37
16
+ 2 2(9437)
16
r r2 16
2(4))
ri + r2
71 +
Answer: The answer is 4
Figure 16: Strategy generated by GPT-3.5 for MATH-Algebra and its execution on a task example.
22

---

MATH- HATHH-Peellgbba
Strategy:
Understand the problem: Read and carefully analyze the given problem understand the requirements and constraints.
Identify the known and unknown information: Determine what information is provided
and what needs to be calculated or determined.
3. Translate the problem into matherattiaa equations statements: Use appropriate
mathematical symbols, equations, or statements to represent the problem.
Solve the equations or statements: Use algebraic techniques to solve the mathematical equations or statements,
Verify the Solution: Substitute the found values back into the original problem to check they satisfy the given conditions.
Answer the question or provide the solution in the appropriate form, such fraction, decimal, or percentage.
Express the solution the required format:
Executing the strategy on a task example:
Question: Pete thinks of number. He doubles it, adds 10, multiplies by 4, and ends up with 120. What was his original number? Solution:
1. Understand the problem: Pete starts with number, performs series of operations it, and ends up with a final result. We need to determine the original number.
2. Identify the known and unknown information:
Known information: Pete ends up with 120 after doubling the number, adding 10, and multiplying by
Unknown information:
The original number Pete started with.
3. Translate the problem into mathematical equations statements:
Let x be the original number Pete started with.
Pete doubles the number: 2x
Pete adds 10: 2x + 10
Pete multiplies by 4: 4(2x 10)
Pete ends up with 120: 4(2x 10) = 120
Solve the equations or statements:
4(2x + 10) 120
Distribute the 4: 8x + = 120
Subtract 40 from both sides: 8x = 80
Divide both sides by 8: x 10
The original number Pete started with is 10.
Verify the Solution: Substitute the found value back into the original problem.
Pete doubles 10: 2(10)=2)
Pete adds 10: 20 10 =
Pete multiplies by 4: 4(30) 120
The final result is indeed 120, which matches the given information.
6. Express the solution in the required format: Answer:
The answer is 10.
Figure 17: Strategy generated by GPT-3.5 for MATH-Preeageeba and its execution on a task example.
23

---

MATTI-Intermediate Algebra
Strategy:
Begin by identifying the type of problem you are given: quadratic polynomial, algehraic equation, geometric problem, etc.
2. If the problem involves polymomial, start by determining the degree of the polynomial and then proceed solve for the unknown coefficients using the given conditions or given equations. the problem involves algebraic equations, simplify the expressions and solve for the unknowns by isolating the variable. Make use of algebraic rules such as distributive property. solving equations with fractions, and solving systems equations if necessary.
4. If the problem involves geometric concepts, analyze the given information and understand the geometric relationships involved. Apply the appropriate formulas theorems and solve for the unknown quantities.
the problem involves inequalities, solve for the range possible values by analyzing the conditions or constraints given. Make use of algebraic manipulation and reasoning skills to determine the range or the maximum/minimmu values.
If the problem involves functions, analyze the properties of the function and any given information determine unknown variables. such finding the maximun/minimun values, solving x-intercepts or y-intercepts, or finding asymptotes.
7. Check your solutions by substituting them back into the original equation or problem statement to ensure they satisfy the given conditions.
Exccuting strategy on example:
Question: Find the quadratic polynomial p(z) such that p(-3) = 10, p(0) = 1, and p(2) = 5..
Solution:
1. Identify the type of problem: quadratic polynomial.
Determine the degree of the polynomial and solve for the unknown coefficients using the given conditions.
Let the quadratic polynomial he p(z) = ax? + hx +
Using the given conditions:
p(-3)- 10 a(33)² b(-3) +c=
p(0)
a(0) 6(0) +
p(2 = = a(2)² ((2) =c=5
Simplifying the equations:
9a 1a-33 3b cc 10
c=
4a + +b++c -
Substituting e = 1 into the equations:
9a 3b + 10
1a 2b 11 =
Simplifying further:
9a 35
4a 1 26 4
3. Solve the system of equations using algebraic manipulation.
Multiplying the second equation by
9a
12a + 66 12
Adding the equations together: 21a 21 Dividing both sides by 21: 1
Substituting a into the sccond equation:
4(1) - 2b 4
4+ 26
=
Dividing both sides by
Substituting a and = into first equation:
9(1) 3(0) =
9 9
Therefore. 1. 0, and 1. Write the quadratic polynomial p(x).
p(z) = 1x2
Simplifying: p(z) x² +1
Answer: The answer is ?
Figure 18: Strategy generated by GPT-3.5 for MATH-Intermediate Algebra and its execution task example.
24

---

MATH--Cunting and Probability
Strategy:
Understand the problem statement:
Read and understand the given problem.
Identify what needs to he calculated or determined.
Make note of any special conditions restrictions mentioned.
Identify the counting or probability concept:
Determine which counting or probability concept needed solve the problem.
Common concepts include permutations, combinations, probability rules, etc.
3. Break down the problem:
Break down the problem into smaller subproblems cases if needed.
Identify any patterns symmetries that simplify the calculations.
Consider any specific conditions or restrictions mentioned in problem.
4. Calculate the required counting or probability:
Apply the appropriate counting or probability formula or rule to calculate the desired quantity.
Use combinations permutations for counting problems.
Apply probability rules like multiplication rule, addition rule, conditional probability, etc., for probability problems.
5. Simplify or reduce the problem:
Look for ways simplify reduce the problem.
Utilize symmetry equivalence eliminate redundant cases.
Consider any combinatorial identities or properties can simplify calculations.
Verify and check the Solution:
Double-ched the calculations to ensure accuracy.
Check if the solution satisfies any specific conditions restrictions mentioned in the problem.
Compare the solution with any provided answers or results.
7. Communicate the solution clearly:
Present the final solution in a clear and concise manner.
Include any necessary explanations steps taken to arrive at the solution.
Express the final answer the required units fraction, decimal, whole number, as specified by the problem.
Practice and review:
Solve additional problems on counting and probability to reinforce the concepts.
Review the solved problems to identify any common patterns strategies that can be applied to similar problems.
Executing the strategy on task example:
Question: Given that (?7) 19448, (:5 24310 and (57 24310, calculate (1.). Solution:
Understand the problem statement:
We are given three values of binomial coefficients: (47) 19448, (17) 24310, and (!5) 24310.
need calculate (2)).
2. Identify the counting or probability concept:
We need use the concept of combinations solve this problem.
Break down the problem:
We can use identity (2) (f=1) + ("r²) break down problem.
Using this identity, we can express (3)) terms of given values: (2))) = -(5) + +.
4. Calculate the required counting or probability:
Using the given values, we calculate (2)) and (35).
(18)
(17 19448 + 24310 = 43758
18
24310 + 24310 = 48620
5. Simplify reduce the problem:
We have calculated the values of (15") and (15)-
We can now substitute these values into the expression for (:) (2)
(2)) + (9)
13758 48620 92378
Verify and check the Solution:
The solution obtained is 92378, which matches the provided answer.
7. Communicate the solution clearly:
The value of (19) is 92378.
Answer: The answer is 92378.
Figure 19: Strategy generated by GPT-3.5 for MATH-Counting and Probability and its execution a task example.
25

---

MATH-Number Theory
Strategy:
1. Understand the problem: Read the problem carefully and identify what is being asked and what information given.
2. Apply basic number theory concepts: Use basic number theory principles such as modular arithmetic, divisibility rules, prime factorization, etc., to simplify the problem find relevant information.
Simplify expressions: Simplify any given expressions or equations to make them easier to work with.
Solve congruences: If the problem involves solving congruences (e.g., finding the remainder when number is divided by another number), the Chinese Remainder Theorem or specific methods for solving modular equations to find the solution.
5. Study patterns and properties: Look for patterns properties in the given problem that might help in finding the solution. This could involve using modular arithmetic properties, Euler's totient function, other relevant principles.
Use prime factorization: the problem involves finding factors or multiples of numbers, the prime factorization of the numbers involved to simplify the calculations.
7. Apply number theory theorems: the problem relates to specific number theory theorems, such as Fermat's Little Theorem Wilson's Theorem, apply those theorems to find the solution.
Check for divisibility: Use divisibility rules to check if number divisible by another number, or find other relevant information that can help in solving the problem. Solve step-by-step: Break down the problem into smaller steps and solve them one by one to find the final solution.
10. Check the Answer: Always double-check your final answer and ensure that it satisfies the given conditions or requirements of the problem.
Executing the strategy a task example:
Question: What is remainder when 2007 divided by 81?
Solution:
Understand the problem: We are asked to find the remainder when 2007 is divided by 81.
2. Apply basic number theory concepts: We can the concept of modular arithmetic to solve this problem. modular arithmetic, we find the remainder when a number is divided by another number.
3. Simplify expressions: There are no expressions to simplify in this problem.
4. Solve congruences: To find the remainder when 2007 is divided by 81, we write it congruence: 2007 = x (mod 81), where x is the remainder we want to find.
5. Study patterns and properties: There no specifie patterns properties consider in this problem.
6. Use prime factorization: We can the prime factorization of 81 simplify the calculations. 81 = 34.
Apply number theory thcorems: There are no specific number theory theorems to apply in this problem.
Check for divisibility: We can check 2007 divisible by 81. If is divisible, then the remainder will be O. Otherwise, we need to find the remainder.
Solve step-by-step:
Divide 2007 by 81: 2007 = 81 = 24 with remainder of 63.
The remainder 63.
10. Check the Answer: The remainder we found, 63, is the same the original answer. Answer: The answer 63.
Figure 20: Strategy generated by GPT-3.5 for MATH-Number Theory and its execution on a task example.
26

---

MATTH-Geometry
Strategy:
To find the area of shaded region in circle sector overlap problem:
a. Calculate the area of each sector using the formula Asectrr = fr200, 0, where r the radius and is the
central angle.
b. Determine the area of the overlapping region by subtracting the area of the triangle formed by the radii from the sum of the areas of the two sectors.
Double-check the calculations and verify obtained answer against correct answer. To calculate the perimeter of a quadrilateral formed by cutting a square with a line:
Find the coordinates of the intersection points between the line and the square. b. Calculate the lengths of the sides of the quadrilateral using the distance formula.
Add up the lengths of the sides find the perimeter.
Double-check the calculations and verify obtained answer against correct answer.
3. To determine the height of cylinder inscrihed in a hemisphere:
Use the Pythagorean theorem to find the radius of the cylinder's 's base by subtracting the height of the cylinder from the radius of the hemisphere.
The height of cylinder is equal to the radius of hemisphere minus radius of cylinder's base.
Double-check the calculations and verify the obtained answer against the correct answer.
4. To reflect parallclogram across the x-axis:
Change the sign of the y-coordinate of each vertex obtain the reflected image of the parallelogram. Double-check the coordinates of the reflected vertices and verify against the correct answer.
5. To reflect a parallelogram across given line:
Find the equation of the given line.
b. Calculate the distance between each vertex of the parallelogram and the given line.
Reflect each vertex across the given line by moving it the same distance on the opposite side of the line.
d. Connect the reflected vertices to form the reflected parallelogram.
Double-check the coordinates of the reflected vertices and verify against the correct answer. To find the ordered pair of reflected point:
Determine the equation the line of reflection.
b. Use the formula for the reflection of point across a line to find the coordinates of the reflected point. Double-check the coordinates of the reflected point and verify against the correct answer.
7. Double-check all calculations and results to ensure accuracy.
Verify the obtained answers against the correct answers.
Review all steps and calculations to identify any errors or discrepancies.
c. Make necessary corrections and adjustments to improve the accuracy of the solution.
Executing the strategy Question: Parallelogram ABCD with A(2, 5). ((4,)) C(6, 5), and D(4, 1) is reflected across the x-axis to A' B' C' and then A' B' C'C'D" D' is reflected across the line = x 1 to A" B" C" D". This is done such that D' is the image of D, and D is the image of D', What is the ordered pair of D in the coordinate plane?
task example:
Solution:
Reflect parallelogram ABCD across the x-axis to obtain parallelogram A' B'
Change the (4, -9) (6 D'(4
sign of the y-coordinate of each vertex: (2, 5)-
2. Reflect parallelogram A' B' C' D' across the line
+ to obtain parallelogram A" ("""""" D": The slope of the line is 1, and the y-intercept is 1. The
Find the equation of the line y equation the line y =
Calculate the distance between cach vertex of parallelogram A' B'C' D' and the line = x 1: Distance from A' to the line = x 1: Substitute the coordinates of A' into the equation of the line: - 2 + -5 5- 3- The distance
5 3) 8.
Substitute the coordinates of B' into the equation of the line: - - = + -9 = 5- The distance is
Distance from B' the line y ==
9 = 14.
Distance from C" to the line y x + 1: Substitute the coordinates of C' into the equation of the
line: - - =
-5 7- The distance is
12.
Distance from D' to the line y
Substitute the coordinates of D) into the equation of the 5- The distance is
-
c. Reflect cach vertex of parallelogram A' across the line = x + by moving the same distance on the opposite side of the line: Reflect A' across the line = x + 1: Move A' 8 units in the opposite direction of the line's 's's normal vector: (2
5 8) = (-6.3
Reflect B' across the line y =
Move B' 14 units in the opposite direction of the line's normal 14) = (-10,
vector: (4 14,
Reflect C" across the line y = x + 1: Move C" 12 units in the opposite direction of the line's normal
vector: - C6 12, -5 + 12) = (-6,
Reflect D' across the line
Move D' 6 units in the opposite direction of the line's normal
(1
1 6) (-2,5)
27
The ordered pair of D" (-2,5).
Answer:
The answer is (-2, 5).

---

MATH-Precalcuute Precalculus
Strategy:
1. Identify the problem or concepi being tested in the given question.
Define any necessary terms formulas related the problem.
3. Simplify or manipulate any given expressions or equations to solve the problem.
Solve the problem step by step with the help of appropriate formulas or techniques.
5. Verify the solution or answer to ensure its correctness.
Repeat steps 1-5 for each problem in the precalculus subject.
Executing the strategy on a task example: Question: The function
f(=) (v1+ + (-2v3- 18i)
represents rotation around some complex number Find c. Solution:
1. Identify the problem concept being tested in the given question.
The problem is asking us to find the complex number e that represents the center of rotation for the given function.
2. Define any necessary terms or formulas related to the problem.
The formula for rotation around complex number is given by f(z) ei c) cc where 9 is the angle of rotation.
Simplify or manipulate any given expressions or equations solve the problem.
The given function can be rewritten as f() = ==tiv 2-/3-18
4. Solve the problem step by step with the help of appropriate formulas techniques.
Comparing the given function with the formula for rotation, we can see that e 1+iv3
2
and 2
-2 18: To find 0, we can take the argument of e00 which is 0 = arg
2
Simplifying the expression inside the argument, we get =1+iv3 -2+2iv3
+ iv3
2
v3 Thhe argument of this complex number is = arctan -1
arctan -33.. Using the
unit circle, we can determine that 0 3.- Therefore, the angle of rotation is - 5. Verify the solution or answer to ensure its correctness. Substituting /
into the formula for rotation, we get f(z) e (2- c. 4
Simplifying this expression, we have f(=) = 2 (cos (- --5 + issn
(= c) c.
Using the trigonometric identities cos(-5) 1 and sin(- - we can further
simplify the expression to f(=) = 2 v3 (z - c) + Distributing and simplifying. /3 3i(z c) + Combining like terms, have f(z)
get f(z) {=
4 3 z + v3
Comparing this expression with the given function, we can
that 1 33i= -l+iv3 and + v3 -2V3-18. Therefore, our solution is correct.
2
Answer: The answer is /3- 56.
Figure 22: Strategy generated by GPT-3.5 for MATH-Precaleulus and execution on a task example.
28

---

StrategyQA
Strategy:
given information and 2. Identify the type of question being asked, whether it is a simple "yes or no" question if there is need to provide specific answer.
Read and understand the question fully by carefully analyzing requirements.
Determine the key elements and concepts mentioned the question that need addressed.
4. Break down the question into smaller parts, if necessary. to identify any hidden information or assumptions.
5. Look for any clues hints within the question that help in finding the answer. Decide the approach comparison, or analysis.
used, whether it requires logical reasoning. knowledge of specific facts,
7. Use prior knowledge and general understanding to answer the question, if applicable.
8. If additional information is required, either use external sources gather relevant data from memory. Organize the information and answer the question clearly and concisely.
10. Review the answer ensure accurately addresses the question and provides correct response.
11. If necessary, double- cuuble-check the answer to ensure aligns with the requirements and request.
12. Finally, present the answer in clear and understandable manner, either as "yes or no" response or specific answer required by the question.
Executing the strategy on task example:
Question: Could Brooke Shields succeed at University of Pennsylvania? Solution:
1. Read and understand the question fully by carefully analyzing the given information and requirements.
The question asking Brooke Shields could succeed the University of Pennsylvania.
Identify the type of question being asked, whether is a simple "yes or no" question or there a need to provide specific answer.
The question requires simple "yes or no" answer.
Determine the key elements and concepts mentioned the question that need be addressed.
Key elements: Brooke Shields, University Pennsylvania succeed.
4. Break down the question into smaller parts, if necessary, to identify any hidden information or assumptions.
No need to break down the question further.
Look for any clues hints within the question that help in finding the
The question does not provide any specific clues or hints.
Decide on the upproach to be used, whether it requires logical reasoning, knowledge of specific facts, comparison, analysis.
The approach requires logical reasoning and knowledge of the University of Pennsylvania.
Use prior knowledge and general understanding to answer question, if applicablle.
The University of Pennsylvania is a prestigious university known for its rigorous academic programs. Brooke Shiclds, as successful actress and model, may have the potential to succeed at the university.
8. If additional information is required, either external sources gather relevant data from memory. No additional information required.
answer the question clearly and concisely. The answer is yes, Brooke Shields could succeed at the University of Pennsylvamia.
Organize the information
10. Review the answer to ensure accurately addresses the question and provides correct response.
The answer accurately addresses the question and provides correct response.
11. If necessary, double-check the answer ensure aligns with requirements and request.
The answer aligns with the requirements and request.
12. Finally, present the answer in clear and understandable manner, cither as "yes or no" response or specific answer required by the question.
Answer: The answer is yes.
Figure 23: Strategy generated by GPT-3.5 for Strategy QA and its execution on task example.
29

---

Date Understanding
Strategy:
Determine the current date based on the given information.
2. Calculate the number of hours or days given in the question.
Determine the time period given
the past or future.
4. Adjust the current date based the time period and direction (past future).
5. Convert the adjusted date to the required format (MM/DD/YYYY) as specified in the question. Provide the final date as the answer to the question.
Executing the strategy on a task example:
Question: The concert was scheduled to be on 06/01/1943, but delayed by one day to today. What is the date 10 days ago in MM/DD/YYYY? Solution:
Determine the current date based on the given information.
The concert was delayed by one day, so the current date 06/02/1943.
2. Calculate the number of days given in the question.
The question asks for the date 10 days ago.
3. Determine if the time period given is in the past or future.
The time period is in the past.
4. Adjust the current date based on the time period and direction (past or future).
Subtract 10 days from the current date: 06/02/1943- 10 days = 05/23/1943.
5. Convert the adjusted date to the required format (MM/DD/YYYY) as specified in the question.
The adjusted date is already in the required format: 05/23/1943. Provide the final date as the answer to the question.
Answer: The answer is 05/23/1943.
Figure 24: Strategy generated by GPT-3.5 for Date Understanding and its execution on task example.
Word Sorting
Strategy:
Identify the list of words that needs to be sorted.
Split the list words into individual words.
3. Sort the individual words alphabetically.
Combine the sorted words back into a single list.
5. Present the sorted list of words as the answer.
Executing the strategy a task example: Question: Sort the following words alphabetically: List: oven costume counterpart
Solution:
Identify the list words that needs to be sorted.
The list of words "oven costume counterpart".
2. Split the list of words into individual words.
The individual words are "oven", "costume", and "counterpart".
3. Sort the individual words alphabetically.
The sorted words are "costume". "counterpart", and "oven".
4. Combine the sorted words back into a single list.
The combined sorted list "costume counterpart oven".
5. Present the sorted list of words as the answer.
Answer:
The answer is costume counterpart oven.
Figure 25: Strategy generated by GPT-3.5 for Word Sorting and its execution on a task example.
30

---

Multi-step Arithmetic
Strategy:
Identify the innermost operations enclosed parentheses.
2. Evaluate the innermost operations first.
Replace the evaluated expression with the result.
4. Repeat steps 1-3 until all parentheses are removed.
Evaluate the remaining arithmetic operations according to the order of operations (multiplication and division before addition and subtraction).
Evaluate all multiplication and division operations from left to right.
7. Replace the evaluated expression with the result. Repeat steps 5-7 until there are no more multiplication and division operations to be performed.
9. Evaluate all addition and subtraction operations from left to right.
10. Replace the evaluated expression with the result.
11. Repeat steps 9-10 until there are no more addition and subtraction operations to be performed.
12. The final result is the solution to the multi-step arithmetic problem.
Executing the strategy on a task example:
Question: ((-9 * 7 x -9) + (4 *
Solution:
Step Identify the innermost operations enclosed in parentheses.
The innermost operation is (4 -9- 8 -4).
Step 2: Evaluate the innermost operations first.
4 -9 -36
-36 =44
-44 44-
Step 3: Replaaee the evaluated expression with the result.
(((-9 * * -9) -40)
Step 4: Repeat steps 1-3 until all parentheses are removed.
The innermost operation is (-9 * * * -9).
Step 5: Evaluate the innermost operations first.
-9 7=
-63 * =
-441 * -- 3969
Step 6: Replace the evaluated expression with the result.
(3969 -40)
Step 7: Repeat steps 5-6 until there are no more multiplication and division operations to be performed.
There are no more multiplication and division operations to be performed.
Step 8: Evaluate all addition and subtraction operations from left to right. 3969 -40 3929
Step 9: Replace the evaluated expression with the result. Answer: The answer is 3929.
Figure 26: Strategy generated by GPT-3.5 for Multi-step Arithmetic and its execution task example.
31

---

Last Letter Concatenation
Strategy:
Identify the given input, which will be string containing multiple words separated by spaces.
2. Split the input string into individual words.
3. Initialize an empty string to store the last letters of the words.
4. Iterate through each word in the list of words.
5. Get the last letter of each word.
6. Concatenate the last letter the string from step
7. Continue steps 4-6 until all words have been processed.
8. Print return the final concatenated string.
Executing the strategy on a task example:
Question: Take the last letters of the words in "Sergey Brin" and concatenate them.
Solution:
1. Identify the given input: "Sergey Brin"
Split the input string into individual words: ["Sergey", "Brin") "]
3. Initialize an empty string: last letters = ""
Iterate through each word in the list of words:
For the word "Sergey":
Get the last letter: last_ let_letter== "y"
Concatenate the last letter to the string: last letters = "y"
For the word "Brin":
Get the last letter: last_ lest__ettr==== = "n"
-Concatenate the last letter to the string: last_ letters = "yn"
5. Print return the final concatenated string.
Answer: The answer is yn.
Figure 27: Strategy generated by GPT-3.5 for Last Letter Concatenation and its execution on task example.
32

---

MATH-Counting and Probability
Strategy:
1. Identify the type of the counting or probability problem presented.
If the problem regarding permutations or combinations, identify all possible outcomes and arrange or select the required number of objects accordingly.
3. If the problem about probability, assess the sample space, the number of possible outcomes and the number of favorable outcomes.
For probability questions, use the formula of probability, which is the number of favorable outcomes divided by the number of total outcomes.
If the problem involves conditions or restrictions, ensure factor in these when calculating probability counting the possible arrangements selections.
If the problem involves complex calculations, break them down into smaller manageable calculations.
Apply the suitable rule or formula according to the type of counting or probability problem, such multiplication rule, addition rule, arrangement rules, rules of permutations and combinations, or probability rules.
8. Compute the result using arithmetic calculation or mathematical formulas, and write down the answer.
9. Review the process and the to ensure accuracy.
Executing the strategy on a task example:
Question:
Given that (?7) 19448, (17) 24310 and (?5) 24310, calculate (19)
Solution:
1. The problem is about combinations, which is type of counting problem.
2. The problem is asking for the number of ways to choose 9 items from a set of 19, which is represented by (29).
3. We are given the values of (?7). (3)) and (0)), which can be used to calculate (19)- n!!
4. The formula for combinations is (:) rl(n-r)!" where is the total number of items, is the number of items choose, and "!' denotes factorial.
5. However, we can use the property of combinations that (*) = -n",), and the relationship
between (1+1) and (f"1) and (+), which is (1+1))
+ (*)
6. Using these properties, we can express (29)
118
(1))
Similarly, (2S) can be expressed (27)
(17) and (15) can be expressed (?7) + (17).
8. Substituting the given values, we get 9
19 (19448 + 243100) + (24310 + 24310).
9. Performing the arithmetic calculation, get (19) 43758 48620 92378.
Answer: The answer is 92378.
Figure 28: Strategy generated by GPT-4 for MATH-Counting and Probability and its execution on task example.
33

---

StrategyQA
Strategy:
Identify the main subject(s) or concept(s) in the question.
2. Identify the relationship(s) action(s) associated with the main subject(s) concept(s).
3. Break down the question into sub-questions based on the identified subjects or concepts and their associated relationships information required to answer the main question.
actions, ensuring each sub-question leads additional
For each sub-question, infer or retrieve relevant information. This come from provided information, general knowledge, or external resources.
Compare and synthesize the information gathered from each sub-question.
6. Based on the synthesis, determine if the information satisfies or contradicts the main question.
7. Formulate yes' or 'no' answer to the main question based on the evaluation. Confirm that the answer accurately addresses the main question. If not, revisit the sub-questions and retrieval process. Record the answer.
Executing the strategy on a task example:
Question: Could Brooke Shields succeed at University of Pennsylvania?
Solution:
1. The main subject in the question is Brooke Shields and the main concept is University of Pennsylvania.
2. The relationship associated with the main subject and concept is Brooke Shields' potential success at the University Pennsylvania.
Sub-questions based on the identified subjects or concepts and their associated relationships or actions could be:
Who is Brooke Shields?
What is the University of Pennsylvania?
What does it mean to succeed at the University Pennsylvania?
4. For each uub-Question:
Brooke Shields a well-known American actress and model.
The University of Pennsylvania is an Ivy League research university in Philadelphia. It is one of the most prestigious universities in the world.
To succeed at the University of Pennsylvania would likely mean to graduate with a degree, possibly with honors or high academic achievement.
5. Comparing and synthesizing the information gathered from each sub-question, we know that Brooke Shields is a successful actress and model, and University Pennsylvania is prestigious university. Success at the university would likely require dedication, intelligence, and hard work.
6. Based on this synthesis, it is possible that Brooke Shields could succeed at the University of Pennsylvania, given her demonstrated success in her career. However, without specific information about her academic abilities aspirations,
cannot definitively say. Given the information available, we can infer 'yes' answer to the main question.
8. The answer 'ees' accurately addresses the main question, given the information available.
9. The answer is recorded.
Answer: The answer is yes.
Figure 29: Strategy generated by GPT-4 for StrategyQA and its execution on a task example.
34

---

Multi-step step Arithmetic
Strategy:
Begin by identifying all the numbers and operations the given problem.
2. Apply the mathematical rules of order operations. The correct order of operations is Parentheses, Exponents, Multiplication and Division (from left to right), Addition and Subtraction (from left right). This rule is often abbreviated PEMDAS.
3. Start solving the problem from the inside of the parentheses first, keeping PEMDAS rule in mind.
While doing multiplication, division, addition, and subtraction (in the order), make sure to change the sign of the number if there is a negative sign before it.
5. After solving the operations inside the parentheses, proceed solving the operations outside the parentheses. Remember still follow the rules and order of operations.
6. Continue to calculate step by step until you reach the final answer.
Executing the strategy on a task example:
Question: ((-9 7 # -9) + (4 *
Solution:
Identify all numbers and operations the given problem. The numbers are -9, -9, 4,
-9, 8, -4. The operations are multiplication (*), addition (+), and subtraction (-).
2. Apply the mathematical rules of order operations (PEMDAS).
3. Start solving the problem from the inside of the parentheses first.
The first parentheses is (-9 7 -9). According to the order of operations, we should do
multiplication first. So, -9 * 7= -63, -63 * 7 -411, -441 -9 3969.
The second parentheses is (4 -9-8
8 -4). According to the order of operations, we should
do multiplication first and then subtraction. So, 4 -9 -36, -36 8 -44, -44 -4 -40. After solving the operations inside the parentheses, proceed solving the operations outside the parentheses. So, 3969 -40 3929.
5. Continue to calculate step by step until you reach the final answer.
Answer: The answer is 3929.
Figure 30: Strategy generated by GPT-4 for Multi-step Arithmetic and its execution on task example.
35

---

CoT Prompt for MATH-Algeber
Calculate v v2x- v14z. Express your answer in simplest radical form in terms of z. Solution: Writing everything in terms of prime factorizations, the given expression v33-5 2- 2²
Question:
/ (2 50 .4) (3.7.x) 10ry2Ir.
Answer: The answer is 10r/21z
v/6-v2 The answer can be written Av/B4C where A, B, C, and D are integers, D is positive, and B is not divisible by the square of any prime. Find the minimum possible value of A + B +
Question: Rationalize the denominator of V32
Solution: We can write the numerator as 4v2, the denominator as v2. Then, we multiply the numerator and denominator by the conjugate of the denominator.
4v2
4+
4+
4
/2
4+
/2
16 2 +
14
8 +
Therefore, A + B + C+ = 8 21.
Answer:
The answer is 21
The quadratic 8r + 12x 14 has two real roots. What is the sum of the squares of these roots? Express your answer common fraction in lowest terms.
Question:
equation 8x² + 12x 14. want find æी xΑ. Note that of + x² = (71 + .2) 2x1 We know that Ø1 + 22, sum of the roots, is equal to =b
Solution: ø1 and ø2 be the roots
which for this equation is =12 =3 Likewise, know that 11 the product of the roots, equal
e, which this equation
- Thus, of + x. (=))
(=)
23
Answer: The answer is 23
Question: What is the product of all the coordinates of all the points of intersection of the two circles
defined by x² 2x + y²² 10y 25 0 and x² 8r y² 10y + 37 02
Solution: Add (-2/2)² and (-10/2)² to the first equation and (-8/2)² and (- 10/2)² the second
equation to that the given equations are equivalent to
+ 2z
-
(y² 10y + 25) = 1, and
(-² 8c + 16) + (y² - 10y + 25) = 4
which equivalent
(x - 11- 5) 12,
(ar 1) 5)2 23
respectively. Hence, the two circles have centers (1,5) and (4, 5) respectively and radii 1 and respectively. Since the centers of the circles are 3 units apart and the sum of their radii is 3, the two circles intersect only point. We can that (2, 5) is the desired intersection point, product is 2- 10.
Answer: The answer is 10
Figure 31: Chain-of-thought (CoT) prompt for MATH-Algebra.
36

---

CoT Prompt for MATH-Prealgebra
Pete thinks of number. He doubles it, adds 10, multiplies by 4, and ends up with 120. What was his original number?
Question:
Solution:
Let Pete's ' original number be æ. If he double it and adds 10, he will have 2x + 10. After multiplying by 4, Pete ended up with 120. From this information, we have the equation: 4(2x 10) 120. Expanding the left side and solving, we find:
8x + 40 = 120
8r 80
I = 10.
Answer:
The answer is
Question: A standard deck of 52 cards has 13 ranks (Ace, 2, 3, 4, 5, 7, 8, 9, 10, Jack,
Queen, King) and 4 suits ( V. O and 4) such that there is exactly one card for any given rank and suit. Two of the suits and ) black and the other The deck is randomly arranged. What is the probability that the top card is a ???
suits ( and 4) are red.
Solution: There are 13 Q and 52 cards total, the probability that the top card is V is
1
52 4'
Answer: The answer is }
Question: number in the set (50, 51, 52, ..,,,, is randomly selected. What is the probability that it is two-digit number? Express your answer common fraction. Solution: To count the number of numbers in this set, we subtract 49 from all of the numbers, giving the set (1, .,5)), making it obvious that there are 950 numbers total. Furthermore, the set (50, 51, 52, ..... 98, 99} corresponds to the more easily counted 21,2,3,... ...,, 49, 50} by subtracting 49. So, the probability of selecting two-digit number is 50
950
I9*
Answer: The answer is 19
Question: A pentagon is drawn by placing an isosceles right triangle on top of square as pictured. What percent of the area of the pentagon is the area of the right triangle? [asy] size(50);
[/asy]
Solution: Let the leg length of the isosceles right triangle be so the hypotenuse of the triangle has length 2. The hypotenuse of the triangle is a side of the square, so the area of
the square is (xv/2)2 = 2x.2 The area of the triangle is (x) = x²/2. So, the area the pentagon is
2
5
2x
Therefore, the fraction of the pentagon's area that is inside the triangle is
2²/2
x²
2
20%.
5x2
/2
2
5
Answer: The answer is 20
Figure 32: Chain-of-thour (CoT) prompt for MATH-Preallebra.
37

---

CoT Prompt for MATH Intermeeeteeeeee Algebra
Question: Find the quadratic polynomial p(r) such that (-33) 10, p(0) 1, and p(2) 5. Solution: Let p(x) ox + be Then from the given information,
9a + c 10,
c c= 1,
4a + 26 c= 5.
Then 9a 3b = and 4a + 2b 4, which reduce to 3a - b 3 and 2a 2. Adding, get 5a Then 4 26 4, b 0. Therefore, p(x) x² Answer: The answer x² +
Question: Calculate: ;
x
Solution: Each of the fractions 3'
5
15 25 d0
21 reduce to 3 and cach of the fractions
12 18 24
9 15
10 20 30
reduce to 3. Therefore, the product all eight fractions is I. Answer: The answer is
Question: 25 x S 25, find maximum value /25 + + 25 x. Solution: By QM-AM,
+
25 ạ
25 + x 25 5,
so v 25 22
10. Equality occurs a
0, so the maximum value
Answer: The answer is 10
Question: The hyperholas
x²
y²
and
y²
2
18
have the same asymptotes. Find N. Solution: In general, for the hyperbola =3
8 = 1, the asymptotes are _ = = = +
Therefore, the asymptotes of the first hyperbola are y = 1. For the hyperbola 3 -े 1, the asymptotes are = or #fr. Therefore, the asymptotes of second hyperbola are y + av2 v/ x.
For the two hyperbolas to have the N = 8.
asymptotes, must have Solving for V gives
[asy] void axes(real x0, real x1, real y0, yl) draw((x0.00-xx11))) EnddArEwdrr):: label("x" ()., (ooel("".(00.y1)..)" for (int i=fioor(x0)+1; isxl; ++i) (i. for (int i=floor(00)+1: icyl; ++1) draw((.)))-<-LLL)):: path[] yh(real a, real b, real h. real k. real x0, real x1. bool upper-true, bool lower=true, pen color=black) real f(real x) return l + a/b sgri(b2 + (x h)²); real g(real x) return 4: - a/b agrt(b² (v. (62-h)3 h)²); if (upper) draw(graph(f, x0, x1),color, Arrows); if (lower) draw(graph(g, x0, x1),color, Arrows); path N graph(f. x0, x1), graph(g. x0, x1); return arr; void xhíreal a, real b, real h, real k, real y0, real yl, bool right=true, bool left=true, pen color=black) | path [ arr yh(a, b. k. h. y0, y1. false, false): if (right) Arrows); if (left) draw(reflect( (I.))
draw((0.y00-
Arrows); size(8em); axes(-8,8, -10, draw((-6,9) (6.-9)66.9) (0669)-(66..)) (-6.,9),dottedd [/asy]
void e(real real b, real h, real k) *aahiit(( 10); xh(2, 0, 8); yh(3**s (Q(""ss(()))ss((())...... Answer: The answer is
Figure 33: Chain-of- of-thouggtt (CoT)) prompt for MATH-Intermediate Algebra.
38

---

CoT Prompt for MATH-COTE CounttCnnning and Probability
The digits 2, 3, 5 and 7 arranged randomly form four-digit number. What is the probability that the number is odd? Express your answer as common fraction.
Question:
Solution: The number formed is odd if and only if its units digit is not 2. Since the digits 2, 3, 5, and 7 are arranged randomly, the probability that 2 is the units digit is 1/4. Therefore, the probability that the number is odd is 1 - 1/4 Answer: The answer is
Question: Eight congruent equilateral triangles, each of different color, are used to construct a regular octahedron. How many distinguishable ways are there to construct the octahedron? (Two colored octahedrons are distinguishable if neither can be rotated to look just like the other.) [asy] import three: import math; unitsize(L.5em) triple pnteeuppooecooooooorerrrrp......
triple
A=(0.0.1); triple B=(sqr((2)22sqqr((22)
triple (=(-str((222
sqr((2)2,0); triple (x( triple F=(0,0,-1); draw(A (eaw(A-B-aa (A-B-FF-cyclee, draw(A-C-DD
cycle); draw(F-C-B-cycle;;
[/asy]
(A) 210
(B) 560 (C) 840 (D) 1260
(E) 1680
Solution: Since the octahedron is indistinguishable rotations, without loss of generality fix face be red.
[asy] size(8cm); defaultpen(0.S); import three; import math; curreapppietiooooottogopppppl..p triple A=(0,0,1); triple (gt triple C=(sgr((2)2 -qmp(2222 triple (= sqrt(2)2,0); triple B=(=sqrt(2/2ssrr1222220 D-cycle); draw(F-CC C- B- C-B-cycle;; draw E sycle.dotcc--linevviih0...777771 There are 7! ways to arrange the remaining seven colors, but there still are three possible rotations about the fixed face, so the answer is 7!/3 1680.
triple F=(0,0,-1); draw(A-B Bee-------eeeee E-cycle); draw(A C-
draw(surface(A
C-
(ycle.rgb(1.6..))moofiih
[asy] size(8em); defaultpen(0.55) import three; import math; currenprojcciiossaatoggppppip triple A=(0,0.1); triple B=(ssrt(2V2.sqr11222200 triple E=(sgrt(2222-sqrr(22222 triple D=(- =qr(2222.-q1((2222.3) triple E=($spn((222sspp((222.00) picture p = new picture, = new picture, s = new picture: B cyclee) Crawp.A-C-D--yee
triple F=(0,0,-1): triple right=(0.1.0);
(raw(p.F-C-B-cycle) Htt.:p.--D----reel..oodd--hhhoho
wwt.saaaaccA.---CCyyyo.rob.....a C-cyylo,.gb dww..p/wwffaacc..BBBB..oooo..... draw(p.surface(A- A-B pd drawir.A---B-cycle draw(r.A-C-D-cyele;; draw(r.F-C-B-cyele
addisalee322))sshii2hiijjjj Braw(s.A-B-E-
cycle); (raw(s.--C-D-cycee)
Frawss,F--B--yccee
Set.ssssuraaee------eyeeee ot noolghh)
add(scale3(2.2) "right)) [/asy]
Answer: The answer is 1680
Question: The Gnollish language consists of words, "splargh," "glumph," "glumh" and "amr." In sentence, "splargh" cannot come directly before "glumph"; all other sentences are grammatically correct (including sentences with repeated words). How many valid 3-word sentences are there Gnollish? Solution: We proceed by counting the complement, or the number of invalid 3-word sentences. A sentence is invalid precisely when is of the form "(word) splargh glumph" or "splargh glumph (word)." There 3 choices for the missing word each sentence, and since each case is exclusive, have total 6 invalid sentences. Since there are 3 3-3.3 = 27 possible 3-word sentences with no restrictions, there are 27- - 6 = 21 that satisfy the restrictions of the problem. Answer: The answer is 21
Question: Given that (17 = 19448, (17) = 24310 and (17) 24310, calculate (19)-
Solution: We can apply Pascal's 'ss identity to get that (10) = (10) + (18), From here, we can apply
twice more to get that (199 = (19) (19) ((7) + (2)) ((2) (H)). Substituting the provided
values of (27. (17), and (17) gives (19) 19448 + 2(2331)) 21310 92378.
Answer: The answer is 92378
Figure 34: Chain-of-thougee (CoT)) prompt for MATH-Counting and Probability.
39

---

CoT Prompt for MATH-Numbe Theory
Question: Let be positive integer such that 9z = 1 (mod 25). What is the remainder
when 11 + z is divided by 25?
Solution: The given information can be expressed by writing = 9-1 (mod 25). Thus wish to compute 11 + 9-1 (mod 25).
Modulo 25, we can write 11 as 11 (9 9- (11 9-1 99 9-1 Thus
11 11 - = 99- +1.9--
100-9-1
0
0 (mod 25).
so the remainder when 11 + x is divided by 25 is O.
Notice that the trick we used here analogous using a common denominator add fractions.
Answer: The answer is
Question: Let be the product of all positive integers less than 4! which are invertible modulo 4!. Find the remainder when is divided by (Here n! denotes 1 X for each positive integer n.)
Solution: We compute that 4! x x 3 2 x = 24. So we want exactly the numbers in the set {1,..., 24} which are divisible by neither 2 nor 3, since an integer is invertible modulo n for some positive integer n and only if ged((a (, n) These turn out to be {1, 5, 7, 11, 13, 17, 19, 23). Then
m = 1- 5- 11 13 17- 19 23
1-55-7 11 (-11) (111)(77)))))) (-5) (-1)
(5-7-11)* (35 11)²
(11-
= (121)²
1
(mod 24)
Answer:
The answer is
Question: Find 2-1 (mod 185), as residue modulo 185. (Give an answer between 0 and 184, inclusive.)
Solution: Since 2 93 = 186 = 1 (mod 185), 2 = 93 (mod 185).
Answer: The answer is 93
Question: What is the remainder when 2007 is divided by 81? Solution: Dividing using long division, we find that 2007 = 81 + 63, so the remainder is 63.
Answer: The answer 63
Figure 35: Chain-of-thought (CoT) prompt for MATH-Number Theory.
40

---

CoT Prompt for MATH-Geometry (Part 1)
Question: Two sectors of circle of radius 12 overlap shown, with P and R as the centers of the respective circles. Determine the area of the shaded region.
[asy]
label("P".(00)) aabe((""".110.323, (110,3923,6) N), label(" Ht" .22077846,0).E), label("S"" (10.3923, (),)); (bbel("60".(0.0))22."" [/asy]
(1ldraw((00.33333.6)---o hwwp://0.......222000ooeooroaba-
label("60"", .(20.7846,0))
Solution: By symmetry. the areas of the two parts of the shaded region are equal. Consider the right part of the shaded region and the left triangle.
(10.3923,)).bbe bln.kkkkiiee (1039))).(0.33233)))oc/ooo (r00:00000.....22200000000222-21 draw((0,0) (10.3923, b.ba...k 6).S); label(" Z". (10.3923.0). ()))): [/asy]
[asy]
(10)).0330)) (9..09999.....0099933.. (10.3923,6).NN) label(" (100.3923,
I); fabel("P".(0.. (".0P")))))W)))) [a.."""
The shaded area is equal to the area of sector PQS minus the area triangle PQS.
Since LPQS 60" and PQ = 12, the area sector PQS
12 24m.
Also, triangle PQS is equilateral with side length 12, so its area
v3 12 = 36v3.
Thus, the area of the right part the shaded region is 24 36/3, so the area of the entire shaded region is
2(24 - 36v3) = 48T 72
Answer: The answer is 18 72v3
The square with vertices (-a, -a), (a, -a). (-a, a), (a, a) is cut by the line y x/2 into congruent quadrilatcrals. The perimeter of one of these congruent quadrilaterals divided by equals what? Express your answer in simplified radical form.
Question:
Solution: The line y will intersect the vertical sides of the square, as shown below: pair A=(- draw axes(Arrows(4)); dot("( (aa" .A.N: dot(" .B..,))).;....... doi(" (a, -a)
[asy] real f(real x) { return x/2; ] import graph; size(6em); real
a.a). B=(aa)) C=(a,-a), D=(-a.-a); draw(A-B-C-D-cyck))
-.)".c.S)) .C.S); dot("( "a"" DDS); draw(shift((10000) "1",0)"""a" """""a"""""""""""""""""""" (-ateps.-a/2-55 (a+2" Arrps/ss
real eps=0.2: dot((8.4)h-8,,-4)) (raw(shift(00.10))* a". (at2"tttppss--"a of the square is x a, so we have y = _ = I. which means that the intersection point with the right side of square is (a, 9) Similarly, the equation of the left side of the square is -a, so we have y= = 3/2 1, which means that the intersection point with the left side of the square (-a, 9) follows that the sides of each quadrilateral have lengths 2a, and Va² (2a)² av5, by the Pythagorean theorem. Hence, the perimeter of the quadrilateral
a/2.pp.. The equation of the right side
+ 2a
3a
+ av5 VV55
2
2
and when this is divided by a, we get 4 + v5. Answer: The answer is v5
Figure 36: Chain-of-thouse (CoT) prompt for MATH-Geometry (Part 1).
41

---

CoT Prompt for MATH- CeETH-yyy (Part 2)
Question: A right circular cylinder with radius 2 is inscribed a hemisphere with radius so that its bases are parallel to the base of the hemisphere. What is the height of this cylinder? Solution: We draw and label diagram as follows:
[asy] size(110); pair O (0,0); pair A (.3,994); pair (.3..075); draw(O-A-B- cycle.heavycyan); label("OO, (""ii""""""""""""""" O)W label(" A..N; label("B",,Bl 3" .B,S); import solids; import three: defaultpennlinewiittto
currentprojection orthographiic550..1 revolution cylinder((O,0,0), .. 991); draw(c.black); (ra0(Arr((00)),.0011118)) [/asy]
Let the center of the hemisphere be O, and let A be a point on the circumference of the top circle of the cylinder. Since the cylinder is inscribed in the hemisphere, A lies on the hemisphere as well, so OA = 5. We drop perpendicular from A to the base of the hemisphere and let it intersect the base of the hemisphere at B. Since the cylinder is right and AB is height of the cylinder, LOBA is right angle, and B lies on the circumference of the bottom circle the cylinder. Thus, OB a radius of the cylinder, so OB We have that ΛOB A is right, so by the Pythagorean theorem, we have
AB = VO - OB2 = V52 2 V21.
Thus, the height of the cylinder V21. Answer: The answer is 221
Question: Parallelogram ABCD with A)2,5), B(4,9), C(6,5), and D 1) is reflected across the z-axis to A' D'C'' and then A'B'C'D reflected across the line A" D This is done such that D' is the image of D, and D" is the image of D'. What the ordered pair of D" the coordinate plane?
+ to
Solution: Reflecting point across the r-axis multiplies its y-coordinate by -1. Therefore, D' D' (4, 1). To reflect across the line y = x + 1, we first translate both the line and the point down one unit that the equation of the translated line is
and the coordinates of the translated point are (4, 2)). To reflect across y x, we switch the z-coordinate and y-coordinate obtain (-2, 4). Translating this point unit up, find that D" (-2,5).
Answer: The answer is (-2,5)
Figure 37: Chain-of of-thought (CoT) prompt for MATH-Geomete (Part 2).
42

---

CoTI' Prompt for MATH-Precalculus (Part 1)
Question: The function
f(z) (-1+ + (-2v3- 18i)
represents rotation around complex number Find Solution: Since rotation around fixes c, the complex number must satisfy f(c) In other words,
(-1 i (-2v3 18i)
Then 2c = (-233 - 181),
(3 iv-iv3)e -2v3 18i.
Then
== 2233 182
i/3
(-2v3 13
(3 (3 -
-6v3 6i - 54i + 18V3
12
12v3 60i
12
v3- 5i.
Answer:
The answer is v3 52
Question: Convert the point (8, in polar coordinates rectangular coordinates.
Solution: In rectangular coordinates, (8, becomes
7
7T
8 coss
8 sin
= (-4v3,
6
6
Answer: The answer is (-4v3, -4)
Figure 38: Chain-of-thought (CoT) prompt for MATH-Precalculuss (Part 1).
43

---

CoT Prompt for MATH--Precalculus (Part 2)
8
-2
Question:
Compute
1 1
3
1
Solution:
We have that
2
0
8 2
(2) + (0)(1)
(2(-2))(00)))
16 -4
5
3
1
+
20(-2)+(-310)
37 -13
16
-4
Answer: The answer is
37
- 13
Question: A line is parameterized by parameter i, so that the vector the line at f 2 is
3
and the vector on the line at / =
Find the vector on the line at l = -7.
Solution: Let the line be
x
fd.
Then from the given information,
2d,
3
3d. -4
We can treat this system as a linear set of equations in and d. Accordingly, we can solve to
get a
-3 and
{-8
2
Hence,
20
(-3
2
+
11
20
-8)
Taking t =77 we get
3
2
--17)
y/
=
20
-8)
=
76
(-17)
Answer: The answer is
76
Figure 39: Chain-of-thought (CoT) prompt for MATH-Prrealcullss (Part 2).
44

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In the abstract and Section |
(Introduction)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 7 (Discussion).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
45

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section 4 (Experiments) and Appendix B (Implementation Details).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
46

---

Answer: [Yes]
Our code is available at StrategyLMM/
Justification:
-ia-aaaoo------oo----eeo-oooooo- bai
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (ht.ps::/nii cc/
public/
http://udieeesseee publicc/uides//Ceee / ssionpoiccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https Ceee//ooeSuuui Submi is cessonoollcso for more details.
/ httpss.ccpubblccc.c /
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section (Experiments).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: In Sections (Experiments), (Analysis), and Appendix A (Additional Analysis), we comprehensively evaluate our framework across diverse array of tasks, LLMs, example groups, and scenarios. This extensive evaluation consistently demonstrates the advantages of our framework. Given the breadth and consistency of these results, we have opted not to include formal significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
47

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Sections (Experiments), (Analysis), and Appendix (Additional Analysis).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://heurrpps.ccoo htt.eerwrpp......oooooocccccccaa
Answer: [Yes]
Justification: This paper adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper aims to enhance the capabilities of large language models in common task-solving scenarios and does introduce privacy, security, or faimess issues.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
48

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In Section (Experiments).
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
license, copyright information, and terms of use package should be provided. For popular datasets, baapeettspppersssee paperswws thoode. com datasstt has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
assets are released,
49

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
50


Reviews:
[{"review_id": "q78xXDLwD6", "reviewer": "Reviewer_CCvY", "summary": "This paper proposes StrategyLLM, a pipeline for improving the few-shot reasoning performance. The main intuition is that when solutions to few-shot exemplars are inconsistent in terms of the reasoning process, the performance can be suboptimal compared to those with consistent solutions. Based on this intuition, StrategyLLM includes 3 prompting-based components: strategy generator, executor, and optimizer. The strategy generator generates task-level reasoning guidelines, the executor generates solutions to few-shot exemplars based on the generated strategies, and the optimizer improves the strategy when the accuracy on examples is lower than a threshold. Experiments on several reasoning benchmarks demonstrate that StrategyLLM outperforms other prompting methods, including CoT and plan-and-solve.", "questions": "1. Please clarify the few-shot setting for StrategyLLM. Specifically, does the few-shot prompt constructed by StrategyLLM contain wrong solutions when the generated strategies do not pass all examples?\n\n2. Have the authors tried self-consistency with a single strategy? How much performance gain is from aggregating multiple strategies compared to sampling from a single strategy?\n\n3. Have the authors tried constructing a single prompt for all MATH problems, and how does it compare to baselines?", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The empirical results are good compared to the baselines prompting methods, and the evaluation covers multiple LLMs.\n\n2 There is a thorough breakdown analysis on different number of samples, different number of optimization steps, gap to the oracle selection accuracy, etc.", "weaknesses": "1. The few-shot setting for StrategyLLM is unclear. Specifically, the paper says that a strategy is applied when it passes an accuracy threshold, which is <1. Does it mean that when using the few-shot prompt constructed by StrategyLLM, some examples have wrong solutions? If this is the case, it is surprising that StrategyLLM still outperforms the baselines.\n\n2. Have the authors tried self-consistency with a single strategy? How much performance gain is from aggregating multiple strategies compared to sampling from a single strategy?\n\n3. On MATH, it is unnatural to construct separate prompts per category. Have the authors tried constructing a single prompt for all MATH problems, and how does it compare to baselines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We address each question as follows:\n\n> The paper says that a strategy is applied when it passes an accuracy threshold, which is <1. Does it mean that when using the few-shot prompt constructed by StrategyLLM, some examples have wrong solutions?\n\nYes, we would like to clarify the following points: (1) In most scenarios, we set the accuracy threshold at 0.75, and the number of examples used is fewer than 8, there can be at most 1 example with an incorrect solution in these scenarios. (2) These incorrect examples may not largely degrade the performance. As pointed out by existing research (e.g., [1]),  the relevance of the solution to the question and the presentation of intermediate reasoning steps can be more critical than the correctness of the final answer.\n\n[1] Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters. ACL 2023. https://aclanthology.org/2023.acl-long.153/\n\n> Have the authors tried self-consistency with a single strategy? How much performance gain is from aggregating multiple strategies compared to sampling from a single strategy?\n\nWe have explored self-consistency using a single strategy. The performance of Meta-Llama-3-8B-Instruct on the CP, StrategyQA, and MA datasets is summarized in the table below. \"StrategyLLM-Single-SC\" refers to self-consistency applied with the best-performing strategy identified, which generates a set of solutions via\ntemperature sampling to obtain multiple answers. The results indicate that StrategyLLM-Single-SC underperforms compared to StrategyLLM-SC and even StrategyLLM, highlighting the advantages of aggregating multiple strategies.\n\n|                    | CP   | StrategyQA | MA   | Avg   |\n|---------------------------|------|------------|------|-------|\n| StrategyLLM               | 24.5 | 74.0       | 64.7 | 54.4  |\n| StrategyLLM-Single-SC | 23.5 | 69.8       | 59.3 | 50.9  |\n| StrategyLLM-SC | 25.0 | 74.0       | 66.0 | **55.0**  |\n\n> Have the authors tried constructing a single prompt for all MATH problems, and how does it compare to baselines?\n\nThank you for your suggestion. We would like to clarify that constructing a distinct prompt for each subject may be more appropriate as the strategy can be more specific to the subject and incorporate more domain-specific knowledge of it. \n\nAdditionally, to address your question, we conduct experiments using a single CoT prompt and a strategy-based prompt, both utilizing the same 7 examples, with each example drawn from a different subject in the MATH benchmark. Both the StrategyLLM and CoT prompts were evaluated on the test sets across all MATH subjects. The results for Meta-Llama-3-8B-Instruct are presented in the table below. The findings indicate that StrategyLLM consistently outperforms CoT across all subjects, demonstrating the advantages of our framework.\n\n|                    | AL   | PA | IA | CP | NT | GE | PC | Avg |\n|---------------------------|------|------------|------|-------|------|-------|------|------|\n| CoT | 36.5 | 44.5 | 4.5 | 23.0 | 18.0 | 16.0 | 13.5 | 22.3 |\n| StrategyLLM               | 39.0 | 48.5 | 8.5 | 24.0 | 22.5 | 18.0 | 17.5 | **25.4** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aUdKeWpbyU", "reviewer": "Reviewer_WbXZ", "summary": "This paper introduces a novel prompting strategy for large language models called StrategyLLM, which it employs to solve a range of tasks in math, commonsense reasoning, word sorting, and last letter concatenation. The approach involves (1) prompting a language model to generate task-specific instance-general strategies (sequences of subtasks) from a few examples of a given task, (2) prompting a language model to execute these strategies in order to grade them, (3) prompting a language model to refine strategies, (4) evaluating the strategies on a validation set to select inference time strategies, and (5) employing one or more of the selected strategies to solve new instances of the tasks of interest. The key attribute of this approach is that the strategies generated are not instance-specific, and indeed the paper finds improved generalization compared with prompting with instance-specific solutions as in the CoT baseline. The paper evaluates the StrategyLLM approach against a selection of standard prompting baselines, finding improvements across all tasks considered and across all models considered, and provides additional analyses.", "questions": "Clarity: Line 85 describes the role of the evaluator, constructing few-shot prompts for a strategy \"using itself and its execution result\". Please clarify the interface of the evaluator (line 122), and specifically what the form of the few shot examples p_{st} are. Providing the prompt used by the evaluator would be a welcome change, as it is the main absent component in the current revision.\n\nSee also the clarify questions about the SolutionLLM baseline in the Weaknesses section of the review.\n\nLet's have some discussion on how two rounds of validation are used before applying the strategy to the task instances of interest. Specifically, the execution phase grades a strategy on a small set of task examples $\\mathcal{E}$ (the same examples used to generate the strategy), and then the evaluator E grades the best performing strategies on a larger validation set $\\mathcal{V}$. Both rounds of validation are being used for filtering the set of strategies. In the paper, these are presented as separate phases of the StrategyLLM method with different type signatures and different descriptions. Clarity-wise, I fear this might be obscure the fact they represent two rounds of validation and filtering.\n\nThe final prompts used during inference include both the strategy generated as well as consistent few-shot prompts (consistent because they all employ the same strategy). Do you have evaluation of relative value of the presence of the strategy vs the presence of the consistent few-shot prompts.\n\nI would encourage you to write out what StrategyLLM is in contrast with StrategyLLM-SC and StrategyLLM-ZS; my assumption is that StrategyLLM employs only one selected strategy, but I don't see this stated explicitly.\n\nTerminology: Why do you describe your approach as involving multiple LLM agents? To me it seems like it is simply an algorithm that calls a single LLM with multiple prompts orchestrated by an (ordinary) algorithm. I understand that there are separate components for G, X, O, E, and inference, and my question is specifically around why you describe each of G, X, O, and E as a separate \"agent\".", "limitations": "The paper includes an adequate Limitation and Impact section, though it does not address societal impacts in particular. Per the checklist response, this is because the paper aims \"to enhance the capabilities of large language models in common task-solving scenarios and does not introduce privacy, security, or fairness issues.\"", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main high-level thrust of the paper, that LLM can generate strategies specific to a task, and then carry out these strategies is an important direction. This paper shows that LLMs can indeed use strategy generation to achieve greater generalization than an LLM performing in-context learning from example solutions. The experiments are adequately wide-ranging and the results are compelling; strategy generation as employed in StrategyLLM yields improvements over the baselines compared against. These baselines are well selected, and while they don't represent the full suite of prompting strategies that people employ with LLMs today, they make for a fair comparison that allows for measuring the effect of the core idea of StrategyLLM, having the LLM generate task-specific strategies.\n\nPrompting methods to elicit improved reasoning from large language models is an active area, ripe with a large number of unexplored ideas. This paper does a good job of placing its approach in the context of some of the recent work in this area. This is not the first paper to employ prompting strategies that share this high-level thrust, and as the paper cites a closely related learning-to-plan approach [15] has previously been explored.\n\nOverall the paper is clearly written, though in the Weaknesses section I remark on how the SolutionLLM baseline should be explained more clearly.", "weaknesses": "The approach in StrategyLLM is modestly complicated. For example, a simpler approach that still tests the core idea of the paper would be to keep the strategy generator and inference steps, but skip the intermediate grading and optimization of the strategies. Would the method work without these extra prompting phases, or are they essential to the method? An ablation would be valuable. I would expect these phases are essential, since they provide a data-driven validation of strategies before they are employed. This raises another natural question which is: is it the data-driven validation of the prompt (in this case, a strategy) that is the main advantage of StrategyLLM, or is it specifically that the strategies are not instance-specific? An experiment that compares with a prompt selection approach analogous to StrategyLLM's (i.e. generating multiple prompts, grading them on a validation set, refining them as needed, using a validation set to select between them, and finally employing SC or ZC to get a final answer), but  generating e.g. few-shot prompts instead of strategies would yield a fairer comparison.\n\nIn my view, the SolutionLLM baseline is not explained clearly. Two points in particular could use clarification. First, state more clearly what is meant by a \"solution\"; this is understandable by looking at the prompt in the appendix, but could use clarification in the text. Second, state explicitly how the SolutionLLM prompt is used; the prompt in the appendix merely outputs the solution to a specific example (I presume this is a dev/validation example), but it is not stated (if I am not mistaken) how this results in answers for novel tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We address each question as follows:\n\n> A simpler approach that still tests the core idea of the paper would be to keep the strategy generator and inference steps, but skip the intermediate grading and optimization of the strategies. Would the method work without these extra prompting phases, or are they essential to the method?\n\nWe would like to emphasize that these phases are indeed essential to the method. Specifically, StrategyLLM relies on the strategy executor to apply the generated strategies to the provided examples. The execution results are necessary for formulating the strategy-based prompts. Additionally, the strategy evaluator plays a vital role in determining the most effective strategy for inference. Without these phases, we would be unable to generate the few-shot prompts necessary for each strategy or to make informed decisions regarding which strategy to employ.\n\n> Is it the data-driven validation of the prompt (in this case, a strategy) that is the main advantage of StrategyLLM, or is it specifically that the strategies are not instance-specific? An experiment that compares with a prompt selection approach analogous to StrategyLLM's (i.e. generating multiple prompts, grading them on a validation set, refining them as needed, using a validation set to select between them, and finally employing SC or ZC to get a final answer), but generating e.g. few-shot prompts instead of strategies would yield a fairer comparison.\n\nWe appreciate your valuable suggestion. In response, we implement a baseline approach, which involves initially generating $n$ prompts, retaining $k$ of these for validation, and subsequently selecting the prompt with the highest validation accuracy for inference. The results of Meta-Llama-3-8B-Instruct on the CP, StrategyQA, and MA datasets are presented in the table below. We can observe that StrategyLLM significantly outperforms this baseline across all datasets, highlighting the benefits of integrating generalizable strategies.\n\n|     | CP   | StrategyQA | MA   | Avg   |\n|------|------|-----|-----|------|\n| Baseline    | 19.5 | 71.0 | 53.3 | 47.9 |\n| StrategyLLM | 24.5 | 74.0 | 64.7 | **54.4** |\n\n> Regarding the SolutionLLM baseline\n\nA solution is a step-by-step reasoning path. In this baseline, we utilize LLMs to generate a solution for each example in the prompt, which is the same as the examples used in the CoT prompt. These examples, along with their respective solutions, are then combined to formulate a few-shot prompt for inference, analogous to the CoT approach.\n\n> Regarding the evaluator and constructing the few-shot prompt\n\nUnlike other agents, the strategy evaluator does not utilize a specific prompt. Instead, to assess the effectiveness of a strategy, the evaluator constructs a few-shot strategy-based prompt tailored to that particular strategy. An illustrative example of the strategy-based prompt is depicted in Figure 3 of our paper. This prompt is structured as follows: **Strategy + (Question_1, Solution_1, Answer_1) + … + (Question_n, Solution_n, Answer_n)**. The solutions, Solution_1 through Solution_n, are derived during the strategy execution phase by applying the strategy to the $n$ examples in the prompt. The strategy evaluator then infers using each prompt on the validation set and calculates its validation accuracy.\n\n\n>  In the paper, the execution and validation are presented as separate phases of the StrategyLLM method with different type signatures and different descriptions. Clarity-wise, I fear this might be obscure the fact they represent two rounds of validation and filtering.\n\nThank you for your observation. We acknowledge that these phases represent two rounds of validation. However, they differ in their approach to obtaining solutions. During the execution phase, we directly prompt the LLM to generate a solution for a specific example by following the strategy. Next, in the validation phase, the strategy and the examples with their solutions are utilized to construct the strategy-based prompt. LLMs use this strategy-based prompt to infer solutions for each validation example. We will clarify these distinctions in the revised version of our paper to enhance understanding.\n\n> Do you have evaluation of relative value of the presence of the strategy vs the presence of the consistent few-shot prompts.\n\nWe conduct experiments to assess inference performance without the presence of the strategy, utilizing only the consistent few-shot prompt. The results for Meta-Llama-3-8B-Instruct are summarized in the table below. The findings indicate that omitting the strategy leads to a decrease in performance. This is because the consistent prompt merely illustrates how to apply the strategy to specific examples and may not encapsulate all the nuances of the strategy itself.\n\n|    | CP | StrategyQA | MA | Avg |\n|-----|------|------|------|-------|\n| StrategyLLM wo/ strategy | 24.5 | 66.0  | 63.3 | 51.3 |\n| StrategyLLM   | 24.5 | 74.0  | 64.7 | **54.4**  |\n\n> I would encourage you to write out what StrategyLLM is in contrast with StrategyLLM-SC and StrategyLLM-ZS; my assumption is that StrategyLLM employs only one selected strategy, but I don't see this stated explicitly.\n\nYes. StrategyLLM employs the most effective strategy identified during our experiments. We will state this clearly in the revised version of our paper. \n\n> Why do you describe your approach as involving multiple LLM agents? \n\nWe follow the terminology used in existing popular agent frameworks such as BabyAGI  (https://github.com/yoheinakajima/babyagi). Multiple agents collaborate to accomplish a task, with each agent assigned a subtask of the task. While the underlying LLMs for these agents can differ, our experiments utilize the same LLM across all agents to ensure a fair comparison.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BFANMeV6uk", "reviewer": "Reviewer_s895", "summary": "This work uses an LLM to describe the strategy that it should use to solve classes of problems, and then applies that strategy to the different test cases as required.  In addition, there is a caching and evaluation process whereby these strategies are improved during in an in-context learning prompt training/refinement cycle.  This approach is tested across a number of datasets and underlying (commercial and open-source) LLMs.", "questions": "* L221-233: Clarification: Is the Strategy Generator the same LLM as being 'tested' - or is the strategy being generated originally by GPT-3.5?", "limitations": "As described in the text, this approach is very dependent on whether the underlying LLM has a 'high level' understanding of how to approach the problem domain.  This is a dependency that somewhat undercuts the broad claims of the work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Well written paper, with a clear idea and thorough experimentation.\n\nGood to see both commercial and open-source models being tested side-by-side.\n\nThe strategy artifacts produced by the model are very interesting in their own right.  They are produced by the 'portable' prompts in the Appendix, however by the time they have been validated/cached/refined, they will likely be model-specific.", "weaknesses": "The actual prompt (Figure 7, in Appendix C) on which *the whole approach relies* should be included in the body of the paper.\n\nText Tweak:\n\n* L560: \"the optimal strategies for various datasets\" - seems unlikely.  Perhaps \"the best strategies discovered ...\" ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We address each question as follows:\n\n> Regarding the actual prompt and text tweak\n\nWe appreciate your suggestions. We will incorporate these revisions in the new version of our paper.\n\n> Is the Strategy Generator the same LLM as being 'tested' - or is the strategy being generated originally by GPT-3.5?\n\nWe would like to clarify that the underlying LLM for the strategy generator, executor, optimizer, and evaluator is the same as the LLM being tested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5qrVZxsvdS", "reviewer": "Reviewer_D2vF", "summary": "The paper proposes a prompt engineering method to address generalizability and consistency issues in existing prompting approaches for LLM-based problem solving. It utilizes four LLM-based agents including strategy generator, executor, optimizer, and evaluator. The proposed method works by generalizing knowledge from instances through induction and applying generalized knowledge to solve a given problem through deduction. \nAuthors evaluate the performance of their method empirically on multiple data sets and against a few alternative methods including self consistency with chain of thoughts. Results show superior performance for the proposed method compared with a few alternative approaches.", "questions": "- Where does the feedback in strategy optimizer come from? Is it automatically generated by LLM? If yes, can you elaborate more on how it's generated? If not, can authors clarify where feedback is coming from?\n- Authors report biggest gain from their method for symbolic reasoning tasks, but Addendum does not provide any examples of such tasks and the corresponding generated strategies.\n- How does strategy LLM generate multiple sets of strategy? Is it by asking it to re-run the same prompt multiple times or does it involve changing the original prompt?", "limitations": "Authors have listed lack of domain specific knowledge in underlying LLMs due to lack of domain specific training data as a limitation of their work. Beyond this limitation, strategy LLM might be susceptible to bias by generating discriminating strategies that may perform well on the train set which is not addressed in the paper.", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "Authors have evaluated their method's performance across multiple data sets and against various alternative methods. In addition, they conduct experiments to assess the universality of their proposed method, the effectiveness of task level strategy for reasoning. They also analyze the cost of strategy-based prompt engineering.", "weaknesses": "The main gap in the paper is the lack of detailed explanation to bridge between the superior performance of the proposed method and other studied methods including CoT through concrete examples and clear intuitions. The intuition explained in the paper relies on generalization and consistency as the main traits of the proposed method. It would have been helpful to use a concrete example showing how alternative methods such as CoT fail to provide the right solution for a given problem while the  method succeeds at doing so. It could be argued that the raw information is already available in the Appendix section, however, the reviewer is asking for interpretation of the way CoT is solving a problem vs. the proposed method by highlighting the generalization and consistency gaps in CoT's approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We address each question as follows:\n\n> The intuition explained in the paper relies on generalization and consistency as the main traits of the proposed method. It would have been helpful to use a concrete example showing how alternative methods such as CoT fail to provide the right solution for a given problem while the method succeeds at doing so.\n\nWe use an example from the test set of the CP dataset in the MATH benchmark for illustration. \n\n**Question**: Suppose that I have 6 different books, 2 of which are math books.  In how many ways can I stack my 6 books on a shelf if I want a math book on both ends of the stack?\n\n**Gold Answer**: 48\n\nThe CoT prompt for the CP dataset is illustrated in Figure 33 on Page 36 of our paper, while the strategy generated by GPT-3.5 is detailed in Figure 18 on Page 22. The predictions of CoT and StrategyLLM are as follows:\n\n**CoT Prediction**:\n```\nWe can treat the two math books as a single entity. So we have 5 entities to arrange on the shelf, which can be done in $5!$ ways. However, within the math book entity, the two math books can be arranged in $2!$ ways. Therefore, the total number of ways to arrange the books is $5! \\cdot 2! = \\boxed{240}$.\nAnswer: The answer is 240.\n```\n\n**StrategyLLM Prediction**:\n```\n1. Understand the problem statement:\n  - We have 6 different books, 2 of which are math books.\n  - We need to stack the books on a shelf in a way that a math book is on both ends of the stack.\n\n2. Identify the counting or probability concept:\n  - We need to use the concept of permutations to solve this problem.\n\n3. Break down the problem:\n  - We have 6 books, 2 of which are math books.\n  - We need to arrange these books on a shelf in a way that a math book is on both ends of the stack.\n  - The math books can be arranged in 2 ways (one on the left end and one on the right end).\n  - The remaining 4 books can be arranged in 4! = 4 x 3 x 2 x 1 = 24 ways.\n\n4. Calculate the required counting or probability:\n  - To calculate the total number of ways to stack the books, we need to multiply the number of ways to arrange the math books with the number of ways to arrange the remaining books.\n  - Total number of ways = 2 (ways to arrange math books) x 24 (ways to arrange remaining books) = 48.\n\n5. Simplify or reduce the problem:\n  - There are no further simplifications or reductions possible for this problem.\n\n6. Communicate the solution clearly:\n  - There are 48 ways to stack the 6 books on a shelf such that a math book is on both ends of the stack.\n\nAnswer: The answer is 48.\n```\n\nThe CoT prediction fails to account for the requirement of placing a math book at both ends, resulting in an incorrect solution. In contrast, the generalizable strategy employed by StrategyLLM allows for a comprehensive understanding of the problem, effective utilization of relevant concepts, and a breakdown of the task. This approach ensures that critical details are not overlooked, leading to the correct answer. Additionally, consistently applying a strategy across few-shot examples illustrates how that strategy can be utilized in various contexts and ensures that the same strategy will be applied to the test instance.\n\nBeyond accuracy, the solutions derived from StrategyLLM are more detailed, interpretable, and accessible, making them particularly suitable for contexts that demand clarity and comprehensibility, such as educational settings.\n\n> Where does the feedback in strategy optimizer come from? Is it automatically generated by LLM? If yes, can you elaborate more on how it's generated? If not, can authors clarify where feedback is coming from?\n\nThe feedback in the strategy optimizer is indeed automatically generated by LLMs. Following the execution of a strategy, we can identify which examples yield correct solutions and which do not. We then prompt the LLMs to analyze the incorrect solutions, providing insights into potential reasons for their failures, such as calculation errors, misunderstandings of the problem, or the application of incorrect formulas. The LLM also offers suggestions for enhancing the strategy, which may include adding subtasks, decomposing complex subtasks into simpler components, or revising ineffective subtasks. Subsequently, we utilize the LLM to refine the strategy based on this feedback.\n\n> Authors report the biggest gain from their method for symbolic reasoning tasks, but Addendum does not provide any examples of such tasks and the corresponding generated strategies.\n\nWe would like to clarify that we have included a strategy and an execution example for the LLC task in Figure 27 on Page 30 of our paper. This example illustrates the application of our method on symbolic reasoning tasks.\n\n> How does strategy LLM generate multiple sets of strategies? Is it by asking it to re-run the same prompt multiple times or does it involve changing the original prompt?\n\nStrategyLLM generates multiple strategies by re-running the same prompt multiple times, utilizing temperature sampling to introduce variability in the outputs and allowing for the exploration of diverse strategies.\n\n> StrategyLLM might be susceptible to bias by generating discriminating strategies that may perform well on the training set, which is not addressed in the paper.\n\nWe emphasize that StrategyLLM is specifically designed to address potential biases present in few-shot prompts and to enhance the generalizability of the prompts through the use of task-level strategies. Our comprehensive results demonstrate that the strategy-based prompts exhibit greater generalizability compared to traditional CoT prompts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After a fruitful discussion, the reviews generally agree that the paper is well written and that the core contribution is clear and insightful. They have also provided a positive evaluation of the experiments, which consider a variety of sensible models and baselines and yield compelling results.  This is a solid paper overall, and the majority of issues brought up by the reviewers have been solved during the rebuttal stage.
Decision: Accept (poster)