Automatic Model Selection with Large Language Models for Reasoning
Authors: James Xu Zhao
Yuxi Xie
Kenji Kawaguchi
Junxian He
Michael Qizhe Xie
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
In-Context Learning
Reasoning
TL;DR: We propose a model selection method to combine two distinct reasoning models, CoT and PAL, and achieve the new SOTA results on GSM8K and SVAMP.

Abstract: Chain-of-Thought (CoT) and Program-Aided Language Models (PAL) represent two distinct reasoning methods, each with its own strengths. CoT employs natural language, offering flexibility and interpretability, while PAL utilizes programming language, yielding more structured and rigorous logic. We introduce a model selection method to combine the best of both worlds by employing a large language model (LLM) to dynamically select between them. Our theoretical analysis underscores the feasibility of this method, which is further corroborated by empirical results. Our proposed method demonstrates significant performance improvements across eight reasoning datasets with Codex, ChatGPT, and GPT-4. Additionally, our method is complementary to self-consistency; when integrated, it can further enhance performance while significantly reducing computation costs. Moreover, we achieve new state-of-the-art results on GSM8K and SVAMP, with respective accuracies of 96.8% and 93.7%.

Automatic Model Selection with Large Language Models for Reasoning
James Xu Zhao', Yuxi Xie Kenji Kawaguchi', .uuxiian He?, Michael Qizhe Xie The Hong Kong University of Science and Technology xu.zhao @ u.u.s.ddu
National University of Singapore
Abstract
2022) have emerged two effective methods that offer different strengths and weaknesses. Essen- tially, CoT dccomposes a reasoning problem into a series of intermediate steps using natural language, making more general, flexible, and comprehen- sible. On the other hand, PAL offers solutions via Python functions, with its step-by-step program- ming code ensuring methodology. The external Python interpreter fur- ther guarantees the computation precision. Intu- itively, combining the strengths both models and selecting between them based on their solutions to a specific problem would lead to enhanced per- formance. However, without access to the ground truth, choosing a better method itself becomes a machine learning problem.
(CoT) and Program- Lideed--iiedd Language Models (PAL.) represent two distinct reasoning methods, each with its own strengths. CoT employs natural language, offering flex- ibility and interpretability, while PAL utilizes programming language, yielding more struc- tured and rigorous logic. We introduce model selection method combine the best of both worlds by employing a large language model (LLM) to dynamically select between them. Our theoretical analysis underscores the fea- sibility of this method, which is further cor- roborated by empirical results. Our proposed method demonstrates significant performance improvements across eight reasoning datasets with Codex. ChatGPT, and GPT-4. Addition- ally, our method is complementary to self- consistency; when integrated, it can further enhance performance while significantly reduc- ing computation costs. Moreover, we achieve new state-of-the-art results on GSM8K and SVAMP, with respective accuracies of 96.8% and 93.7%..
rigorous and structured
In order select among multiple solutions, pre- vious studies have suggested training ranker (Ue- sato et al., 2022). While training dedicated model generally results in improved accuracy, it can also be somewhat cumbersome and entail sig- nificant costs. Conversely, large language models (LLMs) have demonstrated good calibration and have been used to assess the accuracy of their own outputs (Guo et al., 2017; Shinn et al., 2023; Xie et al., 2023). In light of this, propose leveraging the in-context learning capabilities of LLMs for model selection. We direct LLMs to choose be- tween two distinct reasoning models and elucidate their selection rationale.
1 Introduction
Large language models (LLMs) have made impres- sive progresses in numerous fields (Devlin al., 2019; Brown et al., 2020; OpenAI, 2023; Chowd- hery et al., 2022; Bubeck et al., 2023; Wei et al., 2022a) and are often powerful enough to solve problems through single unified method. While convenient, this approach tends to ignore the dis- tinct structures and variations among the problems, which would henefit from using different methods. On the other hand, in human society, individuals with different skill sets excel in various roles, lead- ing to a thriving world as a whole.
We present a detailed theoretical analysis to val- idate our approach. The analysis highlights two primary factors impacting the effectiveness of our method: (1) the significance of the difference be- tween the two models, i.e. the distinct distributions of their respective probabilities of correctly solving problems, and (2) the probability of selecting the correct model. Particularly, a higher overall perfor- mance can be attained when there is a substantial difference between the models being considered, and when there is a high probability of the correct
In the case of reasoning, Chain-of- Thooughf (CoT) (Wei et al., 2022b) and Program-Aider Lan- guage Models (PAL) (Gao et al., 2022; Chen et al.,
OOu code, data and prompts are available at https:// hwtta//cooon/aaaa//aaellee-----

---

Two different ways of solving the problem
(1) Each box of contacts $100 and currently 10% off.
Se the discount on each box is 100 + 0.10 $10.
Therefore, the price of each box after the discount is 100 - $90. Pete buys 2 boxes, so the total cost 2 90 $180.
box single contacts,
buys bowes, gets single-
Problem
eye,
Selection Q
there
90 contacts,
(A) can correctly solve the problet.
The total cost the contacts $180, and there 90 pairs. So the cost each pair of contacts is 1800 $2. So the answer $2.
and
Beccuse (8) calculates the cost per single-use contact, not per pair of contacts.
currently 10% IF buys boxes contact how mach will each pair contacts cost?
solution():
contacts_pet Becpcct___rr__r__e
days_per, _____e___oo__ _ox_
cost 100
discount 0.1
2
discounted box,
box_cost cost box cost (1 discount) total_cest discounted box boxes, bough_ total_conter contacts contacts_pee box boxes bought
total
contacts
result
Figure We propose to perform model selection to combine two distinct methods, CoT and PAL figure illustrates example where PAL makes mistakes about crucial information and therefore fails to answer the question correctly. In contrast, CoT manages to correctly answer the same question. Our selection model successfully chooses the correct solution and provides a brief explanation to support its choice.
model being selected. Furthermore, our analysis affirms that even without exceptional model se- lector, we can still achieve improvement in certain cases. This reinforces our decision to simply em- ploy an LLM for model selection.
plained. Furthermore, natural language supports a broad range of reasoning that may involve common sense and confidence in the reasoning steps. But it might struggle with intricate logical relationships. In contrast, PAL takes structured and accurate ap- proach, breaking down reasoning problems using Python code. However, the deterministic program format of PAL constrains its generalizability and flexibility.
We evaluate our method across eight reason- ing tasks, with CoT and PAL serving as the baseline methods. Our method consistently at- tains performance improvement, when employing Codex (Chen et al., 2021), ChatGPT, and GPT- as backbone LLMs. Its broad applicability is further underscored by performance gains with open-source Llama models (Touvron et al., 2023). the context multiple sample aggregations, our approach significantly reduces the computa- tion costs while achieving notable performance en- hancements. Moreover, our approach attains new state-of-the-as accuracies of 96.8% and 93. 7% GSM8K (Cobbe al., 2021) and SVAMP (Patel al., 2021), respectively.
2.1 Case Study the Differences between CoT and PAL
PAL
Number of Talure Dars
2 Automatic Model Selection with Large Language Models
hrubice
Decompsuttoo
Hodbllly
Undesitering
Figure Comparative analysis of error cases between COT and PAL GSM8K.
In this study, examine reasoning tasks using two baseline models: CoT and PAL. To tackle complex reasoning tasks, CoT leverages an LLM gen- erate several intermediate reasoning steps before arriving at a final answer. Due to its reliance on natural language, the reasoning steps are clearly ex-
To delve deeper into the distinct strengths of CoT and PAI., we analyze 100 instances from GSM8K where CoT and PAL yield different results: 50 cases where CoT is correct and PAL is wrong, and

---

another 50 where the opposite is true. All the CoT and PAL solutions are generated using ChatGPT. Then we divide the error reasons into 5 categories. Examples of each category can be found in Ap- pendix E.
answer APAL. When Acor and Apal yield different results, we move to the model selee- tion phase. The LLM is prompted again, pro- vided with (Q, CCOT, ACoT, OPAL), and tasked to select the better method, alongside generat- ing a brief explanation. Specifically for Llama 2 selection prompts, the LLM is prompted with (Q, CC ACoT. CPAL, APAL) pairs. This selec- tion result, together with the explanation, is rep- resented as S. Ultimately, based on S, the final answer is derived as A E {Acor, Ar...) AAA.).
Computation Precision: The ability to per- form accurate numerical computations.
Logical Decomposition: Breaking down com- plex problems into manageable parts and han- dling logical relationships well.
During the selection phase, LLMs leverage their in-contcxt learning abilities for model selection, i.e., we present the LLM with few in-context ex- emplar pairs (Q, CCor, Acor, CPAL S). An ex- ample of the proposed method is illustrated in Fig- ure excluding the few-shot examples save space. Specifically, we provide the LLM with two distinct reasoning solutions a multiple-choice question format. We expect the LLM to identify errors or gaps within the chains and determine the correct method. Note that there are very few in- stances where the LLM fails to make a choice; in these cases, we randomly select method. The effect and qualitative analysis of the generated ex- planation can be found in Appendix C.
Problem-Solviigg Flexibility; The adaptability in addressing problems using different reason- ing paradigms, such as forward and backward reasoning.
Semantic Understanding: Grasping and inter- preting the problem accurately without over- looking crucial information.
Others: This encompasses miscellaneous er- ror reasons such as redundant calculations and ambiguous questions.
Figure 2 depicts the error distribution across these categories. It reveals notable differences be- tween CoT and PAL. PAL, with its external Python interpreter, ensures computation accuracy. It also excels in logical decomposition, effectively break- ing down problems and employing variables to map logical relationships. On the other hand, CoT is more versatile and flexible, allowing for both for- ward and backward logical reasoning, while PAL is less adept at backward reasoning scenarios. CoT's strength also lies in its superior grasp of natural language, aiding in semantic understanding. This analysis highlights the distinct proficiencies of both models in reasoning, reinforeing our hypothesis.
Owing to the in-context learning capabilities of LLMs, we find that they exhibit reasonable accu- racy in selecting the appropriate method. Further- more, as our subsequent theoretical analysis in Sec- tion 2.3 reveals, highly accurate method selection is not actually necessary for the algorithm to per- form well. Our empirical findings corroborate this, showing that even if there are situations where the model selection is poor, the overall improvement remains substantial.
2.3 Theoretical Analysis
In this section, we conduct a thcoretical analysis to determine under which condition the proposed method could work (and fail).
2.2 Method
Given the distinct advantages of CoT and PAL, it would be beneficial to combine the strengths of both. We propose a systematic approach that combines these models through model selection. Specifically, it contains two stages: solution gener- ation and model selection.
Quantifying error rates Let us denote the error rates of the two base methods, mj and m2, by erri and err2, respectively. Without loss of general- ity, let m, be better base method in the overall performance: i.c., erT1 S erT2. For given ques- tion x, we define Pa. as the probability of choos- ing more accurate method, either mi or m2, for the given x using the proposed approach. Define R(x) = p(correct x, m2) - p(correct x, m.) where p(correct) x, m;) represents the probabil-
For given reasoning question Q, an LLM is prompted generate reasoning chains for CoT, symbolized as Ccor along with its answer Acor. Simultaneously, the I.M is prompted derive reasoning chains for PAL, denoted as CPAL- Em- ploying a Python interpreter, we then get its

---

ity of outputting a correct prediction given input with method m4- Then we can quantify the final error rate err of the proposed method as follows: Proposition 1. For any methods m and m2 with any combining probability function Pas
3 Experiments
3.1 Setup
Datasets and backbones We conduct experi- ments on eight datasets that span range of arithmetic and symbolic reasoning tasks. 7 of these datasets, including GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), ASDIV (Miao et al., 2020), SingleOP, SingleEEQ, AddSub and MultiArith (Konicl-Kedzzorski et al., 2016), are about arithmetic reasoning, while Date Under- standing (Srivastava et al., 2022) focuses symbolic reasoning. To comprehensively eval- uate the effectiveness of our approach, we em- ploy three proprietary LLMs as backbone sys- tems: Codex (code-davinci-000), ChatGPT 3opt-3.5-turbb-03011 and GPT-4 (gpt-4-0314) We also evaluate our method with open-source Llama 2 models on GSM8K.2 Unless stated oth- erwise, we always utilize the same LLM for both base models as well as for the model selector. Prompt design To effectively exploit the in- context learning abilities of the LLMss, we create a set of few-shot examples by manually creating an error in one model's reasoning chain. Few-shot examples in our prompts are based on those in pre- vious works. For each backhone LLM, we use fixed prompt for all the arithmetic benchmarks without specific customizations. Examples of the prompts can be found in Appendix D.
err err - Eg [[[()] (px - 1(((x) < 0})
We refer readers Appendix B for the full proof. Proposition decomposes the possible improve- ment (or deterioration) over base methods in terms of R(z) and Px- quantitatively shows when and how we can expect improvement (or deterioration) based on these two factors. For example, to im- prove over the best base method mi, Proposition I suggests us to choose another base method m2 such that [[(r)] is not too small and Pr is high when [[()] is large. In other words, discourages us from choosing too similar methods as base meth- ods, because for similar methods, [[xx tends to be small and it is challenging to increase Pa even when [R(x) is large due to the similarity. This provides a theoretical motivation for us to use CoT and PAL. instead of combining CoT with another COT.
On the accuracy of selection Define p be the overall probability of selecting a better method: X p= F Theorem shows that can be much worse than that of a random guess to achieve the improvement over the base methods m, and m2; i... err < err and e < err2 can happen with p < 0.5:
Hyperparameters For the results derived using greedy decoding, we use temperature of during both solution generation and model selection. For experiments with self-comsistency. we follow the settings in prior works by setting the temperature at 0.5 for CoT and 0.8 for PAL, during solution generation. During model selection, the tempera- ture is set to 0 for a deterministic result. We refrain from further hyperparameter tuning emphasize the simplicity and efficacy of our method.
Theorem 1. For any > 0, there exist data dis- tribution over two base methods (m1, m2), and combining probability (px) such that err < erri, ert1 S erT2, and p <.
We provide stronger version of Theorem 1 in Appendix A (that implies Theorem 1) and its proof in Appendix B.
Thcorem 1 supports our proposition that, de- spite not training new model for the selection process and with the in-context learning limited to a few-shot prompt, it is possible to achieve im- provement, even if we do not achieve > 0.5 in some instances. Further elaborated in Theorem 2, our analysis indicates that even with a weak selec- tor, improvements attainable, especially when the base models exhibit comparable overall per- formance. This theoretical analysis offers deeper insights into the conditions and strategies for effec- tive performance improvement with the proposed methodology.
3.2 Main Results
The results of our experiments with greedy de- coding are shown in Table 1. First, we find that our proposed method effectively and robustly en- hances performance in most settings across various datasets of different difficulties and with different backbone LLMs, simply by combining two base models. For example, with GPT-4, we achieve
Codex results are obtained in February and March, Chat- GPT in April and May, GPT-4 in May and June 9-13, llama 2 in September 2023.

---

Arithmetic SingleOP
Symbolic
Backbone
Method
GSM8K
SVAMP
ASDIV
SingleEQ
AddSub
MultiArith
Date
CoT
64.4
77.6
80.2
92.7
93.5
88.4
95.7
64.5
Codex
PAL
71.5
79.6
79.1
95.4
96.5
91.9
99.7
77.5
Ours
74.7
82.2
81.6
96.3
96.9
91.6
99.7
79.4
CoT
80.8
83.0
89.3
94.8
97.4
90.4
98.7
69.1
ChatGPT
PAI.
79.2
80.3
83.0
90.7
97.6
89.4
96.3
68.3
Ours
82.6
84.3
89.4
94.8
97.8
90.6
98.7
70.2
CoT
94.6
91.9
92.7
97.2
97.2
93.9
98.0
90.0
GPT-4
PAL
94.0
92.2
90.2
95.2
98.8
94.9
98.5
88.1
Ours
95.6
93.7
93.5
97.3
98.6
95.7
99.0
90.5
Table 1: Results comparison (Accuracy %) 7 arithmetic datasets and symbolic dataset with greedy decoding. We evaluate our methods on Codex, ChatGPT, and GPT-4. The best results are highlighted in bold.
Backbone
Method
SC@5
SCB15
Method
Acc.
Cost
Tokens
# Generated
CoT
85,4
87.4
CoT@ 15
87.4
33.69
21,56M
2.72M
ChatGPT
PAL
80.9
82.4
PAL@ 15
82.4
41.64
27.13M
1.90M
Ours
88.2 (+2.8)
89.2 (+1.8)
Ours@5
88.2
29.29
18.99M
1.61M
CoT
95.6
95.8
CoT@ 40
88.3
89.84
57.32M
7.08M
GPT4
PAL
94.7
95.5
PAL@40
83.5
111.04
72.35M
5,07M
Ours
96.5 (+0.9)
96.8 (+1.0)
Ours@15
89.2
87.87
56,97M
4.83M
Table Results comparison (Accuracy %) on GSM8K with the integration of the Self- Self-Consistency (C@) SC@5 and SC@15 represents and 15 sampled paths respec- tively. The previous state-of-the-are 95.5%, achieved by Zheng et al. (2023a).
C6T@80
88.2
179.68
115.04M
14.48M
PAL@
83.3
222.08
144.64M
10.16M
Ours@20
89.5
117.16
75.96M
6.44M
GSM8K is
Table 3: Computation cost comparison with ChatGPT on GSM8K (1319 data samples). CoT 15 represents CoT with self-consistency (SC) with 15 sampled paths. Cost USD calculated based on OpenAl Pricing'. # Tokens denotes the total number of tokens, including inputs and generation tokens. # Generated denotes the number of generated tokens.
accuracy of 95.6% GSM8K and 93.7% SVAMP without self-consistency.
Second, our results show a considerable improve- ment even when one of the base models performs much worse than the other. For instance, we ob- serve a significant 3.2% improvement over PAL's 71.5% accuracy on GSM8K, even though CoT has lower accuracy of 64.4%.
lection with multiple samples, and whether they complement each other. The results of CoT and PAL with self-consistency are obtained by sam- pling multiple times with the temperature 0.5 and 0.8 respectively. For our method with self- consistency, we execute our algorithm repeatedly, and aggregate results across all samples through majority voting, arriving at the final answer. As demonstrated in Table 2, we achieve substantial improvements over the self-consistency results of both CoT and PAL.
Third, our model's general applicability is fur- ther underscored by its 1.9% improvement the symbolic date understanding task when utilizing Codex the backbone. In fact, even if the ac- curacy difference between two base models as large as 13% on this task, our proposed method still improves the accuracy from 77.5% to 79.4%. Additionally, our method also achieves respectable performance gains with both ChatGPT and GPT-4 this task. It serves evidence of our method's applicability to a spectrum of reasoning tasks, ex- tending beyond mathematical reasoning.
Employing ChatGPT, we attain a pronounced performance improvement with only 5 samples, reflecting 2.8% leap in accuracy. With GPT-4, even though both base models already score around 95%, integrating them with our method leads 96.5% accuracy on GSM8K. Furthermore, we es-
Experiments with self-consistency We aim to investigate the relationship between self- consistency (Wang et al., 2022b) and model se-
tablish the new state-of-the the--he-n art results on GSM8K at 96.8% using 15 samples.
Phaps://opemaa..oo/mriinnng Oct. 2023

---

Baseline LLM
Selector LLM
Acccor
AcePaL
Ours
Improvement
Llama 2 7B
Llama 7B
15.0
13.7
16.2
+1.2
Llama 2 7B
Llama 2 13B
15.0
13.7
16.8
+1.8
Llama 13B
Llama 13B
29.0
33.3
35.3
+2.0
Table Results with Ilama 7B/13B on a subset of GSM8K. The reference CoT accuracies (Touvron et al., 2023) with Llama 2 7B and 13B are 14.6% and 28.7% respectively. Baseline LLM refers to the backbone LLM used for both CoT and PAI solution generation. Selector LLM refers to the backhone L.M used for model selection.
3.3 Computation Cost Comparison
baseline results the 13B model, method achieves respectable 2% performance gains, im- proving accuracy from 33.3% to 35.3%. On the smaller 7B model, we observe the performance im- provement from 15.0% to 16.2% when utilizing the 7B model as the selector. Moreover, when employ- ing the 13B model as the selector, the performance is further improved to 16.8%. These results affirm the broad upplicability of our method.
We extend our analysis to compare the computa- tion costs of model selection with multiple samples versus using self-consistency alone. As depicted in Table 3, our method not only showeases notable performance improvement but also significantly re- duces computation costs, underscoring its efficacy and efficiency.
Our method consistently outperforms both CoT and PAL with SC in terms of accuracy while de- manding less token computation and cost. For in- stance, using only 5 paths, method achieves comparable performance to CoT @80 while only costs 29 USD, which is merely 16% % of expense associated with CoT@ 80. Moreover, when aggre- gating 15 samples, our method surpasses the per- formance of CoT@ 80 at only half the cost.
4 Analysis
In this section, we provide a few analyses to see when and how our method works.
4.1 Influencing Factors
To better understand the reasons for the perfor- mance improvement across various datasets and backbone LLMs, we present the performance im- provement, and the associated influencing factors in Table 5.
Furthermore, our method also elevates the per- formance ceilings. While the result of CoT with SC plateaus around 88.2% and that of PAL with SC saturates around 83.3%, our method breaks through these ceilings, achieving an impressive 89.5% accu- racy using just samples. This result underscores the power and potential of integrating different rea- soning models.
As demonstrated in our theoretical analysis, the performance improvement is jointly determined by the difference between the two combined meth- ods, |R(x)), and the effectiveness of the model selection, p. The results from Table 5 showcase a high expected performance difference between CoT and PAL, and decent selection accuracy with all backbone LLMs, which clarifies the significant performance gains across the datasets.
3.4 Experiments with Llama 2 Models
To explore the broad applicability of our method, we conduct experiments with open-source Llama 2 7B/13B models using a subset of the GSM8K, comprising 600 data samples. We generate CoT and PAL solutions with Llama 2 7B/13B models4. For model selection, models.. The experiments are carried out by using greedy decoding without tuning hyperparameters. As shown in Table 4, our method consistently yields performance enhancements when applied to Llama 2 models. Specifically, compared to the
Firstly, the Aupper Bound of CoT and PAL flects how differently the two base models behave across questions. A larger AUpper Bound signifies larger room for potential improvement. Specif- ically, on GSM8K with ChatGPT, AUpper Bound stands at 8.6%, even though the accuracies of CoT and PAL are closely matched at 80.8% and 79.2%, respectively. Similarly, with GPT-4, A Upper Bound is 2.5% while the accuracies of the two basc models close (94.6% vs 94.0%).
Llama 7B/13B Chat
"htts://hugginngface wttp:iinnwwac.ccomtttttaeamm Laaatallaamaaaaaaaaaaa
Secondly, across the eight datasets with varied complexities, each of the three L.LMs exhibits com- mendable selection accuracy. Notably, both Codex and GPT-4 show a relatively high success selection
"htpss//huggingfice.coo/// www.eca- haamma/aaaua-----eee
https:/hugginggaceccom commeta- -lamaal lama-2- 3hb-chat-
hf

---

Arithmetic SingleOP
Symbolic
Backbone
Metric
GSM8K
SVAMP
ASDIV
SingleEQ
AddSub
MultiArith
Date
Uoped
10.0
8.1
6.5
1.6
0.9
2.0
0.3
4.6
Codex
Success Rate
74.8
72.5
63.7
87.9
88.0
70.0
92.9
87.8
Improvement
+3.2
+2.6
+1.4
+0,9
+0.4
-0.3
0
+1.9
UpperMound
8.6
6.0
3.4
2.4
1.4
0.6
9.8
ChatGPT
Success Rate
60.4
66.4
69.8
58.0
57.1
60.9
75.0
53.6
Improvement
+1.8
+1.3
+0.1
0
+0.4
+0.2
0
+1.1
À Upper Success Rate
2.5
3.6
2.2
2.4
0.6
1.3
0.5
1.1
GPT-4
72.6
68.7
64.2
57.0
69.2
85.7
100
86.7
Improvement +1.0
+1.8
+0.8
+0.1
-0.2
+0.8
+0.5
+0.5
Table 5: We define the following terms: AUpperBound ApCUpper Bound Accm1 where AcCUpper Bound is the upper bound accuracy where we assume a perfect model selection and my is the stronger one of two base models. A pperBound reflects the expected performance difference between the two base models. Success Rate calculates the correct selection rates when either CoT is correct or PAI. correct, i.e., we ignore the cases where both methods are either correct or wrong. Improvement performance improvement achieved over the performance of the stronger base model.
Backbone
ChatGPT CoT CoTV
GPT-4
the prompt. Both these methods' accuracies are similar or higher than the accuracy of PAL.
m1
CoT
PAL
PAL
CCoT
From the results in Table 6, we can find that model selection between CoT and CoTY, or CoT and ComplexCoT, does not lead to substantial per- formance gains, even though the accuracy of Co'V' and ComplexCoT is par with that of PAL. On the other hand, model selection between CoT and PAL results in consistent performance improvements. To understand the reasons behind these outcomes, we further investigate the Aupper Bound and the success selection rate.
AcCmy
80.8
80.8
94.6
94.6
Accma
79.2
79.2
94.0
95.1
Ours
82.6
80.8
95.6
95.2
Improvement
(+1.8)
(+0)
(+1.0)
(+0.1)
AUpee Bound
7.5
2.5
1.7
Success Rate
60.4
52.2
72.6
58.8
Table 6: Other model combinations results GSM8K. CoT' denotes the base CoT model with the temperature of 0.1. CCoT denotes ComplexCoT (Fu al., 2022).
rate. For example, their success rates GSM8K surpass 70%, and on the simpler MultiArith dataset, the selection accuracies exceed 90%.
Firstly, Aupper Bound of CoT-PAL exceeds that of other combinations, CoT-CoT' and ComplexCoT-CoT,, despite their employing two stronger or equivalent two base models. This obser- vation suggests a larger absolute value of the accu- racy difference per question for CoT-PAL. It indi- cates that CoT and PAL perform dissimilarly than other model combinations, which represents a larger [[ As Proposition highlights, without substantial [R()). it is unlikely to achieve sig- nificant performance gain since the improvement component is factored by [R(x)]
Furthermore, as detailed in Theorem 1, perfor- mance improvements are attainable even without a high success selection rate. Our empirical results corroborate it. For instance, on the date understand- ing task employing ChatGPT, despite modest success rate of 53.6%, still achieve 1.1% % per- formance enhancement with large Appper Bound of 9.8%, and small overall performance difference (69.1% % and 68.3%).
Secondly, the success selection rate of CoT-PAL surpasses that of other model combinations. It means that the selector is more likely to select the correct choice when one solution derives from CoT and the other from PAL. In theory, this higher suc- cess rate implies that when [R(z)] is high for a given question x, the success selection probability Pz for CoT-PAL is higher than others.
4.2 Combination between Similar Methods
We choose CoT and PAL as our two base mod- els due to the motivation of combining different strengths of distinct models. We conduct exper- iments examine whether the performance im- proves when combine two similar base models, We use two variants of CoT: CoT' where we set the temperature at 0.1, ComplexCoT (Fu et al., 2022) where we use more complex fhw-shot examples in
These findings support our initial motivation and hypothesis. We choose CoT and PAL as our two

---

LL
PAL LLM
Selector LLM
AcccoT
ACcPAL
Ours
Success Rate
UUpeerBound
ChatGPT
ChatGPT
ChatGPT
80.8
79.2
82.6 (+16)
60.4
8.6
ChatGPT
Codex
ChatGPT
80.8
71.5
81.2 (+0.4)
70.6
7.0
ChatGPT
Codex
GPT-4
80.8
71.5
84.4 (43.6)
84.8
7.0
Table Results with combination of different backbone L.LMs on GSM8K. CoT LLM refers to the backbone LLM used for CoT solution generation.
Codex
ChatGPT
GPT-4
Metric
CoT-PAL
PAL-CoT
COT-PAL
PAI-CoT
CoT-PAL
PAL-CoT
AccCoT
64.4
64,4
80.8
80.8
94.6
94.6
ACCPAL
71.5
71.5
79.2
79.2
94.0
94.0
Ours
69.9(-1.6)
74.7(+3.2)
82.6(+1.8)
81.6(+0.8)
95.6(+1.0)
95.1(+0.5)
Success Rate
56.7
75
60.4
54.6
72.6
63
Ratioc.r 71.9
17.3
89.9
79.7
60.3
53.4
RatiopAL 28.1
82.7
10.1
20.3
39.7
46.6
Table The effect of option order for different backbones on GSM8K. CoT-PAL represents CoT and PAI. being placed choice A and B respectively. Ratiocor indicates the ratio of CoT selected out of all the selections. As stated previously, during analysis, we ignore the cases where both CoT and PAL are either correct or wrong.
base models because they represent distinct rea- soning approaches using natural language and pro- gramming language. We expect these models to exhibit significant difference errors and accura- cics, indicated by a high AUpper Bound- Morcover, we posit that for particular question, the consid- erable difference in errors between two solutions makes it easier for LLMs to select the correct op- tion, leading to higher success rate compared selecting between similar base models like COT-CoT'. This holds true even when different prompts or temperature settings are used.
4.4 The Effect of Option Order
To understand the influence of option order in a multiple-choice question format, we perform ex- periments by modifying the option order. The sults in Table show performance impact from simply changing the choice order. We attribute this to the inherent bias within the LLMs. A similar phenomenon reported by Zheng et al. (2023b). Each of the three backbones seems to exhibit preference for the first option. Particularly notice- able on Codex, in which when CoT is positioned first, it is selected in 71.9% of selections. How- ever, Ratiocor significantly drops to 17.3% when putting CoT second. ChatGPT shows preference for natural language reasoning as it always selects CoT solutions more frequently regardless of its po- sition in the options. Moreover, when CoT is the second choice, RatioçoT decreases from 89.9% 79.7%, which we speculate demonstrates its prefer- ence for the first option. GPT-4 has similar biases akin to ChatGPT, though appears fairer. The op- tion order also affects the success selection rate and further influences the overall performance.
4.3 Combination between Different Backbone LLMs
Considering that Codex is optimized for code gen- eration and ChatGPT performs better in natural language reasoning. we embark on experiments in- tegrating various backbone LLMs, to harness the strengths of distinct LLMs and reasoning models.
As shown Table 7, we achieve 0.4% perfor- mance improvement with CharGPT as the selector, despite a notable 9.3% accuracy disparity between PAL from Codex and CoT from ChatGPT. More impressively, when utilizing GPT-4 the model selector, we get significant 3.6% performance boost, with an 84.8% success selection rate. These results hint the potential benefits when integrating multiple models tailored for specialized domains or tasks.
While Zhao et al. (2021) attempts to address the inherent bias through probability calibration, the task of mitigating the bias in the absence of un- derlying token probabilities remains challenge. Therefore, despite the wide of LLMs in data notation and task evaluation(Xu al., 2023; Wang et al., 2023; Chiang al., 2023), addressing and climinating inherent biases is a critical area that deserves further exploration.

---

5 Related Work
hibit reliable calibration, there a growing number of research emphasizing the use of self-evaluation for verification. Shinn et al. (2023) proposes an approach to provide an agent with dynamic mem- ory and self-reflection capabilities. Madaan et al. (2023) proposes a method to generate outputs from LLMs and refine its previously generated output given its own feedback. Different from these works where the underlying method is the same, in this work, we are interested in combining systems with different strengths and weaknesses through self- evaluation model selection.
Ensemble learning. In machine learning, the strat- egy of combining various models to address sin- gle problem exemplified in techniques such as bagging (Breiman, 1996), boosting (Freund and Schapire, 1997; Chen and Guestrin, 2016; Ke et al.. 2017), and random forest (Ho, 1995; Breiman, 2001). The underlying idea in these methods is that group of weak learners can collectively man- ifest a strong learner. This concept has also found its place deep learning through the use of ensembles. For reasoning, self-consistency sam- ples diverse reasoning paths and chooses the most consistent answer through majority voting (Wang et al., 2022b). Wang et al. (2022a) takes it step further by introducing rationale- augmented ensem- bles, emphasizing rationale sampling in the out- put space. However, typically, ensembling places equal weights on models through majority voting, which may restrict the full potential of the diverse strengths that each model offers.
6 Conclusion
We introduce a method that effectively combines two distinct models, CoT and PAL. by using LLMs for the model selection. Our theoretical analysis supports the feasibility of such model combination, which is validated by empirical results. Our method achieves performance improvements across eight datasets with various backbone LLMs. We reduce the computation cost while attaining notable per- formance improvement with self-consistency. We set new SOTA results on GSM8K and SVAMP. This research represents a significant step towards tapping into the potential of diversity and collabo- ration among models in LLMs.
Reasoning. The research community has made tremendous progress in the field of reasoning. Apart from CoT (Wei al., 2022b) and PAL (Gao al., 2022; Chen et al., 2022), Zhou al. (2022) simplifies complex problems by breaking them down into a series of sub- brobberoblems. Kojima et al. (2022) shows that by simply adding "Let's think step by step" before each answer, LLMs can be competent zero-shot reasoners. Creswell et al. (2022) alternates between selection and inference stages, generating causal reasoning steps to the final answer. Kazemi et al. (2022) proposes backward chaining algorithm that breaks reason- ing down into four sub-models. Paul et al. (2023); Xie et al. (2023); Yao et al. (2023) refine interme- diate reasoning steps by leveraging another critic model or the self-evaluation ability. Recently, Zhou et al. (2023) utilizes GPT-4 Code Interpreter and code-based-self-eee beffeeddeef---errfcaattoo achieving 97.0% ac- curacy on GSMSK, but their work is concurrent with ours. More importantly, the contributions of these methods are distinct from our approach, and the progress made by them could potentially be seamlessly integrated using our method.
In our future work, we aim to expand this frame- work to more reasoning tasks and possibly other domains. An intriguing concept involves explor- ing the use of diverse system instructions to elicit varying model behaviors for model combinations.
Furthermore, a genuine combination of specialized models, each with its strengths and expertise, offers promising avenue for exploration.
Limitation
This work focuses on reasoning tasks, but we be- lieve that extending the exploration of model se- lection into other domains could yield valuable insights. Further, due to the inherent bias of LLMs. our method is affected by the order of options. We encourage the community to explore solutions for bias mitigation. We also anticipate subsequent stud- ies to identify more robust and potent model selec- tors. Finally, our combination strategy currently comprises only the CoT and PAL base models. Fu- ture research could investigate the incorporation of more base models with distinct strengths, with the aim of creating a more powerful synthesis that leverages their unique advantages.
Self-Evaluation. LLM calibration studies reveal that the probabilistic predictions made by current LLMs closely align with the actual frequencies of token occurrences, hence producing well-calibrated predictions for certain tasks (Guo et al., 2017; Ka- davath et al., 2022; Jiang et al., 2020). As LLMs ex-

---

References
Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Leo Breiman. 1996. Bagging predictors. Machine learning, 24:123-140
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- kumar Prabhakaran, Emily Reif, Nan Du Benton C. Hutchinson, Reiner Pope, James Bradhury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Samkaranarayana Pillai, Marie Pellat, Aitor Lewkowycz. Erica Moreira, Re- won Child, Oleksandr Polozov, Katherine Lee, Zong- wei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Helle Hellse-ellstemm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. ArXiv,
Leo Breiman. 2001. Random forests. Machine learning. 45:5-32.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πly Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ArXiv, 0is/2005.14163
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundherg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv, abs/2303.12712.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. ArXiv, abs/21 10. 14168.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Ed- wards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri. Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Moham- mad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Cam,, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford. Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob MeGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluat- ing large language models trained on code. ArXiv, abs/2 107.2107..3374.
Antonia Creswell, Murray Shanahan. and Irina Higgins. 2022.
Exploiting large language models for interpretable logical reasoning. ArXiv, 059/2205.0971..
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. ArXiv, abs/ 0e/11800.4885.
Yoav Freund and Robert E Schapire. 1997. decision- theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1)11199139 19-133.
Yao Fu, Hao-Chun Peng. Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-baut based prompt- ing for multi-step reasoning. ArXiv, abs/2210.0072
Tianqi Chen and Carlos Guestrin. 2016. Xghoost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785. 794.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022. Pal: Program-aided language models. ArXiv, abs/2211. 10435..
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from rea- soning for numerical reasoning tasks. ArXiv, abs/2211.12588.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International Conference on Machine Learning.
Tin Kam Ho. 1995. Random decision forests. In Pro- ceedings of 3rd international conference docu- ment analysis and recognition, volume I, pages 278- 282, IEEE
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon

---

Zhengbao Jiang. J. Araki, Haibo Ding, and Graham Neubig. 2020. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Associa- lionfor Compulational Linguistics, 9:962-977.
Noah Shinn, Beck Labash, and Ashwin Gopinath, 2023, Reflexion: an autonomous agent with dynamic mem- ory and self-reflection. ArXiv, abs/2303. 1336..
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro. Aditya Gupta, Adrià Garriga-Alonso, Agmieszka Kluska, Aitor Lewkowyez. Akshat Agarwal, Alethea Power. Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Annasaheb Rahane, Anan- tharaman S. Tyer, Anders Andreassen, Andrea San- tilli, Andreas Stuhlmuller, Andrew M. Dai, An- drew D. La, Andrew Kyle Lampinen, Andy Zou. Angela Jiang. Angelica Chen, Anh Vuong, Ani- mesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokan- dov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakacs, Bridget R. Roberts. Bao Sheng Loe, Barret Zoph. Bartlomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ek- mekci, Bill Yuchen Lin, Blake Stephen Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argucta, C'esar Ferri Ram 'irez. Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burcch, Chris Waites, Christian Voigt, Christopher D. Manning. Christo- pher Potts, Cindy Tatiana Ramirez, Clara Rivera, Clemencia Siro, Colin Raffel, Courtney Asheraft, Cristina Garbacea, Damien Silco, Daniel H Garrette, Dan Hendrycks, Dan Kilman, Dan Roth. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Gonz' alez, Danny Hermandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, D. Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, De- nis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes. Diganta Misra, Dil- yar Buzan, Dimitri Coelho Mollo, Diyi Yang. Dong- Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman. Elizabeth Barnes, Elizabeth P. Donoway, Ellie Pavlick, Emanuele Rodolà, Emma FC Lam, Eric Chu, Eric Tang, Erkut Erdem. Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan J. Jerzak, Ethan Kim, Bunice Engefu Manyasi, Evgenii Zheltonozhskii, Fan Xia. Fate- meh Siar, Fernando Mart -nez-Plumed Francesca Happ'e. e. François Chollet, Frieda Rong. Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Ger- mán Kruszewski, Giambattista Parascandolo, Gior- gio Mariani, Gloria Wang, Gonzalo Jaimovitch- I L'opez, Gregor Betz, Guy Gur-Ari, Hana Galija- sevie, Han Sol Kim. Hannah Rashkin. Hanna Ha- jishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hubert Wong, Ian Aik-Soon Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, John Kernion, Jacob Hilton. Jachoon Lee, Jaime Fernández Fisac, J. Brooker Simon, James Koppel, James Zheng, James Zou, Jan Koco' n, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Narain Sohl-Dickster Jason Phang.
Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Hcnighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran- Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage. Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt. Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. ArXiv, abs/2207. 27.022....
Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xinyuan Xu, and Deepak Ramachandran. 2022. Lam- bada: Backward chaining for automated reasoning in natural language. ArXiv, abs/2212.1382
Guolin Ke, Qi Meng. Thomas Finley. Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. 2017. Lightgbm: A highly efficient gradient boost- ing decision tree. Advances in neural information processing systems, 30.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Twasawa. 2022. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.
Rik Koncel-Kedde Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In North Ameri- can Chapter of the Association for Computational Linguistics.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. ArXiv, abs/2303. 17651..
Shen- Yun Miao, Chao-Chun Liang, and Keh- Yih Su. 2020. A diverse corpus for evaluating and devel- oping english math word problem solvers. ArXiv, 0bs/2106.11772.
OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Arkil Patel, S. Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In North American Chapter of the Associ- ation for Computational Linguistics.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beat- riz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on in- termediate representations. ArXiv, abs/2304.01904.

---

Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jenni Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Oluwadara Alabi, Ji- acheng Xu, Jiaming Song. Jillian Tang, Jane W Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jorg Frohberg. Jos Rozen, José Hemández-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule. Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubb D. Dhole, Kevin Gimpel, Kevin Ochieng' Omondi, Kory Wallace Mathewson, Kristen Chia- fullo, Ksenia Shkaruta, Kumar Shridhar, Kyle Mc- Donell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang. Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis-Philippe Morency, Luea Moschella, Luca Lam. Lucy Noble, Ludwig Schmidt, Luheng He. Luis Oliveros Col' on, Luke Metz, Lutfi Kerem eSenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Madotto Andrea, Maheen Salcem Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, M Quintana, Marie Tolkiehn, Mario Giulianelli. Martha Lewis, Martin Potthast, Matthew Leavitt, Matthias Hagen, M'aty'a as Schubert, Medina Baitemirova, Melissa Arnaud, Melvin Andrew McElrath, Michael A. Yee. Michael Cohen, Mi Gu, Michael 1. Ivanitskiy, Michael Star- rits, Michael Strube, Michal Swkedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Monica Tiwari, Mo- hit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, T MukundVarma, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas S. Roberts, Nicholas Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha Iyer. Noah Con- stant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares. Parth Doshi, Pas- cale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Pency Liang, Pe- ter W. Chang, Peter Eckersley, Phu Mon Htut, Pi- Bei Hwang. P. Milkowski, Piyush S. Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, QING LYU, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph. Raefer Gabriel, Rahel Habacker, Ram lam'on// Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa. Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhut- dinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib J. Singh, Saif M. Mo- hammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman. Samuel Gruetter, Sam Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh. Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster. Sepideh Sadeghi, Shadi S. Hamdan, Sharon Zhou, Shashank Srivas- tava, Sherry Shi. Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Deb-
nath, Siamak Shakeri, Simon Thormeyer. Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo hwan Lee, Spencer Bradley Torene, Sriharsha Hat- war. Stanislas Dehaene, Stefan Divie, Stefano Ermon, Stella Rose Biderman, Stephanie C. Lin, S. Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen. Tal Schuster, Tao Li, Tao Yu. Tariq A. Ali, Tatsuo Hashimoto, Te-Lin Wu, Theo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, T. N. Kornev, Tim- othy Telleen-Lawcn Titus Tunduny, Tobias Ger- stenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler O'Brien Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Rau- nak, Vinay Venkatesh Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fe- dus. William Saunders, William Zhang, W Vossen. Xiang Ren, Xiaoyu Tong. Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadch, Yair Lakretz. Yang Song, Yasaman Bahri, Ye Ji Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yu Hou, Yun- tao Bai, Zachary Seid, Zhao Xinran, Zhuoye Zhao. Zi Fu Wang, Zijie J. Wang. Zirui Wang, Ziyi Wu. Sahib Singh, and Uri Shaham. 2022. Beyond the imi- tation game: Quantifying and extrapolating the capa- bilities of language models. ArXiv, 0tss/2206.04611
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi. Yasmine Babaei, Niko- lay Bashlykow, Soumya Batra. Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cris- tian Cantón Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes. Jeremy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hos- seini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura. Marie-Anne Lachaux. Thibaut Lavril. Jenya Lee, Diana Liskovich, Yinghai Lu, Yuming Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, lliyan Zarov, Yuchen Zhang, An- gela Fan. Melanie Kambadur, Sharan Narang, Aure- lien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307 ,09288.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Fran- cis Song, Noah Siegel, L. Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome- based feedback. ArXiv, abs/2211.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022a. Aationale-augmeent ensembles in language mod- els. ArXiv, abs/2207.00747.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Huai hsin Chi, and Denny Zhou. 2022b. Self-

---

consistency improves chain of thought reasoning in language models. ArXiv, abs/2203.11171.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Huai hsin Chi. 2022. Least-to-most prompting enables com- plex reasoning in large language models. ArXiv, abs/2205.10655.
Yidong Wang, Zhuohao Yu, Zhengran Zeng. Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xingxu Xie, Wei Ye, Shi-Bo Zhang, and Yue Zhang. 2023. Pandalm: An automatic evalu- ation benchmark for Ilm instruction tuning optimiza- tion. ArXiv, abs/2306.0500
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raf- fel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Met- zler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus. 2022a. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. ArXiv, 199/
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min- Yen Kan, Junxian Hc, and Qizhe Xie. 2023. De- composition enhances reasoning via self-evaluation guided decoding. ArXiv, abs/2305.00633.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley, 2023. Baize: An open-source chat model with parameter-efficient tuning self-chat data. ArXiv, abs/2304.01196.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths. Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601.
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In International Conference on Machine Learning.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023a. Progressive-iee hint prompting improves reasoning in large language models. ArXiv, abs/2304.09797..
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhwang, Zhanghao Wu. Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Haotong Zhang, Joseph Gonzalez, Ion Stoica. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv, 0aas/236.006855.
Aojun Zhou, Ke Wang. Zimu Lu. Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song. Mingjie Zhan, and Hongsheng Li. 2023. Solving challenging math word problems using gpt-4 code interpreter with code-based verificiii....... ArXiv, abs/2308.07921.

---

A A detailed version of Theorem 1
this appendix, we provide a detailed version of Theorem Whereas Theorem 1 only states the existence of problem instances, Theorem 2 constructs such instances concretely: i.e. Theorem implies Theorem 1. Define Mx[A to be the distribution for the expected errors: i.e., an expected error can be written by (X],.. (T * [a])]] for some function f. Define S[X] {ar X R(x)) < 03. Let us denote U[]] as the uniform distribution over X. Given any X, we write x1, T |S[2]] T/n.
Assume that 1 - T
<
Theorem 2. Let Ue [X] = U[X] and be given such that |X| < cc. Let e, E (0,1) and E (0,
such that B = n-T C (0, 1) and À 2 1- (n (n-T 6). Let R and Pa be set such that R(x) =
for E S R(x) = 3 for z E X S(X), (L/T) Pz = X, and (1/(n n- T)) E Pz =
(T/((n - (1) 2) + 8/(n T). Then, we have that err < err,, err S crry, and
5 n
p p=l-a a 122a - 11
In particular, when a 2 0.5, we have p- - as a + 1 and (6/(n-T)) - (with À 1- when a < 0.5, we have p - O as a + O and (6/n) + (with À - 1).
Ie (n-T-8))
The proof of Theorem 2 is presented in Appendix B. Theorem 2 shows that the overall success probability of the selection process can be much worse than random guess achieve the improvement over the base methods m and m2; i.c., err < err, and err I err2 can happen with p < 0.5. Indeed, it is possible to have p -> with the improvement (err < err1 and err S err2) when the size of X is large: when a 2 0.5, we can choose À = 1 Te (n T - 5) with which err < ert1, err1 I err2, and p - as a > and (6/
T)) - O. When a < 0.5, we can choose À = 1 with which err < erT1, err, S erry, and + 0 as + and (8/n) - 0. This supports our proposition that, despite not training a new model for the selection process and with the in-context learning limited to a few-shot prompt, it is possible to achieve improvement, even if we do not achieve Pr 0.5 in some instances.
Theorem also suggests that if the overall performance of two base methods is similar, captured by e, the overall selection process can be weak achieve some improvement, as long as the success selection probability is relatively high when the two methods have very different expected errors (or accuracies) for a given question. In essence, Thcorem suggests a tradc-off: we want [R(x) to be larger when deciding which two base methods m. and m2 to choose, implying that we prefer base methods to perform dissimilarly on X. On the other hand, if two base methods exhibit a substantial expected accuracy difference, then the selection process needs to be stronger to improve the performance (i.e., needs to be larger). However, if the expected accuracy difference between the two base methods is relatively small, increasing the power of the selection process is not that necessary to boost performance.
B Proofs
B.1 Proof of Proposition 1
and acc; - err; for (1,2). Since expected error ||][iinorreett prediction]] P(incorrect prediction) 1 Pccorrect prediction), we have that
Proof. Define acc
acc; = p(correct|m:) = Ex (ccorrect] m))]
where correct represents the event of the correct prediction. Similarly,
2 Σ [p(m: (mmpl/))ccoorcctccc m²)
acc = Er
where pmm,(ar) represents the probability of selecting method m. given & via the proposed method. Thus,
acc - acC1 = E, [p(m, (z)p(correct],, mi) + (o(m//))p(come) m2) - p(correct|æ, m))

---

En (p(mi (x) pccorrect| mi) ( m2)]
Es (p(mg) p=)orrerec))];; z, m2) (1 p(m1) p m1)]
Since 1 p(m1la)) (r) p(me(z),
acc - acc1 = Ex (( (2) m2) - p(masl)pp(correet/a m1)]
Ex [))(m]()))(((
Here, we notice that
p(malar)
Psr
if R(x) 2 0
(1 11-pe
if R(x) <0
[(R(z) 2 0}ps + I([(z) < (1 -
By plugging this into the above equation,
acc acc1 = E.r ((][)(x) 2 0}p- + (R) pz)
< + 0)
= (R(() 2
Since R(x)1(R(r) 2 0} = [ 2 0) and R(z)1{[((z) < 0) = --((()) < 00, we
have that
acc1
E=[[[()) 2 -,[ < <1 Px)]
([([)())(((((())) 0) + ({((2) 0))p= 1(R(x) < 0))
Since (D(R(x) 2 0} + ((((x) < 0}) = for any x.
acc acc1 E=(lR(z)) - (((()) 1 0))]-
B.2 Proof of Theorem 2
Proof. We first confirm that R(x) and Pa. define valid probabilities under the condition of this statement.
For R(x), since e E (0, 1) and 3 E (0, 1). it defines valid probabilities for methods m. and m2- For Pas since À E [0, 1]. it also defines valid probabilities for the case of & E S S[X] For the case of x E X' \ S[X]. since e(T [n - (1) A) + 8/n 2 0, we need to show that e(T/ (n T1) A)B 1 + 8/n S 1.
That is,
<(T((n TT)(1 1)3 1 + 5/(n 7) $
T(1 - -1)3- <n T
B(n T 8)
<>,
cT
which is satisfied by the condition on that A 2 1-
& (n T 8).
Thus, the condition on P. defines
2T
the valid probabilities for both cases of x S[X] and x E X S(X). We now show that err < err1. Invoking Proposition 1,
err erri - Ex [[(x)) (px 1([(x) < <})].
Thus, we have err < erri if E, [[(())) (pz - 1(R(x) < <0))] > 0. This condition can be rewritten as
Ez [[(x2) (pr 1(l(x) < <)])] >
[R(x)) (ez - I(R(x) < <0}]] > 0
ZEX

---

E |R(x)Pp+
Σ |R(x)|p>>
Σ
[R(x)]
ZES
xES[N]
Σ
R(z) + TTX
3
E
Pe >T
ET - -TX =
SCA
E SEA
P=
3
This is satisfied by the condition on p that (1/(n T)) E Excrisix) <(T/(n- T))(1 118-++ 8/(n T) for some 0: i.e.,
ET - X) P2
3 8.
xCA
Therefore, we have that err eir1.
We now show that err, - err2. Similarly to the proof of Proposition I, we define acc; 1 - err; for E {1,2). Then, the inequality errı S err2 holds if ace1 2 acc2. By using correct represent the event of the correct prediction, this condition can be rewritten as
a00 2 ac02
Σ p(corroct/ x m1)] 2 Σ
p(correct|/r m2)]
xCX
xEX
2
Σ
R(x) = (n-T) eT
=EX
T 23
This is satisfied by B cT Thus, we have that err S crr2. Using these, we now compute the p as
n-T
p E Exe-ga 12
[px]
= Σ
E Ps
Pr +
S
xES(X]
TT(1
X)
+6
+ +ax
eT(1 - A)
+ax +
3n
cT(1
1)(n - -T)
+ ax +
cTn.
= (1 A) -a(1 - 1) + ad +
5
= - a + AB2 - 1] +
Finally, we prove the asymptotic behavior using this equation. When a < 0.5, by setting À = 1, we have that
1 - a + 2/2a + = = 2a - 1 + 8 aa
- 0 and (8/n) + 0. When 2 0.5. by setting A - f. (n - T T--),,
8),
we have that
p=1 1= (1 1 B
8
(n T 8))
-6)) [2a
1] +
Tc

---

3
6
a + 2a -12x
11
(n
8)
Te
18
6
a [2a 1] Te
(n T 8)
By defining Q = (n - -6), we have
p= a 122 - 11Q +
Here,
8 Q
cT
(n- T - 6) n- Te
(n - 8)
(n
T
- 6)
Te
T
T
Thus,
6
== [2a 1]
1
T
71
2a 1
2a + + 0
+
T
20 1
a a+6
->
11 1 T
n1
a = T/n - and (6/(n- - T)) - 0: e.g. by setting = ((n - T) and take Ç - 0, with which
(6/(n - (()n)))) = - 0.

---

C
The Effect and Qualitative Analysis of Explanation
Success Rate
Backbone
AcccoT
ACcPAL
Explanation?
Ours Acc.
64.4
71.5
w/o exp
74.7 (+3.2)
74.9
Codex
64.4
71.5
w/ exp
74.6 (+3.1)
74.2
80.8
79.2
w/o exp
81.8 (+1.0)
$5.9
ChatGPT
80.8
79.2
w/ exp
82.6 (+1.8)
60.4
94.6
94.0
w/o exp
95.5 (+0.9)
69.9
GPT-4
94.6
94.0
w/ exp
95.6 (+1.0)
72.6
Table 9: Accuracy and success selection rate with and without explanation GSM8K.
To perform model selection, we provide explanations in the few-shot examples and ask the LLMs generate explanations after making choice, by pointing out why the other choice is incorrect. To investigate the potential role of explanations enhancing the model's selection capability, we conduct experiments on GSM8K by excluding explanations from the answer.
expect to improve the model's selection ability
The results in Table 9 reveal that for backbone LLMs with more powerful in-context learning abilitics, such as Codex and GPT-4, the inclusion of explanations within the answer seems to have negligible influence on the model's 's performance. In other words, Codex and GPT-4 have relatively high success selection rate even when explanations are not given. However, for ChatGPT, which possesses relatively low feo-shot in-context learning capability, the integration of explanations yields a larger improvement in the success selection rate. This observation suggests that for backbone LLMs with weaker in-context learning abilities, explanations can serve as a valuable tool improve the success selection rate.
Qualitative studies on the explanations We are also interested in gaining an intuitive understanding of the algorithm, so we conduct some qualitative analysis on the selection explanations. As displayed in Table 10, in the first example, both ChatGPT and GPT-4 can accurately identify that the PAL solution is correct. However, ChatGPT provides an incorrect explanation, while GPT-4 offers valid one. In the second example, which is relatively simpler, both ChatGPT and GPT-4 successfully perform accurate model selection and provide valid explanations. In the first example, we can see that GPT-4 actually possesses exceptional reasoning capabilities and provides reliable explanations.

---

Courtney atlended a concert and reported that audience was number. However, Kelly went the same concert said that Courtney made mistake of overstating the number people in attendance by 20%. Kelly was right, how many people really attended the concert?
Question:
Answer:
def solution():
Courtney reported people attendance. this number is overstated by 20%, then we need to find actual number of people.
courtney_repors
overstateeentt actual_attendance
0.20
First, we need to find 20% 48.
20%
(20/100) * 9.6.
rourtney_report /(1
overstatement__ercentgg. result (aattualllattennddcc))) return result
Now, we subtract this number from Courtney's reported number to find the actual number of people.
48 9.6= 38.4.
Since we cannot have a fraction a person, we round the number to the nearest whole number. So, the actual number of people in attendance was approxi- mately 38.
So the answer is 38.
ChatGPT: (B) can correctly answer the math prohlem. Because (A) rounds up the result to the nearest whole number, which is not necessary.
GPT-4: (B) can correctly answer math problem. Because (A) incorrectly calculated the actual number people by subtracting the overstatement percentage from the reported number instead dividing it.
Question: How much does it cost you for lunch today at Suhway if pay $40 for foot-long fish sub and thrice much for six-inch cold-cut combo sub?
(A)
(B)
def colution():
Answer:
cost_footlong_ fish = 40
If the foot-long fish sub costs $40. then the six-inch cold-cut comho sub costs 3 $400 = $120.
cost_sixinch.coldeur
fost_footlong_fie total_cost = cost_footlong_f fish cost_sixinch_coldeur result = total_cost return result
So the total cost for both subs is $40 $120 = $160. Therefore, it costs $160 for lunch today at Subway.
ChatGPT: (A) can correctly answer the math problem. Because (B) calculates the cost of the six-inch cold-cut combo sub as one-third of the cost of the foot-long fish sub instead of three times the cost of the foot-long fish sub.
GPT-4: (A) can correctly answer math problem. Because (B) incorrectly calculates the cost of the six-inch cold-cut combo sub as one-third of the foot-long fish sub instead thrice as much.
Table 10: In the first example, CharGPT performs model selection correctly, but gives wrong explanation, while GPT-4 gives a correct explanation. The second example shows where both ChatGPT and GPT-4 select correctly and give the correct explanation.

---

D Examples of Prompts
We show examples model selection prompts used on different tasks with different backbones. For arithmetic reasoning tasks, we employ an 8-shot example for both Codex and Llama 2, and a 5-shot example for ChatGPT and GPT-4. For the date understanding task, we use 6-shot examples for Codex, ChatGPT, and GPT-4. Few-shot examples in the prompts are based on those in previous works (Wei et al., 2022b; Gao et al., 2022; Fu et al., 2022). We only show few examples for each case. Full prompts can be found in our code.
Math Problem: Olivia has $23. She bought five bagels for $3 each, How much money does she have left?
Question: Which of the following two choices correctly answer the math problem?
(A)
(B)
def
solution ():
Answer:
money_ coney_initial = 23
Olivia had 23 dollars.
bagels 5
bagels for 3 dollars each will be 5 * 3 = 15 dollars. So she has 23 5 18 dollars left.
bagel_cost =
noney_spent = bagels bagel_cost
The answer is 18.
money_left = money_initial
money_spent
result money_left
return result
Answer: (A)
Table An example of 8-shot model selection prompts used on arithmetic datasets with Codex.
Date Understanding Problem: 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?
Question: Which of the following two choices can correctly answer the date understanding problem?
(A)
B
def solution ():
A:
# If 2015 is coming in 36 hours, then today is 36 hours before. today = datetime (2015, 1, 1)
If 2015 is coming in 36 hours, then it is coming in 2 days. days before 01/01/2015 is 12/30/2014, so today is 12/30/2014.
relativedelta # One week from today,
So one week from today will be 01/06/2015. So the answer is 01/06/2015.
one wtw_wee__froomoodeee = today +
relativedelta
# The answer formatted with %m/Xd /XY is result one_week_from_toda
strftime (' Km/%d/XY
return result
Answer: (A)
Table 12: example of 6-shot model selection prompts used on Date Understanding task with Codex.

---

System:
You helpful assistant that can identify the correct answer to math problem.
There two choices the math problem. One natural language the math problem. You need to identify which
the question, while the other
Python program answer Either of them correctly choice correctly the math problem. Here onc example how to do it,
Math problem: Olivia $23. She hought five bagels $3 cach. How much money does she have left?
(A)
def solution (): money_initial 23
Answer:
Olivia had 23 dollars.
bagels
bagels for 3 dollars each will 5 dollars. the answer
bagel_cost
So she 23 dollars left.
money_spent bagels bagel_cost money_left money_initial
money_spent
result money_left result
Which of the above two choices can correctly answer the math problem?
(A) can correctly answer the math problem. Because (B) adds the number of bagels to the cost of each bagel instead of multiplying them.
Now it's your turn. Here is another math problem and two choices.
Math Problem: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost more. How many golf halls did he have the end of wednesday?
(A)
(B)
def solution ():
Answer:
solf_balls_initii = 58
Michael started with 58 golf halls. Then after losing 23 tuesday. be had 58 -23 35. After losing 2 more, he had 35 2 37 golf balls.
ftf_ballls_lott_uueddaaa = 23
golf_balls_1osttwedneeee =
golf_balls_lett=
So the answer is 37.
solf_balls_initial
solf_bells_ostttueedda
\golf_balls_loss_eeddssss
result golf_balls_left return result
Which of the above two choices can correctly answer the math problem? (B) can correctly answer the math problem. Because (A) adds 2 more balls after losing 2 more on Wednesday instead of subtracting them.
Table 13: Two examples 5-shot model selection prompts used on arithmetic datasets with ChatGPT.

---

System: You helpful assistant that identify the correct to the math problem.
There are two choices to the same math problem. One uses natural language to answer the question, while the other uses Python code to answer it. Either of them can correctly answer the math problem. You need to identify which choice correctly answer the math problem. Here is one example how to do it,
Math problem: There were nine computers in the server room. Five computers were installed each day, from in the room?
monday thursday. How many computers
(A)
(B)
def solution ():
Answer:
There were originally computers. For each of 4 days from monday thursday, more computers So 5 * 4= 20 computers So there 9 20 29 computers So the answer 29.
computers computers_added =
computers_total.
added.
computers_initial
added.
computers_added result computers_total return result
Which of the above two choices can correctly answer the math problem?
(A) can correctly answer the math problem. Because (B) missed the fact that computers were added each day from monday to thursday.
Now it's your turm. Here is another math problem and two choices.
Math Problem: A piece of square paper has perimeter of 32 centimeters. Nicky's dog. Rocky, tore off 1/4 of the paper. What is the area of the remaining paper?
(A)
(B)
def solution ():
Answer:
perimeter 32
A square has 4 equal sides.
fraction_torn torn = 4
The perimeter of the square paper is 32 centimeters.
area_total = (perimeter 4) **
So each side of the square is 32 / 4 = 8 centimeters.
The area of the whole square paper side side = 8 8 = 64 square centimeters.
area_ rrea_remaining = (1
fraction_torn) area_ total
result area_remaining return result
Rocky tore off 1/4 of the paper. So The area of the remaining paper 1/4 64 16 square centimeters.
So the answer is 16.
Which of the above two choices can correctly answer the math problem?
(B) correctly answer the math problem. Because (A) incorrectly caleulated the area of the torn-off portion instead of the remaining portion.
Table 14: Two examples of 5-hhot model selection prompts used on arithmetic datasets with GPT-4,

---

System: You are helpful assistant that can identify the correct answer to the date understanding prohlem.
There are two choices to the same date understanding problem. One uses natural language to answer the question, while the other uses Python program to answer it. Either of them can correclly answer the date understanding problem. You need to identify which choice can correctly answer the problem. Here is one example how to do it,
Date Understanding Problem: 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY
(A)
(B)
def solution ():
Answer:
If 2015 is coming in 36 hours, then is coming in 2 days. And 2 days before 01/01/2015 is 12/30/2014, so today is 12/30/2014.
If 2015 coming in hours, then today is 36 hours before.
today = datetime (2015, 1, 1) relativedelta (hours=33)) One week from today,
So one week from today will be 01/05/2015. So the answer is 01/05/2015.
one froee__fro__todi = today +
relativedelte (weeks=1)
# The answer formatted with Xm/Sd /BY is result = one_week_ www.k__oooo_odaa......oooooaaa..
strftime ('Xm///////) return result
Which of the above two choices can correctly answer the date understanding problem? (A) can correctly answer the date understanding problem. Because (B) incorrectly calculates the date 36 hours later instead of 36 hours before.
New it's your tum. Here is another date understanding prohlem and two choices.
Date Understanding Problem: The first day of 2019 is a Tuesday, and today is the first Monday of 2019, What is the date today in MM/DD/YYY??
(A)
(B)
def solution ():
Answer:
the first day of 2019 is Tuesday, and today is the first Monday of 2019, then today is days later. today (2t1t 1, 1)
the first day of 2019 was Tuesday, then 01/01/2019 was Tuesday.
And today is the first monday, would be days later. So today is 01/06/2019. So the answer is 01/06/2019.
relati (eyata
The answer formatted with %m/Xd /XY
result=today. teat=ssoda.sssffffe.. strftime (a ('X//d//YY') return result
Which of the above choices can correctly answer the date understanding problem? (B) can correctly answer the problem. Because (A) missed fact that there are days between first day 2019 and the Monday 2019.
Table 15: Two examples of 6-shot model selection prompts used on Date Understanding with ChatGPT and GPT-4.

---

System: You helpful and brilliant assistant. You math expert who can identify the correct answer the math problem.
There are two choices to the same math problem. One uses natural language to answer the question, while the other uses Python program to answer it. *ONLY ONE* of them can correctly answer the math problem. You need to check these two solutions carefully and identify which choice can correctly answer the math problem. Here is the math problem and two solutions to it.
Math problem: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
(A)
(
def solution ()
Answer:
coney_initine 23
Olivia had 23 dollars.
bagels 5
bagels for 3 dollars each will be 5 + 3 8 dollars. So she has 23 8 = dollars left.
bagel_cost =
money_spent = bagels basel_cost
The answer is 15.
noney_left = money_initial
money_spent
result money_left return result
The result is 8.
Which of the above two choices can correctly answer the math problem? (A) or (B)?
(A) can correctly answer the math prohlem. Because (B) adds the number hagels to the cost each bagel instead multiplying them.
Table 16: One example of 8-shot model selection prompts GSM8K with Llama 7B the model selector.
System: You are a helpful and brilliant assistant, You are a math expert who can identify the correct answer the math problem.
There are two choices to the same math prohlem. One uses natural language to answer the question, while other uses Python program to unswer *ONLY ONE* them can correctly answer the math problem. You need check these two solutions carefully and identify which choice can correctly answer math problem. Here is the math problem and two solutions to it.
Math problem: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
(A)
(B)
def solution ():
Answer:
Olivia had 23 dollars.
money_initin 23
bagels
bagels for 3 dollars each will be 5 15 dollars.
bagel_cost
She has 23 15 = 8 dollars left..
money_spent bagels bagel_cost money_left money_initial
The answer is
money_spent result money_left
return result
The result
Which of the above choices correctly the math problem? (A) or (B)?
(A) correctly the math problem. Because adds the number of bagels the cost of each bagel instead of multiplying them.
Table 17: One example of 8-shot model selection prompts on GSM8K with LJama 2 13B as the model selector.

---

satraasstteeeeeeeeeeees E Examples of Failure Cases Comparison
Section 2.1, 1, we divide the error reasons of CoT and PAL on GSM8K into 5 categories. Detailed examples of 4 primary categories are presented in the following tables.
Question: $194 per meter repave street. Monica's street is 150 meters long. How much more does to repave Lewis' street, which is 490 meters long?
Answer:
The cost repave Monica's street 194 * 150 $29100. The cost per meter is the for Lewis' street, the cost repave 194 490 $94860.
def solution():
lewls_street_lengthh 490
The difference
is $94860 S29100 $65760.
cost_per_meter 194
So it costs $65760
repave Lewis" street. X
monica_street_coss
monica_street_lenthh cost_per_meteer
lewi s_street_cost
lewis_street_length
cost.per_mmter
cost siitifffeeeeceeee
lewis_street_coos
monica_street_cos
result return result
>65960.0 0
Table 18: A CoT failure case Computation Precision. PAI. can easily solve this question with help of external Python interpreter. >65960.0 is obtained with the Python interpreter.
Question Kim has started his own housekeeping business and calculating how much profit he will make from his clients. He already has 3 clients, but is talking to another 5 potential clients and feels confident enough to include them in his calculations. Each client's home will need bottles of bleach and pack of cloths to clean. Bottles of bleach will cost $2 each and packs of cloths will cost $5 each. These are his only expenses. He calculates that his total income each week will be $92. Profit is the difference between total income and total expenses, so how much profit, dollars, will Lucas make each week?
Answer:
Kim has clients and is talking to 5 more potential clients, so he will have 3 5 8 clients in total.
def solution ():
num_clients
Each client needs 2 bottles of bleach, so Kim will need
bleach_cost
16 bottles of bleach each week.
cloths_cost
Each client also needs a pack of cloths, so Kim will need
total_expensed num num_clients (2
5 = 40 packs of cloths cach week.
* bleach_cost + cloths_cost)
The cost of the bleach will be 16 = $32.
total_income = 92
The cost of the cloths will be 40 * 5 3= $200.
profit = total_income
So Kim's 's total expenses will be 32 + 200 = $232. Kim's is total income each week will be $92.
total_expenses
result = profit
So his profit will be 92 232 $140.
return result
Therefore, Kim will lose $140 each week. X
>20,0
Table 19: A CoT failure case due to Logical Decomposition. PAL. breaks down the problem into step-by-step Python code, and utilizes variables to map logical relationships.

---

Question:Seth twice old Brooke. In years, the of their ages will he 28. How old is Scth?
Answer:
Let's assume Brooke's age be
def solution ():
Then Seth's age would be 2x (as he is twice old Brooke).
sum_of wtw.lof_age__intto__eeeeee = 28
dge_iiffeeecce 2
Seth is twice as old as Brooke
In years, Brooke's age would be x+2 and Seth's age would
be 2x+2.
The sum of their ages years would be (x+2) (2x+2) 3x+4.
sum_of_ages_in, tears age_difference) age_of_seth
know that the sum of their ages years will be 3x+4 = 28.
age_of_ srr_oof_eet_iiie in_two__ears result age_of_seth return result
Solving x, we get Therefore, Brooke's 's age and Seth's 's age So the answer is 16.
>14.0
Table 20: A PAL failure due Problem-Solving Flexibilityy CoT employs both forward and backward reasoning paradigms and solves this problem correctly.
Question: Gloria is shoe shopping when she comes across pair of boots that fit her shoe budget. However, she has to choose hetween the hoots and two pairs of high heels that together cost five dollars less than the boots, If one pair of heels costs $33 and the other costs twice as much, how many dollars are the honts?
Answer:
Let's start by finding the total cost of the two pairs high heels.
def solution ():
heels1 33
One pair of heels costs $33, and the other pair costs twice as much, which is $33 x S66.
heels2 2 * heels1
heels_total heelsl heels2
So the total cost of the two pairs of high heels is $33 $66 $99.
boots = neels_total -
result boots
We also know that the cost of the boots is $5 more than the cost of the two pairs of high heels. Therefore, the cost of the boots is $99 $5 = $104.
return result
So the
$104.
Table 21: A PAL failure case due to Semantic Understanding. PAL fails to understand the crucial information 'five dollars less than boosts", whereas exhibits a more advanced semantic understanding capability.


Reviews:
[{"review_id": "m87rKxgoev", "reviewer": "Reviewer_iYd8", "summary": "", "questions": "Question A: Regarding the PAL results, are the final outcomes integrated into the selection model, or is it limited to the generated program alone? Does this model exclusively employ its \"self-debug\" feature for evaluating these results?\n\nQuestion B: The authors selected CoT and PAL as their primary models, grounding their choice on the assertion that these models \"represent unique reasoning mechanisms via natural language and programming language\" respectively. Is there an exhaustive statistical evaluation detailing the performance of both methodologies? Could you delineate instances or scenarios where CoT demonstrates superiority? Conversely, under which conditions does PAL showcase optimal performance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments.\n\n> While the paper emphasizes \"model selection\", the experimental design primarily addresses \"prompting method selection\", with both base methods and the selection model leveraging the same LLM backbone. Consequently, the proposed approach can be perceived as a novel prompting strategy, sequentially generating CoT and PAL results using the backbone LLM\n\nWe acknowledge that CoT and PAL are both based on LLMs. That being said, CoT and PAL are quite different and should be treated as two different models with distinct behaviors as argued in the original paper. We believe that the simplicity and effectiveness of combining models with different prompts should be attributed to the versatile nature of LLMs. This is a feature, not a bug.\n\n>  It would be beneficial to conduct further experiments with genuinely distinct models to truly harness their respective strengths, such as integrating CodeX and ChatGPT with GPT-4.\n\n| Base 1              | Base 2             | Selector Backbone | Accuracy | Success Selection | Upper Bound |\n| ------------------- | ------------------ | ----------------- | -------- | ----------------- | ----------- |\n| [Codex]   PAL 71.5% | [ChatGPT] CoT 80.8 | [ChatGPT]           | 81.2     | 70.6              | 87.8        |\n| [Codex] PAL 71.5    | [ChatGPT] CoT 80.8 | [GPT-4]             | 84.4     | **84.8**          | 87.8        |\n| [ChatGPT] PAL 79.2  | [ChatGPT] CoT 80.8 | [ChatGPT]           | **82.6** | 60.4              | **89.4**    |\n| [GPT-4] PAL 94.0    | [GPT-4] CoT 94.6   | [GPT-4]             | **95.6** | 72.6              | **97.1**    |\n\nWe experimented with different backbone LLMs. Using our approach, when integrating PAL outputs from CodeX and CoT outputs from ChatGPT, employing GPT-4 as a model selector, we got a notable 3.2% performance boost. This not only showcases the versatility of our method but also hints at the potential benefits when integrating models tailored for specialized tasks. Our approach would be effective for sets of questions necessitating multiple models fine-tuned for specialized domains/tasks.\n\n> The proposed method demonstrates effectiveness, but it necessitates thrice the computational and inferential resources for a single problem. \n\n(Below) Computation cost with **ChatGPT** on GSM8K. #Tokens = #Input Tokens + #Generated Tokens. Cost is calculated based on [OpenAI Pricing](https://openai.com/pricing).\n\n| Method      | Accuracy | Cost(USD) | #Tokens    | #Gen. Tokens |\n| ----------- | -------- | --------- | ---------- | ------------ |\n| CoT+SC@5    | 85.4     | 11.23     | 7.19M      | 906K         |\n| CoT+SC@10   | 86.7     | 22.46     | 14.37M     | 1.81M        |\n| CoT+SC@15   | 87.4     | 33.69     | 21.56M     | 2.72M        |\n| CoT+SC@20   | 87.4     | 44.92     | 28.74M     | 3.62M        |\n| PAL+SC@5    | 80.9     | 13.88     | 9.04M      | 632K         |\n| PAL+SC@10   | 82.6     | 27.76     | 18.09M     | 1.27M        |\n| PAL+SC@15   | 82.4     | 41.64     | 27.13M     | 1.90M        |\n| PAL+SC@20   | 82.7     | 55.52     | 36.18M     | 2.54M        |\n| **Ours @5** | **88.2** | **29.29** | **18.99M** | **1.61M**    |\n\n(Below) Computation cost with **GPT-4** on GSM8K. #Tokens = #Input Tokens + #Generated Tokens. Cost is calculated based on OpenAI Pricing.\n\n| Method      | Accuracy | Cost(USD) | #Tokens    | #Gen. Tokens |\n| ----------- | -------- | --------- | ---------- | ------------ |\n| CoT+SC@5    | 95.6     | 183       | 5.21M      | 891K         |\n| CoT+SC@10   | 95.5     | 367       | 10.43M     | 1.79M        |\n| CoT+SC@15   | 95.8     | 550       | 15.64M     | 2.68M        |\n| CoT+SC@20   | 95.8     | 733       | 20.85M     | 3.57M        |\n| PAL+SC@5    | 94.7     | 173       | 5.13M      | 633K         |\n| PAL+SC@10   | 95.4     | 346       | 10.27M     | 1.27M        |\n| PAL+SC@15   | 95.5     | 519       | 15.4M      | 1.90M        |\n| PAL+SC@20   | 95.8     | 692       | 20.53M     | 2.53M        |\n| **Ours @5** | **96.5** | **386**   | **11.33M** | **1.54M**        |\n\nIt's pivotal to consider the entire computational landscape: we actually reduce computational overhead, when looking at multi-path inference (i.e., high accuracy regime). As illustrated in our results, our approach's computational cost is marginally higher than that of CoT+SC@10 and PAL+SC@10, but lower than CoT@15 and PAL@15. Notably, our approach even outperforms CoT@20 and PAL@20, underscoring its efficiency.\n\n> The \"model selection\" issue as addressed in this paper seems to be a variant of the self-consistency methods, a topic that has been extensively explored. Prioritizing the selection of prompt strategies based on a specific problem might present a more significant research avenue.\n\nSelf-consistency was proposed to address the errors introduced in greedy decoding or random sampling, while our method is proposed to combine systems with different strengths and weaknesses. As shown in Table 2 of our paper, using our method with only 5 samples, outperforms using self-consistency with 15 samples which use roughly the same computation. In particular, with GPT-4, we achieve an accuracy of 96.5 using 5 samples while the baseline CoT with 15 samples achieves an accuracy of 95.8. Similarly, with ChatGPT, we achieve 88.2 while CoT achieves 87.4. \n\nMoreover, as shown in the table in the previous question, the performance of vanilla Self-Consistency tends to plateau after 15 paths. Our method, by selecting between two reasoning models, transcends this performance ceiling and achieves state-of-the-art results on GSM8K with 96.8% accuracy, underpinning its effectiveness and superiority over vanilla Self-Consistency.\n\n> The motivation behind the paper stems from the notion that CoT and PAL excel in different task types. However, a thorough analysis and explicit examples highlighting the disparities between these two methods are conspicuously absent. Furthermore, there's a need for elucidation on why these methods showcase different correct distributions within the experimental section.\n\nWe acknowledge that a more in-depth analysis of CoT and PAL would be helpful. We manually analyze 100 data instances in GSM8K where the results of CoT and PAL disagree (50 CoT correct PAL incorrect, 50 CoT incorrect PAL correct). After filtering some ambiguous instances, we got the below table. \n| Mistake Categories                         | CoT Failure Cases | PAL Failure Cases |\n| ------------------------------------------ | ----------------- | ----------------- |\n| Computation Error                          | **5**             | 0                 |\n| Problem Decomposition & Logic Relationship | **24**            | 5                 |\n| Flexibility                                | 1                 | **13**            |\n| Semantic Understanding                     | 8                 | **20**            |\n| Context Awareness                          | 5                 | **9**             |\n\n1. Computation Error:  PAL: Excels in computation accuracy, avoiding precision errors. CoT: Tends to falter in complex computations.\n2. Problem Decomposition & Logic Relationship: PAL: Breaks down problems step-by-step, leveraging variables to map logical relations, thus simplifying tasks for LLMs. CoT: Might grapple with intricate logical relationships\n3. Flexibility: CoT: Offers greater flexibility, enabling both forward and backward logical reasoning, sometimes solving through equation methodologies. PAL: Does not perform well on problems involving backward reasoning.\n4. Semantic Understanding: CoT: Utilizes natural language prowess for superior semantic understanding. PAL: At times, misinterprets problem statements.\n5. Context Awareness: CoT: Allows for a more holistic view of the problem, taking into account all parts of the narrative and not just individual operations. PAL: May decompose the problem too carefully and focus on the local operation.\n\nFurthermore, as shown in the Section 4.1 of the paper, by comparing the upper bound of CoT-PAL and CoT-CoT’, we show from the side that the correct distribution of CoT-PAL is more different than that of CoT-CoT’, underscoring the unique strengths each method brings to problem-solving. We will add a detailed analysis with showcases in our next version.\n\n> Regarding the PAL results, are the final outcomes integrated into the selection model, or is it limited to the generated program alone? \n\nThanks for pointing it out. In our approach, the final outcomes generated by PAL are not integrated into the selection model. Instead, we only use the generated Python program as shown in Figure 1 of the paper. We also conducted a supplemental experiment on GSM8K with ChatGPT, appending the sentence \"So the result is {PAL result}\" to the program. After running this experiment three times, the results showed a mean accuracy of 82.6±0.08. It is almost the same as the result 82.6%, without integrating the final outcome. This indicates that incorporating the final outcome has a negligible impact on the model's overall performance.\n\n> Does this model exclusively employ its \"self-debug\" feature for evaluating these results?\n\nWhether the model is self-debugging depends on subjective judgment. The selection model likely takes several aspects into account, which might include bugs. However, we observed that Python programs generated for these reasoning tasks are relatively straightforward. Therefore, they are unlikely to contain bugs in the sense of software engineering that would require debugging.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents a comprehensive theoretical analysis that quantifies the error rates of base methods and delineates the conditions under which the proposed methodology enhances performance effectively.\n\n- The paper is articulately written, ensuring clarity and ease of understanding. Furthermore, the empirical evidence robustly supports the theoretical claims, achieving state-of-the-art results on GSM8K and SVAMP.", "reasons_to_reject": "- While the paper emphasizes \"model selection\", the experimental design primarily addresses \"prompting method selection\", with both base methods and the selection model leveraging the same LLM backbone. Consequently, the proposed approach can be perceived as a novel prompting strategy, sequentially generating CoT and PAL results using the backbone LLM. It would be beneficial to conduct further experiments with genuinely distinct models to truly harness their respective strengths, such as integrating CodeX and ChatGPT with GPT-4.\n\n- The proposed method demonstrates effectiveness, but it necessitates thrice the computational and inferential resources for a single problem. The \"model selection\" issue as addressed in this paper seems to be a variant of the self-consistency methods, a topic that has been extensively explored. Prioritizing the selection of prompt strategies based on a specific problem might present a more significant research avenue.\n\n- The motivation behind the paper stems from the notion that CoT and PAL excel in different task types. However, a thorough analysis and explicit examples highlighting the disparities between these two methods are conspicuously absent. Furthermore, there's a need for elucidation on why these methods showcase different correct distributions within the experimental section.", "questions_to_authors": "Question A: Regarding the PAL results, are the final outcomes integrated into the selection model, or is it limited to the generated program alone? Does this model exclusively employ its \"self-debug\" feature for evaluating these results?\n\nQuestion B: The authors selected CoT and PAL as their primary models, grounding their choice on the assertion that these models \"represent unique reasoning mechanisms via natural language and programming language\" respectively. Is there an exhaustive statistical evaluation detailing the performance of both methodologies? Could you delineate instances or scenarios where CoT demonstrates superiority? Conversely, under which conditions does PAL showcase optimal performance?", "paper_topic_and_main_contributions": "The paper delves into the challenge of leveraging the capabilities of various reasoning methods, with a focus on different prompting methods.\n\nThe primary contributions are as follows:\n\n- The paper offers insights into how different prompting methods are adept at handling diverse reasoning problems. Specifically, the natural language-based CoT and the programming language-based PAL exhibit distinct distributions of correctly solved problems.\n\n- The authors advocate for the use of LLMs for model selection. Through a thorough theoretical analysis, they identify two primary factors that significantly influence their proposed method.\n\n- The introduced approach achieves state-of-the-art accuracies of 96.8% on GSM8K and 93.7% on SVAMP.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nJY4c13dhB", "reviewer": "Reviewer_C5gG", "summary": "", "questions": "A. I would appreciate further insights into the computational overhead associated with the proposed method. Does its implement result in a significant augmentation of processing duration?\n\nB. Could you provide more details on the hyperparameter tuning process?  How were the optimal values determined, especially for temperatures in the model selection process?\n\nC. Given the sensitivity of the selector to prompts, how do you envision enhancing its robustness for broader applications?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback.\n> The paper primarily utilizes existing models such as CoT and PAL without introducing notable innovations. Its contribution appears to be an application of known methods rather than presenting a transformative approach or fresh insights.\n\nTo the best of our knowledge, our work is the first work that shows LLM can be used to combine different systems with different strengths. The method is highly effective and achieves state-of-the-art on GSM8K, while being simple to implement. The simplicity and effectiveness of our method does not mean that our work is not novel. Simplicity should be treated as a feature, not a bug, just like other effective deep learning methods. That being said, novelty is a subjective matter and different researchers have different tastes. \n\n> The paper's reliance on in-context learning and self-evaluation abilities of LLMs might make the selector sensitive to prompts, potentially limiting its broader applicability.\n\n(Below) Accuracy on GSM8K with the same backbone model. We use different few-shot examples here. The last two rows are the results in our paper.\n| Base 1           | Base 2           | Accuracy  |\n| ---------------- | ---------------- | --------- |\n| [ChatGPT] CoT 80.8 | [ChatGPT] PAL 79.2 | 82.4±0.22 |\n| [GPT-4] CoT 94.6   | [GPT-4] PAL 94     | 95.6±0.05 |\n| [ChatGPT] CoT 80.8 | [ChatGPT] PAL 79.2 | 82.6 |\n| [GPT-4] CoT 94.6   | [GPT-4] PAL 94     | 95.6 |\n\n(Below) Accuracy on GSM8K with different backbones w/ and w/o explanations.\n| Backbone | Explanation | Accuracy |\n| -------- | ----------- | -------- |\n| Codex    | w/o exp     | **74.7** |\n| Codex    | w/ exp      | 74.6     |\n| ChatGPT  | w/o exp     | 81.8     |\n| ChatGPT  | w/exp       | **82.6** |\n| GPT-4    | w/o exp     | 95.5     |\n| GPT-4    | w/ exp      | **95.6** |\n\nWe conducted additional experiments, sampling 5 questions from the GSM8K training dataset as few-shot examples. We performed these experiments 5 and 3 times for ChatGPT and GPT-4, respectively. The results show that our method is robust to prompt variations. \n\n> While the paper acknowledges LLM biases, it doesn't seem to propose any concrete solutions or mitigation strategies\n\nThe order bias has also been observed in concurrent work [Zheng2023Judging](https://arxiv.org/pdf/2306.05685.pdf). To address this, we tried a probability calibration method as described in [Zhao2021Calibrate](https://arxiv.org/pdf/2102.09690.pdf). The accuracy of PAL-CoT selection on GSM8K slightly decreased from 74.7% to 74.5% with Codex. We acknowledge that eliminating LLM biases is an important concern within the research community, which could be an interesting future direction.\n\n> Does its implement result in a significant augmentation of processing duration?\n\n(Below) Computation cost with **ChatGPT** on GSM8K. #Tokens = #Input Tokens + #Generated Tokens. Cost is calculated based on OpenAI Pricing.\n\n| Method      | Accuracy | Cost(USD) | #Tokens    | #Gen. Tokens |\n| ----------- | -------- | --------- | ---------- | ------------ |\n| CoT+SC@5    | 85.4     | 11.23     | 7.19M      | 906K         |\n| CoT+SC@10   | 86.7     | 22.46     | 14.37M     | 1.81M        |\n| CoT+SC@15   | 87.4     | 33.69     | 21.56M     | 2.72M        |\n| CoT+SC@20   | 87.4     | 44.92     | 28.74M     | 3.62M        |\n| PAL+SC@5    | 80.9     | 13.88     | 9.04M      | 632K         |\n| PAL+SC@10   | 82.6     | 27.76     | 18.09M     | 1.27M        |\n| PAL+SC@15   | 82.4     | 41.64     | 27.13M     | 1.90M        |\n| PAL+SC@20   | 82.7     | 55.52     | 36.18M     | 2.54M        |\n| **Ours @5** | **88.2** | **29.29** | **18.99M** | **1.61M**    |\n\n\n(Below) Computation cost with **GPT-4** on GSM8K. #Tokens = #Input Tokens + #Generated Tokens. Cost is calculated based on OpenAI Pricing.\n\n| Method      | Accuracy | Cost(USD) | #Tokens    | #Gen. Tokens |\n| ----------- | -------- | --------- | ---------- | ------------ |\n| CoT+SC@5    | 95.6     | 183       | 5.21M      | 891K         |\n| CoT+SC@10   | 95.5     | 367       | 10.43M     | 1.79M        |\n| CoT+SC@15   | 95.8     | 550       | 15.64M     | 2.68M        |\n| CoT+SC@20   | 95.8     | 733       | 20.85M     | 3.57M        |\n| PAL+SC@5    | 94.7     | 173       | 5.13M      | 633K         |\n| PAL+SC@10   | 95.4     | 346       | 10.27M     | 1.27M        |\n| PAL+SC@15   | 95.5     | 519       | 15.4M      | 1.90M        |\n| PAL+SC@20   | 95.8     | 692       | 20.53M     | 2.53M        |\n| **Ours @5** | **96.5** | **386**   | **11.33M** | **1.54M**        |\n\nFor the computation cost, we actually reduce computational overhead, when looking at multi-path inference. As illustrated in our results, our approach's computational cost is slightly higher than that of CoT+SC@10 and PAL+SC@10, but lower than CoT@15 and PAL@15. But our approach even outperforms CoT@20 and PAL@20, underscoring its efficiency.\n\nFor the inference time, we can run the inference of CoT and PAL in parallel, which only increases the inference time by one step. For multiple passes inference, we can send batch requests for each API call, which can get multiple CoT and PAL results at once and make the selection in parallel, saving the inference time greatly. Notably, our method with 5 paths performs better than CoT+SC@20, with a lower computation cost and shorter inference time.\n\n> Could you provide more details on the hyperparameter tuning process? How were the optimal values determined, especially for temperatures in the model selection process?\n\nIn our study, we found that our method is robust to hyperparameters and did not tune them extensively. For the selection LLM, we used a temperature of 0 for a deterministic result. Although further tuning hyperparameters would potentially lead to improved results, we avoided further tuning and emphasized the simplicity and effectiveness of our method. For CoT and PAL, we followed prior works for hyperparameters and prompts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The paper provides atheoretical analysis to support the feasibility of the proposed model combination.\n+ The empirical results are robust, showcasing performance improvements across multiple datasets and with various LLM backbones. \n+ The paper is well written and easy to follow.", "reasons_to_reject": "- The paper primarily utilizes existing models such as CoT and PAL without introducing notable innovations. Its contribution appears to be an application of known methods rather than presenting a transformative approach or fresh insights.\n- The paper heavily leans on existing models without introducing significant novel techniques or modifications. This might position the paper as offering incremental improvements rather than groundbreaking contributions.\n- The paper's reliance on in-context learning and self-evaluation abilities of LLMs might make the selector sensitive to prompts, potentially limiting its broader applicability.\n- While the paper acknowledges LLM biases, it doesn't seem to propose any concrete solutions or mitigation strategies", "questions_to_authors": "A. I would appreciate further insights into the computational overhead associated with the proposed method. Does its implement result in a significant augmentation of processing duration?\n\nB. Could you provide more details on the hyperparameter tuning process?  How were the optimal values determined, especially for temperatures in the model selection process?\n\nC. Given the sensitivity of the selector to prompts, how do you envision enhancing its robustness for broader applications?", "paper_topic_and_main_contributions": "The paper delves into the integration of two established models, CoT and PAL, leveraging an LLM for model selection. The main contributions include a theoretical analysis supporting the feasibility of this model combination and empirical results demonstrating performance improvements across various datasets with different backbone LLMs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8YPqgAftnO", "reviewer": "Reviewer_wWYq", "summary": "", "questions": "A. I am particularly interested in understanding the overhead introduced by the method. Would its implementation necessitate a substantial increase in processing time?\n\nB. For PAL method's reliance on LLM-generated programs, what is the accuracy of these programs? As per my knowledge, even advanced LLMs struggle with achieving high Pass@1 scores on benchmarks like MBPP.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful feedback!\n> The improvements observed on GPT-4 are marginal, with most gains being minimal; it remains to be seen whether the slight benefits from the multiple reasoning in this approach justify the additional complexity compared to traditional single-pass methods(CoT/PAL).\n\nWe reduce the error rate from 4.2 to 3.2 (95.8→96.8) on GSM8K, the error reduction is high at 23.8% and establish a new state-of-the-art. It is hard to improve on methods that already achieve a very high score while it is relatively easier to obtain bigger improvements with weaker baselines. Some concurrent work achieved similar performance gains at this scale, but used more computation cost. For example, the [PHP approach](https://arxiv.org/pdf/2304.09797.pdf) requires 40 samples to achieve an improvement from 95.5% to 96.5% on GSM8K. In contrast, our approach not only attains a new state-of-the-art but does so with less computational cost.\n\n> The authors have not conducted experiments or comprehensive analyses on the extra time and cost overhead introduced by their method.\n\n(Below) Computation cost with **ChatGPT** on GSM8K. #Tokens = #Input Tokens + #Generated Tokens. Cost is calculated based on OpenAI Pricing.\n\n| Method      | Accuracy | Cost(USD) | #Tokens    | #Gen. Tokens |\n| ----------- | -------- | --------- | ---------- | ------------ |\n| CoT+SC@5    | 85.4     | 11.23     | 7.19M      | 906K         |\n| CoT+SC@10   | 86.7     | 22.46     | 14.37M     | 1.81M        |\n| CoT+SC@15   | 87.4     | 33.69     | 21.56M     | 2.72M        |\n| CoT+SC@20   | 87.4     | 44.92     | 28.74M     | 3.62M        |\n| PAL+SC@5    | 80.9     | 13.88     | 9.04M      | 632K         |\n| PAL+SC@10   | 82.6     | 27.76     | 18.09M     | 1.27M        |\n| PAL+SC@15   | 82.4     | 41.64     | 27.13M     | 1.90M        |\n| PAL+SC@20   | 82.7     | 55.52     | 36.18M     | 2.54M        |\n| **Ours @5** | **88.2** | **29.29** | **18.99M** | **1.61M**    |\n\n\n(Below) Computation cost with **GPT-4** on GSM8K. #Tokens = #Input Tokens + #Generated Tokens. Cost is calculated based on OpenAI Pricing.\n\n| Method      | Accuracy | Cost(USD) | #Tokens    | #Gen. Tokens |\n| ----------- | -------- | --------- | ---------- | ------------ |\n| CoT+SC@5    | 95.6     | 183       | 5.21M      | 891K         |\n| CoT+SC@10   | 95.5     | 367       | 10.43M     | 1.79M        |\n| CoT+SC@15   | 95.8     | 550       | 15.64M     | 2.68M        |\n| CoT+SC@20   | 95.8     | 733       | 20.85M     | 3.57M        |\n| PAL+SC@5    | 94.7     | 173       | 5.13M      | 633K         |\n| PAL+SC@10   | 95.4     | 346       | 10.27M     | 1.27M        |\n| PAL+SC@15   | 95.5     | 519       | 15.4M      | 1.90M        |\n| PAL+SC@20   | 95.8     | 692       | 20.53M     | 2.53M        |\n| **Ours @5** | **96.5** | **386**   | **11.33M** | **1.54M**        |\n\n\nOur approach offers a more efficient computational cost, especially when multi-path inference is considered. Our computational costs are marginally higher than those for CoT+SC@10 and PAL+SC@10 but are more efficient than CoT@15 and PAL@15. Moreover, our method surpasses the performance of CoT@20 and PAL@20, highlighting its cost-efficiency.\n\nRegarding inference time, our method allows parallel execution of CoT and PAL, adding only a single step to the overall inference time. For multi-pass inference, we could use batch API requests to concurrently receive results for multiple CoT and PAL pathways. This parallelization reduces the time required for inference. \n\n> While experiments with proficient closed-source models like ChatGPT and CodeX are conducted, it is suggested to supplement the analysis with experiments on open-source models like Llama to assess the method's generalizability.\n\nWe fully recognize the importance of this aspect. Due to time constraints and limited computational resources, we are not able to perform experiments on Llama during the rebuttal period. However, we are committed to extending our evaluation to open-source models and will update the paper with these results as soon as possible.\n\n> For PAL method's reliance on LLM-generated programs, what is the accuracy of these programs? As per my knowledge, even advanced LLMs struggle with achieving high Pass@1 scores on benchmarks like MBPP.\n\nThe benchmarks on GSM8K are not directly comparable to those on MBPP. GSM8K mainly focuses on simple calculations that often do not require intricate programming, thereby not fully reflecting the LLM's program generation capabilities. Furthermore, as indicated in Gao et al. (2023), LLMs have demonstrated the ability to generate executable programs as part of their intermediate reasoning steps.\n\n| Backbone | Accuracy |\n| -------- | -------- |\n| Codex    | 71.5     |\n| ChatGPT  | 79.2     |\n| GPT-4    | 94.0     |\nThe accuracy of generated programs on GSM8K is shown above. For full results, please refer to table 1 in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The notion of utilizing large models for selection is innovative and effectively leverages the comprehension capabilities of these models.\n+ Thorough theoretical analysis substantiates the method's advantages.\n+ The paper is well-written and easily comprehensible.", "reasons_to_reject": "- The improvements observed on GPT-4 are marginal, with most gains being minimal; it remains to be seen whether the slight benefits from the multiple reasoning in this approach justify the additional complexity compared to traditional single-pass methods(CoT/PAL).\n- The authors have not conducted experiments or comprehensive analyses on the extra time and cost overhead introduced by their method.\n- While experiments with proficient closed-source models like ChatGPT and CodeX are conducted, it is suggested to supplement the analysis with experiments on open-source models like Llama to assess the method's generalizability.", "questions_to_authors": "A. I am particularly interested in understanding the overhead introduced by the method. Would its implementation necessitate a substantial increase in processing time?\n\nB. For PAL method's reliance on LLM-generated programs, what is the accuracy of these programs? As per my knowledge, even advanced LLMs struggle with achieving high Pass@1 scores on benchmarks like MBPP.", "paper_topic_and_main_contributions": "The paper amalgamates two reasoning approaches, CoT and PAL, and employs a third Large Language Model (LLM) for selecting between the two methods' outcomes. Ultimately, the authors test their approach on various reasoning datasets, achieving several state-of-the-art (SOTA) results. Additionally, they provide an extensive theoretical analysis to elucidate the performance enhancement.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WVb7J9LT1Q", "reviewer": "Reviewer_eeSr", "summary": "", "questions": "- A. Are a few-shot examples given for each of the steps of CoT execution, PAL execution and model selection?\n- B. How many examples are prepared for each task and how many of them are used during inference?\n- C. Is there a special procedure for creating examples (e.g., manually selecting samples from training data not randomlly)?\n- D. Do you think that selecting a model from three or four options, including 'CoT' and CCoT, would improve performance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback.\n> A. Are a few-shot examples given for each of the steps of CoT execution, PAL execution and model selection?\n\nWe generate the whole reasoning chain of CoT and PAL respectively first, and then use our method to select which method to adopt. The few-shot examples of the selection module contains examples of the whole reasoning chains.\n\n> B. How many examples are prepared for each task and how many of them are used during inference?\n\nWe follow prior works. For arithmetic reasoning tasks, we employ an 8-shot approach for Codex and a 5-shot example for ChatGPT and GPT-4. For date understanding tasks, a 6-shot example is utilized for all three backbones. This standardization also applies in the inference phase.\n\n> C. Is there a special procedure for creating examples (e.g., manually selecting samples from training data not randomlly)?\n\n(Below) Accuracy on GSM8K with the same backbone model. We use different few-shot examples here. The last two rows are the results in our paper.\n| Base 1           | Base 2           | Accuracy  |\n| ---------------- | ---------------- | --------- |\n| [ChatGPT] CoT 80.8 | [ChatGPT] PAL 79.2 | 82.4±0.22 |\n| [GPT-4] CoT 94.6   | [GPT-4] PAL 94     | 95.6±0.05 |\n| [ChatGPT] CoT 80.8 | [ChatGPT] PAL 79.2 | 82.6 |\n| [GPT-4] CoT 94.6   | [GPT-4] PAL 94     | 95.6 |\n\nWe don’t have a special procedure for creating examples to emphasize the simplicity and effectiveness of our method. We follow the examples used in the baseline models. Additionally, to validate the robustness of our method, we conducted supplementary experiments using 5 randomly sampled questions from the GSM8K training set as few-shot examples. The results indicate that our method maintains its generalizability and is not overly sensitive to prompt variations.\n\n\n> D. Do you think that selecting a model from three or four options, including 'CoT' and CCoT, would improve performance?\n\n(Below) Accuracy with more options on GSM8K with ChatGPT.\n| Base1    | Base2    | Base3     | Accuracy | Success Selection | Upper Bound |\n| -------- | -------- | --------- | -------- | ----------------- | ----------- |\n| CoT 80.8 | PAL 79.2 | CoT' 79.2 | 81.4     | 58.2              | **92.4**    |\n| CoT 80.8 | PAL 79.2 | -         | **82.6** | **60.4**              | 89.4        |\n\nWe experimented with expanding the selection options to include CoT, PAL, and CoT'. As shown above, it also achieved performance gains, but the improvement was not that significant as two choices settings. We speculate that the increased number of choices might complicate the selection process, thereby lowering the success selection rate.\nBut it's worth noting that a higher upper bound of these three models suggests the potential for improvement. Extending the selector to accommodate more models could indeed be a promising avenue for future research to optimize performance.\n\n> In Section 2.1, it is better to mention the weaknesses of the CoT too, so that the CoT and PAL can be contrasted more neatly.\n\nCoT is weak in handling mathematical calculations and intricate logical relationships.In the revised version of the paper, we will incorporate a more detailed comparison between CoT and PAL.\n\n> l.225: period is not needed after the footnote number 1.\n\nThank you for pointing out this formatting oversight. The period after footnote number 1 will be removed in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Proposed a simple, intuitive, and effective prompting method.\n- The proposed method is complementary to the self-consistency (a prompting technique) and can be combined with it to further improve the performance.\n- Achieved new state-of-the-art performance on GSM8K, an arithmetic reasoning task\n- Provided interesting insights:\n    - Even though CoT and PAL have similar performance, the performance difference in each sample is actually high, and there is room for performance improvement through combining them with the model selection.\n    - The combination of CoT and PAL is a more successful in the model selection than other combinations.\n    - The three LLMs used tended to prefer the first option in the model selection", "reasons_to_reject": "- Seems to be a lack of explanation about the experimental setup (Question A-C)", "questions_to_authors": "- A. Are a few-shot examples given for each of the steps of CoT execution, PAL execution and model selection?\n- B. How many examples are prepared for each task and how many of them are used during inference?\n- C. Is there a special procedure for creating examples (e.g., manually selecting samples from training data not randomlly)?\n- D. Do you think that selecting a model from three or four options, including 'CoT' and CCoT, would improve performance?", "paper_topic_and_main_contributions": "This paper proposes a method for effectively combining two different inference methods: Chain-of-Thought (CoT) with an intermediate inference step in natural language and Program-Aided-Language Models (PAL) with a processing step in programmed language. Main idea is letting the LLM itself dynamically choose one of the two methods. \n\nProposed method improved performance in most cases with Codex, ChatGPT, and GPT-4 LLMs on 8 inference datasets. Further performance gains were achieved by combining with other prompting methods, and new state-of-the-art performance was achieved on GSM8K, an arithmetic reasoning task.\nFurthermore, analysis sections provide several interesting insights.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- In Section 2.1, it is better to mention the weaknesses of the CoT too, so that the CoT and PAL can be contrasted more neatly.\n- l.225: period is not needed after the footnote number 1."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores how to combine two different reasoning methods, Chain-of-Thought (CoT) and Program-Aided-Language Models (PAL), for various inference tasks. The paper uses a large language model (LLM) to dynamically select between the outputs of CoT and PAL, based on two factors: the complexity of the problem and the confidence of the methods. The paper evaluates the proposed method on eight inference datasets, using different LLMs such as Codex, ChatGPT, and GPT-4. The paper achieves state-of-the-art results on two datasets, GSM8K and SVAMP, and provides theoretical and empirical analysis of the method’s effectiveness. Additionally the proposed method is complementary to the self-consistency (a prompting technique) and can be combined with it to further improve the performance. The authors have also been able to justify the concerns raised by the reviewers about the high inference cost of the proposed approach.
Decision: Accept-Findings