Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning
Authors: Murong Yue
Jie Zhao
Min Zhang
Liang Du
Ziyu Yao
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Natural Language Processing
Reasoning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: The paper investigates approaches of building LLM cascades for saving the cost of few-shot LLMs in reasoning tasks.

Abstract: Large language models (LLMs) such as GPT-4 have exhibited remarkable performance in a variety of tasks, but this strong performance often comes with the high expense of using paid API services. In this paper, we are motivated to study building an LLM "cascade" to save the cost of using LLMs, particularly for performing (e.g., mathematical, causal) reasoning tasks. Our cascade pipeline follows the intuition that simpler questions can be addressed by a weaker but more affordable LLM, whereas only the most challenging questions necessitate the stronger and more expensive LLM. To realize this decision-making, we consider the "answer consistency" of the weaker LLM as a signal of the question difficulty and propose several methods for answering sampling and consistency checking, including one leveraging a mixture of two thought representations (i.e., Chain-of-Thought and Program-of-Thought). Through experiments on six reasoning benchmark datasets, with GPT-3.5-turbo and GPT-4 being the weaker and stronger LLMs, respectively, our cascade pipeline demonstrates comparable performance but reduces about 60% of the cost compared with fully using the stronger LLM.

Published as a conference paper at ICLR 2024
LARGE LANGUAGE MODEL CASCADES WITH MIX- TURE OF THOUGHT REPRESENTATIONS FOR COST- EFFICIENT REASONING
Murong Yue', Jie Zhao", Min Zhang", Liang Du", Ziyu Yao George Mason University Miicosooft Virginia Tech (myye,, zi ziyuyao @mmu. edu {zhaojie, liang. du @mi crosoft. com [minzhang23)
ABSTRACT
Large language models (LLMs) such GPT-4 have exhibited remarkable per- formance in a variety of tasks, but this strong performance often comes with the high expense of using paid API services. In this paper, are motivated to study building an LLM cascade to save the cost of using LLMs, particularly for perform- ing reasoning (e.g.. mathematical, causal) tasks. Our cascade pipeline follows the intuition that simpler questions can be addressed by a weaker but more afford- able LLM, whereas only the challenging questions necessitate the stronger and more expensive LLM. To realize this decision-making, we consider "answer con- sistency" of the weaker LLM as a signal of the question difficulty and propose several methods for the answer sampling and consistency checking, including one leveraging a mixture of two thought representations (i.c., Chain-of-Thouse
(Wei
et al. 2022) and Sfogram-of-Thoughhe (Chen et al., 2022 Gao al. (023h). Through experiments on six reasoning benchmark datasets, with GPT-3.5-turbo and GPT-4 being the weaker and stronger LLMs, respectively, we demonstrate that our proposed LLM cascades can achieve performance comparable to using solely the stronger LLM but require only 40% of its cost. Our codes are available at / tuub com/ Mur LLM MoT cascade
INTRODUCTION
Large language models (LLMs) such GPT-4 have exhibited remarkable performance in reasoning tasks (Rae et al. 2021 Lewkowycz et al.) 2022 Zhong et al., 2023). Because of the intensive computing resources required for training and hosting the LLMs for inference, many such LLMs are only accessible via paid API services, thus leading to high monetary costs. In this work, we are motivated to study strategies for reducing the costs of using LLMs while not sacrificing task performance, particularly for LLMs' applications to reasoning tasks.
Different types and versions of LLMs often come with different capabilities and costs. Typically, LLMs with better performance (termed "stronger LLMs") are more expensive than those with rela- tively worse overall performance (termed "weaker LLMs")). For example, GPT-4 (OpenAl 2023) 30 times more expensive than GPT-3.5-turbo for the output tokens It thus implies a promising so- lution to cost-saving. That is, simple questions could be answered by the weaker but more affordable LLM, whereas only the difficult questions need to be tackled by the more expensive, stronger LLM. Drawing inspirations from here, Chen et al. (023a) explored the idea of "LLM cascades", where a question is always first answered by a weaker LLM, and then optionally routed to stronger LLM when the the weaker LLM's answer is not accepted (Figure I.. To decide this routing, this work suggested fine-tuning smaller LLM score each question along with its answer produced by the weaker LLM. While this approach could work for some tasks, in practice, we observed that it did not yield satisfying performance for intricate reasoning tasks. Intuitively, it is very challenging to evaluate the difficulty and the answer correctness of a reasoning question solely based on its literal expression, even with a large enough I.M,, since the errors could be nuanced despite the reasoning paths appearing promising (Madaan et al. 2023).
https://openal) http://pppeai..oooorrrcccoo

---

Published as a conference paper at ICLR 2024
In this work, we proposed to devise this routing decision maker from different angle, ie., the "answer consistency" of the weaker LLM (Wang et al. 2023). This is inspired by the observation that answers from the weaker LLM tend to be consistent in multiple sampling paths when the ques- tion is easy, but inconsistent when the question is hard. To implement this idea, we proposed two types of methods, a vote-based method that examines if the agreement of multiple answer samples on the majority-voted answer surpasses pre-defined confidence threshold, and a verification- method that checks if the majority-voted answers sampled from different prompts are consistent. To realize the two methods in reasoning tasks, we further investigated multiple strategies for answer sampling, including sampling from a single set versus two sets of task demonstrations. In partieu- lar, we proposed to leverage "mixture of thought (MoT) representations", which samples answers from both Chain-of-Thougee
(Wei et al., 2022 CoT) and sagerram-ofTThuugle (Chen et al. 2022] Gao et al. 2023 PoTT) prompts, emulating how experts can provide diverse perspectives to the same question. This follows the same spirit of ensembling (Rokach) 2010), but is applied to developing LLM cascades for the first time. By pairing different sampling strategies with the two answer consis- tency checking methods (i.e., vote and verification), we end up with ten approaches to implementing the LLM cascade.
To evaluate the proposed approaches, we conducted experiments on six reasoning datasets, covering mathematical, symbolic, and causal reasoning tasks, using GPT-3.5-turbo the weaker LLM and GPT-4 as the stronger one. The experimental results demonstrated the effectiveness of L.M cas- cades based on answering consistency. That is, different approaches that we proposed for LLM cas- cades can generally achieve performance comparable to or even better than fully using the stronger LLM, while they require only half or less relative cost the latter. In particular, our approaches based on a mixture of thought representations achieved comparable task performance with only 40% of the cost of GPT-4. Our results also underscored the effectiveness of sampling answers from di- verse prompt settings, such sampling from different task demonstrations or different thought rep- resentations. Our further analysis revealed that different prompt settings can often provide different opinions for the more complex questions while tending be consistent for easier ones, which allows us to distinguish questions at different difficulty levels more accurately for cascade lecision-making Finally, we also compared our consistency-basee approaches with fine-tuned smaller LLMs (Chen) et al. (023a) as well as other variants that make the routing decisions based on the literal expressions of the question and its answer. Our approaches exhibited strong advantages over all of them.
LLM CASCADES FOR COST-EFFICIENT REASONING
2.1
OVERVIEW OF LLM CASCADES
We leverage cascade of LLMs to save the cost of in-context LLM reasoning, as illustrated in Figure D Specifically, we assume two LLMs. The weaker LLM (denoted as LLM")) yields relatively worse performance but is less costly, whereas the stronger LLM (denoted as LLM") enjoys better task performance but more ex- pensive. Given question Q. the LLM cascade first employs the weaker LLM to obtain an ini- tial answer A This answer, along with other metadata produced by the weaker LLM, will then be fed to a cascade decision maker to decide whether the answer can be accepted as the final one. If the answer is rejected, the stronger LLM should be invoked to provide a more reliable answer A As consequence, the total cost of answer- ing the question becomes
Final Answer
Anewer A
Apcapt
A =
Query
Q
Weaker
Decision Maker
LLM Cascade
Reject
Answer
Firal Answer
A= A
Sterger
Figure 1: Illustration of LLM cascade chaining weaker but cheaper LLM with a stronger but more costly one.
C = C + Cdd + Preject C*,
(1)
where C and O* indicate the costs from calling the weaker and the stronger LLMs, respectively, Odd denotes any cost involved in the LLM cascade decision decision-making process, and 1reeect = holds if and only if the decision maker rejects the answer.
Both LLMs solve the question via few-shot in-context learning, e.g., for the weaker LLM, an answer AT is produced by sampling from PLLM" (A E, |2 E. [Q), where E1 E2||E2l.... EM IQ denotes a concatenation of M task demonstrations and the input question Q, forming the "prompt input" to the L.M (Brown al. 2020). As M task examples are used to demonstrate the task,
2

---

Published as a conference paper at ICLR 2024
it indicates "M-shot in-context learning" of LLM. For reasoning tasks, in practice, LLM will typically be prompted to elaborate on its reasoning process via "thought representations". such as Chain-of-Thought (Wei et al., 2022 CoT) and Bfogram-of-Thougeee
(Chen et al., 2022] Gao et al.
2023 PoTT), where the reasoning process is described step by step via natural language and program- ming language, respectively. The answer (e.g.. a numerical result of the mathematical calculation) can then be extracted from the texts (for CoT) or obtained by executing the code (for PoT).
2.2 ANSWER CONSISTENCY-BASED CASCADE DECISION-MAKING
The core of our LLM cascade is the decision maker, which takes in the output from the weaker LLM, and then decides whether to route to the stronger LLM or not. An ideal cascade decision maker should call the stronger L.MM only when the answer by the weaker LLM is wrong, such that the total cost C' can be minimized without degrading the overall task performance (compared with using the stronger LLM all the time). To this end, we propose two methodologies based the "answer consistency" of the weaker LLM, which we elaborate on below.
Answer Consistency and Sources of Sampling Answer consistency has been found helpful for improving the LLM performance in reasoning tasks (Wang et al. 2023). Instead of greedily de- coding one answer for each question, Wang et al.] (2023) sampled a diverse set of reasoning paths (or thought processes) and then selected the most consistent answer by marginalizing out the sam- pled paths. Drawing inspiration from the prior work, we make the following hypothesis: When the weaker LLM samples highly consistent answers for given question, it reveals high "confidence" in solving this question and its most consistent answer is likely to be correct; in this case, there is thus no need to invoke the stronger LLM.
To realize this intuition, we generalize from Wang et al. (202)) and consider three sources of sam- pling consistency:
In-distribution sampling: As Wang et al. (2023), we consider sampling multiple answers given the same prompt input the weaker L.M.. In practice, this can be achieved by setting non-zero temperature for the weaker LLM.
Sampling from different in-context demonstrations: We further consider sampling answers from two sets of task demonstrations under the same thought representation. For example, to demonstrate the CoT process in mathematical reasoning tasks, Wei et al. (2022) annotated eight math examples the demonstrations and performed 8-shot in-context learning. We additionally annotated another eight examples as the second set of demonstrations, which allowed us to further diversify the sources of answer sampling.
Sampling from different thought representations: While existing literature typically investi- gated either CoT or PoT independently. in this work, we propose to leverage the synergy of both thought representations in single task. We hypothesize that an LLM obtains truly high confi- dence in its problem-solvingg only when it is able to produce a consistent answer agnostic to how the intermediate steps are represented. Therefore, we propose to sample the weaker LLM answers from a "mixture of thought (MoT) representations" which includes both CoT and PoT prompts.
Below, we introduce our methodologies for consistency checking based on the answer samples.
Vote-based decision-making The first method calculates the consistency of the weaker LLM's answer samples by voting. Formally, for a single prompt, we denote the set of answers produced by the weaker LLM for each question Q as (AT AA ...., ...,.AAA AW), where K is the pre-defined number of samples. When sampling from two different prompts, we denote (Ali AY, ..., A.,,A, ) and (4g) A22,.... .... Ank,) as the answer samples produced by each of them, where K₁ and K2 represent the pre-defined sample size for each prompt setting, respectively. Note that for this method, we do not distinguish answers sampled with single prompt or multiple prompts (e.g., samples from different prompts have exactly equal weights when voting). The most consis- tent answer can then be selected the one that most samples agree with, and this answer will also be regarded as the final answer A by the weaker LLM. The decision maker measures the weaker LLM's consistency via the agreement score
Method 1:
ΣK AA"=A" K
ΣK =m==A + +kk 1AA=A"
or
(2)
K1 + K₂
3

---

Published as a conference paper at ICLR 2024
Query Q
"Joelle has orchids and African daisies on her belicory. How potats daleles compared to the orchids""
CoZ-ID-Voch
CoT-2D-Vote
Majority
A
Co
Vote
wont
fridd chickens lotal. Therefore, Fau
Answer:
resiaurant...
Pau ordered x 10
Pol- 1D-1----v 1D-Vote
4x
BoT 20T-20-Vote Vote
N
Consistency
Meesure
fried
Aocept
Ans
Reject
Question: Joaiie has archids and 4 Africn daisies on her bakcon...
11
21 22
Vote-based
Answer:
CoT-2D-Vertle
Weaker LLM
AQuestion: a restaurant...
44
BOT-2D-VerLry
44
*Python code, return ans
4
MoT-ID-WerLfy
11
MoT-2D-Verife
pau crdee==kobe ordez*2
pau cutca -2 -2apaia ordez
Question: Joelie 5 orchids and
Majority
Majoniy Vote
African daisies belcony...
Vate
Pyython code, return ans
Acocpi
Verify-based
An overview of our approaches (6 vote-based and 4 verification-based). We use to represent the answers from PoT and to represent the answers from COT. Demo, is the i-th set of demonstrations.
Figure 2:
The larger the the consistent the weaker LLM's answer samples. In conjunction with pre-defined threshold value т, the decision maker accepts the weaker LLM's most consistent answer A s when T and rejects otherwise. As a result, the total cost of answering a question (Eq [ can vary depending on the threshold.
Method 2: Verification- decision-making In the case of producing samples from two dif- ferent prompt settings (i.e., different demonstrations or thought representations), we propose the second method, which compares the most consistent answers produced by each prompt as the an- swer verification. As previously mentioned, we could obtain two sets of answers from distinct prompts: (ATi A .... A(r,) and (A% A ... 222) Our method then verifies the most consis- tent answers within each prompt, denoted as A and Ag respectively, as follows:
= At AM"
Only when equals 1, i.e., when the two answers are the same, the weaker LLM's answer will be accepted by the decision maker. In this case, the final answer of the weaker LLM will be the same as the two most consistent answers, i.e., A AlT = Azd
In comparison, these two methods have different applicable scenarios. The vote-based method is well-suited for scenarios with pre-defined cost constraints. In such cases, we have the flexibility to tune the threshold to ensure it aligns with the constraint. On the other hand, the verification-batt method is capable of producing relatively optimal results without the need for threshold tuning, although lacks flexibility. We will systematically compare the methods in experiments.
2.3 LLM CASCADES FOR REASONING TASKS
We instantiate the proposed two methods in LLM reasoning tasks with different sampling sources, resulting in 10 approaches, as summarized in Figure 2 Specifically, 6 approaches adopt vote-based decision-making: CoT- 1D-Vote collects K answers sampled from prompting the weaker LLM with the CoT representation, and then calculates the answer consistency for decision-making. fol- lowing Eq 2 Similarly, PoT-ID-Vote bases its decision- making on answers sampled from PoT prompt. To diversify the sources of the answers, for each thought representation, we further consider sampling from two sets of CoT PoT demonstrations, resulting in CoT-2D-Vote and PoT-2D-Vot- respectively. Finally, the vote-based approaches also include two variants lever- aging a mixture of thought (MoT) representations. For MoT-1D-Vote K₁ answers are sampled from the CoT prompt and another K2 from the PoT prompt, and union set of their answers then used to compute the consistency score s. For M e, the procedure is similar, except that the CoT and the PoT prompts are annotated from two sets of demonstration examples.
The verification- based approaches assume answer samples from two different prompts. We instantiate 4 variants, including CoT-2D-Verl fy, where prompt the weaker L.M with
4

---

Published as a conference paper at ICLR 2024
two sets of CoT demonstrations, resulting in two answer sets for decision decision-making making (Eq B) PoT-2D-Veriřy, where we similarly prompt the weaker LLM with two sets PoT demonstra- tions; MoI-ID-Verify where we consider two sets of answers from two thought representation prompts (but on the same set of task demonstration examples); and M fy, which ad- ditionally employs different sets of demonstrations when prompting the weaker L.M with different thought representations.
Cost-Comparrble Sample Size Configuration To fairly compare all approaches in terms of their effectiveness in identifying easy (or correct) hard (or correct) questions, aim to configure their costs to be comparable. Since we use the same prompt to the stronger I.L,, C is agnostic to the specific approach. However, different approaches may require varying costs in calling the weaker LLM (i.e., (tn). For example, even with the same total sampling size (i.e., K = K₁ + K2). 1--V-D--eee and CoT-2D-Vone prompt the LLM once but the latter prompt it twice, doubling the token usages. Therefore, we unify the C by coarse-grasd grarseegr cost alignment. Our analysis suggests different configurations of K for different approaches. We refer readers to Appendix A more details. Since our configuration can only yield comparable costs due to certain simplifications, rather than exact costs, we still report the actual token cost for each approach.
result in different input token usages because the former
EXPERIMENT
3.1 EXPERIMENTAL SETTING
We evaluate our LLM cascade approaches on six datasets, covering (1) mathematical reasoning. including GSM8k (Cobbe et al. 202T), ASDIV (Ling et al. 2017), and TahMWP (Lu et al.) 2023); (2) symbolic reasoning from BIG-Bench Hard (bench authors (023), including DATE and Navi- gate; and (3) causal reasoning, including CREPÉ (Zhang et al. 2023). In our pipeline, we leverage the GPT-335-turbo (4k context) the weaker LLM and the GPT4 (8k context) with CoT self- consistency (Wang et al. 2023, SC) as the stronger LLM. Throughout our experiments, we set the number of task demonstrations as M 8. We use the same demonstration examples prior work (Chen et al.) 2022, Wei et al. 2022). When additional task demonstrations are needed (e.g.. for 2D approaches). we randomly sample examples from the training data and manually annotate them with thought representations. We set the number of sampling paths as K = 20 for GPT-3.5- turbo and K
for GPT-4 The sampling temperature by default is 0.4 for both LLMs. The metrics we use are the task accuracy and the relative cost compared with the cost of GPT-4 with CoT SC (denoted as GPT-4-CoT-SC). A lower relative cost and higher accuracy indicate bet- ter performance. We also compared our approaches with baselines using only the weaker LLM (i.e., GPT 3. 5--oT--CC-- sc, 3.T-3.5-P07 SC) or only the strong LLM in different ways (i.e., GPT-4-CoT-Greedh GPT-4-Pos (it-4-Pood 1ii-4--PoT-rreedd Trrrddy, GPT-4-COT-SC, GPT-4-PoT-SC), Our perimental details can be found in Appendix and reproducible prompts Appendix L
3.2 MAIN RESULTS
Figure illustrates the performance of our proposed approaches. For Vote-based approaches, we draw curves by changing the pre-defined threshold 7 varying from 0.4 to 1. A high value of threshold signifies a more rigorous criterion for trusting the answers from the weaker LLM, making more follows:
examples transferred to the stronger LLM. Our observations
Our pipeline achieves comparable task performance with significantly reduced costs. On av- erage, all of our cascade variants (Vote or vyrify) demonstrate significant cost efficiency. In particular, as shown in the average plot, the four McT variants achieve comparable task perfor- mance (~0.9299 accuracy) to GPT-4-CTT-SC (0.931) while demanding only 40% of its cost. On CREPE. MoT variants even outperform GPT-4-Co--SC (0.885 vs. 0.871) at 47% of its cost. Our approaches based on CoT and PoT also exhibit the capability to save costs while maintaining the overall task performance. For example, COT-2D-Vote achieved 0.924 task accuracy on average but demanded only 57% relative cost. These observations suggest the effectiveness of our cascade decision maker via checking the answer consistency of the weaker LLM.
Sampling from diverse prompt settings helps cascade decision-making. Our results show that variants involving diverse sources of sampling, such as CoT/PoT-2D-Vors and MoT-1D/2D-We
more precisely distinguish between easy and hard reasoning questions,
5

---

Published as a conference paper at ICLR 2024
ASDIV
0.54
0.96
Taak Accuracy
5.92
0.90
0.92
9.90
0.85
0.88
0.50
0.96
0.2
0.8
1.0
0.2
0.4
0.5
0.6
Relative Cost
Relative Cost
Retative Cost
DATE
Navigate
CREPE
0.50
0.98
0.85
Acuraey
0.85
0.92
0.80
5.90
0.00
0.75
0.56
0.6
1.2
0.4
1.2
Average
Legend
GPT-3 S-GAT--C
CoT-20-0te
0.50
GPT4 COT
GPT PUT
-0-
-C-
PDT wote
MoT20 Vot
0.88
0.86
GPT---oT-Greedy
81
PST-2C-vertyy
GPT4 PoT Groody
MoT.10 Vor
0.36
Mar1-DDAee
0.82
POT Vo-eee
0.2
0.4
0.6
0.8
1.b
Relative Cost
Figure 3: Main experiment results over six reasoning datasets. The bottom figure represents the average performance. The exact numerical results are included in Appendix C
compared with their counterparts sampling from single sources, i.c., CoI/PoI-1D-Vors For example, between CoT-2D-Va- and CoT-ID-vote, the former outperforms the latter by 1.4% absolute accuracy under the same relative cost of 0.4 on average.
Mixing thought representations is particularly effective. Furthermore, we find that mixing the two thought representations (i.e., (DD-1D/DD-Vote outperforms decision-making using either of them (i.e., CoT-1D/ / and PoT-1D/2D-vote. This is illustrated by the gap in the average plot and is consistent most datasets except DATE, where many test questions very similar to the demonstration examples. Intuitively, this is because different thought representations can bring in more diverse "opinions" of the weaker LLM the same input question, resembling how a group of experts with diverse perspectives could contribute to more effective results in collab- orative work. can also be viewed as "ensembling" LLMs, which utilizes the intuition that variants of the same model typically share few mistakes (Rokach) 2010). We provide a further investigation of this effect in Section 3.3 We also note that when using MoT, no obvious difference is perceived between using one set (i.e., MoT-iD-Vote) or two sets (i.e., MoT-2D-Vot- of task demonstra- tions. This result reveals that tuning the thought representations more helpful for measuring an LLM's (un)certainty on its answer than tuning the task demonstrations.
Increasing the threshold yields marginal benefits for MoT-1D/2D-Vor the curves of McT-ID 2DD --Voe flatten out, showing that pushing the threshold exceedingly high is unnecessary. This is because even for an easy question, the weaker LLM may still have hal- lucinations in small set of answers. Setting the threshold too high can lead to the decision-making being influenced by them. may result in easy questions being sent to the stronger LLM incorrectly, thus driving up the overall cost. In practice, a threshold that can balance cost and accuracy typi- cally falls between 0.5 and 0.6. When comes to 100T-11/2D-200 between cost and accuracy as the verification method can tolerate few hallucinations.
As costs increase,
fy, we can strike a balance
3.3 ANALYSIS ON MIXTURE OF THOUGHT REPRESENTATION
To understand the effect of MOT, we analyze three vote-based approaches. We first group questions into "easy" and "hard" based on whether the weaker LLM can answer them correctly (i.e., whether the majority-voted answer A is correct or not). For all easy/hard questions, we then calculate the average consistency score following Eq 2 We present the average consistency scores for each approach, categorizing easy and hard questions based whether the weaker LLM can answer them correctly (i.e., Aw being correct). A higher average consistency score indicates our decision- makeer places greater trust in the answer from the weaker I.I.. Hence, effective cascade decision-

---

Published as a conference paper at ICLR 2024
GSMBK
Navigate
Average
conmittency
0.8-
0.8-
0.8
nows
0.5
05-
05-
Question
Hard Question
Call Mhil
Mai 10
Question:
football
kong. back forth
forth
dacides
sho starts to yard and back.
does
How farther winner run Ioser?
Bold
Black runs 200 yards for each back and fort becauce... He tuns 3000 yards total
Answer: I.. (..llgored() Kelly runs to the 40: yard line and back 36 times, su ahu runs 54'80-2720 yards. Thus, Blake runs 3000. 2720-290 yards farther than Kelly
Python (Ignored)
because.... Kolly runs 200 vards first
ci distance covered by kelly Flenggh of Footbal Fiold
becouse 100 2 200, She runs 80 yards eash time she does back and forth the
40-yard line because... She rums 2720 from
GoT2
Japo, Selly
these back and forth She runs 2920 lonal
Answer: ((gnoeedd) Kelly runs 34 (40
coverred by distance, covervedeedd bty__y___lllll. kellly
because... The wineer ran 80 yards more because 8000 2520-80.
2720 Bake 3000
pards farther Kally
(Answar via Python exccution: 8520.0
[ Logic Generation Error
1 Value Grounding Error
Figure 4: Average consistency scores (Top) and case studies (Bottom; simplified for presentation purposes) of various approaches showing the advantage of MoT. Shades highlight similar mistakes by CoT even based different task demonstrations, and dissimilar mistakes by PoT.
maker should reveal relatively higher consistency scores for easy questions and lower ones for hard questions, leading larger "gap" between them.
We show the results in Figure (Top). It is observed that all approaches lead to higher consistency scores on the easy questions than those on the hard questions, which explains their overall effec- tiveness for cascade decision-making Because of involving two different thought representations, McT-1D-Vote tends to have lower consistency score compared with the two CoT approaches. However, it still ends up with larger "gap" in the consistency scores for the easy and the hard questions, which particularly prominent on the Navigate dataset where it gives the best perfor- mance gain (Figure 3). In contrast, CoT-1D-Vote results in the smallest score gap, indicating its weakness in distinguishing between the easy and the hard questions, particularly for Navigate. This weakness is mitigated by diversifying the prompting with two of task demonstrations (i.e.. CoT-2D-Vote) but it still underperforms mixing the thought representations.
Finally, in Figure 4 (Bottom) show that when CoT cannot answer hard question, prompting the weaker LLM with another set of task demonstrations (but still under the CoT representation) often yields the same mistaken answer, which thus results in high consistency score. On the contrary, PoT tends to make mistakes in a very different way and result in a different mistaken answer, which explains the low consistency score of MoT-1D-Vote, More cases are shown Appendix D
3.4
ROBUSTNESS EVALUATION
GSMBK variant
Date variant
CREPE variant
3.26
0.90
CoT-2D-Vote T=0.4
0.94
Accuracy
MoT-1D-bbte T=0.4
CoT-2D-Vote T=0.8
Tosk
0.52
0.8
MoT-1D-Wbte T=088
0.80
0.86
0.50
0:85
2.4
Rolative Date varfant
GSH varfant
CREPE varfant
CoT-2D-Vote K=20
0.54
Accurstyy
Acorriy
MoT-ID-Vbte K-20
0.87
CoT-2D-Vste K-40
Yoak
0.52
0.86
MoT-iD-Vote K=40
0.80
0.85
03
Figure 5: Robustness analysis with varying temperature T and sample size K.
7

---

Published as a conference paper at ICLR 2024
We further analyze if our results are sensitive to the change of the sampling temperature T or the sample size K. We select datasets for each type of reasoning task and conduct experiments with CoT-2D-Vote and MoT-ID-Vote. Our results are shown in Figure [ We first look into the effect when increasing the sampling temperature T from 0.4 (our default setting) to 0.8. For both approaches, increasing their temperature yields comparable or slightly better performance. This is owing to the increased answer diversity when the temperature gets higher. However, in any case, NoT-1D-Vote consistently outperforms CoT CoT-2D-Von 2D-tte. Increasing the sample size K from 20 to 40, on the other hand, leads to rightward shift of the curves, implying that it requires higher cost to achieve the same task accuracy. This can be explained by the higher cost of the larger sample size, whereas increasing the sample size does not contribute to the detection of easy vs. hard questions. Like the results of varying temperatures, MoT robustly outperforms CoT in any case.
3.5
COMPARISON TO EXTERNAL TEXT-BASED VERIFIERS
GSMBK
CREPE
0.55
0,80 -
0.85
0.86
NotttD-bbee
Cali2B-ible
MattlD-Weely
0.75
LUM-O
0.85
0.23
-
Finetunes--
LIM-QA
0.2
0.8
1.0
0.6
0.8
0.2
0.4
0.6
0.8
Tireturss-Q4
Relativee Cost
Relative Cost
Reletive Cost
Figure 6: Comparison with external verifiers showing the advantages of our approaches based on answer consistency. We do not report finetuned verifiers for DATE as it does not have a training set.
As mentioned, prior work implemented the LLM cascade decision-makes by training an external verifier, which scores question and its answer (from the weaker LLM) based on their literal de- scriptions (Chen et al. 2023a Šakota et al. 2023). Related to the above work. Chen et al. (2023b) and Madaan et al. (2023) also showed the promise of prompting LLMs to evaluate their own re- sponses. To perform general comparison with such external verifiers, conducted experiments on GSM8k, DATE, and CREPE with the following baselines: Finetuned- O, which is a RoBERTa- base model (Liu et al. 2020) fine-tuned decide whether question should be routed to the stronger LLM based on its description; Fine tetuundd-Q which works similarly as Finetuned-1 but ad- input; LLM-Q Q, where we instead prompt GPT-3.5-turte as the verifier to judge based on the question de- scription; and LLM 2A, which similarly employs GPT-3.5-turo decide upon the question and the weaker LLM's majority-voted answer. We leave details of the baselines in Appendix E
ditionally takes the majority-voted answer from the weaker LLM
The results in Figure o show that incorporating the external verifiers cannot achieve comparable accuracy with GP T - CCCST For example, on the GSM8k dataset, the highest accuracy with the external verifiers is 0.892, which is way lower than the accuracy (0.958) of GPT-4-CoT-SE and the accuracy (0.951) of our approaches. They also show lower task accuracies than our approaches under the same cost. indicates that the external verifiers cannot yield satisfying results in complex reasoning tasks, which can be due to the intrinsic challenge of deciding question difficulty and answer correctness solely based on their textual descriptions. For the calibration analysis, please refer to the appendix[]
3.6 ADDITIONAL STUDIES
How weak can the weaker LLM be? We evaluate our approaches when adopting LLAMA2-13B
Touvrro et al. 2023) as the weaker LLM and GPT-4 as the stronger LLM. The results are shown in the Appendix F On the DATE dataset, our approaches still works. However, LLAMA2- 13B as the weaker I.L doesn't yield ideal results on GSM8K and CREPF. That is because most of the questions GSM8k and CREPE are excessively complex for LLAMA2-133, Therefore, LLAMA2- 13B often fails to answer the questions consistently across multiple samplings. Hence, the choice of a weaker LLM should be contingent on the task's level of difficulty. When the current weaker LLM struggles with the task, it is advisable to consider switching to a more powerful LLM.
Can the stronger LLM benefit from the weaker LLM hints?
In our LLM cascade, all questions will obtain answers from the weaker L.M,, matter if they will be sent to the stronger L.L
8

---

Published as a conference paper at ICLR 2024
or not. Therefore, an interesting question is whether the answer produced by the weaker LLM (correct or incorrect) can provide "hints" to enhance the stronger LLM. To answer this question, we experimented with S-T--D--erife Fy, where we additionally passed the two inconsistent answers AY" and 13" as hints to the stronger L.MM following the format of prior work (Zheng et al., 2023). We observed that the hints can only yield slight improvement performance on GSM8K and CREPE. Therefore, we conclude that hints from the weaker LLM, when it is uncertain about the question, do not help the stronger LLM. Details are in Appendix G
DATE, but greatly hurt the model
Can method generalize to factual-based tasks? We also explored whether our method can be generalized to factual-based reasoning tasks in Appendix
4 RELATED WORK
Earlier research has delved into techniques to enhance the cost efficiency of LLMs, including quan- tization, pruning, and decoding methods(Bai et al. 2022 Kurtic et al. 2023 Leviathan al. 2022). While many LLMs are closed-source, certain studies have concentrated on how to utilize the API efficiently. Chen et al. (2023a) proposed an LLM cascade, which sends a query to a list of LLMs sequentially if the answers provided by the prior LLMs are regarded unacceptable. A concurrent effort is made by Sakota et al. (023), where the proposed approach determines which LLM to in- voke prior to sending the query any LLM. Both of these approaches employed external verifiers fine-tuned from smaller LMs and therefore demanded substantial training data and additional model learning. In our experiments, we showed the ineffectiveness of these approaches when they were used to save costs for complex reasoning tasks, which echos the discovery of the shortcomings of LLM fine-tuning in recent work (Ni et al. 2023). In contrast, we proposed a novel solution for LLM cascades based on the answer consistency of the weaker LLM, which is training-free and is not limited specific weaker LLM.
Reasoning with LLMs LLMs have demonstrated strong capability in solving reasoning tasks (Rae al.) 2021 Lewkowycz et al.] 2022 Zhong al. 2023). The standard few-shot in- context learning for LLMs is to feed the question to the LLM and let it generate the answer directly (Brown et al.) 2020). Wei et al. (2022) then proposed the Chain-of Thooughht approach, showing that prompting LLMs to think step by step can significantly improve their performance in solving reason- ing problems, as activates the built-in reasoning capabilities of LLMs (Madaan & Yazdanbakhshi 2023). To further facilitate the mathematical steps in a reasoning task, Chen et al. (2022) and Gao al. (023) proposed the Program-of of-Thought prompting, using code as the intermediate step and executing it with an external interpreter to entangle reasoning and numerical calculation. While most existing works studied CoT and PoT independently as alternative approaches, several other works have delved into the synergy between them, e.g., integrating code into CoT (Cheng et al. 2023b He- Yueya et al., 2023), ensembling algebraic and code representations (Imani et al. 2023 Yue et al. 2023), performing automatic selection between CoT and PoT (Zhao et al.) [023 Sim- ilar to these works, our approaches employ CoT and PoT jointly; however, unlike them, we leverage this synergy for the novel application of cost-saving for LLM reasoning. On the other hand, like
LLM cascades, there are also recent works building pipelines chaining multiple LLMs (Varshney &
Baral 2022 Xiong et al. [203a|] Li al. 2023 Lin et al. 2023a), but these works have a different focus on improving the task accuracy, rather than reducing the LLM costs.
Uncertainty of LLMs Our approach involves measuring the uncertainty of the results of LLMs. Multiple concurrent works have investigated using the voting score to evaluate the answer's uncer- tainty (Xiong et al. 2023b Si et al. 2023 Cai et al. (023). However, they dismissed the potential application for different representations of intermediate steps in the evaluation.
5 CONCLUSION
We introduce simple yet efficient and universally applicable economical pipeline to dynamically decide the LLMs in reasoning tasks, so as to save the token costs. Our approach based on checking the answer consistency of the weaker LLM is novel and effective. Our discoveries emphasize that leveraging prompts with a mixture of thought representations in weaker LLM achieves the best performance as it introduces diverse answers. Compared with fully employing the stronger LLM, our pipeline requires approximately 40% of expenses to achieve a comparable result. Future works are listed in Appendix K.

---

Published as a conference paper at ICLR 2024
6 ETHICS STATEMENT
Our research endeavors to build a cost-efficient pipeline in solving reasoning problems in the LLM era. The significance of this work holds the potential benefit diverse range of organizations, particularly those with limited financial resources such as local businesses, educational institutions, and non-profit organizations. By devising cost-effective solutions, we empower these resource- constrained entities to harness the reasoning ability of LLMs cheaply, thereby fostering fairness and inclusivity within the NLP community. This aligns with the broader goal of ensuring that ad- vancements in NLP do not remain exclusive to select few, but are accessible to wider audience, regardless of their financial constraints.
Moreover, for these industry giants, the cost savings facilitated by our findings can also be sub- stantial, particularly when facing extremely large throughputs. By optimizing cost-efficiency, our research contributes to not only economic savings but also environmental sustainability by mitigat- ing the carbon footprint associated with running large-scale computations.
REPRODUCIBILITY STATEMENT
Our pipeline is simple to implement and reproducible. We have documented all the experimental details, both in the main text and the appendix sections. While we cannot include the complete text of every prompt due to their excessive length, we do provide examples of each prompt in Appendix L facilitating the reader's ' comprehension of the style employed. All of our implementations (including the complete prompt scripts, the code for training external verifiers, the code for approach evaluation, etc.) are publicly available at httpt::///// http:://gitttte siggt....... com MurongYuee LLLLLM LLM_MMTTTTMMMMLLMMLLLLLLLLLLIL _oo__caccade
8 ACKNOWLEDGEMENT
This project was supported by Microsoft. It was also partially supported by resources provided by the Office of Research Computing George Mason University (URL: https: //erc. gmu. (edu) and funded in part by grants from the National Science Foundation (Awards Number 1625039 and 2018631). Murong and Ziyu appreciate the funding support from George Mason College of Engineering and Computing.
REFERENCES
Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael Lyu. Towards efficient post- training quantization of pre-trained language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL httpt::t///.... // /ttp:////peeeeeveee
net./ forum? VDRMAAG jjw
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
https::/cpenreet
net wowrurodduuuysssae
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Zefan Cai, Baobao Chang, and Wenjuan Han. Human-in-the-loop through chain-of-thoughh arXiv preprint arXiv:2306.0793 07932, 2023.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint 0rXiv:2305.05/77 2023a.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 2022.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint 20X:::2304.05/228 2023b.
10

---

Published as a conference paper at ICLR 2024
Zhoujun Cheng. Jungo Kasai, and Tao Yu. Batch prompting: Efficient inference with large language model apis. arXiv preprint arXiv:2301.0872.1 2023a.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zeutlemoyer, Noah A. Smith, and Tao Yu. Binding lan- guage models in symbolic languages. In The Eleventh International Conference on Learning Representations, 2023b. URL
net/ fwr.morrmmmmm moiidd1f- wee.dd11111120 H1PV42cbF
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint N.iv:2110.141666 2021.
Luyu Gao. Aman Madaan, Shuyan Zhou, Uni Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL:: Program-aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning. volume 202 of Pro- ceedings of Machine Learning Research, pp. 10764-10799. PMLR, 23-29 Jul 2023. URL https: pittss:/ppron
httppppooccennsseeeee https://proceedings
www. www.piigss.mr.peessee.. wttp//222////// wwwp:///.........2o.2
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transac- tions of the Association for Computational Linguistics (TACL), 2021.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compasitional visual reasoning without training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14953-14962, June 2023.
Joy He- Yueya. Gabriel Poesia, Rose E Wang, and Noah D Goodman. Solving math word problems by combining language models with symbolic solvers. arXiv preprint arXiv:2304.09100 2023,
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398, 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatdield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bow- man, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Ka- plan. Language models (mostly) know what they know, 2022.
Eldar Kurtic, Elias Frantar, and Dan Alistarh. Ziplm: Hardware-aware structured pruning of lan- guage models. arXiv preprint 23Xi::2300.040889 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, 2022. URL https:://pp.
semanticscholer org http://oooooss..02000000
Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Soke Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative rea- soning problems with language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: httt:::://ppnrevieww net forum ? id= IFXTZEXXMT
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Grabes and Naoaki Okazaki (eds.). Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12286- 12312, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18653/ v1/2023.acl acl-n.o..-..-.... URL https: http:p/:///alaaaaooooooo.ooooo/
ww..023..cll1.. 687

---

Published as a conference paper at ICLR 2024
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavat- ula, Prithviraj Ammanabrolu, Yejn Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. In Shirty Conference Neural Information Processing Systems, 2023a. URL. https:// pettps://openreviiee ew. net. forum? dd Rzk3GP HN7
Jianzhe Lin, Maurice Diesendruck, Liang Du, and Robin Abraham. Batchprompt: Accomplish more with less, 2023b.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gen- eration: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 158-167, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P11-1000 URL http:::://aaaaatthhacaa wttp://oyy..rg/11--11111
Yinhan Liu, Myle Out Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Ro(bert)a: A robustly optimized (bert) pre- training approach, 2020.
net http://wwwww.tttttoooooooooooaso S
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023.
Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought it takes net/ 9
two to tango, 2023. URR_htttss///ppenreeee
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine; Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 17651, 2023.
Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. Learning math reasoning from self-sampled correct and partially- correct solutions. In The Eleventh International Conference Learning Representations, 2023. URL
net
OpenAL. Gpt-4 technical report, 2023.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112. 1146, 2021.
Lior Rokach. Ensemble-based classifiers. Artificial intelligence review, 33:1-39, 2010.
Marija Šakota, Maxime Peyrard, and Robert West. Fly-swat or cannon? cost-effective language model choice via meta-modeling. arXiv preprint arXiv:2308. 06077, 2023.
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang. Jordan Lee Boyd-Graber, and Lijuan Wang. Prompting GPT-3 to be reliable. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https::tt/........... http::///ppereviiee. ew. net./ / forum? id 98p5x51 L.5ff
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Neeraj Varshney and Chitta Baral. Model cascading: Towards jointly improving efficiency and accuracy of NLP systems. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Pro- ceedings of the 2022 Conference on Empirical Methods Natural Language Processing. pp- 11007-11021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computa- tional Linguistics. doi: (1118653/1122222emm///p wwweullp-min..... 756. URL https: /atta:://allantholooyyy.
org/2022. emn 1p-main....... 756
12

---

Published as a conference paper at ICLR 2024
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL
net o
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances Neural Information Processing Systems, 2022. URL Forum? Berrm?id=_Vijp _i_ 1000MMSB_BBB__ MeSB_L
net/
Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Diving into the inter-consistency of large language models: An insightful analysis through debate. arXiv preprint arXiv:2305. 2023a.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306. 130.233...1333, 2023b.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05663, 2023.
Li Zhang, Hainiu Xu, Yue Yang. Shuyan Zhou, Weiqiu You, Manni Arora, and Chris Callison- burch. Causal reasoning of entities and events in procedural texts. In Findings of the Associa- tion for Computational Linguistics: EACL 2023, pp. 415-431, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL httpes//aelatttooooyoo/////000
Findings-eacl. 31
Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Qizhe Xie. Automatic model selection with large language models for reasoning. arXiv preprini arXiv:2305. 14333, 2023.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797, 2023.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXhv:2304.0688 2023.
13

---

Published as a conference paper at ICLR 2024
A COST ANALYSIS OF LLM CASCADE APPROACHES
In Section 2.3 we introduced ten approaches to implement the LLM cascade decision maker. As interested in saving the cost of LLM usage, first analyze the token usage of each ap- proach. While different thought representations of the same task example could induce different token usages, the difference is very hard to quantify and also depends on the specific reasoning task. For example, for GSM8K (Cobbe et al. 2021), the PoT representation of the same example is typically shorter than its CoT counterpart, while a reversed comparison observed for DATE and Navigate (bench authors 2023). In addition, different demonstration examples may also consist of different numbers of tokens, which is hard to quantify as well. To provide unified analysis, we assume that every demonstration example for any task, regardless of the thought representation in use, consumes No tokens.
Based on our hypothesis, we conducted a cost analysis, with results summarized in Table [ To give an example, consider comparing the token costs of SPOT-1D-Vort CoT-1D-Vote, the cost of calling weaker LLM is (MC,) + K KCT) x Ni for (OTT-2D-Vote it is (2M + +K, + " x Niok. Here, C.º² and C. are the input and output per-token cost for the weaker LLM, respectively, and M is the number of demonstrations in each prompt. In practice, we set K1 = K₂ = K2p/2, where K2D is the total sample size of two sets of demonstration examples, and the total cost for CoT-2D-Vote and
and CoT-2D- 2D-ve For
then be rewritten to (2MC,, + K2DC.). To C consistent, we set Knn = KID - M x c Similarly, we can get the total sample size of different representations KM The analysis guides us configuring each approach to be cost-compprable" (i.e., leading to similar ("*s). Finally, we note that we do not change the number of task demonstrations (i.e., M) because LLMs are very sensitive to this configuration.
keep the costs of
Method
Certainty Value (e) Definition
Cost Estimation
CoT 1D Vote
Agreement of CoT samples on the majority-voted answer A", = (MC, + KinC.) Nos
EK aay=a=/f
PoT-12-Vate
Agrvement of K. PoT samples the majurity voted answer A", (MCF KiDCT) Nosk
Eis 1 A/
McT-17-Vote
Agreement of K Kaar CaT and 22 PoT samples on the majority-voted 2x(MCT Exger Cx CT) Nuse answer A" = >KK Lap=a=/Katet
COT 2D Vote
Agreement of Kap/2 CoT samplos over 2 sets of task demonstrations on the 2x (MCT Map CH) Nose
majority-voted answer A 8 >>pp 14p /Ka
PoT-2D-Vote
Agreement of Kan/2 PoT samples over 2 sets of task demonstrations on the
2x (MCT' * Og) Nuk
mujority-vute voted unswer A s Eka0 1 - A=/Kan
MoT-2D-Vote
Agreement of Kitor /2 CoT samples on one set of task demos and K K PoT camples on another set of task demos their majority-votes answer A',
2x(MCT+ Kyg Nx
Kaoor 1 / Kooo
CoT CoT-2D--- 2D- 2D-Verisy
Agreement two majority-voted answers At" and Ab" over 2 sets of CoT task demonstrations with Kap examples, 1 Agr
2x (MCT + Eige ) Nuck
Por-2o-Veri.y
Agreement the two mujority voled answers, 14" and and AT" over sets of PoT task demonstrations with Kan examples, 1 =p'
2 (MCT'+ Kyp CT) Nos
MoT-2D-Veričy
Agreement of the two majority-voted answers At" and AS" over one set of CoT and another set of Pot task demonstrations with KMoT examples,
2x + )x
Ay"
MoT-ID-Veridy
Agreement of the two majority-voted answers A. and AL' for CoT and PoT same task demonstrations with Kisor/2 examples,
2x(MCT "xga Nus
Table 1: Investigated approaches and cost calculations. For simplicity, we assume M-shot demon- stration prompting and that every demonstration example has Nook tokens. C.ee is the input price per token and C is the output price per token. To keep consistent costs of using the weaker LLM,
set KMOT = K2D = KID - M x
C
B MAIN EXPERIMENTAL DETAILS
We run each approach two times to reduce variance and report the aver- age results. Since DATE and Navigate do not have a training set, before experiments we sampled shots of examples randomly and annotated them be the second set of task demonstrations (for 2D
Implementation Details
14

---

Published as a conference paper at ICLR 2024
approaches). The remaining examples used consistently across all experiments the test set. We adopt "Python code, return ans" in PoT prompt to let the LLM generate the Python code (Chen et al. 2022) The interpreter used for executing PoT is Python 3.10, with some packages, such to facilitate the execution the generated code.
as Date' TimeTime
Metrics Details We evaluate the methods based on task accuracy and cost efficiency, with lower cost and higher accuracy indicating better performance. For accuracy, when the answer is string, we use exact matching. When the answer number, we relax the evaluation criteria due to po- tential variations in the exact computations carried out by the external interpreter. Following prior work (Chen al. 2022). we adopt the tolerance 0.001. For cost efficiency, we calculate the actual token number based the tiktoken and the total cost for each method and get the relative cost by comparing it with the total cost of GPT 20----CT-SS
C MAIN RESULT TABLES
this section, we show the exact numerical results that are reported in Figure 3
The GSM8K result The DATE result is
in Table 2 The ASDIV result is in Table The TabMWP result is in Table 4
in Table 5 The Navigation result is in Table 6
The CREPE result is in Table The average results
in Table g
Voting Method
CoT-ID-Vort Vote
PoT-ID-Vort Vote
CoT-2D-Von Vote
PoT-2D-Vios
McT-ID-Wes Vote
MoT-2d. Vote
Threshold
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Aec
Cost
Ace
Cost
0.875
0.50
0.896
0.233
0.850
0.179
0.912
0,249
0.872
0.209
0,917
0,235
0.918
0.241
0.35
0.907
0.265
0.861
0.206
0.929
0,309
0.894
0.262
0.943
0.331
0.942
0.334
0.65
0.924
0.356
0.886
0.265
0.945
0.408
0.920
0.342
0,951
0.425
0.952
0.425
0.70
0.929
0.372
0.895
0.292
0.945
0.408
0.920
0.342
0,951
0.425
0.952
0.425
0.941
0.438
0.910
0.346
0.952
0.514
0.933
0.427
0.954
0.528
0.955
0.530
0.90
0.950
0.924
0.410
0.938
0.483
0.600
1.00
0.954
0.660
0.940
0.517
0.956
0.699
0.947
0.565
0.957
0.693
0.957
0.691
Verify Method
CoT:2D. Verify
Verify
Verify
Verify
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
0.926
0.322
0.009
0.311
0.947
0.401
0.949
0.403
Without Cascade
GPT:3-C6T-SC
GPT-S-POT-SC
GPT-4 CoT-Greody
GPT-4
T-Greedy
GPT4 CoT-SC
GPT-4 PoT-SC
Acc
Cost
Age
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
0.842
0.111
0.792
0.078
0.945
0.789
0.942
0.603
0.958
1.000
0.947
0.745
Table 2: Exact numerical results on GSM8k.
hnteer//pppi.rrr/pppiee////LLL
httass//platffomm.ppenaaa wtt.://om/tccceteeeeeeeee
https///peenai.com/prriiit
15

---

Published as a conference paper at ICLR 2024
Voting Method
CoI-ID-Vore
Po Vote
CoT-2D. Vote
PoT-2D-Vote
MoT-ID-Voee
MoT-2d- Vote
Threshold
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
0.40
0.902
0.181
0.872
0.173
0.911
0.200
0.877
0.167
0.913
0.189
0.916
0.184
0,908
0.209
0,878
0.790
0.883
0.188
0.920
0.214
0.55
0.910
0.226
0,882
0.210
0.924
0,262
0.894
0.222
0,930
0.291
0.931
0.290
0.60
0.914
0.243
0.887
0.224
0.926
0,288
0.903
0.251
0.932
0.324
0.933
0.324
0.65
0.917
0.261
0.893
0.230
0.929
0.312
0.912
0.273
0.933
0.346
0.935
0.348
0.70
0.920
0.281
0.897
0.254
0.929
0,312
0.912
0.275
0.933
0.347
0.933
0.348
0.80
0.924
0.311
0.906
0.283
0.930
0.365
0.923
0.328
0,935
0.396
0.934
0.400
0.90
0.927
0.348
0.916
0.318
0.930
0.402
0.927
0.360
0.935
0.430
0.934
0.433
0.930
0.429
0.925
0.930
0.393
Verify Method
CoT-2D-
PoT-2D-Verii
MoT-ID. DeVerff
MrT-2D-Nerns Verify
Cost
Cost
Cost
0.924
0.272
0.908
0.297
0.933
0.357
0.934
0.361
Without Cascade
GPT-3-CoT-SC
GPT-3-POT-SC
GPT-4 Col-Greody
GPT-4
T-Gredy
GPI-4 CoT-SC
GPT-4 PoT-SC
Ace
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Codt
0.887
0.115
0.854
106
0.927
0.742
0.930
0.725
0,933
1,000
0.943
0.927
Table 3: Exact numerical results on ASDIV.
Voting Method
LOT-ID-Vore Vote
PoT-ID- Vote
CoT-2D-Von
PoT-2D-
Vote
McT-ID- Vote
MoT-2d-Von
Threshold
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
0.868
0.141
0.150
0.152
0.892
0.151
0.155
0.922
0.149
0.50
0.880
0.169
0.894
0.177
0.909
0,183
0.897
0.179
0.919
0.193
0.924
0.185
0.35
0.890
0.191
0.899
0.195
0.934
0,273
0.916
0.238
0.951
0.324
0,952
0.329
0,899
0,216
0.905
0.941
0.925
0.288
0.952
0.65
0.910
0.245
0.910
0.236
0.945
0.355
0.935
0.327
0,954
0.409
0.952
0.421
0.70
0.917
0.273
0.913
0.254
0.945
0,355
0.935
0.327
0
0.409
0,952
0.421
0.80
0.927
0.327
0.924
0.292
0.950
0.448
0.946
0.411
0.957
0.484
0.956
0.496
0.90
0.935
0.393
0.934
0.345
0.950
0.513
0.952
0.400
0.955
0.531
0.956
0.348
1.00
0.946
0.535
0.947
0.425
0.950
0.601
0.955
0.524
0.955
0.610
0.958
0.630
Verily Method
CoT-2D. Verify
PdT:2D Veriiy
MoT.
Verify
Mo Verify
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
0.940
0.333
0.920
0.262
0.950
0.342
0.931
0.337
Without Cascade
GPT:3-C6T-SC
GPT-3-POT-SC
GPT-4 Cor-ory
GPT-4 Po T-Greedy
GPT4 CoT-SC
PGT-SC
Acc
Cost
Ave
Cost
Acc
Cust
Acc
Cost
Acc
Cost
Acc
Cost
0.844
0.092
0.849
0.072
0.958
0.819
0.933
0.728
0.961
1.000
0.941
0.849
Table 4: Exact numerical results on TabMWP.
Voting Method
Vote
Pol-ID-Vote
CoT-2D- Vote
PoT-2D Vote
MoT-IDDVoee
MoT-2d-M Vote
Threshold
Cost
Ace
Cost
Acc
Cost
Cost
Acc
Cost
Acc
Cost
0.40
0.737
0.192
0.793
0,205
0.775
0,203
0.829
0.208
0.805
0,207
0.812
0.197
0.50
0.758
0.235
0.806
0.229
0.792
0.240
0.844
0.235
0,821
0.240
0.826
0.225
0.769
0,259
0.872
0,871
0.00
0.779
0,289
0.828
0.285
0.849
0.431
0.877
0.367
0,881
0.417
0.872
0.425
0.65
0.795
0.328
0.337
0.306
0.862
0.489
0.882
0.419
0.889
0.469
0.881
0.484
0.70
0,809
0.357
0.848
0.329
0.862
0.489
0.882
0.419
0.889
0.469
0.881
0.484
0,80
0.835
0.425
0,860
0.378
0.877
0.604
0.890
0.502
0.894
0.569
0.888
0.583
0.90
0.854
0.511
0.877
0.425
0.883
0.666
0.892
0.552
0.895
0.637
0.891
0.654
1.00
0.877
0.657
0.888
0.523
0.888
0.770
0.889
0.625
0.892
0.739
0.892
0.744
Verily Method
CoT-2D- Verify
PoT-2D-Verity Verify
MoT-ID Verrify
DerifyyVenifyy
Ace
Cost
Ace
Cost
Acc
Cost
Acc
Cost
0.850
0.456
0.873
0.392
0.875
0.869
0.459
Without Cuscade
GPT-3-COT-SC
GPT-3-POT-SC
GPT-4 Co1-Gready
GPT-4 Po Por-eddydy
GPT- COT-SC
GPT-4
Ace
Cost
Ace
Cost
Ace
Cost
Ace
Cost
Acc
Cost
Acc
Cost
0.677
0.107
0.770
0.162
0.876
0.750
0.873
1.31
0.886
1.000
0.890
1.402
Table 5: Exact numerical results on DATE.
16

---

Published as a conference paper at ICLR 2024
Voting Method
CoI-ID- Vore
PoT-ID-Vord Vote
CoT-2D-Vor
PoT-2D Vote
MoT-ID- Vore
MoT-2d- Vote
Threshold
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
0.40
0.800
0.132
0.871
0.213
0.898
0.125
0.868
0.192
0.878
0.145
0.880
0.146
D.860
0.134
0.877
0,246
0.898
0.874
0.219
0.153
0.55
U.862
0,140
0,881
0.270
0,913
0,233
0.882
0.261
0,964
0,368
0.964
0.365
0.60
0.864
0.144
0.885
0.293
0.920
0.265
0.892
0.305
0.966
0.402
0.966
0.400
0.05
0.806
0,149
0.889
0.317
0.924
0.282
0.908
0.371
0.968
0.440
0.967
0.437
0.70
0.869
0.159
0.892
0.349
0.924
0.282
0.908
0.371
0.968
0.440
0.967
0.437
0.80
0.876
0.175
0.903
0.432
0.934
0,318
0.926
0.537
0,971
0.545
0.970
0.543
0.90
0.884
0.193
0.920
0.548
0.941
0.339
0.935
0.649
0.973
0.626
0.972
0.624
0.901
0.240
0.942
0.948
0.815
Verify Method
CoT-2D- Verify
PaT-2D-Veriih
MoT- ID. Verify
MoT-2D-Verify
Cost
Cost
Cost
0.929
0,293
0,879
0.208
0.966
0.400
0.966
0.398
Without Cascade
GPI-3-CoT-SC
GPT-3-POT-SC
GPT-4 CoT-Greody
GPT-4 PoG-edyyyy
GPT-4 CoT-SC
GPI-4
Ace
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
0.856
0.128
0.859
0.454
0.977
0,701
0.877
0.966
0.975
1,000
0.878
1.302
Table 6: Exact numerical results on Navigate.
Voting Method
CoT-ID-Vore Vote
PoT-ID-VVot
CoT-2D-Von
PoT-2D- Vote
McT-ID -Dtee
MoT-ld-Van
Threshold
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
0.838
0.188
0.846
0.369
0.874
0.257
0.50
0.843
0.208
0.842
0.485
0.857
0.197
0.855
0.422
0.871
0,295
0.878
0.301
0.35
0.847
0.227
0.846
0,500
0.865
0.241
0.870
0.352
0.833
0.460
0.883
0.471
0.852
0,261
0.846
0.517
0.865
0.869
0.603
0.65
0.856
0,284
0,847
0.537
0.868
0.320
0.869
0.658
0.882
0,573
0.881
0.575
0.70
0.863
0.312
0.849
0.557
0.869
0,369
0.872
0.708
0.879
0.614
0.880
0.620
0.80
0.867
0.374
0.833
0.603
0.868
0.419
0.874
0.754
0.879
0.660
0.878
0.663
0.90
0.873
0.455
0.861
0.683
0.809
0.534
0.874
0.851
0.877
0.760
0.873
0.760
1.00
0.872
0.594
0.868
0.803
0.871
0.621
0.873
0.928
0.873
0.847
0.871
0.846
Verily Method
CoT-2D Verify
PrT:2D Verify
Verify
MoT Verify
Acc
Cost
Acc
Cost
Acc
Cost
Acc
Cost
0.800
0.302
0.874
0.581
0.882
0.534
0.883
0.336
Without Cascade
GPT:3-C6T-SC
GPT-3-POT-SC
GPT-4 CoT-Grrodyy
GPT-4 Po T-Greedy
GPT4 COT-SC
GPT-4PPT-S0
Acc
Cost
Aue
Cost
Acc
Cust
Acc
Cost
Asc
Cost
Acc
Cost
0.820
0.157
0.719
0.226
0.871
0.577
0.876
0.871
0.871
1.000
0.882
1.522
Table 7: Exact numerical results on CREPE.
Voting Method
Vote
Pol-ID-Vote
CoT-2D- Vote
PoT-2D Vote
MoT-ID-Voee Vote
MoT-2d-Wa Vote
Threshold
Cost
Ace
Cost
Acc
Cost
Cost
Cost
Acc
Cost
0.40
0.847
0.167
0.848
0,202
0.872
0,178
0.861
0.193
0.878
0,183
0.883
0.183
0.50
0.857
0.195
0.858
0.231
0.881
0.203
0.871
0.227
0.888
0.216
0.891
0.216
D.864
0.864
0.888
0.00
0.871
0,240
0.871
0.276
0.906
0,312
0.896
0.329
0.927
0,389
0.925
0.391
0.65
0.878
0.263
0.877
0.297
0.912
0.348
0.904
0.371
0.929
0.428
0.92%
0.431
0.70
0.884
0,289
0.882
0.320
0.912
0.354
0.905
0.377
0.929
0.433
0.927
0.436
0,80
0.895
0,336
0,893
0.370
0.918
0.428
0.915
0.465
0.932
0,513
0.930
0.517
0.90
0.904
0.394
0.905
0.436
0.921
0.488
0.920
0.530
0.932
0,576
0.930
0.581
1.00
0.913
0,506
0.918
0.543
0.924
0.567
0.924
0.611
0.931
0.663
0.930
0.567
Verily Method
Col-2D-Verify
CoT-2D-
PoT-2D-Verity Verify
MoT-
Doo--ID-eerify Verify
MoT-2D-Verit
Venify
Ace
Cost
Ace
Cost
Acc
Cost
Acc
Cost
0.905
0.311
0.894
0.925
0.925
0.402
Without Cuscade
GPT-3-C6T-SC
GPT-3-POT-SC
GPI-4 Co1-Greedy
GPT- PoT-edeydy
GPT- COT-SC
GPT-4 PoT-SC
Ace
Cost
Ace
Cost
Ace
Cost
Ace
Cost
Acc
Cost
Acc
Cost
0.821
0.116
0.807
0.117
0.926
0.743
0.905
0.763
0.931
1.000
0.913
1.017
Table 8: Exact numerical results on average across all datasets.
17

---

Published as a conference paper at ICLR 2024
D CASE STUDY
In this section, perform case study learn why MoT can achieve better performance. When CoT makes a mistake, prompting the LLM with another set of CoT task demonstrations often leads to the same mistake. In contrast, PoT more frequently either directly gives correct answer or makes a different mistake than CoT's. Among those cases, 43% fall into the former category that PoT can solve the question, while the remaining 57% fall into the latter that PoT can make different mistake. Here we use CoTI, CoT2 and PoT1 to represent the different prompts. CoTI and CoT2 share the same representation but different demonstrations while CoTI and PoTI share the same demonstration but different representations.
D.1 CASES PoT CAN SOLVE
Here show satisfying that PoT can solve the question. The first shows that the PoT can provide different reasoning paths to avoid the same mistake with CoT and the second case shows that the PoT can avoid the computation errors.
D.1.1 CASE 1
Question: Juan and his brother Carlos are selling lemonade. For each gallon they make it costs 3 for lemons and 2 for sugar. They sell each glass for 0.50 and get 20 glasses per gallon. they made 25 in profit, how much did they spend on lemons?
Gold Answer:
Each gallon sells for total of 10 because 20 0.5
Fach gallon costs to make because 3 + 2
They make off each gallon because 10 5 = 5
They made and sold 5 gallons because 25/5 5 5 They spent 15 on lemons because x 3 = 15
CoT1: Answer: They make 10 each gallon of lemonade because 20 x 0.5 10 They made 2.5 gallons of lemonade because 25/10- 2.5
They spent 7.5 lemons because 2.5 3 7.5
They spent 5 on sugar because 2.5 x 5
They spent 12.5 in total because 7.5 + 12.5
ans = 7.5
CoT2: Answer: They make 10 per gallon because 20 0.5 10
They sold 2.5 gallons because 25/ 25/10- 2.5
They spent 7.5 on sugar because 2.3 2 1.5 7.5
They spent 7.5 on lemons because 2.5 3 = 7.5
ans = 7.5
PoT1:
Python code, return ans
cost_per_gallon 3
Lat___glasee__er_______
20
profit_per_galll - num olasspeaaasss_eeecaalllo * 0.5 -\
cost_per_gallon
total_profit 25
total_gallons total_profit profit_per_gallas
fost_of_lemone total_gallons
ans cost_of_lemons
#ans = 15. (This line is not generated by LLM.)
D.1.2 CASE 2
Question: Robert wants to buy a bookshelf and a sofa and has received two offers. Furniture United's offer includes 1, 350 advance payment and 6 monthly installments of 350 each. Cozy Homes' offer includes a 1, 100 advance payment and 9 monthly installments 250 each. What is the cost difference between the two offers?
18

---

Published as a conference paper at ICLR 2024
Gold Answer: Furniture United's offer is 1,0 (6 x 350) 3, 4,0 Cozy Homes' offer is 100 + (9 x 250) = 3, 350. The difference is 350 3, 350 100
CoT1: Answer: Furniture United's total cost would be 1350 + 6 x 350 3150. Cozy Homes' total cost would 1100 + 250 3350. The cost difference between the two offers is 3350 3350 200. ans 200
CoT2: Answer: The total cost of Furniture United's offer is 1350 + (6 x 350) = 3, 150. The total cost of Cozy Homes' offer is 1100 + (9 250) 350. The cost difference between the two offers 3350 - 3150 200. ans = 200.
PoT1:
1350
= 350
cozy_ homes advance 1100
cozy_homes wttss__oooohhy__ssllleee
homes_advance
250
furniture_united__eee oot_ furniture
paaniture_unitedddom anvance
(6
furniture_united
insta Iment:)
cozy_homes_total__e = cozy_homes_advee
(9
ans
ans 100.0 (This line is not generated by LLM.)
D.2 CASES POT CANNOT SOLVE
In this section, we show some cases that PoT cannot solve but will make different mistake. In case I, the CoT forgot calculate the first 40-yard line but the PoT calculated the first 40-yard but with incorrect grounding value. In case 2, the CoT forgot to remove the lowest score. The PoT removed lowest score but grounded an incorrect number of total tests. can see that although both the CoT and PoT make mistakes, the reasons why they make mistakes different, making the final results different.
D.2.1 CASE
Question: Blake and Kelly are having contest to see who can run the most in 15 minutes. They decide to do it football field that is 100 yards long. Blake runs back and forth 15 times. Kelly runs back and forth once, and then decides that she doesn't want to run next to Blake, so she starts to run to the y0ry line and back. She does this 34 times. How much farther does the winner than the loser?
Gold Answer: Blake 200 yards for each back and forth because 100 2 200 He runs 3000 yards in total because 200 15 = 3000 Kelly runs 200 yards at first because 100 x 2 200 She runs
80 yards each time she does a back and forth the 40-yard line because 40 2 80 She runs 2720 from these back and forth because 34 80 = 2720 She runs 2920 in total because 2720 + 200 = 2920
Blake won because 3000 2920 The winner ran 80 yards more than the loser because 3000 2920 80
CoT1: Answer: Blake runs back and forth 15 times, so she runs 15#100===0000 yards. Kelly runs to the 40-yard line and back 34 times, so she runs 34*80=27000 yards. Thus, Blake runs 3000- 2720=280 yards farther than Kelly. ans 280
CoT2: Answer: Blake runs a total of 15 x 100 x 2 = 3000 yards. Kelly runs 34 x (40 x 2) = 2720
yards. Blake runs 3000 2720 280 yards farther than Kelly. ans 280.
PoT1:
Python code, return ans
sength_os_ffoobblll_ii
100
mun_of_laps_for_bbeer = 15
lim_of_laps_foo____mm = 34
distance balaance_covvreedbyy... blake ofngth_of_footballl__
num_of_laps_ for laaplssffo_bbialee 2
distancee_ooerree_by_elll
[ *\
19

---

Published as a conference paper at ICLR 2024
num_of 1aps for_kelly * 2) (40 num of _i__iap__fee oum_oof_apssfor____ kelly 2) ans = abs wewpancceeeeeeeeeeeeeeeeeeeeeeee by blake - distance_covered by by_kelly) ans 6520.0 (This 1ine is not generated by LLM.)
D.2.2 CASE
Question: Brinley is in Mr. Bert's math class. Mr. Bert gives six tests each semester. He allows the students to remove the lowest score from the average each semester. If Brinley has scores 89, 71, 92, 100 and 86 on the first five tests, what score does she need on the sixth test to get average of 93?
Gold Answer: First, identify the lowest score, which is 71, and remove it. Since Brinley wants score of 93, we know her total score needs to 93 (average) (tests) 465 points So far, Brinley has earned 89 + 92 +100 + 86 367 points The final test must be 465 367 = 98 points.
CoT1: First, we need to find Brinley's current average by adding up her five test scores and dividing by 5: (89 + 71 92 100 86) / 5 = 87.6 We know that Brinley wants her average to be 93, which means she needs to have a total of test scores (since one will be dropped). The total of these 6 scores needs to be 93 6 558 To find out what score Brinley needs the sixth test, we subtract the total of her first five scores from the desired total: 558 (89 + 71 + 92 + 100 + 86) = 120 So Brinley needs to get score of 120 on her sixth test to have average of 93 after dropping her lowest score. ans = 120
CoT2: Answer: Brinley's first five test scores add up 438. To have average of 93, Brinley needs a total of 93 x 6= 558 points for all six tests. To get 558 points after five tests, Brinley needs score 558 438 120 points on the sixth test. 120
PoTI:
Python code, return ans
htt_frrss__fiie_tttss_ssssrrrr ttsttssssccreesssseesssssssssset 89 92 + 100 86 desired_average 93
lowest_score min (89, 71, 92, 100, 86) run_of_best_five_sce
= total_of
-\
Lowest_scora
ans (desired average num of of_tests) sum of best fcores ams = 191. 0 (This line is not generated by LIM.)
E IMPLEMENTATION DETAILS OF EXTERNAL VERIFIERS
Throughout these experiments, our choice of the weaker LLM remains GPT-3.5-turo.. We only per- form our experiment with CoT representation because the finetuned model we utilize is not suitable for understanding Python code.
The input to the verifier can be either the question alone or a concatenation of both the question and the answer produced by the weaker LLM. The question texts include the context, e-g., the goal and context in CREPE. The answer texts contain both intermediate thoughts and the final answer (e.g., an integer mathematical reasoning). We have multiple generation results from the weaker LLM. To avoid excessive input length, we only employ the majority-vote answer and its intermediate thoughts in the test time. Specifically, in the test time, we extract the majority CoT prompts with different demonstrations this setting shows better performance in our method, and then randomly select one collection of intermediate thoughts that result this majority-voter answer.
answer from two
We still use task accuracy and relative cost as the metric. Note that only the question as input doesn't t need the answers from the weaker LLM necessarily. Therefore, the total cost C' (Eq [ for only the question as input becomes:
C= cd + 111-11111ee Irejeea) + + Ireject C
(4)
To use an LLM as the verifier, we have designed two distinct prompts: one for "question diffi- culty prediction", which solely relies the questions (LLM--Q, and another for "answer evalua-
20

---

Published as a conference paper at ICLR 2024
tion", which incorporates both questions and answers (LLM-QA). The difficulty prediction (LIM-Q) prompt entails the LLM's determination of whether a question necessitates redirection to the stronger LLM, based solely the question's 's complexity. If the LLM deems question challenging for dif- ficulty prediction, it triggers forwarding to the stronger I.I.M The answer evaluation (IILMQQA) prompt is to give the L.M both the question and the answer. We then let the L.M rethink the solv- ing steps and generate feedback to evaluate whether the solution is correct. If not, the question will be transferred the stronger LLM. Both prompting methods are implemented with M 8 shots of demonstrations, and we keep the number of "easy" and "hard" questions or the number of "correct" and "incorrect" answers balanced. We set the sampling path as K 20 and temperature T = 0.4 to sample the major answer and then use the self-consistency (SC) to decide the final "difficulty" or "correctness of the answers". The reproducible prompts are the Appendix L Notably, the cost for decision Cd in Eq. is no longer O with an external verifier. For the LLM as verifier, we use GPT-3.5-turbe as it's s much cheaper than GPT-4. We count the number of tokens used for the approaches LLM-Q Q and LLM-Q2 EA.
For finetuning a smaller LLM (RoBERTa-base of Liu et al. (020 as the verifier, we similarly tested two variants, when the input contains only the question (F (Finetuned-O) both the question and the answers (Tinetuned-O split them. The training data for fine-tuning the models come from the original training split of each benchmark dataset. Since DATE does not have training split, we only performed experiments on GSM8k and CREPE. In each training instance, we have K 20 generated responses and we label all of them. We label the generated responses where GPT- 3.- S-CoT-SC correctly predicts the answer positive. Notably, the number of total training cases is the number of sampling paths K = 20 multiplied by the number of questions. To address the data imbalance issue (i.e., most of the training cases are positive), we perform simple under-sampling over the GSM8k. For CREPE, the resulting training set is relatively smaller, we did not perform the under-sampling.
when it includes in which case we use a separation token to
In the training process, we fine-tune RoBERTa-haan with the learning rate 10-5 and batch size 32. We randomly split 10% of the training data to be the development set. The performance of the best accuracy over the development set is shown in Table g One notable approach, predicting "all positive", is to assign positive label to every example in the development set. We could learn that the external verifier could achieve better performance than "all positive" but not significant enough.
Dataset
Method
Dev
All positive
0.5
GSMBk
Based Question
0.592
Based on Question and Answer
0.615
All positive
0.715
CREPE
Based on Question
0.749
Based Question and Answer
0.812
Table 9: Performance of finetuning RoBERTa as verifier over the dev set
Table 10 shows the exact numerical results of those presented in Figure 6 We can learn that the external verifier could boost the performance but not that significant. Comparing LLM-Q with LLM-QA, an interesting observation is that the latter costs less than the former. This may seem counter-intuitive at first glance because LLM-QA includes a much longer input (a concatenation of question and answer) than LLM-Q (only question). However, we found out that for both approaches, their final costs mainly determined by the number of cases transferred to the stronger LLM. That is, when including both the question and the answer as input, LLM QA tends to trust the weaker LLM more frequently, leading fewer cases being routed to the stronger LLM and hence the lower cost. However, both approaches do not yield accuracy as high as our Mbsed approaches.
F LLAMA2-13B AS THE WEAKER LLM
Although GPT-3.5-turbo is considered to be weaker compared to GPT-4, it is still more powerful and expensive than most of the open-source I.LMs. In this section, we tested the performance of
21

---

Published as a conference paper at ICLR 2024
Dataset
Method
Accuracy
Cost
GPT--.--mrbo COT-SC
0,842
0.111
LLM-Q
0.253
LLM-QA
0.862
0.135
GSMRK
Finetuned-Q
Finetuned-QA
0,882
0.262
CoT-2D-Vote threshold=0.7
0.945
0,408
MoT-ID-Vott with threshold-0.5
0.917
0.235
MoT-1D- --te with threshold=0.7
0.951
0,425
Mo'T-1D- Verify
0.401
GPT CoT
0.958
1,000
GPT-3.3-trbb COT-SC
0.676
0.107
LLM
0.750
0.367
DATE
LLM-QA
0.207
0.142
CoT-2D-Voot with Uhresbold=0.65
0.489
MoT-ID-Vote with threshold-0.65
0.889
0.469
MoT-ID-Verify
0.449
OPT-4-TT-SC
0.886
1,000
GPT.3.5-mrb CGT-SC
0.820
0.157
LLM-Q
0.864
0,406
L.M-QQ
0.846
0.169
CREPE
Finetuned-Q
0,323
Finetuned-QAA
0,841
0.322
CoT-2D--ote with Uhreshold=0.55
0.865
0.241
MoT-ID- DDtee with threshold-0.55
0,885
0.460
MoT-ID- Verify
0.882
0.534
GPT-4 CoT-SC
Table 10: Exact numerical results of various external verifiers. Specifically, because DATE does not have a training set, do not report finetuned models' performance on
GSMBK LLAMA2-138
DATE LLAMA2-138
CREPE LLAMA2-138
LaMaz 128 CCT.CC
LaMaa 128 PoT
Accuracy
Task Accuracy
2.80
Tack Accuracy
22-
0.2
McT-1D-2tt
COT Vote
POT2D-W0e
2.10
Col Verly
McT-20-Mte
Pof 20 Wally
3:60
Mol-12-Welly
McT-2D-Welly
1:00
2.4
02
0.4
0.
12
04
3.0
Relative Cost
Relativee Cost
Relative Cost
Figure The performance of using LLAMA2-13B as weaker LLM.
LLAMA2-13B (Touvron al. 2023), open-source and easy-to-deploy LLM, as the weaker LLM and GPT-4 as the stronger LLM.
The result is shown in the Figure 7 Through experimentation, observe that choosing LLAMA2- 13B as a weaker L.LM and applying our strategy on GSM8k and CREPE did not yield ideal results, but performed well on the DATE dataset. On GSM8k and CREPE datasets, accuracy and cost change approximately linearly and we cannot achieve a comparable result with a lower cost, but on the DATE dataset, obtain a curve similar to the curve using GPT-3.5-turbo To analyze the cause, we have selected cases in MoT-1D-Vote whose agreement score (Eq ( is greater than the 0.6, 0.7, and 0.8. We report their task accuracy and proportion among all cases. A lower accuracy in those cases is indicative of reduced overall task performance, while diminished proportion implies an increased cost.
weaker LLM.
The results are displayed in Table 1I Comparing LLAMA2-13B to GPT-3.5-turbo, we observe that LLAMA2-13B exhibits significantly lower proportion. This suggests that LLAMA2-13B is less likely to provide consistent answers across multiple samplings. For example, only 16.3% of the cases have an agreement score greater than the 0.6 threshold on GSM8k in LLAMA2-133 but 77.3% in GPT-3.5-turbo However, the difference in accuracy is slight in GSM8k and DATE. This outcome aligns with our hypothesis that the LLM can consistently answer easy questions but is hesitant to the hard questions. The lower proportions show that the number of easy questions for LLAMA2-13B is
22

---

Published as a conference paper at ICLR 2024
far fewer than that in GPT-3.5-turbo LLAMA2-133 13B cannot solve these complex reasoning tasks well. Therefore, LLAMA2-133 not a suitable choice as a weaker LLM for particular tasks.
Threshold
Dataset
Weak LLM
Accuracy
Proportion
GPT-3.5-turke
0.964
0.773
GSM8K
LLAMA2-133
0.923
0.163
GPT-3.5-turie
0.887
0.721
0.6
DATE
LLAMA2-13B
0.824
0.461
GPI-3.5-ttrbo
0.916
0.722
CREPE
LLAMA2-13B
0.639
0.243
GPT-3.5-turho
0.981
0.707
GSM8K
ILAMA2-13B
0.946
0.093
GPT-3.5-turbo
0.893
0.656
0.7
DATE
LLAMA2-13B 0.893
0.328
GPT-3.5-turbo
0.934
0.655
CREPE
LLAMA2-13B 0.658
0.095
GPT-3.5-turte
0.985
0.639
GSM8k
LLAMA2-13B 1.000
0.070
GPT-3.5-turie
0.900
0.593
0.8
DATE
LLAMA2-13B 0.933
0,203
GPT-3.5 -.ttrbo 0.956
0.588
CREPE
LLAMA2-13B
0.600
0.013
Table 11: The performance of cases where the percentage of the major answer under MoT-1D-Vote be greater than the threshold with LLAMA2-13B as the weaker LLM.
G CAN THE STRONGER LLM BENEFIT FROM THE WEAKER LLM's HINTS?
Dataset
Method
Accuracy
W/o Hints 0.891
GSM8k
W/ Hints 0.867
Wo Hints 0.892
DATE
W/ Hints 0,910
W/o Hints 0.774
CREPE
W/ Hints 0.727
Table 12: The GPT-4-COT-SC accuracy over the instances that A McT-1D-Veri approach
*
A
in the
We conducted an extended experiment to investigate whether progressive hints can further enhance our pipeline. Previous works have proven that progressive hints multiple iterations can enhance the LLM's reasoning performance (Zeeng al. 2023). For example, if the answer is 13 in the first iteration and 14 in the second one, the hint for the third iteration can "Hint: The answers may be close 13, 14" In MoT-1D/ 1D 2D i fy, before the decision maker transfers question a stronger LLM,, we already have two different answers generated by the weaker L.M.. However, the answers from the weaker LLM will be discarded when using the stronger LLM. To incorporate the answers in the leveraging process of the stronger LLM, we follow the prior work (Zheng et al., 2023) to provide these inconsistent answers hints. For example, the main answer 5 with CoT and 12 with РоТ, the hint for the stronger LLM can be "Hint: The answers may be close to 5, 12" We perform the exploration of whether using these answers as hints can improve the stronger LLM's performance. Prompts can be found in Appendix L In the prompts, we acknowledge that the hints may not always be correct. In our exploration, we choose the data that will sent to the stronger LLM by the MoT- -oo--D-Veriff approach. We then assess the GPT 4-oo---o----- C-----C-CC accuracy of these cases, both with and without the use of hints, respectively. We conduct our experiments on GSM8k, DATE, and CREPE datasets.
The results in Table 12 reveal that progressive hints may not necessarily further enhance our pipeline. Adding progressive hints only yields slight improvement the DATE dataset, with diminishing returns on GSM8k and GREPE. The reason is that GPT-4 tends to misled by the incorrect answer, especially when both answers from GPr--35-turbo are incorrect.
23

---

Published as a conference paper at ICLR 2024
accuracy
cost
w/o Batch Prompting
0.947
0.401
w Batch Prompting 0.924
0.325
Table 13: The GSM8k results with and without batch prompting
H CAN BATCHPROMPY FURTHER REDUCE THE COST?
Batch Prompting is method reduce costs by inputting multiple questions into LLM at once (Cheng et al. 2023a Lin et al. 2023b). In this section, we explore whether our method can be used in conjunction with batch prompting. Our experiments are based on the basic batch prompt- ing setup (Cheng et al., (023a). We use MoT-ID-Verist fy for decision making. Specifically, we grouped batch of 4 test questions into each APT call of the weaker L.M,, in addition to the original 8-shot demonstrations. The prompt is shown in Figure 39 Like in our previous experiments, we obtain multiple samples from running the weaker LLM, and the verification-baat baaed method (Eq 3 can then be leveraged independently for each test question. For example, if two of four test instances are rejected by the decision maker, we only feed the rejected two to the stronger LLM. We perform the experiment on the GSM8k dataset.
We show the result in Table 13 The results show that adding batch prompting can indeed reduce costs more. However, using batch prompting slightly affects the accuracy of both the weaker LLM and the stronger LLM. The result indicates that our method is orthogonal to the batch prompting approach.
I CALIBRATION ANALYSIS
In this section, we performed calibration analysis comparing our approaches with baselines. For voting-based method, we take n/ K as the confidence score. For LLM as external verifier, followed the prior works' prompting method (Kadavath et al. 2022) to let the LLM make the deci- sion according to the textual contents FooieowwinFFlliiinnn their setting, i.., T= 1 and T = 2, and donate the variants as LLM-QA T=1 and LLM-QA T=2, respectively. The prompts are shown in Table 31 We also note that, in the original method of Kadavath et al. 2022). the authors used the token probability of an LLM in predicting the answer to be "True" its confidence score. Since we cannot obtain the token probability of GPT-3.5-turbo, the confi- dence score is obtained by sampling the True/False prediction K times and reporting the frequency of "True" (n/K). This will induce significantly more costs than our approach, but we adopted the strategy here for this calibration analysis. We perform the calibration analysis in the GSM8k dataset.
consider two temperature setups,
The calibration results on GSM8k are illustrated in Figure Left). Our major observations are: (1) All decision -making methods yield a monotone calibration curve, implying that when they have higher confidence certain answer, the answer generally more likely to be true. For our three vote-based approaches, this upward trend also supports our initial hypothesis that question is easy the weaker LLM exhibits consistency in its responses across multiple sampling paths; (2) LLM- QA T-2 with larger temperature can lead to the least calibrated decision maker; (3) However, there is no a significant difference among other approaches in terms of their calibration degree (though all variants showed to be better than LLM-QA).
We wanted to note that achieving perfect calibration with n/K K as the confidence score is not nec- essary for our task. To give extreme example, let consider the case that whenever answer has n/ K > 0.5, the answer from the weaker [.LM is correct (accuracy 1.0). In this case, we would have a very poor calibration curve, but pipeline would actually perform very well, because any majority-voter answer it accepts in this case is perfectly true. Another factor to note is the "size" of each bin in the calibration plot (c.g., the number of instances that have confidence scores ranging between 0.6 and 0.7). While this factor will also dramatically impact the LLM cascade performance in practice, cannot be shown in the calibration plot.
'I our experiments in Section 3.5 the I.I.M- QA model is additionally prompted to explain its decision making, hut ignored this explanation following Kadavath KallaaingKadaavahhccommm et al. 2022) for the purpose of calibration analysis.
24

---

Published as a conference paper at ICLR 2024
0.50
08-
0.56
Accuracy
0.6-
Accuracy
0.54
0.52
0.50
0.88
0.2
0.06
0.0
0.2
0.6
2-6
0.8
3.0
0.2
0.4
0.6
0.8
Confidence Score
Confidence Score
Figure 8: The left figure is for the calibration analysis of different approaches. The number of each method is the expected calibration error (ECE). The right figure plots the accuracy of the subset of answers satisfying n/ K greater than the confidence
To give a more direct comparison among different decision-making making approaches, we plot Fig- ure Right), where for each approach we report the accuracy of subset of answers satisfying n/K greater than the confidence score, following the exact way how the approach will be applied in the LLM cascade pipeline. As such, curves on this plot have taken into account the "bin sizes" and can reflect their actual effectiveness. (022). We first observed that as the confidence score grows, there is consistent upward trend in the subset accuracy for all approaches, which echoes the monotone curves the calibration analy- sis. The higher position of MoT compared with other variants and baselines also explains its superior performance in the main experimental results.
similar analysis was also performed by Kadavath et al.)
J CAN OUR METHOD GENERALIZE TO FACTUALITY-BASED TASKS?
In prior studies, the efficacy of our approach in mathematical, symbolic, and causal reasoning has been evidenced. This section explores its potential to be generalized to factual reasoning tasks. Our investigation utilized Strategy (Geva et al., 202T), factoid question answering dataset that requires implicit reasoning over facts. We employed an open-domain question-answering format, presenting only questions without context to the LLM. Our few-shot prompts are shown in Figure 27 and Figure 28
GPT CoT SC GPT PUTSC GPT4-C5T55
Nut-1D
0.75
Accuracy
0.70
0.65
0.60
6.2
0.8
Relative Cost
Figure 9: Performance of MoT-1D-Vote and baselines the dev set of strategyQA.
The result is shown in Figure 9 It shows that our method is still effective on factual reasoning tasks. That is again owing to the fact that using different prompt representations could trigger different reasoning paths, which often results in more trustworthy answers when the two representations agree with each other.
25

---

Published as a conference paper at ICLR 2024
Golden
Question
A curling iron is a tool used to make the hair curly using heat. The sport of curling requires curling brooms, stones (rocks), and curling shoes. curling iron is not necessary in curling.
a curling iron necessary in curling?
PoT
CoT
necessary seuiimmet_ffo_cumng = ["curling stone". "curling broom']
Yes, a curling iron is necessary for curling as is the tool used to create curls in the hair.
equipment_ _eiii__ue__i_ ctapt__udd____uiiiieiee jn_curllng__wwln Burlling_withccrfiinne sathh__urlnggiooo = ["curling iron") ans ("curling iron" in eeneessayyeeuipmment farrttffo_____ererrrrree (orcullinno
Figure 10: An example showing that MoT helps factual reasoning tasks such as StrategyQA.
An example is shown Figure 10 For the question "Is a curling iron necessary in curling?", the golden answer is "No, carling is ice sporí and doesn't II need a curling iron". However, most of the CoT answers are "yes" with hallucinations about the concept "curling". In contrast, most of the PoT answers are "No". The PoT processes typically list the necessary equipment for curling, such as "curling stone" and "broom", and then check "curling iron" on the list. By checking the consistency between CoT and PoT, MoT-1D-Vote is thus able to identify the incorrect or untrustworthy answer. It is worth noting that, judging from the results of Figure PoT is not better than CoT, but the combination of CoT and PoT generates diverse thoughts and answers, instead of leaning towards one kind of thinking, thus reducing errors in factual reasoning. Therefore, we can still leverage consistency checking across MoT prompts in decision- maiinnnn to check if the answer from the weaker LLM is trustworthy in nactual-based reasoning tasks.
K LIMITATIONS AND FUTURE WORKS
Our approach is still subject to some limitations. Firstly, its applicability is confined to tasks where questions have clear answers, such as mathematical reasoning. Secondly, the utilization of our pipeline might lead to increased latency in cases demanding stronger LLM. Thirdly, our method is based on the assumption that the intermediate steps could be expressed via different representations. In addition, if weaker LLM has overconfidence in some incorrect beliefs, our method will fail in these cases.
Considering these limitations, identify some potential avenues for future research. In tackling the initial challenge, we could integrate other metrics, such as semantic similarity, to evaluate the consistency of the general textual generation tasks. To address the latency issuc, it is essential to establish framework for making decisions early for some obviously easy or hard questions. For the third one, believe that how represent the intermediate steps in a specific task relies hu- man designation. Recent research has shown some predefined programming interfaces effectively manage complex tasks, even those that appear unrelated to traditional code generation tasks, such as image understanding (Gupta & Kembhavi 2023). If task's 's intermediate steps cannot be ex- pressed directly via code, we can also introduce diverse answers by leveraging some programming interfaces/tools.
Additionally, due to CoT and PoT being suitable for different tasks, the answer's correctness with different representations varies across tasks, e.g., the CoT answers are more reliable in GSM8k but the PoT answers are more reliable in DATE. The incorporation of learning algorithms to perform weighted voting for specific tasks yields promising improvement.
L FULL PROMPTS
We show the prompts used for this paper according to Table 14 We show one example prompt in each case.
26

---

Published as a conference paper at ICLR 2024
Prompt Type
Dataset
Prompt Table
GSM8K
prompt 15
ASDIV
prompt
TabMWP
prompt 19
CoT
Date
prompt 21
Navigate
prompt
CREPE
prompt
GSM8k
prompl
ASDIV
prompt 18
TabMWP
prompt ??
PoT
Date
prompt
Navigate
prompt 24
CREPE
prompt 26
GSM8k w/ Question
prompt 29
GSM8k w/ Question and Answer
prompt 30
DATE w/ Question
prompt
LLM as External Verifier
DATE w/ Question and Answer
prompt
CREPE w/ Question
prompt 34
CREPE w/ Question and Answer
prompt 33
GSM8k
prompt 36
Learning from hints
Date
prompt 37
CREPE
prompt 38
Table 14: The index of each prompt.
GSM8K CoT
Complete the text, start with 'Anwwer' and the last line starts with
"ans
Question: Manny had 3 birthday cookie pies to share with his 24 + classmates and his teacher, Mr. Keith. If each of the cookie pies were cut into 10 slices and Manny, his elassmates, and Mr. Keith all had piece, how many slices are left?
Answer: There is a total of 3 x 10 = <<3=10=30>>30 cookie slices. There + are 24 + + 1= <<24+1+1=26>>> >6 people who ate the cookie pieces. 4 There is 30 26 <<<3-26<4>>4 cookie slices left. ans =
Table 15: GSM8K COT task demonstrations
27

---

Published as a conference paper at ICLR 2024
GSM8k PoT
Question: Manny had birthday cookic pies to share with his 7 classmates and his teacher, Mr. Keith. If each of the cookie pies were cut into 10 slices and Manny, his classmates, and Mr. Keith all had piece, how many slices left? Python code, return ans num cookie pies =
num slices per cookie pie 10 total-people 24 1 +1 total_slices_ner
total_peoplc
total_slices = num_cookie.pics num wmm_slices.ppr.cooki.... ans totalllsiccss Lotal_slice needed
Table 16: GSM8k PoT task demonstrations
ASDIV CoT
Please answer the math question and return a number the answer.
Question: Isabella 's hair is 18 inches long. By the end of the year her inches long. How much hair did she grow?
hair
Answer: Isabella 's hair is 18 inches long and 24 inches long.
So the length she grow in one year is 24 18 ans 6
Table 17: ASDIV CoT task demonstrations
ASDIV PoT
Question: Isabella 's hair is 18 inches long. By the end of the year her hair is 24 inches long. How much hair did she grow?
Python code, return ans bsabella.hair.beffee before = 18
Isabella hair_aaiees year 24 hair-growth hai.blllaaair.aftte..... ans hair.growth
Seabella.hair.de
Table 18: ASDIV PoT task demonstrations
28

---

Published as a conference paper at ICLR 2024
TabMWP CoT
Read the following table regarding "Coins" and then answer a question:
Name | Number of coins
Braden 76
Camilla 94
Rick
Mary 84
Hector 80
Devin 83
Emily 82
Avery 87
Question: Some friends discussed the sizes of their coin collections. + What is the mean of the numbers?
Explain: Let's think step by step.
The numbers of coins of each one are [76. 94. 86, 84, 80, 83. 82. 87].
So the mean of the numbers is (76699-86--88550-884444---)))) = 648/8 = 81
Answer: 81
Table 19: TabMWP CoT task demonstrations
Strategy QA PoT
Question: Are more people today related to Genghis Khan than Julius 4 Caesar?
# Python code,
children num Julius Caesar =3
ans = Saa.drrr.aummof..nnniiiss... Khan >
=
Table 20: Strategy PoT task demonstrations
DATE CoT
Q:
Today is Christmas Eve of 1937. What the date tomorrow in + MM/DD/YYY??
Explain: Today is the Christmas Eve of 1937. today is 12/24/1937. Today is 12/24/1937. the date tomorrow is 12/25/1937.
A: 12/25/1937
Table 21: DATE CoT task demonstrations
29

---

Published as a conference paper at ICLR 2024
DATE PoT
# Write Python Code solve the following questions. from datetime import date, timedelta from dateutil. relativedelta import relativedelta
# Q: Today is Christmas Eve of 1937. What is the date tomorrow in MM/DD/YYYY?
today is Christmas Eve 1937, then today is 12/24/1937 today = date (1937, 12, 24)
tomorrow
date- today relativedelta (days=1) The answer formatted with % is ans date_tomorrow. strftime "m/%dd/%Y)
Table 22: DATE PoT task demonstrations
Navigate CoT
Following these instructions. if we return to the starting point, return + "yes": : else return 'no'.
Instruction: Take step. Take 2 steps. Take steps. Turn around. Take 4 6 steps. Turn left.
Explain: start-position = [0, 0I, assume the start face is to x-positive
1. Take I step. The current position is [I. 0]
2. Take 2 steps. The current position is [3, 0]
3. Take 3 steps.
current position is [6, 0]
4. Turn around. The face is to x-negative in the following steps
5. Take 6 steps. The current position is [0, 0]
6. Turn left. The face is y-negative in the following steps.
After the steps, the position is [0. 0]. the same as the starting
4 point.
Answer: yes
Table 23: Navigate CoT task demonstrations
30

---

Published as a conference paper at ICLR 2024
Navigate PoT
Following these instructions, if we return to the starting point, return yes"" else return 'no".
def rotate (face direct): face direct = tuple face direct)
mapping dict = {(1, 0): (0. 1), (0. D): (-1, 0), (-1, 0): (0. -1),
(0. -1): (1. 0))
return list (ssapinngg-iic face (ireet))
def right_rotate rotate ( face (acc_direct))
face direct tuple (face_direct) direct)
mapping diet [(1, 0): (0, -1), (0. 1): (1, 0). (-1, 0): (0. 1),
9 (0, -1): (-1, 0))
return list (maapinngg.iin face direec))
def artane face-direct):
face_direct tuple face-
mapping-diet = {(1, 0): (-1, 0), (0, 1): (0, -1), (-1, 0): (1, 0),
9 (0, -1): (0, 1))
return list (mapping diet face direct
def (eveesteps
face-direct, step step.direct="")
new-lix = []
if step-direct == 'left
face_direct left rofttre dfree_)
elif step-direct
'right
firecdirect = right_rotate (face_direct)
elif step.direct = 'backward"; :
face- = around.rolate (face (face_dire) .irrect)
for i in range(len) (e postctionosiii))
new-list. www.iit..ppedccuurtttpooiiiii (current.posiiion
face_direct[i] step)
return nesslist
# Instruction: Take 1 step Take 2 steps. Take 3 steps. Turn around. Take steps. Turn left. Python code. return ans start.position = [0. 0]
current-position
start-position
# assume the start face is to x-positive face_direet [1, 0]
Take step.
current_posile position = move mo (curreet___sstisitioo face 1)
# Take 2 steps.
current.position
steps (apss(cune (current posittion, directi 2)
Take steps.
current position = move steps ccurrent position, face direct, 3) # Turn around.
face_direct- around_ (face_direct)
Take steps.
current position = move move current -osiittoossiioo,... face_ direct, 6)
# Turn left. Now the face is to y-positive
face sire_ddrect = left. facc_directt ans 'yes" if current-position start-position else 'no"
Table 24: Navigate PoT task demonstrations
31

---

Published as a conference paper at ICLR 2024
CREPE CoT
Answer the question. The final answers are "more likely", "equally 4 likely" " and "less likely" "
Goal: Wash sneakers
Current Context: Brush off dirt from the surface of the sneakers. Remove shoelaces. Rinse the shoelaces in soapy water and air dry. Apply mild detergent to the sneakers and rub gently. Question: What's s the likelihood that The sneakers are damp? Explain:
For step 1, after brushing off dirt, the sneakers likely" because there is no change for the dampness of the shoes. For step 2, after removing shoelaces. the sneakers are damp "equally + likely" because there is no change for the dampness of the shoes. For step 3, after rinsing the shoelaces in soapy water and air dry, the sneakers are damp is "equally likely" because dampness
damp is "equally
removed the
shoelace from the sneaker so we don' change sneaker.
For step 4, after applying mild detergent to the sncakers and rubbing gently, the sneakers are damp is "more likely" " because put mild detergent to the sneakers.
Therefore. in the final step the event "more likely". Answer: more likely
Table 25: CREPE CoT task demonstrations
32

---

Published as a conference paper at ICLR 2024
CREPE PoT
Answer the question. The answers are "more likely", "equally likely" 4 and "less likely".
# Goal: Wash sneakers
# Cortent,Contex:: Brush off dirt from the surface of the sneakers.
Remove shoelaces. Rinse the shoelaces in soapy water and air dry. Apply mild detergent to the sneakers and rub gently. # Question: What's s the likelihood that The sneakers are damp?
# Python code
class Wash_Sneakers
# Init from Current_ Contexxt
Brush off dirt from the surface of the sneakers
# Remove shoelaces
# Rinse the shoelaces in soapy water and air dry # Apply mild detergent to the sneakers and rub geatly. def -.init...self).
evento = None # evento is the likelihood that The sneakers 4 are damp.
def ofush-off.dirt
# After brushing off dirt, eventO becomes "equally likely" self eventO = "equally likely"
def remove shoelaces
# After removing shoelaces, eventO becomes "equally likely"
self eventO = "equally likely"
def rinse weaeeseseelaeee...tteerrrrr After rinsing in water, evento becomes "equally likely" self eventO = "equally likely"
def apply_ (eergent_sncaker(seee)]
# After rinsing in water, eventO becomes "more likely"
self falfevver eventO "more likely"
def fallsall.func..nnoorddd (relr:
self brush_off ( self reovee...ssshol
self wwwse..hoollace.iii...... w ()
self aaw.yayddeeeerggttsssseess ()
= Wash Sneakers ()
c. call all funes in order () ans evento
Table 26: CREPE PoT task demonstrations
Strategy QA CoT
Question: Are more people today related to Genghis Khan than Julius Caesar?
Explain: Julius Caesar had three children.
Genghis Khan sixteen children.
The children number of Genghis Khan is larger than Julius Caesar. Answer: Yes
Table 27: StrategyQA CoT task demonstrations
33

---

Published as a conference paper at ICLR 2024
StrategyQA QA PoT
Question: Are more people today related to Genghis Khan than Julius Caesar?
Python code, return ans children num of Julius Caesar 3 children num of Genghis Khan =
children Chilbee.
Table 28: StrategyQA PoT task demonstrations
GSM8K Question Difficulty Prediction
Predict the hardness level of the questions.
Question: Shawna workout goal is 30 situps. On Monday, Shawna only able to 12 situps, so she decided that
would make up for the rest on Tuesday. However, she was oaly able to do 19 situps on
4 Tuesday. How many situps would Shawna have to do on Wednesday to meet her minimum goal and make up for the ones she didn t do?
Level: Hard
Table 29: GSM8k question difficulty prediction task demonstrations
GSM8k Question and Answer Difficulty Prediction
Generate feedback and predict if the generated answer is trustful. Question: Shawna s workout goal is 30 situps. On Monday, Shawna was only able to do 12 situps, so she decided that she would make up for
the rest on Tuesday. However, she was only able to do 19 situps on Tuesday. How many situps would Shawna have do Wednesday meet her minimum goal and make up for
ones she didn do?
Answer: Shawna s goal is 30 situps and she has already done 12 19
4 <<12+19-31>>31 situps. nToo meet her goal, she needs to do 30 - 31
= <<30-31=-1>>-11 situps. nSince she can i do negative situps she doesn 't need do any more situps Feedback: The answer incorrect. The goal 30 situps everyday. + the total is 3+30 = 90 rather than 30.
meet her goal. nans 0
Trustful: No
Table 30: GSM8k question and answer difficulty prediction task demonstrations
34

---

Published as a conference paper at ICLR 2024
Calibration
Question:
Shawna's s workout goal is 30 situps. On Monday. Shawna was only able do 12 situps,
she decided that she would make up for the rest on Tuesday. However, she was only able to do 19 situps on Tuesday. How many situps would Shawna have to do on Wednesday to meet her minimum goal and make for the ones she didn't t do?
Proposed Answer: Shawna's s goal is 30 situps, and she has already done 12 "ctups.Antl meet her goal, she needs to 30 - 31 <<30-31=-1>>-1 situps. nSince she can 't do negative 4 situps she doesn need to do any more situps to meet her goal. nans 0
19 <<12+19=31>>31
Is the proposed answer:
(A) True
(B) False
The proposed answer is: B
Table 31: Calibration task demonstrations
DATE Question Difficulty Prediction
Predict the bardness level of the questions.
Question: : Today is Christmas Eve of 1937. What is the date tomorrow in MM/DD/YYY?
Level: Easy
Table 32: DATE question difficulty prediction task demonstrations
DATE Question and Answer Difficulty Prediction
Generate feedback and predict if the generated answer is trustful.
Question: Today is Christmas Eve of 1937. What is the date tomorrow in + MM/DD//YYY?
Explain: Today is the Christmas Eve of 1937, today is 12/24/1937.
Today is 12/24/1937, the date tomorrow is 12/25/1937.
A: 12/25/1937
Feedback: : The answer is correct.
Trustful: Yes
Table 33: DATE question and answer difficulty prediction task demonstrations
35

---

Published as a conference paper at ICLR 2024
CREPE Question Difficulty Prediction
Predict the hardness level of the questions.
Goal: Making Matcha Green Tea the Traditional Way
Current.Context: Boil 3/4 cup (180 ml) of water and pour it into your matcha bowl. Whisk the hot water with chasen. Discard the hot
water from the bowl. Scoop tsp (1.5 heaping teaspoons or 10 g) 4 of matcha into a fine mesh strainer. Sift the matcha into your 9 emply. dry bowl. Pour boiling water into a leacup. Add small amount of hot water into the matcha bowl and whisk it. likelihood that drink the matcha?
Question: What's
Level: Easy
Table 34: CREPE question difficulty prediction task demonstrations
CREPE Question and Answer Difficulty Prediction
Generate feedback and predict if the generated answer is trustful. Goal: Boating in lake
Current.Context: Rent boat. Find a lake which allows boating. Drive to + the lake and set the boat in lake.
Question: What's s the likelihood that the boat is in lake?
Explain: For step 1, after renting boat, that the boat is in the lake "less likely" because the boat is not yet in the lake For step 2, after finding lake which allows boating.
the boat is + in the lake is "equally likely" because we have found a lake but + the boat is not yet in the lake.
the lake and setting the boat in the lake that the boat is in the lake is "more likely" because have set + the boat in the lake. Therefore in the final step the event is "more likely".
For step 3, after driving
Feedback: The answer is correct.
Trustful; Yes
Table 35: CREPE question and answer difficulty prediction task demonstrations
GSM8k Hints
Given the hints (may not be correct), answer the question..
Question:
Tara has shoebox that is 4 inches tall and 6 inches wide. + She puts square block inside that is inches per side. How many square inches of the box are left uncovered?
Hints: The answer may near to
Answer: The shoebox is 24 square inches because 4 x <<446=24>>24
The block is 16 square inches because 4 x 4 <<4x4=16>>16
There square inches uncovered because 24 16 <<24-16=8>>8
Table 36: GSM8k with hints task demonstrations
36

---

Published as a conference paper at ICLR 2024
DATE Hints
Given the hints (may not correct), answer the question.
Question: Jane was born on the last day Feburary 2000. Today 7 her 16-ycar-old birthday. What is the date 24 hours later in
+ MM/DD/YYYY?
Hints: The answer may be
02/29/2016
03/01/2016.
Answer: Jane was born on the last day of Feburary in 2000, so she + on 02/29/2000.
Today is her 16-year-old birthday. So today is 02/29/2016.
The date 24 hours later than today is 03/01/2016.
03/01/2016
Table 37: DATE with hints task demonstrations
CREPE Hints
Given the hints (which may not be correct), answer the question with "more likely", "equally likely" "less likely "
Goal:
Buy flight ticket from SF to Hawaii
Context: Open google flights and check the flights availability during + the dates wanted. Choose non-stop flights and check if the time slots work. After deciding right time slots and price, proceed to reservation
buy tickets. Got an email confirmation
Question: Jane was born on the last day Feburary
2000. Today 9 her ye-year-old birthday. What is the date 24 hours later in 4 >MM/DD/YYYY?
Hints: The answer may be near "more likely" "less likely".
Explain: For step after checking flights availability that can 4 refund the ticket if it is refundable is "equally likely" because I1 haven purchased the ticket yet.
For step 2, after choosing non-stop flights and checking if the time slots work, that "equally likely" because haven' purchased the ticket yet. For step 3, after deciding on the right time slots and price and + proceeding to buy tickets. that can refund the ticket if it is refundable is "less likely" because have already purchased the ticket.
refund the ticket if it is refundable is
For step 4, after getting an email confirmation of reservation that 4 can refund the ticket if it is refundable is "more likely" because I I can check the terms and conditions of the ticket and see if it is refundable
not.
Therefore. in the final step the event "more likely". Answer: more likely
Table 38: CREPE with hints task demonstrations
37

---

Published as a conference paper at ICLR 2024
Batch Prompting
Question robe takes 2 bolts of blue fiber and half that much white 4 fiber. How many bolts in total does it take?
Question2: Josh decides to try flipping
house. He buys a house for $80,000 and then puts in $50 in repairs. This increased the value of the house by 150%. How much profit did make?
Question3: Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to
help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed.
the afternoon, she gives her chickens another 25 cups of feed. How many cups of feed does she - need to give her chickens in the final meal of the day if the size
of Wendi' s flock is 20 chickens?
Question4: Kylar went the store to buy glasses for his new apartment.
One glass costs S5, every second glass costs only 60%
+ price. Kylar wants to buy 16 glasses How much does he need to pay for them?
:nswerl: takes 222<<11211>11 bolt white fiber nSo the total 4 amount of fabric is 2<l=<<2#1=3>>3 bolts of fabric ans 3
Answer2: The cost of the house and repairs 50,000+50., 1..00000,,000$<450 value of the house by
increased the
120, (22,000)mm. (00))nSo nSo
4 the new value of the house is
4 120, 220,000+80.000 -0000000000000000000 00000:50000000000000000000000000
he made
profit of 1w0.000-1..000=33=000000-----110
ans 70000
Answer3: If each chicken eats 3 cups of feed per day. then for 20 cups of feed + day nIf nIf she feeds the flock 15 cups of feed in the morning, and 25 cups in the afternoon then the final meal would require cups of chicken feed.
+ chickens they would need 3<20=<<<<20=60>>>>>
60-15-25=<<<00----500002-
ans = 20
Answer4: The discount price of one glass is 60/100 * 5
5<<<60/100<53>>>3.aa1 every second glass is cheaper, that Kylar is going buy 16 / <<16/==>>>8 cheaper glasses. nSo for the cheaper glasses. Kylar is going to pay 8 3
for the -price- glasses. Kylar will pay 4 8 5 S<< (SS in total Kylar needs to pay 24 40 $<<24+40=64>>64 for the glasses he wants to buy.
+ B<<B<<3=25>>4AAmAu
ans
Table 39: Batch Prompting task demonstrations
38


Reviews:
[{"review_id": "RO60IgPmrU", "reviewer": "Reviewer_ijvt", "summary": "This work introduces an interesting cascading approach to reduce cost in LLM inference. They devised a method: for easy questions, it will use a cheaper LLM (GPT-3.5). But for really hard questions, we'll use the expensive, stronger LLM (GPT-4). This method consists of a weaker LLM, a stronger LLM, and a decision maker, and they reduced the cost to 40% of the cost in using a stronger LLM for everything. To decide which LLM to use, they check if the simpler version gives consistent answers every time they consider its answer. If it does, the question is probably easy, and they stick with the weaker LLM. But if the answers are all over the place, it means the question is tough, and they switch to a stronger LLM. They tried 10 different strategies using Chain of Thought, Program of Thought, mixture of Thought along with majority vote, and verification-based decision making.to find the optimal way to reduce cost while ensuring equal or better performance.", "questions": "* Instead of just the answer as a hint, what if we give the entire CoT or PoT from one of the prompts as a hint? Will that help?\n* what if we ask multiple questions at once? won't we reduce the cost more?. (2/3 Questions with context as prompt)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* They used unique ways of prompting for better decision-making, especially sampling from different in-context demonstrations and thought representations.\n* In-depth analysis of which strategy worked better and why. Evaluating consistency, robustness, and comparisons to other fine-tuned models gives a deeper understanding of how LLMs work.", "weaknesses": "I haven't found any major weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LKKEIfSsFN", "reviewer": "Reviewer_BfsR", "summary": "This paper addresses the problem of robust and cost-efficient question answering using LLMs. To reduce the cost of accurate question answering, the paper proposes to estimate a weak LLM's uncertainty about its answer, to decide whether to accept the answer or reject it and instead ask a strong (but more expensive) LLM. The paper comprehensively evaluates 10 different approaches to the \"routing\" task, and compares the proposed approach to several baselines. The experiments show that significant cost savings are possible without compromising on task accuracy (relative to always using the strong LLM).", "questions": "* How exactly does the MoT-2D setting work? There are now four prompts, rather than two. In the voting setting, this poses no additional problems, but what about the verification setting? Do all four prompt settings have to agree? Or are multiple prompts \"pooled\" when computing two vote-based answers to compare for verification?\n\n* If temperature 0.8 yields better results (Fig. 5), why is this not your default? Did you try increasing the temperature further (e.g. Temperature 1)?\n\n* In Figure 4, what threshold was used to decide whether answers were consistent or not?\n\n* Why do you think QA-based external verification with GPT-3.5 performed poorly? Does it incorrectly validate many incorrect answers as trustworthy? Have you tried increasing the temperature of the QA-based verifier, to understand the actual distribution the model places on \"yes, trustworthy\" vs. \"no, not trustworthy\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "I found the paper to be of high quality and clarity. Specific strengths include:\n\n* This is a clearly written paper that proposes and comprehensively evaluates a simple technique for reducing the cost of question answering with language models. \n\n* The empirical evaluation is impressively thorough, comparing to many interesting baselines. \n\n* The paper surfaces several interesting ideas, e.g., that the sampling distribution of an LLM alone may be insufficient for evaluating how uncertain it is, but by varying the prompting strategy, it is possible to get a broader distribution over LLM answers (which may more accurately reflect the LLM's uncertainty over the correct answer).\n\n* The paper does not overclaim: it honestly represents itself as a careful empirical study of the value of a particular approach to rational cost-aware decision making in the LLM Q&A setting, and does not overstate its novelty w.r.t. related work.", "weaknesses": "1. The evaluation reports \"end-to-end\" accuracy of the entire cascade under different experimental settings, but does not perform a finer-grained analysis of a key novel component: the uncertainty quantification via sampling. It would be great to see some form of **calibration analysis**: in the vote-based methods, how calibrated is the distribution over sampled answers? That is, for each number 1 <= n <= K, how often are the answers that receive n votes actually correct answers? In a perfectly calibrated model, n/K of the answers receiving n votes (across the entire dataset) would be correct answers. Even without perfect calibration, it is interesting to see if the calibration plot is at least monotone: do answers that receive more votes have a higher probability of being correct? It would be great to see how calibration varies across the various vote-based sampling procedures, and perhaps across different LLM temperatures. \n\nSuch analyses would contribute new evidence on important scientific questions surrounding language models, like the extent to which LLMs \"know what they don't know\", and how this uncertainty can best be quantified. For example, the paper \nhttps://arxiv.org/pdf/2207.05221.pdf reports that explicitly asking an LLM to evaluate the truthfulness of a proposed answer yields a calibrated distribution over the tokens True and False. Does the present paper's \"External Verifier - QA\" setting provide contrary evidence? To evaluate this, it would be helpful to see the calibration of the External Verifier compared to the calibration of the methods this paper proposes. (Also, it would likely be necessary to set the temperature higher than 0.4 -- the other paper reports calibration for temperature 1.0 for base language models, and temperature 2.5 for RLHF-tuned models.)\n\n2. Cost is measured based on the actual cost of using GPT-3.5 and GPT-4. This is not unreasonable (and is the exact calculation that many potential users of this framework might wish to do), but the lack of transparency around OpenAI's pricing model, and how it relates to the actual costs of running strong and weak models, makes it harder to interpret the paper's results. I don't think it's necessary for acceptance, but it would be nice to see whether the results from the paper still hold up when using e.g. Llama 2-7b vs. 70b variants, for some replicable measure of cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EWJfrkxiey", "reviewer": "Reviewer_6oKd", "summary": "This paper introduces the notion of \"model cascades\" which is a method to offload easier problems to weaker models, which saves costs. They propose a simple method: answer consistency of the weaker LLM. Intuitively, this just means when the weaker llm is inconsistent, offload the task, because the model is uncertain. From there, the \"stronger\" llm performs inference on the task to solve it. They find this method that increase performance while decreasing cost by a significant margin.", "questions": "- For tasks that require a specific piece of knowledge are the ever situations where the weaker llm is confident, though incorrect, which causes the task not to be allocated to a more accurate & powerful model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is simple and understandable. It's also quite clear why this would help performance -- namely that in highly uncertain situations, samples from an LLM are likely to diverge and lead to overall increased entropy -- so contributing these cases to a stronger model is likely to lead to improved performance.\n- The evaluation is extremely comprehensive. I appreciate the breadth of evaluation across different reasoning tasks\n- The extended results in 3.6 are quite interesting as well -- clearly stating the limitations around how weak the weak llm can be is useful", "weaknesses": "In general, it's clearly stated throughout the paper that this method is aimed at \"reasoning tasks\" which indicated focus on datasets like gsm8k or big bench hard -- where the model must reason or understanding challenging problems. Nevertheless, I'm a bit concerned about how well this method would generalize to factuality based tasks or tasks that concern reasoning about facts/knowledge. In these situations it may be the case the model is highly confident (though it is incorrect) about a few pieces of knowledge which causes it to fail to reason correctly. Understanding that this paper is mostly about reasoning tasks, I'm still a bit concerned about how this method could be limited by the overconfidence in incorrect knowledge, and I believe it could be useful to evaluate this potential limitation to better inform readers about how this method may be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)