A Comprehensive Evaluation of Tool-Assisted Generation Strategies
Authors: Alon Jacovi
Avi Caciularu
Jonathan Herzig
Roee Aharoni
Bernd Bohnet
Mor Geva
Venue: EMNLP 2023 Findings (2023)
Keywords: tool-assisted
tool-augmented
tool usage
large language models
LLMs
few-shot
TL;DR: A systematic procedure for evaluating few-shot tool-assistance methods in LLMs + empirical results

Abstract: A growing area of research investigates augmenting language models with tools (e.g., search engines, calculators) to overcome their shortcomings (e.g., missing or incorrect knowledge, incorrect logical inferences). Various few-shot tool-usage strategies have been proposed. However, there is no systematic and fair comparison across different strategies, or between these strategies and strong baselines that do not leverage tools. We conduct an extensive empirical analysis, finding that (1) across various datasets, example difficulty levels, and models, strong no-tool baselines are competitive to tool-assisted strategies, implying that effectively using tools with in-context demonstrations is a difficult unsolved problem; (2) for knowledge-retrieval tasks, strategies that *refine* incorrect outputs with tools outperform strategies that retrieve relevant information *ahead of* or *during generation*; (3) tool-assisted strategies are expensive in the number of tokens they require to work---incurring additional costs by orders of magnitude---which does not translate into significant improvement in performance. Overall, our findings suggest that few-shot tool integration is still an open challenge, emphasizing the need for comprehensive evaluations of future strategies to accurately assess their *benefits* and *costs*.

A Comprehensive Evaluation of Tool-Assisted Generation Strategies
Alon Jacovi
Avi Caciularu
Jonathan Herzig²
Roee 2haroni Bernd Bohnet
Mor Geva²
'Bar Ilan University
Gooole Research
Google DeepMind
al alonjacovi@gmail.com
Abstract
Google search (Gao al., 2023a; Press et al., 2023; Nakano al., 2022), Wikipedia search (Trivedi et al., 2022a), a calculator (Schick et al., 2023), or a python interpreter (Paranjape et al., 2023). Often, tool invocations are structured as Chain-offThoughh (Co'T) long-form answers (Wei et al., 2023).
growing area of research investigates aug- menting language models with tools (e.g.. search engines, calculators) to overcome their shortcomings (e.g., missing or incorrect knowl- edge, incorrect logical inferences). Various few-shot tool-usage strategies have been pro- posed. However, there no systematic and fair comparison across different strategies, or between these strategies and strong baselines that do not leverage tools. We conduct exten- sive empirical analysis, finding that (1) across various datasets, example difficulty levels, and models, strong no-tool baselines compet- itive to tool-assisted strategies, implying that effectively using tools with in-context demon- strations is difficult unsolved problem; (2) for knowledge-retrieaal tasks, strategies that refine incorrect outputs with tools outperform strate- gies that retrieve relevant information ahead of or during generation; (3) tool-assisted strate- gies expensive in the number of tokens they require to work- incurring additional costs by orders of magnitude- which does not translate into significant improvement in performance. Overall, our findings suggest that few-shot tool integration still an open challenge, empha- sizing the need for comprehensive evaluations of future strategies to accurately assess their benefits and costs.
Recent work proposed a variety of strategies for interfacing between the LM and the tool, such through demonstrations of API calls (Paranjape et al.. 2023) using the tool to refine the model's output (Gao al., 2023a) see Figure for an overview. But what are the advantages and trade- offs of different TA strategics? For example, some strategies incur significantly higher computation costs than others with little to no improvement in performance. There is gap in the literature on the evaluation of such strategies, in particular against strong baselines and against each other. Concretely, works that report empirical evaluations are often restricted comparisons of single pro- posed strategy against a limited selection of non-Th baselines, using a limited selection of LMs or even a single LM, or focus on evaluating various LMs with a specific TA strategy (Li et al., 2023). Ad- ditionally, comparisons often do not consider the increase in computation that each TA strategy quires, which vary significantly, and have large effect inference time or cost.
1 Introduction
The above issues are only some of the pitfalls we observed in the literature, limiting the scope of current evaluations. In $3, we analyze the literature for common pitfalls and collect a set of guidelines towards a fair and reliable evaluation procedure specifically for TA strategies. Next ($4), we con- duet a study which addresses all of the observed pitfalls, using GPT3, Flan-UL2 and FlaM-PaLM, and complex reasoning benchmarks StrategyQA. MuSiQue, GSM8K, and DROP. We report fair, systematic comparison of five few-shot TA strate- gies across multiple models and demonstrations, and all strategies use the same set of tools.
Augmenting language models (LMs) with tools has been proposed to overcome LMs' inherent weak- nesses (Mialon et al., 2023; Qian et al., 2022), such as the lack of grounding to reliable or up-
dated sources (Jiang al., 2023), incoherent logi- cal ability (Liu et al., 2022; Ling et al., 2023) and arithmetic ability (Gao al., 2023b), among others. This is done through tool-assisted (TA) generation, where LMs are trained or instructed to use external tools, such as search engines over the web-e.g..
Work done during an intersshi at Google Rescarch.

---

(a) Teol-assietance during generation
(b) Tool-assietance after generation
integration is still an open challenge. We call the community to evaluate future strategies systemati- cally, while taking into account the significant costs that these strategies require in comparison to their benefits. Towards this, we provide a set of concrete guidelines for fair and reliable evaluation of TA strategies. Moreover, We release the handcrafted collection of 184 demonstrations used in our study (attached in the supplementary material).
Tool demonstrations question
CoT demonstrations question
tool call
O
generated
tool output
to0l output
answer refinement
answer generated
o,0
Tool-Assisted Language Models
answer generated
O,O
We describe existing few-shot strategies for aug- menting LMs with tools and discuss related work.
Figure I: Illustration of tosl-assistance strategies that invoke tools and insert their outputs into the prompt (a), and strategies that first generate some output, and only use tools to fix and refine it (b).
2.1 Few-shot TA strategies
Strategies for tool usage can be broadly divided into two categories: (a) Using tools during genera- tion and insert the tools' outputs into the model's 's prompt (Figures 1a, 2a): (b) Using tools to refine the LM's output after generation (Figures 1b, 2b). Strategies can be further categorized into settings where the tool is heuristically called in pipeline or called when the model generates pre-specified tool calls. Refer Mialon al. (2023) for a review of the literature on TA strategies and models.
We analyze the study results ($5) and arrive at surprising conclusions: (1) Non-TA baselines are stronger than initially reported. In most cases, TA strategies do not significantly or at all improve on non-TA strategies on popular Question Answering datasets. (2) For retrieval tools in knowledge tasks, TA strategies that fix model output after it is gener- ated perform better than TA strategies that prompt the model to interface with the tool directly dur- ing generation. For calculator tools in calculation- intensive tasks, the relationship is not decisive. (3) TA strategies incur significantly higher computa- tion costs than non-TA baselines by multiplicative factors, and there is no general correlation between computation cost and performance, with the excep- tion that refinement strategies in retrieval settings are more costly than non-refinement strategies.
Among TA strategies of type (a): SelfAsk (Press et al., 2023) decomposes the task into subtasks as simpler questions. such that tool can be called on each question. A related strategy is Demonstrate- Search-Preddis (Khattab et al., 2023). Inline strate- gies such Toolformer (Schick et al., 2023)!, ART (Paranjape et al., 2023), inter alia (Chen al., 2022: Gao al., 2023b; Lyu et al., 2023) demonstrate tool usage with pre-defined words or tokens and tool arguments, halt generation when those tokens and arguments are generated, invoke the tool, and insert its output into the prompt to resume genera- tion. Interleaving Retrieval (Trivedi et al., 2022a) does not directly instruct the model to use tools, but calls the tool each reasoning step, pro- vide the model with additional context for future steps. (Jiang et al., 2023) propose similar strat- egy, opting to re-write each step after using it as a query. There are also strategies such as Decom- posed Prompting (Khot et al., 2023) that are gen- eralizations of the previous strategies.
In $6 we report a fine-grained analysis of the results. We investigate the effect of each example's s difficulty-e.g. very large numbers, or very rare entities) on improvement from tool usage, and find that tools do not systematically improve model per- formance on harder examples, where they were ex- pected to have the strongest improvement. Finally, based on an error analysis of failure cases, we find that the majority of mistakes follow incorrect tool invocations, rather than incorrect tool responses (in the case of the retrieval tool) or incorrect inferences based on correct tool usage.
Among TA strategies of type (b): RARR (Gao et al., 2023a) involves a pipeline designed for knowledge-baad based tasks: verifying the relevance
In conclusion, we conduct an extensive evalua- tion of few-shot TA strategies, finding that previous estimates of tool-usage performance is not repre- sentative. Overall, this suggests that few-shot tool
"Shiikk et al. primarily discusses tool usage with training. We adapt only the few-shot strategy in our experiments.

---

Who lived longer, Muhammad Ali or Alan Turing?
SelfAsk
Inline
Interleaving
Are follow up questions needed here:
Search(How old Muharmad Ali when he died?)
longer,
Muhammad All or Alan Turing?
up: old wass Muhammad Ali when he died?
Muhammad Ali died at the age of 74.
Document Muhammad died the age of 74,
Muhammad Ali was 74 years old when he died.
Intermediate answer: Muhammad All died at the age
Document 2: Alan Turing was only 41 took
Search(How old was Alan Turing when he diec?) >>
up: when he died?
Turing
Muharmaad All was 74 years old when he died.
Turing was only own life.
Intermediate Alan Turing was only 41 when he took his own life.
Turing was 41 years old when he died.
Alan Turing 41 years old when he died.
Check & Fix
RARR
Muhammaad Ali 40 years old when died. Alan
Muhammad
years old he diad. Alan Turing 41
Turing was 41 years old when he died. So the final answer Muhtammd Ali
years old when he died. So the final answer Muhammmad Ali
Search(Muhammad
years when he
How old Muhammad when he died? Search(How old was Muhammed All when he died?) >>
(b)
Muhammad All died at the age of 74,
Muhammad diod the age of 74.
Corrected: Muhammad All 74 old when
Relevant, corrected: Muhammad Ali 74 years old when he died.
Q2: How oid was Alan Turing when he died? Search(How Alan Turing when died?)
Search(Alan Turing was 41 years old when he died.)
Alan Turing was only 41 when he took his own life.
Alan Turing only 41 when he look his life.
Correct: Alan Turing was 41 years old when he died.
Relevant, correct: Alan Turing was 41 years old when he died.
SelfAsk Baseline
Inline Baseline
Chain of Thought Baseline
follow up questions needed here:
Search(How old was Muhammad All when he died?) >> Muhemmad All was he Muhammad All was 74 years old when he died. Search(How old was Turing he died ?) > Alan Turing was 41 years Alan Turing was years old when he died.
Muhammad All 74 years old when he died. Turing was 41 years old when he died.
Yes. Follow up: How old was Muhammad All when he died?
(c)
years
Intermediate answer: Muhammad died the age of 74.
Follow up: How was Alan Turing when died?
Intermediate answer: Alan Turing only when own life.
So the final answer is: Muhammad All
Figure 2: Overview of the TA strategies implemented in this work. Blue text marks tool queries, tool responses are in turquoise cells, refinement is in orange cells and dashed arrows, and yellow cells are LM generations.
and factuality of each claim by generating ques- tions based on the claim, retrieving snippets that answer these questions, and checking if the answers match the information the claim. If not, the claim is refined to match the snippets. Check & Fix, a method we introduce this work, uses each CoT step as search query, and checks whether the step is entailed by the retrieved snippets by prompting the model to classify this entailment. This strategy is similar to Jiang al. (2023, contemporaneous work), which additionally uses low-confidence fil- tering but omits the entailment verification.
are trained to use specific tools (Parisi et al., 2022; Hao et al., 2023; Patil et al., 2023). These methods are constrained to the tools seen during training, and require data (annotated, bootstrapped, or syn- thetically constructed) tool demonstrations.
Other tool-assisted neural networks. There is adjacent research on augmenting neural networks, in ways besides textual interfaces, with tools (e.g.. Andor et al., 2019; Jacovi et al., 2019) training differentiable subnetworks that heavily mimic tools (Neclakantan et al., 2017; Trask et al., 2018).
Evaluation Pitfalls
2.2 Related Work
Training LMs use tools. While we are primar- ily concerned with few-shot tool assistance of LM generation, the literature also explores LMs which
While there a plethora of 'A strategies ($2.1), 1),, no systematic comparison of these strategies has been conducted. Research that proposes TA strategies in

---

Fitfall
Recommendation
(1) Coupling the TA strategy and the tool together.
Comparisons of TA strategies should use the same tools across strategies.
(2) Forcing no-tool baselines framework of the TA strutegy.
The optimal way solve the task without tools may different from solving the task with tools: No-tool baselines should include multiple variants of both free-form and structured strategies, to ensure the TA strategies
given advantage.
(3) Using one model across all comparisons.
Different models may hehave differently when it comes to using tools effectively, based on their training data. Multiple models should be tested, if possible.
(4)
Using one prompt and set of demonstrations across all comparisons.
Multiple different sets of demonstrations should be used to get reliable estimates of few-shot performance.
(5) Not considering TA strategy costs.
TA strategies can he efficient or inefficient with regards to the prompt tokens and generation tokens they require to work, with respect to no-tool baselines or with respect to each other. The differences can be significant ($5). Comparisons TA strategies should factor the computation cost the strategy, which we term as token efficiency.
Table I: Summary of evaluation pitfalls 'TA
TA strategies (83) and recommendations to mitigate them.
few-shot settings is often not focused on evaluat- ing properties of those strategies, but other aspects of LM capabilities (Press et al., 2023; Gao et al., 2023a), usage in particular strict contexts (Paran- jape et al., 2023), evaluating various LM models themselves with particular strategy (Mialon et al.. 2023), and so on.
(4) Using one prompt and one set of demon- strations across all comparisons. Few-shot eval- uation is known to be unreliable when using a single set of demonstrations as a single prompt (Perez et al., 2021). Furthermore, some prompts used in TA strategy evaluations particular, CoT Seemonttaations-apaee so often on the internet that they are suspected to be part of the models' training data, further compromising their function (Jacovi et al., 2023).
Below we collect observations from the litera- ture that demonstrate the limited evaluation scope of TA strategies, in an effort to establish a set of criteria for future evaluations to be reliable and fair (a summary is provided in Table 1).
(5) Not considering TA strategy costs. In many cases, the TA strategy requires significantly more compute than no-tool baselines, and different TA strategies also require different amounts of compu- tation. Computation cost is not traditionally con- sidered in comparisons.
(1) Coupling the TA strategy and the tool to- gether. Comparisons may vary the tools and meth- ods together (c.g., a TA strategy A with a tool A versus a TA strategy B with a tool B).
(2) Forcing baselines to the framework of the TA strategy. Typical baselines given TA strategy are to apply that strategy while letting the model generate the tool's output instead of the tool, and using CoT prompting. However, the optimal way to solve the problem without tools may not be the same as the TA strategy in question. In this work, we implement three different baselines ($4) and find that there is clear winner among two of them (we explore this empirically in $5).
Experimental Setup
Our goal is to conduct a fair and reliable compari- son of TA strategies, without being influenced by properties of specific models, tools or prompts. To this end, we focus on few-shot tool usage, popular TA scheme that allows flexibility around using new tools and adapting tools to specific tasks.
In what follows, we describe our experimental setup. What guides this experimental setup is to perform comprehensive, rigorous evaluation with- out the pitfalls of $3. Our evaluation covers 5 differ- ent TA strategies, 4 recent LMs, 4 complex reason- ing datasets, 3 few-shot prompts, and 2 tools. For each TA strategy + dataset model combination. we run three experiments with a different number of demonstrations. Overall, our evaluation includes an execution of 342 experiments, each of which
(3) Using one model across all comparisons. Of- ten, a single model is chosen to use as the under- lying model for the TA strategy. This limits the insights from the evaluation this model in par- ticular, since conclusions may not carry over to other models. In this work, we find that the best- performing strategies vary significantly across dif- ferent LMs (we explore this empirically in 85).

---

generates 250 (GPT-3) or 500 (non-GPT-3) long- form answers. Additional implementation details are in Appendix A.
test set has 229 examples), and use it for perfor- mance evaluation of UL2, Flan-PaLM-540B and Flan- Pan-Paa For GPT-3, we use a subset of 250 examples of that set, due to cost. We use stan- dard evaluation measures for every dataset (F1 in the case of MuSiQue). We provide data examples in Appendix A.
Tool-assisted strategies. We evaluate the TA strate- gies shown in Figure 2: SelfAsk. Inline, Interleav- ing, C&F and RARR. We additionally include vari- ants of SelfAsk and Inline where the model is sep- arately called to summarize tool output in relevant context, as can often be very long (SelfAskQA and InlineQA; see Appendix A for details). Finally, in the retrieval settings, we use Top-1 retrieval for all models, and additionally Top-5 retrieval for the Flan-PaL-M5540 model (see "Models" below) to check whether additional retrieved information improve performance despite the significantly longer input and processing cost.
Models. We evaluate the methods across four LMs: Flan-UL2-20B (Tay et al., 2023), GPT-3 (text-davinci-on)) 000)) (Brown et al., 2020), Flan- PaLM-550B and Flan-PaLM-62B (Chung et al., 2022). We omit GPT-3 experiments on RARR and Interleaving due to cost. Importantly, our focus is not in comparing performance of these models, but to use them as samples of different model instances and training schemes against which to compare different TA strategies.
For SelfAsk and RARR we use the original im- plementation provided by the methods' creators. We implement Interleaving (Trivedi et al., 2022a). as at the time of this research no implementation was available. Importantly, this implementation yields similar performance to that of existing ap- proaches that combine CoT with retrieval from Wikipedia by He et al. (2022); Jiang al. (2023) (see full results in Appendix B). Additionally, Jiang al. (2023, Figure 4) implemented methods that apply retrieval and refinement over generated CoT that are similar to C&F and achieve similar per- formance to ours, as well (see Appendix B). For Inline, we are not aware of reports on few-shot performance of a similar strategy in the literature.
Tools. We strictly the same tools across all strategies, to ensure fair comparison: Google Search (Press et al., 2023; Schick et al., 2023; Lewis al., 2021) for knowledge tasks, and a cal- culator (Schick et al., 2023; Qin et al., 2023) for the calculation tasks. RARR, SelfAsk and Interleav- ing are designed for retrieval settings only, while Inline and Check & Fix can be used in all settings. For the retrieval settings using Google Search and Flan-PaLMM
test retrieval with both the top 1 and top 5 tool-retrieved snippets: The two formats are designed to cover both cases where a shorter tool output may prevent the model's answer from degenerating, and a longer tool output may help the model with more relevant information.
Baseline strategies. We use no-tool versions of SelfAsk, Inline, and standard CoT prompting. The SelfAsk and Inline baselines simply involve giv- ing the model the prompts used for the tool-based versions, while disabling tool calls (such that the model generates the output in- i--lace of the tools). These are the baselines used by Press et al. (2023) and Schick et al. (2023) respectively.
Few-shot demonstrations. In order to overcome bias from using demonstrations from prior work that were likely seen during training (Jacovi et al., 2023), re-annotate prompts for all TA strate- gies, datasets and tools. We randomly sample 8 examples from each dataset's 's training set, and an- notate each example with demonstrations for each TA strategy. Some of the strategies call the model multiple times with different prompts (e.g., Check & Fix, RARR), which requires separate annota- tions. This effort results in a total of 184 annotated demonstrations, which we release resource for future works on TA generation. From each set of demonstrations, we then construct three separate prompts- --shot,, 5-shot and r-shot-randomlly sampled from the original 8 demonstrations, to get a better estimation of few-shot performance.
We consider tasks that require complex reasoning, where models could potentially benefit from external tool usage. Specifically, we use Strat- egyQA (Geva et al., 2021) and MuSiQue (Trivedi et al., 2022b), which require reasoning about en- tity knowledge, and GSM8K (Cobbe et al., 2021) and DROP (Dua et al., 2019) that evaluate arith- metic reasoning. In DROP we select examples that have numerical answers. We randomly sample 500 examples from the development set of each dataset (with the exception of StrategyQA, whose
Datasets.

---

DROP
GSM8K
MusiQue
StrategyQA
Flan-PaLM 5408
tool without
(a)
Fian-UL2-208
GPT-3
0
50
50
50
50
Score
DROP
GSMBK
MusiQue
StrategyQA
Flan-PaLM-54008
without with refinement
(b)
Flan UL2 20B
GPT-3
50
50
50
50
Figure 3: A comparison of evaluation scores across Tool usage via refinement of generated text vs. tool usage during generation, where the generated text contains tool arguments conditioned on tool outputs. The dark line marks the confidence interval among samples.
($5): (a) No-tool baselines TA strategies; (b)
5 Comparative Results
Model
Dataset
Best strategy
Organization of the results. Due to the Tool vs. no tool. Previous work that propose TA strategies found that using such strategies consis- tently improve performance in comparison no- tool baselines (Press et al., 2023; Jiang al., 2023; Trivedi et al., 2022a, inter alia).
GPT-3
StrategyQA
Buseline-Inling
GPT-3
DROP
Baseline- Inline
GPT-3
GSM8K
Check & Fix
GPT-3
MuSiQue
Inline
Strategy QA
Baseline-CoT
Flan-PaLMM-
DROP
Baseline- -nninne-Inlie
Flan-Pal.MM-3
GSM8K
Baseline-Inling
Figure 3 shows that the TA strategies do not im- prove performance over the no-tool baselines in our selection of datasets. The figure shows results against the average of the different few-shot scores, though observe similar trends when using the maximum of scores as well. Full results in Appendix B. Similarly to us, Gao al. (2023a, $6.2) found that StrategyQA performance slightly decreased with tools in RARR compared to no- tool baselines for PaLM-540B (Chowdhery et al.. 2022), and Jiang et al. (2023, $6.2) found that per- formance decreased StrategyQA in two settings comparable
Flan-Pat Man-PaLMM5540B
MuSiQue
RARR-Tope Top5
Flan-U1.2-208 U1.2-20B
Strategy
Baseline-Inline
Flan UL2-20B
DROP
Baseline- Inline
Flan- UL2-20B
GSMBK
Inline
-L2-00
MuSiQue
Baseline-Cof
Flan-Pal M-6622
StrategyQA
Baseline-
Flan-PaLM-62BB
DROP
Baseline-Col
Flan-Pal.M-6i M-6-2-
GSMBK
Inline
PaLM-62B
MuSiQue
Check & Fix
Table 2: For each combination of dataset and model. derive the best-performing strategy the aver- age score across the few-shot prompts. Notably, the best-performing strategy varies across different models, datasets or prompts, which means that is necessary to evaluate over all axes to get better estimation of general performance.
implementations of Interleaving and Check & Fix with GPT-3.
We conclude that for the settings in this work, the no-tool baselines are stronger than initially expected based on the literature. More research is required to investigate whether this relationship holds in other contexts, though we note that the datasets and models used in our experiments are common in TA research (Mialon al., 2023).
that different baselines obtain different results, and relying on only single baseline in evaluation does not necessarily provide good estimation for no-tool performance (recommendation (2)). Also, the best-performing strategies vary significantly across models, which highlights the importance of using multiple models for evaluation (recommen- dation (3)) for illustration, we report the highest- performing strategies in each setting in Table 2, to
Additionally, experiments provide empirical justification Recommendations (2) and (3) in $3. First, we find that the CoT and Inline base- lines outperform each other at a roughly equal rate, and neither emerges as a clear winner. This shows

---

Prompt tokens
Prompt tokens (empirical)
TA strategy
Retrieval
Calculator
(canonical)
GPT-3
Flan-Pal. Pal.M-540RR
GPT-3
Plan-Pal.MM--44
Baseline
T
353
353
1418
801
SelfAsk
[n++ k++1
2281
1399
2
SelfAskQA
((2n + k)
3589
2736
Inline
[(++ k(al)
1793
1775
3453
1083
InlineQA
1(2n + k))
3375
3672
Check
f(2n + k)
3839
3547
7548
3647
RARR
3m(t + 1)
4729
Interleaving
f(n+ 2
3221
Table 3: Average number of prompt tokens per strategy (5-shot), with the CoT prompt length, t the number of tool calls, as the tool's output length. Flan-PaLMM-540B has a shorter context window than GPT-3, which limits prompt length. The canonical formula for RARR favorably assumes a single verification question.
Answer tokens (canonical)
Answer tokens (empirical)
TA strategy
Retrieval
Calculator
GPT-3
Mlan-Pal.M-5400
GPT-3
Flan-Pal.M-5-000
Baseline
m
44
42
58
88
SelfAsk
m
20
72
SelfAskQA
2m
59
64
Inline
m
103
248
62
102
InlineQA
2m
114
256
Check fix 2m
177
177
RARR
3m
181
Interleaving
m
72
Table 4: Average number of answer tokens across the 5-shot experiments, for each strategy. The RARR formula assumes single verification question step.
show that the overall conclusion be distorted by choosing particular model strtegy Extended details are Appendix B.
and generated tokens, which have direct effect the cost of the TA strategy. Notably, the cost of TA strategy depends on various variables, includ- ing model size, GPU type, caching optimizations. vocabulary size, beam scarch size, and so on. How- ever, token counts can serve as plausibly generic proxy for the purpose of comparing the cost of dif- ferent TA strategies, as other factors are roughly equal across strategies, long as the same models and tools used. We consider prompt tokens and generated tokens separately, as they often have different consequences on cost.
Tool during generation post-generation refinement. In Figure 3 compare the strategies that use tools during generation against the strate- gies that first generate an answer, and then use tools to improve the answer. For retrieval tasks, refinc- ment clearly outperforms non-refinement strategies, but the same does not apply to the calculation tasks. We conjecture that planning calculations ahead of time during generation is more aligned with LM pretraining data, based on internet text, than plan- ning retrieval queries in similar contexts.
Tables 3, 4 show both canonical and empirical comparisons across TA strategies with regards token efficiency. The canonical comparison is function of the relevant variables in the "canonical" setting where the model was expected to answer
Token efficiency. TA strategies typically eval- uated in terms of task performance and properties such factuality and logic correctness. We argue that computational cost another important factor to consider. Specifically, we propose to evaluate to- ken efficiency, that is, the amount of prompt tokens
'Dpendding on model architecture and quantity of times reusing the same prompt, prompt processing cost can be op- timized, whereas the token generation cost varies with other factors such as vocabulary size.

---

the question perfectly, and use the tool perfectly as intended. Across all TA strategy experiments, we found no general correlation between token efficiency and performance. Concretely: (1) All TA strategies are significantly more expensive than the no-tool baselines by orders of magnitude, while not incurring improvement worthy of this extra cost. Empirically, using tools in each case can incur extra costs by a factor of 5x to 10x for prompt processing, and 2x to 5x for generation. (2) The refinement strategies are more expensive than the no no-refinement strategies. So while they improve performance for retrieval tasks, it comes at a cost.
amples in the retrieval setting for all models, but in the calculation setting, this only manifests for Flan- Man-UUL--20B (implying that the larger models are more robust to the numerical ranges in GSM8K and DROP), Overall, in all cases tool use does not improve upon the baselines even when controlling for the harder cases where tools are expected to be more useful. This conclusion is aligned with our analysis in $6.3, which shows that the common errors stem from incorrect tool arguments, more than correct tool arguments but incorrect inferences based on them. Flan-UL2 with a calculator is an exception, where tool use indeed helps, though moreso on the easier examples, likely due to a higher rate of correct arguments to the calculator.
6 Analytical Results
We discuss further analyses of our results, find- ings that (a) our observations generally hold across different levels of example difficulty, and (b) most prediction errors of tool-augmented LMs stem from incorrect inputs to the tool and bad outputs from it, and not from a lack of tool usage.
6.2 Tool Usage Statistics
A possible explanation for the similar performance of no-tool baselines could be lack of tool usage. To check this, we aggregate usage over the differ- ent TA strategies, and find that the models indeed use tools in the majority of the cases; 70%-80% in SelfAsk, and >90% in others (see Appendix B). We also investigate usage across other axes, such as models and number of demonstrations, and find similar trends. However, the datasets and tasks we investigate are designed to benefit from the tools in all cases, which shows that few-shot demonstra- tions are not always sufficient in inducing tool use in models. In particular, the SelfAsk strategies re- ceive the lowest tool use, being the strategies that use natural language to query whether to use the tool (the answer begins with "Are follow up ques- tions needed here:' to which the model answers "No" in the cases where the tool is not used).
6.1 Example Difficulty
It has been shown that LMs have difficulty solv- ing problems involving long-tail entities (Kandpal et al., 2022; Mallen et al., 2022) and complex math- ematical reasoning challenges (Mishra al., 2022: Imani al., 2023). Accordingly, we ablate the results from $5 along the following axes exam- ple difficulty, in order to understand how tools can affect performance on difficult examples. We pro- vide an overview of the trends here, and extended results are available in Appendix B.
Measures of difficulty. We investigate the effec- tiveness of tool-usage across varying levels of ex- ample difficulty, which we approximate in two axes: (A) Long-tail entities (retrieval): Following Mallen et al. (2022), we extract the entities from the ques- tion and associated gold answers in StrategyQA and MuSiQue, and use the corresponding entity Wikipedia page views a measure of popular- ity. (B) Large numbers (calculation): We segment the examples in the calculation tasks based on the range of the median and largest number in the ex- ample (question and gold solution in GSM8k, or question and context paragraph in DROP).
6.3 Error Analysis
We sampled 50 instances for which error was made by the TA models, randomly across the 5-shot experiments, and categorized them across three categories: (A) Incorrect tool input; (B) incorrect tool output; (C) incorrect model inferences based on correct tool usage. Error B applies only to the retrieval settings, where the retrieval tool (Google Search in our case) retrieved a wrong or irrelevant snippet. The errors were distributed approximately to 60% (A), 10% (B), and 30% (C) in the retrieval setting, and 80% (A) and 20% (C) in the calculation setting. Li al. (2023) reported an error analysis for tool-assistance in dialogue customer assistance settings, with similar conclusions regarding error A, although emrors B and C do not apply in their
Results. Performance across increasing levels of entity popularity and computation complexity, with different LMs and TA strategies, are shown in Fig- ure 4a and Figure 4b, respectively. We find that performance uniformly decreases for harder ex-

---

Flan-UL2-20B
Paan-PaLM-5408
Tlan-PaLM--oe
GPT-3
without tool with tool
score
0.4
entity popularity (by page views)
Flan-UL2-20B
Flan-PaLM-5408
Flan-PaLMM622
GPT-3
0.75
without
with tool
score
0.50
(b)
0.25
0.00
0.0
0.2
0.4
0.6
0.8
0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8
0.0 0.2 0.4 0.6 0.8
computation complexity (by numeric range)
Figure 4: We analyze performance of the strategies across example difficulty as defined by the existence of rare or common entities in the retrieval settings (via percentile of page views) and small or large numbers the calculation settings (via percentile of numeric range). In (a), lower page views imply higher difficulty, and in (b), larger numbers imply higher difficulty.
(no-tool baselines TA strategies), conditioned
context, and other error types manifest instead.
Limitations
Our results suggest that the majority of errors not due to the incorrect tool responses (i.e., issues with Google Search as choice of retriever), and overall more influenced by incorrectly invoking tools to begin with, in comparison to invoking them correctly but composing the solution incorrectly.
While our study aims provide comprehensive evaluation of TA strategies, there are some limita- tions. First, recent work (Dodge et al., 2021; Magar and Schwartz, 2022; OpenAI, 2023) suggests that examples from public datasets, like those used our evaluation, may have leaked to the training data of recent LMs. Such contamination can introduce biases to the evaluation, such as lack of need for external tools. We not aware of alternatives without this issue the time of this writing.
7 Conclusions and Takeaways
We conduct a comprehensive assessment of few- shot tool augmentation strategies for LMs, cover- ing hundreds of experiments with multiple LMs, datasets, and tools. Our experiments show that cur- rent ussge integration approaches are presently a false promise; prompting strategies that do not use tools typically obtain similar task performance, without the high cost of tool execution. Controlling for example difficulty, where tools are expected to provide the most benefit, does not explain the rela- tive strength of the no-tool basclines. Instead, the observe related to incorrect usage of the tools to begin with (i.e., generating incorrect arguments the tool).
Second, due the high cost of executing large LMs in exhaustive evaluation, single experiment for each combination of TA strat- egy, model, dataset, and number of demonstrations, However, given the sensitivity of models to the demonstrations (Perez et al., 2021), future work should extend this evaluation to use multiple sets of demonstrations for each such combination.
ran only
Last, while our findings show that non-tool mod- els often perform on par with existing TA strategies, our setting favors tool usage. For example, our tasks only require a single type of tool such that the model does not need to choose between multiple tools. Future work that investigates when and how tools can improve performance should consider more realistic evaluation settings, for example, by considering tasks where the model may need use multiple types tools together, or tasks where tools may sometimes give unhelpful answers.
primary
Our findings call for more robust evaluation of future TA strategies, primarily in more practical settings where models are not expected leverage inherent abilities solve tasks. To this end, our work provides concrete evaluation guidelines, such as employing stronger baselines and factoring in computation costs.

---

References
and Jason Wei. 2022. Scaling language models.
Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. 2019. Giving BERT a calculator: Finding opera- tions and arguments with reading comprchension. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Procexsing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 5947- 5952, Hong Kong, China. Association for Computa- tional Linguistics.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems. CoRR, abs/21 10.1468..
Jesse Dodge, Maarten Sap. Ana Marasović. William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colos- sal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are seo- learners. CoRR, 0ls/2005.14165.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In North American Chapter of the Association for Computa- tional Linguistics.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks.
Luyu Gao, Zhuyun Dai. Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vin- cent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu, 2023a. Rarr: Researching and re- vising what language models say, using language models.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes. Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meisre-ellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2023b. Pal: Program-aided language models.
Mor Geva, Daniel Khashabi. Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristo- tle Use Laptop? A Question Answering Bench- mark with Implicit Reasoning Strategies. Transac- tions of the Association for Computational Linguis- tics (TACI.).
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu, 2023. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings.
Hangfeng He, Hongming Zhang, and Dan Roth. 2022. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301. 003033.
Shima Imani, Liang Du and Harsh Shrivastava. 2023, Mathprompter: Mathematical reasoning using large language models. arXiv preprint 303.5.2333.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le,
Alon Jacovi, Avi Caciularu. Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contami- nation by evaluation benchmarks.
Alon Jacovi, Guy Hadash, Einat Kermany, Boaz Carmeli, Ofer Lavi, George Kour, and Jonathan Be- rant. 2019. Neural network gradient-based learning

---

of black-box function interfaces. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yo Yu, Asli Celikyilmaz. Edouard Grave, Yann LcCun, and Thomas Scialom. 2023. Augmented language mod- els: survey.
Zhengbao Jiang. Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation.
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. 2022. NumGLUE: A suite of funda- mental yet challenging mathematical reasoning tasks. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pages 3505-3522 Dublin, Ireland. Association for Computational Linguistics.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge. arXiv preprint or Xx1v:2211..08111
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Pous, and Matei Zaharia. 2023. Demoostrate--sarc-ppreecce Composing retrieval and language models for knowledge-intensive nlp.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou. Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser- assisted question question-answering answering with human feedback.
Tushar Khot, Harsh Trivedi. Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks.
Arvind Neelakantan, Quoc V. Le, Martin Abadi, An- drew McCallum, and Dario Amodei. 2017. Learning natural language interface with neural programmer. In International Conference Learning Representa- tions.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2021. intensive nlp tasks.
generation for knowledge-
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wor Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844-9855, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Minghao Li. Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api- bank: A benchmark for tool-augmented llms.
Zhan Ling, Yunhao Fang, Xuanlin Li. Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. 2023. Deductive verification of chain-of-thought reasoning.
OpenAl. 2023. Gpt-4 technical report.
Bhargavi Paramjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi- step reasoning and tool-use for large language mod- els.
Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te-Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and An- drew M. Dai. 2022. Mind's eye: Grounded language model reasoning through simulation.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Caltison-Burch. 2023. Faithful chain-of- thought reasoning.
Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. Talm: Tool augmented language models.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and
Joseph E. Gonzalez. 2023. Gorilla: Large language model connected with massive apis.
Inbal Magar and Roy Schwartz. 2022. Data contamina- tion: From memorization exploitation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Pa- pers). pages 157-165, Dublin, Ireland. Association for Computational Linguistics.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das. Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Inves- tigating effectiveness and limitations of paramet- ric and non- parametric memories. arXiv preprint arXiv:2212.1055
Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. 2022. Limitations of language models in arithmetic and symbolic induction.

---

Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen. Ning Ding. Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang. Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang. Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang. Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023. Tool learning with foundation models.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. U12: Unifying language learning paradigms.
Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, and Phil Blunsom. 2018. Neural arithmetic logic units.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022a. Interlcaving retrieval with ofain-of-thought reasoning for knowledge- intensive multi-step questions.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022b. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Fdl Chi, Quoc Lee, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.

---

A Implementation Details
Inline and InlineQA. The Inline strategy format largely mimics the Toolformer format by Schick et al. (2023), but can also be cast into the ART framework by Paranjape et al. (2023) or the Decom- posed Prompting framework by Khot et al. (2023). In general, the strategy simply calls for generating the tool call in predefined format-in in our case, square brackets and the tool name. The tool is in- voked with the arguments generated by the model inside the brackets, and the tool's output is inserted into the model. Our implementation is based on the inference code implemented by Schick et al. (2023), although notably, we focus on few-shot us- age, and do not perform the tool-usage pretraining step that largely concerns the referenced work.
A.1 Tool-Assisted Strategies.
General Details. In all cases, if the tool invoca- tion fails (e.g., with an ill-formatted calculation, or a null response from Google Search), the model is used to generate the tool's output instead. For all retrieval settings using Google Search, we test both Top-1 and Top-5 retrieval: The two formats are designed to cover both cases where a shorter tool output may prevent the model's answer from degenerating, and longer tool output may help the model with more relevant information. Illustrative examples of the data are available in Table 5.
SelfAsk and SA SelfAsk involves de- composing each question into series of simpler sub-questions, and calling the tool directly for each sub-question. The tool's output is inserted into the prompt as an intermediate answer. When the model generates a step that begins with the string "So the answer is:", it is expected to generate an answer that builds on the previous intermediate an- swers which were tool outputs. In this work, we use Google Search as the tool as in the original work by (Press et al., 2023).
We implement two variants: Inline, which uses a tool called "Search" that appends the retrieved snip- pet or calculation output directly into the prompt, and InlineQA, which uses tool called "QA" which calls the model with a separate prompt in order to summarize the retrieved snippet into a concise an- swer, identically to the aforementioned SelfAskQA variant. As with the SelfAsk and SelfAskQA vari- ants, among Inline and InlineQA in the knowledge- based tasks, neither consistently outperforms the other in particular.
Our SelfAsk implementation reuses the origi- nal implementation by Press et al. (2023). Since Self-Ask is designed specifically for knowledge- based QA, we only evaluate this strategy for the knowledge tasks MuSiQue and StrategyQA.
Interleaving. The Interleaving Retrieval strategy (Trivedi et al., 2022a) proposes use each rea- soning step by the model in its CoT answer as a query to retrieval model. The retrieved snippet is then added to the prompt in order to provide ad- ditional information to the model. The structure for each demonstration becomes: (1) All retrieved documents thus far; (2) The question; (3) The gen- erated answer thus far (see Trivedi et al., 2022a for details). In this way, the tool is used heuris- tically without explicit demonstrations from the model, but the generation of the answer at each CoT step is still conditioned on tool usage based on the previous steps.
The SelfAskQA variant involves calling the model for each pair of sub-question and retrieved snippet that (hopefully) contains its answer. This method of recursively calling the model with dif- ferent a differrent prompt as if it were another tool is a technique proposed by Khot et al. (2023). We collect all sub-questions from the SelfAsk prompts in order to construct QA prompts (using the tool to retrieve supporting snippets). The model is called with the QA prompts in order to answer each sub- question based on its snippet. The SelfAskQA variant in essence summarizes each Google Search snippet, which can be as long as a paragraph, into a short answer to the given sub-question, effectively simplifying and shortening the overall answer. Among the two SelfAsk implementations, nei- ther decisively outperforms the other: SelfAskQA outperforms SelfAsk for GPT-3 and FlaM-PaLMM- 62B on both MuSiQue and Strategy QA, but for PaL--LLL and Flan-UL2 UL2-20B the relation- ship flips.
Check & Fix. lightweight variant of refinement based on tools in comparison to RARR, and comparable to contemporancoussl proposed (Jiang et al., 2023): After each CoT step, the step is checked for accu- racy using tool, and if found inaccurate, a new fixed step is generated to replace it.
We propose this strategy as more
In the retrieval setting, each step is verified and fixed by prompting the model to classify whether the step is contradicted by the retrieved paragraphs,

---

Model
Dataset
Strategy
Exumple
Model Answer
GPT-3
QArategyQA
Baseline CoT
Question: Was worman Prime Minister directly before or after Stankey Baldwin? Answer: No.
Stanley Baldwin was Prime Minister of the United King- dom from 1923 to 1929. The woman Prime Minister directly before bim was Margaret Thateber, who served from 1979 to 1990. The womun Prime Minister directly after him Theresa May, who served from 2016 2019. So the answer is no.
Mlan- Mlan-U12
MuSiQue
SelfAsk
Question: What administrative territorial entity contains the place where KPRM is licensed to broadcast to? Answer: Hub- bard County.
Are follow questions needed here: Yes. What is the place where KPRM is licensed to hraadcast Interme- diate answer: KPRM is licensed to breadcast to the city of Houston. Follow up: What administrative territorial cutity contains the city of Houston? Intermediate answer: The city of Houston is in Texas state. Follow up: What the adninistrative territorial entity that contains Texas state? Intermediate auswer: Texas stare is in the United States, the final answer The United States
Flan Pal.M-625
DROP
Fix
Context: [omisted paragraph about the 2000 censis] Ques- tion: How many houscholds and families reside the county cording to the 2000 cemsus? An- swer: 26484
26484 households and 12068 Famiiles reside the county. There were 40543 people, 15416 houscholds, and 11068 families residing in the county. Thas, 26484 12068 38552. So the final answer is: 38552.
Flan- Pauu-PaLM--SN M-00
GSMBK
Inline
How many girls are in the school if 40% of school population is made up of 240 boys? Answer: 360
Question:
40% school population made up 240 boys. then 1% is made up of [Calculator 2200 / 40) -> 6.0] 6 boys. Therefore, the total school population is made up of [CalculatorG * 100) -> 600] 600 students. the school population made up of 600 students, and 40% of them are hoys. then the remaining 60% are girls, total of [Calculator(50/ 100) 600 -> 360) 360 girls. So the final answer is: 360.
Table 5: Illustrative examples of various datasets, strategies and model outputs. The answers from the Interleaving, Check & Fix and RARR models are of the same format as the CoT baseline.
and if so, to generate the fixed step based on demon- strations. In the calculation setting, each step is first heuristically checked for whether it contains a calculation, and if so, the calculation is inserted into the calculator tool, and the model is prompted to verify whether the tool output is consistent with the calculation in the text. If this is incorrect, the model generates the fixed step. In both cases, the answer generation continues where the fixed step completely replaces the original incorrect step.
information from the passage. This is done via prompting the LM with few-shot examples.
Evidence Retrieval: For each generated ques- tion, the Google Search tool is utilized to re- trieve the top-k passages that are related to the question. In this work, we evaluate both Top-1 and Top-5.
3. Evidence Ranking: The retrieved evidences are next ranked using a query-document rel- evance model scorer. Unlike the original RARR implementation (Gao al., 2023a), which uses the GTR retrieval model (Ni et al., 2022), we instead implement the scorer via few-shot LM prompting, as suggested by the authors. The output of this stage is thus the top-1 ranked evidence.
RARR. RARR (Retrofit Attribution using Re- search and Revision, Gao et al., 2023a) was pro- posed as a post processing method for refining any text, including LM saaii--f--thuughy outputs. This is done via automatically finding attribution for each claim in the text, and post-editing the output to fix unsupported content while preserving the original output as much as possible. Our RARR implementation reuses the original implementation by Gao et al. (2023a).
4. Agreement Phase: Given a triplet of a text, question, and an evidence, this phase deter- mines whether both the text and the question imply the same answer to the question. This is implemented via few-shot LM prompting using a chain-of- of-thought style prompt.
The RARR process involves the following steps. with each considered as a separate tool:
1. Question Generation: First, they generate a series of questions that cover various aspects of a passage, referred to as passage x. The questions generated aim to verify and attribute
5. Editing Phase: If the previous Agreement Phase outputs disagreement between the text and the evidence, the (text, question, evidence) triplet is fed to a model that outputs a revised

---

Model
Dataset
Best haseline
Model
Usage (%)
GPT-3
StrategyQA
Inline
Flan-Pal.M--40B
70.9
GPT-3
DROP
Inline
Flan-PaLM-62D
80.6
GPT-3
GSM8K
CoT
Flan- Man-UUL--20B
82.6
GPT-3
MuSiQue
Inline
GPT-3
95.1
Flan-UL2-20B
StrategyQA
Inline
Plan-UU2-200B
DROP
Inline
Table 7: Note that RARR and Interleaving guaran- teed to use tools so they are omitted.
Flan-ULL2-20B
GSM8K
CoT
Flan-UL2-20B
MuSiQue
CoT
Flan Man-PaLM-540BB
StrategyQA
CoT
Strategy
Usage (%)
Flan-Pal.M-54000
DROP
Inline
Check
92.9
Flan-PalMM-540B
GSM8K
Inline
SelfAsk
80.4
Mlan-PaLM--540B
MuSiQue
CoT
SelfAskQA
72.8
Flan-PaLM--60 PaLM-62B
StrategyQA
CoT
Inline
99.9
Tlan-Pal.M-611
DROP
CoT
InlineQA
96.1
1lan-Pal,M-622
GSM8K
Inline
Bl-6-P MuSiQue
CoT
Table 8: Overview of average rate of tool usage across experiments. Note that RARR and Interleaving guar- anteed to use tools.
Table 6:
For each combination of dataset and model, we derive the best-performing baseline on the average score across the few-shot experiments. There is no clear winner: Two of the baselines achieve the best score in 50% of cases.
each tool call, and instead generates the entire an- swer. This is the original no-tool baseline in Schick et al. (2023).
version of the text, considering the discrep- ancy between the previous text and the evi- dence, This is implemented via few-shot LM prompting using a similar thhon-tf-thought style prompt from the previous stage (see Gao et al., 2023a for the exact prompting template). The agreement and editing phases run itera- tively until there are no needed revisions, de- tected in the Agreement Phase.
B Extended Results
We provide the full results for our experiments (described in $4) in $B.1, and further analysis of TA strategy performance and tool usage in $B.2.
B.1 Full Experiment Results
Tables 9, 10 detail our experiment results. Tables 11, 12, 13, 14 detail average and max aggrega- tions over the few-shot prompts. As mentioned. we sample 500 examples for Flan-PaLMM-62B Flan- PaLM-540B and Flan-UL2-20B experiments, and 250 for GPT-3 experiments, with the exception of Strategy QA whose test set has 229 examples.
A.2 Baselines
Chain-of-Thought.. The CoT baseline is the stan- dard baseline proposed by Wei et al. (2023) and implemented as a baseline by Press et al. (2023); Paranjape et al. (2023), inter alia. Often, the demon- strations used for this baseline are those originally published by Wei et al. (2023). In this work we annotate sample of examples with CoT an- swers for the purpose of better estimation of CoT few-shot performance, and release our annotations.
For DROP and MuSiQue, we report the F1 mea- sures using the evaluation scripts provided by Dua et al. (2019); Trivedi et al. (2022b) respectively. For GSM8K, we normalize the numerical answers and measure exact-match. For StrategyQA, normal- ize the answers (for capitalization, prefix and suffix punctuation, and so on) and measure exact-match to "yes" and "no".
Self-Ask. The Self-Ask baseline uses the Self- Ask tool demonstrations, but does not invoke the tool after each "Follow up:" call, and instead gener- ates the entire answer. This is the original no-tool baseline in Press et al. (2023).
Best-perfor pestrprfforinng strategies and baselines in each setting. In Tables 2, show the best- performing baseline and best-performing general strategy for each setting of model and dataset, among the average scores across the three few-shot
Inline. The Inline baseline uses the Inline tool demonstrations, but does not invoke the tool after

---

experiments. For strategies in general (Table 2). we see that the winning strategies vary significantly for different models, which supports Guideline (3)
in Table 1.
The distribution among the baselines is split 50%-50% among CoT and Inline. When consid- ering each few-shot experiment separately (i.e.. not taking the average), the distribution is 60.0%, 37.5%, and 2% for Baseline-Coi. Baseline-Inline and Baseline-SelfAsk respectively for which base- line achieves the hest-performing score. This sup- ports Guideline (2) Table 1.
B.2 Analysis
Example Difficulty. Figures 5, 6 show extended results for the example difficulty analyses in $6. Here we consider the median of each difficulty metric-i.e., the difficulty across all entities or numbers in the example- rather than the minimum maximum, as well as the ablation of refinement strategies against no-refinement strategies, We ad- ditionally checked for two alternative axes: opera- tion complexity (addition and substraction as "easy" examples, and multiplication and division as "hard" examples) and popularity links rather than popular- ity views. The trends we observe in the main paper hold in all of these cases.
Tool Usage, Tables show aggregate tool age percentages over multiple axes. Overall, few- shot demonstrations induce tool usage the major- ity of cases, though not completely so (i.e., below 100%).

---

MusiQue
StrategyOA
Fia5-PaLM:5408
without tool
Flan-UL2-208
GPT-3
Score
MusiQue
StrategyOA
Flan PaLM
without tool
Flan-PaLM-628
Flan-UL2-208
GPT-3
50
50
score
GSMBK
MusiQue
StrategyQA
Hon--PLMM-3-00
without refinement
with refinement
Fla--Pa
(b)
Flan UL2-20B
GPT:3
Score
DROP
GSMBK
MuSiQue
StrategyQA
without refinement
Fla--UL2-208
GPT-3
50
50
50
Figure 5: An extension of Table 3 with results for both the average across few-shot experiments (a-b) and the maximum across few-shot experiments (c-d)-icc, the maximum between 3-shot, 5-shot and 7-shot for each experiments setting.

---

MuSiQue
Strategy QA
Strategy
Model
3-shot
5-shot
7-shot
3-shot
5-shot
7-shot
RARR
Man-PPLM-
34.86
35.09
34.14
80.35
81.22
80.79
RARR
Flan-UL2 -L2-20
13.40
12.01
12.98
55.90
40.17
42.79
RARR
Flan-PaLLM-221
23.60
23.42
24.07
75.98
77.73
77.73
Baseline-Cot T
Pan-PaaM--540B
33.07
33.36
33.80
79.91
84.28
82.10
Baseline-CoT
Plan-UL--20B
15.14
16.50
16.10
67.25
71.62
72.05
Baseline-Cor
GPT-3
27.37
29.31
30.25
70.74
71.62
71.62
Baseline-CoT
Flan- Min-PaLM-62B
23.60
23.42
24.27
75.98
79.04
80.35
Baseline-SelfAAs
Flan Pan-PaLMM-540B
25.80
25.34
24.31
76.86
73.36
75.55
Baseline-SelíAsk
HL2-0B2-200B
11.40
11.52
11.52
34.06
48.47
53.71
Baseline-SelfAsk
GPT-3
27.98
28.13
29.80
72.05
74.24
73.36
Baseline-SelíAsk
Flan Mann-2LLM-62B
5.28
9.52
5.43
58.95
75.98
74.24
Baseline-Inline
Flan Man--aaMM-540B
30.39
30.71
31.19
71.62
79.91
72.49
Baseline-Inline
Plan-UL2-20B
13.66
13.33
9.74
72.05
68.56
71.18
Baslline-Inline Inline
GPT-3
29.11
30.33
28.15
70.31
75.98
78.60
Baseline-Inlimg
Man-PaaM--621
23.42
22.69
21.86
75.11
73.36
75.55
SelfAsk
Man-PaLM-5500
20.02
23.14
23.26
71.62
71.18
73.80
SelfAsk
Flan -L--2U2--00B
11.86
7.68
7.41
49.78
25.76
23.14
SelfAsk
GPT-3
24.38
24.15
22.33
64.19
67.25
65.94
SelfAsk
Mann-PaLM-633
13.79
14.80
12.68
67.25
67.69
66.38
SelfAskQA
Pa55540
21.08
21.92
22.91
71.62
69.43
73.80
SelfAskQA
Flan-UL22 UL2-20B
8.53
5.35
2.30
47.16
17.03
11.79
SelfAskQA
GPT-3
32.74
31.30
30.34
65.50
67.69
70.31
SelfAskQA
Man-PaLM-62BB
15.42
17.49
14.51
67.25
68.12
69.00
InlineQA
Man-PALM-5540B
31.86
32.78
32.10
70.31
72.93
73.36
InlineQA
Flan UL22-20
18.07
17.94
1.56
71.18
70.31
56.77
InlineQA
GPT-3
34.90
36.65
31.32
70.31
72.05
70.31
InlineQA
-lan-PaLLM-22B
12.52
11.65
10.55
61.14
63.32
61.57
Check & Fix
Flan- PaLM-540B
30.73
33.17
33.48
80.35
80.79
78.17
Check & Fix
Flan-UL2-205
10.90
11.77
13.52
52.40
60.70
69.87
Check & Fix
GPT-3
29.66
32.95
32.26
72.05
73,80
70.74
Check & Fix
Flan-PaLMM662B
25.21
26.39
26.47
75.55
71.18
76.42
Inline
Flan PaLM--5000
18.97
24.42
22.61
74.24
74.24
75.11
Inline
Flan- UL2-200
14.70
14.93
14.78
48.47
52.84
44.98
Inline
GPT-3
28.85
31.03
33.54
70.31
69.43
68.56
Inline
9.95
9.45
13.32
54.59
68.56
70.31
Interleaving
Flan- PaLM-----550BB
23.71
21.29
20.51
76.86
78.60
75.98
Interleaving
Flan-PaLMM-22B
23.43
23.71
24.42
74.67
71.62
74.24
RARR-Top5
36.12
35.40
35.44
80.35
79.91
79.91
SelfAskQA-Tops
Man-PaLM-550B
19.75
21.60
21.99
69.87
70.31
72.05
Inline-Top5
Paan-PaLM-5503
32.67
34.53
31.69
65.50
77.73
72.93
Check & Fix-Top5
Maan-PaLM-54000
31.74
32.68
33.87
78.60
81.66
81.22
Table 9; Results for the knowledge-retrieval tasks of MuSiQue and Strategy MuSiQue scores are scores. Missing cells, such as "Interlcaving" with Flan-UL.2-2BB, experiments where the model failed to converge to answer.

---

DROP
GSM8K
Strategy
Model
3-shot
5-shot
7-shot
3-shot
5-shot
7-shot
Baseline-Con
Maan-Pal.M-5A0
77.2
75.0
74.2
67.4
70.8
70.8
Baseline-
Flan-UU.22-00B
7.2
27.2
26.2
Baseline- ToT
GPT-3
57.6
55.6
55.6
58.8
58.0
58.4
Baseline-
Flan- PaLM-62B
65.6
63.6
59.2
47.4
46.2
47.4
Baseline-
77.8
75.6
74.4
69.8
72.6
71.2
Baseline- Inline
-Ln-UUL-200B
3.6
5.6
3.6
Baseline- Inline
GPT-3
57.6
66.0
59.6
51.6
54.0
53.2
Baseline- Iaseline-nnline
Flan-Pan PaLM-62B
59.0
64.0
59.2
48.8
47.8
48.0
Inline
Flan-PaLMM-555 PaLM-540B
76.2
75.2
74.4
61.4
61.8
70.6
Inline
Blan-UL2-205
26.6
26.2
26.0
Inline
GPT-3
56.8
66.0
45.2
50.8
52.4
52.8
Inline
MMn-Pa
57.0
64.0
57.8
48.8
47.8
48.2
Check & Fix
76.0
73.6
45.0
68.4
70.4
70.2
Check & Fix
-ln--U1.2-20B
23.2
25.8
23.2
Check & Fix
GPT-3
54.8
54.4
54.8
56.0
58.4
61.6
Check & Fix
Flan-PaLMM-22B
65.0
63.6
44.2
46.8
44.0
46.6
Table 10: Results for the calculator settings of DROP and GSM8K. We omit Flan- UL2-20B results on DROP, as the model could not converge solve the task with prompts, likely since each example in this task very long.
Stralegy
Aggregation
Model
MuSiQue
StrategyQA
Baseline-Cof
Max
GPT-3
30.2
71.6
Baseline-Con
Average
GPT-3
29.0
71.3
Baseline-Cof
Max
Flan-U1.2-200
16.5
72.1
Baseline-Cor
Average
Flan-UT.2-20B
15.9
70.3
Baseline-Coo
Max
Flan-PaLM--672
24.3
80.3
Baseline- CoT
Average
Flan MaanPaLMM-225
23.8
78.5
Baseline-Cofi
Max
Flan-Pal-M-5400
33.8
84.3
Baseline-CoT
Average
Flan-Pal.MM-50B
33.4
82.1
Baseline-SelfAss.
Max
GPT-3
29.8
74.2
Miseline-SelfAae
Average
GPT-3
28.6
73.2
Baseline-SclfAsk
Max
Flan-U1.2-20B
11.5
53.7
Baseline-SelfAsk
Average
Flan-UT.2-20B
11.5
45.4
Maxeline-SelfAak
Max
Flan-PaLM-66BB
9.5
76.0
Baseline- SelfAsk
Average
Flan 1an-PaLM-22B
6.7
69.7
Baseline-Sefie
Selfnsk
Max
Tlan-PaLM-54001
25.8
76.9
Baseline-SelfAsk
Average
Flan-PaLMM-540B
25.1
75.3
Baseline-Innine
Max
GPT-3
30.3
78.6
Daseline-Inlimg
Average
GPT-3
29.2
75.0
Baseline-Iniee Inline
Max
Flan- -Ln-UU2
13.7
72.1
Baseline-In Inline
Average
Flan-UT.2-20B
12.2
70.6
Maselnne-Iniine
Max
M-n-PaLMM62BB
23.4
75.5
Baseline- Inline
Average
Man-PaLM-622
22.7
74.7
Baseline-Inlin
Max
31.2
79.9
Baseline-Inling
Average
Flan-Pal.M-5400
30.8
74.7
Table I: Aggregations by few-shot prompt results Table (baselines).

---

Fian-UL2-208
Fian-PaLM-540B
Fian-PaLM-628
GPT-3
without tool with tool
score
(a)
0.2
0.0
0.2
0.4
0.6
0.8
0.0 0.2 0.4 0.6 0.8
0.0 0.2 0.4 0.6 0.8
0.0 0.2 0.4
0.6 0.8
entity popularity (by page views)
-lan-UL2-20B
Flan-PaLM-5408
Flan-PaLM-628
0.75
without tool
with tool
0.50
(b)
0.00
0.2
0.4
0.8
0.0 0.2 0.4
0.6
0.8
0.0
0.4 0.6
0.0
0.2
0.4
0.6
computation complexity (by numeric range)
Flan-UL2-208
Flan-PaLM-540R
Flan-PaLM-622
GPT-3
0.6
without oo
with to
Score
(c)
entity popularity (by page views)
Flan-UL2 Man-ULL-200
Hlan-PaLM-540B
Flan-PaLM-660
GPT-3
without
tool
score
0.50
(d)
0.2
0.4
0.8
0.4
0.4 0.6
0.0
D.6
computation complexity (by numeric range)
Fian-UL2-20B
Flan-PaLM-5408
Flan-PaLM-628
GPT-3
0.6
without to
with tool
score
(e)
0.2
0.0
0.2
0.4
0.6
D.8
0.0 0.2 0.4 0.6 0.8
0.0 0.2 0.4 0.6 0.8
0.0 0.2
0.4
0.6 0.8
entity popularity (by page links)
Flan-UL2-20B
Flan-PaLM-5408
Flan-PaLM-62B
GPT-3
0.6
without refinement
with refinement
0.4
0.2
0.0
0.2
D.4
0.6
D.B
0.0 0.2 0.4 0.6 0.8
0.0 0.2 0.4 0.6 0.8 entity popularity (by page views)
0.0 0.2 0.4 0.6 0.8
Figure 6: An extension of Table 4. (a-b) refer to taking the minimum of entity page views to ablate examples that have rare entities, and maximum of numbers to ablate examples with large numbers. (c-e) take the median both cases, and (f) shows the results when comparing TA strategies between refinement and non-refinement types.

---

Strategy
Aggregation
Model
MuSiQue
StrategyQA
Interleaving
Max
Mla--PaLM-62B
24.4
74.7
Interleaving
Average
Maa--PaMM-62B
23.9
73.9
Interleaving
Max
Flan-Pal.M-540B
23.7
78.2
Interleaving
Average
Man-PaLMM5540B
21.8
77.0
RARR
Max
-la--UU2--20B
13.4
55.9
RARR
Average
Mlan-UL.2-20B
12.8
46.3
RARR
Max
Flan-PaLMM-62B
24.1
77.7
RARR
Average
Flan-PaLM--66
23.7
77.1
RARR
Max
Mlan-Pal.M-5A4
35.1
81.2
RARR
Average
Maan-PaLM-5505
34.7
80.6
RARR-Top5
Max
Flan-Pal-M-5400
Flan-Pal
36.1
80.3
RARR-Top5
Average
Flan-PaLMM-540B
35.7
80.1
Check & Fix
Max
GPT-3
32.9
73.8
Check & Fix
Average
GPT-3
31.6
72.2
Check & Fix
Max
Flan-UL2-20B
13.5
69.9
Check & Fix
Average
Plan-UL2-40
12.1
61.0
Check & Fix
Max
Mann-PaLM-62B
26.5
76.4
Check & Fix
Average
Maan-PLM-6225
26.0
74.4
Check & Fix
Max
Flan-Pal-M-5A4
33.5
80.8
Check & Fix
Average
Flan-PaLLM-540B
32.3
79.6
Check & Fix-Top5
Max
Flan-Pal
Pal.M-54085
33.9
81.7
Check & Fix-TopS
Average
Flan-PaLM-54000
80.5
Table 12: Aggregations by few-shot prompt of the results in Table (TA strategies).

---

Strategy
Aggregation
Model
MuSiQue
Strategy QA
SelfAsk
Max
GPT-3
24.4
67.2
SelfAsk
Average
GPT-3
23.6
65.8
SelfAsk
Max
11.9
49.8
SelfAsk
Average
Plan-UL2-20BB
9.0
32.9
SelfAsk
Max
Flan-PaLM-62B
14.8
67.7
SelfAsk
Average
Flan-PaLMM-62B
13.8
67.1
SelfAsk
Average
Flan-Pal-M-540B
22.3
72.2
SelfAsk
Max
Flan-Pal-M-5408
23.4
74.2
SelfAskQA
Max
GPT-3
32.7
70.3
SelfAskQA
Average
GPT-3
31.5
67.8
SelfAskQA
Max
Plan-UL2-2205
8.5
47.2
SelfAskQA
Average
Man-UL2-20BB
5.4
25.3
SelfAskQA
Max
Flan-PaLMM-62B
17.5
69.0
SelfAskQA
Average
Flan-Pal_M-62B
15.8
68.1
SelfAskQA
Max
Flan-Pal-M-540B
22.8
75.1
SelfAskQA
Average
Flan-Pal-M-540R
21.9
71.9
SelfAskQA-Tops
Max
Plan-Pal.M-544
22.0
72.1
SelfAskQA-Top5
Average
Flan-PaLMM-540R
21.1
70.7
InlineQA
Max
GPT-3
36.7
72.1
InlineQA
Average
GPT-3
34.3
70.9
InlineQA
Max
Flan-UL22-00
18.1
71.2
InlineQA
Average
Flan-U1.2-20B
12.5
66.1
InlineQA
Max
M-66-B
12.5
63.3
InlineQA
Average
Flan-Pal.M-622
11.6
62.0
InlineQA
Max
Plan-PaL.M-400
32.4
73.4
InlineQA
Average
Flan-PaLMM-540B
32.1
72.2
Inline
Max
GPT-3
33.5
70.3
Inline
Average
GPT-3
31.1
69.4
Inline
Max
Flan-UL2-20BB
14.9
52.8
Inline
Average
Flan-U1.2-20B
14.8
48.8
Inline
Max
Plan-PaL.M-62.
13.3
70.3
Inline
Average
Flan-PaL.M-622
10.9
64.5
Inline
Max
24.3
74.7
Inline
Average
Jan-PaLMM5540B
22.0
74.2
InlineQA-Top
Max
Flan Man-PaLM-5540B
34.5
77.7
InlineQA-Tope
Average
Mlan-PaLM--50B
33.0
72.1
Table 13: Aggregations by few-shot prompt of the results Table (TA strategies).

---

Strategy
Aggregation
Model
DROP
GSMSK
Bascline-CoT
Max
GPT-3
57.6
58.8
Baseline-CoT
Average
GPT-3
56.3
58.4
Baseline-Cor
Max
Flan-UL2-20B
27.2
Bascline-CoT
Average
Flan -LL-UUL--00B
20.2
Baseline-CoT
Max
Flan-Pal.M-62B
65.6
47.4
Bascline-CoT
Average
Flan-PaLM-622B
62.8
47.0
Baseline-CoT
Max
Flan-Pal.M-5400B
77.2
70.8
Baseline-CoT
Average
Flan-Pal.M-540B
75.5
69.7
Bascline-Inline
Max
GPT-3
66.0
54.0
Baseline-Inline
Average
GPT-3
61.1
52.9
Bascline-Inline
Max
Flan-UL2-200
9.2
5.6
Bascline-Inline
Average
Flan-UL2-20B
4.2
4.3
Baseline-Inline
Max
Flan-Pal-M-62B
64.0
48.8
Bascline-
Average
Man-PaLM-62B
60.7
48.2
Baseline-Inline
Max
Man-P50.M-5500
77.8
72.6
Baseline-Inile
Average
Flan-Pal.M-550B
75.9
71.2
Check Fix
Max
GPT-3
54.8
61.6
Check & Fix
Average
GPT-3
54.7
58.7
Check & Fix
Max
MannUUL2-20B
25.8
Check Fix
Average
Flan-UL2-20B
24.1
Check & Fix
Max
Flan-PaLMM-62B
65.0
46.8
Check Fix
Average
Flan-PaLMM-62B
57.6
45.8
Check & Fix
Max
Flan-PaLM--540B
76.0
70.4
Check & Fix
Average
Flan-PaLM-55400
64.9
69.7
Inline
Max
GPT-3
66.0
52.8
Inline
Average
GPT-3
56.0
52.0
Inline
Max
Mann-LL--205
26.6
Inline
Average
Flan-UL2-20B
26.3
Inline
Max
Flan Man-PaLM-62B
64.0
48.8
Inline
Average
Flan-PaLMM-62B
59.6
48.3
Inline
Max
Flan-PaL.M--50BB
76.2
70.8
Inline
Average
Flan-PaLM--500
75.3
64.5
Table 14: Aggregations by few-shot prompt of the results in Table 10.


Reviews:
[{"review_id": "HJxnxrPBw6", "reviewer": "Reviewer_YEbL", "summary": "", "questions": "Q1: how tool use during generation and post-generation refinement influence long-tail entities retrieval and large-numbers calculation respectively? Could you please give more examples to illustrate this question?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and insightful comments! We appreciate the affirmation that our work will be fundamental to future evaluations, and the proposals for improvements to the paper.\n\nWe address below the two concerns raised in the review:\n\n#### **(1) On the scope of the example difficulty analysis:**\nIn the paper we analyzed the effectiveness of TA strategies across varying levels of example difficulty, where difficulty was measured using entity popularity and numerical range, following previous work [1, 2]. We ran two additional analyses:\n1. Based on complexity of operations - between addition, division, multiplication and so on.\n2. Based on entity rarity as an alternative measure of difficulty, via rarity of entity links in Wikipedia.\n\nIn both cases, tools are expected to improve performance on the “harder” examples, but we did not observe statistically significant improvement in most settings, as with the analyses in the paper. We will add these results to the camera-ready version, and hope that this sufficiently increases the scope of this analysis. If there are any suggestions for other axes for testing performance across increasing difficulty levels, we are more than happy to analyze them!\n\n#### **(2) On prompt/generation processing costs:**\nThank you for this comment. We agree that this discussion can be improved, and we will do so for the camera-ready version. Specifically, the token counting metrics (canonical formulas, and empirically 5x to 10x for prompts, 2x to 5x for generation) are directly equivalent to monetary payment per token charged by popular model APIs such as OpenAI, Google and MosaicML, which means that using TA strategies will increase costs by the observed factors based on the additional tokens. We will add more precise details and definitions here. \n\nWhen the model can be deployed locally, the token counts are not a direct measure for processing cost. This cost depends on model size, GPU type, optimizations such as caching prompt embeddings, vocabulary size, beam search size, self-consistency implementation, and so on. However, token counts can serve as a plausibly generic proxy for the purpose of comparing different TA strategies, since all other factors are roughly equal across strategies, as long as the same models and tools are used. Regardless, we will add these discussions and formalizations to the paper, so future evaluations can make better cost judgements. \n\n#### **Regarding Q1:**\nThe appendix (e.g., fig. 6) has this analysis, which we can expand and mention in the main paper. There are not many conclusions we can make here, except that refinement’s improvement over non-refinement in retrieval settings does not appear to change significantly between easier or more difficult examples, and is different depending on the specific model.\n\nThanks!\nAuthors\n\n[1] Yuan et al., 2023, https://arxiv.org/abs/2304.02015 \n\n[2] Mallen et al., 2023, https://arxiv.org/abs/2212.10511", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In this paper, the evaluation schema is set up by measurements and the some of the authors’ findings can be referenced for more robust evaluation of future TA strategies. In addition, the paper is logically written.", "reasons_to_reject": "Measures of difficulty only consider the long-tail entities retrieval and the large numbers calculation, so it may not be comprehensive enough. Also, the definition of prompt processing cost and how it relates to token efficiency are not explained clearly in the section 5, so the argument “using tools in each case can incur extra costs by a factor of 5x to 10x for prompt processing, and 2x to 5x for generation” is not convincing.", "questions_to_authors": "Q1: how tool use during generation and post-generation refinement influence long-tail entities retrieval and large-numbers calculation respectively? Could you please give more examples to illustrate this question?", "paper_topic_and_main_contributions": "This paper presents a comprehensive evaluation of how different tool-assisted strategies influence the LLM performance. The contributions of this paper are as follows: firstly it summarizes the evaluation pitfalls of tool-assisted (TA) strategies as a set of criteria which can facilitate reliable and fair evaluation work in the future; secondly, it clearly demonstrates the experimental setups by grouping the models and datasets with TA strategy and distinguishing the during-generation and post-generation refinement for the tool usage. This work will serve as fundamentals for future evaluation of TA strategies.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None", "Justification_for_Ethical_Concerns": "None"}, "extra_text": {}}, {"review_id": "BcgF9KdH6d", "reviewer": "Reviewer_uECc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and positive review! We greatly appreciate the reviewer's recognition of the  effort that went into the literature review of existing approaches, and the extensive experimental scope.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper provides a comprehensive assessment of the existing LLM tool augmentation strategies,  which has not been explored before.   It exposes the pitfalls of the existing evaluation setups which can help the NLP community in designing better evaluation systems for LLMs.\n\n2)  Along with documenting the pitfalls, the paper provides concrete evaluation guidelines and recommendations to mitigate the pitfalls, such as employing stronger baselines and factoring in computation costs.\n\n3) The paper performs an impressive number of experiments (324 to be precise) using different LLMs, difficulty levels of examples, datasets, and tools. The experiment setups are also sound.\n\n4) The paper contains a thorough literature review of the existing approaches. The paper is also well-written and easy to understand.\n\n", "reasons_to_reject": "I do not have any reasons to reject the paper. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper evaluates the existing tool augmentation strategies for LLMs. Existing work in the field lacks thorough evaluation of these strategies with strong baselines and with each other. This paper points out the pitfalls in the evaluation setups and conducts a comprehensive assessment of these augmentation strategies covering hundreds of experiments with multiple LMs, datasets, and tools. \nResults show that current tool-usage integration approaches are not as good as they are portrayed: the improvement obtained by augmenting LLMs is low and not stable. Further, these tools incur extra costs by a factor of 5x to 10x for prompt processing, and 2x to\n5x for generation.\n", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "otrKo2q1I4", "reviewer": "Reviewer_2Ti3", "summary": "", "questions": "1. In Figure 3, the legend presents \"w/\" and \"w/o\" refinement (baselines), but the caption describes as \"tool usage via refinement of generated text\" vs \"tool usage during generation\". Which one is correct? For a more clear presentation, there seems to be three types of methods, baselines (w/o tool), tool-assisted during generation, tool-assisted refinement. Can they be put into the same figure? \n2. Why does adopting calculator lead to a decrease of performance in DROP and GSM8K? It seems to be contradictory to some previous work. Is there any additional explanation or supportive analysis for this issue? \n3. In the caption of Table 10, what do you mean \"Flan-UL2-20B could not converge\"? It is supposed to be few-shot in-context learning, why will the model converge without training?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and very detailed review, and for the constructive suggestions!\n\nOverall, the review contains points regarding the general writing and organization of the paper (with one exception, point 2 in “reasons to reject”), all of which we will address in a manuscript update. Thanks for this valuable constructive feedback. We address these concerns below in detail, enumerated to the “reasons to reject” points:\n\n#### **(1) Organization and writing:**\nWe appreciate the reviewer’s constructive suggestions, which we will address in the revised manuscript. Specifically, we will follow the reviewer’s recommendations in the “presentations improvements” section and add the following (a) a high-level summary of section 4, (b) making the references to the appendix more specific such that it is easy to find the relevant parts, (c) clarifying the reviewer’s questions 1 and 3 about the figures and tables (see our answers below), and (d)  improving the tables’ layouts in the appendix and splitting them to sub-sections. \nPlease let us know if you have any further suggestions, thanks!\n\n#### **(2) Types of tools:**\nPlease note that the scope of our experiment setting was prohibitively large due to requiring hundreds of different experiments in order to get sufficiently robust conclusions - we found that every aggregation axis in the “pitfalls” overview mattered. Adding one more type of tool would multiply the experiments by a multiplicative factor (requiring 150-200 more experiments, depending on the tool). This is unfortunately very costly and time-consuming to execute. However, we believe that this is not a compromise since our work is primarily concerned with showing sufficient evidence that this kind of rigorous evaluation is necessary, rather than comparing tool-usage performance of specific tools. In addition, we believe that there is also value in the particular choice of tools we made (i.e., search and a calculator), as these are arguably of the most widely used and studied tools.\n\n#### **(3) Contradictory findings:**\nOur findings do not actually contradict previous work in many cases - for example, we replicate results for SelfAsk when using the same model, prompt, datasets, and baselines as in that work (including their observation that SelfAsk wins over the baseline). Where possible, we use the same code written by the original authors, as well (we use our own implementation only when public code is not available). The contradicting conclusion comes from performing the experiments by aggregation across many models, settings and prompts, which serves to strengthen our discussion about the pitfalls in section 3. We will add this discussion to the paper.\n\n#### **(4) Results organization**\nThank you for this valuable feedback. We prioritized visual analyses of the results, rather than presenting full tables due to lack of space. For the camera-ready version, we will incorporate aggregated results of the full table from the appendix into the main paper to provide a more organized and concrete overview of the results in the main paper.\n\n#### **Regarding the questions:**\n\nWe answer the “questions to the authors” below in-depth, which primarily concern communication and writing. We will revise the paper to incorporate these suggestions and make all the information here clearer.\n\nQ1. “Tool usage during generation” (as opposed to tool usage after generation) refers to “w/o refinement”. We will clear this up in the paper. This is a binary categorization of TA strategies which we referred to in multiple ways, which caused this confusion. [Category 1] includes strategies like SelfAsk which invoke a tool before or during model generation, while [Category 2] refers to strategies like Check&Fix which first generate some answer without a tool, then invoke a tool to refine/revise the generated text. The goal of this categorization is to get meaningful general insights across multiple TA strategies, in the interest of giving more resilient conclusions. \n\nQ2. We see two possible explanations: \n\n(A) Previous research traditionally uses a single model (commonly GPT-3), a single dataset, a single prompt, and a single no-tool CoT baseline. We report empirical evidence in our work that such specific choices can give conclusions that do not generalize. For example, for Flan-UL2 and US on GSM8K specifically, “Inline” was the best performing strategy (tab. 7) winning over the baselines, and this does not contradict previous papers’ results. Our contradicting conclusions come from aggregating results over many settings. Note that in some previous work (e.g., Toolformer), improvements with tools were demonstrated with fine-tuned TA models, rather than few-shot TA.\n\n(B) For DROP and GSM8K, as well as many popular calculation tasks, it appears that newer and bigger models can already perform the calculations well without tools, so tools become unnecessary in those cases, and only bring further complications. We can see this in fig. 4 where the larger models do not lose performance even on calculations with large numbers (in GSM8K and DROP, these numbers are approx. in the thousands), while Flan-UL2 does. \n\nQ3. Here we meant that few-shot prompting with DROP demonstrations on Flan-UL2 does not manage to learn the task sufficiently (negligible performance). This is consistent with previous work, and likely due to the fact that DROP examples include long and difficult paragraphs in addition to questions about those paragraphs. \n\n#### **Regarding typos and grammar:**\nThank you for the detailed comments. We will implement all of them with our best effort, including a reorganization of Appendix B and rewriting of captions (see our detailed response to the organization and writing concern above).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work investigates a timely and important problem in the era of LLMs. \n2. The analysis of pitfalls in existing studies is valuable.\n3. This work conducts a comprehensive and systematic evaluation with various tool-assisted strategies across multiple models and complex reasoning benchmarks. \n4. The extensive analysis provides some new insights for the future studies on tool-assisted generation. ", "reasons_to_reject": "1. The organization of this paper can be further improved. There are some severe issues regarding the readability of this paper. The analysis of experimental results is a bit messy, where the illustration (figure/table) is hard to be interpreted. Despite many presented evaluation results, it is not sure whether it is useful for supporting the conclusion, since there is no related discussions or it is difficult to locate the discussions. \n2. The types of investigated tools are limited (only two: Google search and a math calculator). It is questionable regarding the generality of the conclusion. \n3. There are some interesting findings that are contradictory to the existing findings from other papers, but they lack of detailed explanations or supportive analysis for the results. \n4. Although it is good to adopt various models and tool-assisted strategies for evaluation, the comparison is rather messy. From the results in the tables, it just list a bunch of results but it is difficult to make a clear comparision. ", "questions_to_authors": "1. In Figure 3, the legend presents \"w/\" and \"w/o\" refinement (baselines), but the caption describes as \"tool usage via refinement of generated text\" vs \"tool usage during generation\". Which one is correct? For a more clear presentation, there seems to be three types of methods, baselines (w/o tool), tool-assisted during generation, tool-assisted refinement. Can they be put into the same figure? \n2. Why does adopting calculator lead to a decrease of performance in DROP and GSM8K? It seems to be contradictory to some previous work. Is there any additional explanation or supportive analysis for this issue? \n3. In the caption of Table 10, what do you mean \"Flan-UL2-20B could not converge\"? It is supposed to be few-shot in-context learning, why will the model converge without training?", "paper_topic_and_main_contributions": "This paper conducts a comprehensive evaluation of tool-assisted generation strategies. In specific, the authors first analyze the shortcomings of existing analysis and evaluation on tool-assisted strategies from five perspectives. To overcome these issues, the authors design a systematic evaluation scheme to evaluate five tool-assisted strategies across multiple models and complex reasoning benchmarks. The empirical analysis reveals several findings regarding the usage of tool-assisted strategies. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Section 4 can be summarized and its details can be reorganized into the Appendix, so that it can leave more space for the result analysis. \n2. The reference to Appendix should be more clear. For example, there are many parts presented with \"in Appendix B\". However, it is difficult to locate which part of Appendix B. \n3. Many captions and legends of the figures and tables are also unclear, as posted in the questions before. \n4. The comparison is also messy, which is difficult to tell which two methods are supposed to be compared. For the tables in appendix, it would be better to add some hline to separate several sub-session for specific comparisons or just use multiple sub-tables. "}}]

Author rebuttal:

Meta-review: This paper investigates how tool-assisted strategies impact LLM performance. It provides comprehensive empirical evaluation comparing strategies that use tools and those that do not leverage any tools, using multiple LMs, benchmarks, and tools. The findings, particularly that the current tool usage approaches are not as good as they are portrayed, and that they also incur extra costs, are interesting, novel, timely, and can benefit the community for future research on tool-assisted efforts with LLMs.

Concerns include the organisation and readability of the paper, and the limited types of tools that are investigated. The author responses provide further insights on the latter that have the potential to increase the scope of the contributions of this paper.
Decision: Accept-Findings