Tool Decoding: A Plug-and-Play Approach to Enhancing Language Models for Tool Usage

Authors: Chenheng Zhang, Mingqing Xiao, Yifei Wang, Lizhe Fang, Haihan Zhang, Zhouchen Lin
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, tool usage, decoding method

Abstract:
Despite the significant advancements in large language models (LLMs), their tool-use capabilities remain limited. This limitation stems from the fact that existing approaches often merely adapt strategies designed for basic natural language tasks, overlooking the specific challenges inherent in tool usage, such as precise tool selection, strict predefined formats, and accurate parameter assignment.
To bridge this gap, we conduct a fine-grained analysis of the tool usage process, breaking it down into three critical stages: tool awareness, tool selection, and tool call. Our analysis reveals that most failures stem from selection errors, format violations, and parameter mis-assignments.
Building on these insights, we propose \textbf{Tool Decoding}, a novel, training-free approach that directly incorporates tool-specific information into the decoding process. Tool Decoding employs constrained decoding to ensure format correctness and eliminate hallucinations, while leveraging order consistency to improve parameter accuracy through structured sampling and a majority-voting mechanism. This approach effectively addresses many common tool-use errors in a plug-and-play manner, allowing for seamless generalization to new tools as long as they are accompanied by well-structured documentation to guide the decoding process. 
Experimental evaluations on benchmarks like API-Bank and BFCL V2 • Live show that Tool Decoding leads to significant improvements across a diverse set of more than 10 models, including both generalist and tool-finetuned models. Almost all models demonstrate performance gains exceeding 70\% on both benchmarks. Among the 7B-level models, five outperform GPT-3.5 on key tasks, with two even surpassing GPT-4.

Decision: withdrawn
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730303561264

thread_type:
official_review

<thread_object>
creator: reviewer_cykv

<content>
summary:
The authors propose a constrained decoding workflow to improve LLMs' utilization of external tools. LLMs need to recognize when to enter tool mode, which tool to select, and how to invoke the tool with the appropriate arguments. Errors in any of these steps can result in an incorrect response. To address this, the authors introduce a constrained decoding approach that restricts the output vocabulary to include only tool names when the LLM enters tool mode. After a tool is selected, multiple candidate arguments (key-value pairs) are generated by varying the parameter order, and majority voting is used to select the final value for each parameter. The tool is then called, and the LLM continues generating its response. The authors evaluate their tool decoding method on the API-Bank and BFCL datasets across several LLMs, finding that their approach outperforms standard greedy and beam search decoding techniques.

strengths:
The main benefit of the proposed constrained tool decoding approach is that it is training-free, making it easier to integrate into existing inference pipelines. 
Constrained decoding provides a significant accuracy boost over greedy and beam search methods in terms of tool usage. 
The authors conduct an error analysis to examine the impact of different error types across various LLMs, helping to identify key bottlenecks. 
Additionally, the idea of generating multiple values for each parameter by varying the order is a compelling approach.

weaknesses:
The proposed constrained decoding method increases latency, so it would have been beneficial to include a comparison in terms of latency impact. Additionally, incorporating some naive baselines, such as those using alternative search strategies like epsilon sampling or prompt engineering, would add value. As noted in the related work section, other constrained decoding algorithms are available, so a comparison to existing methods would have been interesting. Furthermore, the absence of stronger instruction-tuned LLMs as base models is a limitation.

questions:
Could you please address the below questions:
1. Why UltraTool is used in Figure 2, while API-Bank is used in Figure 3?
2. Why the awareness error changes between the configurations with and without tool decoding in Figure 6?
3. Why LLM is used to generate optional parameters instead of supplying them directly, as is done with required parameters?
4. Provide a more detailed explanation on how accuracy is computed in the paper?
5. Report the success rate with and without constrained decoding, specifically indicating the proportion of samples with correct awareness, selection, and tool call?

Additional comments:
1. Figure 5 is missing accuracy numbers for Gemma.
2. Reporting accuracy numbers in Table 3 rather than error reduction might provide easier interpretation.
3. “Notably, tool call presents the greatest complexity, with even powerful models like Qwen1.5-72B achieving less than 60% accuracy in this stage” is a bit misleading because just by reading the text it appears that none of the model is able to achieve greater than 60% accuracy, which is not true because 32B model achieve around 70% accuracy. Maybe you can consider re-phrasing it to “Notably, tool call presents the greatest complexity, with the best performing model achieving around X% ”
</content>
</thread_object>

</thread>
<thread>
creation date:
1730316201252

thread_type:
official_review

<thread_object>
creator: reviewer_fh4s

<content>
summary:
This paper introduces a method to improve LLMs' tool-use capabilities with constrained decoding. They claim that current LLMs struggle with tool awareness, tool selection, and tool call. To address these, the author uses constrained decoding to reduce selection and format errors, moreover they also use order consistency to enhance parameter accuracy with structured sampling and majority voting. 
This plug-and-play approach enables seamless adaptation to new tools only with the api documentation.
Experimental results on API-Bank and BFCL V2 Live benchmarks show high performance improvements across different models.
This work demonstrates the effectiveness of specialized decoding methods tailored to tool usage.

strengths:
1. This work introduces an innovative, training-free approach to address tool-use challenges in language models. With constrained decoding and order consistency, the LLM can have much better performance over the two benchmark.
2. The paper is well-organized and uses clear figure to present the ideas, like constrained decoding. The figures break down workflows and methods, which helps improve readability.
3. This method provide a way to improve the LLM tool ability without training. This method may be important across diverse applications, enabling more efficient and versatile LLM deployment in real-world settings.

weaknesses:
1. The author only compares to standard methods like greedy and beam search. However, I believe the author should do more comparison with recent tool-use improvement methods, such as state-tracked constrained decoding and reranking method or other in-context learning methods.
2. The models involved in the paper are mostly under 7B, (only dicuss 7B > models in the Figure 2), yet discussion about 30B, or even 70B model is limited in the paper. I am wondering whether the method still works or how much the method will improve when the model gets larger.
3.  Since the method involves structured sampling and majority voting, maybe the author can discuss its computational efficiency compared to baseline. This would give a more picture of its practical trade-offs in real-time applications.

questions:
See questions in Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730571285913

thread_type:
official_review

<thread_object>
creator: reviewer_hs7t

<content>
summary:
The paper presents Tool Decoding, a training-free approach to improve LLMs’ tool-use capabilities, addressing common errors in tool awareness, tool selection, and tool call. By using constrained decoding for format accuracy and order consistency for parameter correctness, Tool Decoding significantly boosts tool-augmented performance on benchmarks e.g. API-Bank and BFCL V2 • Live, even surpassing GPT-4 in some cases. This adaptable approach enhances LLMs' effectiveness in real-world tool use without requiring additional training.

strengths:
1. The paper presents Tool Decoding a novel, training-free approach to enhancing large language models (LLMs) in tool use. Unlike conventional methods relying on extensive fine-tuning, this plug-and-play solution integrates tool-specific data into the decoding phase. Tool decoding approach includes constrained decoding and order consistency modules, which is shown to effectively mitigate different tool errors e.g. Key error, Value error, and Format error.

2. The paper conducts an error analysis across three stages of tool usage—Tool Awareness, Tool Selection, and Tool Call. The analysis is done, using both API-Bank and BFCL V2 • Live datasets, which are benchmark datasets well-suited to assessing tool-augmented LLMs.

3. They showed the effectiveness of Tool Decoding by applying it to a variety of both general-purpose and tool-specialized models, evaluating them on the API-Bank and BFCL V2 • Live benchmarks. The experimental results show that Tool Decoding substantially improves performance across all models. Nearly all models achieve performance gains above 70% on both benchmarks. Among the 7B parameter models, five surpass GPT-3.5 in critical tasks, with two even outperforming GPT-4.

weaknesses:
1. Limitation of Novelty as a long paper: constrained decoding for tool usage in LLMs has been employed in several prior works (e.g., [Domino](https://arxiv.org/abs/2403.06988): is proposed to optimize for general constrained text generation tasks with a focus on grammar and token alignment, [TOOLDEC](https://arxiv.org/pdf/2310.07075): eliminates syntax errors by constraining token choices using FSMs, focusing on maintaining tool syntax), and the specific approach to constrained decoding in this paper does not differ significantly.  As a result, the primary novelty of this paper lies in the order consistency mechanism, which offers only a limited improvement (Table 6).

2. Missing Baselines: The tool-finetuned models presented in the results table are outdated, and more recent, more powerful models (e.g., [Hermes Function Calling](https://github.com/NousResearch/Hermes-Function-Calling) and [Gorilla](https://github.com/ShishirPatil/gorilla)) are not included for comparison.

questions:
1. Can you please further describe the differences of your approach (specially constrained decoding module) to previous work on constraint decoding for tool calling in LLMs?

2. Is there any technical reason that more recent tool-fine tuned models are not added for comparison?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671703563

thread_type:
official_review

<thread_object>
creator: reviewer_pjpg

<content>
summary:
This paper addresses the limitations of LLMs in tool usage. Previous works often overlook the specific requirements of tool selection, format adherence, and parameter accuracy. The authors propose a training-free approach called "Tool Decoding," designed to improve tool-use precision through constrained decoding. Experimental results demonstrate that Tool Decoding achieves improvements in tool-use accuracy across benchmarks (API-Bank and BFCL V2), with open-weight models performing comparably to GPT-4 in some cases. However, the paper lacks a thorough comparison with other existing methods (only compared with beam and greedy search) and provides no report on latency, which could be important for practical applications.

strengths:
- Achieved comparable results to GPT-4 for open-weight models with the help of Tool Decoding on the API-Bank and BFCL V2 benchmarks.
- The method does not require any training and primarily relies on constrained decoding to ensure format correctness and eliminate hallucinations.

weaknesses:
- Comparison with other methods? The authors compare their method only with Beam Search and Greedy Search. Are there no other methods available, such as those in https://arxiv.org/pdf/2401.06201 (01.2024), trained models, etc.?
- For the error analysis section, I believe it does not sufficiently acknowledge that other researchers have reported similar problems. It is written as though this paper is the first to report them. For example, the API-Bank [paper](https://arxiv.org/pdf/2304.08244) provides a thorough analysis at a similar level of granularity in Section 7.3, which I believe has not been adequately acknowledged.

questions:
- Value errors increased in most scenarios after applying Tool Decoding (add to Limitation section); on the other hand, key errors are not a major challenge. Tool Decoding primarily addresses format errors and partially resolves selection errors. I understand how it can help with selection errors by decoding tool names under constraints. However, for format errors, the example in Table 1 shows a missing parenthesis. How would Tool Decoding help in this case?

- Another important factor in decoding tools is latency in seconds, but I did not find any comparison. Although there is no major comparison with other methods, some latency figures for the tool itself would be beneficial.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732343410307

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A summary of paper updates

comment:
Following the reviewers’ suggestions, we have updated the paper with key revisions highlighted in orange. The main changes are as follows:
1. **Section 2:** Revised based on feedback from reviewer PjPG and reviewer CYKv.
2. **Section 4:** Presented results combining Tool Decoding with in-context learning.
3. **Appendix C:** Included latency analysis for our method.
4. **Appendix D:** Added error analysis for two 70B-level models.
5. **Appendix E:** Added results combining Tool Decoding with two 30B-level generalist models and one powerful tool-finetuned models.
6. **Appendix F:** Added comparisons with existing constrained decoding methods.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper breaks down LLMs' tool usage into three stages - tool awareness, selection, and invocation - and identifies common failure modes at each step. The authors propose Tool Decoding, a simple constrained sampling approach that improves tool usage without extra training. The method restricts vocabulary during tool selection and uses majority voting across different parameter orderings during invocation. A key advantage is that it only needs API documentation to work with new tools. Experiments on API-Bank and BFCL V2 show substantial gains across different LLMs, sometimes matching GPT-4's performance. 

In the initial round of reviews, the main criticism focused on the lack of comparative analysis. In the rebuttal, the authors expanded their discussion in Section 4 and Appendices E and F, adding comparisons with sampling methods like epsilon-sampling and with models fine-tuned for tool usage, including Gorilla-OpenFunctions, Toolformer, and DeepSeek-Coder. They also address reviewer fh4s’s concern by testing larger models in Appendix E. Another issue was the lack of latency analysis, which the authors have now fixed by providing detailed latency results in Appendix C.

After reviewing these revisions and engaging in follow-up discussions, all reviewers have agreed that the paper is suitable for acceptance.