ToolGen: Unified Tool Retrieval and Calling via Generation

Authors: Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, Haonan Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Agent, Tool Learning, Virtual Token

Abstract:
As large language models (LLMs) advance, their inability to autonomously execute tasks by directly interacting with external tools remains a critical limitation. Traditional methods rely on inputting tool descriptions as context, which is constrained by context length and requires separate, often inefficient, retrieval mechanisms. We introduce ToolGen, a paradigm shift that integrates tool knowledge directly into the LLM’s parameters by representing each tool as a unique token. This enables the LLM to generate tool calls and arguments as part of its next token prediction capabilities, seamlessly blending tool invocation with language generation.  Our framework allows the LLM to access and utilize a vast amount of tools with no additional retrieval step, significantly enhancing both performance and scalability. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents that can adapt to tools across diverse domains.  By fundamentally transforming tool retrieval into a generative process, ToolGen paves the way for more versatile, efficient, and autonomous AI systems. ToolGen enables end-to-end tool learning and opens opportunities for integration with other advanced techniques such as chain-of-thought and reinforcement learning, thereby expanding the practical capabilities of LLMs

Decision: accept (poster)
Reviewer scores: 5, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1730374760987

thread_type:
official_review

<thread_object>
creator: reviewer_a87g

<content>
summary:
This paper proposes ToolGen, a generative tool/function calling framework. The concrete methods are: (1) virtualizing tools by virtual tokens; (2) memorizing tools with training data of (tool docs, tool virtual tokens); (3) learning tool retrieval with training data of (user queries, tool virtual tokens); (4) and finally, finetuning tool agent with tool calling trajectories. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents that can adapt to tools across diverse domains.

strengths:
Overall, I like this paper very much.  

1. ToolGen is now paradigm for fundamentally transforming tool retrieval into a generative process, which is a very important topic for function calling or LLM-based agents.  
2. The methods are sound and resonable; the paper presentation is clear.  
3. The experimental results show that the methods are effective compared with several strong baselines.

weaknesses:
1. Traditional retrieval and generation methods for function calling can handle dynamic tools. If the tool set is changing, could ToolGen be used (without retraining)?  

2. A few reltated works are not mentioned or compared. For example, TPTU and TPTU-V2 [1,2,3,] used demo retriever and fintuner besides the tool retrieval, which may be more powerful than the traditional retrieval and generation methods. They can be a strong baseline for comparision.  

3. It seems that ToolGen uses a more complex process (i.e., (1) virtualizing tools by virtual tokens; (2) memorizing tools with training data of (tool docs, tool virtual tokens); (3) learning tool retrieval with training data of (user queries, tool virtual tokens); (4) and finally, finetuning tool agent with tool calling trajectories) to prepare for tool calling. Therefore, the suprior of ToolGen may come from more training of LLMs? Could the authors give more explination on this?


[1] TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents. FMDM Workshop at NeurIPS 2023.   
[2] TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems. LLMAgents Workshop at ICLR 2024.  
[3] TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Industry Systems. EMNLP 2024 Industry Track.

questions:
see the above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730570388360

thread_type:
official_review

<thread_object>
creator: reviewer_sicu

<content>
summary:
The paper introduces ToolGen, a novel framework designed to address the limitations of LLMs in interacting with an extensive array of external tools. This framework shifts existing paradigms by embedding knowledge about various tools directly into the LLM’s vocabulary, using unique tokens to represent each tool. This allows the model to generate tool calls and arguments as part of the language generation process, eliminating the need for separate tool retrieval systems and seamlessly integrating tool usage within LLM capabilities. ToolGen incorporates a three-stage training process to enable the LLM to learn how to use tokenized tools.

The main contribution of this paper is introducing the ToolGen framework including training and inferencing, and proving that ToolGen have comparable result with current tool retrieval systems.

strengths:
**Originality:**
The paper introduces a novel approach to tool retrieval by representing each tool as a unique virtual token directly integrated into the LLM’s vocabulary. This method eliminates the need for auxiliary retrievers, making the retrieval process more seamless and efficient. The concept of transforming tool retrieval into a generative task is novel and presents a possible solution to the scalability challenges faced by some existing methods.

**Quality:**
The authors have conducted a comprehensive set of experiments to validate their claims, showing thoroughness in their evaluation. These experiments include various comparisons and detailed ablation studies to assess the impact of different components of their approach. The robustness of their methodology enhances the credibility of their results and conclusions.

**Clarity:**
The paper is well-structured and written in a clear and concise manner. It systematically outlines the problem, the proposed solution, and the experimental validations. Each section transitions smoothly into the next, making it easy for the reader to follow the logic and understand the contributions of the work.

**Significance:**
The research addresses a critical issue in the field of LLM agents: managing and retrieving tools from a large set of tools efficiently. Given the increasing complexity and number of tools available, the proposed method provides a solution. This work has implications for the development of more autonomous and efficient AI systems, potentially benefiting some applications where tool interaction is essential.

weaknesses:
**Substantive Assessment of Weaknesses:**

**Cost and Efficiency Claims:**
The key claim of "significantly less cost and higher efficiency" does not hold up under scrutiny. The authors have not substantively demonstrated that their framework is less costly or more efficient than existing methodologies. The ToolGen framework necessitates a three-stage training process, which does not inherently suggest reduced costs. Furthermore, the paper lack data or experiments to substantiate the claim regarding efficiency improvements over other approaches.

**Role of the Memorization Stage:**
In Section 4.4, the table shows that the memorization stage plays a relatively minor role in the three-stage training process. However, the authors assert that this stage is beneficial for generalization, with further discussion supposedly found in Appendix F. There is, however, no such discussion present in Appendix F. I would recommend authors to do more experiments to validate the significance of the memorization stage and its impact on generalization.

**Hallucination Comparison:**
Section 5.4 contains statements that do not fully make sense. The claim that ToolGen experiences no hallucination is primarily due to the use of a constrained decoding strategy. Theoretically, it is impossible to encounter hallucination, as defined by the authors, under this constraint. This does not constitute a fair comparison with other frameworks. The authors should either provide results without this constraint and compare them or demonstrate why other frameworks are unable to implement a similar strategy.

**Performance Benchmarking:**
ToolGen is not currently the best-performing framework. Several baselines in the paper outperform ToolGen in various categories, as shown in Table 1 and Table 4. Notably, even the proposed but ultimately unused semantic tokenization approach outperforms the authors' current method, as shown in Table 2. This discrepancy confuses me.

questions:
- As stated earlier, why don't take the semantic representation approach as your final approach as it performs better (5 wins / 4 loses in table 2) than atomic approach? Saving tokens might not be a sufficient reason. Chain-of-thought reasoning process should produce a lot more than what atomic approach can save here.

- In Table 1: "For ToolGen in the In-Domain setting, we allow the generation space to include all tokens, which is a more challenging scenario compared to other models." What if you don't include all tokens? Since ToolGen can't outperform IterFeedback in the In-Domain setting currently, will it be able to outperform it without this additonal challange?

- In section 4.4,  the authors are assessing the impact of different training stages, but the table shows "−memorization", "−retrieval training", "−constraining". If I understand correctly, constraining is an decoding strategy used in inference stage, while the third training stage "end-to-end agent-tuning" does not show up.

Please also pay attention to the questions and suggestions in the "Weaknesses" part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662035467

thread_type:
official_review

<thread_object>
creator: reviewer_pqud

<content>
summary:
The paper introduces ToolGen, a novel framework that aims to enhance the interaction between large language models (LLMs) and external tools. ToolGen shifts away from traditional tool retrieval methods and instead integrates tool knowledge directly into the LLM's parameters. This is achieved by representing each tool as a unique token, allowing the LLM to generate tool calls and arguments seamlessly as part of its text generation process.

strengths:
1. ToolGen elegantly combines tool retrieval and execution into a single generative process, eliminating the need for separate retrieval mechanisms. This streamlines tool interaction and enhances efficiency, particularly as the number of tools increases.
2. The use of constrained beam search during inference effectively restricts the output to valid tool tokens, significantly reducing the generation of nonexistent tools, a common issue in LLM-based agents.
3. ToolGen demonstrates its capacity to effectively handle a large repository of over 47,000 real-world tools, highlighting its scalability compared to existing methods that struggle with vast tool sets.
4. The authors spend good amount of effort comparing different indexing method, and the result is clear.

weaknesses:
1. The advantage of ToolGen which combines tool retrieval and execution into a single generative process introduces limitation together with its efficiency. Since the tools are integrated into the system as tokens, the extension of new tools become inefficient. For every new tool/API, new token need to be added and the documentation finetuned into the model. Also, consider the case that when the tool/APIs get updated, the maintenance of all the tool/APIs, making sure they are up to date is a quite challenging task. On the contact, with a retriever would make adding and maintaining tool/APIs very easy.

questions:
1. The paper highlights the efficiency of atomic indexing due to each tool being represented by a single token. However, as the number of tools grows significantly larger, could this approach lead to vocabulary explosion issues? 
2. In the current setting, the vocabulary consists of 128K regular token and 47K tool/API tokens already, how does it affect the performance of LLM itself?
3. How does the performance of ToolGen vary with the size and complexity of the underlying LLM? Would a larger model lead to better generalization ability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716266852

thread_type:
official_review

<thread_object>
creator: reviewer_iej7

<content>
summary:
This paper proposes ToolGen, a finetuned LLMs that can use various tools during the conversations with the users. ToolGen incorporates new 47K tokens for tools into the Llama-3-8B. Through the tool virtualization, tool memorization, retrieval training, and end-to-end agent tuning, ToolGen correctly select the right tools in the context on  the ToolBench evaluation, achieving better performance than retrieval and end-to-end baselines.

strengths:
- [S1] ToolGen outperform or achieves competitive performance among retrieval and end-to-end baselines on ToolBench.
- [S2] ToolGen can natively invoke 47K tools following the context.

weaknesses:
- [W1] The technical novelty is limited. Using special tokens for tools and incorporating them into the original vocabularies are widely-known approach (e.g. Toolformer: https://arxiv.org/abs/2302.04761, ToolkenGPT: https://arxiv.org/abs/2305.11554). The contribution of this paper is scaling this up to 47K tools, but it's very straight forward and I'm not confident if the ICLR community would be interested in it.
- [W2] Releted to [W1], the results of ToolLlama-3 in Section 5 is unclear to me. Why ToolLlama-3 is not as good as ToolGen, even using the same data and models? Could you clarify the difference between two?
- [W3] How about retrieving tools by LLMs itself (without retriever)? For instance, if we use the LLM with millions of extremely long context (such as Gemini), we may not need to rely on neural retrievers. It would be interesting to include long-contect LLMs as a baseline.
- [W4] It would be important to compare other capability of LLMs before/after tool token incorporation. Current draft seems to lack the analysis after combining 47K tokens to LLMs.
- [W5] In Figure1, I didn't understand the distinction between ToolGen's Retrieval Task and Agent Task. Both have the input "I want some popular video games." How are they differentiated?

questions:
See the Weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733212780385

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all reviewers for their detailed reviews and insightful feedback. Their comments will help to make our work more solid and comprehensive. Below is the summary of new results and updates made based on feedback from reviewers:

- We analyzed the latency and cost for ToolGen and different baselines, showing that ToolGen is significantly more effective compared to same performance-level baselines.
- We added discussions on tool maintenance and tool set change in Appendix B.
- We discussed how the general capabilities change in Appendix F.
- We added long-context LLM baseline in experiments.
- We compared atomic and semantic encoding to show atomic encoding's superiority in efficiency, hallucination and performance.
- We added and discussed how ToolGen's performance and generalization changes as the size changes in Appendix H.
- We added more comprehensive experiments and discussion to emphasize the role of tool memorization stage.
- We added more related work.

**Further clarification about our contributions**

First, we express our gratefulness to Revewer A87g, who has given highly positive and encouraging feedback to our work. We really appreciate Reviewer A87g's support for our work and we discuss more about our contribution and novelty here, which is also still concerned by Reviewer ieJ7:
- While the scalability to 47K tools is a notable aspect, we emphasize that ToolGen is not just about scaling but about redefining tool retrieval and usage as a unified generative task, which is also stated by Reviewer A87g. This shift addresses practical challenges, such as context length limitations and tool hallucinations, with a systematically designed pipeline integrating memorization, retrieval, and end-to-end tuning.
- Regarding scalability, while further expansions in tool tokens might appear as a natural progression, the methodology and infrastructure we propose (e.g., constrained decoding, multi-stage training) offer **first** foundational insights that extend beyond mere scaling and are applicable to future challenges in integrating even broader sets of tools.
- Regarding the interests of the community, the ToolGen framework is not confined to Tool Learning itself. In fact, it contributes to broad research topics.  For example, in areas which can be enhanced significantly by incorporating tools, such as reasoning, question-answering, information retrieval, RLHF, optimization is often hindered by the separation between tool selection and tool usage. ToolGen offers an option to internalize related tools into their parameters, therefore enabling end-to-end approaches for them to integrate massive and complex tools during training; In In-Context Learning, ToolGen makes it possible to do few-shot generation with a large set of available tools.

Therefore, we believe that ToolGen not only addresses existing challenges but also contributes to other research directions and inspires future works. We thank all reviewers' efforts to help improve our work and we are committed to incorporating further insights and feedback to enhance the impact and rigor of our contributions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces ToolGen, a novel framework that integrates tool knowledge directly into the parameters of a Large Language Model (LLM) by representing each tool as a unique token.  This allows the LLM to generate tool calls and arguments seamlessly, eliminating the need for separate tool retrieval systems.  ToolGen is trained in three stages: tool memorization, retrieval training, and end-to-end agent tuning.  The authors claim this framework enhances the interaction between LLMs and external tools, particularly with large tool sets.    

Strengths: The paper proposes a novel approach to tool retrieval and execution by representing tools as virtual tokens, streamlining the process.  The use of constrained beam search effectively reduces the generation of nonexistent tools.  The authors demonstrate the scalability of ToolGen by incorporating over 47,000 real-world tools, and how it can more easily deal with context length limitations and tool hallucinations.

Weaknesses: (1) Dealing with new tools at test-time or major updates in tool APIs would necessitate retraining, which has been added as a discussion in appendix.  (2) Degradation in general instruction following performance of LLMs when fine-tuning heavily on tool use trace, which would be critical for practical deployment. (3) Post-hoc tool use is common for deployment due to latency introduced by tool calls during generation, which might need to carefully handled with ToolGen.

Reasons for Acceptance/Rejection: The paper present a novel perspective on tool use -- generative tool calling - which can tap into the generation abilities of LLMs. Moreover, the authors have addressed the reviewers' concerns by providing additional explanations and committing to further experiments. The revised version of the paper, incorporating the planned improvements, is expected to be a valuable contribution to ICLR.

Minor suggestion: Work in similar spirit to ToolGen has been going on in generative verifiers / reward models (GenRM, Cloud) which would be worth discussing.