ToolGen: Unified Tool Retrieval and Calling via Generation

## Abstract

Abstract As large language models (LLMs) advance, their inability to autonomously execute tasks by directly interacting with external tools remains a critical limitation. Traditional methods rely on inputting tool descriptions as context, which is constrained by context length and requires separate, often inefficient, retrieval mechanisms. We introduce ToolGen, a paradigm shift that integrates tool knowledge directly into the LLM’s parameters by representing each tool as a unique token. This enables the LLM to generate tool calls and arguments as part of its next token prediction capabilities, seamlessly blending tool invocation with language generation.
Our framework allows the LLM to access and utilize a vast amount of tools with no additional retrieval step, significantly enhancing both performance and scalability. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents that can adapt to tools across diverse domains.
By fundamentally transforming tool retrieval into a generative process, ToolGen paves the way for more versatile, efficient, and autonomous AI systems. ToolGen enables end-to-end tool learning and opens opportunities for integration with other advanced techniques such as chain-of-thought and reinforcement learning, thereby expanding the practical capabilities of LLMs 1 1 1 Code and data are available at: https://github.com/Reason-Wang/ToolGen .

## 1 Introduction

Large language models (LLMs) have demonstrated impressive capabilities as interactive systems, adept at processing external inputs, executing actions, and autonomously completing tasks (Gravitas, ; Qin et al., ; Yao et al., ; Shinn et al., ; Wu et al., ; Liu et al., ). Among the various methods enabling LLMs to interact with the world, tool calling via APIs has emerged as one of the most common and effective approaches. However, as the number of tools grows into the tens of thousands, existing methods for tool retrieval and execution struggle to scale efficiently.

A common approach in real-world scenarios is to combine tool retrieval with tool execution, where a retrieval model first narrows down the relevant tools before passing them to the LLM for final selection and execution (Qin et al., ; Patil et al., ). While this combined method addresses the challenge of handling vast numbers of tools, it has notable limitations: retrieval models often rely on small encoders that fail to fully capture the semantics of complex tools and queries, and separating retrieval from execution introduces inefficiencies and potential misalignment between stages of task completion.

Moreover, LLMs and their tokenizers are pretrained primarily on natural language data (Brown et al., ; Touvron et al., ), leaving them with limited intrinsic knowledge of tool-related functionalities. This gap in knowledge results in suboptimal performance, especially when the LLM must rely on retrieved tool descriptions for decision-making.

In this study, we introduce ToolGen, a novel framework that integrates real-world tool knowledge directly into the LLM’s parameters and transforms tool retrieval and execution into a unified generation task. Specifically, ToolGen expands the LLM’s vocabulary with tool-specific virtual tokens and trains the model to generate these tokens within a conversational context, allowing the LLM to leverage its pre-existing knowledge more effectively for both retrieving and calling tools.

Specifically, each tool is represented as a unique virtual token within the LLM’s vocabulary. Building upon a pretrained LLM, ToolGen’s training process consists of three stages: tool memorization, retrieval training, and agent training. In the tool memorization stage, the model associates each virtual tool token with its documentation. During retrieval training, the model learns to generate relevant tool tokens based on user queries. Finally, in end-to-end agent-tuning, the model is trained to act as an autonomous agent, generating plans and tools, and determining the appropriate parameters to complete tasks. By calling tools and receiving feedback from external environments, the model can handle user queries efficiently and integratively. Figure 1 shows comparison between ToolGen and traditional paradigms.

We demonstrate ToolGen’s superiority in two scenarios: a tool retrieval task, where the model retrieves the correct tool for a given query, and an LLM-based agent task, where the model completes complex tasks involving real-world API calls. Leveraging a dataset of 47,000 real-world tools, ToolGen achieves performance comparable to the leading tool retrieval methods, but with significantly lower cost and greater efficiency. Additionally, it surpasses traditional tool learning paradigms, highlighting its potential for advancing more effective tool usage systems.

Figure: Figure 1: Comparison between previous retrieval-based methods and our ToolGen. Previous methods use a retriever to retrieve relevant tools based on similarity matching, which are further put into prompts for LLMs to select. ToolGen can retrieve tools by generating tool tokens directly. ToolGen can also complete the task without relying on any external retriever.
Refer to caption: x1.png

ToolGen represents a paradigm shift in tool interaction by merging retrieval and generation into a single, cohesive model. This innovation sets the stage for a new generation of AI agents capable of adapting to a vast array of tools across diverse domains.
Additionally, ToolGen opens new opportunities for integrating advanced techniques like chain-of-thought reasoning and reinforcement learning with the ability to use tools in a unified generation way, expanding the capabilities of LLMs in real-world applications.

In summary, our contributions are:

- •
A novel framework, ToolGen, that integrates tool retrieval and execution into the LLM’s generative process using virtual tokens.
- •
A three-stage training process that enables efficient and scalable tool retrieval and API calling within ToolGen.
- •
Experimental validation demonstrates that ToolGen achieves comparable performance to current best tool retrieval methods with significantly less cost and higher efficiency and surpasses traditional tool learning paradigms across large-scale tool repositories.

## 2 Related Work

### 2.1 Tool Retrieval

Tool retrieval is essential for LLM agents in real-world task execution, where tools are usually represented by their documentation. Traditional methods like sparse (e.g., BM25 (Robertson et al., )) and dense retrieval (e.g., DPR (Karpukhin et al., ), ANCE (Xiong et al., )) rely on large document indices and external modules, leading to inefficiencies and difficulty in optimizing in an end-to-end agent framework.
Some work has explored alternative methods. For example, Chen et al. () rewrite queries and extract their intent, targeting unsupervised retrieval settings, though the results are not comparable to supervised approaches. Xu et al. () propose a method that iteratively refines queries based on tool feedback, improving retrieval accuracy but increasing latency.

Recently, generative retrieval has emerged as a promising new paradigm, wherein models directly generate relevant document identifiers rather than relying on traditional retrieval mechanisms (Wang et al., ; Sun et al., ; Kishore et al., ; Mehta et al., ; Chen et al., ). Motivated by this, ToolGen represents each tool as a unique token, allowing tool retrieval and calling to be framed as a generation task.
Beyond simplifying retrieval, this design integrates smoothly with other LLM and LLM-based agent features like chain-of-thought reasoning (Wei et al., ) and ReAct (Yao et al., ). By consolidating retrieval and task execution into a single LLM agent, it reduces latency and computational overhead, leading to more efficient and effective task completion.

### 2.2 LLM-Agents with Tool Calling

LLMs have shown strong potential in mastering tools for various tasks. However, most existing works focus on a limited set of actions (Chen et al., ; Zeng et al., ; Yin et al., ; Wang et al., ). For instance, Toolformer (Schick et al., ) fine-tunes GPT-J to handle just five tools, such as calculators. While effective for narrow tasks, this approach struggles in real-world scenarios with vast action spaces. ToolBench (Qin et al., ) expands the scope by introducing over 16,000 tools, highlighting the challenge of tool selection in complex environments.

To perform tool selection, current methods often use a retriever-generator pipeline, where relevant tools are retrieved and then utilized by the LLM (Patil et al., ; Qin et al., ). However, pipelined approaches face two major issues: error propagation from the retrieval step and the inability of LLMs to fully understand and use tools via simple prompting.

To mitigate these issues, researchers have tried representing actions as tokens, converting action prediction into a generative task. For example, RT2 (Brohan et al., ) generates tokens representing robot actions, and Self-RAG (Asai et al., ) uses special tokens to decide when to retrieve documents. ToolkenGPT (Hao et al., ) introduces tool-specific tokens to trigger tool usage, a concept closest to our approach.

Our approach differs from ToolkenGPT in several ways. First, we focus on real-world tools that require flexible parameters for complex tasks (e.g., YouTube channel search), while ToolkenGPT is limited to simpler tools with fewer inputs (e.g., math functions with two numbers). Additionally, ToolkenGPT relies on few-shot prompting, whereas ToolGen incorporates tool knowledge directly into the LLM through full-parameter fine-tuning, enabling the model to retrieve and execute tasks autonomously. Finally, our experiments involve a much larger tool set—47,000 tools compared to ToolkenGPT’s 13–300.

Other studies, such as ToolPlanner (Wu et al., ) and AutoACT (Qiao et al., ), have used reinforcement learning or developed multi-agent systems to enhance tool learning or task completion (Qiao et al., ; Liu et al., ; Shen et al., ; Chen et al., ).
We do not compare our model with these approaches for two reasons: (1) Most of these works rely on feedback mechanisms, either through Reflection (Shinn et al., ) or a reward model, which is similar to ToolBench’s evaluation design, where an LLM serves as an evaluator without access to ground truth answers. However, this is not the focus of our study, and our end-to-end experiment does not rely on such feedback mechanisms. (2) Our method is not in conflict with these approaches; instead, they can be integrated. Exploring this integration is left for future work.

## 3 ToolGen

In this section, we first introduce the notations used throughout the paper. Then we detail the specific methods of ToolGen, including tool virtualization, tool memorization, retrieval training, and end-to-end agent tuning, as illustrated in Figure 2. Lastly, we describe our inference approach.

Figure: Figure 2: An illustration of ToolGen framework. In tool virtualization, tools are mapped into virtual tokens. In the following three-stage training, ToolGen first memorizes tools by predicting tool tokens based on their documentations. Then it learns to retrieve tools by predicting tool tokens from queries. Finally, pipeline data, i.e., trajectories, are used to finetune the retriever model from the last stage, resulting in the ToolGen Agent model.
Refer to caption: x2.png

### 3.1 Preliminaries

Given a user query $q$, tool learning aims to resolve $q$ using tools from a large tool set $D=\{d_{1},d_{2},\ldots,d_{N}\}$, where $|D|=N$ is a large number, making it impractical to include all tools in $D$ in the LLM context. Therefore, current research typically uses a retriever $R$ to retrieve $k$ relevant tools from $D$, denoted as $D_{k,R}=\{d_{r_{1}},d_{r_{2}},\ldots,d_{r_{k}}\}=R(q,k,D)$, where $|D_{k,R}|\ll N$. The final prompt is then the concatenation of $q$ and $D_{k,R}$, denoted as
$Prompt=[q,D_{k,R}]$. To complete a task (query), an LLM-based agent usually adopts a four-stage paradigm (Qu et al., ) iteratively: generates a plan $p_{i}$, selects a tool $d_{si}$, determines tool parameters $c_{i}$, and collects feedback from the tool(s) $f_{i}$. We denote these steps for the $i$-th iteration as $p_{i},d_{s_{i}},c_{i},f_{i}$. The model continues iterating through these steps until the task is completed, at which point the final answer $a$ is generated. The entire trajectory can be represented as $Traj=[Prompt,(p_{1},d_{s_{1}},c_{1},f_{1}),\ldots,(p_{t},d_{s_{t}},c_{t},f_{t}
),a]=[q,R(q,D),(p_{1},d_{s_{1}},c_{1},f_{1}),\ldots,(p_{t},d_{s_{t}},c_{t},f_{
t}),a]$. This iterative approach allows the model to dynamically adjust and refine its actions at each step based on the feedback received, improving its performance in completing complex tasks.

### 3.2 Tool Virtualization

In ToolGen, we virtualize tools by mapping each tool to a unique new token through a method we call atomic indexing. In this approach, each tool is assigned a unique token by expanding the LLM’s vocabulary. The embedding for each tool token is initialized as the average embedding of its corresponding tool name, ensuring a semantically meaningful starting point for each tool.

Formally, the token set is defined as $T={\mathrm{Index}(\mathrm{d})\ |\ \forall d\in D}$, where $\mathrm{Index}$ is the function mapping tools to tokens. We demonstrate that atomic indexing is more efficient and can mitigate hallucination compared to other indexing methods, such as semantic and numeric mappings, discussed in Section 4.3 and 5.4.

### 3.3 Tool Memorization

After assigning tokens to tools, the LLM still lacks any knowledge of the tools. To address this, we inject tool information by fine-tuning it with tool descriptions as inputs and their corresponding tokens as outputs, which we call tool memorization. We use the following loss function:

$$ $\mathcal{L}_{tool}=\sum_{d\in D}-\log p_{\theta}(\mathrm{Index}(d)|d_{doc})$ $$

where $\theta$ denotes the LLM parameters, and $d_{doc}$ represents the tool description. This step equips the LLM with basic knowledge of the tools and their associated actions.

### 3.4 Retrieval Training

We then train the LLMs to link the hidden space of virtual tool token (and its documentation), to the user query space, so that LLM can generate correct tool based on a user’s query. To achieve this, we fine-tune the LLM with user queries as inputs and corresponding tool tokens as outputs:

$$ $\mathcal{L}_{retrieval}=\sum_{q\in Q}\sum_{d\in D_{q}}-\log p_{\theta^{\prime} }(\mathrm{Index}(d)|q)$ $$

where $\theta^{\prime}$ represents the LLM parameters after tool memorization, $Q$ is the set of user queries, and $D_{q}$ is the set of tools relevant to each query. This results in the ToolGen Retriever, which can generate the appropriate tool token given a user query.

### 3.5 End-to-End Agent-Tuning

After retrieval training, the LLM is capable of generating tool tokens from queries. In the final stage, we fine-tune the model with agent task completion trajectories. We adopt a similar inference strategy as Agent-Flan (Chen et al., ), in instead of generating Thought, Action, and Arguments together as ReAct. Our pipeline follows an iterative process, where the LLM first generates a Thought, and the corresponding Action token. This token is used to fetch the tool documentation, which the LLM uses to generate the necessary arguments. The process continues iteratively until the model generates a “finish” token or the maximum number of turns is reached. The generated trajectory is represented as $Traj=[q,(p_{1},\mathrm{Index}(d_{s_{1}}),c_{1},f_{1}),\ldots,(p_{t},\mathrm{
Index}(d_{s_{t}}),c_{t},f_{t}),a]$. In this structure, relevant tools are no longer required.

### 3.6 Inference

During inference, the LLM may generate action tokens outside the predefined tool token set. To prevent this, we designed a constrained beam search generation that restricts the output tokens to the tool token set. We applied this constrained beam search for both tool retrieval, where the model selects tools based on queries, and the end-to-end agent system, significantly reducing hallucination during the action generation step. A detailed analysis can be found in Section 5.4. The implementation details can be found in Appendix C.

## 4 Tool Retrieval Evaluation

### 4.1 Experimental Setup

We use pretrained Llama-3-8B (Dubey et al., ) as our foundation model, with a vocabulary size of 128,256. Using the atomic indexing approach, we expand the vocabulary by an additional 46,985 tokens following the tool virtualization process, resulting in a final vocabulary size of 175,241. We fine-tune the model using the Llama-3 chat template with a cosine learning rate scheduler, applying a 3% warm-up steps. The maximum learning is $4\times 10^{-5}$. All models are trained using Deepspeed ZeRO 3 (Rajbhandari et al., ) across 4$\times$A100 GPUs. We train 8 epochs for tool memorization and 1 epoch for retrieval training.

#### Dataset

Our experiments are based on ToolBench, a real-world tool benchmark containing more 16k tool collections, each containing several APIs, resulting in a total of 47k unique APIs. Each API is documented with a dictionary, containing the name, description, and parameters for calling the API. A real example is shown in Appendix A. We take each API as an action and map it to a token. Our retrieval and end-to-end agent-tuning data are converted from the original data in ToolBench. Details can be found in Appendix G. Although each tool may consist of multiple APIs, for simplicity, we refer to each API as a tool in this paper.

We follow the data split of Qin et al. (), where 200k (query, relevant API) pairs are divided into three categories: I1 (single-tool queries), I2 (intra-category multi-tool queries), and I3 (intra-collection multi-tool instructions), containing 87,413, 84,815, and 25,251 instances, respectively.

#### Baselines

We compare ToolGen with the following baselines:

- •
BM25: A classical unsupervised retrieval method based on TF-IDF, which retrieves documents based on term similarity with the query.
- •
Embedding Similarity (EmbSim): Sentence embeddings generated using OpenAI’s sentence embedding model; specifically text-embedding-3-large in our experiences.
- •
Re-Invoke (Chen et al., ): An unsupervised retrieval method with query rewriting and document expansion.
- •
IterFeedback (Xu et al., ): BERT-based retriever with gpt-3.5-turbo-0125 as a feedback model with iterative feedback for up to 10 rounds.
- •
ToolRetriever (Qin et al., ): A BERT-based retriever trained via contrastive learning.

#### Settings

We conduct experiments under two settings. In the first, In-Domain Retrieval, the retrieval search space is restricted to tools within the same domain. For example, when evaluating queries from domain I1, the search is limited to I1 tools. This aligns with ToolBench settings. The second, Multi-Domain Retrieval, is more complex, with the search space expanded to include tools from all three domains. In this case, models are trained on combined data, increasing both the search space and task complexity. Unlike ToolBench, this multi-domain setting reflects real-world scenarios where retrieval tasks may involve overlapping or mixed domains. This setup evaluates the model’s ability to generalize across domains and handle more diverse, complex retrieval cases.

#### Metrics

We evaluate retrieval performance using Normalized Discounted Cumulative Gain (NDCG) (Järvelin & Kekäläinen, ), a widely used metric in ranking tasks, including tool retrieval. NDCG accounts for both the relevance and ranking position of retrieved tools.

**Table 1: Tool retrieval evaluation across two settings: (1) In-Domain, where models are trained and evaluated within the same domain; and (2) Multi-Domain, where models are trained on all domains and evaluated with the full set of tools across all domains. BM25, EmbSim, and Re-Invoke are unsupervised baselines without training. IterFeedback is retrieval system with multiple models and feedback mechanism. ToolRetriever is trained using contrastive learning, while ToolGen is trained with next-token prediction. Results marked with * were not implemented by us and are copied from their original paper, and hence only in the In-Domain setting. For ToolGen in the In-Domain setting, we allow the generation space to include all tokens, which is a more challenging scenario compared to other models. Best results in each category are bolded.**
| Model | I1 | I2 | I3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NDCG1 | NDCG3 | NDCG5 | NDCG1 | NDCG3 | NDCG5 | NDCG1 | NDCG3 | NDCG5 |  |
|  | In-Domain |  |  |  |  |  |  |  |  |
| BM25 | 29.46 | 31.12 | 33.27 | 24.13 | 25.29 | 27.65 | 32.00 | 25.88 | 29.78 |
| EmbSim | 63.67 | 61.03 | 65.37 | 49.11 | 42.27 | 46.56 | 53.00 | 46.40 | 52.73 |
| Re-Invoke* | 69.47 | – | 61.10 | 54.56 | – | 53.79 | 59.65 | – | 59.55 |
| IterFeedback* | 90.70 | 90.95 | 92.47 | 89.01 | 85.46 | 87.10 | 91.74 | 87.94 | 90.20 |
| ToolRetriever | 80.50 | 79.55 | 84.39 | 71.18 | 64.81 | 70.35 | 70.00 | 60.44 | 64.70 |
| ToolGen | 89.17 | 90.85 | 92.67 | 91.45 | 88.79 | 91.13 | 87.00 | 85.59 | 90.16 |
|  | Multi-Domain |  |  |  |  |  |  |  |  |
| BM25 | 22.77 | 22.64 | 25.61 | 18.29 | 20.74 | 22.18 | 10.00 | 10.08 | 12.33 |
| EmbSim | 54.00 | 50.82 | 55.86 | 40.84 | 36.67 | 39.55 | 18.00 | 17.77 | 20.70 |
| ToolRetriever | 72.31 | 70.30 | 74.99 | 64.54 | 57.91 | 63.61 | 52.00 | 39.89 | 42.92 |
| ToolGen | 87.67 | 88.84 | 91.54 | 83.46 | 86.24 | 88.84 | 79.00 | 79.80 | 84.79 |

### 4.2 Results

Table 1 presents the tool retrieval results. As expected, all trained models significantly outperform the untrained baselines (BM25, EmbSim, and Re-Invoke) across all metrics, demonstrating the benefit of training on tool retrieval data.

Our proposed ToolGen model consistently achieves the best performance across both settings. In the In-Domain setting, ToolGen delivers highly competitive results, achieving comparable performance to the IterFeedback system, which uses multiple models and a feedback mechanism. ToolGen, as a single model, outperforms ToolRetriever by a significant margin in all metrics and even surpasses IterFeedback in several cases, such as NDCG@5 for domain I1 and NDCG@1,@3,@5 for I2.

In the Multi-Domain setting, where the search space is larger and performance generally drops, ToolGen remains robust, outperforming ToolRetriever and maintaining superiority over other baselines. This demonstrates that ToolGen, despite being a single model, is capable of competing with complex retrieval systems like IterFeedback, showcasing its ability to handle complex real-world retrieval tasks where domain boundaries are less defined.

### 4.3 Indexing Method Comparison

While ToolGen uses atomic indexing for tool virtualization, we explore several alternative generative retrieval approaches. In this section, we compare it with the following three methods:

- •
Numeric: Map each tool to a unique number. The resulting token is purely numeric, offering no inherent semantic information, but providing a distinct identifier for each tool.
- •
Hierarchical: This method clusters tools into non-overlapping groups and recursively partitions these clusters, forming a hierarchical structure. The index from the root to the leaf in this tree-like structure represents each tool, similarly to Brown clustering techniques.
- •
Semantic: In this approach, each tool is mapped to its name, using the semantic content of the tool names to guide the LLM. The tool’s name provides a meaningful representation directly related to its function.

Figure: Figure 3: The distribution of the number of subtokens per tool varies across different indexing methods. Atomic indexing ensures that each tool is a single token, while numeric indexing encodes tools into N tokens for tools numbered in $(10^{N-1},10^{N}]$. In contrast, both semantic indexing and hierarchical indexing produce a variable number of subtokens, with semantic indexing having more outliers with significantly longer sequences.
Refer to caption: x3.png

The implementation details are described in Appendix B.

First, we conducted an analysis of the number of subtokens required to represent each tool for the different methods, as shown in Figure 3. The figure highlights the superiority of atomic indexing, where each tool is represented by a single token, whereas other methods require multiple tokens. This efficiency allows ToolGen to reduce the number of generation tokens and inference time in both the retrieval and agent scenarios.

Next, we examined the effectiveness of different indexing methods. As shown in Table 2, semantic indexing demonstrates the best retrieval performance across various metrics and scenarios, while atomic indexing closely follows in many cases. We attribute this to the fact that semantic indexing aligns better with the pretraining data of LLMs. However, this advantage diminishes as the training data and type increase. For example, in Section 5.3, we show that atomic indexing achieves better end-to-end results. Taking all these factors into account, we choose atomic indexing for ToolGen tool virtualization.

**Table 2: Retrieval evaluation for different indexing methods in Multi-Domain setting. Best results are bolded and second best results are underlined.**
| Model | I1 | I2 | I3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NDCG1 | NDCG3 | NDCG5 | NDCG1 | NDCG3 | NDCG5 | NDCG1 | NDCG3 | NDCG5 |  |
| Numetric | 83.17 | 84.99 | 88.73 | 79.20 | 79.23 | 83.88 | 71.00 | 74.81 | 82.95 |
| Hierarchical | 85.67 | 87.38 | 90.26 | 82.22 | 82.70 | 86.63 | 78.50 | 79.47 | 84.15 |
| Semantic | 89.17 | 91.29 | 93.29 | 83.71 | 84.51 | 88.22 | 82.00 | 78.86 | 85.43 |
| Atomic | 87.67 | 88.84 | 91.54 | 83.46 | 86.24 | 88.84 | 79.00 | 79.80 | 84.79 |

**Table 3: Ablation study for tool retrieval. We assess the impact of removing retrieval training, tool memorization, and constrained beam search on ToolGen’s performance, respectively.**
| Model | I1 | I2 | I3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NDCG1 | NDCG3 | NDCG5 | NDCG1 | NDCG3 | NDCG5 | NDCG1 | NDCG3 | NDCG5 |  |
| ToolGen | 87.67 | 88.84 | 91.54 | 83.46 | 86.24 | 88.84 | 79.00 | 79.80 | 84.79 |
| $-$memorization | 84.00 | 86.77 | 89.35 | 82.21 | 83.20 | 86.78 | 77.00 | 77.71 | 84.37 |
| $-$retrieval training | 10.17 | 12.31 | 13.89 | 05.52 | 07.01 | 07.81 | 03.00 | 04.00 | 04.43 |
| $-$constraining | 87.67 | 88.79 | 91.45 | 83.46 | 86.24 | 88.83 | 79.00 | 79.93 | 84.92 |

### 4.4 Ablation

We perform an ablation study to assess the impact of different training stages of ToolGen, as shown in Table 3. The results indicate that retrieval training is the crucial factor for tool retrieval performance, as it directly aligns with the retrieval task where inputs are queries and outputs are tool tokens.
Removing tool memorization leads to a minor performance drop although it plays a role in improving generalization, which we will discuss further in Appendix F.
Similarly, constrained beam search, while not a major contributor to retrieval task, helps prevent hallucinations, making it useful for end-to-end agent tasks, see Section 5.4.

## 5 End-to-End Evaluation

### 5.1 Experimental Setup

We make several modifications to the trajectory data from ToolBench to fit it into ToolGen framework. For example, as ToolGen does not require explicit selection of related tools as input, we remove this information in the system prompt. Further details are provided in Appendix G. Following this, we fine-tune the retrieval model using the reformatted data, resulting in an end-to-end ToolGen agent.

#### Baselines

- •
GPT-3.5: We use gpt-3.5-turbo-0613 as one of our baselines. The implementation is the same as used in StableToolBench (Guo et al., ), where the tool calling capability of GPT-3.5 is used to form a tool agent.
- •
ToolLlama-2: Qin et al. () introduced ToolLlama-2 by fine-tuning Llama-2 (Touvron et al., ) model on ToolBench data.
- •
ToolLlama-3: To ensure a fair comparison, we fine-tuned Llama-3, the same base model used in ToolGen, on the ToolBench dataset, creating the ToolLlama-3 baseline. In the rest of this paper, we refer to ToolLlama-3 as ToolLlama to distinguish it from ToolLlama-2.

#### Settings

- •
w/ Ground Truth Tools (G.T.) Following Qin et al. (), we define ground truth tools for a query as those selected by ChatGPT. For ToolLlama, we directly input the ground truth tools in the prompt, consistent with its training data format. For ToolGen, which is not trained on data with pre-selected tools, we add a prefix during the planning phase: I am using the following tools: [tool tokens], where [tool tokens] are virtual tokens corresponding to the ground-truth tools.
- •
w/ Retriever In the end-to-end experiments, we use a retrieval-based setting. For baselines, we use the tools retrieved by ToolRetriever as the relevant tools. In contrast, ToolGen generates tool tokens directly, so no retriever is used.

All models are finetuned using a cosine scheduler with maximum learning rate set to $4\times 10^{-5}$. Context length is truncated to 6,144. The total batch size is set to 512. We further use Flash-Attention (Dao et al., ; Dao, ) and Deepspeed ZeRO 3 (Rajbhandari et al., ) to save memory.

ToolGen and ToolLlama follow different paradigms to complete tasks. ToolLlama generates Thought, Action, and Parameters in a single round, while ToolGen separates these steps. For ToolGen, we set a maximum of 16 turns, which allows for 5 rounds of actions and 1 final round for providing the answer. We compare this to ToolLlama, which operates with a 6-turn limit.

Additionally, we introduce a retry mechanism for all models to prevent early termination, the details are introduced in Section D. Specifically, if a model generates a response containing give up or I’m sorry, we prompt the model to regenerate the response with a higher temperature.

#### Metrics

For end-to-end evaluation, we use StableToolBench (Guo et al., ), a stabilized tool evaluation benchmark that selects solvable queries from ToolBench and uses GPT-4 (OpenAI, ) to simulate outputs for failed tools. We employ two metrics to assess performance: Solvable Pass Rate (SoPR), which is the percentage of queries successfully solved, and Solvable Win Rate (SoWR), which indicates the percentage of answers outperforming those generated by a reference model (GPT-3.5 in this study). Additionally, we provide micro-average scores for each category.

**Table 4: End-to-end evaluation performance on unseen instructions under two settings. For R. setting, GPT3.5 and ToolLlama use ToolRetriever, while ToolGen does not use external retriever. For all results, SoPR and SoWR are evaluated three time and reported with mean values.**
| Model | SoPR | SoWR |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| I1 | I2 | I3 | Avg. | I1 | I2 | I3 | Avg |  |
|  | w/ Ground Truth Tools (G.T.) |  |  |  |  |  |  |  |
| GPT-3.5 | 56.60 | 47.80 | 54.64 | 50.91 | - | - | - | - |
| ToolLlama-2 | 53.37 | 41.98 | 46.45 | 48.43 | 47.27 | 59.43 | 27.87 | 47.58 |
| ToolLlama | 55.93 | 48.27 | 52.19 | 52.78 | 50.31 | 53.77 | 31.15 | 47.88 |
| ToolGen | 61.35 | 49.53 | 43.17 | 54.19 | 51.53 | 57.55 | 31.15 | 49.70 |
|  | w/ Retriever (R.) |  |  |  |  |  |  |  |
| GPT-3.5 | 51.43 | 41.19 | 34.43 | 45.00 | 53.37 | 53.77 | 37.70 | 50.60 |
| ToolLlama-2 | 56.13 | 49.21 | 34.70 | 49.95 | 50.92 | 53.77 | 21.31 | 46.36 |
| ToolLlama | 54.60 | 49.96 | 51.37 | 51.55 | 49.08 | 61.32 | 31.15 | 49.70 |
| ToolGen | 56.13 | 52.20 | 47.54 | 53.28 | 50.92 | 62.26 | 34.42 | 51.51 |

### 5.2 Results

Table 4 presents the end-to-end evaluation performance of various models in two settings: using Ground Truth Tools (G.T.) and a Retriever (R.). In the G.T. setting, ToolGen achieves the best average SoPR score of 54.19, outperforming GPT-3.5 and ToolLlama, with SoWR also highest for ToolGen at 49.70. In the Retriever setting, ToolGen maintains its lead with an average SoPR of 53.28 and SoWR of 51.51. ToolLlama shows competitive performance, surpassing ToolGen on some individual instances. An ablation study of end-to-end ToolGen is provided in Appendix G

**Table 5: End-to-end evaluation for different indexing methods.**
| Indexing | SoPR | SoWR |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| I1 | I2 | I3 | Avg. | I1 | I2 | I3 | Avg |  |
| Numeric | 34.76 | 29.87 | 46.99 | 35.45 | 25.77 | 33.02 | 29.51 | 28.79 |
| Hierarchical | 50.20 | 45.60 | 32.79 | 45.50 | 38.04 | 43.40 | 29.51 | 38.18 |
| Semantic | 58.79 | 45.28 | 44.81 | 51.87 | 49.69 | 57.55 | 26.23 | 47.88 |
| Atomic | 58.08 | 56.13 | 44.81 | 55.00 | 47.85 | 57.55 | 29.51 | 47.58 |

### 5.3 Indexing Method Comparison

Similar to indexing method comparison for retrieval task (Section 4.3), Table 5 presents a comparison of different indexing methods for the end-to-end agent task. In this setting, constrained decoding is removed, allowing the agent to freely generate Thought, Action, and Parameters. From the results, we observe that the Atomic method achieves the best performance among the four indexing methods. We attribute this to the higher hallucination rates in the other methods, as discussed in Section 5.4.

Figure: Figure 4: The hallucination rates of generating nonexistent tools across different models are shown. ToolGen does not generate any nonexistent tools when using constrained decoding. However, without this constraint, ToolGen generates 7% non-tool tokens during the Action generation stage with atomic indexing, and even more with semantic indexing. For ToolLlama and GPT-3.5, despite being provided with five ground truth tools in the prompt, hallucinations still occur. Without any tools specified in the prompt, ToolLlama generates over 50% nonexistent tool names.
Refer to caption: x4.png

### 5.4 Hallucination

We evaluate model hallucination in tool generation within an end-to-end agent scenario. To do this, we input a query in the format the models were trained on. Specifically, for ToolGen, we input a query directly and prompt the model to respond using the ToolGen agent paradigm (i.e., sequentially generating Thought, Tool, and Parameters). We tested Actions decoding without the beam search constraints described in Section 3.6. For ToolLlama and GPT-3.5, we input the query along with 5 ground truth tools. In all settings, we report the proportion of generated tools that do not exist in the dataset out of all tool generation actions. Figure 4 shows the hallucination rates of nonexistent tools for different models. From the figure, we observe that, despite being provided with only five ground truth tools, ToolLlama and GPT-3.5 may still generate nonexistent tool names. In contrast, ToolGen, with constrained decoding, does not hallucinate at all due to its design.

## 6 Conclusions

In this paper, we introduced ToolGen, a framework that unifies tool retrieval and execution in large language models (LLMs) by embedding tool-specific virtual tokens into the model’s vocabulary, transforming tool interaction into a generative task. By incorporating a three-stage training process, ToolGen equips LLMs with the ability to efficiently retrieve and execute tools in real-world scenarios. This unified approach sets a new benchmark for scalable and efficient AI agents capable of handling vast tool repositories. Looking ahead, ToolGen opens doors for integrating advanced techniques like chain-of-thought reasoning, reinforcement learning, and ReAct, further enhancing the autonomy and versatility of LLMs in real-world applications.

## Appendix A Real Tool Example

Figure 5 shows a real tool example. Each tool is a collection of several APIs. In our experiments, the following fields are used: "tool_name" is the name of the tool. "tool_description" describes tool related information such as the functionality of the tool. In each API, "name" is the name of the API. "description" describes API related information. "method" is the http method for calling the API. "required_parameters" are parameters that must be filled when calling the API. Optionally, "optional_parameters" can be set for extra parameters.

Figure: Figure 5: A real tool example. The tool contains one API. We have removed unnecessary fields for simplicity.
Refer to caption: x5.png

## Appendix B Tool Virtualization Implementation

ToolGen adopts a single and unique token to represent a tool, which shows its superiority for tool retrieval and tool calling. We also introduced other methods to index a tool, including semantic, numeric, and hierarchical. The following is a detailed implementation of how we implement each indexing.

#### Atomic

indexing is the method we use in ToolGen. Compared to other methods, it takes a single token as a tool and does not hallucinate to nonexistent tools. We use <<tool name&&api name>> to combine the tool name and api name to form a single token. For example, for the example in Appendix A, the resulting token is <<Youtube Hub&&Get Video Details>>.

#### Semantic

indexing maps each tool to the name used in ToolBench, which is also a combination between tool name and API name. However, the name can be tokenized into multiple tokens so that the model can perceive its semantic meanings. For the example in Appendix A, the resulted mapping is get_video_details_for_youtube_hub.

#### Numeric

indexing maps each tool to a unique number. We first get a list of all tools, with a length about 47,000. For all tools, we use a five digit number separated by space to represent the tool. If the example in Appendix A is the 128th element in the list, we use 0 0 0 1 2 8 to represent the tool. Since Llama-3 tokenizer encodes each number

#### Hierarchical

also maps each tool into a number. Different from Numeric indexing, we inject structure information into the tool representation by iterative clustering. During each iteration, we cluster tools into ten clusters, where each cluster is assigned a number from 0 to 9. For each cluster, we repeat this clustering process until there is only one tool in the cluster. These steps form a clustering tree. We take the number from root to the leaf as the representation to the tool in that leaf. The example in Appendix A may be assigned a number longer than five digits, such as 0 1 2 2 3 3 3.

## Appendix C Constrained Beam Search Detail

During retrieval and completing end-to-end agent tasks, we use constrained beam search to limit the generated actions to be valid tool tokens. The detailed steps are shown in Algorithm 1. The basic idea is to limit the searching space during beam search step. To achieve this, we need to first build a disjunctive trie, where each node represents a tool token id. Children of the node are all feasible ids following the current id. Using this tree, we can determine all possible next token ids based on current searched ids. During beam search step, we mask out all other unfeasible tokens’ logits, forcing possible ids to be sampled or searched.

For retrieval, this can be directly applied during generation. For end-to-end agent tasks, since we have decomposed a inference step into three conversational turns, we can easily detect when ToolGen needs to generate an action, therefore apply the constraint. Figure 6 shows an end-to-end inference example of ToolGen, where there is no relevant tools for ToolGen to choose. It can generate the tool token directly and complete the task.

**Table 6: Dataset statistics for the three-stage training. For tool memorization, there are some repeated tools, resulting the number of samples slightly larger than the tools we used.**
| Dataset | Tool Memorization | Retrieval Training | End-to-End Agent-Tuning |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| I1 | I2 | I3 | All |  |  |  |
| #num | 49,936 | 194,086 | 222,783 | 72,833 | 489,702 | 183,336 |

Figure: Figure 6: An inference example of ToolGen. A system prompt is given first with no relevant tools. Then user gives the task query. ToolGen generates the Thought, we then use user role to hint the model to generate the action. After generating the action, we use user again to give the tool documentation. The model will generate tool inputs based on this documentation.
Refer to caption: x6.png

## Appendix D Retry Mechanism

For reproducibility, we have adopted several techniques such as fixing random seeds and setting temperatures to zero for decoding. However, we find this results in some problems for the whole task inference. Models tend to give up early while not trying enough for possible tools. And they are likely to say sorry when giving the final answer, which affects the end-to-end evaluation. Since our goal is to evaluate the tool usage capability, we want to mitigate this negative impact that is more related to summary ability. We use a retry mechanism, which simply regenerates the turn when models try to give up or say sorry.

Figure: Algorithm 1 Constrained Beam Search

## Appendix E Ablation

Table 7 shows the ablation results for end-to-end evaluation. For unseen instructions, ToolGen Agent shows a slightly better performance without tool memorization or retrieval training. However, for unseen tools, training without the first two stages causes a drop in both SoPR and SoWR. This demonstrates that the first two stage training plays a role in generalization capability of ToolGen, and retrieval training is more significant compared to tool memorization.

**Table 7: Ablation results for ToolGen end-to-end evaluation. Here Inst. represents unseen queries (instructions) and Tool. and Cat. mean unseen tools during training.**
| Model | SoPR | SoWR |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | I1-Inst. | I2-Inst. | I3-Inst. | Avg. | I1-Inst. | I2-Inst. | I3-Inst. | Avg. |
| ToolGen | 54.60 | 52.36 | 43.44 | 51.82 | 50.31 | 54.72 | 26.23 | 47.28 |
| w/o retrieval training | 56.95 | 46.70 | 50.27 | 52.42 | 49.69 | 50.94 | 34.43 | 47.27 |
| w/o memorization | 56.03 | 47.96 | 57.38 | 53.69 | 49.08 | 59.43 | 34.43 | 49.70 |
|  | I1-Tool. | I1-Cat. | I2 Cat. | Avg. | I1-Tool | I1-Cat. | I2 Cat. | Avg. |
| ToolGen | 56.54 | 49.46 | 51.96 | 52.66 | 40.51 | 39.87 | 37.90 | 39.53 |
| w/o retrieval training | 49.47 | 40.31 | 37.90 | 42.84 | 36.71 | 30.07 | 36.29 | 34.18 |
| w/o memorization | 58.86 | 46.19 | 49.87 | 51.70 | 37.34 | 38.56 | 42.74 | 39.32 |

## Appendix F Generalization

For ToolGen Agent, we measure the performance on queries requiring tools that the model hasn’t been trained on. Table 8 shows the end-to-end evaluation of models on unseen tools. ToolGen Agent underperforms ToolLlama, also indicating worse generalization capability in completing full tasks. The generalization problem is prevalent in generative retrieval and beyond the scope of this paper. Therefore, we leave this for future work.

**Table 8: Generalization results of ToolGen. We test and compare the performance of ToolGen with other models on queries require unseen tools during training.**
| Model | Setting | SoPR | SoWR |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| I1-Tool. | I1-Cat. | I2 Cat. | Avg. | I1-Tool | I1-Cat. | I2 Cat. | Avg |  |  |
| GPT-3.5 | GT. | 58.90 | 60.70 | 54.60 | 58.07 | - | - | - | - |
| ToolLlama | GT. | 57.38 | 58.61 | 56.85 | 57.68 | 43.04 | 50.31 | 54.84 | 49.04 |
| ToolGen | GT. | 52.32 | 40.46 | 39.65 | 47.67 | 39.24 | 38.56 | 40.32 | 39.30 |
| GPT-3.5 | Retrieval | 57.59 | 53.05 | 46.51 | 52.78 | 46.20 | 54.25 | 54.81 | 51.58 |
| ToolLlama | Retrieval | 57.70 | 61.76 | 45.43 | 54.96 | 48.73 | 50.98 | 44.35 | 48.30 |
| ToolGen |  | 56.54 | 49.46 | 51.96 | 52.66 | 40.51 | 39.87 | 37.90 | 39.53 |

## Appendix G Adapt ToolBench Data to ToolGen

Our ToolGen data are adapted and converted from ToolBench data. Specifically, we adopt the tool documentations as the data for tool memorization training, where the input is tool document and the output is the corresponding tokens.

For retrieval training, we use the data in ToolBench that are annotated for tool retrieval, where a query was annotated with several relevant tools. We take the query as input, and convert relevant tools into virtual tokens. These tokens are then used as outputs for retrieval training.

For end-to-end agent-tuning, we use the interaction trajectories as the sources and make the following conversions: (1) Each trajectory contains available tools in system prompt for solving the query. When completing the task, ToolLlama relies on the retrieved tools in system prompt to solve the task, while ToolGen can generate tools directly. Therefore, we remove the tools in system prompt. (2) We replace all tool names in the trajectory to corresponding virtual tool tokens. (3) In original trajectories, agent model generates Thought, Action, Action Input sequentially (also referred ReAct). We decompose the whole ReAct into three conversational turns. During the first turn, the agent model generates a Thought, and we use a user to prompt the model to generate an action. In second turn, the model generates the action, which are virtual tool tokens. We then fetch the document corresponding to those tokens, so the model knows which parameters to specify. In third turn, the model generates parameters for the tool.

The number of samples in each dataset is shown in Table 6. Samples of tool memorization and retrieval training are shown in Figure 7. A sample of end-to-end agent-tuning is shown in Figure 8.

Figure: Figure 7: Datasets examples for tool memorization and retrieval training. We use user role to represent inputs and assistant role to represent outputs.
Refer to caption: x7.png

Figure: Figure 8: An example for end-to-end agent-tuning.
Refer to caption: x8.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: See the Weaknesses above.
soundness: 2
strengths: - [S1] ToolGen outperform or achieves competitive performance among retrieval and end-to-end baselines on ToolBench.
- [S2] ToolGen can natively invoke 47K tools following the context.
summary: This paper proposes ToolGen, a finetuned LLMs that can use various tools during the conversations with the users. ToolGen incorporates new 47K tokens for tools into the Llama-3-8B. Through the tool virtualization, tool memorization, retrieval training, and end-to-end agent tuning, ToolGen correctly select the right tools in the context on  the ToolBench evaluation, achieving better performance than retrieval and end-to-end baselines.
weaknesses: - [W1] The technical novelty is limited. Using special tokens for tools and incorporating them into the original vocabularies are widely-known approach (e.g. Toolformer: https://arxiv.org/abs/2302.04761, ToolkenGPT: https://arxiv.org/abs/2305.11554). The contribution of this paper is scaling this up to 47K tools, but it's very straight forward and I'm not confident if the ICLR community would be interested in it.
- [W2] Releted to [W1], the results of ToolLlama-3 in Section 5 is unclear to me. Why ToolLlama-3 is not as good as ToolGen, even using the same data and models? Could you clarify the difference between two?
- [W3] How about retrieving tools by LLMs itself (without retriever)? For instance, if we use the LLM with millions of extremely long context (such as Gemini), we may not need to rely on neural retrievers. It would be interesting to include long-contect LLMs as a baseline.
- [W4] It would be important to compare other capability of LLMs before/after tool token incorporation. Current draft seems to lack the analysis after combining 47K tokens to LLMs.
- [W5] In Figure1, I didn't understand the distinction between ToolGen's Retrieval Task and Agent Task. Both have the input "I want some popular video games." How are they differentiated?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. The paper highlights the efficiency of atomic indexing due to each tool being represented by a single token. However, as the number of tools grows significantly larger, could this approach lead to vocabulary explosion issues? 
2. In the current setting, the vocabulary consists of 128K regular token and 47K tool/API tokens already, how does it affect the performance of LLM itself?
3. How does the performance of ToolGen vary with the size and complexity of the underlying LLM? Would a larger model lead to better generalization ability?
soundness: 2
strengths: 1. ToolGen elegantly combines tool retrieval and execution into a single generative process, eliminating the need for separate retrieval mechanisms. This streamlines tool interaction and enhances efficiency, particularly as the number of tools increases.
2. The use of constrained beam search during inference effectively restricts the output to valid tool tokens, significantly reducing the generation of nonexistent tools, a common issue in LLM-based agents.
3. ToolGen demonstrates its capacity to effectively handle a large repository of over 47,000 real-world tools, highlighting its scalability compared to existing methods that struggle with vast tool sets.
4. The authors spend good amount of effort comparing different indexing method, and the result is clear.
summary: The paper introduces ToolGen, a novel framework that aims to enhance the interaction between large language models (LLMs) and external tools. ToolGen shifts away from traditional tool retrieval methods and instead integrates tool knowledge directly into the LLM's parameters. This is achieved by representing each tool as a unique token, allowing the LLM to generate tool calls and arguments seamlessly as part of its text generation process.
weaknesses: 1. The advantage of ToolGen which combines tool retrieval and execution into a single generative process introduces limitation together with its efficiency. Since the tools are integrated into the system as tokens, the extension of new tools become inefficient. For every new tool/API, new token need to be added and the documentation finetuned into the model. Also, consider the case that when the tool/APIs get updated, the maintenance of all the tool/APIs, making sure they are up to date is a quite challenging task. On the contact, with a retriever would make adding and maintaining tool/APIs very easy.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - As stated earlier, why don't take the semantic representation approach as your final approach as it performs better (5 wins / 4 loses in table 2) than atomic approach? Saving tokens might not be a sufficient reason. Chain-of-thought reasoning process should produce a lot more than what atomic approach can save here.

- In Table 1: "For ToolGen in the In-Domain setting, we allow the generation space to include all tokens, which is a more challenging scenario compared to other models." What if you don't include all tokens? Since ToolGen can't outperform IterFeedback in the In-Domain setting currently, will it be able to outperform it without this additonal challange?

- In section 4.4,  the authors are assessing the impact of different training stages, but the table shows "−memorization", "−retrieval training", "−constraining". If I understand correctly, constraining is an decoding strategy used in inference stage, while the third training stage "end-to-end agent-tuning" does not show up.

Please also pay attention to the questions and suggestions in the "Weaknesses" part.
soundness: 1
strengths: **Originality:**
The paper introduces a novel approach to tool retrieval by representing each tool as a unique virtual token directly integrated into the LLM’s vocabulary. This method eliminates the need for auxiliary retrievers, making the retrieval process more seamless and efficient. The concept of transforming tool retrieval into a generative task is novel and presents a possible solution to the scalability challenges faced by some existing methods.

**Quality:**
The authors have conducted a comprehensive set of experiments to validate their claims, showing thoroughness in their evaluation. These experiments include various comparisons and detailed ablation studies to assess the impact of different components of their approach. The robustness of their methodology enhances the credibility of their results and conclusions.

**Clarity:**
The paper is well-structured and written in a clear and concise manner. It systematically outlines the problem, the proposed solution, and the experimental validations. Each section transitions smoothly into the next, making it easy for the reader to follow the logic and understand the contributions of the work.

**Significance:**
The research addresses a critical issue in the field of LLM agents: managing and retrieving tools from a large set of tools efficiently. Given the increasing complexity and number of tools available, the proposed method provides a solution. This work has implications for the development of more autonomous and efficient AI systems, potentially benefiting some applications where tool interaction is essential.
summary: The paper introduces ToolGen, a novel framework designed to address the limitations of LLMs in interacting with an extensive array of external tools. This framework shifts existing paradigms by embedding knowledge about various tools directly into the LLM’s vocabulary, using unique tokens to represent each tool. This allows the model to generate tool calls and arguments as part of the language generation process, eliminating the need for separate tool retrieval systems and seamlessly integrating tool usage within LLM capabilities. ToolGen incorporates a three-stage training process to enable the LLM to learn how to use tokenized tools.

The main contribution of this paper is introducing the ToolGen framework including training and inferencing, and proving that ToolGen have comparable result with current tool retrieval systems.
weaknesses: **Substantive Assessment of Weaknesses:**

**Cost and Efficiency Claims:**
The key claim of "significantly less cost and higher efficiency" does not hold up under scrutiny. The authors have not substantively demonstrated that their framework is less costly or more efficient than existing methodologies. The ToolGen framework necessitates a three-stage training process, which does not inherently suggest reduced costs. Furthermore, the paper lack data or experiments to substantiate the claim regarding efficiency improvements over other approaches.

**Role of the Memorization Stage:**
In Section 4.4, the table shows that the memorization stage plays a relatively minor role in the three-stage training process. However, the authors assert that this stage is beneficial for generalization, with further discussion supposedly found in Appendix F. There is, however, no such discussion present in Appendix F. I would recommend authors to do more experiments to validate the significance of the memorization stage and its impact on generalization.

**Hallucination Comparison:**
Section 5.4 contains statements that do not fully make sense. The claim that ToolGen experiences no hallucination is primarily due to the use of a constrained decoding strategy. Theoretically, it is impossible to encounter hallucination, as defined by the authors, under this constraint. This does not constitute a fair comparison with other frameworks. The authors should either provide results without this constraint and compare them or demonstrate why other frameworks are unable to implement a similar strategy.

**Performance Benchmarking:**
ToolGen is not currently the best-performing framework. Several baselines in the paper outperform ToolGen in various categories, as shown in Table 1 and Table 4. Notably, even the proposed but ultimately unused semantic tokenization approach outperforms the authors' current method, as shown in Table 2. This discrepancy confuses me.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: see the above.
soundness: 3
strengths: Overall, I like this paper very much.  

1. ToolGen is now paradigm for fundamentally transforming tool retrieval into a generative process, which is a very important topic for function calling or LLM-based agents.  
2. The methods are sound and resonable; the paper presentation is clear.  
3. The experimental results show that the methods are effective compared with several strong baselines.
summary: This paper proposes ToolGen, a generative tool/function calling framework. The concrete methods are: (1) virtualizing tools by virtual tokens; (2) memorizing tools with training data of (tool docs, tool virtual tokens); (3) learning tool retrieval with training data of (user queries, tool virtual tokens); (4) and finally, finetuning tool agent with tool calling trajectories. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents that can adapt to tools across diverse domains.
weaknesses: 1. Traditional retrieval and generation methods for function calling can handle dynamic tools. If the tool set is changing, could ToolGen be used (without retraining)?  

2. A few reltated works are not mentioned or compared. For example, TPTU and TPTU-V2 [1,2,3,] used demo retriever and fintuner besides the tool retrieval, which may be more powerful than the traditional retrieval and generation methods. They can be a strong baseline for comparision.  

3. It seems that ToolGen uses a more complex process (i.e., (1) virtualizing tools by virtual tokens; (2) memorizing tools with training data of (tool docs, tool virtual tokens); (3) learning tool retrieval with training data of (user queries, tool virtual tokens); (4) and finally, finetuning tool agent with tool calling trajectories) to prepare for tool calling. Therefore, the suprior of ToolGen may come from more training of LLMs? Could the authors give more explination on this?


[1] TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents. FMDM Workshop at NeurIPS 2023.   
[2] TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems. LLMAgents Workshop at ICLR 2024.  
[3] TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Industry Systems. EMNLP 2024 Industry Track.

Meta-review: The paper introduces ToolGen, a novel framework that integrates tool knowledge directly into the parameters of a Large Language Model (LLM) by representing each tool as a unique token.  This allows the LLM to generate tool calls and arguments seamlessly, eliminating the need for separate tool retrieval systems.  ToolGen is trained in three stages: tool memorization, retrieval training, and end-to-end agent tuning.  The authors claim this framework enhances the interaction between LLMs and external tools, particularly with large tool sets.    

Strengths: The paper proposes a novel approach to tool retrieval and execution by representing tools as virtual tokens, streamlining the process.  The use of constrained beam search effectively reduces the generation of nonexistent tools.  The authors demonstrate the scalability of ToolGen by incorporating over 47,000 real-world tools, and how it can more easily deal with context length limitations and tool hallucinations.

Weaknesses: (1) Dealing with new tools at test-time or major updates in tool APIs would necessitate retraining, which has been added as a discussion in appendix.  (2) Degradation in general instruction following performance of LLMs when fine-tuning heavily on tool use trace, which would be critical for practical deployment. (3) Post-hoc tool use is common for deployment due to latency introduced by tool calls during generation, which might need to carefully handled with ToolGen.

Reasons for Acceptance/Rejection: The paper present a novel perspective on tool use -- generative tool calling - which can tap into the generation abilities of LLMs. Moreover, the authors have addressed the reviewers' concerns by providing additional explanations and committing to further experiments. The revised version of the paper, incorporating the planned improvements, is expected to be a valuable contribution to ICLR.

Minor suggestion: Work in similar spirit to ToolGen has been going on in generative verifiers / reward models (GenRM, Cloud) which would be worth discussing.

Decision: Accept (Poster)