ToolGen: Unified Tool Retrieval and Calling via Generation
Authors: Renxi Wang
Xudong Han
Lei Ji
Shu Wang
Timothy Baldwin
Haonan Li
Venue: ICLR 2025 Poster (2025)
Keywords: Agent
Tool Learning
Virtual Token
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Unified tool retrieval and calling by transforming tools into virtual tokens

Abstract: As large language models (LLMs) advance, their inability to autonomously execute tasks by directly interacting with external tools remains a critical limitation. Traditional methods rely on inputting tool descriptions as context, which is constrained by context length and requires separate, often inefficient, retrieval mechanisms. We introduce ToolGen, a paradigm shift that integrates tool knowledge directly into the LLM’s parameters by representing each tool as a unique token. This enables the LLM to generate tool calls and arguments as part of its next token prediction capabilities, seamlessly blending tool invocation with language generation.  Our framework allows the LLM to access and utilize a vast amount of tools with no additional retrieval step, significantly enhancing both performance and scalability. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents that can adapt to tools across diverse domains.  By fundamentally transforming tool retrieval into a generative process, ToolGen paves the way for more versatile, efficient, and autonomous AI systems. ToolGen enables end-to-end tool learning and opens opportunities for integration with other advanced techniques such as chain-of-thought and reinforcement learning, thereby expanding the practical capabilities of LLMs

Published as a conference paper at ICLR 2025
TOOLGEN: UNIFIED TOOL RETRIEVAL AND CALLING VIA GENERATION
Renxi Wang!a Timothy Baldwin 12,5
Xudong Han 1.2
Lei Ji
Shu Wang
Haonan Li
LLibrAl Mohamed bin Zayed University of Artificial Intelligence
3 Microsoft
*University of California, Los Angeles 'The University of Melbourne
(renxi. wang, xudong. han, timothy ba 1 Ldwin, haonan. 1i Jembzuai. ac.ae lei j ji@microsoft. com
shuwang0712@ucc edu
ABSTRACT
As large language models (LLMMs) advance, their inability to autonomously exe- cute tasks by directly interacting with external tools remains a critical limitation. Traditional methods rely on inputting tool descriptions as context, which is con- strained by context length and requires separate, often inefficient, retrieval mech- anisms. We introduce ToolGen, a paradigm shift that integrates tool knowledge directly into the LLM's parameters by representing each tool unique token. This enables the LLM to generate tool calls and arguments as part of its next token prediction capabilities, seamlessly blending tool invocation with language gener- ation. Our framework allows the LLM to access and utilize a vast amount of tools with no additional retrieval step, sigmificantly enhancing both performance and scalability. Experimental results with over 47, 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task comple- tion but also sets the stage for new era of AI agents that can adapt to tools across diverse domains. By fundamentally transforming tool retrieval into a generative process, ToolGen paves the way for more versatile, efficient, and autonomous AI systems. ToolGen enables end-to-end tool learning and opens opportunities for integration with other advanced techniques such as chain-of-thougee and rein- forcement learning, thereby expanding the practical capabilities of LLM
INTRODUCTION
Large language models (LLMs) have demonstrated impressive capabilities interactive systems, adept at processing external inputs, executing actions, and autonomously completing tasks (Gravitas) Qin et al.) 2023 Yao et al.) 2023; Shinn et al. 2023] Wu et al.) 2024a] Liu et al.) [024,] Wang 2024b cp. Among the various methods enabling LLMs to interact with the world, tool calling via APIs has emerged as one of the most common and effective approaches. However, as the number of tools grows into the tens of thousands, existing methods for tool retrieval and execution struggle
2023
ct al.
to scale efficiently.
common approach real-world scenarios is to combine tool retrieval with tool execution, where retrieval model first narrows down the relevant tools before passing them to the LLM for final selection and execution (Qin et al. 2023 Patil et al., (023). While this combined method addresses numbers of tools, it has notable limitations: retrieval models often
the challenge of handling rely on small encoders that fail to fully capture the semantics of complex tools and queries, and separating retrieval from execution introduces inefficiencies and potential misalignment between stages of task completion.
Moreover, LLMs and their tokenizers are pretrained primarily on natural language data (Brown et al., 2020 Touvron et al., (023). leaving them with limited intrinsic knowledge of tool-related functionalitties This gap knowledge results in suboptimal performance, especially when the L.M must rely on retrieved tool descriptions for decision-making.
Data and code available https://ggituub.eeott httpp///ggthoo.ccc/////eee/////

---

Published as a conference paper at ICLR 2025
popular
some popular
"Game search
Video garnes.
Similarity
video gomes
popular video gumes.
ToolRetriever
Available Tools:
Game Search
Game Search
Top Movies
Game Search
Game Hall of Fame
Glame Soarhh
'g': "popular video games"
Gio Wenther
Embed
Came Hall of
G-5
Game Hall of Fame
Tooll Jama
Retrieval Task
Agent Task
want popular
want pogular Vidco garmes.
vidoo games.
TookGen
need to call search popular video games. Search
Steken Game, Searchs
Stoken
Came Searche
Steken
Hall
Fames
"4" "popular video games"
Figure 1: Comparison between previous retrieval-based methods and our ToolGen. Previous meth- ods use retriever to retrieve relevant tools based on similarity matching, which are further put into prompts for LLMs to select. ToolGen can retrieve tools by generating tool tokens directly. ToolGen can also complete the task without relying on any external retriever.
In this study, we introduce ToolGen, a novel framework that integrates real-world tool knowledge directly into the LLM's parameters and transforms tool retrieval and execution into a unified gen- eration task. Specifically, ToolGen expands the I.IM's vocabulary with tool-specific virtual tokens and trains the model to generate these tokens within conversational context, allowing the LLM to leverage its pre-existing knowledge more effectively for both retrieving and calling tools.
Specifically, each tool is represented as unique virtual token within the LLM's vocabulary. Build- ing upon pretrained LLM, ToolGen's 's training process consists of three stages: tool memorization, retrieval training. and agent training. the tool memorization stage, the model associates each virtual tool token with its documentation. During retrieval training, the model learns generate rel- evant tool tokens based on user queries. Finally, in end-to-end agent-tuning, the model is trained to autonomous agent, generating plans and tools, and determining the appropriate parameters to complete tasks. By calling tools and receiving feedback from external environments, the model can handle user queries efficiently and integratively. Figure shows comparison between ToolGen and traditional paradigms.
We demonstrate ToolGen's superiority in scenarios: tool retrieval task, where the model trieves the correct tool for given query, and an LLM-based agent task, where the model completes complex tasks involving real-world API calls. Leveraging a dataset of 47,000 real-world tools, Tool- Gen achieves performance comparable to the leading tool retrieval methods, but with significantly lower cost and greater efficiency. Additionally, it surpasses traditional tool learning paradigms, highlighting its potential for advancing more effective tool usage systems.
ToolGen represents paradigm shift in tool interaction by merging retrieval and generation into a single, cohesive model. This innovation sets the stage for new generation of Al agents capable of adapting to a vast array of tools across diverse domains. Additionally, ToolGen opens new op- portunities for integrating advanced techniques like chain-of-though reasoning and reinforcement learning with the ability to use tools in a unified generation way, expanding the capabilities of L.LMs in real-world applications.
In summary, our contributions are:
A novel framework, ToolGen, that integrates tool retrieval and execution into the LLM's generative process using virtual tokens.
A three-stage training process that enables efficient and scalable tool retrieval and API calling within ToolGen.
2

---

Published as a conference paper at ICLR 2025
Experimental validation demonstrates that ToolGen achieves comparable performance to current best tool retrieval methods with significantly less cost and higher efficiency and surpasses traditional tool learning paradigms across large-scale tool repositories.
RELATED WORK
2.1 TOOL RETRIEVAL
Tool retrieval essential for LLM agents in real-world task execution, where tools are usually represented by their documentation. Traditional methods like sparse (e.g.. BM25 (Rohertson et al.) 2009 and dense retrieval (e.g., DPR (Karpukhin et al. 2020), ANCE (Xiong et al. (021)) rely on large document indices and external modules, leading to inefficiencies and difficulty in optimizing in an end-to-end agent framework. Some work has explored alternative methods. For example,
Chen et al. (2024b) rewrite queries and extract their intent, targeting unsupervised retrieval settings, though the results are not comparable to supervised approaches. Xu et al. (2024) propose a method that iteratively refines queries based on tool feedback, improving retrieval accuracy but increasing latency.
Recently, generative retrieval has emerged a promising paradigm, wherein models directly generate relevant document identifiers rather than relying on traditional retrieval mechanisms (Wang
2022 Sun al. 2023b Kishore al. 2023b Mchta et al. 2023b; Chen et al. 2023c).
Motivated by this, ToolGen represents each tool as a unique token, allowing tool retrieval and calling be framed generation task. Beyond simplifying retrieval, this design integrates smoothly with
other LLM and LLM-based agent features like chain-of of-thhough reasoning (Wei et al.) 2023) and
ReAct (Yao et al.) 2023). By consolidating retrieval and task execution into single LLM agent, reduces latency and computational overhead, leading to more efficient and effective task completion.
2.2 LLM-AGENTS WITH TOOL CALLING
LLMs have shown strong potential in mastering tools for various tasks. However, most existing works focus on limited set of actions (Chen al., 2023a Zeng et al. [2023] Yin al., 2024] Wang et al. 2024a). For instance, Toolformer (Schick et al.) (023) fine-tunes GPT-J to handle just five tools, such as calculators. While effective for narrow tasks, this approach struggles
real-world
scenarios with vast action spaces. ToolBench (Qin et al.) (023) expands the scope by introducing over 16,000 tools, highlighting the challenge of tool selection in complex environments.
To perform tool selection, current methods often use retriever-generate pipeline, where relevant tools are retrieved and then utilized by the LLM (Patil et al. 2023 Qin et al.) 2023). In addition, TPTU (Ruan et al.)) proposes structured framework for LLM agents and evaluates their task plan- ning and tool usage abilities. Furthermore, TPTU-v2 (Kong et al. 2024) builds an LLM Finetuner to enhance agent performance with curated datasets and demo selector select relevant demonstra- tions. They set a ficxible and superior paradigm compared to traditional retrieval- based paradigm. However, pipelined approaches face two major issues: error propagation from the retrieval step and the inability of LLMs to fully understand and use tools via simple prompting.
To mitigate these issues, researchers have tried representing actions as tokens, converting action prediction into generative task. For example, RT2 (Brohan et al. 2023) generates tokens repre- senting robot actions. and Self-RAG (Asai al. 2023) uses special tokens to decide when to retrieve
documents. ToolkenGPP (Hao et al. 2023) introduces tool-specific tokens to trigger tool usage, concept closest to our approach.
Our approach differs from ToolkenGPT in several ways. First, we focus on real-world tools that require flexible parameters for complex tasks (e.g.. YouTube channel search), while ToolkenGPT is limited to simpler tools with fewer inputs (e.g., math functions with two numbers). Additionally, ToolkenGPT relies on few-shot prompting, whereas ToolGen incorporates tool knowledge directly into the LLM through full- full-parameter fine-tuning, enabling the model to retrieve and execute tasks autonomously. Finally, our experiments involve a much larger tool set set-47 47,000 tools compared to TTolkenGPTTs 13-300. Detailed comparison and other related work can be found in Section
3

---

Published as a conference paper at ICLR 2025
Tool Virtualization
Memorization
Retrieval Training
Pipeline Training
Add Embeekiing
LLM
Tool Description
Query
Trajdtrics
Tools
+ Tokens
LLM
ToolGen
ToolGen Agent
Expand Vocabolary
Tokenizer
Retriever
Tool Tokens
Tool Tokens
Instraction Finetune
Tools
Indexing
Scarch games query.
Could recommend
movies?
Movics
Numeric
Hierarchical
Game Sonth
00000
LLL
00000
Hall Fame:
00002
<token Game Seardo
Top, Mivics
00003
I wast some popular video
ToelGem: Fint, need use" steken Giare Seaccs" sarch games
Semantic
Atomic
Top.
User: Generate the action.
Gunte
Search
ToolGen:
Seacher
Gitt
Wenther
Stakes Get Weathern
Document: (name:
Came. Htll
Fames
Obneraation: Here wore papulae video games
An illustration of ToolGen framework. In tool virtualization, tools mapped into virtual tokens. In the following staee-stage training. ToolGen first memorizes tools by predicting tool tokens based on their documentations, Then it learns to retrieve tools by predicting tool tokens from queries. Finally, pipeline data, i.e., trajectories, are used to finetune the retriever model from the last stage, resulting in the ToolGen Agent model.
Figure 2:
TOOLGEN
In this section, we first introduce the notations used throughout the paper. Then we detail the specific methods of ToolGen, including tool virtualization, tool memorization, retrieval training, and end-to- end agent tuning, as illustrated in Figure Lastly, we describe our inference approach.
3.1 PRELIMINARIES
Given a user query 4. tool learning aims to resolve & using tools from a large tool set = (di.dg,... dx ). where []] N large number, making impractical to include all tools D in the LLM context. Therefore, current research typically uses retriever R to retrieve k relevant tools from D, denoted as Da, {dr, dry.-.. dr R(q, k. D), where |x.nn < N. The final prompt is then the concatenation of and Da, denoted as Prompt task (query), an LLM-based agent usually adopts a four four-stage paradigm (Qu et al. (024) iteratively: generates plan p;, selects tool d.i. determines tool parameters C, and collects feedback from the tool(s) ft. We denote these steps for the i-th iteration as pi, da,, G, Sú. The model continues iterating through these steps until the task is completed,
44 DR.R
]- To complete
which point the final answer is generated. The entire trajectory can be represented as Traj = [Prompt. (pI, de, C1 fi)....... (Pt. da,, Ct2 ft), (g, R((g, D), (pi, d C1, fi),..., (pe, da,, Ct: fa), a). This iterative approach allows the model to dy- namically adjust and refine its actions at each step based on the feedback received, improving its performance in completing complex tasks.
3.2 TOoL VIRTUALIZATION
In ToolGen, we virtualize tools by mapping each tool to a unique new token through a method call atomic indexing. In this approach, each tool is assigned unique token by expanding the LLM's vocabulary. The embedding for each tool token initialized as the average embedding of its corresponding tool name, ensuring semantically meaningful starting point for each tool.
4

---

Published as a conference paper at ICLR 2025
Formally, the token set is defined T = Index(d) AA E D, where Index is the function mapping tools to tokens. We demonstrate that atomic indexing is more efficient and can mitigate halluci- nation compared to other indexing methods, such semantic and numeric mappings, discussed in Section and 5.4
3.3 TOOL MEMORIZATION
After assigning tokens to tools, the LLM still lacks any knowledge of the tools. To address this, we inject tool information by fine-tuning it with tool descriptions as inputs and their corresponding tokens as outputs, which we call tool memorization. We use the following loss function:
Ltoot
Σ - -llog (Index(d) (doe)
dED
where 0 denotes the LLM parameters, and ddoe represents the tool description. This step equips the LLM with basic knowledge of the tools and their associated actions.
3.4 RETRIEVAL TRAINING
We then train the LLMs to link the hidden space of virtual tool token (and its documentation), the user query space, so that LLM can generate correct tool based on a user's query. To achieve this, we fine-tune the LLM with user queries as inputs and corresponding tool tokens as outputs:
Σ Σ - log po (Index(d)lg)
Pretrievat
@EQ dED,
where 8' represents the LLM parameters after tool memorization, Q is the set of user queries, and D, the set of tools relevant to each query. This results in the ToolGen Retriever, which can generate the appropriate tool token given a user query.
3.5 END-TO-END AGENT-TUNING
After retrieval training, the LLM capable of generating tool tokens from queries. In the final stage, we fine-tune the model with agent task completion trajectories. We adopt a similar inference strategy as Agent-Flan (Chen al., (002c)), in instead of generating Thought, Action, and Argu- ments together as ReAct. Our pipeline follows an iterative process, where the LLM first generates a Thought, and the corresponding Action token. This token is used fetch the tool documentation, which the LLM uses to generate the necessary arguments. The process continues iteratively until the model generates a "finish" token or the maximum number of turns is reached. The generated trajectory is represented as Traj [q, (P1, Index(d,,) c1. fi).... (pt, Index(d,). ct fi), a). In this structure, relevant tools are no longer required.
3.6 INFERENCE
During inference, the LLM may generate action tokens outside the predefined tool token set. To prevent this, we designed constrained beam search generation that restricts the output tokens to the tool token set. We applied this constrained beam search for both tool retrieval, where the model selects tools based on queries, and the end-to-end agent system, significantly reducing hallucination during the action generation step. detailed analysis can be found in Section 5.4 The implementa- tion details can be found in Appendix
4 TOOL RETRIEVAL EVALUATION
4.1 EXPERIMENTAL SETUP
We use pretrained Llama-3-8B (Dubey et al. 2024 as our foundation model, with a vocabulary size of 128,256. Using the atomic indexing approach, we expand the vocabulary by additional 46,985 tokens following the tool virtualization process, resulting in a final vocabulary size 175,241. We fine-tune the model using the Llama-3 chat template with cosine learning rate scheduler, applying
5

---

Published as a conference paper at ICLR 2025
3% warm-up steps. The maximum learning is x 10 10-5. All models are trained using Deepspeed ZeRO 3 (Rajbhandari et al. 2020) across 4xA100 GPUs. We train epochs for tool memorization and epoch for retrieval traming.
Dataset Our experiments are based on ToolBench, a real-world tool benchmark containing more 16k tool collections, each containing several APIs, resulting in a total of 47k unique APIs. Each API is documented with a dictionary, containing the name, description, and parameters for calling the API. A real example is shown in Appendix [ We take each API as action and map it to a token. Our retrieval and end-to-end agent-tuning data are converted from the original data in ToolBench. Details be found in Appendix K Although each tool may consist of multiple APIs, for simplicity, we refer to each API as a tool in this paper.
We follow the data split of Qin et al. (2023), ) where 200k (query, relevant API) pairs are divided into three categories: II (single-tool queries), 12 (intra-category multi-tool queries), and 13 (intra- collection multi-tool instructions). containing 87,4 87,413, 84,815, and 25,251 instances, respectively.
Baselines
We compare ToolGen with the following baselines:
BM25: A classical unsupervised retrieval method based on TF-IDF, which retrieves docu- ments based on term similarity with the query.
Long-Context LLMs: We concatenate tools into a long prompt gpt -4,, and prompt it to choose from the pool. Limit by context length, we cannot input all 47k tools, so we use 2k tools with ground truth tools included.
Embedding Similarity (EmbSim): Sentence embeddings generated using OpenAl's sen- tence embedding model; specifically text www-eeeeeddinn---1ee------ in experiences.
Re-Invoke (Chen et al., 2024b): An unsupervised retrieval method with query rewriting and document expansion.
IterFeedback (Xu et al.
2024): BERT-based retriever with gpt -pt-3.555. 3...-.rbbtubooo 0125 as
feedback model with iterative feedback for up 10 rounds.
ToolRetriever (Qin et al. 2023): : A BERT-based retriever trained via contrastive learning.
Settings We conduct experiments under two settings. In the first, In-Domain Retrieval, the re- trieval search space is restricted to tools within the same domain. For example, when evaluating queries from domain 11, the search is limited to Il tools. This aligns with ToolBench settings. The second, Multi-Domain Retrieval, is more complex, with the search space expanded to include tools from all three domains. In this case, models are trained on combined data, increasing both the search space and task complexity. Unlike ToolBench, this multi-domain setting reflects real-world scenarios where retrieval tasks may involve overlapping or mixed domains. This setup evaluates the model's ability to generalize across domains and handle more diverse, complex retrieval cases.
Metrics We evaluate retrieval performance using Normalized Discounted Cumulative Gain (NDCG) (Järvelin & Kekäläinen 2002), widely used metric in ranking tasks, including tool re- trieval. NDCG accounts for both the relevance and ranking position of retrieved tools.
4.2 RESULTS
Table I presents the tool retrieval results. As expected, all trained models significantly outperform the untrained baselines (BM25, EmbSim, and Re-Invoke) across all metrics, demonstrating the ben- efit of training on tool retrieval data.
Our proposed ToolGen model consistently achieves the best performance across both settings. the In na-Domain setting. ToolGen delivers highly competitive results, achieving comparable performance to the IterFeedback system, which uses multiple models and a feedback mechanism. ToolGen, as a single model, outperforms ToolRetriever by significant margin in all metries and even surpasses IterFeedback in several cases, such as NDCG@S for domain and NDCG@ @, @5 for 12.
In the Molti-Domain setting, where the search space is larger and performance generally drops, ToolGen remains robust, outperforming ToolRetriever and maintaining superiority over other base- lines. This demonstrates that ToolGen, despite being single model, is capable of competing with

---

Published as a conference paper at ICLR 2025
Table Tool retrieval evaluation across two settings: (1) In-Domain, where models are trained and evaluated within the same domain; and (2) Multi-Domain, where models are trained on all domains and evaluated with the full set of tools across all domains. BM25, EmbSim, and Re-Invoke are unsupervised baselines without training. IterFeedback is retrieval system with multiple models and feedback mechanism. ToolRetriever trained using contrastive learning, while ToolGen trained with next-token prediction. Results marked with * were not implemented by and are copied from their original paper, and hence only in the In- in-Domain setting. For ToolGen the In- D--Dommi setting, allow the generation space to include all tokens, which is more challenging scenario compared to other models. Best results in each category are bolded.
11 NDCG3
12 NDCG3
13
Model
NDCGI
NDCGS
NDCG1
NDCGS
NDCG1
NDCG3
NDCGS
In-Domain
BM25
29.46
31.12
33.27
24.13
25
27.65
32.00
25.88
29.78
Long-Context Context LLM*
32.22
42.87
52.14
25.39
33.91
46.07
25.11
32.57
44.03
EmbSim
63.67
61.03
65.37
49.11
42.27
46.56
53.00
46.40
52.73
Re-Invoke
69.47
61.10
54.56
53.79
59.65
59.55
Iterf werdeckd
90.70
90.95
92.47
89.01
87.10
91.74
ToolRetriever
80,50
79.55
84,39
71.18
64.81
70.35
70.00
60.44
64.70
ToolCGen
89.17
90.85
92.67
91.45
88.79
91.13
87.00
85.59
90,16
Multi-Domain
BM25
22.77
22.64
18.29
20.74
22.18
10.00
10.08
EmbSim
54,00
50.82
55,86
40.84
36.67
39.55
18.00
17.77
20.70
ToolRetriever
72.31
70.30
74.99
64.55
57.91
63.61
52,00
39.89
42,92
ToolGen
87.67
88.84
91.54
83.46
86.24
88.84
79.00
79.80
84.79
complex retrieval systems like IterFeedback, showcasing ability to handle complex real-world retrieval tasks where domain boundaries are less defined.
4.3 INDEXING METHON COMPARISON
While ToolGen uses atomic indexing for tool virtualization, we explore several alternative generative retrieval approaches. In this section, compare it with the following three methods:
Numeric: Map each tool to unique number. The resulting token is purely numeric, offering no inherent semantic information, but providing a distinct identifier for each tool.
Hierarchical: This method clusters tools into non-overlapping groups and recursively parti- tions these clusters, forming a hierarchical structure. The index from the root to the leaf this tree-like structure represents each tool, similarly Brown clustering techniques.
Semantic: In this approach, each tool is mapped to its name, using the semantic content of the tool names to guide the LLM. The tool's name provides a meaningful representation directly related to its function.
The implementation details are described in Appendix D
Token Length Distribution
300
First, we conducted an analysis of the num- ber of subtokens required to represent each tool for the different methods, as shown in Fig- ure 3 Atomic indexing ensures each tool to be single token, while numeric indexing en- codes tools into N tokens for tools numbered in N10 10 In contrast, both seman- tie indexing and hierarchical indexing produce variable number of subtokens, with seman- tie indexing having more outliers with signifi- cantly longer sequences. The figure highlights the superiority atomic indexing, where each tool is represented by single token, whereas other methods require multiple tokens. This efficiency allows ToolGen to reduce the number of generation tokens and inference time in both the retrieval and agent scenarios.
25
20
15
10
Atomic
Semantic
Numeric
Hierarchical
Figure 3: The distribution of the number of subtokens per tool varies across different index- ing methods.

---

Published as a conference paper at ICLR 2025
Table 2: Retrieval evaluation for different indexing methods in Multi-Domaal setting. Best results are bolded and second best results are underlined.
II
12
13
Model
NDCG1
NDCG3
NDCG5
NDCG1
NDCG3
NDCG5
NDCGI
NDCG3
NDCGS
Numetric
83.17
84,99
88.73
79.20
79.23
83.88
71.00
74.81
82.95
Hierarchical
85.67
87.38
90.26
82.22
82.70
86.63
78.50
79.47
84.
Semantic
89.17
91.29
93.29
83.71
84.51
88.22
$2.00
78.86
85,43
Atomic
87.67
88.84
91.54
86.24
79.00
79.80
84.79
Table 3: Ablation study for tool retrieval. We assess the impact of removing retrieval training, tool memorization, and constrained beam search ToolGen's performance, respectively.
12
13
Model
NDCG1
NDCG3
NDCG5
NDCGI
NDCG3
NDCGS
NDCG1
NDCG3
NDCG5
ToolGen
87.67
88.84
91.54
83,46
86.24
88.84
79.00
79.80
84.79
memorization
84,00
86.77
89.35
82.21
83.20
86.78
77.00
77.71
84.37
retrieval training
10.17
12.31
13.89
5,52
7.01
7.81
3.00
4.00
4,43
construining 87.67
88.79
91.45
83,46
86.24
88.83
79.00
79.93
84.92
Next, we examined the effectiveness of different indexing methods. As shown Table semantic indexing demonstrates the best retrieval performance across various metrics and scenarios, while atomic indexing closely follows in many cases. We attribute this to the fact that semantic indexing aligns better with the pretraining data of LLMs. However, this advantage diminishes the training data and type increase. For example, in Section 5.3] we show that atomic indexing achieves better end-to-end results. We also show that combining constrained beam search with semantic indexing will cause biased tool usage, which is detailed in Section E.2
Taking all these factors into account, choose atomic indexing for ToolGen tool virtualization.
4.4 ABLATION
We perform an ablation study to assess the impact of different training stages of ToolGen, as shown The results indicate that retrieval training is the crucial factor for tool retrieval per- formance, it directly aligns with the retrieval task where inputs are queries and outputs tokens. Removing tool memorization leads to a minor performance drop although it plays role in improving generalization, which we will discuss further in Appendix []] Similarly, constrained beam search, while not a major contributor to retrieval task, helps prevent hallucinations, making it useful for end-to-end agent tasks, Section 5.4
in Table 3
tool
END-TO-END EVALUATION
EXPERIMENTAL SETUP
We make several modifications to the trajectory data from ToolBench fit it into ToolGen frame- work. For example, as ToolGen does not require explicit selection of related tools as input, we remove this information the system prompt. Further details are provided in Appendix K Fol- lowing this, fine-tune the retrieval model using the reformatted data, resulting in an end-to-end ToolGen agent.
We use gpt-3. 3pt-3.5-turbo- 5-turbo-06665 one of our baselines. The implemen- tation is the same as used in StableToolBench (Guo et al. 2024), where the tool calling capability of GPT-3.5 is used to form tool agent. ToolLlama-2: Qin et al. (2023) introduced ToolLlama-2 by fine-tuning Llama-2 TTovvro et al. 2023) model on ToolBench data. ToolLlama-3: To ensure fair comparison,
Baselines GPT-3.5:
fine-tuned Llama-3, the same base model used in ToolGen, the ToolBench dataset, creating the ToolLlama-3 baseline. In the rest of this paper, we refer to ToolLlama- as ToolLIama Jama distinguish from Tooll lama-..
8

---

Published as a conference paper at ICLR 2025
Table 4: End-to-end evaluation performance on unseen instructions under two settings. For R. setting, GPT3.5 and ToolLIama use ToolRetriever, while ToolGen does not use external retriever. For all results, SoPR and SoWR are evaluated three time and reported with mean values.
SoPR
SoWR
Model
13
Avg.
12
Avg
w/ Ground Truth Tools (G.T.)
GPT-3.5
56.60
47.80
54.64
50.91
TooaL
53.37
41.98
46.45
48.43
47.27
59.43
27.87
47.58
ToolLIama
55.93
48.27
52.19
52.78
50.31
53.77
31.15
47.88
ToolGen
61.35
49.53
43.17
54.19
51.53
57.55
31.15
49.70
w/ Retriever (R.)
GPT-3.5
51.43
41.19
34.43
45.00
53.37
53.77
37.70
50.60
ToolLlama-2
56.13
49.21
34.70
49.95
50.92
53.77
21.31
46.36
TooILIama
54.60
49.96
51.37
51.55
49.08
61.32
31.15
49.70
ToolGen 56.13
52.20
47.54
53.28
50.92
62.26
34.42
51,51
define ground truth tools for query as those selected by ChatGPT. For Tama we directly input the ground truth tools in the prompt, consistent with its training data format. For ToolGen, which is not trained on data with pre-selected tools, we add a prefix during the planning phase: I am using the fol foll Lowirg wing tcols: [tool tokens], where [tool tokens] are virtual tokens corresponding to the ground-truth tools. w/ Retriever In the end-to-end experiments, we use retrieval- based setting. For baselines, we use the tools retrieved by ToolRetriever as the relevant tools. contrast, ToolGen generates tool tokens directly, so no retriever is used.
Settings w/ Ground Truth Tools (G.T.) Following al. (2023).
10-5 Context length is truncated to 6, 1.14. The total batch size is set to 512. We further use Flash- Attention (Dao et al. 2022 memory.
All models are finetuned using a cosine scheduler with maximum learning rate
Dao 2024) and Deepspeed ZeRO (Rajbhandari et al. 2020) to save
ToolGen and TooILlama follow different paradigms to complete tasks. TooILlama generates Thought, Action, and Parameters in single round, while ToolGen separates these steps. For Tool- Gen, we set a maximum 16 turns, which allows for 5 rounds of actions and final round for providing the answer. We compare this to TooaLlama, which operates with 6-turn limit.
Additionally, we introduce a retry mechanism for all models to prevent early termination, the details introduced in Section G Specifically, if model generates response containing give up I'm sorry, we prompt the model to regenerate the response with higher temperature.
Metries For end-to-end evaluation, we use StableToolBench (Guo et al. 2024), stabilized tool evaluation benchmark that selects solvable queries from ToolBench and uses GPT-4 (OpenAl 2024) to simulate outputs for failed tools. We employ two metrics to assess performance: Solvable Pass Rate (SoPR), which the percentage of queries successfully solved, and Solvable Win Rate (SoWR), which indicates the percentage of answers outperforming those generated by reference model (GPT-3.5 in this study). Additionally, we provide micro-average scores each category.
5.2 RESULTS
Table presents the end-to-end evaluation performance of various models in two settings: using Ground Truth Tools (G.T.) and Retriever (R.). In the G.T. setting, ToolGen achieves the best average SoPR score 14,19, outperforming GPT-3.5 and ToolLlama, with SoWR also highest for ToolGen at 49.70. In the Retriever setting, ToolGen maintains its lead with average SoPR of 53.28 and SoWR of 51.51. ToolLlama shows competitive performance, surpassing ToolGen on some individual instances. An ablation study of end-to-end ToolGen is provided in Appendix

---

Published as a conference paper at ICLR 2025
Table 5: End-to-end evaluation for different indexing methods.
SoPR
So
Indexing
11
12
Avg.
11
12
13
Avg
Numeric
34.76
29.87
46.99
35.45
25.77
33.02
29.51
28.79
Hierarchical
50.20
45.60
32.79
45.50
38.04
43.40
29.51
38.18
Semantic
58.79
45.28
44.81
51.87
49.69
57.55
26.23
47.88
Atomic
58.08
56.13
44.81
55.00
47.85
57.55
29.51
47.58
5.3 INDEXING METHOD COMPARISON
Similar to indexing method comparison for retrieval task (Section 4.3 Table 3 presents compari- son of different indexing methods for the end-to-end agent task. In this setting, constrained decoding removed, allowing the agent to freely generate Thought, Action, and Parameters. From the re- sults, we observe that the Atomic method achieves the best performance among the four indexing methods. We attribute this to the higher hallucination rates in the other methods, as discussed in Section 5.4]
5.4 HALLUCINATION
54.45
We evaluate model hallucination in tool gener- ation within end-to-end agent scenario. To do this, we input a query in the format the mod- els were trained on. Specifically, for ToolGen, we input a query directly and prompt the model respond using the ToolGen agent paradigm (i.e., sequentially generating Thought, Tool, and Parameters). We tested Actions decoding without the beam search constraints described in Section 3.6] For ToolL-ama and GPT-3.5, we input the query along with 5 ground truth tools. In all settings, we report the propor- tion of generated tools that do not exist in the dataset out of all tool generation actions. Fig- ure 4 shows the hallucination rates of nonexis- tent tools for different models. From the fig- ure, observe that, despite being provided with only five ground truth tools, ToolLlama and GPT-3.5 may still generate nonexistent tool names. In contrast, ToolGen, with constrained decoding, does not hallucinate all due to its design.
50
Eallmrinttoo Rate
12
8.31
7,28
3.51
2.35
0.00
0. 0.49
0.36
Tuakiion
we countrai
Atomue
Semautic
Mummes
Hierarchicl
Thell
Tball Jama epo E.Z
GFT3.
Figure 4: The hallucination rates of generat- ing nonexistent tools across different models are shown. ToolGen does generate any nonexis- tent tools when using constrained decoding. How- ever, without this constraint, ToolGen generates 7% non-tool tokens during the Action generation stage with atomic indexing, and even more with semantic indexing. For ToolI and GPT-3.5, despite being provided with five ground truth tools in the prompt, hallucinations still occur. Without any tools specified in the prompt, ToolLlama gen- erates over 50% nonexistent tool names.
CONCLUSIONS
In this paper, we introduced ToolGen, a framework that unifies tool retrieval and execution in large language models (LLMs) by embedding tool-specific virtual tokens into the model's vocabulary, transforming tool interaction into a generative task. By incorporating a three-stage training process, ToolGen equips LLMs with the ability to efficiently retrieve and execute tools in real-world sce- narios. This unified approach sets new benchmark for scalable and efficient AI agents capable of handling vast tool repositories. Looking ahead. ToolGen opens doors for integrating advanced techniques like chain-of of-thought reasoning, reinforcement learning, and ReAct, further enhancing the autonomy and versatility of L.LMs in real-world applications.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. wttphh:ss/ttiiiiiiieeeeeeeeaeeee
abs/2310. 11511
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez. Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexan- der Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong. Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RI-2: Vision-languagg-actionn models transfer web knowledge to robotic control, 2023. URL
https://arxi:: org. h abs///bs////// 15818
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.2 1787, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL htl.ntpsss/prcccd.ngsssttt ieu.s..s........................ (ewrdeeffcbb6741ooffooeoo..ee
www.://eet.AAAsaattttt..o Abstract. html
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao.
Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05911 2023a.
Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. Continual learning for generative retrieval over dynamic corpora. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 306-315, 2023b.
Jannuu Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. Continual Learning for Generative Retrieval over Dynamic Corpora. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM '23, pp. 306-315, New York, NY, USA, 2023c. Association for Computing Machinery. ISBN 9798400701245. doi: 10.114575533780.6144881. URL https:://dl...... acm. http://l...cmmoo///o/////o///// 3614821
Junzhi Chen, Juhao Liang, and Benyou Wang. Smurfs: Leveraging multiple proficiency agents with context-efficiency for tool planning, 2024a. URL https::/arxiv..rg/ wttp://///abb//0.../.
05955
Yanfei Chen, Jinsung Yoon, Devendra Singh Sachan, Qingze Wang, Vincent Cohen-Addad, Mo- hammadhossein Bateni, Chen-Yu Yu Lee, and Tomas Pfister. Re-invoke: Tool invocation rewriting for zero-shot tool retrieval. arXiv preprint arXiv:2408.01875, 2024b.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang. Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models, 2024c. URL htttttttt////attt
org/ abs /2033 12881
Tri Dao. FlashAttention 2: Faster attention with better parallelism and work partitioning. In Inter- national Conference Learning Representations (ICIR), 2024.
11

---

Published as a conference paper at ICLR 2025
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with 10-awareness. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Gravitas. AutoGPT, 2023.
URL httpss//ggttaub..coo//gggffffff Saa.iiiiiiiatt--attaaat
AUt
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Tooleehho: Towards Stable Large-Scale Benchmarking on Tool Learn- ing of Large Language Models, 2024. URL (tRt:s/tt:/// //te:////xxiii wrg/abs 2403.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL 8fdla8 c882cd45t 64 958ca (2155588662444433333 www.:aafaa-asttttttttaaoaa
http://ppppppspp....ppp wttpppppccppppepppeeeeee fi wetles/papeee httple/sapprr/22///eee
html
Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422-446, 2002.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bon- nie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, 2020. Association for Computational Linguistics. doi: 10. 3665:////020.mmmmmmmmmoo https::/acclanthol.ooo .rg. 2020.02200000 emnlp in.ii.551 550
URL
Varsha Kishore, Chao Wan, Justin Lovelace, Yoav Arizi, and Kilian Q Weinberger. Incdsi: incre- mentally updatable document retrieval. In International Conference on Machine Learning, pp. 17134-17134. PMLR, 2023a.
Varsha Kishore, Chao Wan, Justin Lovelace, Yoav Artzi, and Kilian Q. Weinberger. Incdsi: In- crementally updatable document retrieval. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd, Sivan Sabato, and Jonathan Scarlett (eds.), International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 17122-17134. PMLR, 2023b. URL htttps://proceed.nng
mlr malr.press. / wtt./kissooeee....
html
Yilun Kong, Jingqing Ruan, YiHong Chen, Bin Zhang, Tianpeng Bao, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, Xueqian Wang, et al. Tptu-v2: Boosting task planning and tool usage of large language model-based agents in real-world systems.
Yilun Kong, Jingqing Ruan, Yihong Chen, Bin Zhang, Tianpeng Bao, Shi Shiwei, Du Qing, Xiaoru Hu, Hangyu Mao, Ziyue Li, et al. Tptu-v2: Boosting task planning and tool usage of large language model-based agents in real-world industry systems, 2024.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, et al. Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents, arXiv preprint (XXv:2308.05660 2023.
Sanket Vaibhav Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Q Tran, Jinfeng Rao, Marc Najork, Emma Strubell, and Donald Metzler. Dsi++: Updating transformer memory with new documents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8198-8218 2023a.
12

---

Published as a conference paper at ICLR 2025
Sanket Vaibhav Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Jinfeng Rao, Mare Najork, Emma Strubell, and Donald Metzler. DSI++ Updating transformer memory with new documents. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8198-8213, Singapore, 2023b. Association for Computational Linguisties. doi: 10.18653/v1/20222 URL atttps:/aclaattholoe http://aaaattoooooooo///a....ooa 1
10.8653311/00333mmmmmmmm
org.
OpenAl. Gpt-4 technical report, 2024. URL UBL//ttpes///axxv..0////////3//0
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023.
15334
Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch for qa via self- planming, 2024. URL
abs 2401 05268
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li. Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating Large Language Models to Master 16000+ Real-world APIs, 2023. URL wttpps://axxx...ottm org abs/ 16789
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. arXiv preprint arXiv:2405.17935, 2024.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. URL bettp:://aaxii..rr///bbba 02054
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@) Information Retrieval, 3(4):333-389 2009.
Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, et al. Tptu: Task planning and tool usage of large language model-based ai agents.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL http:att///a.....oo///// 0477 04761
Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. Small 1lms are weak tool learners: A multi-llm agent, 2024. URL https:
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 8634-8652 2023.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 4xX11:2408.03314, 2024,
Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang. Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. Learning to tokenize for generative retrieval, 2023a. URL htt.https:// //w.x//aivvvvv/ org/ abs /204 04171
13

---

Published as a conference paper at ICLR 2025
Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. Learning to tokenize for generative retrieval. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, IAA, USA, December 10 16, 2023, 2023b. URL htt:hhttp////ppeeeee nips cc ccppaperrepppeeeeeppee files/paper/20223 Abstractttttttt..... -onfeooonneeeeeeeeeeeeee html
Qwen Team. Qwen2.5: A party of foundation models, September 2024, httpttttps//qnnn/tt wathub.io/blle gwwpp///wee22.//
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL htt.::ttt///tt.tt. //apxii wwt.oosg///oa/ooooooooo.oeoe 2wt0. 13971
Renxi Wang. Haonan Li, Xudong Han, Yixuan Zhang, and Timothy Baldwin. Learning from failure: Integrating negative examples when fine-tuning large language models as agents. arXiv preprint arXiv:2402.11651, 2024a.
Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song- Chun Zhu, and Hangxin Liu. Lim3: Large language model- task and motion planning with motion failure reasoning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 12086- 12092, 2024b. doi: 10. 0011009/R0S583392222242 10800133333
Shu Wang, Lei Ji, Renxi Wang. Wenxiao Zhao, Haokun Liu, Yifan Hou, and Ying Nian Wu. Explore the reasoning capability of Ilms in the chess testbed, 2024c. URL htttp:://axxi... org/
abs/ 20as2443
Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuging Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. A neural corpus indexer for document retrieval. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA November 28- December 9, 2022, 2022. URL http: / /ttp:::/paapes...om httpp/pppprsss..m/ wttp:e/siipps.ccpaapeeellleeeeee wimpperrffileeeeeeeeeeeeeeeeeeee /ttpa////aaappee////eepm2 waw.aaad--aaaattttooteeeeeeee
html
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou, ofain-of-thouggee prompting elicits reasoning in large language models, 2023. httlh:: wwwp:iv.oor//..o abs/ /tt.b////2211110
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. In COLM, 2024a.
Qinzhuo Wu, Wei Liu, Jian Luan, and Bin Wang. ToolPlanner: Tool Augmented LLM for Multi Granularity Instructions with Path Planning and Feedback, 2024b. URL bttps://arxiv.
orrg/bb// 14826
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for I1m problem-solving. In The 4th Workshop on Mathematical Reasoning and Al at N4.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL URL httphtts////peeeeeeii net./ Forum? 1d=zeF faa.zeeFrggyrra fg
Qiancheng Xu, Yongqi Li, Heming Xia, and Wenjie Li. Enhancing tool retrieval with iterative feedback from large language models. arXiv preprint arXiv:2406. 17465, 2024.
14

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12380-12403, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18633v/// 1024.acl-long.6777 URL https: ac aaaatnaooooyyy org 2024 acl welelllonge 670
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang. Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttun- ing: Enabling generalized agent abilities for 1lms, 2023.
15

---

Published as a conference paper at ICLR 2025
A
MORE RELATED WORK
Previous work include Toolformer and ToolkenGPT, already employed vocabulary expansion for tool learning. The main difference between our work and the others is: previous studies primarily demonstrate that through SFT (in Toolformer) or adding new tool tokens with pre-computed embed- dings (in ToolKenGPT), LLMs can learn to use a very small number of tools. However, in real-world tool-calling (agent) scenarios, previous methods require listing available tools in the prompt, which greatly limits their practical use. Examples can be seen in Figure
Other studies, such ToolPlanner (Wu et al. 2024b) and AutoACT (Qiao al., 2024), have used reinforcement learning or developed multi- systems to enhance tool learning or task completion
(Qia et al., 2024 Liu al. 2023 [Shen al. 2024 Chen al.) (024a). We do not compare our model with these approaches for two reasons: (1) Most of these works on feedback mechanisms, either through Reflection (Shinn et al. 2023) or a reward model, which similar to ToolBench's
evaluation design, where an LLM serves as an evaluator without access to ground truth answers. However, this not the focus of our study, and our end-to-end experiment does not rely on such feedback mechanisms. (2) Our method is not in conflict with these approaches; instead, they can be integrated. Exploring this integration is left for future work.
ToolkenGPT
Example
Answer following questions <add>, <subtract>, <multiply>, <divide>, Spower,, esqit>, <log?, <lem>, <god>, <ln>, <choose>, <remainder>> and <permulate>: Question: A coin is tossed times, what the probability of getting exactly heads?
Example 2:
am household robot and I take actions from [[FIN]', '[SIT]. '[SWITCHO]J, '[TURNTO]: '[LOOKAT]', [[TYPE], [[WLLKJ, "[LIE], '[GRABJ', "READD'. [[WATCH], "POINTATY" *[TOCCHJ' "[SWITCHOFFJ, "[OPENJ, [PUSH]", "PLUTORIAACK], "[CLOSFJ, "[DRINK]'.
Toolformer
Your task to complete given piece of text by using a Machine Translation API. You can do so by writing "[MT((txx)]" where text is the text to be translated into English.
Here some examples:
Input: He has published one book: O homem suprimido ("The Supressed Man")
Output: He has published book: homem suprimido [MI(O homem suprimido)]
("e Supressed Man")
ToolGen
Example (ours, the same example is provided the paper appendix) You are an AutoGPT, capable of utiliziig numerous tools and functions to complete the given task. 1.ristt, will provide you with the task description, and your task will commence.
2.At each step, you need to determine the next course of action by generating an action. ... (no specific tool API mentioned in instruction)
Could you please fetch addresses for the postcode IDN? would like know the number items found, the district, ward, county, country, and geocode details (castings, northings, latitude, and longitude).
Figure 5: Real examples from ToolkenGPT, Toolformer, and ToolGen (ours). Both ToolkenGPT and Toolformer describe tools available in the prompt, while ToolGen does not require tools been mentioned in its prompt.
16

---

Published as a conference paper at ICLR 2025
B TOOL EXTENSION AND MAINTENANCE
In ToolGen and other generative retrieval systems, tools or documents are embedded into the model's parameters. Therefore, how to add and maintenance new tools/documents become challenging. For ToolGen, not only generates the proper tool, but also fetches the documentation for that tool. If there are minor changes that the tool usage scenarios keep the same (e.g. small parameter changes), can still generate the tool and rely on the fetched documentation to do further tasks.
For vast changes that the usage scenarios are different adding totally new tools, we admit that ToolGen is not able to utilize these tools. However, this inefficiency exists and is persistent for generative retrieval systems Sun al. (2023a); Chen al. (2023b); Mchta et al. (2023a). Current methods to adapt these changes include continual training and constrained optimization (Mehta) al. 2023a Kishore et al. 2023a), which we believe could also be applied to ToolGen alleviate the above challenges.
Despite that ToolGen is inefficient of adopting to tools, its unified design lead to unique advan- tages such as easy integration with Chain-of-Thouse (Wei et al., (023). Reinforcement Learning with Human Feedback (Ouyang et al., 2022), and inference time scaling (Brown al., 2024 Snell et al. 2024
Wu et al.). We leave the problem of maintaining and adding tools to future work.
C REAL TOOL EXAMPLE
Figure shows a real tool example. Each tool is collection of several APIs. In our experiments, the following fields are used: "cool_name" is the name the tool. "tool_de desooipeittiioooo describes tool related information such as the functionality of the tool. In each API, "name" is the name of the API. "description' describes API related information. "method" is the http method for calling the API. "required parameters' parameters that must be filled when calling the API. Optionally, "options pa paraeters can be set for extra parameters.
"tool_ -nnmammm"""r "YouTube Hub",
"tool_ descrrpption "Fetch all details about single video likes, views, title, thumbnail etc. thome_url":"" "ntt:"""ttts://rappdppp con/ htt.:fffccaaaaeeettooo////////to "host"
"api list":[
"nane":"Get Video Details",
"ur1 "url:""htps://youuuu www.:hb.....ppp..cccmmmmmmmmmm
"nascriptionn""Fett. all basic information about video",
"method": "GET", "required ptweredd_aaaaerrt_____
"name":"id"
"SyRI"=""TRIN""
"STRING",
"description":""
"default": "FD6SSEYTRR"""
}, [aatooaa__prrmmeeerss[]]]]
}
]
Figure 6: A real tool example. The tool contains API. We have removed unnecessary fields for simplicity.
17

---

Published as a conference paper at ICLR 2025
D TOOL VIRTUALIZATION IMPLEMENTATION
ToolGen adopts a single and unique token to represent tool, which shows its superiority for tool retrieval and tool calling. We also introduced other methods index a tool, including semantic, numeric, and hierarchical. The following is a detailed implementation of how we implement each indexing.
Atomic indexing is the method use in ToolGen. Compared to other methods, it takes sin- gle token as a tool and does not hallucinate to nonexistent tools. We use <<tool name &api name>> combine the tool name and api name to form single token. For example, for the example in Appendix C the resulting token <<Youtube Hub&&Get Video Details>>.
Semantic indexing maps each tool the name used in ToolBench, which also combination between tool name and API name. However, the name can be tokenized into multiple tokens so that the model can perceive its semantic meanings. For the example Appendix C is wtt__vide_ddettaeeeeee wtt.oeddeaallloooooooooooeeeeeee for fou_yout ube_hub
the resulted mapping
Numeric indexing maps each tool to a unique number. We first get a list of all tools, with a length about 47,000. For all tools, we use five digit number separated by space represent the tool. If the example in Appendix iiis the 128th element the list, we use 0 1 2 8 to represent the tool. Since Llama-3 tokenizer encodes each number separately, numeric indexing will lead to tool tokens with same number of sub-tokens.
Hierarchical also maps each tool into number. Different from Numeric indexing, we inject structure information into the tool representation by iterative clustering. During each iteration, we cluster tools into ten clusters, where each cluster is assigned a number from to 9. For each cluster, repeat this clustering process until there is only one tool in the cluster. These steps form clustering tree. We take the number from root to the leaf as the representation to the tool in that leaf. The example in Appendix C may be assigned number longer than five digits, such 0 2 3 3.
E CONSTRAINED BEAM SEARCH
E
IMPLEMENTATION
During retrieval and completing end-to-end agent tasks, we use constrained beam search limit the generated actions to be valid tool tokens. The detailed steps are shown in Algorithm [ The basic idea is to limit the searching space during beam search step. achieve this, we need first build a disjunctive trie, where each node represents tool token id. Children of the node are all feasible ids following the current id. Using this tree, we can determine all possible next token ids based on current searched ids. During beam search step, we mask out all other unfeasible tokens" logits, foreing possible ids to be sampled or searched.
For retrieval, this can be directly applied during generation. For end-to-end agent tasks, since have decomposed a inference step into three conversational turns, we can easily detect when Tool- Gen needs to generate an action, therefore apply the constraint. Figure 7 shows end-to-end infer- ence example of ToolGen, where there is no relevant tools for ToolGen to choose. It can generate the tool token directly and complete the task.
E.2 BIAS ANALYSIS
For semantic indexing, the prevalent constrained beam search introduces bias toward tools with more subtokens after tokenization. Traditionally, beam search retrieves the top-k decoding sequences at each step, and the sequence probability is computed by multiplying the probabilities of each token (given previous tokens) in the sequence and then averaging by the token count. Consider the following example with two tools:
ToolA: get music from [get, music, from, usal
18

---

Published as a conference paper at ICLR 2025
Algorithm Constrained Beam Search
1: 1. Build Disjunctive Trie
2: Input: Set of tool token ids {Ids), Ids2,.... ..: Ids..
3: Initialize Root + {}
4: for each sequence Ids in the set do
5: Level < Root
6: for each token id in Ids do
7: if id Level then
8: Level[id] + 0
9: end if
10: Level < Level[id]
end for
12: end for
Trie Root
14: 2. Constrained Beam Search
Inputs: Initial Inputids; Beam width k; Language model LM
16: Output: Searched Beams
17: Initialize Beams + [[Inputdds, root T)]
18: while Beams is not empty do
19: Initialize NewBeams +-
20: beam scores <
21: for each (beam, node) in Beams do
22:
if beam ends with eos cos_token_ii then
Output beam and remove beam from beams 24: Continue
23:
25:
end if
score LM(beam)
27: feasible ids <- children of node in T
28:
Mask out ids not in feasible ids from score
29:
beam_scores <- beam_score + [score]
30:
end for
31:
Toplds, Groups Top k token ids and their groups from beam scores for each id, group in zip(Toplds, Groups) do
32:
33:
NewBeam beams[group] + [id]
34:
NewNode + node.child(id)
35:
Append (NewBeam, NewNode) to NewBeams
36:
end for
Beams NewBeams
38:
end while
ToolB: www.muusfrroo..mmmmmmmmmm www..spwiiioaa....oo..........ea bladhdlincecnseeee (a long-tail name) -> [get, music, from, spain, black, singer, ...]
After tokenization, the first three tokens are identical. For the fourth token, suppose usa has probability of 0.7, and spain has probability of 0.3. However, after spain, ToolB has a long tail of tokens with no alternatives, resulting in all subsequent tokens having probability of 1.
How should the best tool for decoding determined this scenario? Using the traditional method, ToolB's iss sequence probability increases its unique number of tokens grows with each time step. This results in that tools with more unique tokens will have a higher probability to retrieved.
As shown above, semantic tool name encoding tends to produce many long- name tools, and hence such bias becomes severe. However, there no common solution to this type of bias. Note that this problem also exists in constrained natural language decoding. Still, because language candidates are typically very large, and constraints are usually associated with ban list tokens or words, this issue is not usually considered.
Based on the above observations, we made the hallucination comparison in paper, with a setting of non constrained beam search (to avoid length bias) for other encoding methods. For atomic
19

---

Published as a conference paper at ICLR 2025
System: You are an AutoGPT, capable utilizing numerous tools functions complete given task.
1.Firtt, will provide you with the task description, and your task will commence.
2.At each step, you need to determine the next course of action by generating an action token.
3. Following the loken, you will receive the documentation of the action correponnding to token. You need Subsqquenty, you will make decisions about
generate the input of the action, transitioning you
steps, and repeat this process.
A.After several lterations of generating actions and inpuls, you will ultimately complete the task and provide your final answer.
I recently visited Viva City and I was impreased by the menu. would like to know more about the menu items at veriie 7677662266666 c09731. you provide and other relevant details? Please the language 'zh-hans' menu details.
menu category, price, availahility, rules, items
I need call the Viva City Documentation& & MeennDeeillss""" function retrieve the menu details for The reuson calling function to provide user with menu calegory, price, availability, riles, items in and other relevant details. The function called with the arguments "lang" set to "zh-hans" and "is id" set (587669276668666000 c097331 1" to ensure that the menu details are provided in the desired language and for specific venue.
verue 337769276b556660000
Generate the action.
Vivi City Documentation& Saaieiattttinn&&MMmmeeeeeeeee
Please give the input. Here is the documentation: ('name': VecViva City Documentation&& VenueDetailgoos description': GGt venue details using KV (need resync to get latest data) and filter dynamic multi language data based on query params.. 'roquired:: [f ('namee; 'id', 'type': 'STRING, deseription': 'default': (Bc46776bb004448111od
('name': 'lang', 'type: 'STRING', 'deerriptonn: ", 'dofault': 'zh-hanss)] J. 'optional': []}
I'n "is 74776692706ss66000000000o "/n "lang": "sh-hams" [m]
'language': 'ch-huns'' 'menu details:: ('sections:: [[section name': of - gt) 'items': ("item name': " 'description:: 116288 NEE 'price:: 12.99),
('emorr: 'response': ('menu jul: (tt66/992776500000000000000
('item name': 2803 descriptionn. i # " price': 15 15,9911;, l'section name': E.E, litems':
[[item name': 2 (R J. deseription': " AE 'price': 18.99), ('item name': S
deseription:: 1 2992000
'price: 22.99]]:]. 'spccial notes': 1 M000 W "3)
The menu details for venue 5
are follows:'n-
category:
cuisineln.
Price: in Ieem I: 12.99 RMBin Item 2: 15.99 RMB Item 18.99 RMB Item 4: 22.99 RMBin
Availability: in Item 1: Available'n Irem 2: Availablein Item 3; Available'n Item 4: Available'n- Rules: in
Item I: No specific rules mentioned'n Item 2: No specific rules mentioned'a Item 3: No specific rules Item
specific mentionedin. Items 3: Set of Item Set 2'm- Other relevant details; in Special noles: The menu is subject change and the prices are subject change. Please check with the venue for the most up-to-date information. IninPlease lot me know if there's anything else can assist you with!
mn Item
2: Set
Generate the action.
Finish
Please give the input. Here is the documentation: l'rqquired:: [1, 'optionall: [['name:: 'give answer, 'description'; Ourput the answer' 'typo'. 'string'), ('name: 'give_up and restart' 'description': 'Unable to handle the task from this step'. 'ype': "stinng) }]]
|rettur type": "give answer", "final answer": "The menu details for venue 5576:927M656660001 are as follows:in- Mena category: Chinese cuisine'm- Price:n Item I: 12.99 RMBin Ttem 2: 15.99 RMB Item 3: 18.99 RMB'n Item 4: 22.99 RMB'a- Availabilitylnn Item Available'n Item 2: Availablein Item 3: Available'n Item 4: Available'a- Rules'n Item 1: No specific rules mentioned' Item 2: No specifie rules
mentioned'a Item 3: No specific rules mentioneduu Item 4: No specific rules mentionedu- Items in setin Item I: Set of Item 2: Set of 2'n Item Set of Item 4: Set of 2'- Other relevant details:in Special notes: The menu subject to change and prices are subject to change. Please check with the venue the most up-to-date information.") )
Figure 7: An inference example of ToolGen. A system prompt is given first with relevant tools. Then user gives the task query. ToolGen generates the Thought, we then use user role to hint the model to generate the action. After generating the action, documentation. The model will generate tool inputs based on this documentation.
user again give the tool
encoding, hallucination and bias do not exist even with constrained decoding (not beam search) because each tool is represented by single token, ensuring unbiased and deterministic decoding.
20

---

Published as a conference paper at ICLR 2025
F INTEGRATE INSTRUCTION-FOLLOWII DATA
To ensure the model's general ability is not lost after tool-specific task training, we incorporated general caaruction-follwinng data OpenHermes-2.5 into each training stage of ToolGen We used 50:50 ratio of our tool data and instruction-follow following data, resulting in model called ToolGen- Instruct, indicating its capability to follow general instructions.
We evaluate the models general ability using 6 widely used LLM evaluation benchmarks, including ARC-easy, ARC-challenee, Commonsense QA, Hellaswag, Winograde, and GSM8K, all with 3- shot in-context examples. Results shown in Table
Table 6: 3-shot evaluation results across different NLP benchmark tasks, including ARC Challenge, ARC Easy, Commonsense QA, Hellaswag, Winograde, and GSM8K. The average performance (AVG.) is calculated as the mean across all tasks.
Method
ARC-C
ARC-E
CSQA
Hellaswag
Winograde
GSMSK
AVG.
Llama-3 50.34
80.09
69.37
60.13
73.32
49.28
63.76
Llama-3-Inst
57.16
85.31
78.05
58.69
76.24
74.45
71.65
ToolGen 20.14
33.88
19.66
31.61
54.62
1.74
26.94
ToolGen-Ine 51.62
82.49
78.79
56.33
73.16
62.02
67.40
From the table, we can see that ToolGen's general capability is limited. The original Llama got average score of 63.76, while ToolGen obtained almost random results with score of 26.94, showing a significant gap. However, ToolGen-Instn Instruct shows significant improvement, resulting in 67.4 average score.
To check ToolGen-Instructss tool learning results, we conduct the end-to-end evaluation, see Table Surprisingly, we find that its average performance improved by 3-5 percentage points compared to ToolGen. We did manual comparison and found that ToolGen-Instruct performs better on final output summarization
provides more positive responses to the user after tool calling. From this observation, we conclude that general-purpose training can help ToolGen provide more helpful responses. We leave further exploration future work.
Table 7: End-to-end agent evaluation of ToolGen and ToolGen-Instann
SoPR
SoWR
Method
11 Inst.
I2 Inst.
13 Inst.
AVG.
I1 Inst.
12 Inst.
13 Inst.
AVG.
ToolGen
56.13
52.20
47.54
53.28
50.92
62.26 60.38
34.42
51.54
ToolGen-Inst
63.09
55.03
54.10
58.84
51.53
50.81
54.24
G RETRY MECHANISM
For reproducibility, we have adopted several techniques such as fixing random seeds and setting temperatures to zero for decoding. However, we find this results some problems for whole task inference. Models tend to give up early while not trying enough for possible tools. And they are likely to say sorry when giving the final answer, which affects the end-to-end evaluation. Since our goal is to evaluate the tool usage capability, want to mitigate this negative impact that is more related to summary ability. We use retry mechanism, which simply regenerates the turn when models try to give up say sorry.
21

---

Published as a conference paper at ICLR 2025
H TOOLGEN FOR DIFFERENT SIZES OF LLMS
We also investigate how ToolGen's performance changes as the sizes of base models change. Llama- 3 series of models are not suitable as we can only use the 8B model and the 70B is too large for Therefore, select Qwen2.5 (Team 2024) with sizes of 1.5B, 3B, 7B, and 14B. As shown in Figure 8 models with larger sizes achieve better performance in tool retrieval and agent tasks. When the model size reaches 7B, the performance tends to plateau. And scaling it 14B does not necessarily improve performance. While for generalization, larger models do not show better generalization capability.
Tool Retrieval
End-to-End Evaluation
Generalization Evaluation
SoPR
52
90
56
SoWR
50
88
54
48
46
44
50
84
42
82
NDCG@1
40
-
SoPR
ND C
46
38
SoWR
1.5B
7B
14B
1.5B
3B
14B
1.5B
3B
7B
Figure 8: Performance and generalization of ToolGen with different sizes of LLMs the base model. For tool retrieval, the performance is calculated as the average score I1, 12, and 13 domains. For end-to-end and generalization evaluation, is based on the average score of unseen instructions and tools respectively.
ABLATION
Table&shows the ablation results for end-to-end evaluation. For unseen instructions, ToolGen Agent shows slightly better performance without tool memorization retrieval training. However, for unseen tools, training without the first two stages causes a drop in both SoPR and SoWR. This demonstrates that the first stage training plays role in generalization capability of ToolGen, and retrieval training is more significant compared to tool memorization.
Table Ablation results for ToolGen end-to-end evaluation. Here Inst. represents unseen queries (instructions) and Tool. and Cat. mean unseen tools during training.
Model
SoPR 12-Inst.
SoWR
1I-Inst.
13-Inst.
Avg.
11-Inst.
12-Inst.
13-Inst.
Avg.
ToolGen
54.60
52.36
43.44
51.82
50.31
54.72
26.23
47.28
w/o retrieval training
56,95
46.70
50,27
52.42
49.69
50.94
34.43
47.27
w/o memorization
56.03
47.96
57.38
53.69
49.08
59,43
34,43
49.70
I1-Tool.
I1-Cat.
12 Cat.
Avg.
I1-Tool
I1-Cat.
12 Cat.
Avg.
ToolGen
56.54
49,46
51.96
52.66
40.51
39.87
37.90
39.53
w/o retrieval training
49.47
40.31
37.90
42.84
36.71
30.07
36.29
34.18
memorization
58.86
46.19
49.87
51.70
37.34
38.56
42.74
39.32
22

---

Published as a conference paper at ICLR 2025
J GENERALIZATION
In our three-stage training process, the data in the tool memorization stage encompasses all tools. However, the training data in the second and third stages has limited tool coverage. This reflects practical scenario where we may have access to the names and documents of more tools, but less coverage of the use cases of these tools in iterative training data.
We measure the model's generalization, in this case, refers to the ability to correctly retrieve or use tools that were not included the training data for stage crucial role in achieving this, as without it, the retrieval or tool-using capabilities learned in later stages may not generalize to these unseen tools.
We believe that stage plays
First, for ToolGen Agent, we measure the performance on queries requiring tools that the model hasn't been trained on. Table ssoows the end-to-end evaluation of models on unseen tools. ToolGen Agent underperforms TooaLlama, indicating weaker generalization capability in completing tasks.
Tool Memorization for Generalization It can been seen in Table that tool memorization stage plays a role in generalization, which is validated by the drop on performance of unseen tools after removing this stage. However, this drop relatively small. We noticed that the retrieval training dataset contains approximately 500k samples, and the end-to-end training consists of 183k samples significantly more than the total number of tools (47k). This could result in most tools being seen during other training stages, which may affect the investigation of how memorization contributes to generalization.
-
To validate how the memorization stage influences the generalization abilities of ToolGen, first train ToolGen on a domain of retrieval data, and testing on another domain. Table 10 shows the tool retrieval results of ToolGen, which trained on I1 domain retrieval data and test on 12 and 13 domain. The table demonstrates that tool memorization also plays an important role, without which will lead poor generalization in tool retrieval.
We then train ToolGen on fewer retrieval data, and test its end-to-end performance on unseen tools. Table I shows the results of ToolGen on unseen tools with volume of 10%, 50%, and 100% retrieval data. As the volume of retrieval data decreases. the importance of tool memorization increases.
Table Generalization results ToolGen. We test and compare the performance of ToolGen with other models queries require unseen tools during training.
SoPR
SoWR
Model
Setting
11-Tool.
I1-Cat.
12 Cat.
Avg.
11-Tool
11-Cat.
12 Cat.
Avg
GPT-3.5
GT.
58.90
60.70
54.60
58.07
ToolI.ama
GT.
57.38
58.61
56.85
57.68
43.04
50.31
54.84
49.04
ToolGen
GT.
52.32
40.46
39.65
47.67
39.24
38.56
40.32
39.30
GPT-3.5 Retrieval
57.59
53.05
46.51
52.78
46.20
54.25
54.81
51.58
ToolI.lama
Retrieval
57.70
61.76
45.43
54,96
48.73
50.98
44.35
48.30
ToolGen
56.54
49.46
51.96
52.66
40.51
39.87
37.90
39.53
Table 10: Retrieval results of ToolGen trained 11 domain and tested 12 and 13 domain.
I2
Setting
NDCG1
NDCG3
NDCG5
NDCGI
NDCG3
NDCG5
w/o memorization
11.28
16.72
19.37
12.00
15.07 40.98
18.15
w/ memorization
40.86
50.37
55.38
33.00
49.97
23

---

Published as a conference paper at ICLR 2025
Table 11: End-to-end performance of ToolGen on unseen tools trained with 10%, 50%, and 100%
retrieval data respectively.
SoPR
SoWR
Setting
10%
50%
100%
10%
50%
100%
w/o memorization
36.60
48.67
51.70
26.68
37.32 37.60(+0.28)
39.32
w/ memorization
40.99(+4.38)
49.07(=0.40)
52.66(+0.96)
34.23(+7.555
39.53(+0.21)
K ADAPT TOOLBENCH DATA TO TOOLGEN
Our ToolGen data are adapted and converted from ToolBench data. Specifically, we adopt the tool documentations as the data for tool memorization training, where the input is tool document and the output the corresponding tokens.
For retrieval training, use the data in ToolBench that are annotated for tool retrieval, where query was annotated with several relevant tools. We take the query as input, and convert relevant tools into virtual tokens. These tokens then used outputs for retrieval training.
For end-to-end agent-tuning, we use the interaction trajectories as the sources and make the follow- ing conversions: (1) Each trajectory contains available tools in system prompt for solving the query- When completing the task, ToolLlama relies on the retrieved tools in system prompt to solve the task, while ToolGen can generate tools directly. Therefore, we remove the tools in system prompt. (2) We replace all tool names in the trajectory to corresponding virtual tool tokens. (3) In orig- inal trajectories, agent model generates Thought, Action, Action Input sequentially (also referred ReAct). We decompose the whole ReAct into three conversational turns. During the first turn, the agent model generates Thought, and we use user to prompt the model to generate an action. In second turn, the model generates the action, which are virtual tool tokens. We then fetch the document corresponding those tokens, so the model knows which parameters to specify. In third turn, the model generates parameters for the tool.
The number of samples in each dataset is shown in Table 12 retrieval training are shown in Figure 9 A sample of end-to-end agent-tuning is shown in Figure I0)
Samples tool memorization and
Dataset statistics for the three-stage training. For tool memorization, there are some repeated tools, resulting the number of samples slightly larger than the tools we used.
Table 12:
Retrieval Training
Dataset
Tool Memorization
End-to-End Agent-Tuning
T1
I2
13
All
#num
49,936
194,086
222,783
72,833
489,702
183,336
24

---

Published as a conference paper at ICLR 2025
Tool Memorization
Tool Name: Thai Drivers License OCR, Tool Description: Extract information on Thai driver's license and return text results such as driver's license number and personal information. Api Name: Driver's License Api Description: Extract the information on Thai driver's license and return text results such as driver's license number and personal information.
User:
Assistant: <haahai Drivers License OCRDriver's License>>
Retrieval Training
My friends and organizing hackathon 'web development and 'mobile app development', We need some inspiration and guidance. Can you fetch the top stories on these topics from Modium.com?
User:
Assistant: Mediimmm&& Saaddimm&&&/eeecclloopisssss
Datasets examples for tool memorization and retrieval training. We use user role to repre- sent inputs and assistant role represent outputs.
Figure 9:
25

---

Published as a conference paper at ICLR 2025
End-to-End Agent-Tuning
System:
You are un AutoGPT, capable of utilizing numerous tools and functions to complete the given task.
F.First, will provide you with the task description. and your task will commence.
2.A1 each step, you need determine the nexi course actiiio by generating an action loken.
3. F.Fllowiing the token, you will receive the documentation of the action corresponding to the token, You need generate the input of the action, transitioning you to new state. Subsequentl,, you will make decisions about the next steps, and repeat this process.
4.A several iterations of generating actions and inputs, you will ultimarely complete the task and provide your final answer.
Remember:
T.ehe state changes irreversible, and you
previous state.
2. Keep your actions concise, limiting them to best suits the current query.
You can make multiple attempts. If you plan to try different conditions continuously, perform condition per iry.
you believe you have gathered enough information, generale the action "c<4Finish>o with argument give answwrr" provide your answer the task.
you feel unable handle task this step, generate action "<<Finish>> argument give up up_and and resturt".
Let's Bogin!
Task description: You should use actions to help handle the real time user querys. Remember:
JAUSAYS generate "iiissss the of task. And the final answer should contain enough information to show to the user. you can't bandle the task, you find that actions always fail(the function not valid now), use action << Finishe> with give up and restart.
2. 2,Only generate actions and inputs.
User:
Could you please fetch the addresses for the postcode PPLIDNN! ! would like to know number of litems found, the districi, ward, county, country, and gevcode details (eastings, northings, latitude, and longitude). Regin!
Assistant:
Dased on the user's query, they want to feich addresses for the postcode 'LLIDN' and retrieve specific details like the found, district, ward, county, country, and geocode details. The current API for function 'find postcode' with the argument 'postcode' set 'PLIIDN: This API is useful bacause it provide addresses based on given UK postcode. By using this API, will be able to retrieve the roquired information for the user's query.
User:
Generate the action.
Assistant:
<x-Find By PostCode&&Find By PostCodexs
User:
Please give the input. Here the documentation: ('name'; Se<Find By PostCode&& By Post 'description:: "Find PostCade" API get audresses for the given UK PosiCade, This API provides JSON interface to seurch UK addresses for a postende., 'required:: [['name:: ppostcode', 'type:: 'string', 'description:: ", 'default;; "PLIIDNN)], 'optional:: []]
Assistant:
postcode": "PIIIDD""
Tool:
"" "r, "response": : "('status' 200, 'message'; 'Success', 'noOfftems': 37, 'result:: ('pestcode: 'LLI IDN, 'district'; PPlymouth, 'ward:: Peter and Waterfronf, 'county': 'Devon', 'country:: 'England'; 'gsocode"; ('castings:: '248225;
'northings': 00044200 'lattitude': '50,370001, 'longitade": -4, 1383977.) 'adrrcsss": [['o box:: 'line 2 'posteode type": 'ST, 'line 3:." 'su orgamisation indicator""" 'building number' '5, 'udpmn: 18911184 'throoghhfare: 'SL. Andrews Cross', "line 1t:5 Andrews Cross', ppssteude outcode': 'LL'' 'postende compact': PLIIDNN 'dependant thoroughfare!."
'premise': 'S. 'post town': PPLYMOUH. 'building name': ". 'organisation. name": 'postcode'; 'PLI IDN. department name": "posccode incode:: 'IDN', 'sddress:: 15 St. Andrews Cross, PLYMOUTH, PL1 IDN'. 'douhle dependant locality: ", Sub building name'; 'dependunt locality:: 'delivery, deliivrry_point_ suffix' (4T'), /'po_box': 'line 2:: 2 St. Andrews Cross!, 'p..
User:
Generate the action.
Assistant:
exFinishe>
User:
Please give the input. Here the documentation; l'required:: [I 'optional': ['name': 'give answer, 'description': 'Outpur the answer'. 'type. 'string),, ('name'; 'give up _p___pp_nddd__aaae and restart', 'dscrriptinn. 'Unablo to handle the task from this stop', 'type: [t
Assistant:
"retum, type": "give junswer""
"final answer": 'The number of items found: 37nnDittric:: Plymouthin Ward: Si Peter and the WaterfrontinC DevuntCCoustt::: accoedd details: Eastings: 248025 Northings: 0054420mn- Latitude: 50,3 30.37000''- Longitude: -4.138397'
Figure 10: An example for end-to-end agent-tuning.
26


Reviews:
[{"review_id": "3GjPCo2L2k", "reviewer": "Reviewer_ieJ7", "summary": "This paper proposes ToolGen, a finetuned LLMs that can use various tools during the conversations with the users. ToolGen incorporates new 47K tokens for tools into the Llama-3-8B. Through the tool virtualization, tool memorization, retrieval training, and end-to-end agent tuning, ToolGen correctly select the right tools in the context on  the ToolBench evaluation, achieving better performance than retrieval and end-to-end baselines.", "questions": "See the Weaknesses above.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- [S1] ToolGen outperform or achieves competitive performance among retrieval and end-to-end baselines on ToolBench.\n- [S2] ToolGen can natively invoke 47K tools following the context.", "weaknesses": "- [W1] The technical novelty is limited. Using special tokens for tools and incorporating them into the original vocabularies are widely-known approach (e.g. Toolformer: https://arxiv.org/abs/2302.04761, ToolkenGPT: https://arxiv.org/abs/2305.11554). The contribution of this paper is scaling this up to 47K tools, but it's very straight forward and I'm not confident if the ICLR community would be interested in it.\n- [W2] Releted to [W1], the results of ToolLlama-3 in Section 5 is unclear to me. Why ToolLlama-3 is not as good as ToolGen, even using the same data and models? Could you clarify the difference between two?\n- [W3] How about retrieving tools by LLMs itself (without retriever)? For instance, if we use the LLM with millions of extremely long context (such as Gemini), we may not need to rely on neural retrievers. It would be interesting to include long-contect LLMs as a baseline.\n- [W4] It would be important to compare other capability of LLMs before/after tool token incorporation. Current draft seems to lack the analysis after combining 47K tokens to LLMs.\n- [W5] In Figure1, I didn't understand the distinction between ToolGen's Retrieval Task and Agent Task. Both have the input \"I want some popular video games.\" How are they differentiated?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bvYRPJ2LiA", "reviewer": "Reviewer_pQUD", "summary": "The paper introduces ToolGen, a novel framework that aims to enhance the interaction between large language models (LLMs) and external tools. ToolGen shifts away from traditional tool retrieval methods and instead integrates tool knowledge directly into the LLM's parameters. This is achieved by representing each tool as a unique token, allowing the LLM to generate tool calls and arguments seamlessly as part of its text generation process.", "questions": "1. The paper highlights the efficiency of atomic indexing due to each tool being represented by a single token. However, as the number of tools grows significantly larger, could this approach lead to vocabulary explosion issues? \n2. In the current setting, the vocabulary consists of 128K regular token and 47K tool/API tokens already, how does it affect the performance of LLM itself?\n3. How does the performance of ToolGen vary with the size and complexity of the underlying LLM? Would a larger model lead to better generalization ability?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. ToolGen elegantly combines tool retrieval and execution into a single generative process, eliminating the need for separate retrieval mechanisms. This streamlines tool interaction and enhances efficiency, particularly as the number of tools increases.\n2. The use of constrained beam search during inference effectively restricts the output to valid tool tokens, significantly reducing the generation of nonexistent tools, a common issue in LLM-based agents.\n3. ToolGen demonstrates its capacity to effectively handle a large repository of over 47,000 real-world tools, highlighting its scalability compared to existing methods that struggle with vast tool sets.\n4. The authors spend good amount of effort comparing different indexing method, and the result is clear.", "weaknesses": "1. The advantage of ToolGen which combines tool retrieval and execution into a single generative process introduces limitation together with its efficiency. Since the tools are integrated into the system as tokens, the extension of new tools become inefficient. For every new tool/API, new token need to be added and the documentation finetuned into the model. Also, consider the case that when the tool/APIs get updated, the maintenance of all the tool/APIs, making sure they are up to date is a quite challenging task. On the contact, with a retriever would make adding and maintaining tool/APIs very easy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qypI39Sq9C", "reviewer": "Reviewer_SiCu", "summary": "The paper introduces ToolGen, a novel framework designed to address the limitations of LLMs in interacting with an extensive array of external tools. This framework shifts existing paradigms by embedding knowledge about various tools directly into the LLM’s vocabulary, using unique tokens to represent each tool. This allows the model to generate tool calls and arguments as part of the language generation process, eliminating the need for separate tool retrieval systems and seamlessly integrating tool usage within LLM capabilities. ToolGen incorporates a three-stage training process to enable the LLM to learn how to use tokenized tools.\n\nThe main contribution of this paper is introducing the ToolGen framework including training and inferencing, and proving that ToolGen have comparable result with current tool retrieval systems.", "questions": "- As stated earlier, why don't take the semantic representation approach as your final approach as it performs better (5 wins / 4 loses in table 2) than atomic approach? Saving tokens might not be a sufficient reason. Chain-of-thought reasoning process should produce a lot more than what atomic approach can save here.\n\n- In Table 1: \"For ToolGen in the In-Domain setting, we allow the generation space to include all tokens, which is a more challenging scenario compared to other models.\" What if you don't include all tokens? Since ToolGen can't outperform IterFeedback in the In-Domain setting currently, will it be able to outperform it without this additonal challange?\n\n- In section 4.4,  the authors are assessing the impact of different training stages, but the table shows \"−memorization\", \"−retrieval training\", \"−constraining\". If I understand correctly, constraining is an decoding strategy used in inference stage, while the third training stage \"end-to-end agent-tuning\" does not show up.\n\nPlease also pay attention to the questions and suggestions in the \"Weaknesses\" part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "**Originality:**\nThe paper introduces a novel approach to tool retrieval by representing each tool as a unique virtual token directly integrated into the LLM’s vocabulary. This method eliminates the need for auxiliary retrievers, making the retrieval process more seamless and efficient. The concept of transforming tool retrieval into a generative task is novel and presents a possible solution to the scalability challenges faced by some existing methods.\n\n**Quality:**\nThe authors have conducted a comprehensive set of experiments to validate their claims, showing thoroughness in their evaluation. These experiments include various comparisons and detailed ablation studies to assess the impact of different components of their approach. The robustness of their methodology enhances the credibility of their results and conclusions.\n\n**Clarity:**\nThe paper is well-structured and written in a clear and concise manner. It systematically outlines the problem, the proposed solution, and the experimental validations. Each section transitions smoothly into the next, making it easy for the reader to follow the logic and understand the contributions of the work.\n\n**Significance:**\nThe research addresses a critical issue in the field of LLM agents: managing and retrieving tools from a large set of tools efficiently. Given the increasing complexity and number of tools available, the proposed method provides a solution. This work has implications for the development of more autonomous and efficient AI systems, potentially benefiting some applications where tool interaction is essential.", "weaknesses": "**Substantive Assessment of Weaknesses:**\n\n**Cost and Efficiency Claims:**\nThe key claim of \"significantly less cost and higher efficiency\" does not hold up under scrutiny. The authors have not substantively demonstrated that their framework is less costly or more efficient than existing methodologies. The ToolGen framework necessitates a three-stage training process, which does not inherently suggest reduced costs. Furthermore, the paper lack data or experiments to substantiate the claim regarding efficiency improvements over other approaches.\n\n**Role of the Memorization Stage:**\nIn Section 4.4, the table shows that the memorization stage plays a relatively minor role in the three-stage training process. However, the authors assert that this stage is beneficial for generalization, with further discussion supposedly found in Appendix F. There is, however, no such discussion present in Appendix F. I would recommend authors to do more experiments to validate the significance of the memorization stage and its impact on generalization.\n\n**Hallucination Comparison:**\nSection 5.4 contains statements that do not fully make sense. The claim that ToolGen experiences no hallucination is primarily due to the use of a constrained decoding strategy. Theoretically, it is impossible to encounter hallucination, as defined by the authors, under this constraint. This does not constitute a fair comparison with other frameworks. The authors should either provide results without this constraint and compare them or demonstrate why other frameworks are unable to implement a similar strategy.\n\n**Performance Benchmarking:**\nToolGen is not currently the best-performing framework. Several baselines in the paper outperform ToolGen in various categories, as shown in Table 1 and Table 4. Notably, even the proposed but ultimately unused semantic tokenization approach outperforms the authors' current method, as shown in Table 2. This discrepancy confuses me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kskQFPF1Tb", "reviewer": "Reviewer_A87g", "summary": "This paper proposes ToolGen, a generative tool/function calling framework. The concrete methods are: (1) virtualizing tools by virtual tokens; (2) memorizing tools with training data of (tool docs, tool virtual tokens); (3) learning tool retrieval with training data of (user queries, tool virtual tokens); (4) and finally, finetuning tool agent with tool calling trajectories. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents that can adapt to tools across diverse domains.", "questions": "see the above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Overall, I like this paper very much.  \n\n1. ToolGen is now paradigm for fundamentally transforming tool retrieval into a generative process, which is a very important topic for function calling or LLM-based agents.  \n2. The methods are sound and resonable; the paper presentation is clear.  \n3. The experimental results show that the methods are effective compared with several strong baselines.", "weaknesses": "1. Traditional retrieval and generation methods for function calling can handle dynamic tools. If the tool set is changing, could ToolGen be used (without retraining)?  \n\n2. A few reltated works are not mentioned or compared. For example, TPTU and TPTU-V2 [1,2,3,] used demo retriever and fintuner besides the tool retrieval, which may be more powerful than the traditional retrieval and generation methods. They can be a strong baseline for comparision.  \n\n3. It seems that ToolGen uses a more complex process (i.e., (1) virtualizing tools by virtual tokens; (2) memorizing tools with training data of (tool docs, tool virtual tokens); (3) learning tool retrieval with training data of (user queries, tool virtual tokens); (4) and finally, finetuning tool agent with tool calling trajectories) to prepare for tool calling. Therefore, the suprior of ToolGen may come from more training of LLMs? Could the authors give more explination on this?\n\n\n[1] TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents. FMDM Workshop at NeurIPS 2023.   \n[2] TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems. LLMAgents Workshop at ICLR 2024.  \n[3] TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Industry Systems. EMNLP 2024 Industry Track.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)