From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions
Authors: Changle Qu
Sunhao Dai
Xiaochi Wei
Hengyi Cai
Shuaiqiang Wang
Dawei Yin
Jun Xu
Ji-Rong Wen
Venue: ICLR 2025 Oral (2025)
Keywords: Large Language Model
Tool Learning
Learning from Experience
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: This paper proposes a novel framework aimed at dynamically adjusting and optimizing tool documentation based on the interaction feedback between LLMs and external tools.

Abstract: Tool learning enables Large Language Models (LLMs) to interact with external environments by invoking tools, serving as an effective strategy to mitigate the limitations inherent in their pre-training data. In this process, tool documentation plays a crucial role by providing usage instructions for LLMs, thereby facilitating effective tool utilization. This paper concentrates on the critical challenge of bridging the comprehension gap between LLMs and external tools due to the inadequacies and inaccuracies inherent in existing human-centric tool documentation. We propose a novel framework, DRAFT, aimed at Dynamically Refining tool documentation through the Analysis of Feedback and Trials emanating from LLMs' interactions with external tools. This methodology pivots on an innovative trial-and-error approach, consisting of three distinct learning phases: experience gathering, learning from experience, and documentation rewriting, to iteratively enhance the tool documentation. This process is further optimized by implementing a diversity-promoting exploration strategy to ensure explorative diversity and a tool-adaptive termination mechanism to prevent overfitting while enhancing efficiency. Extensive experiments on multiple datasets demonstrate that DRAFT's iterative, feedback-based refinement significantly ameliorates documentation quality, fostering a deeper comprehension and more effective utilization of tools by LLMs. Notably, our analysis reveals that the tool documentation refined via our approach demonstrates robust cross-model generalization capabilities.

Published as a conference paper at ICLR 2025
FROM EXPLORATION TO MASTERY: ENABLING LLMs TO MASTER TOOLS VIA SELF-DRIVEN INTERACTIONS
Changle Qu' Sunhao Dai', Xiaochi Wei Hengyi Cai?, Shuaiqiang Wang", Dawei Yin', Jun Xu'; Jo-no Wen' 'Gaoling School of Artificial Intelligence, Renmin University of China, BBidd Inc. [changlequ, sunhaodai, junxu, jrwen @ruc. edu. en, weix mixxiiocch @ba du. com caihengyi@ien ac. cn, shqiang. wang@gmail com, y yindawe acm. org
ABSTRACT
Tool learning enables Large Language Models (LLMs) to interact with external environments by invoking tools, serving effective strategy to mitigate the limitations inherent in their pre-training data. In this process, tool documentation plays crucial role by providing usage instructions for LLMs, thereby facilitat- ing effective tool utilization. This paper concentrates on the critical challenge of bridging the comprehension gap between LLMs and external tools due the inadequacies and inaccuracies inherent in existing camann-centris tool documenta- tion. We propose a novel framework, DRAFT, aimed Dynamically Refining tool documentation through the Analysis of Feedback and Trials emanating from LLMS' interactions with external tools. This methodology pivots on an innovative trial-and-error approach, consisting of three distinct learning phases: experience gathering, learning from experience, and documentation rewriting, to iteratively enhance the tool documentation. This process is further optimized by implementing a diversity-promon promoting exploration strategy to ensure explorative diversity and a tool-adaptive termination mechanism to prevent overfitting while enhancing effi- ciency. Extensive experiments on multiple datasets demonstrate that DR FT''s iterative, feashack-based refinement significantly ameliorates documentation qual- ity, fostering deeper comprchension and more effective utilization of tools by LLMs. Notably, our analysis reveals that the tool documentation refined via our approach demonstrates robust cross-model generalization capabilities.
INTRODUCTION
Tool learning Mialon et al. 2023 Qin al. 2023b; Schick et al. 2024 Qu al. 2024b), which integrates external tools with large language models (LLMs), has significantly enhanced the capability of L.MS to address complex real-world tasks (Nakano et al. 2021 Qin et al.) 2023a M. Bran et al.. 2024). By leveraging external tools, LLMs are able to mitigate the limitations of outdated pre-training data and the text-in-text-out interface, enabling them to access up-to-date information, interact with dynamic environments. and take actions beyond their original scope (Zhuang et al. 2024 Wang ct al. 2024a). To effectively utilize these external tools, LLMs are typically provided with tool documentation context (Shen et al. 2024 Song al. 2023 Xu at. 2023). This documentation provides essential information on how tools function, their potential uses, and the ways in which they can be leveraged solve complex tasks. By incorporating tool documentation within the task instructions, LLMS can leverage their in-context learning abilities to understand and utilize the tools efficiently (Wei al. 2022 Hsieh et al., 2023). Therefore, tool documentation is an indispensable component driving the success of tool learning, serving as bridge between LLMs and external tools.
However, existing tools primarily originate from pre-established. human-engineered code reposi- tories and are not explicitly tailored for the utilization of LLMs from their inception, let alone the corresponding tool documentation.
fact, orchestrating an ideal documentation for an external tool that adapts to the specific requirements of LLMs remains a challenging endeavor. First, the original human-crafter tool documentation is typically created with human intuition in mind, and
Jun Xu is the corresponding author.

---

Published as a conference paper at ICLR 2025
Incomplete Decumentation
What this tool used for
000
0
"name" "GetSearchimgge"
Why did it go wrong??
Redundant Decumentation
Tears name" "mageed""
000
O O
00
000
O
O
Vescripton" Ttrage klnnffir.
Emor resource not
What is the key information? ?
(b)
Iaaccuraee Ducumentation
"usccrision "A code generater
(
? Asiton
Tic
Lose
cevelopers
Why
match the deecription?
standl Hern are the renoone:
Oscumentation& ia easker to under
Restllench- TMDB
92.55
12.5%
"deschiplza"" " "SQUARE Ta businesses
"DDtaill and Clarity":
"Enrer Handling"
RextBench-Spoify 77.5%
5%
pefformare API the markel
to build sustainablee productsl With the most
"Faramelers Explanation":
Provided"
TouiBench
87.9%
11.2%
(a)
Actusl return
teapol.
(c)
0.7%
Figure 1: (a): An illustration showcasing issues in current tool documentation. including incomplete, redundant, and inaccurate information. (b): An illustration of how LLMs incrementally enhance their understanding of tool usage through iterative exploration, transitioning from untested functionalities (gray) to discovered capabilities (white) via a anal-rnd-error process. (c): The comparison between the raw and improved tool documentation regarding its usefulness on three tool learning datasets, highlighting that DRAFT is able to produce tool documentation that is more favored by LLMs.
often fraught with inconsistency and ambiguities, as primarily caters to human understanding and usually lacks the precision required for machine interpretation (Chemero) 2023 Yuan et al. 2024). As illustrated Figure [ (a). incomplete documentation makes it difficult for LLMs to clearly understand the purpose of tool and when it should be invoked, while redundant documentation, containing excessive irrelevant information, obscures key details and increases token consumption in prompts. Additionally, inaccurate documentation that does not reflect the tool's actual capabilities can lead to discrepancies between the tool's outputs and its description, leading to the potential misuse of the tool by LLMs. These issues obstruct the effective utilization of tools by LLMs. Second, manual modification of these documentations, even with meticulous revisions, struggle to fully encompass all aspects of tool usage, since discerning the specific scope tool can manage and identifying its edge use-cases often necessitates considerable hands-on experience. For example, as depicted in Figure D (a), incomplete documentation also fails mention certain constraints on the parameters, and the LLM, being unaware of this undocumented constraint, generates an eтror when invoking the tool with an invalid parameter value. Addressing these issues by manually correcting or enhancing tool documentation is time-consuming and labor-intensive. Moreover, this may not scale to sub- stantial number of tools effectively. Furthermore, the dynamic nature of tool development further exacerbates this issue, as functionalities of tools are frequently updated, deprecated, or extended. Maintaining an up-to-date and accurate representation of such evolving functionalities within the tool documentation becomes arduous task. This misalignment between the tool documentation with the current state of the tool hinders the efficient and correct utilization of tools by LLMs.
Humans, the contrary, acquire tool proficiency through repeated interactions and hands- on periences, capable of maintaining an updated comprehension of these tools despite their evolving functionalities. In light of this, this paper proposes DRA AFT, conceptualized automate the ad- justment and optimization of tool documentation based on the feedback derived from the LLM's interaction with the tool, aiming to bridge the comprehension gap between LLMs and external tools.
More concretely, DRAFT implements a trial-and-error methodology to incrementally improve the tool documentation. As shown in Figure DRAFT orchestrates three dynamically interlinked phases, which collectively facilitate the iterative process of documentation enhancement. It first undertakes the simulation of potential tool application scenarios, crafting explorative instances and capturing tool execution outcomes through a designed explorer. Subsequently, the analyzer dissects the prevailing documentation, amalgamating insights from the explorer's 's findings and feedback to moot documentation modification propositions. Finally, the rewriter amalg amalgamates these insights, refining the tool documentation while simultaneously guiding further explorative pursuits by the explorer. exploration strategy to ensure diversity
To optimize this process, we design a diversity-promottin in exploration, thus providing a wider range of samples for subsequent rewriting. Recognizing that different tools vary in complexity for LLMs, we introduce a tool-adaptive termination mechanism to improve efficiency during modifications by halting the iterative process once the documentation aligns with the comprehension of I.LMs, thereby saving time and resources while preventing overfitting.
2

---

Published as a conference paper at ICLR 2025
O Tool
Diversitypromottng Exploration Strategy
Tool Name: GET
Tool
Description: Added the [transations] (Handpeint CSMMCCXXIGGs
Explorer Mr
Experience Gathering
Similarity Constraint
Query: Cia you give ae details about the
person with ID 12345?
Tuol-ase Exampled
Explored Queries
New Generated
Tool Parumetens: ('ID"-11245")
Observation: "hiography"" "hintday" 11919-11 29", "deathday": 11996-01 01",
Query
Generate Query
use
Self-Reflection
"gender": 2, "homepage" mall ...)
Learning from Experience
Tool-adaptive Termination
Here Include Response Fields;... Clarity Parametens: ... 3. Explain "append, to
Tool
Examples
Tool-use
S
Mechanism
esponse": 4. Update Recent Change
M
Analyze Decumentation Problem
Tool Doct
Tool
Tool
Remove Inelevant Information:
5. Provide Example Usage:
Documentation Rewriting
Dacl
reteves information about person using their unique ID. The resporse inchules The API supports the appe nd respaaaess" patameter... Example usage: Direction Next Exploration:
Rewriting:
AB
Tool
Examples
Toul-use
Doc
Doc2
Tool
Tool Dac2
Teol
Dac2
Tool
Rewriter M
Comprehensive Rewriting
Figure 2: Schematic illustration of our proposed self-driven iterative improvement framework, DRAFT. The left part illustrates the three distinct learning phases in our framework, where trial refers to single iteration of the process. The middle part shows the two specialized mechanisms designed to optimize the process. The right part depicts an example of using DRAFT to iteratively modify the tool documentation (a complete revision trajectory is presented in Appendix EE
Through this collaborative, iterative framework, the documentation progressively morphs, aligning more coherently with LLMs operational requisites, and thus empowering LLMs to leverage external tools more effectively in their problem-solving endeavors.
In summary, our contributions are as follows: (1) We highlight that due to the inherent understanding gap between LLMs and humans, inefficiencies and inaccuracies within existing tool documentation hamper the effective utilization tools by LLMs. (2) We introduce a novel framework, DRAFT, designed to dynamically adjust and optimize tool documentation based on the interaction feedback between LLMs and external tools, which significantly bridges the gap between them by enabling the LLMS better comprehend and utilize the tools their disposal, thereby enhancing the overall tool-using capabilities of LLMs. (3) Extensive experiments demonstrate that DRAFT significantly improves the quality of tool documentation and enhances the ability of LLMs utilize external tools.
METHODS
In this section, we will first introduce the overview of DRAFT, and then provide detailed explanations of the three included learning stages. The learning algorithm is presented in Algorithm
2.1 OVERVIEW OF DRAFT
To address the challenges of inadequate, ambiguous, and outdated tool documentation that hinder LLMs from effectively utilizing external tools, we propose DRAFT, framework that iteratively refines tool documentation to bridge the comprehension gap between LLMs and tools. As illustrated in Figure 2 DRAFT operates through three interconnected phases: experience gathering, learning from experience, and documentation rewriting. In the experience gathering phase, an explorer simulates diverse tool usage scenarios, collecting data how the LLM interacts with the tool based on the current documentation, thus uncovering misunderstandings and limitations. The learning from experience phase involves an analyzer examining this data to identify discrepancies between intended and actual tool usage, pinpoint anbiguities or inaccuracies in the documentation, and propose targeted improvements. In the documentation rewriting phase, a rewriter integrates these insights to update the documentation, enhancing clarity and alignment with the tool's functionalities. Through this trial-and-error framework, DRAFT is capable of simulating the process by which humans acquire proficiency in tool usage through repeated interactions and hands-on experiences, thereby automating the creation of tool documentation specifically designed for LLMs. Furthermore, by employing the diversity-promoting promoting exploration strategy and tool-adaptive termination mechanism, DRAFT efficiently converges on optimized documentation, enabling LLMs to utilize external tools more effectively despite the initial documentation shortcomings.
3

---

Published as a conference paper at ICLR 2025
Algorithm 1:
The Learning Algorithm of DRAFT
Input: Raw tool documentation set DD iteration round I, similarity threshold termination threshold т.
Output: Revised tool documentation set D.
Initialize the revised tool documentation D for raw tool documentation E D do
for 1 I do
// Experience Gathering 2.2
Instruct Explorer generate exploratory instance e, using Eq.
while ma sim(e" e > Φ do
Instruct Explorer to generate end
exploratory instance
Instruct Explorer to capture the outcomes of tool execution r:
10
// Learning from Experience encecinne (.))
1I
Instruct Analyzer to learn from experience and provide suggestions for modifications using Eq. // Documentation Rewriting 234
Instruct Rewrites revise the documentation based on experience and suggestions get revised tool documentation f and propose new exploration directions d; using Eq.
14
Calculate the similarities A between 4 and t; using Eq.
15
if A > 7 then
Break
17
end
18
end
Updating the revised tool documentation set + DUt.
19
20
end
return D
2.2 EXPERIENCE GATHERING
In the experience gathering phase, we design an Explorer ME to simulate plausible scenarios in which the tool may be utilized. This approach paral- lels the manner in which individuals investigate the potential applications of a new tool when they are unable to comprehend the accompanying manual.
Explorer
Prompt:
Your tnk is arower the user's query as best you can. You love nerras
tool calling help with the following (Dccomentation)
your reponse:
Now you love the charee to explore the avsillabll tnels, by ...
Her is example: ("User Query", "; "Furameiers"::: :f
Hil Memory
Below ure queries have already explored: [Esplored queries)
Specifically, at the i-th iteration, the Explorer Mg generates exploration instance e; based on the cur- rent tool documentation f4-1. next-step exploration direction d,-1 from the Rewri ter MR, and the pre-
explore quarice
Here are some siggestions to explore the took: (Suggestions)
Figure Prompt template for explorer.
vious history H4 = ri < i}. which consists of prior exploration instances ezi and their corresponding return results of the tool r<i. This process formalized as follows:
= :gg=M((t-1.dd H.),
(I)
where e; consists of a user query ef related the tool and the necessary parameters e₁ The initial tool documentation is denoted as to. representing the raw documentation provided in the dataset. After generating e,, the Explos orer invokes the tool to obtain the result r; returned by the tool.
Given that tool utilization often involves complex parameter ranges, combinations, and potential error sources, is crucial to ensure diversity in the exploration phase to cover wide spectrum of possible scenarios (Hong et al.) 2018 Friedrich et al. (2009). To address this, besides maintaining a record of all previously explored queries-which which we provide to the Explorer to instruct to generate instances that differ from those already generated-ww we also implement a diversity-promon proomotin exploration strategy:
Similarity Constraint. When generating
instance, the Explorer calculates the cosine similarity between the new generated query eς and all prior queries eÿ for j < i, using embedding The similarity is computed as:
vectors obtained from OpenAl's sext--mbedding-uda-aee
max sim(e? e²) < ját
4

---

Published as a conference paper at ICLR 2025
where sim(..-) denotes the cosine similarity function, and ø is predefined threshold controlling the allowed similarity. This constraint ensures that the new query is sufficienlly different from all previous queries, promoting diversity in exploration.
Self-Reflection. If the similarity constraint is not satisfied (i.c., the new query is too similar to previous ones), the Explorer r engages in self-reflection (Shinn et al.) 2024). It discards the current instance and analyzes the reasons for the overlap, adjusting its approach to generate a new query that explores different aspects of the tool. This iterative self-reflection process continues until the Explorer produces produces an instance that meets the diversity criterion.
By incorporating this peremoity-roomoiing exploration strategy, we enhance the exploration coverage of the Explorer, allowing it to investigate broader range of tool functionalities and edge cases. This comprehensive set of exploration instances crucial for identifying potential misunderstandings and gaps in the tool documentation, thereby providing valuable experiential data for subsequent analysis and documentation rewriting, Figure Aigure3iilusste illustratee our prompt template for Explorer.
2.3 LEARNING FROM EXPERIENCE
Analogous to how humans Iearn coam-acquirngg familiar- ity with new tools through practical experiences and then consulting manuals deepen understanding- the insights gained during the experience gathering phase provide foundation for informed and targeted enhance- ments to the documentation. Thus, building upon the experience gathered in the first phase, the second phase focuses on analyzing this data to refine the tool docu- mentation. this phase, we introduce an Anall yzer MA designed to identify and address issues within the
Analyzer
de Task Prompt:
based proviske suggesion modifying ducumeniation
current documestation, explored querics pontmnttrs, and the resals returnd by the tool You have access to the following 1oos: [Teel Decumestation)
Below are explored queries the required parameters and the ourpur of kol (Explored examples)
Here an cumple: ("Segaestioss"
au Memory Mechanisms:
The followigg the histry of you modifying the sool description: [History]
Based shove
provide
Figure 4: Prompt template for analyzer. current tool documentation, thereby guiding the Rewriter in making effective revisions.
Formally, at the i-th iteration, the Analyzer MA takes the following inputs: current tool docu- mentation 1;-1, exploration instance e,, tool feedback v; provided by the Explorer Mr. and the history of documentation revisions T, {t, j < i}. Then the Analyzer MA analyzes these inputs to identify issues and generate revision suggestions s,
S (a T:).
To ensure that the Analyzez can provide high-quality and relevant revision suggestions, we establish several evaluation criteria, including consistency with tool outputs, comprehensiveness, and concise- ness without irrelevant information (as detailed in Appendix(C). These criteria enable Ana 1 yyeer to identify and assess issues present within the existing tool documentation. By considering the historical evolution of the documentation through the revision history Tú the Analyzer gains valuable insights into past updates, helping it avoid redundant or repetitive suggestions and focus on areas that still require improvement. The prompt template for Analyzer is illustrated in Figure 4
Furthermore, the Analyzer delivers its feedback in natural language, offering detailed and nuanced guidance the Rewriter for subsequent updates. This approach contrasts with providing mere scalar feedback, as it ensures the Rewri ter receives comprehensive insights that facilitate accurate and effective revisions, ultimately enhancing the clarity and usability of the tool documentation.
2.4 DOCUMENTATION REWRITING
Building upon the experiences gathered and the revision suggestions obtained from the previous two phases, the final phase focuses on refining the tool documentation to enhance its clarity, accuracy, and usability, ensuring it aligns with the comprehension capabilities of LLMs. This phase also provides suggestions for future exploration directions in the next iteration of the experience gathering phase.
Specifically, design Rewriter MR to synthesize information from the exploration instances e; and the corresponding tool return results r3 provided by the Explorer Mg, as well as the revision suggestions s; from the Analyzer MA. It is important to note that the Rewriter MR also takes into account the rewrite history T., which includes all previous versions of the tool documentation up to iteration i. By integrating these inputs, the Rewri ieer Mn produces an updated version of the
5

---

Published as a conference paper at ICLR 2025
tool documentation i, and provides suggestions for the next round of exploration directions d,. This process is formalized as:
d = M e,, T:).
(4)
By incorporating the revision history into its process, the Rewriter ensures that each version of the documentation builds upon its predecessors, resulting in continuous improvements in clarity, accuracy, and usability. The prompt template to get Rewriter shown in F
Furthermore, recognizing that different tools vary
Rewriter
in their complexity and the case with which LLMs
Hf Task Prompt:
refine doscription
can comprehend them (Qin et al. 2023b; Osiurak &
panmetes you've explored, outpots provided
suppestions provided. You have accoss the following tools: [Too]
Heinke 2018), we implement a tool-adaptive ter-
Decumentation] ...
mination mechanism to adaptively determine when
Bdoo se explosed querios, the noquinal prramcters and the ouiguts of the not jisppoored examples)
to cease modifications for each tool. Analogous to recipes requiring different levels of expertise, some tools may reach optimal documentation faster than others. We consider the iterative process to have con- verged when there is minimal change between two consecutive versions of the documentation, indicat- ing that the Rewr i ter has sufficiently aligned the documentation with the LLM's understanding.
Additionalyy here some sugaestions for updaiing the wool description: (Sugration))
Here example: "Rewritten description "Sugzestions explorieg"; !"]
Hil Memory Meclannssms:
The following is the hisory of you modifying the tool deseriptoo: (Ulsstry] Roood on the information above, rewrite the wol deveription and privide constrocive suggestions for explorarion
Figure 5: Prompt template for rewriter.
Specifically, inspired by Wieting al. (2019), we measure the degree of change A between iterations by caleulating both the word-match metric (e.g., BLEU score (Papineni et al. 2002 and the manantic-matei metric (e.g., cosine similarity of embeddings):
sim(e), et_1) - (LEU(4,,44)
(5)
where et and ei-1 are the embedding vectors of t; and - obtained using OpenAl's text-embedding- ada-002 The function sim(..) calculates the cosine similarity between the semantic embedding vectors of two documentation versions, and (LEU(;,)) measures the n-gram overlap between them.
If A exceeds predefined termination threshold r, we stop the iterative modifications.
This tool-adaptive termination mechanism offers several advantages: First, it enhances efficiency by ceasing iterations when the documentation is adequately aligned with the LLM's comprehension, conserving computational resources and time. Second, it prevents unnecessary modifications that could lead overfitting, thus optimizing the quality of the documentation. By employing both the BLEU score and cosine similarity, we ensure a balanced assessment of structural and semantic alignment, ultimately yielding high-quality documentation tailored for effective LLM utilization.
2.5 DRAFT's STRENGTHS
In this section, we outline the key strengths of the proposed DRAFT framework. First and foremost, DRAFT operates in fully automated manner, which significantly reduces resource consumption in comparison to the time-consuming and labor-intensive manual modifications that are typically required. Furthermore, by employing a trial-and -nrr-ord-rrror methodology. DRAFT continuously updates tool documentation based on feedback regarding tool usage obtained from LLMs, thereby enhancing the alignment between tool documentation and the operational understanding of L.MMs. Additionally, DRAFT is capable of dynamically maintaining an accurate and up-to-date representation of evolving features within the tool documentation as the tools develop. It also provides inherent explainability, as the entire process is presented in natural language. Users can easily track the history of modifications and seamlessly integrate expert insights into the updating process.
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Datasets. To verify the effectiveness of DRAFT, we conduct experiments on two benchmarks: ToolBench and RestBench. ToolBench (Qin et 2024 large-scale benchmark of real-world

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparison of different methods on three datasets. Win% is caleulated by comparing each method with ReAct. The term "-" means that Easy'Too has not been implemented on the Spotify dataset. The best result for each LLM is in bold.
RestBench- TMDDB
RestBench-Spotify
ToolBench
Model
Method
CP%
Win %
CP%
Win%
CP%
Win %
ReAct
48.00
50.00
24.56
50.00
35.00
50.00
DFSDT
50.00
68.00
35.08
61.40
37.00
84.00
GPT-4o-mini
EasyTool
56.00
75.00
42.00
85.00
DRAFT (Ours)
62.00
82.00
43.85
78.94
47.00
88,00
ReAct
72.00
50.00
26.31
50.00
41.00
50.00
DESDT
74.00
38.00
63.15
61.40
42.00
54.00
Llama-3-70B
EasyTool
76.00
64.00
46.00
60.00
DRAFT (Ours)
86.00
64.00
66.66
64.91
53.00
62.00
ReAct
71.00
50.00
28.07
50.00
37.00
50.00
DFSDT
74.00
61.00
64.91
56.
41.00
73.00
GPT-4o
TosyTool
79.00
62.00
45.00
77.00
DRAFT (Ours)
88.00
71.00
70.17
84.21
51.00
78.00
APIs collected from RapidAPI and BMTools, commonly used to evaluate the capability of LLMs in tool usage. Due to budget constraints, we focus on the most challenging subset of ToolBench, namely I3-Instruction, which contains complex requests requiring multiple tools from different categories. RestBench (Song et al.) 2023) a benchmark consisting of two real-world scenarios: TMDB, which includes 54 movie-related APIs, and Spotify, which has 40 music-related APls.
Evaluation Metrics. Following previous work (Song et al. 2023 Qin et al. 20224 Yuan et al. 2024
we evaluate performance using two widely adopted metrics: (I) Correct Path Rate (CP%) which measures the proportion of instances where the model gedee-eeee sequence of tool calls contains the ground truth tool path as subsequence, allowing for straightforward accuracy assessment. (2) Win Rate (Win %), which evaluates effectiveness through pairwise comparisons by a ChatGPT-basee evaluator, capturing nuanced performance differences not reflected by rule-based metrics.
Baselines. Following Qin et al. (2024) and Yuan et al. (2024), we compare our method with widely adopted baselines, including: (1) ReAct (Yao al., (022). which integrates reasoning with action, enabling LLMs not only to justify their actions but also to refine their reasoning processes based on feedback from the environment, (2) DFSDT (Qin et al. 2024), which addresses the issue of error depth-first search strategy to enhance decision-mall makiing accuracy. (3)
propagation by incorporating EasyTool (Yuan et al.) 2024), which achieves more concise tool descriptions by using ChatGPT to directly rewrite the documentation and incorporate guidelines, thereby enhancing the comprehension of L.LMs regarding tool functions and parameter requirements.
Implementation Details. For the main experiments, we the GPT-40 the backbone model for DRAFT, which means that we employ this model to refine the tool documentation by incorporating its tool usage feedback. We the similarity threshold & 0.9, termination threshold to 0.75, and maximum iteration count to 5. We select three of the latest LLMs to ensure our evaluation reflects the current state of the field including the closed-source models GPT-4o and GPT-4o-mini as well as the soen-source model Llama-3-70B* Our code is available at https 7/gi thub. com/ wowm:/uchagggel1/// |DRAF
3.2 EXPERIMENTAL RESULTS
We present experimental results in Table I Based on these results, have the following observations: While Easy Tool can slightly improve experimental performance, it does not incorporate the experience feedback from LLMs to iteratively revise the tool documentation. As result, it cannot fully align with the understanding of LLMSs. In contrast, our method effectively addresses these issues and achieves more significant improvements. We observe that all LLMs achieve better performance
https:/pllafoomm wwwp://mmmmpomiiiimmmmm htwp://wwwoooooooooooooo/ooaaaaa http:///tiiio.o....ccoooooooummn

---

Published as a conference paper at ICLR 2025
Connec Race5%))
64.0
5412
Conrect Patk Ro6
40.0
920
62.0
620
45.0
82.0
-0.0
5300
60.0-
500
44.0
76.0
8500
28.0
780
47.0
700
850
15.0-
7800
40.0
700
540
54.0-
RanDeac--HOR -o0--d------ Warh
740
280
Reuteenc Spazt: Warl
6600
Toaflanah Whrt:
820
724
35.0-
35.
830
Ieeration rounds
Deration rounds
Deration rounds
(a) GPT-44-mini
Conner Path R31e%))
00.0
164
Coneer Path (tsee(9)
Coreer koteet))
51.0
65.1
06.0
64.
05
6600
62.0
64.0
64.0-
1220
Raleys
64.0
64.
620
82.0-
6000
62.0-
62.0
48
6300
00.0-
G20-
<c00
51.0
13.0
Reolwwch TuTE Wiits
36.0
580
krulinnc Spariyy Wosk
5600
44.0
Traflanh Work
5500
26.0
540
360
20
5400
Dieration rounds
Ttenation rounds
Ttention rounds
(b) Llama-3-70b
Porh Rauo%)
740
Conrect Pach Maie %5)
62.0
860
18.00
-724
70
Correct Rater
-2
221
86.0
Rateyso
Ran
20.0
21.0
64.0-
1880
62.0-
14.0
40.0
140
02.0-
66.0
58.0
7000
44.0
7300
003-
248-
28,0
62.0
300
62.0
22.0
620
Deration rounds
Iteration rounds
Iteraion rounds
(c) GPT-4o
Figure 6: A comparative analysis of performance based varying numbers of iteration rounds.
when using the tool documentation modified by DRAFT. This indicates that although our tool documentation was revised through utilizing tool usage feedback from a single model, it exhibits robust cross-model generalization capabilities. Notably, on the ToolBench dataset, the GPT-4o-mini enhanced with DRAFT even surpasses the performance of the GPT-4o without DRAFT. These improvements demonstrate the effectiveness of DRAFT, which can be attributed to the fact that ARAFT iteratively learns from the tool usage experiences of LLMS to refine the tool documentation. In this way, DRAFT creates tool documentation specifically aligned with the understanding of LLMss
3.3 FURTHER ANALYSIS
How does the number of iteration rounds affect the performance of tool learning? A key feature of ARAFT'S is its iterative refinement of tool documentation with tool-adaptive termination mechanism. We examine the effectiveness of these designs by analyzing how the number of iteration rounds influences the performance of downstream tool learning. The results presented in Figure 6 demonstrate general trend where performance is enhanced with an increasing number of iterations, followed by a subsequent decline. This suggests that iterative modifications are crucial for enhancing tool documentation quality and the ability of L.LMs utilize tools effectively. Such modifications facilitate the exploration of a wider array of examples and the incorporation of additional feedback derived from the tool usage experiences of LLMs, ultimately refining the tool descriptions to achieve superior performance. However, a decline in performance is observed after a certain number of iterations. This decline may be attributed to the introduction of redundant information as the number of iterations increases, potentially leading to overfitting. Therefore, we implement a tool -ddativee termination mechanism to prevent performance degradation and ensure optimal results.
Does using other models as back- bones also ensure cross-model gen- eralization? Our preliminary experi- ments have demonstrated that employ- ing GPT-40 as the backbone to re- fine tool documentation by integrat- ing its own usage feedback results in revised documentation that exhibits cross-model generalization, enhancing the performance of other models. This observation prompts
90.0
85.0
80.0
75.0
70.0
65.0
Correct
60.0
$5.0
50.0
45.0
40.0
35.0
mini
Figure 7: An analysis of cross-model generalization.
8

---

Published as a conference paper at ICLR 2025
an inquiry into whether using other models as a backbone would similarly yield cross-model general- ization. To investigate this, we conduct experiments on RestBench- TMDB using Llama-3- 3-70BB as backbone, which led to the generation of a new set of tool documentation informed by its usage feedback. As illustrated in Figure 7 our findings indicate that it also demonstrates cross-model generalization, enhancing the tool usage capabilities across all L.MMs. This may result from decoder- only models sharing similar transformer structures and common pre-training corpora, allowing them to achieve consensus on knowledge mastery and needs. Furthermore, we also find that employing GPT-40 as a backbone yields superior performance compared to LLama-3770B. This suggests that our method can benefit from continuous improvements in foundational models.
Are the two mechanisms we proposed truly effective for DRA We conduct ablation studies on MetBBench-TMDDD using GPT-4o to assess the impact of the two mechanisms in- corporated within DRAFT. The results presented in T highlight the significance of each mechanism: w/o diversity refers to a variant that the explorer generates exploration in- stances without applying similarity constraint, meaning that all generated instances are considered. The absence of the diversity-promotte exploration strategy leads notable per- formance drop, underscoring the need for its implementation to enhance exploration diversity. w/o
Table 2: Ablation study of the pro- posed DRAFT.
TMDB
Mchtods
CP%
Win
DRAFT
88.00
71.00
w/o diversity
84.00
69.00
w/o adaptive
80.00
68.00
adaptive refers to a variant that does not terminate prematurely based on the iteration conditions; rather, it guarantees that each iteration of the tool documentation completes the designated maximum number of iteration rounds before ceasing operations. The significant decline in performance ob- served in this variant further emphasizes the benefits of introducing the tool-adaptive termination mechanism, which serves to prevent overfitting while improving efficiency. The results with other LLMs and datasets demonstrate similar trends and are provided in Appendix B
Does the modified tool documenta- tion improve the performance of tool retrieval? In world scenar- ios, there are often numerous tools, making it impractical to input the de- scriptions of all tools into LLMs (Qu jet al., 2024a). Therefore, effective tool retrieval is crucial for down- stream tool learning. In this analy- sis, we evaluate whether the modified tool documentation not only enhances the ability of LLMs to use tools effec- tively but also improves the effectiveness of tool retrieval. The results presented in Table 3 indicate that the revised documentation enhances the performance tool retrieval methods, including the sparse retrieval method BM25 (Robertson al. 2009) and the dense retrieval method Contriever (Izac- ard et al. 202T). This demonstrates that the modifications not only improve the readability of the documentation but also enhance its semantic quality, thereby improving the different stages of the whole pipeline for tool learning.
Table 3: Comparison of tool retrieval performance between raw documentation and our method. We report NDCG@ I and NDCG@ 10.
TMDB
Spotify
Retriever
Documentation
@1
@10
@1
@ 10
Raw
24.0
35.0
43.9
53.9
BM25
DRAFT
29.0
39.4
43,9
54.2
Raw
29.0
40.4
45.6
49.6
Contriever
DRAFT
31.0
44.1
47.4
49.2
Does the tool documentation modified by DRAFT also improve human comprehension? In the previous experiments, we have demonstrated that tool documentation modified by DRAFT can help LLMs better understand and utilize external tools. Next, we aim to verify whether this improvement extends to human understanding of the tools as well. Specifically, we conduct a human evaluation, inviting three well-educated doctor students to evaluate the raw tool documentation and the documentation modified by DRAFT based on three criteria: completeness: which documentation more comprehensive. conciseness: which documentation is clearer and more concise, accuracy: which documentation reflects the tool's functionality more accurately (details are presented in Appendix A). We randomly sampled 50 cases from RestBench and ToolBench for this evaluation.
The results, as shown in Table indicate that the tool documentation modified by DRAFT demon- strates significant improvements, particularly in terms of completeness and accuracy (some cases are presented in Appendix (D). This can be attributed to the fact that DRAFT incorporates feedback from LLMs based on their tool usage experience, which provides execution results during the revision process, ensuring better alignment between the documentation and the actual tool behavior. Moreover,

---

Published as a conference paper at ICLR 2025
Table 4: Human evaluation comparing the quality of the raw documentation with ours.
Dataset
Completeness
Conciseness
Accuracy
Ours
Raw
Equal
Ours
Raw
Equal
Ours
Raw
Equal
RestBench
40%
16%
44%
36%
20%
44%
30%
0%
70%
Ours
Raw
Equal
Ours 56%
Raw
Equal 40%
Ours 56%
Raw 0%
Equal
ToolBench
68%
4%
28%
4%
44%
our design of a tool-adantive termination mechanism effectively prevents unnecessary redundancy, leading to more concise documentation. Additionally, the results suggest that ToolBench tools have significantly lower documentation quality than those in the RestBench dataset, as the readability of ToolBench tools exhibited a more substantial improvement after modification by DRAFT. This further highlights the necessity of revising the existing tool documentation.
RELATED WORK
Tool Learning. Recent studies have highlighted the potential of LLMs to utilize external tools in addressing complex problems (Qu et al. [024bb Wang et al. 2024d). With the aid of external tools,
LLMs can obtain up-to-date information Nakano al. 2021 Gou et al. 2024a b), enhance domain- specific knowledge (M. Bran et al. 2024 Zhang et al. 2024a), process mu multi-modal information (Surís) et al. 2023 Gao et 20224),. and more. Existing tool learning approaches can be categorized into two types: tuning-based and tuning- free methods (Gao et al. (024b). Tuning-based based methods enhance the tool-using capabilities of LLMs by fine-tuning them on tool-related datasets (Patil al., 2023
Hao et al., 2024 Yang et al. 2024 Qin et al., 2024 Liu et al. 2024). However, this approach is only applicable to open- open-source models and requires substantial computational resources. In contrast,
tuning-free methods provide LLMs with tool documentation and few demonstrations (Wei et al..
2022 Hsieh et al. 2023 Paranjape et al. 2023 Du et al.) 2024; Shi et al.) 2024). relying on the in- context learning ability to understand how to use tools. This approach requires no additional training and allows for the plug-and-ple
integration of external tools. However, this method necessitates high-quality tool documentation that is aligned with the comprehension of LLMs (Yuan et al. 2024] Chen et al. 2024). In this paper, we propose a method to align with L.MMs understanding and improve the quality of tool documentation enhance the tool-using capabilities of LLMs.
Learning from Feedback. Recent studies show that LLMs can improve their initial responses through self-correction, leading to improved performance (Shinn al. 2024 Madaan al. 2024 Pan et al.)
2024; Huung et al.] 2024). However, some researchers observe that relying exc scclusively on self- correction without external feedback may yield minimal improvements or worsen performance (Zhao iet al. (0224). In contrast, incorporating learning from feedback has been shown to improve various tasks (Jin al. 2023 Gao al. 2024a] Wang et al. 2024c| Pan al., 2024, Welleck al., 2022
Zhang et al.; 2024b). The forms of feedback are categorized into scalar and natural language types (Gou et al. (2024a). Scalar feedback provides goaise-grained information and typically serves
as reward signal in reinforcement learning frameworks (Ziegler et al.) (019),, while natural language feedback provides detailed insights and is used in prompts for LLMs enhance performance (Jin) et al. 2023). The sources of feedback are diverse, including humans (Ouyang al., (022), critic models (Nathani et al.). external tools WWang et al., [2024b] Qiao et al.) [0224). and even the LLM itself. To ensure that tool documentation genuinely reflects the purpose of the tool, obtain feedback by actually using the tool to get the returned results, thereby producing high quality tool documentation.
5 CONCLUSION
In this paper, we highlight that the misalignment between the existing, primarily numan-centrr tool documentation and the interpretive requirements of LLMS acts as a pivotal barrier obstructing the full potential of tool learning with LLMs. To remedy this, inspired by trial-and-error, we introduce DRAFT, dynamic and self-improving framework specifically designed to iteratively refine tool documentation based on direct interactions and feedback loops between LLMs and external tools. Through extensive experimentation, markedly enhances the alignment between tool documentation and the operational understanding of LLMss, thereby fostering more effective tool usage.
findings substantiate the assertion that our proposed DRAFT
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was funded by the National Key R&D Program of China (2033YF41008704) the National Natural Science Foundation of China (62472426), PCC@RUC, fund for building world-class uni- versities (disciplines) of Renmin University of China. Work partially done at Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education.
REFERENCES
Authony Chemero. Llms differ from human cognition because they are not embodied. Nature Human Behaviour, 1828-1889, 2023.
Guoxin Chen, Zhong Zhang, Xin Cong, Fangda Guo, Yesai Wu, Yankai Lin, Wenzheng Feng, and Yasheng Wang. Learning evolving tools for large language models. arXiv preprint arXiv:2410.06617. 2024.
Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reflective, hierarchical agents for large-scale api calls. In Proceedings ICML 2024, 2024.
Tobias Friedrich, Pietro Oliveto, Dirk Sudholt, and Carsten Witt. Analysis of diversity-preserving mechanisms for global exploration. Evolutionary Computation, 17(4):455-476, 2009.
Jinglong Gao, Xiao Ding, Yiming Cui, Jianbai Zhao, Hepeng Wang, Ting Liu, and Bing Qin. Self- evolving gpt: A lifelong autonomous experiential learner. In Proceedings of the Association for Computational Linguistics: ACL 2024, 2024a.
Shen Gao, Zhengliang Shi, Minghang Zhu, Bowen Fang. Xin Xin, Pengjie Ren, Zhumin Chen, Jun Ma, and Zhaochun Ren. Confucius: Iterative tool learning from introspection feedback by masy-to-difficull curriculum. In Proceedings of 38th Conference on Artificial Intelligence (AAAI), 2024b.
Zhi Gao, Yuntao Du Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song- Chun Zhu, and Qing Li. Clova: A closed-loop visual assistant with tool usage and update. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024c.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with iooe-interaetive critiquing, In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024a.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. In Proceedings the 12th International Conference on Learning Representations (ICLR), 2024b.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems, 36, 2024.
Zhang-Wei Hong, Tzu-Yun Shann, Shih- Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Lee. Diversity-dri -driven exploration strategy for deep reinforcement learning. Advances in neural information processing systems, 31, 2018.
Cheng-Yo Yu Hsich, Si-An Chen, Chun- Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yo Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. arXiv preprint wxxiv2338..00677 2023.
Xiang Huang, Sitao Cheng. Shanshan Huang, Jiayu Shen, Yong Xu, Chaoyun Zhang, and Yuzhong Qu. Queryagent: A reliable and efficient reasoning framework with environmental feedback based self-correction. In Proceedings of the Association for Computational Linguistics: ACI. 2024, 2024.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint 2 18, 2021.
11

---

Published as a conference paper at ICLR 2025
Di Jin, Shikib Mehri, Devamanyu Hazarika, Aishwarya Padmakumar, Sungjin Lee, Yang Liu, and Mahdi Namazifar, Data-efficient alignment of large language models with human feedback through natural language. arXiv preprint arXiv:2311.14543 2023.
Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al. Toolace: Winning the points of Ilm function calling. arXiv preprint w09v:2409.00920, 2024.
Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. Trunsactions Machine Learning Research, TMLR 2023, 2023.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju. William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint
2021
Deepak Nathani, David Wang, Liangming Pan, and William Yang Wang. Maf: Multi-aspect feedback for improving reasoning in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023.
François Osiurak and Dietmar Heinke. Looking for intoolligence: A unified framework for the cognitive study of human tool use and technology. American Psychologist, 73(2):169, 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang. and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12:484-506. 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 1031155/0733033.073333 URLattps http://aaaaaaaooloorr////2-/---0 wtt.:or///P201000000
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.0901 2023.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint a3Xiv:2305. 15334, 2023.
Shuofei Qiao, Honghao Gui, Huajun Chen, and Ningyu Zhang. Making language models better tool learners with execution feedback. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL- HITT), 2024.
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Webepm: Interactive web search for chinese long-form question answering. In Proceedings of ACL 2023. Association for Computational Linguistics, 2023a. URL WtRLtttp:://aat 06849
12

---

Published as a conference paper at ICLR 2025
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint wrX40:2304.08854, 2023b.
Yujaa Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, al. Toolllm: Facilitating large language models to master 16000+ real- world apis. In Proceedings of the 12th International Conference on Learning Representations (ICIRR), 2024.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang. Dawei Yin, Jun Xu, and Ji- Rong Wen. Towards completeness-orieneed tool retrieval for large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024a.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen. Tool learning with large language models: A survey, arXiv preprint arXiv:2405. 17935, 2024b.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 2009.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
Yongliang Shen, Kaitao Song. Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024.
Zhengliang Shi, Shen Gao, Xiuyi Chen, Lingyong Yan, Haibo Shi, Dawei Yin, Zhumin Chen, Pengjic Ren, Suzan Verberne, and Zhaochun Ren. Learning to use tools via cooperative and interactive agents. arXiv preprint TrXiv:2403.03031 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Restgpt: Connecting large language models with real-world applications via restful apis. arXiv preprint arXiv:2306.0666 06624, 2023.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11888-11895 2023.
Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. Lims in the imaginarium: Tool learning through simulated trial and error. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024a.
Xingyao Wang, Zihan Wang. Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. In Proceedings of 12th International Conference Learning Representations (ICLR)., 2024b.
Yutong Wang, Jiali Zeng, Xuebo Liu, Fandong Meng. Jie Zhou, and Min Zhang. Taste: Teaching large language models to translate through self-reflection. In Proceedings of the Association for Computational Linguistics: ACL 2024, 2024c.
Zhiruo Wang, Zhoujun Cheng, Hao Zhu, Daniel Fried, and Graham Neubig. What are tools anyway? a survey from the language model perspective. In Proceedings of the 1th COLM., 2024d.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of th-thuught prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 -44337, 2022.
13

---

Published as a conference paper at ICLR 2025
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053, 2022.
John Wieting, Taylor Berg-Kirkpatrrik, Kevin Gimpel, and Graham Neubig. Beyond bleu: training neural machine translation with semantic similarity. arXiv preprint arXiv: 06694, 2019.
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. arXiv preprint 165004 2023.
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36, 2024.
Shunyu Yao. Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models, arXiv preprint arXiv:2210.03629, 2022.
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. Easytool: Enhancing lim-based agents with concise tool instruction. arXiv preprint arXiv:2401.06200 2024.
Beichen Zhang, Kun Zhou, Xilin Wei, Xin Zhao. Jing Sha, Shijin Wang. and Ji-Rong Wen. Evaluat- ing and improving tool-augmented compuration-intensiiee math reasoning. Advances in Neural Information Processing Systems, 36, 2024a.
Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. Self-contrast: Better reflection through inconsistent solving perspectives. In Proceedings of the Association for Computational Linguistics: ACI. 2024, 2024b.
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents experiential learners. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 19632-19642, 2024.
Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. T Efficient action space navigation in large language models with a search. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint
2019.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A DETAILS OF HUMAN EVALUATION
We recruit three doctoral students familiar with the domain evaluate the raw tool documentation and its revised versions generated by DRAFT and pay accordingly. They are selected based on their familiarity with the domain and their capability to assess the quality of tool documentation accurately, They are asked to evaluate the documentation through pair-wise comparison based on the following three criteria: Completeness: Which documentation more thoroughly describes the tool's potential uses and key information, such as its usage and parameter descriptions, without missing critical details. 2. Conciseness: Which documentation is clearer and more concise, avoiding irrelevant or redundant information. 3. Accuracy: Which documentation describes the tool functionality more accurately and is free from incorrect information. The Fleiss' Kappa statistics for completeness, conciseness, and accuracy are 0.76, 0.71, and 0.72 RestBench, while 0.77, 0.69, and 0.71 on ToolBench, indicating high agreement among the three annotators.
B MORE EXPERIMENTS
We also conduct analysis experiments on three datasets using other LLMs. As illustrated in Table 5 both GPT-4o-mini and LLama-3- 3-70B display trends that are consistent with those observed in GPT-4o. The absence our proposed diversity-promotiin exploration strategy and tool-adaptive termination mechanism results in decline in performance, thereby underscoring the importance of our design innovations. An intuitive explanation posits that, in the absence of the diversity-prot proomoting exploration strategy, the examples generated during each round of exploration may exhibit significant similarity. This similarity could render multiple iterations indistinguishable from a single iteration, thereby undermining the advantages of repeated exploration. Likewise, the absence of the tool-adaptive termination mechanism may result in excessive iterations, which could produce redundant information and contribute to overfitting.
Table 5: An ablation study of the proposed model, DRAFT, was conducted across three LLMs
utilizing three distinct datasets.
RestBench-TTMN TMDD
RestBench-Spotify
ToolBench
Model
Method
CP%
Win %
CP%
Win %
CP%
Win
DRAFT (Ours)
62.00
82.00
43.85
78.94
47.00
88.00
GPT-4o-mini
w/o diversity
60.00
79.00
42.10
70.
42.00
86.00
w/o adaptive
55.00
77.00
38.59
57.89
36.00
87.00
DRAFT (Ours)
86.00
64.00
66.66
64.91
53.00
62.00
Llama-3-70B
w/o diversity
84.00
62.00
63.15
63.15
48.00
61.00
w/o adaptive
79.00
55.00
59.64
57.89
44.00
56.00
DRAFT (Ours)
88.00
71.00
70.17
84.21
51.00
78.00
GPT-4o
w/o diversity
84.00
69.00
64.91
73.68
44.00
72.00
w/o adaptive
80.00
68.00
59.64
68.42
46.00
76.00
C DETAILED PROMPTS
Table provides comprehensive overview of the prompts utilized during the three learning stages of DRAFT, which include experience gathering, learning from experience, and documentation rewriting.
D CASE STUDY
D.1 TOOLBENCH
Table 7 displays the comparison of original tool documentation and modified versions using DRAFT in cases from the ToolBench dataset. Each case highlights specific issue found in the raw
15

---

Published as a conference paper at ICLR 2025
documentation, including incompleteness, ambiguity, redundancy, and inaccuracy, all of which are effectively addressed by our proposed method.
D.2 RESTBENCH
Table displays displays the comparison of original tool documentation and modified versions using DRAFT in some cases from the RestBench dataset. Although the overall quality of the tools in RestBench is generally superior to that of ToolBench, there remain significant issues, including irrelevance, ambigu- ity, and incomplete information. This further highlights the necessity of employing our method, which iteratively refines the documentation through interactions with the tools. By systematically addressing these issues, approach guarantees that the documentation evolves in way that enhances clarity, relevance, and completeness. This, in turn, minimizes the likelihood of misunderstandings or misuse of the APIs and enhances the ability of LLMs to utilize external tools.
E REVISION TRAJECTORY
Table and Table 10 respectively present the revision trajectory of tool documentation through the first, second, and third rounds of modifications using DRAFT. An analysis of these tables reveals that the original tool documentation is characterized by the inclusion of irrelevant information and a notable absence of guidance regarding the necessity of a valid ID, as well as procedures for addressing errors associated with the provision of invalid ID. These observations underscore significant deficiencies in the original documentation that necessitate revision.
Through an analysis of the revision trajectory, becomes evident that during the first round, the experience of encountering an invalid ID is documented. The analyzer promptly identifies the issue and provides suggestions for improvement. Subsequently, the rewriter utilizes this feedback to make appropriate updates to the tool documentation, recommending that future exploration should concentrate on retrieving results associated with a valid ID. By the third iteration, the exploration successfully yields results with a valid ID), leading to further updates in the documentation. This process underscores the necessity of conducting multiple iterations. Furthermore, in the absence of diversity-promoting
diversity- promoting exploration strategy designed facilitate diverse exploration, the system may repeatedly encounter similar scenarios, thereby hindering the discovery of valid IDs and obstructing effective updates. This situation emphasizes the critical importance of implementing diversity- promoting exploration strategy.
Table II and Table 12 show the revision trajectories from the fourth and fifth rounds, respectively. Upon analysis, it is evident that as the number of iterations increases, the documentation for the tool becomes increasingly lengthy, leading the inclusion of redundant information. This redundancy may impede the effective utilization of the tool by LLMs, thereby underscoring the necessity of the tool-adaptive termination mechanism that we have developed.
16

---

Published as a conference paper at ICLR 2025
Table 6: The full prompt for three learning stages of DRAFT.
Experience Gathering
1 Task prompt #/ Your task is to answer the user's query as best you can. You have access to the following tools, which you can use via API call to help with your your response; (Tool Documentation)
Now you have the chance to explore the available APIs. You can do this by 1) synthesizing some natural user query that calling the API could help, 2) extracting the parameters needed to call these APls from the gener- ated query, and 3) Here, you can focus on queries that only require calling the API once.
Now, first input your synthesized user query. You should make the query natural- for example, try to avoid using the provided API descriptions or API names in the query, as the user does not know what APIs you ha- ve access to. However, please make sure that the user query you generate includes the parameters required to call the API, for which you need to generate random information. For required parameters like IP address, I- ocation, coordinates, etc., provide specific details. For example, instead of simply stating 'an address', provi- de the exact road and district names. Please note that if the required parameters are ID or username, which y- ou do not know what are valid, you should use the default parameters provided in the API documentation di- rectly. Also try make the query specific as possible. Next you need to extract the parameters needed to call the APIs from your generated user queries based on the provided API documentation descriptions. Here is an example: ("User Query": "Parameters"s [ }} 1 Memory Mechanisms */
Below are queries you have already explored: (Explored queries)
Based these, try explore queries that can help you understand the API further, Avoid synthesizing quer- ies that are too close to existing ones. Here are some suggestions to explore the API: (Suggestions) Now you know bit more about the API. You can synthesize another user query to explore the API a bit fur- ther and consolidate your understanding of the API, based on things that you discovered about this API. You should cherish the opportunity explore, cach time is precious. Therefore, you should generate new exp- lorations that are different from previous ones as much as possible.
Learning from Experience
Task prompt
You task is to provide suggestions for modifying the tool documentation based on the current tool document- ation, the explored query and parameters, and the results returned by the tool. You have access to the follow- ing tools: (Tool Documentation)
Please note that the existing tool documentation may be incomplete or noisy. Previously, you generated some user queries and required parameters to explore this API based on the API documentation. Now, you will provided with the output of this API under these parameters. You need to consider the following when provi- ding suggestions: For instance, consider whether the current description is consistent with the actual results returned by the tool, whether the description is comprehensive, and whether it is concise and free of irreleva information. Provide suggestions for modifications based on these aspects. Below are explored queries, the required parameters and the outputs of the tool: (Explored examples) Here is an example: ( ("Suggestions":
1 Memory Mechanisms */
The following the history of you modifying the tool description: (History) Based on the above information, provide more constructive suggestions.
III: Documentation Rewriting
Task prompt "/
Your task is to refine the description of the tool based on the queries and parameters you' ve explored, the ou- tputs provided by the tool, as well as the suggestions provided. You have access to the following tools:
(Tool Documentation)
Please note that the existing tool documentation may be incomplete or noisy. The revised description should focus solely on the functionalities of the API, omitting any irrelevant details. Below are explored queries, the required parameters and the outputs of the tool: (Explored examples]
Based on the feedback provided, here some guidelines for updating the tool description: (Suggestions) Due to the limited number of explorations you can perform, you need to value each opportunity. What aspec- is of this API would you like to explore next? Please provide some suggestions for your next query generati- on. Just give direction of the next exploration, don't give full example of the exploration directly. Here is an example: "Rewritten description": : "Suggestions for exploring":
f Memory Mechanisms */
The following the history of you modifying the tool description: (History] Based on the above information, provide more constructive suggestions.
17

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of original tool documentation and modified versions using DRAFT in some cases from the ToolBench Dataset.
Raw Tool Description (Incomplete)
("Tool Name": "GetSearchlmage" "description": """
Ours
("Tool Name": "GetScarchlmage" "description": "Retrieve an image using specified image identifier search context ID, with optional parameters for folder and storage.""
Raw Tool Description (Ambiguous) { ("Tool Name": "il", "description": "Turkish plates. 1 to 81")
Ours
("Tool Name": "il", "description" : "Provides location details for Turkish plate numbers based on the plate number parameter, including city, district, neighborhood, and postal code." )
Raw Tool Description (Redundant)
{ ("Tool Name": "QR Code Image Generator", "description": "A QR code generator API is tool that enables developers to generate QR codes within their applications. QR codes wwo-dimensional barcodes that can scanned using a smartphone camera and decoded using QR code reader software. The API allows developers to easily integrate QR code generation functionality into their existing applications, such as mobile apps, weh apps, and software. API can be used to generate QR codes for variety of purposes, such Contact information: Generate QR code containing your contact information, such your phone number email Liddress.Links: Create QR code that links website, Tube video, other online Content.Payment Create QR code making payment through mobile wallet or payment app. Events:: Generate a QR code for an event, such as a concert or conference, to provide attendees with all the necessary information. Coupons: Create a QR code for coupon or promo code to be redeemed at physical store or online. Overall, QR code generator API is versatile tool that can help businesses and individuals streamline their processes and improve the user experience for their customers.")
Ours
(Toool Name": "QR Code Image Generator". "description": QR code generator enables developers to generate QR codes for various purposes, such as contact information, links, payments, events, and coupons. Users should be prepared for potential connectivity issues. including connection timeouts, and are advised to contact the API provider for support in case of errors like 'Connection timed out'. Implementing retry mechanisms and checking network connectivity can help handle such situations effectively.""
versatile tool that
Raw Tool Description (Inaccurate)
(Toool Name": "SQUAKE". "description": "SQUAKE helps businesses to build sustainable products! With the most performant API in the market, we help travel, mobility, and logistics companies calculate carbon emissions and purchase certified climate contributions in real-time Implemented within mins!", "Return Results:" "I'm teapot.")
Ours
(Toool Name": "il", "description": "A tool that checks the health status of specific system and provides relevant information enor messages. may occasionally return unexpected or humorous responses.", "Return Results:" "I'm in a teapot.")
18

---

Published as a conference paper at ICLR 2025
Table 8:
Comparison of original tool documentation and modified versions using DRAFT in some cases from the RestBench Dataset.
Raw Tool Description (Irrelevant)
("Tool Name": "GET trending media_type _ttpttttteee_eeeeeeeeeeeeeeeeeee time_ mwnddow" "description": "Get the daily weekly trending items. The daily trending list tracks items over the period a day while items have 24 hour half life. The weekly list tracks items over a 7 day period, with a 7 day half ##f Valid Media Typesl **Media Type** ** **Description** || -I ll all Include all movies. TV shows and people in the results as global trending list. Il movie Show the trending movies in the results. tv I Show the trending TV shows in the results. person Show the trending people in the results, liiiiii Valid Time Windowsl *eme Window **Decrription** I| 1- ll 1ay View the trending list for the day. week View the trending list for the week. ["]
Ours
("Tool Name": "GET trending media type time window". " "description": "This API provides access to trending items on daily or weekly basis, including movies, TV shows, or people. The 'media type' parameter is required and specifies the type of media: 'movie" for movies, 'tv" for TV shows, 'person' for people, or 'all' to include all types. The 'time_ window' parameter also required and can be to 'day' for daily trends or 'week' for weekly trends. The API response includes list of trending items with fields such as 'id", 'name", 'original_ _ame"" "media_tyyr typee, 'popularity.. 'gender', "known_foo snoow_for__ooe for_department' 'profile_patis and 'known_for" For each 'known_for for item, details like 'title', 'original title', 'overview', proster_path', "media_ type'. "original_languagee,, 'genre_ick ids', 'popularity' 'release date'' wrot__average and "vote_count" are included. Pagination supported, allowing navigation through results using the 'page' parameter, with "total pages' and 'total results' indicating the number of pages and total results available. The API can filter content based on language preferences ('original language") ) and content restrictions (" ("adult')."
Raw Tool Description (Ambiguous)
{ ('Tool Name": "GET BOEET_peroon_peeeee peeron_id_tv_oooe lv_credits", "description": "Get the TV show cred- its for person. You query for some extra details about the credit with the [credit (aaee//HeoeooooaPaAAAAAAAACCCCC
Ours
("Tool Name": "GET person person__erroo id IV_ ceedeits". "description": "This API retrieves TV show credits for a person using their unique "person id', It returns two main categories of credits: 'cast' and 'crew'. Each credit includes details such as the TV show's 's name, first air date, overview, popularity, and the person's rolc (c.g., Director). The 'crew' category provides additional information like department, job, and episode count, valid pprroonid' required, and if an invalid non-existent 'person_id is provided, the API will return an error message. The tool does not require any additional parameters. Ensure that the 'person id' correctly formatted and exists in the database to avoid errors. Additional details about the TV show credits can be queried using the credit method.")
Raw Tool Description (Error Message) { ("Tool Name": "EET_tv_latest" "description": "Get the most newly created TVV show. This is live response and will continuously change.")
Ours
(Toool Name": "GET latest", "description": "Retrieve live, comprehensive details about the most recently created TV show, including its name, air dates, episodes, production status, genres, networks, production companics, and countrics. The tool also provides information on the latest and upcoming episodes, as well as metadata like popularity and vote statistics. Note that some fields may be empty if the information is not available at the time of the query.")
Raw Tool Description (Incomplete)
( (Toool Name": "GET genre_te tv list". "description": "Get the list of official genres for TV shows.")
Ours
(TTool Name"; "GET "GET__enre___ _t__ttss""", "description": "Retrieve the official list of TV show genres from The Movie Database (TMDb), This tool provides JSON object containing an array genre objects, each with an 'id' and a 'name". is specifically designed for TV show genres and does not require any input parameters.") ")
19

---

Published as a conference paper at ICLR 2025
Table 9: Revision trajectory of tool documentation through the first two iterations of modifications using DRAFT.
Raw Tool Documentation
("tool name"; "GET "GET_peeroon Lirioon_rsoo____ sorooondd_tt__eeee ji_____reitts" "tool description": "Get the TVV show credits for a pers- on. You can query for some extra details about the credit with the [credit Deetiod|/eedhponntssWWEEEBE NZSicLN).")
First Iteration
1 Explorer
("User Query": "Can you tell me about TV shows that actor John Doe has been a part of?", "url": 'htp::/api.thhmmovvii wwwehooied..rrg///ress.e..e
[person_iidoooo itl/dddyceddiiitttseeesessssssse _cceeittee" "Parameters": (), "AP[ Response": ("success":
false, "status_od :oaosss___eee___eeeeeeeee 6, "status, message".s "Invalid id: The pre-requisite id is invalid or not found.")
t Analyzer */
The tool description should clarify that a valid peeronn id' is required to retrieve TV show credits for a pers- on. The current description does not mention the necessity of this parameter, which is crucial for the tool to function correctly. Additionally, the description should specify that the tool will return an crror message if an invalid non-existent person_id mosson_id' is provided. This will help users understand the importance of providi- ng correct input and what to expect in case of error.
Rewriter
{ "Rewritten description": "Retrieve TV show credits for person using their unique 'person_id id'.. A valid person_id _dd is required to access this information. If an invalid or non-existen non-existent "person id' is provided, the tool will return error message indicating the issue. Additional details about the TV show credits can be queried using credit method.".
"Suggestions for exploring": "To further explore this API, it would he heneficial to understand the structure of the response when a valid 'person_it "ioonndd''ar is used. This includes the types of information returned about each TV show credit, such as the role played, the show's 's title, and the dates of involvement. Additionally, explor- ing how obtain a valid "persnn id' for given actor actress could be useful, well understanding any rate limits or restrictions on the number of queries that can be made.")
Second Iteration
Explorer
("User Query": : "What TV shows has actress Jane Smith been involved in throughout her career?", "url": hta.apiitheemmiiccorr///////
(prrson /tt///ttdd///// adl/ttyccrediss. credits". "Parameters":: " (), "API Response"; ("success";
false, :o 6, "status message": "Invalid id: The pre-requisite id is invalid not found.")
Analyzer
The tool retrieves TV show credits for a person using their unique 'person_id . It is essential to provide valid 'person_id m_ddddd''" to access this information. If an invalid or non-existent 'person_id" _id'd' is provided, the tool will return error message with a status code and status message indicating the issue. The tool does require any additional parameters. Additional details about the TV show credits can be queried using the credit method. Ensure that the pperson is correctly formatted and exists in the database to avoid errors.
Rewriter #/
{ "Rewritten description": "This API retrieves TV show credits for person using their unique "person id'. valid "person_ id' is essential to access the information. If an invalid non-existent "person, id' provided, the API will return an error message with a status code and status message indicating issue. The tool docs not require any additional parameters. Ensure that the 'person_id id" correctly formatted and exists in the database to avoid eиrors. Additional details about the TV show credits can be queried using the credit method." ", "Suggestions for exploring": : "To further explore this API, consider testing with known valid 'person id' to observe the structure and details of successful response. Additionally, explore how the APT handles edge cases, such as a person that exists hut has no TV credits. Investigate the format and type of additional details that can be retrieved using the credit method to understand the full capabilities of the API.")
20

---

Published as a conference paper at ICLR 2025
Table 10: Revision trajectory of tool documentation in the third round of modifications using DRAFT.
Third Iteration
1 Explorer */
("User Query": "Can you provide the TV show credits for the person with ID 12345?", "url": "http://api. hntemovicchbingg///rr//////////
credits", "Parameters": (). "APL Response": ("cast": 0. 2igp07772aNeeel222ooXXXXSsA
genre_ids": [10759, 18, briginna_languugee:: "en", original_nnmee "Knight "crew": H. "adult": false, "hackdrop_pathe [ "US"" }, "original_ language":
10765,801, "id"; 2384, Rider". "overview": "Michael Long, undercover police officer, is shot while investigating case and left for dead by his assailants. He rescued by Wilton Knight, a wealthy, dying millionaire and inventor who arranges life-saving surgery, including a new face and a new identity-that of Michael Knight. Michael then given a special computerized and indestructible car called the Knight Industries Two Thousand (nicknamed KITT), and a mission: apprehend criminals who are beyond the reach of the law. The series depicts Michael's exploits he and KITT battle the forees evil on behalf of the Foundation for Law and Government.", "popularity": 204.362, "poster_path"
"I57NX//Twp5mmPxiix4 Xin.TTrDiiiiB jpg", "first air date": "1982-09-26" "name": "Knight Rider", "vote_ave aate_avrreee.. 7.5, "vote coteecc_ott 1227, "credit id": "SSh-fiaec3a26844c00022ce "department": "Directing", "episode_ count": tt Analyzer */
episoddeccoutt": 1, "job": "Director")]
The API retrieves show credits for a person using their unique 'person id', returns two main categories of credits: "cast' and 'crew", with details such as the TV show's name, first air date, overview, popularity, and the person's role (c.g., Director). The 'crew' category includes additional information like department, job, and episode count. A valid 'person_id" is required, and if an invalid or non-existent 'person_ie idd is provided, the API will return an error message. The tool does require any additional parameters. Ensure that the "person id' correctly formatted and exists the database avoid errors. Additional details about the TV show credits can be queried using the credit method.
t Rewriter $/
{ ("Rewritten description": "This API retrieves TV show credits for person using their unique "person_id" id'. returns two main categories of credits: 'cast' and 'crew', Each credit includes details such as the TV show's name, first air date, overview, popularity, and the person's role (c.g.. Director). The 'crew" category provides additional information like department, job, and episode count. A valid 'person_id jnd is required, and if an invalid or non-existent 'person_is id' is provided, the API will return error message, The tool does not require any additional parameters. Ensure that the 'person_ id' is correctly formatted and exists in the datab- ase to avoid errors. Additional details about the TV show credits can be queried using the credit method.", "Suggestions for exploring": "To further explore the API, consider querying for different 'person id' values see how the API handles various cases, such as when person has only cast credits only crew credits. Additionally, explore how the API responds to edge cases, such as when person has no TV eredits all. Investigate the structure of the 'cast' category to understand the type of information provided there, as the current exploration focused on the 'crew" category. Finally, test the API's crror handling by intentionally using invalid person id' values to observe the error messages returned.")
21

---

Published as a conference paper at ICLR 2025
Table 11: Revision trajectory of tool documentation in the fourth round of modifications using DRAFT.
Fourth Iteration
Explover
("User Query": "Can you look up the TV show credits for the person with ID 99999 and let me know if they have any crew roles?", "url": "http://api.themar "PI Response": ["success": false, "status code": 34, "status message": "The resource you requested could not be [ound."1]
credits", "Parameters" ().
Analyzer #/
The API retrieves TV show credits for a person using their unique 'personidd It categorizes credits into 'cast" and 'crew', providing details such the TV show's name, first air date, overview, popularity, and the person's 's role. For crew credits, includes additional information like department, job, and episode count. valid 'person id' required to access this information. an invalid non-existent 'person id' provided, the API will return an error message with a status code and status message indicating the issue. The tool does not require any additional parameters. Ensure that the pprrsnn __ddd is correctly formatted and exists in the database. Additional details about the TV show credits can be queried using the credit method. The API response will include 'success' flag and, unsuccessful, a "status_ee code' and "status, message" indicate the problem. Rewriter
( ("Rewritten description": "This API retrieves TV show credits for person using their unique 'person_id _iooo_dd... It categorizes eredits into 'cast' and 'crew', providing details such as the TV show's name, first air date, overv- iew, popularity, and the person's 's role. For crew credits, includes additional information like department, job, episode count, A valid 'person "person id' is provided, the API will return an error message with a status code and status message indicating the issue. The tool does not require any additional parameters. Ensure that the 'person id' is correctly formatted and exists in the database. Additional details about the TV show credits can be queried using the credit method. The API response will include 'success' flag and, if unsuccessful, s. message' to indicate the problem.",
is required access information.
invalid non-existent
kaauu_codd and 'statu-
"Suggestions for exploring": : "To further explore this API, consider querying with a valid 'person_id id' to exa- mine the structure and content of a successful response. This will help understand the data fields returned for both 'cast' and 'crew' credits. Additionally, explore edge only cast credits or only crew credits see how the API handles these scenarios. Investigating for a 'person id' associated with a highly popular TV show could also provide insights into how popularity metrics are represented.")
such querying with 'person_ id' that has
response
22

---

Published as a conference paper at ICLR 2025
Table 12: Revision trajectory of tool documentation in the fifth round of modifications using DRAFT.
Fifth Iteration
1t Explorer //
I ("User Query": "Can you look up the TV show credits for the person with ID 24680 and let know if they have any credits in both cast and crew roles?". "url":
credits". "Parameters": (), "APL_R, Response", (" cast": ΠI. "crew": It "adult": false, bhackdrop_paahh "/d1 "Jd1ww4488777 dL. JLgXCKKPOe&AOO4.pppp ge": "en", priginallnameel "Jesus of Nazareth", "overview": "Dramatizes the Birth, Life, Ministry, Crucifix- ion, and Resurrection of Jesus Christ, largely according the Holy Bible's New Testament Gospels.". ", "popu- www.an/22nnRZZZzzannnnnnnnn jpg". "first air date": "1977-03-27", "n- ame". "Jesus of Nazareth", "vote average": 7.727, "vote_ ccuntt: 479, "credit id": 5591477722c333364230000 100", "depaatmment" "Sound", caasode_ccumtt" 4, "job": "Sound Recordist") )]. "id": 24680]
genre_ids", ids": [18]. "id": 19649, "origin_country". : I"IT", "GB"I,
larity": 38.929, "poster_pali path":
1e Analyzer */
The API retrieves show credits for a person using their unique "person_id" . categorizes credits "cast" and 'crew', , providing details such as the TV show's 's name, first air date, overview, popularity, and the person's 's role. For crew credits, includes additional information like department, job, and episode count. A valid 'person_ie id' is required to access this information. If an invalid or non-existent 'person_id id" is provided, the API will return error message with status code and status message indicating the issue. The toal does requi- re any additional parameters. Ensure that 'person is correctly formatted and exists in database. Note that the API response does not include a 'success' flag: instead, it provides the 'cast' and 'crew' lists directly. Additional details about the TV show credits can be queried using the credit method.
1 Rewriter */
("Rewritten deseription": "This API retrieves TV show credits for person using their unique "person_id" categorizes credits into 'cast' and 'crew', providing details such as the TV show's 's name, first air date, overvi- ew, popularity, and the person's role. For crew credits, it includes additional information like department, job, and episode count. A valid pperson_id id' is required access this information. If an invalid non-existent 'p- erson id' is provided, the API will return error message with a code and status message indicating the issue. The tool does not require any additional parameters. Ensure that the "person id' correctly format- ted and exists in the database. The API response does not include a 'success' flag; instead, it provides the 'cast' and "crew' lists directly. Additional details about the TV show credits can be queried using the credit method." ", "Suggestions for exploring": "Consider exploring how the API handles edge cases, such when person has no TV credits at all when they have credits in both cast and crew roles. Additionally, investigate if the API provides any sorting or filtering options for the credits, such as by date or popularity. Another area to explore could be the level of detail provided in the 'vverviww' field and whether it varies significantly between differ- ent TV shows.")
23


Reviews:
[{"review_id": "o0LorTHcnI", "reviewer": "Reviewer_C1G3", "summary": "This paper highlights the importance of proper tool documentation to improve tool learning. Due to incomplete documentation or consideration of human intuition, tool documentation may not include all the information to be useful for an LLM. To resolve these issues, the paper proposed DRAFT that iteratively gathers ‘experience’, learns from it, and rewrites the documentation for future usage.", "questions": "1. The paper highlights how the count of iteration impacts the performance of DRAFT mentioning - ‘a decline in performance is observed after a certain number of iterations. This decline may be attributed to the introduction of redundant information as the number of iterations increases, potentially leading to overfitting’. Is it possible to measure the count of redundant information in the system? It would be nice to see some sort of hallucination error measurement on top of DRAFT.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. DRAFT significantly outperforms across two benchmarks and outperforms three baselines.\n2. It also verifies the results through a small human study — showing its effectiveness in practical use cases.\n3. The paper provides an in-depth analysis of the impact of iteration count on the performance as well as the improved performance for retrieval in the rewritten documentation.", "weaknesses": "1. According to the first contribution highlighted in the introduction, DRAFT highlights ‘inefficiencies and inaccuracies within existing tool documentation hamper the effective utilization of tools by LLMs’. However, the paper does not show any quantitative evaluation of this claim rather than explaining the problems associated with it. Hence, I think the ‘highlight’ itself may not be a novel contribution as much as the framework.\n2. For ‘tool-adaptive termination mechanism’, is it possible to measure the impact of different threshold on the performance of DRAFT?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C7vHJ17Up8", "reviewer": "Reviewer_JQEK", "summary": "This paper addresses the tool usage challenge for LLMs. The core idea is to develop a explore-analyze-rewrite approach to refine tool documentation to improve LLMs' ability to use tools that may have lower quality documentation. To enhance tool documentation refinement, DRAFT encourages diverse queries to try out different queries to trigger different tool behaviors; these execution results are then leveraged into the analysis phase for documentation rewriting. \n\nDRAFT is evaluated on ToolBench and RestBench, results on CP% and WIN% shows that the refined document improves task completion, especially compared to closely related EasyTool. Detailed analysis of iteraction rounds, ablation results on diversity and adaptive shows significance of features in DRAFT. Also, it's interesting to learn that refined document could also potentially benefit tool retrieval and human understanding.", "questions": "Check above. In general, this paper is well written, and the experiments are quite convincing to me. Please check above for clarifications about (1) parameter space exploration, (2) analysis of example quality and (3) types of documentations DRAFT best at targeting comparing to others.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper provides a well designed approach for improving LLMs' tool using ability without additional training. The document refinement approach, while not new, is well designed considering how to leverage LLMs' self-reflection ability and trial-and-error approach to enhance document quality. This paper has the following strengths:\n\n1. Leveraging diversity of exploration and trial-and-error to compensate potential limitation of the the initial document. Where other refinement approach like EasyTool may not be able to obtain such extra information.\n2. The evaluation has fair comparison with related work to highlight performance gain. The detailed analysis show additional benefits of this approach.", "weaknesses": "This paper can be improved upon the following two areas:\n\n1. Please elaborate more in detail about Explorer design and evaluation. DRAFT relies on diverse queries/parameters to trigger different tool behaviors. It looks like the query generation depends on LLM to sample parameters automatically based on tool signature as provided by toolbench. It would be ideal to analyze how diverse in terms of parameter coverage, especially for certain types of parameter like arrays / strings. I imagine if the document is indeed incomplete as mentioned in the intro, these special cases may be less likely going to be hit during exploration. While existing benchmarks conveniently provide well defined parameter domain, it would worth the authors to include in the discussion about how to generalize this for tool documents without good parameter documentation.\n\n2. The exploration-revision loop essentially helps the system to collect more input-output examples that previously doesn't exist in the document. A qualitative analysis between examples generated by revision-only based tool like EasyTool and new examples generated by DRAFT would be helpful to understand this nuance difference.\n\n3. The intro motivates different types of documentation limiations (incomplete, redundant, inaccurate), it would be good to provide qualitative feedback about which scenarios DRAFT is best at improving, especially comparing to prior tools.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "43wMQhHkk2", "reviewer": "Reviewer_CrZf", "summary": "This work introduces a new prompting workflow for improving tool documentation iteratively called DRAFT. While prior tool documentation is typically iterated on manually and laboriously, this work focuses on automating the process of improving documentation via a three step procedure that can be repeated multiple times. The outputs of prior phases (e.g. Explorer, Analyzer) are fed into subsequent phases (Analyzer, Rewriter), where each phase produces prior experiences and filtering that is then used to refine the prompt further. To encourage diverse exploration and ensure that DRAFT terminates at some point, the authors introduce new strategies that encourage such convergence. Following the experimental setup of prior works, the authors show that DRAFT outperforms previous interactive prompting methods and run ablations demonstrating the effect of different parameters and effects of DRAFT.", "questions": "- Equation 1: What is M? Is this supposed to be A, the Explorer?\n- Line 209: How do the two components come together to make up the diversity promoting exploration strategy? Two different constraints were explained, but how do they come together to promote diversity? Is this explicitly included in Algorithm 1 somewhere? Also, is there any citation or support for why self-reflection would promote diversity? How do we know this works?\n- Equation 3: Again, what is M?\n- Line 281 - “Analogous to recipes requiring different levels of expertise, some tools may reach optimal documentation faster than others”. Any citation for this? What is “optimal” in this case? Also, what does faster mean? Is it in terms of the number of trials? Or the number of explorations needed?\n- Equation 5: How did you come up with this? Ignoring empirical performance, why does similarity score + BLEU score work for measuring the degree of change? Why are they weighted equally (e.g. why not weigh surface form similarity / BLEU higher than semantic similarity?)\n- Experimental Setup: How many rounds of DRAFT iterations were run (did DRAFT ever not terminate within a certain number of steps)? Were the initial documentations the original ones from DFSDT or ReAct?\n- Typo Line 326 - “EayTool”\n- The human study detailed in Line 473 onwards feels like it could be better explained. Are the doctor-al (Line 477) students part of the author group? What examples or guidelines did the human annotators use to judge the three criteria correctly? Did the human annotators generally agree with each other (Fleiss Kappa score)?\n- Section 2 read a bit repetitive at times - was this section written with the assistance of a language model? It may be easier to read with less redundancy. Specifically, the beginnings of each of the subsections essentially repeat what was stated in the introduction and prior subsections.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation established by the authors is sound and makes sense. In addition, the formalizations introduced in Section 2 make it easy to follow how the outputs on one prompt feed into the next. The use of an exploration phase is interesting, similar to fuzzing of a program for software engineering tests. The experiments are sound and it follows prior works. The related work covered by the authors is quite comprehensive. The ablations are complete, as the authors do a good job examining the several parameters of DRAFT.", "weaknesses": "- While they are clearly thoughtfully made, I would recommend Figures 1 and 2 be simpler to make it easier for a reader to understand what is going on. The captions would also benefit from being more specific about what is going on in the diagram. For instance, for Figure 2, I did not understand what “Trail” (I think this was supposed to be “Trial”?) refers to, is it an iteration of explorer/analyzer/rewriter?\n- The methodology section, while clear in how the symbols relate to one another, feels somewhat ungrounded in terms of why certain equations or prompt templates were used. For instance, on line 299 - “when the documentation is adequately aligned with comprehension” - what does this mean? What is adequately aligned? I think introducing more citations and explanations for why contributions such as the diversity and termination mechanisms would be needed for a more rigorous justification of why this method is interesting, and not just a prompt based workflow that happens to work.\n- DSPy (Khattab 2022) performs prompt optimization with multiple rounds of trial/error against a validation set. What makes DRAFT different from DSPy as a prompt optimization technique?\n- Although the method is proven to be effective in the experimental results, I think it would be nice to see, qualitatively, the effects of better documentation on downstream performance. While it seems that DRAFT improves the quality of the documentation, it is not obvious why better documentation improves empirical performance. For instance, it may also be possible that more “concise” documentation does not necessarily lead to improvements in downstream performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)