Learning Evolving Tools for Large Language Models
Authors: Guoxin Chen
Zhong Zhang
Xin Cong
Fangda Guo
Yesai Wu
Yankai Lin
Wenzheng Feng
Yasheng Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Tool Learning
Monte Calro Tree Search
Large Language Models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Tool learning enables large language models (LLMs) to interact with external tools and APIs, greatly expanding the application scope of LLMs. However, due to the dynamic nature of external environments, these tools and APIs may become outdated over time, preventing LLMs from correctly invoking tools. Existing research primarily focuses on static environments and overlooks this issue, limiting the adaptability of LLMs in real-world applications. In this paper, we propose ToolEVO, a novel framework designed to enhance the adaptive and reflective capabilities of LLMs against tool variability. By leveraging Monte Carlo Tree Search, ToolEVO facilitates active exploration and interaction of LLMs within dynamic environments, allowing for autonomous self-reflection and self-updating of tool usage based on environmental feedback. Additionally, we introduce ToolQA-D, a benchmark specifically designed to evaluate the impact of tool variability. Extensive experiments demonstrate the effectiveness and stability of our approach, highlighting the importance of adaptability to tool variability for effective tool learning.

Published as a conference paper at ICLR 2025
LEARNING EVOLVING TOOLS FOR LARGE LANGUAGE MODELS
Guoxin Chen', Zhong Zhang" Xin Conge* Fangda Guo', Yesai Wu?, Yankai Lin', Wenzheng Feng', Yasheng Wang 'Institute of Computing Technology, Chinese Academy of Sciences "Tsinghua University RRemmin University of China "Huawei Noah's Ark Lab chenguoxin22 ucas ac. cn { (2hongzhang, concxin ett.ennnn999essst wtw.getu..iiga........
ABSTRACT
Tool learning enables large language models (LLMs) to interact with external tools and APIs, greatly expanding the application scope of LLMs. However, due to the dynamic nature of external environments, these tools and APIs may become outdated over time, preventing LLMs from correctly invoking tools. Ex- isting research primarily focuses static environments and overlooks this issue, limiting the adaptability of LLMs in real-world applications. In this paper, we propose ToOLEVO, a novel framework designed to enhance the adaptive and re- flective capabilities of LLMs against tool variability. By leveraging Monte Carlo Tree Search, TOoLEVO facilitates active exploration and interaction of LLMs within dynamic environments, allowing for autonomous self-reflection and self- updating of tool usage based on environmental feedback. Additionally, we intro- duce ToolQA-D, a benchmark specifically designed to evaluate the impact of tool variability. Extensive experiments demonstrate the effectiveness and stability approach, highlighting the importance of adaptability to tool variability for effective tool learning
INTRODUCTION
Lotest APE Usage
MCT5
Collected API Usage
(Acossiblc In LLMG
API Monnet Finsh, API Paraararret
Besntagee
Dyrani Ensrronmnnt
AMM Name: Aestenclgeria Paramatene:
"Quar": "Aaddaa Bruakiaer Macding 2022-0116
3023111
"Lapoodd" "Acella Draal Marking
Tyensge
"Aeeha back even with crllggao Sepienter
Resporas:
ILLM Agent
Rarpenoe
Refiextion
Sala Rogl
di ca dingaan typtoaiin 17, 3022 eren
beakktet mxce up whollid
Evie wert
Tool Usoge
Figure 1: (Left) An example of inconsistent usage (name, parameters, or response formats) between the collected APIs available to LLMs and the latest APIs deployed on the server. The collected APIs may become outdated over time. (Right) An overview of our TOOLEVO. The LLM engages with the dynamic environment using MCTS for fine-tuning against tool variability, reflecting and updating tool usage based on environmental feedback. Each node in MCTS contains an API invocation.
Tool learning aims to augment large language models (LLMs) with tools and APL Schick al, 2023 Qin et al, 2023a Guo et al. 2024). This augmentation enables LLMs to interact with external tackle diverse array of complex tasks (Qiao et al.) 2024b Yang et al., [002bb] Lu et al. 2024) and having become a vital component in building LLM agents [XAgent] 20233 Chen et al.] 2024c] Qian et al. 2024).
environments and real-world applications, thereby expanding their capabilities
A critical challenge often overlooked is the inherent dynamism of external environment. In the realm of tool learning, dynamic environments are primarily manifested as tool variability, which include changes in API names, parameters, response formats. APIs are subject to continual evolution due to various factors, such as version updates, optimizations, and deprecations. This rapid and frequent
* Corresponding author.
I 'Ou code is available at httpp:://gtthui httpttthcccccc//ccen/////11 Evo
We use the tools and APIs interchangeably.

---

Published as a conference paper at ICLR 2025
change of tools is difficult to capture in timely manner. As a result, there is a discrepancy between the APIs that LLMs have learned to use and those deployed in the real-world environment, which leads to LLMs being unable to correctly invoke the tools, as illustrated in Figure I (left).
Typical tool learning approaches first fine-tune the LLMs with massive tool usage data to learn the behavior of tool invocation. Then, during the inference stage, they provide the required tools for the tasks by offering a tool manual through zero-shot prompting or demonstrating tool usage through few-shot prompting (Qin et al. 2023b Wang et al. 2024 Yang al. 2024b). However, this paradigm presents a serious risk: the specified tools in the prompt do not keep pace with changes in the external environment, the model will incorrectly invoke the outdated APIs instead of the latest APIs, leading to a collapse in performance, as shown in Figure 2 A straightforward solution to this problem collect and update the latest APIs real time, which is time-consuming and resource-intensive Therefore, in this work, we aim to improve the model's adaptability in tool learning to better handle the complexities of dynamic external environments.
Consistent APls
Chunged APGs
To address the above issues, we propose TOOLEVO, a novel framework that facilitates the adaptive and re-
flective capabilities of LLMs through active exploration
against tool variability. Inspired by mature biological
Accuracy (%)
systems such humans (Johnson-Frey 2003 Smitsman eet al. 2005), we posit that the key to adapting to tool variability lies in active interaction with the dynamic en-
22
vironment, coupled with self-reflection and self-updating of tool usage from trial and error. As shown in Figure | (right), we actively expose LLMs to dynamic environ- ments, enabling autonomous adaptation to tool variabil- ity. Specifically, TOOLEVO manages the extensive ac- tion space in dynamic environments using Monte Carlo Tree Search (MCTS), while reflecting on and updating existing tools which may be outdated- based on environment feedback. This approach allows LLMs to understand tool variability through fine-tuning on these autonomously explored trial and error, rather than merely memorizing the invocation patterns of existing tools. Fur- thermore, for research purposes, we construct new benchmark ToolQA-D based on ToolQA (Zhuang et al. variability. Extensive experiments have significantly demonstrated the effectiveness and stability of our approach in adapting to tool variability. Our contributions are summarized as follows:
Llama3722
Lbmmn-88
L.bma3-8B
L-
(Siatie-FTT) (Ouns)
Figure 2: Impact of tool variability. "Consistent APIs" refer APIs that are consistent between LLMs and servers. "Changed APIs" refer APIs accessi- ble to LLMs that are outdated over time. "Static-SFi SFF supervised fine-tuning on tool usage data that has no adaptabil- ity to tool variability. Our method suc- cessfully adapts to API changes.
2023b) to investigate the impact of tool
To our knowledge, we are the first to investigate the impact of tool variability on the performance of LLMs, which is crucial for ensuring their reliability and adaptability in real-world applications.
We propose a self-adaptive framework ToOLEVO designed specifically to address tool variabil- ity. Specifically, we actively enhance the interaction and exploration of LLMs within dynamic environments through MCTS. In this way, ToOLEVO empowers LLMs to understand tool vari- ability through trial and error, rather than merely replicating existing tool invocation patterns.
For research purposes, have constructed the first benchmark for tool variability, ToolQA-D. Based on this benchmark, we comprehensively analyze the impact of various API changes (such names, parameters, and response formats) on LLMs, thereby promoting further research.
Through extensive experiments under various tool variability settings, we demonstrate the effec- tiveness of our approach and highlight the importance of adaptability for effective tool learning.
PRELIMINARIES
2.1 TASK FORMULATION
In this paper, we consider the scenario tool variability, which is both prevalent and demanding in practical applications. The input for this task consists of the task description D and the collected
APIs P. = (P., P?..... ..... p=) However, due to tool variability, the APIs actually deployed on the server, denoted as P. [P. P2 p....., Pm) may differ from P. in terms of API names, parameters
2

---

Published as a conference paper at ICLR 2025
response formats. In practice, the environment will provide various types of feedback, defined as O, including task completion states, API responses, and API error messages. The objective this task is to successfully complete tasks under tool variability based environmental feedback.
2.2
DEFINITIONS OF KEY ELEMENTS IN MCTS
MCTS (Shapiro 2003) Browne et al. 2012) is a heuristic search algorithm designed for decision pro- cesses, particularly suited for scenarios with large action spaces and uncertain outcomes in dynamic environments. In the context of our proposed TOOLEVO, MCTS plays a pivotal role in encouraging LLMs actively engage with dynamic environments. We formulate the tool variability task within MCTS as a multi-step Markov decision process (MDP) (Puterman, 1990) in which:
State s, E S: represents the current context of the LLM's operational environment, which con- sists of the task description D, the currently available API usage Pe. all actions and its environ- ment feedback taken along the search path from the root node to the current node. The MCTS begins with initial state sp, which includes the input task D and the collected API usage P,.. Action a E A: defined as an API invocation in the REACT (Yao et al., 2023) format, which consists of thought (textual analysis), tool invocation (APIs), and observation (environmental feedback). A detailed template is listed in Appendix A.
We employ the LLM to generate
actions at each state, represented as To (aalss):) = LLM ag st), leading to a transition to a new state St+11 by concatenating 8; and ae, such that $t+1 Cat (st, dz).
Dynamic Environment: In alignment with real-worke scenarios, we define the dynamic environ- ment as everything excluding the L.LM agent. Firstly, the APIs P. deployed in the environment may differ from the APIs Pa provided to the L.M.. Secondly, the dynamic environment will provide the following information (Robbes et al. 2012 Brito et al. 2018):
Task Completion State: Upon task completion, the environment provides feedback indicating whether the task was successful (reward r 1) or failed (reward r -1). Specifically, we assess task completion state through the evaluation toolkit (Zhuang et al.] (203b)) We assign
rewards to the search path (tool trajectories) based on the task completion state.
task is successful
(1)
if task is failed
API Response: The API response refers specifically to the information returned following successful API invocation. Different APIs provide various functionalities, thereby enabling LLMs to perform complex tasks.
API Error Message: API error message can be categorized into two main types: invocation errors and deprecation errors.
Invocation Errors: These errors arise from incorrect API names or parameter settings and are not related to API deprecation. To address such errors, the model must possess self-reflective capabilities, allowing to adjust its input successfully invoke the API. Deprecation Errors: These indicate that an API has been removed in the current version, with recommendation utilize newer API instead. To address such errors, the model must process tool update capabilities based on environment feedback.
3 METHODOLOGG
3.1 OVERVIEW
As illustrated in Figure (right), we propose the TOOLEVO framework, which aims to actively immerse the LLM dynamic environments with the help of MCTS. This enables the LLM to autonomously reflect and update existing tool usage. rather than merely executing rigid tool invocations. Through actively exploring dynamic environments, the LLM accumulates trial-and- error experiences for fine-tuning, enhancing its understanding and adaptability tool variability.
3.2 INTERACTION WITH DYNAMIC ENVIRONMENTS
Considering the risk that the API usage P. provided in the prompt may become outdated over time, our TOOLEVO encourages the I.I to interact with dynamic environments through MCTS,
3

---

Published as a conference paper at ICLR 2025
thereby enhancing its adaptive and reflective capabilities regarding tool variability. Specifically, customize the four key operations of MCTS as follows:
Selection: During the selection process, traverse the tree from the root to a leaf node using the PUCT algorithm (Rosin 201T). This algorithm selects the most promising node to explore, striking balance between exploring new states and exploiting known valuable states, which can be represented as:
N(s)
PUCT(,, a) = Qs, a) + Cpect P(s, 1+ N(s
where Gpuct is a constant that controls the balance between exploration and exploitation; Q(s, a) and P(s, a) are the Q-value and prior probability of taking action a in state s, respectively. Additionally, (s) and N (s, a) denote the visit counts for state and for taking action from state respectively.
Expansion: Once an expandable leaf node is selected, the current state 8t is used as input to further expand the tree. Candidate actions sampled using the policy To (i.e., the LLM), follows:
a, ......... af = (T(a))));
where k is the number of expansion nodes. Each action of represents either an API invocation or an update in API usage after self-reflection, as detailed in Section 3.3 Upon executing the correspond-
ing API, the expanded new state s/+1 = Cat($t, at) is derived from the current state 8t.
Simulation (Cached Rollout): In the simulation step, a rollout strategy is employed to simulate a complete episode from one of the newly added nodes 1/: to obtain an accurate reward (Silver ct al. 2016). Inspired by Bellman et al. (2015) and He et al. (2023), we propose a cached rollout strategy enhance efficiency, as detailed in Appendix A.33 In cached rollout strategy, each node (including state, action, reward, and so on) in the simulated episode is cached in the tree. However, these cached nodes remain invisible to the tree during the selection phases. Before each expansion or simulation step, the cache is checked for the current state to either reuse stored results if available or perform a new expansion or simulation as needed. This approach reduces redundant calculations and significantly improves efficiency by avoiding the need to rollout from scratch in every iteration.
Backpropagation: After the simulation, the reward is propagated back along the path from the selected leaf node to the root, updating the visit counts N and the Q-values of these nodes as follows:
(4) By applying these updates, MCTS incrementally refines its search strategy based on feedback from the dynamic environment, leading to more accurate and informed decisions in subsequent selections.
(s. + Q(s a) + ass +1 (r a(s, ((s a) + N(s, a) +
3.3 SELF-REFLECTION AND TOOL-UPDATE
With the customized MCTS described above, our TOOLEVO actively engages LLMs in both explo- ration and exploitation within dynamic environments. To address tool variability, we incorporate the self-reflection and tool-update module into MCTS based on environmental feedback. This allows the LLM not only to deal with invocation errors through self-reflection but also to autonomously summarize new tool usage through the tool-update module to update API usage Pa in the prompt, ultimately fostering a deeper understanding and robustness in the face of tool variability.
Self- RefleRefectin In our ToOLEVO, the self-reflection module generates verbal reflections based on external environmental feedback, providing valuable insights for future corrections. It is important to note that, as shown in Figure [ the self-reflection module and the LLM agent are the same LLM and we do not rely on a more powerful model, such as GPT-4. The external environment typically provides various API error messages, including invocation errors and deprecation errors (Robbes) et al., 2012; Brito et al. 2018). The purpose of encouraging L.M to interact with the external environment through MCTS enable the model to effectively utilize the environmental feedback. rather than backtracking or stopping when encountering errors as in previous work (Qin et al. 2023bb Guo et al. 2024). Specifically, when an error message is encountered, we use this error state s, as input, allowing the model to reflect on the cause of the error and try to solve it.
ayef, ofef Lee <a(a)s)); 1/41 Cat (ee, where ayet, is the reflective action based on the enror state 8t. We will append these new reflective states s141 after 8t instead of interrupting further exploration of this error state.
(5)
4

---

Published as a conference paper at ICLR 2025
invoke tool
modify imput porameters
invoke tool
try tool
summonize new tool usoge.
update Pa the prompt
Tasa Reflexion
TPR
LLM Agent
LLM Agent
Refiexion
Reflexion
S142
0+++1
Tool Usage
invocation errors
Return Information
Toal Usage
deprecation
Return Information
New Tool Usoge
Invocation Errors
Deprecation Errors
Figure 3: Examples of self-reflection and tool update. Invocation errors indicate that the input parameters of the API need to be corrected. In contrast, deprecation errors suggest that the input parameters are correct, but the API has been deprecated, necessitating an update in API usage.
The self-reflection module analyzes the causes of error states based on various envi- ronmental feedback, such API invocation errors. In the context of tool variability,, we specifically focus on errors arising from API changes, referred to as deprecation errors (Zhou & Walker 2016). These errors highlight inconsistencies between the API usage referenced in the prompt and the API usage deployed on the server. Unlike invocation errors, deprecation errors indicate that the invoked API outdated and require updating the API usage (Sawant et al. 2018).
Tool-Update
Therefore, we propose tool update module based on self-reflection, as illustrated in Figure Our tool update module requires the LLM to summarize the usage of new tools and incorporate them into Pa after successfully invoking the new tools based environmental feedback. This process allows the LLM to gradually adapt to the dynamic tool environment (Pe + (Pe + P.)) during exploration. Specifically, we implement the tool update module system tool, referred to as UpdateTool (Appendix A.5). In this context, the tool update module acts like memory mechanism (Zhang) et al. 2024), enabling the model to summarize and modify the API usage in the prompt based on newly acquired knowledge. This enhancement not only helps improve the accuracy of subsequent tool invocations but also fosters more seamless integration of new tools into the LLM's workflow.
www.desoll
which serves to update the descriptions of existing tools
3.4 SELF-IMPROVEMEN FROM TRIAL AND ERROR
In our TOOLEVO, we employ MCTS to encourage the LLM to interact with dynamic environments, thereby collecting trial-and-error experiences (trajectories from the root node to the terminal node in the tree) involving self-reflection and tool updating. Through these experiences, we aim for the model master the behavior of self-reflection and self-updating of tool usage in dynamic environ- ment, thereby enhancing adaptability to tool variability rather than merely learning the specific tool invocation. We evaluate the quality of these experiences based whether the task is successfully completed, allowing the LLM Tg to improve itself through the successful experiences y*:
= arg min - Tog ID, ID,P.). P.).
(6)
Notably, failed experiences be further leveraged through preference learning (Ratailov et al., 2023). However, our work emphasizes the importance of interaction with dynamic environments. We focus solely SFT for fair comparison following the prior research (Zhuang al. 2023b).
4 TOOLQA-D
To our knowledge, we are the first to investigate the impact of tool variability on tool learning. leading to the absence of benchmarks in this domain. For research purposes, we have constructed the first benchmark for tool variability, termed TooIQA-D based on ToolQA (Zhuang et al. 2023b). We define the original APIs of ToolQA (Zhuang et al.) (202bb) as Pe (collected APIs), which may be outdated. For simulating tool variability, employ GPT-4 to randomly modify the collected API usage, including names, parameters, and response formats. This results in two new sets of API usage: Pa,, and P Pooon The purposes of different sets of API usage are outlined follows:
'There are several compelling reasons Appendix for developing this dataset based ToolQA
5

---

Published as a conference paper at ICLR 2025
On the prompt side:
For all methods, Pa used to demonstrate tool usage.
On the server side:
Deploying Pe represents the most common setup in existing studies. This setup maintains the static environment where prompt-provided APIs are consistent with server-deploeed APIs. Deployying P... introduces variability in the tools. This setup allows us to collect trial-and-error experiences in dynamic environment.
Deployying Pevon also introduces tool variability but in an out-of-distribution context. The training data and prompts for all methods do not contain any API information from Pr00o
-
Ultimately, our TooIQA-D comprises 7 datasets and 3 sets of API usage (Pe. Pen. and Paooo), ac- companied by a total of 6,234 and 5,884 training samples, 700 and 700 development samples, and 700 and 730 test samples for Easy and Hard difficulty respectively.
5
EXPERIMENTS
Table 1: Experimental setup regarding
the APIs accessible the LLM during the training and inference stages.
5.1 EXPERINMENTAL SETUP
APIs on servers
Static-SFT
Ours
Dataset Setup We conduct extensive experiments on ToolQA-D to investigate tool variability. In this work, we aim to demonstrate the importance enabling LLMs interact with dynamic environments during the train- ing phase, rather than merely memorizing how to use the tools in static environment. As shown in Table [ we utilize Ps.. as our training environment while evaluating performance in three different environments of (1) Pe to demonstrate that self-improve in tool variability can still adapt to static environments effectively, even without specifically training on the provided tools. (2) P.... demonstrate that, with the help of our ToOLEVO, LLM can reflect on and autonomously master the new tool usage through interactions with dynamic environments. (3) adaptability to tool variability, which is completely dif-
Train Env.
Test
Train Env.
Test
Env.
Env.
P
x
v
P
x
/
/
/
P PSOOp
X
X
/
P Proon to demonstrate the generalizability ferent from P.... Notably, in all experiments, the LLM can only access Pe in the prompt. Both Ps, and Pa 9000 are agnostic to the LLM and can only learned from environmental feedback.
Baselines We compare our approach with typical tool-learning methods that only consider static environments to highlight the importance of accounting for tool variability. We compare our TOOLEVO with: (1) Proprietary models: models: ChatGPT, GPT-4 (Achiam al. 2023), GPT-4o, GPT-4o- mini (OpenAl (024), and Claude-3.5-Somnee (Anthropic, 2024); (2) Open- Open-source source models: Llama3- series (Dubey et al. (224) and Qwen2-series (Yang al. (2024)): :): (3) Static supervised fine-tuming (Static-FFT) method where the supervised data focuses exclusively on tool usage in static environ-
ments, as previous studies (Zhuang et al., 2023b Qin et al., 2023b Wang et al. 2024).
Implementation Details We briefly summarize the implementation details in this section, with further claboration available in Appendix A.6 Through our TOOLEVO, we encourage the LLMs to interact with the dynamic environment of P... and accumulate tria--and-error experiences, resulting in approximately 30k tool trajectories. Subsequently, we fine-tune these models on the collected experiences to enhance their adaptability within dynamic environments. For our experiments, utilize Llama3-8B (Dubey et al. (024) and Qwen2-7B (Yang et al. (002a)) as the base models. Note that the collection of tool trajectories and the training process are conducted separately for the dif- ferent base models. For Static- SFT (Zhuang et al.) 2023b), we collect the tool trajectories in a static environment (P.) through our TOOLEVO and train it as our strong baseline. All methods perform greedy decoding and utilize 3-shot learning based on the REACT format (Yao et al.) (023) (see Ap- pendix DD.. all experiments, we only provide the API usage of Pa along with its demonstrations in the prompt, but the API usage deployed on the server may vary in different setups.
5.2 MAIN RESULTS
Performance Static Environment (P, in Prompt and P. on Server) We first compare the performance of method in the static environment Pe, as shown in Table 2 It is noteworthy that

---

Published as a conference paper at ICLR 2025
Table 2: Main results on the static environment (P. in Prompt and P. on Server). Bold indicates
best performance and underline indicates second-best performance among open-source models.
Agenda
Airbub
Coffee
Dblp
Flights
Seirex
Yelp
Average
Models
Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard
Fasy Hard
Proprictary models
GPT-3.5
40.0
52.0
60.0
36,0
0.8
12.0
17.0
72.0
44,0
2.0
0.0
72.0
52.0
46.0
28.8
GPT-4
48.0
48.0
80.0
36.0
96.0
9.2
48.0
24.0
60.0
32.0
4.0
8.0
68.0
60.0
57.7
31.1
GPT-4o
49.0
68.0
88.0
36.0
91.0
1.5
23.0
28.0
76.0
48.0
3.0
8.0
83.0
64.0
59.0
36.2
GPT-4o-mini
50.0
68.0
76.0
44.0
78.0
1.5
20.0
28.0
52.0
40.0
4.0
0.0
68.0
72.0
49.7
36.2
Claude
65.0
80.0
81.0
48.0
84.0
9.2
59.0
36.0
79.0
52.0
3.0
4.0
86.0
88.0
65.2
45.3
Open-source models
Llama3-70B 703--77t-e-ntt
55.0
55.0
76.0
27.0
94.0
3.8
32.0
32.0
59.0
26.0
0.0
2.0
86.0
44,0
57.4
27.1
I.ama3 88-BBBBnne Instruct
25.0
21.0
68.0
20.0
59.0
0.8
19.0
20.0
24.0
17.0
0.0
1.0
54.0
25.0
35.5
14.9
Static-FFT (Llama3-8B)
68.0
65.0
95.0
33.0
100.0
0.8
55.0
32.0
86.0
24.0
0.0
88.0
50.0
70.2
29.5
TOOLEVO (Llama3-8B)
70.0
57.0
96.0
30.0
88.0
1.5
49.0
34,0
69.0
36.0
1.0
3.0
95.0
$1.0
66.7
30.3
Swen2-72H-Instreen
55.0
46.0
79.0
32.0
95.0
1.5
42.0
39.0
70.0
26.0
3.0
1.0
81.0
45.0
60.7
27.2
Qwen2-7BB -n-sttee Instruct
40.0
35.0
59.0
12.0
94.0
3.1
31.0
25.0
46.0
13.0
0.0
1.0
68.0
15.0
48.2
14.8
Static-SFT (Qwen2-7B)
68.0
55.0
97.0
34,0
98.0
4.6
50.0
37.0
75.0
29.0
1.0
92.0
53.0
68.8
30.8
TOoLEVO (Qwen2-7B)
76.0
50.0
94.0
33.0
95.0
6.1
51.0
38.0
84,0
45,0
2.0
8.0
93.0
41.0
70.7
31.5
Table 3: Main results on the dynamic environment (Pe in Prompt and Pa,, on Server).
Average Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard
Agenda
Airbmb
Coffee
Dblp
Flights
Scirex
Yelp
Models
Proprictary models
GPT-3.5
32.0
52.0
60.0
16.0
72.0
0.0
12.0
7.0
40.0
13.0
0.0
0.0
56.0
16.0
38.8
14.9
GPT4
56.0
32.0
88.0
20.0
64.0
0.0
44.0
10.0
28.0
20.0
0.0
0.0
69.0
67.0
49.8
21.3
GIT-4o
40.0
52.0
84.0
28.0
76.0
0.0
40.0
16.0
68.0
40.0
0.0
4.0
76.0
48.0
54.8
26.8
GPT-4o-mini
44.0
40.0
88.0
16.0
76.0
0.0
24.0
28.0
60.0
28.0
0.0
4.0
48.0
28.0
48.5
20.5
Claude-3.5-
60.0
48.0
92.0
17.0
80.0
8.0
48.0
36.0
80.0
56.0
4.0
4.0
80.0
76.0
63.5
35.0
Open-source madels
Llama3-70B-Inste Instruct
55.0
40.0
78.0
12.0
70.0
2.3
31.0
22.0
42.0
18.0
4.0
3.0
87.0
35.0
52.4
18.9
-amma--88-Inshmee
23.0
21.0
63.0
10.0
44.0
0.0
21.0
13.0
16.0
10.0
1.0
2.0
53.0
13.0
31.5
9.8
Static-FFT (Llama3-8B)
53.0
10.0
49.0
6.0
14.0
44.0
11.0
15.0
29.0
0.0
86.0
34,0
37.2
13.0
TOoLEVO (Llama3-8B)
61.0
53.0
95.0
26.0
88.0
4.6
50,0
32.0
74.0
34,0
2.0
5.0
93.0
48.0
66.2
28.9
Qwen2-72B Deent-eet-nsttruet
56.0
38.0
73.0
11.0
78.0
0.0
42.0
28.0
54.0
12.0
1.0
2.0
75.0
34,0
54.1
18.4
Qwen2-7B- Instruct
32.0
30.0
60.0
8.0
68.0
0.8
32.0
16.0
32.0
12.0
1.0
0.0
66.0
28.0
41.5
13.5
Static-FFT (Qwen2-7B)
49.0
27.0
52.0
21.0
35.0
41.0
18.0
25.0
0.0
29.0
40.8
17.3
TooLEVO (Qwen2-7B)
66.0
41.0
94,0
36.0
97.0
5,4
46.0
39.0
85.0
41.0
1.0
8.0
92.0
54.0
68.7
32.1
Table 4: Main results on the OOD dynamic environment (Po in Prompt and Paop on Server).
Agenda
Airbnb
Coffee
Dblp
Flights
Scirex
Yelp
Average
Models
Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard
Proprictary models
GPT-3.5
40.0
50.0
56.0
20.0
52.0
0.0
4.0
13.0
24.0
19.0
0.0
1.0
44.0
32.0
31.4
19.2
GPT-4
60.0
40.0
52.0
23.0
42.0
0.0
44.0
30.0
20.0
30.0
0.0
64.0
40.0
40.3
23.2
GPT-44
36.0
44.0
92.0
21.0
80.0
0.0
40.0
28.0
64.0
36.0
0.0
0.0
80.0
42.0
56.0
24.4
GPT-4o-mini
40.0
39.0
84.0
14.0
76.0
0.0
32.0
28.0
64.0
30.0
1.0
36.0
40.0
47.4
21.7
Claude-3.5-Sonee
64.0
47.0
84.0
26.0
83.0
4.0
41.0
36.0
72.0
47.0
2.0
4.0
76.0
61.0
60.2
32.1
Open-source models
1.ama3-708- 10n--777------tuct
56.0
28.0
61.0
8.0
65.0
0.0
38.0
23,0
26.0
18.0
3.0
8.0
53.0
29.0
43.1
16.3
Llama3-8B -B-sstuu
27.0
7.0
7.0
9.0
1.0
0.0
28.0
13.0
2.0
9.0
1.0
4.0
9.0
14.0
10.7
8.0
Static-SFT (Llama3-8B)
58.0
9.0
23.0
8.0
11.0
0.0
24.0
6.0
8.0
11.0
3.0
19.0
26.0
20.4
9.0
TOOLEVV (11m
71.0
49.0
65.0
29.0
88.0
2.3
51.0
34.0
63.0
33.0
2.0
4.0
91.0
47.0
61.6
28.3
Qwen2-72B-Instruce
52.0
33.0
39.0
11.0
71.0
0.0
42.0
29.0
22.0
11.0
2.0
4.0
60.0
34.0
41.1
17.8
Qwen2-7B -IIIttuuct
37.0
22.0
55.0
7.0
74.0
1.5
26.0
14.0
37.0
3.0
1.0
3.0
67.0
27.0
42.4
11.1
Static-FFT (Qwen2-7B)
37.0
31.0
63.0
8.0
68.0
0.7
35.0
17.0
58.0
22.0
1.0
73.0
39.0
47.8
16.9
Toor TOOLEVO EVO (Qwen2-71))
68.0
48.0
89.0
14.0
81.0
6.9
48.0
34.0
85.0
33.0
3.0
6.0
83.0
52.0
65.3
27.7
our method does not undergo fine-tuning on the tool trajectories regarding Pe. while other baselines, including in-context learning and Static-SFT methods, benefit from corresponding demonstrations of Pa.. Nevertheless, our method substantially outperforms these baselines and achieves comparable even better performance than Static-SFT. which has been directly fine-tuned on Pa. This finding highlights that, even in the absence of fine-tuning with tool trajectories of Pe. trial-and-error expe- riences focus on tool variability can still enhance the tool-using capabilities in static environments.
7

---

Published as a conference paper at ICLR 2025
Performance in Dynamic Environment (Pe in Prompt and Pa.. on Server) Secondly, we com- pare the performance in the dynamic environment Po.. as shown in Table and reach the following conclusions: (1) The interference caused by outdated API usage from Pe in the prompt significantly impacts performance, especially for the Static-SFT method. Compared to the consistent API usage between prompt and server, most methods exhibit a significant performance degradation, especially Static-SFT. However, our method using the 7B model outperforms both the Static-SFT and 72B models, which primarily focus on static tool usage, (2) Leveraging the environmental feedback, our TOOLEVO can explore the usage of new tools Ps.. by interacting with the dynamic environment, even when the prompt only contains the outdated API usage Pe. Notably, our TOOLEVO does not rely on more powerful models, such as GPT-4, to recognize tool variability. Instead, it continuously interacts with the environment to collect trial-and-error experiences, demonstrating the feasibility of autonomous exploration in addressing tool variability and inspiring future research.
Performance in OOD Dynamic Environment (Pe in Prompt and Peoon on Server) As shown in Table 4 we evaluate the performance in the out-of-domain (OOD) dynamic environment P PoooD which is the most important setting to demonstrate the effectiveness of our method. We derive the following conclusions:
(1) In this setup, our method achieves substantial improvements over other baselines by a significant margin. This finding indicates that our TOOLEVO empowers the model with the ability to self-reflect and self-update its existing tool usage based on environmental feedback, rather than merely memorizing existing invocation patterns. (2) The Static-SFT model, trained exclusively on tool trajectories of static API usage, is significantly affected by tool variability. The stereotypes induced by Static-SFT lead to extreme confidence in the tool usage provided in the prompt, thereby rendering it ineffective in handling tool variability, as shown in error analysis C.3
The role of tool variability in the training phase. When we disregard tool variability in the training phase and focus solely on the API usage provided in the prompt (as observed with Static- SFT in Table 3 and the model's performance is significantly compromised by tool variability. Dynamic environments enable models to better adapt to tool variability and master how to reflect on errors and update existing tool usage in more challenging environments, which often overlooked by previous work (Wang et al. 2024 Guo et al., 2024). In the absence of a dynamic environment, the model tends to lazily focus on how to use APIs provided in the prompt. This limitation hinders the model from reaching its full potential, resulting in a significant decline in performance.
In summary, the observations derived from the three different settings (Table 2 3 and 4 indicate that: (1) tool variability have severely negative impact on LLMs, which needs to be taken seriously in future work. While the Static-SIT method has excellent performance in the static environment, suffers significant performance degradation in the dynamic environments (Pa,, e.. and P Poom)) Both pro- prietary and open- open-source source models exhibit similar phenomena, experiencing substantial fluctuations in performance. (2) Learning how to use a tool in dynamic environment will yield more benefits than simply imitating tool usage in a static environment. Focusing exclusively on tool usage in static en- vironments will cause the model to develop stereotypes, causing an excessive trust in the tool usage provided in the prompt. In contrast, our method significantly enhances performance stability in the face of tool variability, while still exhibiting excellent performance in static environments (Table 2. even without specific training for those scenarios. We conduct comprehensive case studies and error analysis to facilitate future research, as detailed in Appendix C
5.3 ANALYSIS ON TOOL VARIABILITY
In this section, we utilize the Llama3 series (Dubey al.) 2024 base models to further investigate the detailed impacts of tool variability on performance, including changes in API names, parameters, and response formats. All changes are re-randomized by GPT-4 in this section (Appendix B.2
API Name As illustrated in Figure 4 we examine two key aspects of API name: textual variations and random insertion of special characters. Our findings are summarized as follows: (1) Compared with other models, Static-SFT model exhibits more fluciuations in the performance regarding tool variability. This instability can be attributed to the stereotype of API usage presented in the prompt being always correct, which arises from its focus on tool usage in static environments (Zhuang et al., 20236) Guo et al.] 2024). (2) Without any modification to the API names, the insertion of special characters poses significant challenges for tool-using abilities of L.Mss. It is noteworthy that
8

---

Published as a conference paper at ICLR 2025
TrelQA
D-Hard
TrelQ&--EBy
Tealo---BBard
(8B)
(708)
Figure 4: Analysis on API name.
Figure 5: Analysis on API parameters.
only insert special characters between words, ensuring that the tokenizer's ability to encode the text remains intact (e.g.. "Fetch.Agende Data rather than "Fe.tchAgen.dalt aatattadd
API Parameters As illustrated in Figure 5 we assess the effects of textual variations and the random insertion of special characters API parameters. Our observations are as follows: (1) Compared with API name, changes in API parameters have more subsiantial influence perfor- mance. An intuitive explanation for this finding is that the LLM may struggle to recognize subtle variations in API parameters when the API name remains constant. (2) An intriguing phenomenon that, even under the same changes, the model's performance tends to decline more significantly challenging tasks (ToolQA-D-Hard) relative to simpler tributed to the increased cognitive load required for difficult tasks, which leads the model to allocate more attention to task execution, thereby making it more susceptible overlooking tool variability.
(TooIQA-D-Faas
This can be at-
API Response Formats Furthermore, we in- vestigate the impact of changes in the API response formats on tool-using capabilities of
Changed TaalQa D-Bladd
TrelQA-D D-EEy
Acuree (5)
LLMs, as shown in Figure Our findings indi- cate that the API response format exerts min- imal impact on the performance, primarily be- cause the challenges associated with tool learn- ing are more related to invoking the tools rather than processing the returned information. In ToolQA-D-Hard, changes in response formats lead to a decline performance due to the inher- ent challenges of the tasks. However, this decline noticeably smaller compared to the impact induced by changes in API names and parameters.
Llana3 (108))
Likaaa
Statie SFT (M15)
Ours
Llama3
Elama3 Stade- SET (H15)
Oue
(NB)
(HB)
(2019)
(R3)
(MIS)
Figure 6: Analysis on API response formats.
5.4 ABLATION STUDY
In this section, we conduct a comprehensive ablation study to investigate the role cach component in our TooLEVO. Since the tool update module is based the self-reflection module, conduct ablation experiments with the following setting: (1) "w/o tool update": During the interaction with the dynamic environment,
remove the tool update module. This means that after successfully invoking the new tools, the LLM does not summarize the usage of the new tools nor update Pa in the prompt. (2) "w/o self-reflection": In this scenario, we remove self-reflection regarding invocation errors. However, reflection on deprecation errors is retained; otherwise, the model would struggle handle tool variability effectively. All above ablation experiments are conducted in the OOD dynamic environment (P sooo)) as detailed in T
Table 5: Ablation Study
Agenda
Airbub
Coffee
Dblp
Flights
Scirex
Yelp
Average
Models
Easy
Hard Easy
Hard Easy
Hard
Easy
Hard Easy Hard Easy
Hard
Easy
Hard Easy
Hard
TOoLEVO
71.0 49,0
65.0
29,0
88.0
2.3
51.0
34,0
63.0
33.0
2.0
4.0
91.0
47.0
61.6
28.3
w/o tool update
66.0
41.0
71.0
12.0
75.0
0.0
47.0
29.0
46.0
29.0
1.0
3.0
82.0
39.0
55,4
21.9
w/o self-reflection
24.0
47.0
19.0
51.0
0.0
46.0
20.0
28.0
18.0
2.0
52.0
26.0
41.7
15.5
Ablation on Tool-Update serve decline in performance across various datasets. These results suggest that, without the tool
When the tool update module is removed ("w/o tool update"), we ob-

---

Published as a conference paper at ICLR 2025
update module, the model struggles to adapt to new tools in dynamic environment. The tool update module can be conceptualized as an experience summary derived from the environment feedback. Updating tool usage P. in the prompt with the new tool usage summarized by the model significantly facilitates the subsequent invocation, thereby gradually adapting to the new environment.
The removal of the self-reflectio module ("w/o self-reffection") re- sults in more pronounced declines in performance. The results indicate that the self-reflection mod- ule plays a crucial role in enhancing the LLM's understanding and corrective abilities during the interaction process. Although reflection on deprecation errors is preserved, the lack of reflection on invocation errors still leads to a sigmificant drop in performance. The decline in the model's re- flective capabiliitiss severely affects its decision-makinn process, thereby underscoring the critical importance of reflective learning dynamic environments.
Ablation on Self- RefleRefletton
These ablation experiments collectively demonstrate that each component of TOOLEVO is crucial to its overall effectiveness. The interplay between tool updates, self-reflection, and dynamic en- vironments significantly enhances the model's performance across various datasets. Notably, the ability to learn from environmental feedback is essential for effectively managing tool variability.
RELATED WORK
Tool Learning Recently, tool learning has demonstrated impressive potential for extending the
capabilities of LLMs through various APIs (Schick et al. 2023 Qin et al. 2023b Tang et al. 2023
Yang et al. [2023 Zhuang et al.
[223a] Huang et al. [0023] Shen al. [024] Yang et al. 2024b; Qiao
ct al. 2022bb Song et al. 2024: Qiao et 2024a Qu et al. 2024 Sun et al. [224;) Yu et al. 2025
Chen al.) 2025 Qu et al. (025). Current approaches can be divided into two categories: fine-
tuning and in-context learning (ICL). Fine-tuning- based approaches (Parisi et al.] |022| et al.)
2023 Yang et al., 2023) construct high-quality tool chains for training LLM to use a specific set of
tools, while ICL-based approaches (Zhuang et al. 2023a Huang et al. 2023 Liang et al. 2024) directly incorporate sophisticated tool descriptions and usage demonstrations into the input context. However, existing work predominantly focuses enabling LLMs to proficiently master API usage in static environments, often neglecting the inherent dynamic nature of the API ecosystem. Our experiments reveal that both outdated tool descriptions and demonstrations in the context, well as stereotypes formed through fine-tuning, will lead to performance degradation in the face of tool variability. In this paper, we aim bridge this gap and explore the impact of various API changes.
Monte Carlo Tree Search MCTS, which integrates the principles of Monte Carlo sam- pling (Shapiro 2003) with tree search, has emerged as seminal search algorithm for decision- making processes. Its ability to effectively balance exploration and exploitation in complex envi- ronments particularly noteworthy (Browne et al. 2012). The AlphaGo series (Silver et al. 2016] 2017 SSchiitwieser et al., 2020) have demonstrated the efficacy of MCTS in the context of game- playing environments.
the realm of large language models, MCTS plays a critical role in various tasks, such as text generation (Zhang et al. 2023 Liu et al. 2023). mathematical reasoning (Zhul ct al., 2023] Trinh et al. 2024 Chen et al.] 2024a b) and so on. this paper, we explore dynamic environments and leverage MCTS to enhance the interaction between LLMs and the environment, thereby accumulating rial-and-erroo experiences for tool variability.
7 CONCLUSION
In this paper. we have investigated the impact of tool variability on the tool-using capabilities of LLMs. which is often overlooked in existing studies. We find that existing methods that focus ex- clusively on static environments tend to reinforce stereotypes and are more susceptible to tool vari- ability. To address these issues, we propose TOOLEVO, a MCTS- -based framework that encourages LLMs to interact with dynamic environments and actively reflect on and update the usage of existing tools based on environmental feedback. This approach enables LLMs to understand tool variability through trial-and-error experiences, rather than merely memorizing invocation patterns of existing tools. Additionally, we have constructed ToolQA-D, the first benchmark specifically designed for evaluating the impact of tool variability. Through extensive experiments, we have demonstrated that TooLEV EVO effectively handles tool variability, both in in-domain and -ut-oi-domain settings.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the Postdoctoral Fellowship Program of CPSF (Grant No. GZB20230343 and Grant No. GZC20240831) and China Postdoctoral Science Foundation (Grant No.
2023M741945).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.055 03777744 2023.
Anthropic. Claude 3.5 Sonnet, 2024.
URL hette::/ww..aaaarrpii...oom
Richard E Bellman, Stuart E Dreyfus, and et. al. Applied dynamic programming, volume 2050.
Princeton university press, 2015.
Gleison Brito, Andre Hora, Mareo Tulio Valente, and Romain Robbes. On the use replacement messages in api deprecation: An empirical study. Journal of Systems and Software, 137:306-321,
2018.
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 4(1):1-43,20) 2012.
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: Process supervision without process. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural In- formation Processing Systems. volume 37, pp. 27689-27724. Curran Associates. Inc., 2024a. URL
file/ 1ttle//oaaeeaaaccceeeeeo00011111 hacpr//a00001eecbbbccc------eeee
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimiza- tion for mathematical reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7889- 7903, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. org.://00oorr//222/////
doi: 10. 18653/v1/2024, 200555/w/222244mddnneeem-----1 findings-emnlp.463. URL httph:ttps//.
findings-ent emannng--mnlpp emmlp 463/
Guoxin Chen, Minpeng Liao, Peiying Yu, Dingmin Wang, Zile Qiao, Chao Yang, Xin Zhao, and Kai Fan. C-3po: Compact glug-and-play proxy optimization achieve human- retrieval- augmented generation. arXiv preprint urXiv:2502.0660
2025.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024c. URL htttps://operrevit net/forum wtt::/oommmmmaagogooaa
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869/ 2023.
Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reflective, hierarchical agents for large- scale api calls. arXiv preprint arXiv:2402.04225. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 207. 2024.
11

---

Published as a conference paper at ICLR 2025
Zhicheng Guo, Sijie Cheng, Hao Wang. Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models. arXiv preprint arX1v:2403.07714, 2024.
Guoliang He, Zak Singh, and Eiko Yoncki. Mets-geb: Monte carlo tree search good e-graph builder. In Proceedings of the 3rd Workshop on Machine Learning and Systems, pp. 26-33, 2023.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhemqiang Gong, et al. Metatool benchmark for large language models: Deciding 2023.
whether to use tools and which to use. arXiv preprint
Scott H. Johnson-Frey. What's special about human tool use? Neuron, 39(2):201 2003.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang. and Yongbin Li. Api-bank: A comprehensive benchmark for cool-augmented llms. arXiv preprint arXiv:2304. 231v:2304.08244, 2023.
Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foundation models with millions apis. Intelligent Computing. 3:0063, 2024.
Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Making ppo even better: Value-guided monte-carlo tree search decoding. arXiv preprint arXiv:2309. 15028, 2023.
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, al. Toolsandbox: stateful, conversational, interactive evaluation benchmark for Ilm tool use capabilities. arXiv preprint arXiv:2408.04666 2024.
OpenAl. GPT-4o System Card, 2024.
URL https://openat com indx
gpt 40 sys-ssttte--ardd/
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint 12Xi::2005.12255. 2022.
Martin L Puterman. Markov decision processes, Handbooks in operations research and management science, 2:331-434, 1990.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang. Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Commu- nicative agents for software development. In Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 5174-15186 Association for Computational Linguistics, 2024. doi: 10011663/V1122224ALL-LLO 0ww.1....6663//1 /ttp://...oorr....116331/13111//
URL
https://doi.orge
acl ong 810
Shuofei Qiao, Honghao Gui. Chengfei Lv, Qianghuai Jia, Huajun Chen, and Ningyu Zhang. Making language models better tool learners with execution feedback. In NAACL, pp. 3550- 3568, 2024a. doi: 10.18653/1/2022.maaall.
long.ll-ong.195. 195. URL
org/
2024 naacl loogllogg... 195
Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch via self- planning. arXiv preprint wrXiv:2401.05266 2024b.
12

---

Published as a conference paper at ICLR 2025
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang. Cheng Qian, Runchu Tian, Kumlun Zhu, Shihao Liang. Xingyu Shen. Bokai Xu, Zhen Zhang. Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang. Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang. Cheng Yang. Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models. CoRR. abs/2304. 08354, 2023a. doi: 10448550/RXXV.2334.0055
URL
org//110...
18550/arX1:0 2304. 08354)
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, et al. Toolllm: Facilitating large language models master 16000+ real-world apis. arXiv preprint arXiv:2307.1677 2023b.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen. From exploration mastery: enabling Ilms to master tools via self-driven interactions. arXiv preprint ar wrXiv:2410.08/111 2024.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198343, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2023.
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the interna- fional conference for high performance computing, networking, storage and analysis, pp. 1-14, 2021.
Romain Robbes, Mircea Lungu, and David Röthlisberger. How do developers react to api depreca- tion? the case smalltalk ecosystem, In Proceedings of the ACM SIGSOFT 20th International Symposium the Foundations of Software Engineering, pp. 1-11, 2012.
Christopher D. Rosin. Multi-armed bandits with episode context. Ann. Math. Artif. Intell., 61)); 203-230, 2011.
doi: 10.1077/110472-01-99338 URL https:/ //cp:::://o....// http:////e...../...000..000
100722001-9258-
Anand Ashok Sawant, Mauricio Aniche, Arie van Deursen, and Alberto Bacchelli. Understanding developers' needs on deprecation as language feature. In Proceedings of the 40th International Conference on Software Engineering, pp. 561-571, 2018.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2023.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-600 2020.
Alexander Shapiro. Monte carlo sampling methods. Handbooks in operations research and man- agement science, 10:353 425, 2003.
Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. Small 1lms are weak tool learners: A multi-llm agent. arXiv preprint arXiv:2401.07322. 07324, 2024.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwiesser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484448) 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 35047666;:34443559 2017.
13

---

Published as a conference paper at ICLR 2025
Ad W Smitsman, Ralf FA Cox, and Raoul M Bongers. Action dynamics in tool use. Stone knapping: The necessary conditions for a uniquely hominin behaviour, pp. 129-144, 2005.
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-basee trajectory optimization for Ilm agents. arXiv preprint 2rXhv:2403.02502 2024.
Jimin Sun, So Yeon Min, Yingshan Chang, and Yonatan Bisk. Tools fail: Detecting silent emrors in
faulty tools. arXiv preprint arXiv:2406. /9228, 2024.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.053301 2023.
Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995)4476 482, 2024.
Boshi Wang. Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. Lims in the imaginarium: Tool learning through simulated trial and error. CoRR, abs/2403 044444646044, 2024. doi: 10.48550/
ARXIV.2403.04746. URL/ttpss / http///i..ot/ org. wwt.oo/.//.......... 48550 arxivv http:/awwwvv........ 2403.04748
XAgent. Xagent: An autonomous agent for complex task solving, 2023.
URL https://
github. cww 2aomm/OpeBBBB/Aeenn
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1066 2024a.
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36, 2023.
Seungbin Yang, ChaeHun Park, Tachee Kim, and Jaegul Choo. Can tool-augmented large language models be aware of incomplete conditions? arXiv preprint arXiv:2406.12300 2024b.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR. OpenReview.net, 2023.
URL httphtttss////eeeeeeii
net/ forum wwmmdiiddwwwwwulll SaIEE___llULLL-
Donglei Yu, Yang Zhao, Jie Zhu, Yangyifan Xu, Yu Zhou, and Chengqing Zong. SimulPL.: Aligning human preferences in simultaneous machine translation. The Thirteenth International Confer- ence Learning Representations, 2025. URL httph:: http::///ppeeeeee..eomo wwweeeewwwwwweeeeeeeeeeeeeeeeeee feretfformmm
XBF63bHDZW
Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua Tenenbaum, and Chuang Gan. Planning with large language models for code generation. arXiv preprint arX1v:2303.05510, 2023.
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404. 135001 2024.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403. 13372, 2024. http://///a/x orr. abs/ 13372]
Jing Zhou and Robert J Walker. Api deprecation: a retrospective analysis and detection method for code examples on the web. Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 266-271 277, 2016
Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. Solving math word problems via cooperative reasoning induced language mod- els. In Anna Rogers, Jordan Boyd-Grabes and Naoaki Okazaki (eds.), ACL, pp. 4471-4485 Association for Computational Linguistics, 2023. doi: 10.18653/v /2tp//22......moommme 245. URL
https://acl antho ogy. acl long. 245
14

---

Published as a conference paper at ICLR 2025
Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. Toolchain Efficient action space navigation large language models with a* search. arXiv preprint 2rXtv:2310.13227, 2023a.
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for LLM question answering with external tools. In Advances in Neural Information Processing Systems, 2023b. URL lttp:///pppaeeennnnppp...
i oc wepppprr wwlessppapal pa2pr/22pp |ac.g|22224455000000000022000000
and
Berrchmarks.htm
15

---

Published as a conference paper at ICLR 2025
CONTENTS OF APPENDIX
Implementation Details
17
A.T
Action and Node Details
17
A.2
Dynamic Environment
17
A.3
Cached Rollout Strategy
18
Algorithm Details
19
A.5
Tool Update Module Details
19
A.6
Additional Implementation Details
20
More Details in TooIQA-D
20
Benchmark Details
20
B.2 Details of Analysis tool variability
24
Case Study
25
C Cases of Self-reflection
25
C.2 Cases of Tool Update
26
C.3 Error Analysis
27
D Prompts
28
D.I Prompt for API Modification
28
D.2 Prompt for TOOLEVO
29
16

---

Published as a conference paper at ICLR 2025
APPENDIX
A IMPLEMENTATION DETAILS
A.1 ACTION AND NODE DETAILS
2023b), we steer the model to perform an API invocation in the REACT format Yao al. 2023). which includes thought (textual analysis), tool invocation (APIs), and observation (environmental feedback). For the sake of clarity, we will consider O as part of a, Here an example:
Following previous work (Zuang et al.
REACT format our Action a;:
Thought: To answer this question, I should first load the database containing coffee price information. The database named 'coffee' seems be the relevant (textual analysis)
Action: LoadDB (API name)
Action Input: ("DBName": "coffee") (API parameters)
Observation: We have successfully loaded the coffee database, including the following columns: Date, Open, High, Low, Close, Volume, Currency, (envi ronment feedback, O+)
where the tool invocation consists of "Action" (specifying the API name) and "Action Input" (defin- ing the parameters for the API). "Thought", "Action", and "Action Input" are generated by the LLM and the "Obvervation" is obtained from the environment feedback.
Additionally, in our work, we define the Node in the Monte Carlo Tree as shown in Figure [ (right) to record information, such as the action a, the Q-values, the visiting counts N, depth, and etc. Each node is defined to only contain a single action.
A.2 DYNAMIC ENVIRONMENT
As previously discussed, we define the dynamic environment as everything excluding the LLM.
Here examples of these errors:
An example of invocation errors:
Thought:
To answer this question, I should first load the database containing coffee price information. The database named 'coffee' seems be the relevant
Action:
LoadDB
Action Input: ("LoadDBName": "coffee") (error in parameter)
Observation: Your action is filtered due some emorr in content. Please assume all the actions permitted this environment try again. (invocation error message)
17

---

Published as a conference paper at ICLR 2025
example of deprecation errors:
To answer this question, I should first load the datahase containing coffee price information. The database named "coffce" seems be the relevant one.
Thought:
Action:
LoadDB
Action Input: ("DBName": "coffee") (The API invocation correct, but an outdated API.)
Loame is deprecated. Please use Initialize- Dulabas(DataaaseNaaeee] param example: ("DatabaseName": "flights") instead. (deprecation error message)
Observation: Error:
A.3 CACHED ROLLOUT STRATEGY
In MCTS, estimating the expected return for each state has consistently been a focal point of search (Shapiro 2003; Browne et al. 2012). AlphaGo (Silver et al. 2016) achieves efficient rollout through faster small model, while AlphaGo Zero (Silver et al. 2017) estimates the expected turn directly using the value model. However, in the context of tool learning, particularly when considering tool variability, both approaches face significant challenges. For the former, employing a small model for rollouts may yield biased reward estimates, as it may lack the comprehensive tool-using capabilities inherent LLMs. Additionally, this approach does not adequately mitigate redundant computations (Schrittwieser et al., (2020). For the latter, employing value model to esti- mate expected rewards dynamic environments is fraught with difficulties inherent to tool learning scenarios, especially tool variability. To address these limitations, we introduce a cached rollout strategy inspired by (Bellman et al., (015)) and (He et al., 2023). Our approach involves storing subtrees from rollouts to circumvent redundant computations, while ensuring these cached subtrees remain invisible to the MCTS. This maintains the integrity of the search process, as illustrated in obtain more accurate rewards compared to the
Figure At the same time, through rollout, value model in dynamic environments.
if select this node next time
Rollout
(1) check the cache
(2) directly use cached node, instead re-generating again
Cachedi
rrewadd
Cached
Figure 7: An example of cached rollout. (Left) During the rollout process, we will cache the sub- trees. These cached subtrees remain invisible during the MCTS process, particularly in the selection phase. This approach allows to obtain precise rewards though rollout. (Right) When we want to expand a node, we check the cache and directly use the cached node, avoiding the need to regenerate it, which helps prevent redundant computations.
18

---

Published as a conference paper at ICLR 2025
A.4 ALGORITHM DETAILS
Algorithm S Ddelineates our customized MCTS process. To accumulate valuable trial-and-error expe- riences, we actively enhance the interaction and exploration of LLMs within dynamic environments through MCTS. With the help of our self-reflection module, TOOLEVO generates verbal reflections of the eπrог state s, and attempts to provide valuable insights solve it (Lines 8-9), instead of in- terrupting further exploration of this error state. Moreover, we introduce a tool-update module to summarize the usage of new tools after successfully invoking it based on environmental feedback and self-reflection (Lines 14-15). It allows the LLM to incrementally adapt to the dynamic tool envi- ronment during the exploration phase. Through our customized MCTS, we seamlessly integrate the critical ability interact with dynamic environments into the tool chain. Subsequently, we employ instruction fine-tur tuning to enable the model to master this capability, thereby significantly enhancing the adaptability to tool variability.
Using Algorithm | we can construct a complete Monte Carlo tree that captures the model's in- teractions within dynamic environments. Multiple methodologies can be employed to enhance the model's s adaptation to tool dynamics, including: (1) extracting positive trajectories (successful paths from root leaf nodes) for supervised fine-tuning, (2) constructing contrastive pairs for prefer- ence learning methods such as DPO, or (3) implementing RLHF by training reward models with positive-negative sample pairs. As our research primarily focuses highlighting the crucial role of dynamic environments in tool learning, we adopt the most straightforward upprocch-superviseed
fine-tuning- to improve the model's capability to adapt to tool variability.
Algorithm Customized MCTS in TOOLEVO
Require:
Task description D, Initial collected APIs Pc, Policy model To, the number of expansion nodes k, Max depth T. Max simulations M.
1: Build the initial state 80 [D,
root node.
2: Initialize the set of leaf nodes C = {
3: while M do
4: St f- Selecce the best leaf node in C' by PUCT (Eq, 2p.
b Selection
5: if depth of 8, > T then
6: C CC {
7:
Continue
8:
if 8; is error state then
9:
Generate the reflective action by policy: a'er, oed...... d
aeef = TT(a(ss)
D Self Reflection Module
10:
else
11:
Generate candidate actions by policy: at, a² ...., af (o(as))). for at in {at, .f....,,,,, a²) or (a Lui, Re,, do
12:
13:
Get the expanded new state 11 Cat (s,, a.).
> Expansion
14:
if of is UpdateTool [newtool _desocoo__cecccoooooooo
Pap] then
Get P. 8t and Update Pe (Pe + Po;} in ti 1 + Randomly select new node from { s 141
b Tool- Tool-Update Module
16:
17:
+ Do cached rollot for s/4 and get the reward.
b Simulation
18:
Update the Q-values and the visit counts N along the path from s1t1 to the root node
(ss,a) + ((s,a) + Misa)i (r a(s,a)): ((s,a) + N(s +1. Backpropagation (st41, st....,, s
19:
C+ (C)
20:
M M
return Monte Calro Tree of D
A.5
TOOL UPDATE MODULE DETAILS
In our framework, we implement the tool update module _desc], which serves to update the descriptions of existing tools. This
system tool, referred to as
UpdateTool [newtoo: [newtool.desc), tool requires the model to autonomously summarize the usage of the new tool based environmen- tal feedback after successfully invoking the new tool. Following this, it updates the tool usage P. in the prompt. The tool update module can be viewed memory mechanism (Zhang et al. 2024
19

---

Published as a conference paper at ICLR 2025
where the model stores its own behavior and environment feedback into prompts facilitate future tool invocations. This adaptive process not only enhances the model's 's efficiency but also contributes to its adaptability tool variability. For further insights into self-reflection and tool update modules, we provide detailed case studies in Appendix
A.6 ADDITIONAL IMPLEMENTATION DETAILS
Parameter Details For fair comparison, utilize the same prompt and tool-update module, for all methods, including our TOOLEVO and all baselines, as detailed in Appendix D We only provide API usage of Pe in the prompt and use Pe., as the deployed API usage on the server. To facilitate interaction with the dynamic environment and collect trial-and-error experiences, we construct 20 trees for each question. We employ 3-shot learning to guide the pretrained model (Dubey et al.) 2024 Yang et al.) 2024a) in these interactions. All few-shot instances remain consistent across all methods and contain only the demonstrations of Pe. without information on tool variability. For MCTS, we set Cpuct to 1.25, consistent with Silver et al. (2016). We limit the maximum depth of each tree to 15, and set k to 5, which indicates that we will expand child nodes during the expansion phase. The final correct tool trajectory, defined the path from the root node to the leaf node, is denoted as yt according to the evaluation toolkit (Zhuang et al. 2023b). We randomly select up to correct tool trajectories per question, resulting in the collection of 27,823 and 32,034 trial-and- error experiences through Llama3-8B and Qwen2-7B for training, respectively. For self-improved training. we configure a batch size of 512, a learning rate of 2c-5, and specify the training epoch of 8. The training template corresponds to either llama or qwen, in accordance with guidelines from Dubey al. (2024); Yang et al. (2024a). We the maximum sequence length to 1024 and use cos ine learning rate scheduler with warm up rate of 0.03. Given the limited number of hyperparameters in our work, we believe that the provided details are sufficient for reproducing our study.
Baselines For Static-SFT, we collect correct tool trajectories using 3-shot learning and use Pe as the deployed API usage on the server, which is static environment. To ensure fair comparison, similarly to our TOOLEVO, we construct 20 trees to collect tool trajectories in the static environment and randomly select maximum of correct tool trajectories per question. Ultimately, we collect 35,830 and 37,985 tool trajectories through Llama3-8B and Qwen2-7B for training, respectively. All other training parameters of Static-SFT kept identical those used in our method to maintain comparability. For proprietary models employed in our comparisons, we use the following versions of each model: ChatGPT (gpt-30 5-turbo- 0125), GPT-4 (gpt-4 -etturrutoooooo 2024 04-09), GPT-4o (gpt-4o-)) 40-202--2222-- 202--0224-00 08- (0)0 GPT-4o-mini (gpt-64 1es--4o-minnn mini 2024 07 -8), and Claude- 3.5-Sonnet
Experiment Environments All experiments are conducted on Ubuntu 22.04 equipped with
NVIDIA A100 GPUs Our code mainly depends on python 3.1 1 and PyTorch 2.3. The pre- trained language models are derived from boggingFFace l We use I.l Llama-Factora (Zheng) jet al.) 2024) as the training framework and vLLM (Kwon et al. (023) as the inference frame- work. We trained all models with DeepSpeed ZeRO Stage2 (Rajbhandari et al. 2021 and Mlash-Attentinn
(Dao 2023).
B MORE DETAILS IN TOOLQA-D
B. BENCIIMARK DETAILS
For research purposes. we have constructed the first benchmark for tool variability, termed TooIQA- D. based on ToolQA (Zhuang al., 2023b). There are several compelling reasons for developing this dataset based on ToolQA:
Modifiability: Most existing benchmarks (Qin et al. 2023b Guo et al. 2024 provide APIs through platforms like RapidAPI (link), which limits our ability to control modify these APIs.
hittps://ww.python.
https://pttorch.c.
https://huggingee
20

---

Published as a conference paper at ICLR 2025
This constraint inhibits the implementation of tool variability. In contrast, ToolQA enables to directly modify the API services, facilitating the exploration of dynamic tool behaviors.
Unlike most existing benchmarks (Li et al., 2023 Tang et al., 2023 Du al. 2024), TooIQA emphasizes results rather than the sequence of API invocations. This distinction critical, as modifying the API in these benchmarks necessitates re-annotating the entire se- quence of API invocations. By focusing results, ToolQA eliminates the need for large-scale re-annotation, thereby facilitating a more flexible and efficient evaluation tool variability.
Result-oriented:
Sophisticated yet Comprehensive: Although ToolQA comprises only 12 tools, it offers a diverse range of functionalities, including text tools, database tools, graph tools, code tools, and system tools. This comprehensive yet manageable set of tools stands in stark contrast benchmarks that incorporate thousands of APIs (Qin et al.) 2023b Tang et al.) (023). where implementing tool variability for initial explorations can be exceedingly challenging. Consequently, ToolQA provides a balanced and practical platform for our pioneering research into tool variability.
Challenge: ToolQA poses significant challenges by encompassing 7 datasets across two lev- els of difficulty: easy and hard. These tasks cannot be accomplished solely with the internal knowledge of LLM; rather, they necessitate the integration of external knowledge through tool utilization (Zhuang et al. (023b). Therefore, the performance on these tasks directly reflects the tool-using capabilities of the models.
Table 6: Dataset statisties of TooIQA-D.
# Training set
# Test set
Context
Data Name
Easy
Hard
Easy
Hard
Flight
1087
950
100
100
Temporal
Coffee
804
1167
100
130
Yelp
1097
793
100
100
Spatial
Airbnb
1000
818
100
100
Social
Dblp
1000
963
100
100
Science
Scirex
348
493
100
100
Personal
Agenda
998
800
100
100
Table 6 provides detailed statisties for TooIQA-D. We use the dataset processed by Zhuang et al. (023b) as our test set, and reprocess a batch of data according to the settings established by Zhuang al. (2023b) as our training set. We ensure that there is no overlap between the training, and test sets. Different datasets cater to different contexts and require various tools, making ToolQA-D particularly suitable for exploring tool variability in initial research For more API settings, please refer to Appendix B
As discussed in Section we have constructed ToolQA-D, the first benchmark designed to investi- gate tool variability based on ToolQA (Zhuang et al.) (023b). We denote the original API usage of ToolQA as Pa and employ the GPT-4 to randomly modify Pa in terms of API names, parameters, and response formats, resulting in two new sets of API usage: Pss,, and Proco Note that the prompt is constrained solely to the collected API usage Pa. which may be outdated, while Ps,. and PeocD are deployed on the server that is agnostic to LLMs. Furthermore, the prompts do not contain any information about tool variability unless the model autonomously interacts with the external envi- ronment. In other words, deliberately refrain from indicating in the prompts that the API may become deprecated. Moreover, in our TooIQA-D, we ensure that the modified APIs retain similar meanings and adhere CamelCase conventions, aligning with real-world scenarios.
Specifically, we will maintain the system tools a static condition (e.g., "Finish", which re- lies solely the LLM and is independent on the external environment). We enable GPT-4 to randomly alter the names or parameters of the APIs, including textual variations and the random insertion of special characters. Here, we primarily focus on the underscore ("-"), as it is the most commonly encountered case. We will further investigate the impact of other special characters in Section 5.3 Additionally, we will slightly alter the functionality and the invocation patterns
"The ToolQA-D benchmark provided httptttp///// thub. com cwm/Chen woooo//oooooooo KVO
21

---

4. FilterDB: Published as a conference paper at ICLR 2025
of the APIs. For instance, for "Retrievekgenda"" will introduce the "return.num" parameter in Pe., and Prom to control the number of returned results. For "FilterDB", will modify the original string format ("NAME=Chao Zhang, Date<=2004-01-16V) for filtering conditions to a dictionary format, which requires the model to explicitly enumer- "NAME=Chao Zhang", "condition2". "Date<< 2000 1000-01-10 01-16")). Similarly, we will change the dictionary format back to a string for- mat for cases like "EdgeCheck" Additionally, we will impose formatting requirements parameters. For example, we will require the Python code begins with the phrase "The Python code is as follows "; otherwise, will be deemed failed invocation. Through this ap- proach, we aim for the LLM not rigidly replicate the API usage as provided in the prompt, but rather to genuinely possess effective tool-using capabilities. The specific API changes are illustrated below, with red text indicating changes compared to Pe-
ate each filtering condition (f" (f" conditiion
API
Pc
1. RetrieveAgenda: Action. Retrievekgenda
Action Input: ("keyword"; "Amelia Breakfast Meeting 2022/01/16"}
2. RetrieveScirex:
Action: RetrieveScirex
Action Input: ("keyword"; "F-Measure score of the EAST method on IC15 dataset for Scene.Text.Detection task")
3. loadn::
Action: LoadDB
Action Input: ("DEName": "airbab")
Action: FilterDB
Action Input: [ {"condition"; "NAME-Chao zhang, Date<=2004-01-165
5. GetValue: Action: GetValue
Action Input: "column.name"" "price, service fee")
6. LoacGraph:
Action: LeadGraph
Action Input: ( ("GraphName": "dblp")
7. NeighbourCheck:
Action: NeighbourCheck
Action Input: ("GraphName"; "PaperNet", "Node": "Blockchain Simulators:
A Systematic Mapping Study")
8. NodeCheck:
Action: NodeCheck
Action Input: ("GraphName": "AuthorNet", "Node": "Nicola Ferrier")
9. RogeCheck:
Action: EdgeCheck
Action Input: ("GraphName"; "AuthorNet", "Nodei": "Domingo Biel",
"Node2": "Arnau Doria-Cerezo")
10. SQuInterpreter:
Action: SQLInterpreter
Action Input: ("SQL": "SELECT Volume FROM coffee. coffe.dcoffee.data WHERE Date
= 2200001114";""
11. PythonInterpreter::
Action: PythonInterpreter
Action Input: ("Python"; "page_start, page.end = 1, 6
numpaages page.end page_start + print (num_pages)
22

---

Published as a conference paper at ICLR 2025
12. Finish:
Action: Finish
Action Input: {"answer"; "6"}
Pain
1. Retrievekgenda: Action: Agenda
Action Input: ("Query"; "Stephen's s Opera performance", "return.numt 3}
2. RetrieveScirex:
Action: Fetchscirexiatt.
Action Input: ("Querylextt": "Mean loU score the method")
3. LoacDB: Action: InitializeDatapees
Action Input: ("DatabaseName": "airbob")
4. FilterDB: Action: spply.Databbssoilteers
Action Input: ("conditionl"; "NAME=Chao Zhang", "condition2"; "Datec=2004-01-65))
5. GetValue:
Action: FetchValue.Rykey
Action Input: ("columni"; "price", "column2"; "service fee", "Returnkesult"; "True"]
6. LoadGraph:
Action: InitializeGraphatt.
Action Input: N"Graph.Name": "dbip")
7. NeighbourCheck:
Action: Verify.Neighhourrl
Action Input: N "AuthorNez", "graphNode": "Chao Zhang", "ReturnResult"; "True"]
8. NodeCheck:
Action: ValidateGraphhon
Action Input: Name:: "AuthorNe:", "graphNode": "Chao Zhang")
9. EdgeCheck:
Action: ValidateGraphedge
Action Input: ("Graph.Name"; "AuthorNet",
"NodeInfos"; "FirstNode [Chao Zhang], SecondNode [Weihong Lin]")
10. SQLInterpreter:
Action: ExccutesQiquery
Action Input: ("SQLCommand"; "The SOL code is follows:
SELECT Volume FROM coffee. coffeeecofffe.date WHERE Date '2000-01-14'-"
11. #ython/nterpretes
Action: Execute Python Script
Action Input: ("PythonCode": "The Python code is follows: import numpy as np print (mp.mean/ (1247.0, 253.0, 230.01))*)
12. Finish:
Action: Finish
Action Input: ("answer"; "6"}
P
1. RetrieveAgenda: Action: call_Hetrieve_
Action Input: ("searchilerm" "Ameli ia Hreakfast Meeting 2022/01/161
23

---

Published as a conference paper at ICLR 2025
"passage.nun": 3)
2. RetrieveScirex:
Action: CallRetrieveonscirex
Action Input: ("querykeyword"; "E-Keasure score of the EAST method"]
3. LoadDB: Action: Init_DB
Action Input: ( ("DatabaseName": "airbab")
4. FilterDB: Action: berilter Onbatabase
Action Input: ("filterCriterial", "NAME-K. John", "filterCriteria2": "Date<=200s-02-16")
5. GetValue:
Action: Extract.Value
Action Input: ("fieldNamc1"; "price", "ficldNanc2"; "service fee", "RoturnValue"; "True")
6. LoadGraph:
Action: Import_Graph
Action Input: ("Graph"; "dbip")
7. NeighbourCheck:
Action: Get.NeighbourListt
Action Input: ("Graph"; "AuthorNet", "Vertex"; "x. John", "Returnvalue"; "True"]
8. NodeCheck:
Action: TheNods
Action Input: ("Graph"; "AuthorNet", "Vertex"; "x. John")
EageCheck:
Action: Inspect_TheRdges
Action Input: ("CheckInfod Infos": "Graph [AuthorNet:), Vertexl [K. John],
Vertex2 [P
10. SOLInterpreter:
Action: ProcessSQIQuery
Action Input: ("SOL.Query"; "This is the SQL code: SELECT Volume FROM coffee.coffen WHERE Date 22000-01-14';")
11. PythenInterpreter: Action: Process.Python.CCo
Action Input:
Code": "This is the Python code:
import numpy np
print (np.mean (124700 253.0, 230.011)"}
12. Finish:
Action: Finish
Action Input: ("answer": "6")
B.2 DETAILS OF ANALYSIS ON TOOL VARIABILITY
As discussed in Section 5.3 we re-prompt GPT-4 to randomly generate modifications to the original API for specific change. We do not modify the system tools, such Finish and since they are system-level tools that do not rely on external servers. Here the example:
Changes on API Name
Original API -> Changed API
Textual Variations:
24

---

Published as a conference paper at ICLR 2025
Retrievedgenda -> PetchAgenda; LoadDB -> LoadDatabase;
Retrievešcirex FetchScirex FilterDB -> FilterDatabase
GetValue -> RetrieveValue;
LoadGraph -> ImportGraph
MeighbourCheck -> neighborrVrriiccttiio
NodeCheck -> NodeVerificateo
EdgeCheck -> EdgeVerifiicatin
SQu.nterpreter -> SQLProcessor
PythonInterpreter -> PythonProcesscry
2. Random Insertion of SpecialCharacter:
ketrieveAgenda -> Retrieve_kgenda LoadDB -> Load@DB;
RetrieveSci rex Retrieve- FilterDB -> Filter DB
GetValue -> Get@Value;
LoadGraph -> Load#Graph
MeighbourCheck -> Ceigh)
NodeCheck -> Node&Check
EdgeCheck Edge#Check;
SOLInterpreter SQL.Interpreter
PythonInterpreter -> Python- Sython-Iaterpreter)
Changes on API Parameters
Original Parameters -> Changed Parameters
Textual Variations:
keyword -> SearchQuery;
DBName -> TheDDName column.name -> SelectedColumn Node Nodeldentifiee return num -> numberOtResuits: SQL -> DatabaseQuery
condition -> FilterCondition;
GraphName fraphIdentifier;
ReturnResult OutputResult;
NodeInfos -> NodeDetails;
Python -> ScriptCode;
Random Insertion SpecialCharacter:
keyword >> keyword?;
DBName -> DB@Name
condition -> #condition;
column_name -> columnaname
GraphName Graph@Name;
Node _Node
ReturnRcsult Return&Resul;;
return_nun zeturnsnum;
NodeInfos -> NodcHInfos;
SQL -> SQL$
Python -> sPython;
Changes API Response Formats
Original Response Format:
"<INFORMATION HERE>"
Changed. Response Format: ("State": "Success" (or "Failed"), "Kessage": <INFORMATION HERE>}
C CASE STUDY
C.I CASES OF SELF-REFLECTION
In our TOOLEVO, require the model to engage in self-reflection upon encountering errors, rather than backtracking or stopping reasoning as previous work (Schick et al. [023] Qin et al. (023h) Self-reflection is reflected in the subsequent steps of thought and action, shown in Equation Here is an example of self-reflection:
An example of Self-Reflection
<SOMK PREV OUS STEPS HERE
Thought: The observation indicates that there type error when attempting subtract the integer (1245) from the float (1240.0). This that one of the values is not being properly converted integer before subtraction. need to adjust my code accordingly. (There was type error
25

---

Published as a conference paper at ICLR 2025
in the previcus step, and the model starts to reflect on the cause of this error.)
Action: Execute PyccuntePythonnSeeee Script
Action Input: ("PythonCode": "The Python code is follows:
crs.time "1245'
dep.time = *1240.0 0"
time_diff abs (int (crs.time) int (dep_time)
print (time_diff)
"}
ValueError: invalid literal for int( with base 10: '1240.0' (The model successfully resolved the TypeError through self-reflection, but triggered a new ValueError.)
Observation:
The observation indicates that the value '1240.0' cannot be converted into integer using 'int(". This because contains a decimal point, which is not compatible with integer conversion. the values are stored as strings, need to ensure they are properly formatted before performing arithmetic operations.
Thought:
('The model once again in engages in self-ref based envi ronmental feedback, analyzes the cause of the error, and provides potential solutions.)
on
Action: Execute Execute_Python.Seee Python.Script
Action Input: ("PythonCode": "The Python code is follows:
ers.time '1245'
dep.time 11240.0"
time_diff - abs (int (crsitime) int (dep_t www.iieeepppp.......... split (1) [0]))
print (time fi ")
Observation: 5
Thought: The Python code has successfully calculated the difference between the CRS recorded departure time and actual departure time, which is 5 minutes. Now, can provide this answer.
Action: Finish
Action Input: ("answer": "5")
Observation: Answer is CORRECT
In this example, we observe that based on environmental feedback, the model effectively identifies the causes of errors through self-refiection and corrects its actions, which is erucial for tool variabil- ity. It is worth noting that our approach does not rely on more powerful models, such as GPT-4, but fully leverages the model's capabilities learn from interactions and feedback from the dynamic environment.
C.2 CASES OF TOOL UPDATE
As discussed in Appendix A.5 we have transformed the tool update module into system tool:
UodateTool. Here is an example of tool update module:
example of Tool Update
<SOMB PREVI RWVIO ous STEPS HERE
Thought: With the opening and closing prices for 2012-03-08. calculate the percentage change
coffee price. The formula calculate percentage change (Cllosing Price Opening Price) /
26

---

Published as a conference paper at ICLR 2025
Opening Price) 100.
Action: Pythonlnterpreter (The model attempts to invoke the outdated API.)
Action Input: ("Python":
((float (189.35) #loat (189.7)) / fleat (189. 7)) 100
"}
Observation: Error: Pythonlateerpeeer((Pyyoooi is deprecated and removed in future releases. param example: ("PythonCode": "The Python code is as
Use Execute PwtponnnStripppppppoorrrro follows: /nimport numpy as np' (orrnttpp.mean((247.0 253.0, 230.01))*) instcad. on errors.)
Thought: PythonInterpreter has been deprecated. should use the Execute_Python_SS function to execute the Python code for calculating percentage change. (Self-Reflech base deprecation errors.)
Action: Execute PyccunePyttonSSeripp
Action Input: ("PythonCode": "The Python code is as follows:
percentage_change ((float (189. 35) float (189.7) 7))
float (1) 100
print (round (percentage_charrg) 2)
")
Observation: -0.18
The Execute Execute.Python. Pattnnyythhnniiiti API works intended, and have successfully calculated result. Now, we need to Finish task using the calculated result. First, let's update the tool description for the new API.
Thought:
Action: Tool (Tool Update)
"Execute Python Python.SSript(Pyilti Scrip((PythonCodee, which an updated version of PythonInterpreter and return the execution result according to the python code. For example, ("PythonCode': 'The Python code is as follows: (nfllos::nperreettgee change ((float(189) 189.35) floaa(189.7)) float(189.7)) 100',0 apprintrooun(peeennttgge.chgg 2)) }. (Successfully updated P. the prompt.)
Action Input: pneewtoll_dse""
Observation: The deseription for the new tool has been updated successfully.
<SOMB NEXT STEPS HERE
Through the tool updating module, the LLM can update the usage of tools based environmental feedback, allowing it to gradually adapt dynamic environments. The tool updating module also serves memory mechanism, summarizing environmental feedback and its exploration results to update API usage in the prompt, thereby facilitating subsequent invocations.
C.3 ERROR ANALYSIS
To facilitate future research, conducted comprehensive error analysis of performance in the "Pocco environment. We investigate gpt- 4o to-mini and the Llama3 series, such as Llama3-72B- Instruct, Static-SFT (Llama3-8B) and our TOOLEVO (Llama3-8B), by randomly selecting 50 error samples for analysis, as detailed in Table We have identified three main types of errors:
Invocation Error in New Tools: refers to the challenges that the model encounters when explor- ing the usage of new tools. Since only outdated API usage is provided the prompt, the model is more prone to make mistakes when using new tools in a dynamic environment. There are mainly two reasons:
Invalid Invocation: refers to instances where the model encounters invocation errors while invoking tools, such incorrect API names parameters, leading to failed tasks.
27

---

Published as a conference paper at ICLR 2025
Repeated Use of Deprecated Tools: indicates the repeated invocation of deprecated APIs as provided within the prompt. In this error, the model fails to realize from the environmental feedback that the API has been deprecated.
Planning Error: refers to the failure to complete the task due to planning errors. this error, tools based environmental feedback.
the model can successfully invoke
Invocation Looping: The model believes that the API does not return key information and keeps invoking new tools to complete the task. However, the model has missed the key information or did not return the key information due improper parameter settings.
Incorrect Output: The model employs series of tool invocations but ultimately arrives an incorrect result.
Other Error: refers error causes aside from invocation errors of new tools and planning emrors:
Instructions Not Followed: This means that the model does not follow the REACT format we specified (see Appendix A.IIfr details), resulting in task failure.
Tool Misuse: After encountering a deprecation error, the model hallucinates nonexistent tools and keeps invoking them, which ultimately leads to task failure.
Table Error Analysis
Invocation Error in New Tools
Planning Error
Other Error
Error Type
Invalid Invocation
Repeated Use of
Invocation Looping
Incorrect Output
Instructions Not Followed
Tool
Deprecated Tools
Misuse
GPT-4o-mini
36.0%
6.0%
16.0%
16.0%
14.0%
12.0%
-lama3-72B-Instruee
68.0%
6.0%
12.0%
4.0%
6.0%
4.0%
Statie-SFT
18.0%
78.0%
4.0%
0.0%
0.0%
0.0%
TOoLEVO
30.0%
2.0%
42.0%
26.0%
0.0%
0.0%
From Table 7 we have the following observations: (1) GPT-4o-mini is able to recognize deprecation errors and attempts to invoke new tools accordingly. However, may still overlook some details, leading to invocation errors, and GPT-4o-mini might not adhere to the specified format (Instructions Not Followed), resulting in task failure. (2) Llama3-72B-Instee
is able recognize deprecation errors. However, it tends to exhibit more errors when invoking new tools. (3) Static-SFT focuses on mastering tool usage in a static environment, making it easier to repeatedly invoke deprecated APIs (Repeated Use of Deprecated Tools). (4) From Table 4 method demonstrates strong ability to adapt to tool variability, significantly surpassing the basclines. However, our method still encounters issues such invocation errors with new tools and inaccuracies in assessing key information from the API responses. As a preliminary exploration of tool variability, we aim to leverage error analysis inform future work, ultimately making tool learning applicable real-world scenarios.
D PROMPTS
D. PROMPT FOR API MODIFICATION
To simulate tool variability scenarios effectively, we leverage the capabilities of GPT-4 to system- atically generate diverse and realistic API modifications. This approach ensures comprehensive representation of potential API variability.
You are tasked with modifying API and parameter names while adhering to the following guidelines:
Semantic Preservation! Maintain the original semantic meaning of the API and parameter names.
Naming Convention: Adhere camelCase naming convention for both API and parameter names.
Special Character Insertion: Randomly insert special characters (e.g.,
-r -, 8, 8, #) between words in the nanes. Do not insert special characters within individual words.
word Modi fication: Randomly replace words with synonyms related
28

---

Published as a conference paper at ICLR 2025
terms. You may add or words, ensuring the overall meaning is preserved.
Consistency; Ensure that the medified names remain consistent with common programming practices.
Logical Structure: Ensure that each modification is unique and that the overall structure of the API remains logical and intuitive developers.
<EXAMPLES HERE
Please apply these modi fications the following 1ist parameter names:
<ORIGINAL API AND PARAMETER>
<model's a output here>
D.2 PROMPT FOR TOOLEVO
Following Zhuang et al. 2023b), we employ few-shot learning to guide the pretrained model in
executing tool invocation, ensuring that the outputs adhere to the format of thougha/action/action input/observation Yao et al. 2023). as illustrated in the following example.
You are a powerful agent with excel excellent lent capabi 1 lities in using tools. Answer the questions best you can, You have access to the following tool:
(1) RetricveAgenda [keyword], which retrieves the agenda related to keyword.
(2) Retricvescirox [keyword], which retrieves machine learning papers'
paragraphs related to keyword.
(3) (DBdCa[) which loads the database DBName and returns the database. The DBName can be one of the following: flights/coffee/aiiid /yelp/agenda.
(4)
which filters the database DeNane by the column
column_name the relation (e.g., =, ?, etc.) and the value, and returns the filtered database.
(5) GetValue [column_name], which returns the value of the column column name in the database DBName.
(6) (radGraph[GraphNaeeee the graph. The GraphName can be one the following: PaperNet/Authozte (7) NeighbourCheck [GraphName, Node], which lists the neighbours of the node Noác in the graph GraphName and returns the neighbours.
which Icads the graph GraphName and returns
(8) NodeCheck GGrapNNae,, Nodel, which returns the detailed attribute information of Node.
(9) EdgeCheck [GraphName, Nodel, Node2], which returns the detailed attribute information on of the edge between Nodel and Node2. (SQU),
(10) St..ntterreeer [SQL], which interprets the SQL, query SQL. and returns the result.
(11) FythenInterpreter |Python), which interprets the Python code Python and returns the result.
(12) dinish[answer (13) UpdateTool [newtoolddesc), which updates the description of the tool.
which returns the answer and finishes the task.
Please adhere to the guidelines as follows:
1. When sclving problem, you should think step by step, where each step coput//bssevation.
includes 3 mini-steps Thought/Actien/Attan
2. If some steps require the use of tools, you should accurately specify
the tool names as well as the settings of the parameters.
3. If some step requires accurate calculation, you should write Python
code and execute for accurate result.
4. when you discover that a toel has been deprecated and successfully
invoke the corresponding new teel for the first time, please use
29

---

Published as a conference paper at ICLR 2025
UpdateTocl to add description the tool.
Upon completing the task, you should call Finish [answer] to return the answer.
6. Flcase use the following template.
Question: the input question
Thought: the text analysis
Action: the action to take, should be tool.namss
Action Input: the parameters of tocls, should be in JSON format
Observation: the result of the tecl invocation.
(this Thought/Acciin//Atiin Input/Observation can repeat N times) Hore are some examples:
<few-shot examples here>
Now! It's your turn.
Question: <QUESTION HERE>
<model's s cutput here>
30


Reviews:
[{"review_id": "X7QKRSwVu8", "reviewer": "Reviewer_hYZV", "summary": "While LLMs have been equipped to interact with external environments, improving their functionality, in many cases, these \"tools\" or external environments evolve in an unpredictable way inducing incorrectness in the LLM output. This paper presents a framework called ToolEVO that uses MCTS (Monte Carlo Tree Search) to acquire feedback from LLMs through autonomous exploration and interaction. The feedback is then used to adapt the LLM to the changing environment.", "questions": "1. Think about a more efficient design. \n2. The LLM should not be adapted. It is the adaptability module (which is the interface) that should be adapted. \n3. Need precise runtime statistics (including response times).", "limitations": "", "rating": 1, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Considers adaptation of LLMs to the changing environments which is a real problem for existing LLMs", "weaknesses": "1. The method is computationally inefficient and unrealistic in a real-world setting \n2. A simple method of accessing the APi's through a proxy can do the same job in a much more efficient way. \n3. There have been a lot of work on using design patterns to deal with evolving environments. The authors seem to be unaware of that literature. As a result, they have designed a cumbersome method that is not likely to work in practice.  \n4. In addition, the LLM itself might be modified. Thus the adaptation module needs to be separate from the LLM (in other words, the LLM does not need tom adapt). Tuning the adaptation module (which will be much smaller) should do the job. \n5. The design is flawed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Aon4PGDVgu", "reviewer": "Reviewer_Haos", "summary": "Large language models (LLMs) are typically trained on stationary datasets, but the world is constantly evolving. For example, the API of a programming tool can be updated or deprecated in new releases. An LLM trained on the old version of the API will produce incorrect outputs when interacting with the new API. This paper proposes a framework for adapting large language models to a variable external environment, in particular a variable API / tool. \n\nFor an LLM to adapt to new APIs different from those in the training set, it needs to interact with the environment and receive feedback. To this end, the authors propose to use online Monte-Carlo Tree Search (MCTS) to generate language actions to execute in the environment. The environment then returns the API outputs/error messages as well as a 0/1 reward at the end of the interaction sequence as feedback. In addition to MCTS, the authors propose two more mechanisms to improve adaptation: (1) self-reflection, where the model tries to explain the reason for encountering an error (as opposed to stopping right there); (2) tool-update, where the model generates a description of the updated API and add it to the context.\n\nThe authors evaluate their method, ToolEvo, on a curated benchmark of API / tool variability. There are three sets of APIs, $\\mathcal P_{c}$, $\\mathcal P_{s_\\text{in}}$  and $\\mathcal P_{s_\\text{OOD}}$. $\\mathcal P_{c}$ is the API seen in the training set, $\\mathcal P_{s_\\text{in}}$ is a slightly modified API that uses terminology seen in $\\mathcal P_{c}$, and $\\mathcal P_{s_\\text{OOD}}$ is completely out of domain. Compared to a suite of baselines that are either only pretrained or supervised finetuned on $\\mathcal P_{c}$, ToolEvo demonstrates significantly better adaptation capability, achieving higher success rates across the benchmark. \n\nTo summarize, the paper makes two contributions: (1) it proposes a framework for adapting LLMs to variable external environment, (2) it introduces a benchmark to evaluate such adaptation capabilities.", "questions": "1. How much overhead does MCTS add to the inference time?\n2. Can you compare ToolEvo to a method that uses online RL to adapt to the new API? If it's not possible, why?\n3. Can you compare ToolEvo to a version of the Static-SFT combined with self-improve and tool-update? The goal is to understand if MCTS contributes to the overall performance. According to Table 5, the main performance gain comes from self-improve and tool-update, and the MCTS-only version gets about the same performance as Static-SFT.\n4. The overall algorithm is quite vague. I would suggest adding an algorithm box in the main text.\n5. In line 275, I would suggest adding a description of $\\mathcal P_{s_{\\text{in}}}$  and $\\mathcal P_{s_{\\text{OOD}}}$ right away instead of referring to the appendix.\n6. According to Section 5.1 Implementation Details, the models are fine-tuned on a dataset of interactions with $\\mathcal P_{S_{in}}$. Can you expand on the reason behind this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper addresses an important research problem in the community, i.e., adapting LLMs to an evolving external environment. \n2. The proposed approach based on MCTS is grounded in a rich literature, and the self-reflection and tool-update mechanisms are novel contributions. \n3. ToolEvo exhibits strong empirical results on the benchmark, outperforming a suite of proprietary, open-source, and fine-tuned LLMs.\n4. The released benchmark could be useful for future research in this direction.", "weaknesses": "1. The setting is rather contrived. The authors curate three predefined sets of APIs, but the APIs don't change *within an evaluation episode*. So performing well on the test set doesn't necessarily mean the method adapts to a *constantly* evolving environment. In fact, The proposed approach wouldn't apply to a constantly evolving environment, as the Q value in the MCTS is not adaptive.\n2. The evaluation is potentially unfair. For a LLM to adapt to a new set of APIs, it needs to receive feedback. The baselines only have access to the API outputs and error messages, but ToolEval also has access to a reward at the end of each episode, which is privileged information. The authors don't compare to methods that make use of the reward information. \n3. The main contribution might not be the reason for the performance gain. According to Table 5, once they remove self-reflection and tool-update, the performance is about the same as supervised fine-tuning. This suggests MCTS may not contribute to the overall performance, despite it being introduced as a main contribution. \n4. The presentation has much room for improvement. To a general reader, too much context is deferred to the appendix, making the initial read particularly rough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gpc85zfPae", "reviewer": "Reviewer_zgUw", "summary": "The paper proposes ToolEVO, an algorithm for improving LLM tool use in dynamic environments. Specifically, the paper explores whether LLMs can call APIs to complete tasks, when the APIs can change over time. The change causes the API information in the static prompt to become outdated. The proposed method assumes the static prompt is not changeable and then proceeds to develop techniques to adapt the prompt and / or model using recently observed data. This is done via a combination of techniques (MCTS, Tool update, Self-reflection). The primary contributions of the paper are algorithmic and empirical. The paper also proposes a new dataset, ToolQA-D, constructed by random mutations of the ToolQA dataset. Experiments on ToolQA-D comparing ToolEVO using open-access LLMs to SOTA LLM baselines (closed and open access) are included along with ablation studies and other empirical analyses.", "questions": "1. Using the Average-Hard column in Tables 2, 3 and 4, is it reasonable to claim that Claude-3.5-Sonnet outperforms all methods, including ToolEVO? If yes, why is the ToolEVO result highlighted using the bold formatting in Lines 335, 338, 351, etc.?\n\n2. Assuming the prompts of the proprietary baselines in Tables 2, 3 and 4 are only shown the outdated API names and descriptions (P_C), is it possible to create a baseline that does a quick update of its prompt using the latest API names and descriptions before test-time evaluation / inference? How might this baseline perform?\n\n3. How exactly is MCTS used at inference time? Please consider using an illustrative example task (API invocation).\n\n4. What is the sample complexity and computational considerations (time and token costs) of ToolEVO? How does it compare with those of the proprietary LLMs?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper tackles an important and interesting problem of adapting LLMs to be able to invoke changing APIs correctly. This is a practical and important use case for LLMs. Progress here is likely to be of interest to the community.\n\n- The paper considers a number of strong LLM baselines consisting of SOTA closed and open-access LLMs and performs a large set of computationally intensive experiments. While it's not clear if ToolEVO outperforms these, it's still useful to see relative performance. The results seem to suggest that LLM tool use is rapidly improving.", "weaknesses": "- Overall, I found the problem setup and proposed methods a little challenging to understand in detail. The implementation details of MCTS and its various enhancements (cached rollout, inference, computational costs) are not clearly described. More on this below. \n\n- I didn't find the problem setup convincing. Restricting the LLM to only use P_C in the static prompt (Line 296) seems very limiting. Why can't the LLM's prompt be continuously optimized using recent API data in a separate process? That is what the ToolUpdate / SystemTool appears to do anyway. Please discuss this in more detail. Also, is it possible to include a baseline consisting of a proprietary LLM that is allowed to update its static prompt using the latest API invocation data, before each task?\n\n- Given the above, the use of a full anytime planner like MCTS with cached rollouts is an interesting choice. As mentioned above, I'm not sure exactly how MCTS is used at inference time. Given that the state $s_t$ includes a full history, it's unclear to me if any tree node is visited more than once. Please consider including an illustrative example on a single task starting from the root. I wasn't able to find any discussion of computational costs (time, token), hyper-parameters and other standard MCTS implementation details. As a result, I found it very challenging to assess the algorithmic novelty and contributions of the proposed method.\n\n- The methodology used to construct ToolQA-D from ToolQA isn't well motivated. Line 274 states \"we employ GPT-4 to randomly modify the collected API usage\". How exactly is this done? Why is this a good idea versus other mechanisms (e.g., using actual API versions of existing libraries)? Appendix A.3 doesn't say. Please motivate this choice and provide implementation details, assuming the goal is to propose ToolQA-D as a benchmark dataset for designing and evaluating tool-using LLMs.\n\n- The presentation of the results in Table 2, 3 and 4 is a bit confusing. The bold formatting typically suggests best performance but that doesn't seem to be the case here. For example, on Line 335, ToolEVO's 30.3 score is in bold text but the higher score of 45.3 for Claude-3.5-Sonnet on Line 331 is not. Why is this? More generally, the proprietary LLMs, using only P_C, seem to perform best wrt Average-Hard (rightmost column) in Tables 2, 3 and 4. Is this correct? If yes, does that make the baseline proprietary LLM the best method? Please consider including a detailed discussion of the main results in Tables 2, 3 and 4 with an appropriate caption.\n\n- The presentation and text gets a bit hand-wavy at times. Some examples below.\n  - (Line 140) \"rather than merely executing rigid tool invocations\"\n    - What does this mean? How do we know this is what the baseline LLM is doing?\n  - (Line 398) \"the stereotypes induced by Static SFT\"\n    - What stereotype is being referred to here?\n  - (Line 406) \"the model tends to lazily focus on how to use APIs provided in the prompt\"\n    - What does this mean?\n\n- Overall, the paper seems to have a few major technical issues. As a result, I don't think it's quite ready for publication at this time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MqiQ6k2XV6", "reviewer": "Reviewer_VQDs", "summary": "The paper tackles API usage of LLMs in the context of changing API specifications. First, the paper introduces a new dataset “ToolQA-D”, an adaptation of ToolQA with two new versions of API specifications that can be used to simulate a dynamic API change. Second, the paper proposes a method to improve LLM tool usage in such a setting: ToolEVO. They use pre-trained models to interact with the dynamic environment and then fine-tune their model using successful runs from those pretrained models. Their method also is encouraged to self-reflect and has the option to update the API documentation used in the prompt. They evaluate their model empirically on the ToolQA-D dataset and compare it to static approaches (not trained in a dynamic environment) and non-finetuned models. They also provide an ablation study regarding the self-reflection and tool-update modules of their method.", "questions": "- How was GPT-4 used to modify the API usage, and how is $P_{S_{ood}}$ completely different from $P_{S_{in}}$? Is the change in API different for every training/test example or is there one change in $P_{S_{ood}}$ and on in $P_{S_{in}}$ (is the variable name change same for all examples in each set)? Is it just the same dataset but with a different change in API parameter names?\n- What are the numbers in table 2-4 exactly? Are those the number of successful test cases solved (100 according to the appendix)? If so, why is every problem exactly x.0 except for “Coffee hard”. What is different there that there are decimal points? Also the authors talk about significant margins: Are there any statistical tests performed to support that claim? Are those results even from multiple runs?\n- How is the tool-update model used during testing? If the method updates the usage of the API in the first test case correctly, can it use the updated API documentation for the consequent tests? Or is it reset per test instance? \n- Does this method also generalize to real out-of-distribution tests? E.g., how would a model be able to adapt to an API that was not included in the training set?  \n\nMinor comments:\n- Figure 2: Why is there no drop in performance indicator for your method, but for the others?\n- It is unclear from the text and captions of table 2-4 what bold and underlined indicates. I would assume that bold is best performance and underline is second best?\n- The abbreviation SFT is never explained.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper tackles an important topic and provides promising results. Their method outperforms the presented baselines and could improve API usage for LLM agents in the future.", "weaknesses": "My main concerns are with the presentation and some open questions that i would want to be answered to better judge the results before publication:\n- The paper is at times hard to follow, and I found myself constantly looking into the Appendix to be able to follow. Even then some details are still unclear. (See questions)\n- The presentation of the results could also be clearer and more convincing. (See questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)