Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees
Authors: Sijia Chen
Yibo Wang
Yi-Feng Wu
Qing-Guo Chen
Zhao Xu
Weihua Luo
Kaifu Zhang
Lijun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Tool Usage
Direct Preference Optimization
Tree of Thought
Primary area: generative_models

Abstract: Tool-augmented large language models (LLMs) leverage tools, often in the form of APIs, to improve their reasoning capabilities on complex tasks. This enables them to act as intelligent agents interacting with the real world. The recently introduced ToolLLaMA model by Qin et al. [2023] utilizes the depth-first search-based decision tree (DFSDT) mechanism for multi-step reasoning with $16000+$ real-world APIs, effectively enhancing the performance of tool-augmented LLMs compared to traditional chain reasoning mechanisms. However, their approach only employs successful paths from decision trees (also called inference trees) for supervised fine-tuning (SFT), missing out on the potential learning opportunities from failed paths. Inspired by this, we propose an inference trajectory optimization framework based on preference learning to address this limitation. We first introduce a novel method for constructing step-wise preference data from tree-like expert trajectories, which leverages the previously ignored failed explorations in the decision trees. In the subsequent training phase, we first fine-tune the LLM with successful tool-usage expert trajectories and then apply direct preference optimization (DPO) with the preference data to update the LLM's policy, resulting in our ToolPrefer-LLaMA (TP-LLaMA) model. This approach not only enhances the utilization of original expert data but also broadens the learning space of the model. Our experiments demonstrate that by obtaining insights from errors in inference trees, TP-LLaMA significantly outperforms the baselines across almost all test scenarios by a large margin and exhibits better generalization capabilities with unseen APIs. At the same time, TP-LLaMA has also demonstrated superior reasoning efficiency compared to the baselines, making it more suitable for complex tool-usage reasoning tasks.

Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees
Sijia Chen 1.2,+ Yibo Wang 12.+ Yi-Feng Wu Qing-Guo Chen
Zhao Xu² Weihua Luo Kaifu Zhang Lijun Zhang!:4.0
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
School of Artificial Intelligence, Nanjing University, Nanjing, China
Alibaba Interaatoonal Digital Commerce *Pazhou Laboratory (Huangpu), Guangzhou, China {chensj, wangyb, changlj)@@amdda nju..du..c edu. [yixin. qingguo cag,, changgong. x2, weihua. luowh, kaifu. zkf )all lelaaaaabaaiinn..oom
Abstract
Tool- lool-augmented large language models (LLMs) leverage tools, often the form of APIs, to improve their reasoning capabilities on complex tasks. This enables them to as intelligent agents interacting with the real world. The recently introduced TooILLaMA model by Qin et all [2023] utilizes the depth-first search- based decision tree (DFSDT) mechanism for multi-step reasoning with 16000- real-world APIs, effectively enhancing the performance of nool-augmented L.LMs compared to traditional chain reasoning mechanisms. However, their approach only employs successful paths from decision trees (also called inference trees) for supervised fine-tuning (SFT), missing out on the potential learning opportunities from failed paths. Inspired by this, we propose inference trajectory optimiza- tion framework based on preference learning to address this limitation. We first introduce novel method for constructing step-wise preference data from tree-like expert trajectories, which leverages the previously ignored failed explorations in the decision trees. In the subsequent training phase, first fine-tune the L.M with successful tool-usage expert trajectories and then apply direct preference op- timization (DPO) with the preference data update the LLM'' policy, resulting (TP-LLaMA) model. This approach not only enhances the utilization of original expert data but also broadens the learning space of the model. Our experiments demonstrate that by obtaining insights from errors in in- ference trees, TP-LLaMA significantly outperforms the baselines across almost all test scenarios by large margin and exhibits better generalization capabilities with unseen APIs. At the same time, TP-LLaMA has also demonstrated supe- rior reasoning efficiency compared to the baselines, making it more suitable for complex tool-usage reasoning tasks.
in our ToolPrefer-LLaMe
Introduction
In recent years, large language models (LLMs) have exhibited impressive capabilities in various areas, including language understanding and generation, multi-modal content learning and reason- ing, and even embodied intelligence task processing [Brown al 2020 Alayrac et al,, 2022 Zeng
ct all 002
2022
Chen et all 2014, Lu ct 20245 Lf 2024, Zhang et al. 2024
Cao et all
Mazzagha all 2024 Despite these notable strengths, these models still
"Equal contribution. Work done during the internship at Alibaba International Digital Commerce.
1 CCoresponniig author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Environment
ToolPreference (Ours)
Porse
Rapidopi Nub APIS
Successful Steps
Failure Steps
User Queries
Explore
Depthrfirst Searck-hused
Collect
Successful Trajactories
DPO
TP-LLOMA (Ours)
SFT WaMM with SFT
Pre-troin LLM
Figure 1: Our Inference Trajectory Optimization Framework.
face significant challenges, such as lack of access to real-time information [Komeili et all, 2021
and difficulties in precise mathematical tasks [Patel et alll 2021, Lu all, 20236] The develop- ment of tool-augmented LLMS tackles these challenges by enabling LLMs to interact with external tools (often in the form of APIs), significantly enhancing their capabilities. This advancement al- lows LLMs to serve as efficient intermediaries between users and a large ecosystem of applications. Notably, tool-augmented LLMs based on the ChatGPT [Brown eLail [020] and GPT-4 [Achiam et al. [023] have achieved outstanding results by using few-shot or zero-shot prompts to activate the LLM's inherent cool-usage abilities [Deng all 2023, Lu et all 20238, Lin all 20221]. De- spite this progress, some studies demonstrate that open-source LLMs still exhibit significant gap their capacity to utilize external tools compared
of-te-o--the-rr (SOTA) closed-source models like GPT-4 [[iu et all, 20241 Wang et all [022N]. To bridge this gap, aligning these open-source LLMs with tool-usage downstream tasks is essential.
Currently, most efforts to align open-source LLMs with tool-usage downstream tasks rely on su- pervised fine-tuning (SFT) with expert trajectory datasets, which trains LLMs to learn strategies for subsequent actions based on previous actions and observations [Panl el al MI2A, Schick et al 2023]. Early studies in this field typically have limitations such restricted variety of APIs, the reliance on single-tool scenarios, and the use of simple reasoning methods [[We et all 2022, Yen all 2023, all (023). The recent work by Qin et al. [2023]], which focuses on the scene of LLM's multi-step reasoning with external tools, solves the above limitations. They introduce an instruction tuning dataset called ToolBench, which includes over 16, 000 real-world APIs and various realistic instructions, along with expert trajectories annotated by ChatGPT based on depth- first search- based decision tree (DFSDT) reasoning mechanism. They then perform SFT training on LLaMA with this dataset to create the TooILLaMA model, which shows remarkable performance. However, TooALLaMA's training is still based on expert behavior cloning, potentially limiting explo- ration of the target space and leading to suboptimal strategies. Additionally, although their expert trajectories are structured as DFS trees, only successful trajectories are utilized in the SFT training, which neglects valuable insights from failed attempts and results in low data utilization.
As the saying goes, "a fall into a pit, a gain in your wit", effective human learning involves not only drawing lessons from success but also from failures. Inspired by this, we propose new in- ference trajectory optimization framework for developing tool-augmented L.MSs as illustrated in Figure MI which enhances the tool learning process by incorporating previously ignored failure ex- ploration information via preference learning. Specifically, using the tree-like expert trajectories from ToolBench IQin et all, (023), we first parse each pair of branch nodes along the success- ful trajectory in the decision tree into a preference sample pair, thereby constructing a step-wise tool-usage preference dataset named ToolPreference. Subsequently, after conducting SFT training on the pre-trained LLM with successful trajectories, we employ the direct preference optimization (DPO) method [Rataloy et all [023] with the ToolPreference dataset to further align the LLM with tool-usage downstream tasks, and thus obtain our model, named ProlPrefer-LLaMe -TLMA.. Our strategy improves the utilization of expert data and simultaneously broadens the learning space.
Thhe dataset is available http:://eennnhe...aaaaeeeeeeeses

---

Our experiments are conducted on the test tasks from ToolBench. To evaluate the performance, we adopt two metrics: the pass rate, which measures the probability of the model successfully providing an answer within limited steps; and the win rate, which quantifies the likelihood that the evaluator will prefer the model's 's responses. From the experiment results, we have the following findings:
Across all test scenarios, TP-LLaMA consistently surpasses TooILLaMA and other base- lines, with average pass rate improvement of at least 12% and a win rate that outperforms nearly all other models by an average 4%. These results demonstrate that learning from failed attempts can significantly enhance the decision-making ability of LLMs. Addition- ally, our model shows superior generalization to unseen APIs.
Efficiency experiments show that our model requires an average of only 22.62 steps for DFSDT inference, compared to 32.06 steps for the SFT model. This enhancement stems from our method's 's ability to avoid unnecessary branch explorations in DFSDT reasoning.
Our ablation experiments verify that the effectiveness of our preference dataset and infer- ence trajectory optimization framework has nothing to do with the base model itself. Better results can still be obtained after replacing the base model with Mistral-7B [Mingg et all 20233, Qwen1.5577h [ll 2023]. and Gemma-7B [Team et all 2022
In summary, this work aims to enhance the performance of LLMs on multi-step reasoning with ex- ternal tools by integrating insights from errors in tree-like reasoning trajectories and employing step- wise preference pairs for preference learning. Our key contributions include: (i) A novel method for constructing step-wise preference data from tree-like expert trajectories, which may provide inspira- tion for future research; (ii) The proposal of using the preference learning method DPO to optimize the LLM's tool-usage ability, along with the development of the TP-LLaMA model; (iii) Extensive experimental evaluations and in-depth analyses of the TP-LLaMA model, providing evidence of its effectiveness and validating its performance across various dimensions.
2 Related work
In this section, we brietly review recent progress on tool-augmented large language models and the development of preference learning.
luoeend large language models. Over the past year, extensive research has been dedi- cated to developing tool-augmented LLMs, which exhibit improved reasoming abilities across vari- ous tasks by integrating external tools [Patil et anl 2023. Lu et ali 2023a. Schiik et all, 2023, Lm at. 20224). The workflow for tool-augmented LLMs typically involves four key stages: task plan- ning, tool selection, tool calls, and response generation. Early research mainly uses few-shot or zero-shot prompting methods to activate LLM's inherent tool-usage abilities, often employing GPT as the LLM agent manage several external tools such AI models, web search, Python, and more [Shen et all, 202KE, Lu all all, 20265 While GPT performs well with external tools, open- source L.MMs like ILaMA often struggle with direct tool usage and need additional task alignment. Therefore, subsequent research often utilizes Instruction-tuning datasets annotated with tool calls to train open-source models, enhancing their ability to use tools. At the same time, these studies continue to explore a wider range of tools and scenarios [Schick cl
2024 Patil all MIZE))
One of the most comprehensive efforts in this field is by Qin all [12X] They initially collect 16, 464 real-world APIs across 49 categories, then utilize ChatGPT to automatically generate in- structions that could invoke these APIs, and annotate expert trajectories to create high-quality instruction tuning dataset named ToolBench. During the annotation, they employ the DFSDT rea- soning mechanism to broaden the search space and enhance reasoning capabilities. By fine-tuning LLaMA on ToolBench, they develop LaMA, which has shown a compelling capability to handle both single-tool and complex multi-tool instructions.
Preference learning Preference learning uses human preferences from feedback data to assist decision-making. The earliest research in this field employs specially designed neural networks to help agents optimize action choices based on structured human guidance in programming lan- guages [Maclin and Shavlik, [19%]. Subsequent studies shift focus to learning from numerical re- wards provided by humans and performing reinforcement learning based on the prediction of these
3

---

rewards [[sblll al 2006, Knox and Stone 200X. Knox 2012 This approach finds applications
in areas like embodied intelligence Pilaskki et all BUTL Suay and Chernova, 20TT| and dialogue systems AEL all, 2016). The introduction of wreference-basee reinforcement learning marks key milestone the field, which uses qualitative human preferences, often in the form of rankings,
to guide the optimization of policy models [Akrorr et all 20T1. Cheng et all 20T Following this idea, Chrstano el all [2012Z] propose reinforcement learning from human feedback (RLHF), where a reward model is derived from human preferences to enhance reinforcement learning. This technique is later extended to natural language generation tasks [Kreutzer el all, 2018, Ziegler et all
201%] advancing the integration of preference learning with LLM research [Ouyang et al. 2022
3
Preliminaries
In this section, we start by formally defining the problem setup, and then we introduce key knowl- edge about preference learning methods, which is relevant to our approach.
3.1 Problem setup
In this work, we use an iterative paradigm for the LLM's multi-step reasoning with external tools. where the model selects each tool call based on the previous response, rather than pre-p pre-planning all tool calls at the start. Formally, we define as a state transition process. The environment consists of a set of available tools T = {Ti, T2..... Tn J. each with specific functionalities accessible through API calls. The task begins with an initial instruction I, usually consisting of user query and system prompt. At each reasoning step t. the LLM processes the current context S,, defined as:
S. = (T, H.}
where H, is the previous history, which includes the API decisions made AaddeAA1," A and the API responses received RR1, .... R: H, = ((A1,
RI).. (A), The LLM then generates an action decision A, based on this context, specifying both the tool T, E T to use and its parameters, After the tool executes, the response RG is generated and used to update the context. The reasoning process continues until the LLM determines that the task complete and produces final output O to answer the original query gives up the task.
(A--1, Re-1))-
3.2 Direct Preference Optimization
Preference learning has gained growing attention in LLM research. Its main goal is to opti- mize model outputs based on human (or expert) preferences, better aligning the model's behavior with the expectations of real-world applications. Assume there is preference dataset defined as D = {(x1(0) (0) .t (8) )>==.....! (D), where (0()) denotes the i-th prompt, yo and y/ denote the cor- responding preferred and dispreferred output respectively. Moreover, the notation 3/ > 31 x indicates that yo is preferred than y for prompt Because the true distribution of human prefer- ences is inaccessible, we assume it is generated by a latent reward model y (xx,, y), where higher rewards indicate stronger preferences. Then, according to Ratailoy all [2023] the human prefer- ence distribution p" can be captured by the Bradley-Terry (BT) model [Bradley and Tenry, 1952
exp (r" v1))
pa (y1
y2 [z)
a (r" (z. r" (z, v2)).
exp (r* (x,, v1)) exp (r* (2,
where a is the logistic function. Obviously, we can estimate the parameters of the reward model via maximum likelihood estimation (equivalent to minimizing the negative log-likelihood)::
Cn(re, D) = ((z. [logar (r& (w, r (I, v)))
(1)
where rs is parameterized reward model.
To optimize the inference trajectories of LLM based on human preference, a popular method in recent LLM research is Reinforcement Learning from Human Feedback (RLHF) [Christano et all 2017. Duyang et all 2022].. In the RL phase of this method, the optimization goal is
max m Ey-P P.y. (y(a) (Ir y)] (D [To (y) xx) || (y [x)],

---

where r is the reward model learned before, Ra is the policy model we need to optimize. 3 is weighting parameter that controls the deviation from the base reference policy model qee (i.e., the LLM after SFT training). In practice, is also initialized to the L.M after SFT. RLHF will use reinforcement learning methods (such as PPO [Schulman et all 201Zl) to optimize (4) and update the LLM's strategy, with (1, v) providing reward feedback. Additionally, some research in multi- step reasoning scenarios trains process reward models to evaluate each step instead the entire output [Ma ala RIRK Wang all RIZEE]. However, RLHF incurs significant computational overhead, long training times, and potential instability [Shenn et al. 2023a, Ratailov et all 2023), making it less suitable for general tool-usage tasks.
Therefore, we choose more convenient and faster approach that can also effectively align the model's 's preferences - Direct Preference Optimization (DPO) [Ratalov all 2023], which elim- inates the need learn the reward model and directly uses preference data to optimize the L.M. Specifically, the optimal solution of (2) can be written as
Z(x) Teet (y x)) exp (E)(C.N))
1
Ty (y x))
(3)
where Z(x) Ey Tee (y 2)) exp (fr(z, v) is the partition function Ratai ov et all 2023)
We
rearrange (G)) to express r(z, y) in terms of ,,, and Teet
r(z, v) Blog (y a)) Tret (y x))
BlogZ(r).
Substituting (+) into (II), finally get the learning goal of DPO
Coro (To. mee)) (z. > JD
logo
(Blog
To (Yw x)) Tee (yw x))
Blog (31 x))
nee (y1 x
where we is a parametrized policy that need to optimize. As result, the optimization objective of DPO avoids additional learning of the reward model and the RL process while maximizing the final reward, which is more suitable for our general tool-usage scenarios.
4 Our method
In this section, we introduce our inference trajectory optimization framework, beginning with an overview of the framework, followed by description of the preference data construction process.
4.1 The framework
Our framework is composed of two key stages: dataset construction and training. In the dataset construction stage, we create a tool-usage preference dataset, named ToolPreference, which is de- rived from the tree-like expert trajectories in Toolbench [Qin et all 2023). The specific process for constructing this dataset will be detailed in section !
Remark It is important emphasize that our preference data construction approach is not lim- ited to Toolbench and can be adapted to any tree-structured multi-step instruction -tniintinng dataset, offering flexibility for various upplications.
In the training stage, we first perform SFT training on a pre-trained LLM using resampled version of the instruction-tumn ttunongttnnnnnggo data from Toolbench (refer to Remark Ø for the resampling process). SFT training has been commonly adopted in previous research to enhance tool-augmented L.MMs. How- ever, mere cloning expert behavior through SFT is insufficient, explore the environment, and can result in suboptimal strategies. To address this, after the SFT train- ing, we further perform DPO training on the model with the ToolPreference dataset. This additional preference learning enhances the models reasoning capabilities when interacting with external tools and aligns its decision-making preferences with human preferences.
this method fails adequately
4.2 Preference data construction
Before introducing our preference data construction method, we first describe the dataset structure and expert trajectory format used in ToolBench [Qin et all 2023].
5

---

Dataset structure. ToolBench consists of two main components: API information data and instruction tuning data. The API information data is sourced from RapidAPI Hub and includes 3, 451 tools across 49 categories, with total of 16, 464 APIs (as each tool can have multiple APIs). Each API entry contains detailed information such as the name, description, HTTP method, URL, required and optional parameters, and executable code snippets for API calls. This comprehensive data enables LLMs to perform few-shot infer- ence with effective API calls. The instruction-tuning data includes various single-tool or multi-tool instructions as well as corresponding annotated expert trajectories, generated in self-instruction method by ChatGPT.
Expert trajectory format. While traditional LLMs often use sequential reasoning meth- ods like chain-of-thought (CoT) [Weretall 2022]. which follow a single path to comple- tion, ToolBench adopts depth-first search (DFS) reasoning approach. As shown in the left half of Figure D. expert trajectories in ToolBench are structured as decision trees with each tree node representing an L.M decision about an API call. Based on the tree struc- ture, ToolBench implements DFS reasoning using two techniques. First, it defines two additional functions: one is "Finish with final answer", where the LLM concludes it has gathered enough API responses to provide a correct answer and terminate the reasoning process, and the other is "Finish with giving up", where the LLM feels unable to proceed with the task, abandons the current path and returns to a previous node. Second, diversity prompts used to expand the search space. When expanding child nodes, the LLM will be prompted with information about previously explored child nodes of the same layer, and explicitly encouraged to generate different ones. Consequently, the LLM is allowed to either abandon the current path and restart from a previous step or proceed along a more promising path, exploring until answer is reached or the node limit is reached.
We employ the second release of ToolBench which includes over 120, 000 expert trajectories. Our approach is designed based on the motivation of improving data utilization. Although the tree- like expert trajectories in ToolBench extensively search the answer space. only successful paths are used in their training, neglecting valuable insights from failure paths. To address this, we extract preference decision pairs from each tree-like expert trajectory. After filtering out trajectories without failed exploration branches, we explore two different construction methods:
Path-wise means using an entire success path and an entire failure path in the same de- cision tree to form a preference pair. As shown in the upper right part of Figure [ (0, 9, 12, 13, 14, 15) is the success path of the decision tree, and (0. 1,2), (0, 5, 6). (0, 10,3,77 7, 8), (0, 10, 11) are 4 failure paths, so their Cartesian product can constitute a path-wise preference dataset, where > denotes the left part is preferred than the right part.
Step-wise means using each branch node along the success path in the tree and its corre- sponding pair of child nodes (which must contain child node on the success path) to con- struct a preference pair. As shown in the lower right part of Figure Z. (0, 9, 12, 13, 14, 15) is the success path of the decision tree, while and 9 are nodes with branches along the success path. Therefore, (0, 9) > (0, 1), (0,9) > 3), and (0, 9, 12) > (0, 9, 10) can respectively form a preference pair.
Although it is intuitive and common to use path-wise preference samples, this approach is not well- suited to our task scenario. Theoretically, may limit the model to only differentiate between correct and incorrect final responses to specific instructions, resulting in poor generalization with unseen instructions or tools. From an engineering perspective, learning preferences for an entire path at once is inconsistent with the model's reasoning mechanism of inferring the next API call based on the response of the previous API execution each time, which makes it inherently unsuitable for the DFSDT reasoning mechanism.
In contrast, the step-wise design highlights the differences between each reasoning step. providing the model with more fine-grained process supervision. Theoretically, this method can better ad- just the model's reasoning process and enhance its generalization performance. It is also a more suitable fit for implementation within the DFSDTT reasoning framework. Consequently, we create 69, 393 pairs of preference samples from ToolBench step-wise manner. Each pair is formatted
hhtcps//rapidapp.com///a https:/gghhuu..cm///opee////tltt

---

Depth-first Search-based Decision
Pathosise Preference Data Construction
Instriction
Success Path
N
Failure
Foilure
Fuluuee
Step Preference Construction
6 Failure
14
15 Success
Figure 2: Depth-first search-based decision tree and two preference data construction methods
as [Instruction, Input, Dutput). The Instruction includes the system prompt, detailing the DFSDT reasoning task and the relevant API documentation. The Input contains the user query and the reasoning history up to the current step, while the Output presents a preferred and dispre- ferred reasoning step for the given input. Additionally, to prevent information leakage, we carefully remove any diversity prompts from each node's information during parsing.
Remark 2 To ensure a rigorous comparison between models with and without preference learning in subsequent experiments, we also do not directly use the instruction-luning datasel provided by Toolbench during the SFT phase. Instead, we filter out expert trajectories lacking failed exploration branches, as these could not be parsed into preference samples. and resampled the remaining data to create our SFT training set. This ensures the training data distribution remains consistent across models, regardless of whether preference learning is applied.
5 Experiments
In this section, we investigate the performance of our inference trajectory optimization framework. We first introduce the experiments settings in Section 5.I. We then present the main results in Section 5 the efficiency experiments in Section 5.3. and the ablation experiments Section 5.4
5.1 Experiments settings
Evaluation metrics. Since model uses APIs from the online platform RapidAPI Hub, there may be changes such as version updates or service termination over time, making difficult to provide fixed solution path for each test instruction. Following Qin et [2023]] use pass rate and win rate as evaluation metrics in our experiments. The pass rate represents the proportion that the model successfully gives answers within certain number of reasoning actions (set 200 in experiment). Gi Specifically, a sample is considered passed if the reasoning trajectory finishes with the "Finish with final answer" API call. Additionally, we filter out samples that yield meaningless answers using a predefined set of feature keywords, such as "sorry", "apologize", etc. The win rate measures the likelihood that the solution path provided by the test model is preferred over the reference solution path for the same instruction. We use the answers given by ChaGGPTDDFSDT as the reference solution paths and employ ChatGPT to determine preference." D
'During experiment process, noticed that ToolBench has been updated with revised definition of pass rate 1Qmn et all 2izz! The definition we use in the main text follows the original version, while the revised definition and corresponding results will be provided in Appendix Bil The ChatGPT version used in the experiments in the main text is 3pt-3.5-turbo. 16k.
7

---

Table 1: Main Experiment Results. Avg represents the average pass rate or win rate of the 6 test scenarios. win rate higher than 50% means the model performs better than ChaGGPTDDFSDT.
Pass Rate
Model
GI-Ins.
G1-Tool
G1-Cat.
G2-Ins.
G2-Cat.
G3-Ins.
Avg
ChatGPT
0.52
0.55
0.60
0.51
0.51
0.21
0.48
Davinci
0.49
0.47
0.45
0.40
0.27
0.29
0.40
TooILLaMA
0.54
0.62
0.47
0.54
0.17
ILaMA with SFT
0.47
0.53
0.72
0.48
0.63
0.35
0.53
TP-LLaMA (ours)
0.55
0.65
0.80
0.62
0.67
0.61
0.65
Win Rate
Model
G1-Ins.
G1-Tool
G1-Cat.
G2-Ins.
G2-Cat.
G3-Ins.
Avg
ChatGPT
Davinci
0.37
0.37
0.35
0.35
0.29
0.54
0.38
LaMA
0.55
0.53
0.57
0.56
0.52
0.68
0.57
LLaMA with SFT
0.54
0.51
0.56
0.65
0.57
0.81
0.61
TP-LLaMA (ours)
0.56
0.59
0.54
0.70
0.64
0.86
0.65
Training settings. For the 2-epoch SFT training, we randomly sampled 11, 142 instances from the expert-annotated data in ToolBench after removing those without failed exploration branches. The batch size is 16 and the learning rate is le-5 during SFT training. For the 1-epoch DPO training, we randomly sample 8, 202 preference data pairs from ToolPreference dataset, the batch size is the learning rate is 1e-6 and 3 = 0.5 in (Q). It is important to note that our sampling is performed at the instruction level, which means that samples corresponding to the same instruction are either all included in the training set or none are included. We provide a detailed explanation of our design choices for training hyperparameters in Appendix A.I All our experiments are conducted on single machine equipped with 8 NVIDIA A100 GPUs with 80G memory.
Testing settings. We investigate six scenarios same as Qin al [023]: GI-Cat., GI-Ins., G1- Tool, G2-Cat., G2-Ins., and Gn-Ins.. The specific meanings are as follows: (1) G1: instructions that
only use a single tool; (2) G2: instructions that use intra-category multi-tools; (3) G3: instructions that use inter-ctteoryy multi-tools; (4) Cat. (Category): unseen tools that belong to the unseen category of tools the training data; (5) Ins. (Instruction): unseen instructions for the same set of tools in the training data; (6) Tool: unseen tools that belong to the same category of tools in the training data. Each test scene contains 200 test samples, except G3-Ins., which contains 100 test samples. The six test scenarios have different task difficulties and generalization challenges, which can well reflect the comprehensive performance of models.
Baselines. We compare model with several models without preference learning. Among them, we select the expert model ChatGPT and OpenAl Sictt-Davinci-03 (Davinci for short) as basclines. In addition, we also show the results of TooILLaMA and the model trained by SFT using resam- pled SFT training set (LLaMA with SFT for short) for comparison. Note that all models here are combined with DESDT for inference. In addition, regarding the TooILLaMA results, directly use the reasoning answers of LoMALLaMA on test sets provided by ToolBench's GitHub repository to calculate pass rates and win rates.
5.2 Main results
We employ LLaMA-2-7B as the base model of our training framework and finally obtain our model, named MaMPrefer-LLMMM. (TP-LLaMA). The context length of LLaMA-2-7B is extended to 8192 tokens to accommodate our tool-usage reasoning tasks. The main results are shown Table We have the following important observations:

---

Table 2: Efficiency Results of TP-LLaMA. Imp denotes the improvement of TP-LLaMA over LLaMA with SFT in terms of the average steps.
Model
G1-Ins.
G1-Tool
GI-Cat.
G2-Ins.
G2-Cat.
G3-Ins.
Avg
Imp
LLaMA with SFT
32.82
34.60
31.45
31.98
35.05
26.44
32.06
TP-LLaMA (ours)
24.54
24.19
23.85
23.98
23.53
15.61
22.62
29.44%
TP-LLaMA significantly outperforms LLMs without preference learning in terms of pass rate, demonstrating the best performance across all six test scenarios, with an average im- provement of over 12% compared to models not optimized using preference data.
Regarding win rate, TP-LLaMA also exhibits competitive performance, just 3% below TooILLaMA in the GI-Cat. scenario, while achieving the best results in all other scenarios.
Furthermore, TP-LLaMA shows strong performance in more challenging task scenarios such as G2-Cat., G2-Ins., and G3-Ins., maintaining effectiveness similar to that simpler tasks. Notably, in the G3-Ins. scenario, TP-LLaMA's pass rate increased by over 26%, proving that our DPO training process using preference data significantly enhances the model's ability to handle complex multi-tool tasks.
Although we use the provided reasoning answers of TooILLaMA from ToolBench's GitHub repos- itory to calculate its rates, the results indeed differ from those reported in their paper IQun all 2023]. This may be due to the reasoning answers version not matching the one used in their paper or differences in the evaluation environment settings. However, it's important to emphasize that our results remain valid and reliable. We apply consistent settings across all models, so their relative dif- ferences are meaningful. Overall, our experimental results indicate that through preference learning, TP-LLaMA can master various tool-usage instructions better and exhibits stronger generalization capabilities to unseen tools, categories, and instructions.
5.3 Efficiency Evaluation
We also evaluate the inference efficiency of TP-LLaMA on six test scenarios and employ the average number of DFSDT inference steps required for samples that ended with the Finish function as the metric. From Table D we can find that LLaMA with SFT requires an average of 32.06 steps for reasoning, while our TP-LLaMA only requires an average of 22.62 steps of reasoning in all test scenarios, with an improvement of 29.44%. These results clearly demonstrate that the inference efficiency of TP-LLaMA is remarkably superior that of the model trained only with success trajectories. This advantage arises from our step-wise preference data, which allows the model to identify the most optimal decisions at each step of reasoning through DPO training. a result, the model avoids the exploration of unnecessary sub-optimal branches in the decision tree, thereby increasing reasoning speed and efficiency.
5.4 Ablation experiments
In the ablation experiments, to verify the effectiveness of our framework, we further replace LLaMA- 2-7B with other base models, including Mistral-7B, Qwenl.5-7B, and Gemma-7B. The results shown in Table and Table
From Table B. no matter which base model is used, training on preference data can always bring gains the performance the model, which verifies the modee--nneeendent effectiveness of our framework. Specifically, in terms of pass rates, models that have learned from expert errors im- prove by at least 8% on average compared to those that only receive training on success trajectory information. Similarly, in terms of win rates, models with insights from preference data generally outperform those without preference learning. Table further confirms that our method significantly improves model inference efficiency by large margin, up average of 33.35%.

---

Table 3: Ablation Performance Experiment Results. Avg represents the average pass rate or win rate of the test scenarios. A win rate higher than 50% means the model performs better than ChaGGPT+DFSDT.
Pass Rate
Model
GI-Ins.
G1-Tool
G1-Cat.
G2-Ins.
G2-Cat.
G3-Ins.
Avg
Mistral with
0.70
0.43
0.42
0.53
0.46
0.27
0.47
TP-LLaMA (Mistral)
0.71
0.53
0.55
0.70
0.64
0.57
0.62
Qwen with SFT
0.69
0.51
0.51
0.66
0.55
0.49
0.57
TP-LLaMA (Qwen)
0.77
0.53
0.60
0.72
0.61
0.65
0.65
Gemma SFT
0.67
0.44
0.49
0.44
0.47
TP-LLaMA (Gemma)
0.80
0.48
0.61
0.70
0.65
0.68
0.65
Win Rate
Model
G1-Ins.
G1-Tool
G1-Cat.
G2-Ins.
G2-Cat.
G3-Ins.
Avg
Mistral with SFT
0.52
0.47
0.55
0.61
0.61
0.72
0.58
TP-LLaMA (Mistral)
0.53
0.50
0.57
0.62
0.64
0.74
0.60
Qwen with SFT
0.53
0.52
0.52
0.64
0.66
0.75
0.60
TP-LLaMA (Qwen)
0.54
0.54
0.58
0.66
0.67
0.81
0.63
Gemma with SFT
0.58
0.54
0.53
0.50
0.62
0.73
0.58
TP-LLaMA (Gemma)
0.61
0.57
0.58
0.65
0.67
0.75
0.64
Table 4: Ablation Efficiency Experiment Results. Imp denotes the improvement of TP-LLaMA over LLaMA with SFT in terms of the average steps.
Average Number of Steps in One Successful Path
Model
G1-Ins.
G1-Tool
G1-Cat.
G2-Ins.
G2-Cat.
G3-Ins.
Avg
Imp
Mistral with SFT
28.92
26.65
30.22
25.69
26.58
25.24
27.22
TP-LLaMA (Mistral)
25.30
25.01
23.36
23.51
20.74
16.42
22.39
17.74%
Qwen with SFT
35.74
34.66
36.85
32.74
36.18
37.93
35.68
TP-LLaMA (Qwen)
25.12
23.83
24.49
23.84
26.92
22.18
24.40
31.61%
Gemma with SFT
27.49
22.77
24.10
18.70
20.52
21.19
22.46
TP-LLaMA (Gemma)
17.15
13.88
15.91
13.63
13.30
15.94
14.97
33.35%
Conclusion and future work
In this work, we propose a novel inference trajectory optimization framework that leverages pref- erence learning enhance the performance of tool-augmented LLMs. We first built step-wise tool-usage preference dataset, 'ooolPreference, using our proposed new preference data construction method convert previously ignored failed explorations tree-like expert trajectories into valu- able training data. After initial SFT training on the LLM, we use ToolPreference for DPO training further refine the LLM's strategy, resulting in our TP-LLaMA model. Our extensive comparative experiments prove that TP-LLaMA significantly outperforms the baseline models in nearly all test scenarios by learning from single-step errors in inference trees. TP-LLaMA also exhibits superior generalization capabilities and efficiency. Furthermore, ablation experiments confirm the model- independent effectiveness of our framework. In future work, we will try to explore tool-learning research with more complex, human-like reasoning mechanisms, and incorporate preference learn- ing for further optimization. We also aim to extend our research to multimodal scenarios to evaluate the broader effectiveness of our approach.
10

---

Acknowledgments and Disclosure of Funding
This work was partially supported by NSFC (U23A20382, 62122037), and the Collaborative Inno- vation Center of Novel Software Technology and Industrialization.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint, arXiv.2303.08774, 2023.
Riad Akrour, Marc Schoenauer, and Michele Sehag. Preference-based policy learning. In Pro- ceedings of the 201 European Conference on Machine Learning and Knowledge Discovery in Databases (ECML. PKDD), pages 2011.
Maan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual lan- guage model for few-shot learning. In Advances in Neural Information Processing Systems 35 (NeurlPS), pages 23716-23736, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge,
Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint, arXiv:2309. 16609, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biomerrika, 39(3/4):324 345, 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 (NeurlPS), pages 1877-1901, 2020.
Qinglong Cao, Zhengqin Xu, Yuntian Chen, Chao Ma, and Xiaokang Yang. Domain-controoled prompt learning. In Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI), pages 936-944, 2024a.
Qinglong Cao, Zhengqin Xu, Yuntian Chen, Chao Ma, and Xiaokang Yang. Domain prompt learning with quaternion networks. In Proceedings of the 41st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26637-2666
2024b.
Qi Chen, Bowen Zhang, Gang Wang, and Qi Wu. Weak-eval-Stree Strong: Evaluating and eliciting lateral thinking LLMs with situation puzzles. In Advances in Neural Information Processing Systems 37 (NeurIPS), 2024.
Weiwei Cheng, Johannes Fürnkranz, Eyke Hüllermeier, and Sang-Hyeun Park. sinterence-baseee policy iteration: Leveraging preference learning for reinforcement learning. In Proceedings of the 201 European Conference Machine Learning and Knowledge Discovery Databases (ECML PKDD), pages 312-327, 2011.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30 (NIPS), pages 4302-4310, 2017.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. In Advances in Neural Information Processing Systems 36 (NeurIPS), pages 28091-2811 14, 2023.
Layla El Asri, Bilal Piot, Matthieu Geist, Romain Laroche, and Olivier Pietquin. Score-based inverse reinforcement learning. In Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 457-465, 2016.
Charles Lee Isbell, Michael Kearns, Satinder Singh, Christian R Shelton, Peter Stone, and Dave Kormann. Cobot in lambdamoo: An adaptive social statistics agent. Autonomous Agents and Ageti-Agent Systems, 3(13):227-354, 2006.

---

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint, MxXi:2231.00882 2023.
W Bradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforcement. In Proceedings of the 7th IEEE International Conference on Development and Leurning (ICDL), pages 292-297, 2008.
William Bradley Knox Learning from human-generate reward. Ph.D. dissertation, 2012.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL.),
pages 1460-8478, 2021.
Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for pruuence-to-seqeenee reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1777-1788, 2018.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference Machine Learning (ICML), pages 19730-19742, 2023.
Bootstrapping language-image
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. In Advances in Neural Information Processing Systems 37 (NeurIPS), 2024.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. MiniCache: KV cache compression in depth dimension for large language models. In Advances in Neural Information Processing Systems 37 (NeurIPS), 2024a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, al. Agentbench: Evaluating Ilms agents. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024b.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language mod- els. In Advances in Neural Information Processing Systems 36 (NeurIPS), pages 43447-43478, 2023a.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pages 14605-14631, 2023b.
Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint, arXiv: 2405.20797, 2024.
Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Let's 's reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint, LrXiv:2310.10080 2023.
Richard Maclin and Jude W Shavlik. Creating advice-taking reinforcement learners. Machine Learning, 22(1):251-228 1996.
Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Aaron Courville, and Sai Rajeswar. Multimodal foundation world models for generalist embodied agents. In Proceedings of the 41st International Conference on Machine Learning (ICML) Multi-modal Foundation Model meets Embodied Al Workshop, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (NeurIPS), pages 27730-27744, 2022.
12

---

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACL), pages 2080-2094, 2021.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint, arXiv:2305. 15334, 2023.
Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard S Sutton. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In Proceedings of the 2011 IEEE International Conference on Rehabili- tation Robotics (ICORR), pages 1-7, 2011.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan. Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint, W.KIv:2307.167899 2023.
Yujia Qin. Shihao Liang, Yining Ye, Kumlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong. Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating large language models to mas- ter 16000+ weal-world APIs. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model. In Ad- vances in Neural Information Processing Systems 36 (NeurIPS), pages 53728-53741, 2023.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems 36 (NeurlPS), pages 68539-68551, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, arXiv: 101117007774777 2017.
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint, utXiv:2309.15022 2023a.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yucting Zhuang. Hug- ginggpt: Solving ai tasks with chatgpt and its friends in hugging face. In Advances in Neural Information Processing Systems 36 (NeurIPS), pages 38154338 144-38180. 2023b.
Halit Bener Suay and Sonia Chemova. Effect of human guidance and state space size on interactive reinforcement learning. In Proceedings of the 20th IEEE International Symposium Robot and Human Interactive Communication (Ro-Man), pages 1-6, 2011.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprins, arXiv:2403.0829. 2024.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhi- fang Sui. Math-shepherd: Verify and reinforce 1lms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 9426 9439, 2024a.
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating ilms in multi-turn interaction with tools and language feedback. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Ch thought prompting elicits reasoning in large language models. In Advances 2022.
in Neural Information Processing Systems 35 (NeurIPS), pages
13

---

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In Proceedings of the 11th Interna- lional Conference on Learning Representations (ICIR), 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm- 130b: An open bilingual pre-trained model. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
Tianyu Zhang, Suyuchen Wang, Lu Li, Ge Zhang, Perouz Taslakian, Sai Rajeswar, Jie Fu, Bang Liu, and Yoshua Bengio. VCR: Visual caption restoration. arXiv preprint, arXiv:2406.06462, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint, arXiv: 19t.:::990008993.. 2019.
14

---

Appendix
A
Experimental details
In this section, we supplement some details of the experiments in the main text, including the training details in Appendix A.II the API information format in Appendix AA.2 and the ToolPreference sample example in Appendix A.
A.1 Details for training
Training hyperparameters We provide explanation of our design choices regarding training hyperparameters, specifically the sizes of the training and test sets. We first filter 42, 49,192 tree-like expert trajectories with branching nodes from Toolbench, which leads 69,393 DPO samples and 184,816 SFT samples after processing (as each instruction may correspond to multiple samples). After allocating a small part validation set, we sample training sets of different sizes based on these samples. The sampling methods we tried include "by instruction" and "by sample". For sampling by instruction, the size of the SFT training set ranges from 2,500 10,000 queries, and the size of the DPO training set ranges from 5,000 to 392 queries, yielding nine combinations. For sampling by sample, the size of SFT varies from 10,000 to 183,561, and the size DPO varies from 10,000 to 68,951, yielding seven combinations. We conduct small-scale tests based on these different training settings and find that increasing the size may lead to decreased model performance in scenarios with strong generalization, such as G3-ins. (e.g., with settings (SFT: 44,412, DPO: 41 J1,226), the pass rate drops to 0.36), possibly due to overfitting. Consequently, we select the set (SFT: 11,142, DPO: 8,202) } in our final experiments.
Computation time consumption With NVIDIA A100 GPUs, our SFT training phase takes an average of 4.6 hours, and the DPO training phase takes an average of 3.2 hours. In the inference phase, each API call takes about 3.4 seconds, and each query takes about 48.7 seconds. Computation time varies due to task complexity, network conditions, and API service status.
A.2 Details for API Information
Below we provide a detailed document of the API collected in ToolBench to help readers understand the format and content of API information.
API Information Sample
"name": "Get Character By ID"
urr"; "Get individual character by ID\n Options AnAn- Limit \u2192 Limit amount of responses received \n- Step \u2192 Skip amount of characters", "method": "GET"
requireed_parmmeterss [
"name": "id"
"type": "NUMBER", "description"; "default";
].
[]
"code": "import requests
" "teat_endpee _ess__endoint":: {
"err": "Please enter a valid number for character id.
15

---

A.3 Details for ToolPreference
Here we show an example preference pair in ToolPreference. For the sake of brevity, we have omitted some less important information, including some rules in the instruction, API parameter information, and some response content.
Preference Sample Pair
Instruction
You are AutoGPT, you can use many tools (functions) the following task....... Specifically, you have access to the following APIs:
{"name":
storaabioaa,... amazon_date
_scraper", "description": "This the subfunction for tool scraper', you can use this tool. The description of this function is: This endpoint get the product.".
general product description of any parameters":
serriessffor..o por__abiola_aaa amazon_daaaaaae data ("name": "description": "This is the subfunction for tool _scraper", wwwaaaaaanaanaaeeeeaeeeeee acraper' you
this tool. The description of this function is: This endpoint gives you the review of the product you enter.", "parameters";
("name": iet__aazon_prodduse becplucttoffeeeeeeeeeeeeeeee wwr_roorrlraaaaaaeeeeelerrrrrale amazon amazon_data _ata _scraper", "description": "This the subfunction for tool daaizoo_ddat__crpperr you can use this tool. The description of this function is: This endpoint gets the offer of any product.", "parameters": ......}
Input
("fron": "user", "value": "Can you fetch the product details, revieus, and offers for the product with ID B08L5NPENG? Also, provide with the search results for the product 'Apple MagSafe Charger "Bargr".Begiin"))
Begin!")
{"fron":
"assistant", "value": "Thought: Action: get amazon cacpa__rruu__ddeeeeee details for aar_abiola__mma amazon scraper Action Input: ('productid'; 'B08L5NP6NG "}
("fron": "function", "value": "('name': 'Apple MagSafe Charger', 'product_infonn iiniutttiiooomaiiis.... ('brand': 'Visit the Apple Store', 'brand_uri") 'https:// amazon com/stores/Appe AAplee/ full_descriptionn
=ast bln' }}}
'pricing': '0',
{"fro=": "assistant", "value": "Thought: Action: det__mazon_prrdue Senduct_revies_____ for.asiollaaaaaa___aaaaaaaaaaee_ amazon acrape Action Input: ('productid'; BB88L5NPGNG "}
("fron": "function", "value": "......
16

---

Preference Sample Pair
Output
{"from": "assistant", "value": "Thought: Action:
net_amazon_proee offe__fferrreeeeeeeeeeeeeeeeeeee for sorr_abillaa_ amala_amazon, dctape
Action Input: { ('productid'; BB88L5NP6NG }}"}
{"from": "assistant", "value": "Thought: Action:
grr_amazon_prrdutte detaalssfoo__abileeeeeeeeeee aaaaaaaaaaaaaaaaaaaaaaa Laan__dtt__eeee scrap_ Action Input: ('productid'; 'B08L5NP6NG }}"}
B
Supplementary experiment results
In this section, we first supplement the experiments on another defined pass rate Appendix B. Then conduct qualitative analysis of our method in Appendix B. giving specific case study.
B.1 Another definition of pass rate
In the second version of ToolBench IQin et all 2024 a new pass rate definition using GPT-assisted evaluation is introduced. For each query, it first checks if the "Finish with Final Answer" API is called: if not, it's considered a failure. it is, GPT evaluates whether the answer resolves the query. If successful, it is marked with the available APIs. If it isn't, it is still considered a pass; otherwise, it is marked as a failure. Due to the ToolBench API server being offline temporarily, access APIs for evaluation experiments with the new pass rate definition. To maintain consistency in API status, we utilize TooILLaMA's open-source modef to perform reasoning on the test sets, instead of reusing the reasoning answers from its GitHub repository. Similarly, we re-run tests for other models using our RapidAPI accounts. We employ gpt-3.5-turbo-166 and gpt-3.5-turbe 106 as GPT evaluators, with the results shown in Table N.
pass. If not, GPT further assesses whether the query solvable
use our own RapidAPI accounts
First, TP-LLaMA still outperforms the models without preference learning, further validating the effectiveness of our method. However, the absolute pass rates depend heavily on the specific GPT version. We observe notable differences in preferences and consistency across GPT versions. After repeating the evaluation of each sample 7 times, we find that spt-3.5-turbo-- 106 is more likely to mark sample as passed, while 3pt-3.5-turbo- 16k tends to judge it as not passed. This difference mainly stems from how each version assesses whether a query is solvable. Additionally, gpt-3.5- turbo-16k shows greater consistency across the evaluations, meaning it is more likely produce the same inference repeatedly. This highlights the importance of selecting the appropriate GPT version for evaluation, as relative scores may more meaningful than absolute ones.
Furthermore, we observe that the gap between TP-LLaMA and LaMALLaMA narrows under the new evaluation. We believe this is due during their respective training processes. TP-LLaMA tends to avoid giving up on reasoning and attempts partial answers, whereas TooILLaMA is more likely to abandon task entirely, leading to a complete failure. However, this gap narrows due to the use of GPT to evaluate whether the task solvable. (2) During this supplementary experiment, our RapidAPI accounts have access limits (some APIs even can only be accessed 5 times per month per account), reducing the number of valid samples the test sets. This particularly affects complex multi-tool reasoning tasks, where TP-LLaMA usually excels, making its performance gains appear smaller.
factors: (1) The models have different preferences formed
Additionally, the results report for LoolLLaMA still lower than those in Qin et all |022) likely due to shifts in the distribution of real-world APIs, which may make certain test samples unsolvable. Moreover, some features TooILLaMA learned from past environments may fully
17

---

Table 5: New Pass Rate Experiment Results
Model
GI-Ins.
G1-Tool
Gl-Cat.
G2-Ins.
G2-Cat.
G3-Ins.
Avg
gpt-3.5-turbo-16k
TooILLaMA
0.29
0.35
0.40
0.33
0.29
0.25
0.32
LLaMA with SFT
0.22
0.29
0.39
0.28
0.29
0.28
0.29
TP-LLaMA (ours)
0.27
0.37
0.48
0.35
0.36
0.35
0.36
gpt-3.5-turbo-1106
TooILLaMA
0.35
0.42
0.39
0.63
0.59
0.68
0.51
LLaMA with SFT
0.33
0.40
0.40
0.64
0.55
0.60
0.50
TP-LLaMA (ours)
0.37
0.40
0.41
0.65
0.63
0.69
0.53
align with current conditions, resulting in reduced performance. In the future, we can further explore ways to enhance the models performance stability in evolving environments.
B.2 Case study
We further illustrate the effectiveness of preference learning in improving the tool-usage capabilities of LLMs with case study focused on the G3-Ins. scenario. To begin, we present the query along with the relevant API documentation
Case Study: Query and Relevant APIS
I'm organizing film festival and I'm looking for award-winning films. Can you search for videos related to "award-winning" on Vimeo? Additionally, fetch the related people in the "film festival" category to invite them as judges. Finally, provide me with streaming link for a YouTube video with the ID "Uxxajl.WzzqQ"
Query:
Related A AI Documentation (parameter information is omitted):
Name: getrellaeedchamells baaichhannels_forrr.o for_vimeo Description: Get Related Channels.
Name: searchvideos, ffrr_fo______ vimeo Description: Search for videos.
Name: getrellteddpople. soaprdpppppe_frrr_eeeroe for for_vimeo vimeo
Description: Get list of related people for category.
Name: download lalooad_tteeeeee stream for www.ooostreeraeeettteee
Description: Stream or download info.
Name: Finish
Description: you believe that you have obtained a result that can answer the task, please call this function to provide the final answer. Alternatively, if you recognize that you are unable to proceed with the task in the current state, call this function restart. Remember: you must ALWAYS call this function at the end of your attempt, and the only part that will be shown to the user is the final answer, so it should contain sufficient information.
During our experiments, the first three APIs above are temporarily inaccessible since they are real- world. So the best response to this query is YouTube video link with the ID 'UxxajL WwzqY', completing part of the task. We then briefly present the reasoning trajectories from TP-LLaMA and LLaMA with SFT. Note that each reasoning path is limited to at most API calls here.
18

---

Case Study: Reasoning Trajectories from TP-LLaMA
The reasoning tree of TP-LLaMA includes 5 paths and show the successful path below.
Step 1 call [searchvideos __t____r________ _or__forreeee__ vimeo] with arguments ("format": "json". "query": "award- 1, get the response "h..The consumer key passed was not valid..."
Step 2 call Igstrelatedchannels fortrersssrrr__eeeeeeeee for lorffforvvvnnnn _iimo] with arguments ("category": "film festival", "format": "json"), ), get the response "...The consumer key passed was not valid..."
Step 3 call [download stream for for_rtsttrrt _t__ysttteammmmm y loriadd_yuubbeeeeee videos] with arguments ("is_id"; "UxxajLWwzqY") get the response ...'id": "UxxajLWwzqy" 'title": 'Icona Pop Love It (feat. Charli XCX) [OFFICIAL VIDEOJ',
Step 4 call [Finish] with arguments ("return type": "give answer", "final answer": "The streaming link for the YouTube video with ID UxxajLWwzqy is [link]..." "), get the response "successfully giving the final answer"
Case Study: Reasoning Trajectories from LLaMA with SFT
The reasoning tree of LLaMA with SFT includes paths. There is successful path since the model finally gives up. All the paths are listed below (the numbers in brackets represent the unique id each node).
Path 1
searchvideos for for_fforrviaor vimeo (1) >> searchvideos for vimeo ffr__rirroeooo (2)
searchvideos for vimeo (3) getrelatedpeople lfree_ffo______ vimeo (4)
Path 2 searchvidsos_for_ vimee (1) >> searchvideos for vimeo (2)
searchvideos for
vimeo (5)
Path 3 searchvideos_for vimeo (1)
>> searchvideos fordddsssffr__eee vimeo (2)
searchvideos_fe
for for_ vimeo (6) pentreelatedpopple.. sacteeopppeeoorrr_eeeeeeeee for_vimee (7)
Path 4 searchvideos_for for vimeo (1) searchvideos for for_foorvviiio vimeo (6) searchvideos for sie__vimeoo (8)
>> searchvideos for vimeo (2)
dearchvideessfoor_iinneee
Path 5 searchvideos_form for_iimeeforrrvioooooo vimeo (1) > searchvideos for, vimeo (9)
searchvideos_for_ picorrvvmmmeo (10) getrelatedpeoplee wimeo (11)
Path 6 searchvideos_ vimeo (1) searchvideos for vimeo (9)
searchvideos_for_vuee foreesfoorrveee (10) searchvideos for_iimeoorrreooooooo (12)
Path 7 searchvideos for vimeo
(1) >> searchvideos focteeessffo____e for for_fforrre_rr vimeo (9)
searchvideos_fout htwptiddoofffor_ieieed vimmovimmooo (13) Finish with give up and restart (14)
Path 8 searchvideos_forr_
(1) gttrelattedhammel__frr.ro
(15)
>> Finish (16)
We observe that because LLaMA with SFT repeatedly tries inaccessible APIs (possibly using differ- ent arguments) without first accessing the accessible YouTube API, it finally mistakenly chooses to give up reasoning and is unable to give partial answer. In contrast, TP-LLaMA successfully calls the YouTube API to provide the best possible answer while using fewer inference steps.
19

---

C Limitations
While this work demonstrates promising results, it also has some limitations. First, the performance of our approach relies on the quality of the decision tree. We parse preference pairs from trajectories that experts naturally explore, though the quality of these trajectories still requires evaluation. Man- ually introducing suboptimal branches at specific nodes might provide a more effective approach. Additionally, our method currently does not compare preferences between steps on failure paths, suggesting room for improved data utilization. Finally, our approach requires inputting all histori- cal information along the path at each reasoning step, which can be time-consuming. Implementing summary steps during reasoning could help streamline interaction text, assist the model in extracting relevant information, and improve reasoning efficiency.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in Appendix C
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have fully disclosed all the information needed to reproduce the main experimental results in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our data is available at http:://wggayaaacoooaaeeeeeeeees
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (tttps://mips.ccm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
wwabbbmisssoooooiicy.. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We have specified all the training and test details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally in experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provide sufficient information on the computer resources needed to reproduce the experiments in Section [S]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps////ourrpps.ccc//b//////td
Answer: [Yes]
Justification: The research we conduct in our papers always complies in all respects with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We think there is no societal impact of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g,, deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not carry high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In section 5i, we have cited the original paper. We have also stated the version of the asset is used with a URL.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode com/datasets, has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: Our data is available at https:/wwggngsscoooodaassssssees
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection. cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "rOLVP0M1JA", "reviewer": "Reviewer_z4Vk", "summary": "This paper proposes a method to collect preference data from decision trees, making use of the failed trajectories which are previously ignored.  Based on ToolBench (https://arxiv.org/abs/2307.16789) and the decision trees provided in the original dataset, a preference dataset, ToolPreference, is built with the proposed method. DPO is used to optimize LLaMA-2-7B on the new dataset, and the resulting model, TP-Llama, achieves state-of-the-art performance on ToolBench.", "questions": "The result of ToolLLaMA reported in this paper is different from the original paper. l feel like it's due to different definitions of \"pass rate\", but can you give an explanation on this?", "limitations": "The paper lacks a thorough discussion on its limitations. For example, it seems that the performance of the method highly depends on the quality of the decision trees, while \"ETO\" mentioned in the paper doesn't.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Though the idea of the paper is not completely novel, it's the first to apply such ideas on this specific domain. \n2. The paper is well written. The results are sound and solid. \n3. The paper clearly expresses its own contribution. Yet, it still needs to clarify a few issues regarding the results of baseline methods.\n4. The paper is of moderate significance. It provides a new preference dataset for LLM agents.", "weaknesses": "1. The paper is related to the works on process reward, e.g., https://arxiv.org/abs/2310.10080, https://arxiv.org/pdf/2312.08935, yet does not discuss about it in the related works section.\n2. Two different methods of preference dataset construction are discussed in the paper (path-wise and step-wise), but there is no result supporting that discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the constructive reviews! We will revise our paper accordingly.\n\n------\n\n**Q1**. \"The paper is related to the works on process reward, e.g., arxiv: 2310.10080, arxiv: 2312.08935, yet does not discuss about it in the related works section.\"\n\n**A1**. Thank you for pointing out these important references. Both the two reference works train a process reward model to evaluate each step in the multi-step reasoning task. Our motivation is similar to theirs, but instead of explicitly performing reinforcement learning on the model with process rewards, we construct a preference dataset based on the characteristics of DFSDT to optimize the preference of the model. Their works provide inspiration for us to improve the way we construct preference data in the future. We will cite them and add more discussions in the revised version.  \n\n------\n\n**Q2**: \"Two different methods of preference dataset construction are discussed in the paper (path-wise and step-wise), but there is no result supporting that discussion.\"\n\n**A2**: Thanks for the suggestion. Due to code difficulty and time constraints, we are unable to provide experiment results here, but we will add comparative experiments in the revised version. Below we briefly give some theoretical explanations.\n\n+ Our model employs step-wise reasoning via a depth-first search, making only one decision (i.e, API call) per interaction, and each step is based on the previously obtained API feedback. The path-wise method focuses on learning an entire reasoning path, which may cause the model to generate the entire decision sequence at once without receiving synchronous feedback from APIs. We believe this approach is fundamentally unsuitable for our step-wise reasoning paradigm. Using a path-wise preference dataset might negatively impact the model's single-step inference performance.\n+ The recent work of Lai et al., 2024 [1], who compared vanilla DPO and step DPO, reveals that step-wise preference optimization benefits long-chain reasoning. We are confident that similar results would be observed in the context of complex tool usage.\n\n------\n\n**Q3**: \"The result of ToolLLaMA reported in this paper is different from the original paper. l feel like it's due to different definitions of \"pass rate\", but can you give an explanation on this?\"\n\n**A3**: Thanks for your question. The ToolLLaMA results we reported indeed differ from the original paper, but this disparity is not due to different definitions of pass rate. Below, we give an explanation.\n\n+ During our experimental process, we noticed that the original paper had been updated with a revised definition of pass rate and corresponding results. Given resource constraints, we followed the first version's definition [2], using their provided test answers of ToolLLaMA on GitHub (reproduction_data.zip) to compute pass rates. Moreover, it is important to emphasize that we could not reproduce their reported results with either definition.\n+ Actually, several researchers on GitHub have also reported inconsistencies between their results and the original results, without a response from the original authors yet. Potential reasons for these differences may include: (1) differences in ChatGPT APIs, despite we use the same version with them, (2) differences in the evaluation environment configurations.\n+ While our reported results do not match those in the original paper, it is important to emphasize that our experimental outcomes are still valid and credible. Our evaluations used a consistent setup for all models, so the relative differences between models are meaningful. Moreover, the second version's pass rate is revised upon the first version, and thus, the first version's pass rate also holds a certain reference value.\n\nWe will continue our efforts to contact the original authors to align our evaluation settings. Any updates will be included in the revised version of our paper.\n\n------\n\n**Q4**: \"The paper lacks a thorough discussion on its limitations. For example, it seems that the performance of the method highly depends on the quality of the decision trees, while \"ETO\" mentioned in the paper doesn't.\"\n\n**A4**: Thanks for the suggestion. We will include a thorough discussion of limitations in the revised version. As noted, the performance of our method relies on the quality of decision trees. In addition, we can not compare preferences between steps on unsuccessful paths, leaving room for data utilization improvement. In future work, we aim to create a general reward model for better preference data construction and filtering. Additionally, our method requires inputting all historical decision information at each inference step, which is time-consuming and increases the difficulty of understanding. To address this, we plan to add a summary step in the reasoning process to shorten interaction text and help the model extract useful information, improving reasoning efficiency.\n\n------\n\n[1] Lai, Xin, et al. Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs. *ArXiv preprint*, arXiv:2406.18629.\n\n[2] Qin, Yujia, et al. Toolllm: Facilitating large language models to master 16000+ real-world APIs. *ArXiv preprint*, arXiv:2307.16789v1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eVKd6Vh4t6", "reviewer": "Reviewer_d56d", "summary": "The authors aimed to enhance the tool-augmented language model (LLM) by efficiently learning from the failed explorations that previous works had overlooked. Step-wise preference data were extracted by pairing the branch nodes with the corresponding child nodes from the successful path in the tool-using decision tree. A base model was fine-tuned using DPO on this new dataset, in addition to SFT using only successful trajectories, resulting in ToolPrefer-LLaMA. Evaluation metrics included pass rate and win rate, and assessments across six test scenarios demonstrated superior performance compared to baselines, along with a reduction in inference steps. Furthermore, the authors conducted an ablation study on several base models to validate the effectiveness and generalizability of their method.", "questions": "- Computation Time Clarification: Could you specify the computation time on the mentioned infrastructure? \n\n- Error Node Handling: How do you plan to address the issue of \"error\" nodes in the preference pair generation method? Could you provide examples or a detailed explanation?\n\n- Evaluation Criteria: Can you clarify the criteria for what is considered a false positive and the details of the filtering process? This would help in understanding the robustness of the evaluation.\n\n- Dataset Settings: Are the test scenarios and dataset settings identical to those used by ToolBench, or were they resampled, and what scale? This information is necessary for reproducibility and context.\n\n- ChatGPT Version Specification: Which specific version of ChatGPT was used in the baseline? This is important for replicating the results.\n\n- ToolLLaMA Results Usage: How were the ToolLLaMA results used in your comparison? Were the scores used directly, or were the answers recalculated? Clarifying this would help in understanding the validity of the comparison.\n\n- Same amount of post-SFT compute on different dataset: Would you consider conducting additional comparative experiments to address this? if not, could you provide specific reasons why you believe this is not a concern?", "limitations": "See above", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a method to leverage the inherent preferences embedded in failed trajectories from existing benchmark datasets, without imposing too much computational overhead. It improves data utilization.\n- The ablation study demonstrates the efficiency and portability of this method across different base models, indicating its potential for generalizability and immediate applicability to existing systems.\n- This method is practical and logically intuitive, making it likely to be adopted in future tool-augmented training processes.", "weaknesses": "- Handling of \"Error\" Nodes in the ToolBench Dataset\nThe proposed preference pair generation method does not account for \"error\" nodes in the original ToolBench dataset (see Figure 4. from https://arxiv.org/abs/2307.16789). This oversight could lead to the generation of incorrect preference pairs, such as `0-9-12(api1_failed) > 0-9-10(api1_succeed)`, potentially skewing the results.\n\n- Pass Rate Metric\nThe authors mentioned in Section 4.1 that the pass rate evaluation does not verify whether the given answer meets the requirements of the instruction. This could result in inflated scores. Additionally, the criteria for filtering out false positives and the details of the filtering process are not specified, which is a significant oversight.\n\n- Evaluation Settings\nThe paper introduces six test scenarios that closely resemble those described in ToolBench. However, the detailed testing dataset settings are not specified. Clarification is needed on whether these datasets are the same as those used by ToolBench or if they have been resampled.\nThe specific version of ChatGPT used in the baseline is not mentioned. Knowing the exact version (e.g., gpt-3.5-turbo-0125, gpt-4-0613, gpt-4-0125-preview) is essential for replicating the results and understanding the capabilities of the baseline model.\n\n- ToolLLaMA Results Report\nThe paper states that ToolLLaMA results were \"directly used,\" but it is unclear how this was done. If the scores were used directly, they do not match those in the referred publication. If the answers were used for score recalculation, the discrepancy in test performance, such as the pass rate for ToolLLaMA in G3-Ins. (0.17) versus I3-Inst. (66.0%), needs to be explained.\n\n- Data benefit or more post-SFT training\nThe DPO fine-tuned models on the ToolPreference dataset outperform both commercial models and open-source ToolLLaMA, as well as in-house SFT-trained LLaMA. This improvement might stem more from increased computational investment than from the dataset's inherent benefits. To clarify this, experiments should compare DPO fine-tuned SFT-trained models using different datasets: ToolPreference versus successful paths from the ToolBench dataset. This approach would maintain consistent compute resources and DPO methodology, allowing for a direct comparison of how dataset choice impacts performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the constructive reviews! We will revise our paper accordingly.\n\n------\n\n**Q1**: \"Computation Time Clarification: Could you specify the computation time on the mentioned infrastructure?\"\n\n**A1**: Below, we specify the computation time:\n\n+ SFT training: 4.6h for 2 epochs.\n+ DPO training: 3.2h for 1 epoch.\n+ Inference: 3.4s per API call and 48.7s per query. Computation time varies due to task complexity, network conditions, and API service status.\n\n------\n\n**Q2**: \"Error Node Handling: How do you plan to address the issue of \"error\" nodes in the preference pair generation method? Could you provide examples or a detailed explanation?\"\n\n**A2**: Thanks for the valuable insight. On the one hand, most error nodes appear on successful paths because some APIs are temporarily invalid due to their real-time nature, so the model can't get useful responses while generating correct API calls, not necessarily model decision errors. If the APIs become responsive later, the decisions would yield correct results. On the other hand, the error nodes on successful paths from incorrect API calls or parameter errors are relatively few, thus having minimal impact on results.\n\nNonetheless, we plan to address this issue in the revised version as follows:\n\n+ Use GPT to assess the legality of API call names and parameters, as well as evaluate the reasonableness of preference pairs.\n+ Filter out error nodes caused by API response failures in the preferred nodes and assess the impact on results.\n\n------\n\n**Q3**: \"Evaluation Criteria: Can you clarify the criteria for what is considered a false positive and the details of the filtering process? This would help in understanding the robustness of the evaluation.\"\n\n**A3**: Thanks for the question, we will add more details in the revised version. Here’s a brief explanation:\n\n+ False positives are instances where \"Finish with final answer\" is called but the response lacks substantive meaning (e.g., “I'm sorry, but there are no new arrivals on different platforms”). In practice, we collect common features in false positives (e.g., \"sorry\", \"apologize\") and filter responses accordingly to exclude them.\n\n+ Pass rate assesses if answers are provided and meaningful. While some scores may be inflated, removing false positives limits this impact. Additionally, in win rate evaluation, we compare the quality of our model reasoning answers with expert annotations, using ChatGPT to assess whether each answer solves the problem. Therefore, combining the results of pass rate and win rate can make a reliable evaluation of the model.\n\nWe will improve the pass rate evaluation based on your suggestions and include relevant results in the revised manuscript.\n\n------\n\n**Q4**: \"Dataset Settings: Are the test scenarios and dataset settings identical to those used by ToolBench, or were they resampled, and what scale?\"\n\n**A4**: Thanks for this suggestion, we will add details in the revised version. Our test datasets come from ToolBench's open source on GitHub. Test scenarios and dataset settings are identical to theirs: G1-Ins., G1-Tool, G1-Cat., G2-Ins., G2-Cat. each contain 200 samples, G3-Ins. has 100 samples.\n\n------\n\n**Q5**: \"ChatGPT Version Specification: The specific version of ChatGPT used in the baseline is not mentioned. Knowing the exact version is essential for replicating the results and understanding the capabilities of the baseline model.\"\n\n**A5**: We will specify the ChatGPT version in the revised version. \n\n+ For evaluation of win rates, we used the gpt-3.5-turbo-16k-0613 version. \n+ The pass rate results we reported for ChatGPT in baselines are computed using test answers provided by ToolBench's open source on GitHub.\n\n------\n\n**Q6**: \"ToolLLaMA Results Usage: How were the ToolLLaMA results used in your comparison? Were the scores used directly, or were the answers recalculated? Clarifying this would help in understanding the validity of the comparison.\"\n\n**A6**: Sorry for our unclear expression, which may have caused difficulties in understanding. We recalculated the scores of ToolLLaMA using test answers provided by ToolBench's open source on GitHub. Due to space limitations, please check A3 to Reviewer z4Vk for more discussions.\n\n------\n\n**Q7**: \"Same amount of post-SFT compute on different dataset: Would you consider conducting additional comparative experiments to address this? if not, could you provide specific reasons why you believe this is not a concern?\"\n\n**A7**: Since DPO is a preference optimization method, we can't use only successful paths from the ToolBench for preference training. And because there are no other preference datasets developed based on ToolBench, we can not directly compare how dataset choice impacts the performance, but we will try to construct preference datasets in other ways for comparative experiments in the revised version.\n\nCurrently, we can provide a preliminary comparative experiment to show that the performance improvement should not come from increased compute resources. Specifically, we created a new SFT dataset using the same original samples and size as ToolPreference, and performed one epoch SFT training based on the same post-SFT model. The pass rates we observed were lower than those of the DPO fine-tuned model and most even falling below the post-SFT model: \n\n|                | G1-Ins. | G1-Tool | G1-Cat. | G2-Ins. | G2-Cat. | G3-Ins. | Average |\n| -------------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| LLaMA with SFT | 0.47    | 0.53    | 0.72    | 0.48    | 0.63    | 0.35    | 0.53    |\n| TP-LLaMA       | 0.55    | 0.65    | 0.80    | 0.62    | 0.67    | 0.61    | 0.65    |\n| New Model      | 0.34    | 0.54    | 0.60    | 0.39    | 0.52    | 0.18    | 0.43    |\n\nThis may be a result of overfitting. Clearly, simply increasing the amount of post-SFT compute does not enhance the model's learning of the data.\n\n------", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hPVppFospQ", "reviewer": "Reviewer_9yS8", "summary": "Empowering Large Language Models (LLMs) with external tools has been considered as an important research direction to extend LLMs' task scope. However, during the tool utilization of LLMs, it usually relied on successful tool execution path. Therefore, how to instruct LLMs to employ successful paths is an important element for using tools. To address this point, this paper introduces a ToolPreference dataset that collect successful and failure paths from ToolBench, and then deploy a preference-based fine-tuning based on the proposed ToolPreference dataset. Experimental results demonstrate that the proposed method can achieve better performance in using tools.", "questions": "1. Can the proposed method be generalized to other benchmarks? It seems that the proposed method is just applied to ToolBench.\n2. Did you involve any human evaluation? Pass rate and win rate are just two objective metrics to evaluate whether the model can execute the answer path and whether the solution path is preferred. It cannot guarantee the obtained answer is optimal. Therefore, human evaluation is also required in this task.", "limitations": "Please see my above comments.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Strengths**\n\nThe contribution of this paper can be summarized as two parts:\n1. This paper introduces a ToolPreference dataset.\n2. This paper introduces a fine-tuning method by using DPO algorithm based on the proposed ToolPreference dataset.", "weaknesses": "**Weaknesses**\n1. The contribution of this paper is limited. Its main contribution is based on DFSDT and then use DPO method for fine-tuning. Compared with ETO, it just replaces the feedback from environments by using ToolPreference. \n2. In the proposed ToolPreference dataset, it just uses successful path and failure path as preferred and dispreferred data. However, I think such a selection is too easy as model will never choose any dispreferred data. I think directly training on these preferrence data can also produce better performance. \n3. Besides, Tool using also require model to be generalized to other scenarios. Is the proposed model also suitable for other scenarios?\n4. The problems in tool usage usually involve multiple aspects, mentioned by ToolQA [a], like parameters incorrection, choose incorrect tools. I think besides to train tool preference, how to enable LLMs to successfully use Tools is more important. And authors can also analyze whether the proposed method can alleviate these errors when using Tools.\n\n\n**Minor issues**\n1. I didn't see any dataset about ToolPreference in supplemental materials. I think if this dataset is considered as a contribution, it should also be reviewed. \n\n[a] A Dataset for LLM Question Answering with External Tools", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the constructive reviews! We will revise our paper accordingly.\n\n------\n\n**Q1**: \"In the proposed ToolPreference dataset, it just uses successful path and failure path as preferred and dispreferred data. However, I think such a selection is too easy as model will never choose any dispreferred data. I think directly training on these preferrence data can also produce better performance.\"\n\n**A1**: Thanks for the comment. Below we give an explanation of our approach.\n\n+ The \"preferred\" and \"dispreferred\" data in ToolPreference actually reflect human preferences based on whether the reasoning path is successful, rather than the model's tool preferences. Because humans always prefer paths that lead to successful reasoning and more correct answers. By constructing the preference dataset this way, we aim to align human and model preferences.\n+ Additionally, expert trajectory data for our SFT training comes from the same DFSDT samples as ToolPreference, meaning our SFT training includes all preferred data. However, our results indicate that training only on preferred data leads to poorer performance than utilizing ToolPreference for DPO training.\n\n------\n\n**Q2**: \"Tool using also require model to be generalized to other scenarios. Is the proposed model also suitable for other scenarios?\"\n\n**A2**: Thanks for the question. ToolBench contains a wide range of tool usage scenarios across 49 categories, such as news, music, and advertising. Some categories are reserved for testing and do not appear during training. So we have evaluated our model's generalization ability based on ToolBench in our paper, specifically from three perspectives:\n\n1. Instruction Scenario: Evaluating the model's reasoning for new instructions.\n2. Tool Scenario: Assessing the model's application of unseen tools within known categories.\n3. Category Scenario: Evaluating the model's ability to use tools from new categories.\n\nSince different tool categories focus on different task scenarios, the results in our paper, particularly from the Category Scenario, offer valuable insights into our model's generalization performance, which outperformed baseline models. \n\n------\n\n**Q3**: \"The problems in tool usage usually involve multiple aspects, mentioned by ToolQA [a], like parameters incorrection, choose incorrect tools. I think besides to train tool preference, how to enable LLMs to successfully use Tools is more important. And authors can also analyze whether the proposed method can alleviate these errors when using Tools.\"\n\n**A3**: Thanks for the comment. We provide a brief explanation below.\n\n+ Firstly, it is important to clarify that our ToolPreference dataset construction relies on correct tool and parameter selection, as expert trajectories that succeed within a limited path length are more likely to select the right tools and input correct parameters.\n+ As explained in **A1**, ToolPreference reflects human preferences rather than a mere model preference toward tools. Our goal in optimizing based on ToolPreference is to enable models to make better choices more rapidly while minimizing unnecessary exploration.\n+ Our experimental results indicate that our method effectively reduces errors in tool selection and parameter generation, as our model achieves higher rates with fewer reasoning steps. However, quantitatively analyzing these error reductions is challenging due to the need for extensive human annotation, and we will try to include this in our revised version.\n\n------\n\n**Q4**: \"I didn't see any dataset about ToolPreference in supplemental materials. I think if this dataset is considered as a contribution, it should also be reviewed.\"\n\n**A4**: Thanks for the suggestion. We will make our dataset public after the paper is accepted.\n\n------\n\n**Q5**: \"Can the proposed method be generalized to other benchmarks? It seems that the proposed method is just applied to ToolBench.\"\n\n**A5**: Our method can be applied to other benchmarks. Here’s a brief overview of our process:\n\n1. Based on the DFSDT framework, using GPT to generate annotated reasoning trees for training samples of any benchmark.\n2. Using the expert annotations from the previous step, create a preference dataset through our proposed method.\n3. Perform SFT and DPO training based on pre-train models. SFT uses instruction-tuning data from successful paths in the expert annotations, and DPO employs the preference dataset.\n\nThe above process is universal and does not depend on Toolbench, so it can be applied to other benchmarks. Furthermore, we plan to integrate an API retriever in future work, allowing TP-LLaMA to be used across benchmarks without additional training.\n\n------\n\n**Q6**: \"Did you involve any human evaluation? Pass rate and win rate are just two objective metrics to evaluate whether the model can execute the answer path and whether the solution path is preferred. It cannot guarantee the obtained answer is optimal.\"\n\n**A6**: Thanks for your suggestion, we will enhance our evaluation in the revised version accordingly. Due to the variety of APIs in ToolBench, human evaluation is complex and time-consuming. And some queries yield open answers, making it challenging for human annotators to determine the optimal answer. So we didn't involve large human evaluation but used the common method of taking ChatGPT as an expert to evaluate. \n\nIn addition, it should be emphasized that the win rate measures both the preference for solution paths and final answers. Specifically, in win rate evaluation, when both our model's answer and expert annotations are not empty, we use ChatGPT to assess whether each answer solves the problem before comparing reasoning paths. Qin et al., 2024 [1] have shown that the win rate correlates well with human preferences, making it a valid metric for our assessment.\n\n------\n\n[1] Qin, Yujia, et al. Toolllm: Facilitating large language models to master 16000+ real-world APIs. *ArXiv preprint*, arXiv:2307.16789v1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3NCodc60gQ", "reviewer": "Reviewer_jBLW", "summary": "This paper proposes a novel framework for improving tool-augmented large language models by incorporating insights from errors in inference trees. It constructs preference data from expert reasoning trajectories and finetune LLMs accordingly. Experiments demonstrate that the finetuned model significantly outperforms baselines in terms of pass rate and win rate, exhibiting better generalization capabilities with unseen APIs.", "questions": "1. See weakness 1, I'd like the authors to provide some case studies on different task scenarios.\n\n2. I'd like the authors to provide a more detailed explanation of their design choices for training hyperparameters:\n\n(a) Could the authors elaborate on how they determine the optimal number of expert trajectories sampled for SFT training (11142 in the paper)?\n\n(b) Similarly, how was the number of preference pairs used for DPO training decided (8202 in the paper)?\n\nTo strengthen the analysis, it would be valuable to include ablation studies investigating the impact of these two hyperparameters on model performance.\n\n3. Aside from ToolBench, there are some other benchmarks that can test the tool reasoning capability, e.g. APIBench. Does TP-LLaMA still perform better than the other baselines?", "limitations": "None.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is straightforward. The construction of a preference dataset makes full use of the failed attempts and can improve data utilization.\n2. The ablation experiments are sound. The paper conducts ablation experiments on multiple open source LLMs and the results all demonstrate the superiority of the proposed method.", "weaknesses": "1. The paper can benefit from some case studies. Though numbers suggest that TP-LLaMA can master tool-usage instructions better and exhibits stronger generalization capabilities, case studies comparing the reasoning trajectories between TP-LLaMA and LLaMA w/ SFT on different task scenarios is needed to prove the effectiveness of DPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the constructive reviews! We will revise our paper accordingly.\n\n------\n\n**Q1**: \"The paper can benefit from some case studies. Though numbers suggest that TP-LLaMA can master tool-usage instructions better and exhibits stronger generalization capabilities, case studies comparing the reasoning trajectories between TP-LLaMA and LLaMA w/ SFT on different task scenarios is needed to prove the effectiveness of DPO.\"\n\n**A1**: Thanks for your suggestion. Here we provide an example of the G3-Ins. scenario. First, we give the query and related API documentation:\n\n+ **query**: I'm organizing a film festival and I'm looking for award-winning films. Can you search for videos related to 'award-winning' on Vimeo? Additionally, fetch the related people in the 'film festival' category to invite them as judges. Finally, provide me with a streaming link for a YouTube video with the ID 'UxxajLWwzqY'.\n\n+ **related API documentation** (parameter information is omitted):\n\n  + **name:** getrelatedchannels_for_vimeo, **description**: Get Related Channels.\n  + **name:** searchvideos_for_vimeo, **description**: Search for videos.\n  + **name**: getrelatedpeople_for_vimeo, **description**: Get a list of related people for a category.\n  + **name**: download_stream_for_ytstream_download_youtube_videos, **description**: Stream or download info.\n  + **name**: Finish, **description**: ... \n\n  (Ps: during our experiment, the first three APIs are temporarily not accessible.)\n\nNext, we briefly present the reasoning trajectories of TP-LLaMA and LLaMA w/ SFT respectively. Note that we limit a path to contain at most 12 nodes (ie. at most 4 API calls), and a tree contains at most 200 nodes. We require the model to answer as much content of the query as possible.\n\n+ The reasoning tree of TP-LLaMA includes 34 nodes and 5 paths. The successful path is: \n\n  1. call [searchvideos_for_vimeo] with arguments {\"format\": \"json\", \"query\": \"award-winning\"}, get response \"...The consumer key passed was not valid...\", \n\n  2. call [getrelatedchannels_for_vimeo] with arguments {\"category\": \"film festival\", \"format\": \"json\"}, get response \"...The consumer key passed was not valid...\", \n\n  3. call [download_stream_for_ytstream_download_youtube_videos] with arguments {\"is_id\": \"UxxajLWwzqY\"}, get response \"...'id': 'UxxajLWwzqY', 'title': 'Icona Pop - I Love It (feat. Charli XCX) [OFFICIAL VIDEO]', ...\",\n\n  4. call [Finish] with arguments {\"return_type\": \"give_answer\", \"final_answer\": \"The streaming link for the YouTube video with ID UxxajLWwzqY is [link]...\"}, get response \"successfully giving the final answer\". \n\n     (Ps: here we use [link] to replace the real URL in the final answer, because external links are not allowed in rebuttal.)\n\n+ The reasoning tree of LLaMA w/ SFT includes 49 nodes and 8 paths. There is no successful path. Here are some of the failed paths:\n\n  1. searchvideos_for_vimeo -> searchvideos_for_vimeo -> searchvideos_for_vimeo -> getrelatedpeople_for_vimeo\n\n  2. searchvideos_for_vimeo -> searchvideos_for_vimeo -> searchvideos_for_vimeo -> getrelatedchannels_for_vimeo \n\n  3. searchvideos_for_vimeo -> searchvideos_for_vimeo -> searchvideos_for_vimeo -> Finish with give up and restart\n\n  4. ...... (omit due to space limit)\n\n  Because the model repeatedly tried inaccessible APIs (possibly using different arguments) without accessing [download_stream_for_ytstream_download_youtube_videos] first, it failed to give part of the answer in the end.\n\nBased on this case, we can see that TP-LLaMA can reduce unnecessary attempts of failed paths, explore uncalled APIs, and obtain accessible answers faster. Due to the space limit, we will discuss case studies in other scenarios in the revised version.\n\n------\n\n**Q2**: \"I'd like the authors to provide a more detailed explanation of their design choices for training hyperparameters... To strengthen the analysis, it would be valuable to include ablation studies investigating the impact of these two hyperparameters on model performance.\"\n\n**A2**: Thanks for the question. We provide an explanation of our design choices regarding training hyperparameters:\n\n+ We initially filtered 42,192 reasoning trees with branching nodes from Toolbench, which led to 69,393 DPO samples and 184,816 SFT samples (as each query may correspond to multiple samples).\n\n+ After allocating a small part as a validation set, we sampled training sets of different sizes based on these samples. The sampling methods include: by query and by sample. For sampling by query, the SFT training set ranged from 2,500 to 10,000 queries, and the DPO training set ranged from 5,000 to 32,192 queries, yielding nine combinations. For sampling by sample, SFT varied from 10,000 to 183,561, and DPO from 10,000 to 68,951, yielding seven combinations.\n+ We tested most of these training sets and observed that increasing the size may lead to decreased model performance in scenarios requiring generalization, such as G3_instruction (e.g., with settings {SFT: 44,412, DPO: 41,226}, the pass rate dropped to 0.36), possibly due to overfitting. Consequently, we selected the set {SFT: 11,142, DPO: 8,202} finally. We will include the ablation studies in the revised version.\n\n------\n\n**Q3**: \"Aside from ToolBench, there are some other benchmarks that can test the tool reasoning capability, e.g. APIBench. Does TP-LLaMA still perform better than the other baselines?\"\n\n**A3**: Thanks for the question. Currently, TP-LLaMA relies on relevant API documentation provided for each query in instructions, which is not available in most benchmarks like APIBench. So we are temporarily unable to directly test its performance on other benchmarks. In our future work, we will add an API retriever to replace the pre-annotated relevant APIs, and then test the effect of TP-LLaMA on other benchmarks.\n\n------", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a method to enhance tool-augmented large language models (LLMs) by leveraging preference data extracted from both successful and failed trajectories. It introduces the ToolPreference dataset, constructed using preference data from the ToolBench dataset. A base LLM is then fine-tuned using DPO on this new dataset, leading to TP-Llama. Experiments demonstrate superior performance of TP-Llama compared to baselines across various metrics and test scenarios.

The paper is strong in the sense that it proposes an intuitively straightforward technical method for an important problem, and experiments demonstrate that the method is effective. The paper is also clearly written. 

Reviewers raised several important questions regarding the discussion of related work, method design, and experiments. The authors provided extensive amounts of clarification and some new results in their rebuttal, resolving most of the key concerns. 

I strongly recommend that the authors update the camera-ready based on the author-reviewer discussion, clarifying their technical design and adding new results and case studies.
Decision: Accept (poster)