Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees

Authors: sijia  chen
yibo  wang
yi-feng  wu
qing-guo  chen
zhao  xu
weihua  luo
kaifu  zhang
lijun  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:42:11

Abstract: Tool-augmented large language models (LLMs) leverage tools, often in the form of APIs, to improve their reasoning capabilities on complex tasks. This enables them to act as intelligent agents interacting with the real world. The recently introduced ToolLLaMA model by Qin et al. [2023] utilizes the depth-first search-based decision tree (DFSDT) mechanism for multi-step reasoning with $16000+$ real-world APIs, effectively enhancing the performance of tool-augmented LLMs compared to traditional chain reasoning mechanisms. However, their approach only employs successful paths from decision trees (also called inference trees) for supervised fine-tuning (SFT), missing out on the potential learning opportunities from failed paths. Inspired by this, we propose an inference trajectory optimization framework based on preference learning to address this limitation. We first introduce a novel method for constructing preference data from tree-like expert trajectories, which leverages the previously ignored failed explorations in the decision trees. Specifically, we generate a step-wise preference dataset, ToolPreference, from the ToolBench dataset for tool learning. In the subsequent training phase, we first fine-tune the LLM with successful tool-usage expert trajectories and then apply direct preference optimization (DPO) with ToolPreference to update the LLM's policy, resulting in our ToolPrefer-LLaMA (TP-LLaMA) model. This approach not only enhances the utilization of original expert data but also broadens the learning space of the model. Our experiments demonstrate that by obtaining insights from errors in inference trees, TP-LLaMA significantly outperforms the baselines across almost all test scenarios by a large margin and exhibits better generalization capabilities with unseen APIs. At the same time, TP-LLaMA has also demonstrated superior reasoning efficiency compared to the baselines, making it more suitable for complex tool-usage reasoning tasks.

Main paper: 
 introduction: 
 In recent years, large language models (LLMs) have exhibited impressive capabilities in various areas, including language understanding and generation, multi-modal content learning and reasoning, and even embodied intelligence task processing [Brown et al., 2020 , Zeng et al., 2023 , Alayrac et al., Figure 1: Our Inference Trajectory Optimization Framework. 2022 , Li et al., 2023 , Lu et al., 2024 , Cao et al., 2024a ,b, Mazzaglia et al., 2024] . Despite these notable strengths, these models still face significant challenges, such as a lack of access to real-time information [Komeili et al., 2021] and difficulties in precise mathematical tasks [Patel et al., 2021 , Lu et al., 2023b] . The development of tool-augmented LLMs tackles these challenges by enabling LLMs to interact with external tools (often in the form of APIs), significantly enhancing their capabilities. This advancement allows LLMs to serve as efficient intermediaries between users and a large ecosystem of applications. Notably, tool-augmented LLMs based on the ChatGPT [Brown et al., 2020] and GPT-4 [Achiam et al., 2023] have achieved outstanding results by using few-shot or zero-shot prompts to activate the LLM's inherent tool-usage abilities [Deng et al., 2023 , Lin et al., 2024 , Lu et al., 2023a] . Despite this progress, some studies demonstrate that open-source LLMs still exhibit a significant gap in their capacity to utilize external tools compared to state-of-the-art (SOTA) closed-source models like GPT-4 [Liu et al., 2024 , Wang et al., 2024b] . To bridge this gap, aligning these open-source LLMs with tool-usage downstream tasks is essential.Currently, most efforts to align open-source LLMs with tool-usage downstream tasks rely on supervised fine-tuning (SFT) with expert trajectory datasets, which trains LLMs to learn strategies for subsequent actions based on previous actions and observations [Patil et al., 2023 , Schick et al., 2023] . Early studies in this field typically have limitations such as a restricted variety of APIs, the reliance on single-tool scenarios, and the use of simple reasoning methods [Wei et al., 2022 , Yao et al., 2023 , Patil et al., 2023] . The recent work by Qin et al. [2023] , which focuses on the scene of LLM's multi-step reasoning with external tools, solves the above limitations. They introduce an instruction tuning dataset called ToolBench, which includes over 16, 000 real-world APIs and various realistic instructions, along with expert trajectories annotated by ChatGPT based on a depthfirst search-based decision tree (DFSDT) reasoning mechanism. They then perform SFT training on LLaMA with this dataset to create the ToolLLaMA model, which shows remarkable performance. However, ToolLLaMA's training is still based on expert behavior cloning, potentially limiting exploration of the target space and leading to suboptimal strategies. Additionally, although their expert trajectories are structured as DFS trees, only successful trajectories are utilized in the SFT training, which neglects valuable insights from failed attempts and results in low data utilization.As the saying goes, "a fall into a pit, a gain in your wit", effective human learning involves not only drawing lessons from success but also from failures. Inspired by this, we propose a new inference trajectory optimization framework for developing tool-augmented LLMs as illustrated in Figure 1 , which enhances the tool learning process by incorporating previously ignored failure exploration information via preference learning. Specifically, using the tree-like expert trajectories from ToolBench [Qin et al., 2023] , we first parse each pair of branch nodes along the successful trajectory in the decision tree into a preference sample pair, thereby constructing a novel step-wise tool-usage preference dataset named ToolPreference. Subsequently, after conducting SFT training on the pre-trained LLM with successful trajectories, we employ the direct preference optimization (DPO) method [Rafailov et al., 2023] with the ToolPreference dataset to further align the LLM with tool-usage downstream tasks, and thus obtain our model, named ToolPrefer-LLaMA (TP-LLaMA). Our strategy improves the utilization of expert data and simultaneously broadens the learning space.Our experiments are conducted on the test tasks from ToolBench. To evaluate the performance, we adopt two metrics: the pass rate, which measures the probability of the model successfully providing an answer within limited steps; and the win rate, which quantifies the likelihood that the evaluator will prefer the model's responses. From the experiment results, we have the following findings:• Across all test scenarios, TP-LLaMA consistently surpasses ToolLLaMA and other baselines, with an average pass rate improvement of at least 12% and a win rate that outperforms nearly all other models by an average of 4%. These results demonstrate that learning from failed attempts can significantly enhance the decision-making ability of LLMs. Additionally, our model shows superior generalization to unseen APIs.• Efficiency experiments show that our model requires an average of only 22.62 steps for DFSDT inference, compared to 32.06 steps for the SFT model. This enhancement stems from our method's ability to avoid unnecessary branch explorations in DFSDT reasoning.• Our ablation experiments verify that the effectiveness of our preference dataset and inference trajectory optimization framework has nothing to do with the base model itself. Better results can still be obtained after replacing the base model with Mistral-7B [Jiang et al., 2023] , Qwen1.5-7B [Bai et al., 2023] , and Gemma-7B [Team et al., 2024] .In summary, this work aims to enhance the performance of LLMs on multi-step reasoning with external tools by integrating insights from errors in tree-like reasoning trajectories and employing step-wise preference pairs for preference learning. Our key contributions include: (i) The careful design and construction of a new tool-usage preference dataset -ToolPreference, providing a valuable resource for the community; (ii) The proposal of using ToolPreference to optimize the LLM's tool-usage ability with the DPO training, along with the development of the TP-LLaMA model;(iii) Extensive experimental evaluations and in-depth analyses of the TP-LLaMA model, providing evidence of its effectiveness and validating its performance across various dimensions. 
related work: 
 In this section, we briefly review recent progress on tool-augmented large language models and the development of preference learning.Tool-augmented large language models. Over the past year, extensive research has been dedicated to developing tool-augmented LLMs, which exhibit improved reasoning abilities across various tasks by integrating external tools [Patil et al., 2023 , Lin et al., 2024 , Lu et al., 2023a , Schick et al., 2023] . The workflow for tool-augmented LLMs typically involves four key stages: task planning, tool selection, tool calls, and response generation. Early research mainly uses few-shot or zero-shot prompting methods to activate LLM's inherent tool-usage abilities, often employing GPT as the LLM agent to manage several external tools such as AI models, web search, Python, and more [Shen et al., 2023b , Lu et al., 2023a] . While GPT performs well with external tools, opensource LLMs like LLaMA often struggle with direct tool usage and need additional task alignment. Therefore, subsequent research often utilizes instruction-tuning datasets annotated with tool calls to train open-source models, enhancing their ability to use tools. At the same time, these studies continue to explore a wider range of tools and scenarios [Schick et al., 2023 , Patil et al., 2023] .One of the most comprehensive efforts in this field is by Qin et al. [2023] . They initially collect 16, 464 real-world APIs across 49 categories, then utilize ChatGPT to automatically generate instructions that could invoke these APIs, and annotate expert trajectories to create a high-quality instruction tuning dataset named ToolBench. During the annotation, they employ the DFSDT reasoning mechanism to broaden the search space and enhance reasoning capabilities. By fine-tuning LLaMA on ToolBench, they develop ToolLLaMA, which has shown a compelling capability to handle both single-tool and complex multi-tool instructions.Preference learning Preference learning uses human preferences from feedback data to assist decision-making. The earliest research in this field employs specially designed neural networks to help agents optimize action choices based on structured human guidance in programming languages [Maclin and Shavlik, 1996] . Subsequent studies shift focus to learning from numerical rewards provided by humans and performing reinforcement learning based on the prediction of these rewards [Knox, 2012 , Isbell et al., 2006 , Knox and Stone, 2008] . This approach finds applications in areas like embodied intelligence [Pilarski et al., 2011, Suay and Chernova, 2011] and dialogue systems [El Asri et al., 2016] . The introduction of preference-based reinforcement learning marks a key milestone in the field, which uses qualitative human preferences, often in the form of rankings, to guide the optimization of policy models [Akrour et al., 2011 , Cheng et al., 2011] . Following this idea, Christiano et al. [2017] propose reinforcement learning from human feedback (RLHF), where a reward model is derived from human preferences to enhance reinforcement learning. This technique is later extended to natural language generation tasks [Kreutzer et al., 2018 , Ziegler et al., 2019] , advancing the integration of preference learning with LLM research [Ouyang et al., 2022] . 
preliminaries: 
 In this section, we start by formally defining the problem setup, and then we introduce key knowledge about preference learning methods, which is relevant to our approach.In this work, we use an iterative paradigm for the LLM's multi-step reasoning with external tools, where the model selects each tool call based on the previous response, rather than pre-planning all tool calls at the start. Formally, we define it as a state transition process. The environment consists of a set of available tools T = {T 1 , T 2 , . . . , T n }, each with specific functionalities accessible through API calls. The task begins with an initial instruction I, usually consisting of a user query and a system prompt. At each reasoning step t, the LLM processes the current context S t , defined as:S t = {I, H t }where H t is the previous history, which includes the API decisions made{A1 , • • • , A t-1 }, and the API responses received {R 1 , • • • , R t-1 }: H t = {(A 1 , R 1 ), . . . , (A t-1 , R t-1 )}The LLM then generates an action decision A t based on this context, specifying both the tool T i ∈ T to use and its parameters. After the tool executes, the response R t is generated and used to update the context. The reasoning process continues until the LLM determines that the task is complete and produces a final output O to answer the original query or gives up the task.Preference learning has gained growing attention in LLM research. Its main goal is to optimize model outputs based on human (or expert) preferences, better aligning the model's behavior with the expectations of real-world applications. Assume there is a preference dataset defined as D = {(x (i) , y(i) w , y (i) l )} i=1,...,|D|, where x (i) denotes the i-th prompt, y(i)w and y (i) l denote the corresponding preferred and dispreferred output respectively. Moreover, the notation y w ≻ y l | x indicates that y w is preferred than y l for prompt x. Because the true distribution of human preferences is inaccessible, we assume it is generated by a latent reward model r * (x, y), where higher rewards indicate stronger preferences. Then, according to Rafailov et al. [2023] , the human preference distribution p * can be captured by the Bradley-Terry (BT) model [Bradley and Terry, 1952] :p * (y 1 ≻ y 2 | x) = exp (r * (x, y 1 )) exp (r * (x, y 1 )) + exp (r * (x, y 2 )) = σ (r * (x, y 1 ) -r * (x, y 2 )) ,where σ is the logistic function. Obviously, we can estimate the parameters of the reward model via maximum likelihood estimation (equivalent to minimizing the negative log-likelihood.):EQUATIONwhere r ϕ is a parameterized reward model.To optimize the inference trajectories of LLM based on human preference, a popular method in recent LLM research is Reinforcement Learning from Human Feedback (RLHF) [Christiano et al., 2017 , Ouyang et al., 2022] . In the RL phase of this method, the optimization goal isEQUATIONwhere r ϕ is the reward model learned before, π θ is the policy model we need to optimize, β is a weighting parameter that controls the deviation from the base reference policy model π ref (i.e., the LLM after SFT training). In practice, π θ is also initialized to the LLM after SFT. RLHF will use reinforcement learning methods (such as PPO [Schulman et al., 2017] ) to optimize (2) and update the LLM's strategy, with r ϕ (x, y) providing reward feedback. Additionally, some research in multistep reasoning scenarios trains process reward models to evaluate each step instead of the entire output [Ma et al., 2023 , Wang et al., 2024a] . However, RLHF incurs significant computational overhead, long training times, and potential instability [Shen et al., 2023a , Rafailov et al., 2023] , making it less suitable for general tool-usage tasks.Therefore, we choose a more convenient and faster approach that can also effectively align the model's preferences -Direct Preference Optimization (DPO) [Rafailov et al., 2023] , which eliminates the need to learn the reward model and directly uses preference data to optimize the LLM. Specifically, the optimal solution of (2) can be written asEQUATION)whereZ(x) = ∑ y π ref (y | x) exp ( 1 β r(x, y))is the partition function [Rafailov et al., 2023] . We rearrange (3) to express r(x, y) in terms of π r and π ref :EQUATION)Substitute ( 4) into (1), we can finally get the learning goal of DPOL DPO (π θ , π ref ) = -E (x,yw,y l )∼D [ logσ ( βlog π θ (y w | x) π ref (y w | x) -βlog π θ (y l | x) π ref (y l | x) )] ,where π θ is a parametrized policy that we need to optimize. As a result, the optimization objective of DPO avoids additional learning of the reward model and the RL process while maximizing the final reward, which is more suitable for our general tool-usage scenarios. 
our method: 
 In this section, we introduce our inference trajectory optimization framework, beginning with an overview of the framework, followed by a description of the preference data construction process.Our framework is composed of two key stages: dataset construction and training. In the dataset construction stage, we create a tool-usage preference dataset, named ToolPreference, which is derived from the tree-like expert trajectories in Toolbench [Qin et al., 2023] . The specific process for constructing this dataset will be detailed in section 4.2.Remark 1 It is important to emphasize that our preference data construction approach is not limited to Toolbench and can be adapted to any tree-structured multi-step instruction-tuning dataset, offering flexibility for various applications.In the training stage, we first perform SFT training on a pre-trained LLM using a resampled version of the instruction-tuning data from Toolbench (refer to Remark 2 for the resampling process). SFT training has been commonly adopted in previous research to enhance tool-augmented LLMs. However, mere cloning expert behavior through SFT is insufficient, as this method fails to adequately explore the environment, and can result in suboptimal strategies. To address this, after the SFT training, we further perform DPO training on the model with the ToolPreference dataset. This additional preference learning enhances the models reasoning capabilities when interacting with external tools and aligns its decision-making preferences with human preferences.Before introducing our preference data construction method, we first describe the dataset structure and expert trajectory format used in ToolBench [Qin et al., 2023] .• Dataset structure. ToolBench consists of two main components: API information data and instruction tuning data. The API information data is sourced from RapidAPI Hub3 and includes 3,451 tools across 49 categories, with a total of 16,464 APIs (as each tool can have multiple APIs). Each API entry contains detailed information such as the name, description, HTTP method, URL, required and optional parameters, and executable code snippets for API calls. This comprehensive data enables LLMs to perform few-shot inference with effective API calls. The instruction-tuning data includes various single-tool or multi-tool instructions as well as corresponding annotated expert trajectories, generated in a self-instruction method by ChatGPT. • Expert trajectory format. While traditional LLMs often use sequential reasoning methods like chain-of-thought (CoT) [Wei et al., 2022] , which follow a single path to completion, ToolBench adopts a depth-first search (DFS) reasoning approach. As shown in the left half of Figure 2 , expert trajectories in ToolBench are structured as decision trees with each tree node representing an LLM decision about an API call. Based on the tree structure, ToolBench implements DFS reasoning using two techniques. First, it defines two additional functions: one is "Finish with final answer", where the LLM concludes it has gathered enough API responses to provide a correct answer and terminate the reasoning process, and the other is "Finish with giving up", where the LLM feels unable to proceed with the task, abandons the current path and returns to a previous node. Second, diversity prompts are used to expand the search space. When expanding child nodes, the LLM will be prompted with information about previously explored child nodes of the same layer, and explicitly encouraged to generate different ones. Consequently, the LLM is allowed to either abandon the current path and restart from a previous step or proceed along a more promising path, exploring until an answer is reached or the node limit is reached.We employ the second release of ToolBench4 , which includes over 120, 000 expert trajectories. Our approach is designed based on the motivation of improving data utilization. Although the treelike expert trajectories in ToolBench extensively search the answer space, only successful paths are used in their training, neglecting valuable insights from failure paths. To address this, we extract preference decision pairs from each tree-like expert trajectory. After filtering out trajectories without failed exploration branches, we explore two different construction methods:• Path-wise means using an entire success path and an entire failure path in the same decision tree to form a preference pair. As shown in the upper right part of Figure 2 , ⟨0, 9, 12, 13, 14, 15⟩ is the success path of the decision tree, and ⟨0, 1, 2⟩, ⟨0, 3, 4, 5, 6⟩, ⟨0, 3, 7, 8⟩, ⟨0, 9, 10, 11⟩ are 4 failure paths, so their Cartesian product can constitute a path-wise preference dataset, where ≻ denotes the left part is preferred than the right part. • Step-wise means using each branch node along the success path in the tree and its corresponding pair of child nodes (which must contain a child node on the success path) to construct a preference pair. As shown in the lower right part of Figure 2 , ⟨0, 9, 12, 13, 14, 15⟩ is the success path of the decision tree, while 0 and 9 are nodes with branches along the success path. Therefore, ⟨0, 9⟩ ≻ ⟨0, 1⟩, ⟨0, 9⟩ ≻ ⟨0, 3⟩, and ⟨0, 9, 12⟩ ≻ ⟨0, 9, 10⟩ can respectively form a preference pair.Although it is intuitive and common to use path-wise preference samples, this approach is not wellsuited to our task scenario. Theoretically, it may limit the model to only differentiate between correct and incorrect final responses to specific instructions, resulting in poor generalization with unseen instructions or tools. From an engineering perspective, learning preferences for an entire path at once is inconsistent with the model's reasoning mechanism of inferring the next API call based on the response of the previous API execution each time, which makes it inherently unsuitable for the DFSDT reasoning mechanism.In contrast, the step-wise design highlights the differences between each reasoning step, providing the model with more fine-grained process supervision. Theoretically, this method can better adjust the model's reasoning process and enhance its generalization performance. It is also a more suitable fit for implementation within the DFSDT reasoning framework. Consequently, we create 69, 393 pairs of preference samples from ToolBench in a step-wise manner. Each pair is formatted Remark 2 To ensure a rigorous comparison between models with and without preference learning in subsequent experiments, we also do not directly use the instruction-tuning dataset provided by Toolbench during the SFT phase. Instead, we filter out expert trajectories lacking failed exploration branches, as these could not be parsed into preference samples, and resampled the remaining data to create our SFT training set. This ensures the training data distribution remains consistent across models, regardless of whether preference learning is applied.Our ToolPreference is currently one of the few large-scale preference data sets suitable for a wide range of tasks in the tool-usage domain. We believe that it provides valuable reference data for future research on improving the tool-usage capabilities of LLMs. 
experiments: 
 In this section, we investigate the performance of our inference trajectory optimization framework. We first introduce the experiments settings in Section 5.1. We then present the main results in Section 5.2, the efficiency experiments in Section 5.3, and the ablation experiments in Section 5.4.Evaluation metrics. Since our model uses APIs from the online platform RapidAPI Hub, there may be changes such as version updates or service termination over time, making it difficult to provide a fixed solution path for each test instruction. Following Qin et al. [2023] , we use pass rate and win rate as evaluation metrics in our experiments. The pass rate represents the proportion that the model successfully gives answers within a certain number of reasoning actions (set to 200 in our experiment) 5 . Specifically, a sample is considered passed if the reasoning trajectory finishes with the "Finish with final answer" API call. Additionally, we filter out samples that yield meaningless answers using a predefined set of feature keywords, such as "sorry", "apologize", etc. The win rate measures the likelihood that the solution path provided by the test model is preferred over the Baselines. We compare our model with several models without preference learning. Among them, we select the expert model ChatGPT and OpenAI Text-Davinci-003 (Davinci for short) as baselines.In addition, we also show the results of ToolLLaMA and the model trained by SFT using our resampled SFT training set (LLaMA with SFT for short) for comparison. Note that all models here are combined with DFSDT for inference. In addition, regarding the ToolLLaMA results, we directly use the reasoning answers of ToolLLaMA on test sets provided by ToolBench's GitHub repository to calculate pass rates and win rates.We employ LLaMA-2-7B as the base model of our training framework and finally obtain our model, named ToolPrefer-LLaMA (TP-LLaMA). The context length of LLaMA-2-7B is extended to 8192 tokens to accommodate our tool-usage reasoning tasks. The main results are shown in Table 1 . We have the following important observations:• TP-LLaMA significantly outperforms LLMs without preference learning in terms of pass rate, demonstrating the best performance across all six test scenarios, with an average improvement of over 12% compared to models not optimized using preference data.• Regarding win rate, TP-LLaMA also exhibits competitive performance, just 3% below ToolLLaMA in the G1-Cat. scenario, while achieving the best results in all other scenarios.• Furthermore, TP-LLaMA shows strong performance in more challenging task scenarios such as G2-Cat., G2-Ins., and G3-Ins., maintaining effectiveness similar to that in simpler tasks. Notably, in the G3-Ins. scenario, TP-LLaMA's pass rate increased by over 26%, proving that our DPO training process using preference data significantly enhances the model's ability to handle complex multi-tool tasks.Although we use the provided reasoning answers of ToolLLaMA from ToolBench's GitHub repository to calculate its rates, the results indeed differ from those reported in their paper [Qin et al., 2023] . This may be due to the reasoning answers version not matching the one used in their paper or differences in the evaluation environment settings. However, it's important to emphasize that our results remain valid and reliable. We apply consistent settings across all models, so their relative differences are meaningful. Overall, our experimental results indicate that through preference learning, TP-LLaMA can master various tool-usage instructions better and exhibits stronger generalization capabilities to unseen tools, categories, and instructions.We also evaluate the inference efficiency of TP-LLaMA on six test scenarios and employ the average number of DFSDT inference steps required for samples that ended with the Finish function as the metric. From Table 2 , we can find that LLaMA with SFT requires an average of 32.06 steps for reasoning, while our TP-LLaMA only requires an average of 22.62 steps of reasoning in all test scenarios, with an improvement of 29.44%. These results clearly demonstrate that the inference efficiency of TP-LLaMA is remarkably superior to that of the model trained only with success trajectories. This advantage arises from our step-wise preference data, which allows the model to identify the most optimal decisions at each step of reasoning through DPO training. As a result, the model avoids the exploration of unnecessary sub-optimal branches in the decision tree, thereby increasing reasoning speed and efficiency.In the ablation experiments, to verify the effectiveness of our framework, we further replace LLaMA-2-7B with other base models, including Mistral-7B, Qwen1.5-7B, and Gemma-7B. The results are shown in Table 3 and Table 4 .From Table 3 , no matter which base model is used, training on preference data can always bring gains to the performance of the model, which verifies the model-independent effectiveness of our framework. Specifically, in terms of pass rates, models that have learned from expert errors improve by at least 8% on average compared to those that only receive training on success trajectory information. Similarly, in terms of win rates, models with insights from preference data generally outperform those without preference learning. Table 4 further confirms that our method significantly improves model inference efficiency by a large margin, up to an average of 33.35%. prove that TP-LLaMA significantly outperforms the baseline models in nearly all test scenarios by learning from single-step errors in inference trees. TP-LLaMA also exhibits superior generalization capabilities and efficiency. Furthermore, ablation experiments confirm the model-independent effectiveness of our framework. In future work, we will try to explore tool-learning research with more complex, human-like reasoning mechanisms, and incorporate preference learning for further optimization. We also aim to extend our research to multimodal scenarios to evaluate the broader effectiveness of our approach.Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint, arXiv:1909.08593, 2019.Output { " from " : " assistant " , " value " : " Thought In this section, we first supplement the experiments on another defined pass rate in Appendix B.1.Then we conduct a qualitative analysis of our method in Appendix B.2, giving a specific case study.In the second version of ToolBench [Qin et al., 2024] , a new pass rate definition using GPT-assisted evaluation is introduced. For each query, it first checks if the "Finish with Final Answer" API is called; if not, it's considered a failure. If it is, GPT evaluates whether the answer resolves the query.If successful, it is marked as a pass. If not, GPT further assesses whether the query is solvable with the available APIs. If it isn't, it is still considered a pass; otherwise, it is marked as a failure. Due to the ToolBench API server being offline temporarily, we use our own RapidAPI accounts to access APIs for evaluation experiments with the new pass rate definition. To maintain consistency in API status, we utilize ToolLLaMA's open-source model7 to perform reasoning on the test sets, instead of reusing the reasoning answers from its GitHub repository. Similarly, we re-run tests for other models using our RapidAPI accounts. We employ gpt-3.5-turbo-16k and gpt-3.5-turbo-1106 as GPT evaluators, with the results shown in Table 5 .First, TP-LLaMA still outperforms the models without preference learning, further validating the effectiveness of our method. However, the absolute pass rates depend heavily on the specific GPT version. We observe notable differences in preferences and consistency across GPT versions. After repeating the evaluation of each sample 7 times, we find that gpt-3.5-turbo-1106 is more likely to mark a sample as passed, while gpt-3.5-turbo-16k tends to judge it as not passed. This difference mainly stems from how each version assesses whether a query is solvable. Additionally, gpt-3.5turbo-16k shows greater consistency across the 7 evaluations, meaning it is more likely to produce the same inference repeatedly. This highlights the importance of selecting the appropriate GPT version for evaluation, as relative scores may be more meaningful than absolute ones.Furthermore, we observe that the gap between TP-LLaMA and ToolLLaMA narrows under the new evaluation. We believe this is due to two factors: (1) The models have different preferences formed during their respective training processes. TP-LLaMA tends to avoid giving up on reasoning and attempts partial answers, whereas ToolLLaMA is more likely to abandon a task entirely, leading to a complete failure. However, this gap narrows due to the use of GPT to evaluate whether the task is solvable.(2) During this supplementary experiment, our RapidAPI accounts have access limits (some APIs even can only be accessed 5 times per month per account), reducing the number of valid samples in the test sets. This particularly affects complex multi-tool reasoning tasks, where TP-LLaMA usually excels, making its performance gains appear smaller.Additionally, the results we report for ToolLLaMA are still lower than those in Qin et al. [2024] , likely due to shifts in the distribution of real-world APIs, which may make certain test samples unsolvable. Moreover, some features ToolLLaMA learned from past environments may not fullyCase Study: Reasoning Trajectories from TP-LLaMAThe reasoning tree of TP-LLaMA includes 5 paths and we show the successful path below.Step 1 call [searchvideos_for_vimeo] with arguments {"format": "json", "query": "award-winning"}, get the response "...The consumer key passed was not valid..."Step 2 call [getrelatedchannels_for_vimeo] with arguments {"category": "film festival", "format": "json"}, get the response "...The consumer key passed was not valid..."Step 3 call [download_stream_for_ytstream_download_youtube_videos] with arguments {"is_id": "UxxajLWwzqY"}, get the response "...'id': 'UxxajLWwzqY', 'title': 'Icona Pop -I Love It (feat. Charli XCX) [OFFICIAL VIDEO]', ..."Step 4 call [Finish] with arguments {"return_type": "give_answer", "final_answer": "The streaming link for the YouTube video with ID UxxajLWwzqY is [link]..."}, get the response "successfully giving the final answer"Case Study: Reasoning Trajectories from LLaMA with SFTThe reasoning tree of LLaMA with SFT includes 8 paths. There is no successful path since the model finally gives up. All the paths are listed below (the numbers in brackets represent the unique id of each node). We observe that because LLaMA with SFT repeatedly tries inaccessible APIs (possibly using different arguments) without first accessing the accessible YouTube API, it finally mistakenly chooses to give up reasoning and is unable to give a partial answer. In contrast, TP-LLaMA successfully calls the YouTube API to provide the best possible answer while using fewer inference steps.EQUATIONQuestion: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Our data and code are currently undergoing the companys privacy review.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We have specified all the training and test details in Section 5.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not reported because it would be too computationally in our experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have provide sufficient information on the computer resources needed to reproduce the experiments in Section 5.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research we conduct in our papers always complies in all respects with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We think there is no societal impact of our work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper does not carry a high risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: In section 5, we have cited the original paper. We have also stated the version of the asset is used with a URL.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://rapidapi.com/hubhttps://github.com/OpenBMB/ToolBench.gitDuring our experiment process, we noticed that ToolBench has been updated with a revised definition of pass rate[Qin et al., 2024]. The definition we use in the main text follows the original version, while the revised definition and corresponding results will be provided in Appendix B.1The ChatGPT version we used in the experiments in the main text is gpt-3.5-turbo-16k.https://huggingface.co/ToolBench/ToolLLaMA-2-7b-v2 
 
 Appendix:  


Decision: This paper presents a method to enhance tool-augmented large language models (LLMs) by leveraging preference data extracted from both successful and failed trajectories. It introduces the ToolPreference dataset, constructed using preference data from the ToolBench dataset. A base LLM is then fine-tuned using DPO on this new dataset, leading to TP-Llama. Experiments demonstrate superior performance of TP-Llama compared to baselines across various metrics and test scenarios.

The paper is strong in the sense that it proposes an intuitively straightforward technical method for an important problem, and experiments demonstrate that the method is effective. The paper is also clearly written. 

Reviewers raised several important questions regarding the discussion of related work, method design, and experiments. The authors provided extensive amounts of clarification and some new results in their rebuttal, resolving most of the key concerns. 

I strongly recommend that the authors update the camera-ready based on the author-reviewer discussion, clarifying their technical design and adding new results and case studies.

Reviews: review: paper_summary: paper_summary: This paper proposes a method to collect preference data from decision trees, making use of the failed trajectories which are previously ignored.  Based on ToolBench (https://arxiv.org/abs/2307.16789) and the decision trees provided in the original dataset, a preference dataset, ToolPreference, is built with the proposed method. DPO is used to optimize LLaMA-2-7B on the new dataset, and the resulting model, TP-Llama, achieves state-of-the-art performance on ToolBench.
strength_weakness: strength_weakness: 1. The paper is related to the works on process reward, e.g., https://arxiv.org/abs/2310.10080, https://arxiv.org/pdf/2312.08935, yet does not discuss about it in the related works section.
2. Two different methods of preference dataset construction are discussed in the paper (path-wise and step-wise), but there is no result supporting that discussion.
questions: questions: The result of ToolLLaMA reported in this paper is different from the original paper. l feel like it's due to different definitions of "pass rate", but can you give an explanation on this?
limitations: limitations: The paper lacks a thorough discussion on its limitations. For example, it seems that the performance of the method highly depends on the quality of the decision trees, while "ETO" mentioned in the paper doesn't.
paper_summary: paper_summary: The authors aimed to enhance the tool-augmented language model (LLM) by efficiently learning from the failed explorations that previous works had overlooked. Step-wise preference data were extracted by pairing the branch nodes with the corresponding child nodes from the successful path in the tool-using decision tree. A base model was fine-tuned using DPO on this new dataset, in addition to SFT using only successful trajectories, resulting in ToolPrefer-LLaMA. Evaluation metrics included pass rate and win rate, and assessments across six test scenarios demonstrated superior performance compared to baselines, along with a reduction in inference steps. Furthermore, the authors conducted an ablation study on several base models to validate the effectiveness and generalizability of their method.
strength_weakness: strength_weakness: - Handling of "Error" Nodes in the ToolBench Dataset
The proposed preference pair generation method does not account for "error" nodes in the original ToolBench dataset (see Figure 4. from https://arxiv.org/abs/2307.16789). This oversight could lead to the generation of incorrect preference pairs, such as `0-9-12(api1_failed) > 0-9-10(api1_succeed)`, potentially skewing the results.

- Pass Rate Metric
The authors mentioned in Section 4.1 that the pass rate evaluation does not verify whether the given answer meets the requirements of the instruction. This could result in inflated scores. Additionally, the criteria for filtering out false positives and the details of the filtering process are not specified, which is a significant oversight.

- Evaluation Settings
The paper introduces six test scenarios that closely resemble those described in ToolBench. However, the detailed testing dataset settings are not specified. Clarification is needed on whether these datasets are the same as those used by ToolBench or if they have been resampled.
The specific version of ChatGPT used in the baseline is not mentioned. Knowing the exact version (e.g., gpt-3.5-turbo-0125, gpt-4-0613, gpt-4-0125-preview) is essential for replicating the results and understanding the capabilities of the baseline model.

- ToolLLaMA Results Report
The paper states that ToolLLaMA results were "directly used," but it is unclear how this was done. If the scores were used directly, they do not match those in the referred publication. If the answers were used for score recalculation, the discrepancy in test performance, such as the pass rate for ToolLLaMA in G3-Ins. (0.17) versus I3-Inst. (66.0%), needs to be explained.

- Data benefit or more post-SFT training
The DPO fine-tuned models on the ToolPreference dataset outperform both commercial models and open-source ToolLLaMA, as well as in-house SFT-trained LLaMA. This improvement might stem more from increased computational investment than from the dataset's inherent benefits. To clarify this, experiments should compare DPO fine-tuned SFT-trained models using different datasets: ToolPreference versus successful paths from the ToolBench dataset. This approach would maintain consistent compute resources and DPO methodology, allowing for a direct comparison of how dataset choice impacts performance.
questions: questions: - Computation Time Clarification: Could you specify the computation time on the mentioned infrastructure? 

- Error Node Handling: How do you plan to address the issue of "error" nodes in the preference pair generation method? Could you provide examples or a detailed explanation?

- Evaluation Criteria: Can you clarify the criteria for what is considered a false positive and the details of the filtering process? This would help in understanding the robustness of the evaluation.

- Dataset Settings: Are the test scenarios and dataset settings identical to those used by ToolBench, or were they resampled, and what scale? This information is necessary for reproducibility and context.

- ChatGPT Version Specification: Which specific version of ChatGPT was used in the baseline? This is important for replicating the results.

- ToolLLaMA Results Usage: How were the ToolLLaMA results used in your comparison? Were the scores used directly, or were the answers recalculated? Clarifying this would help in understanding the validity of the comparison.

- Same amount of post-SFT compute on different dataset: Would you consider conducting additional comparative experiments to address this? if not, could you provide specific reasons why you believe this is not a concern?
limitations: limitations: See above
paper_summary: paper_summary: Empowering Large Language Models (LLMs) with external tools has been considered as an important research direction to extend LLMs' task scope. However, during the tool utilization of LLMs, it usually relied on successful tool execution path. Therefore, how to instruct LLMs to employ successful paths is an important element for using tools. To address this point, this paper introduces a ToolPreference dataset that collect successful and failure paths from ToolBench, and then deploy a preference-based fine-tuning based on the proposed ToolPreference dataset. Experimental results demonstrate that the proposed method can achieve better performance in using tools.
strength_weakness: strength_weakness: **Weaknesses**
1. The contribution of this paper is limited. Its main contribution is based on DFSDT and then use DPO method for fine-tuning. Compared with ETO, it just replaces the feedback from environments by using ToolPreference. 
2. In the proposed ToolPreference dataset, it just uses successful path and failure path as preferred and dispreferred data. However, I think such a selection is too easy as model will never choose any dispreferred data. I think directly training on these preferrence data can also produce better performance. 
3. Besides, Tool using also require model to be generalized to other scenarios. Is the proposed model also suitable for other scenarios?
4. The problems in tool usage usually involve multiple aspects, mentioned by ToolQA [a], like parameters incorrection, choose incorrect tools. I think besides to train tool preference, how to enable LLMs to successfully use Tools is more important. And authors can also analyze whether the proposed method can alleviate these errors when using Tools.


**Minor issues**
1. I didn't see any dataset about ToolPreference in supplemental materials. I think if this dataset is considered as a contribution, it should also be reviewed. 

[a] A Dataset for LLM Question Answering with External Tools
questions: questions: 1. Can the proposed method be generalized to other benchmarks? It seems that the proposed method is just applied to ToolBench.
2. Did you involve any human evaluation? Pass rate and win rate are just two objective metrics to evaluate whether the model can execute the answer path and whether the solution path is preferred. It cannot guarantee the obtained answer is optimal. Therefore, human evaluation is also required in this task.
limitations: limitations: Please see my above comments.
paper_summary: paper_summary: This paper proposes a novel framework for improving tool-augmented large language models by incorporating insights from errors in inference trees. It constructs preference data from expert reasoning trajectories and finetune LLMs accordingly. Experiments demonstrate that the finetuned model significantly outperforms baselines in terms of pass rate and win rate, exhibiting better generalization capabilities with unseen APIs.
strength_weakness: strength_weakness: 1. The paper can benefit from some case studies. Though numbers suggest that TP-LLaMA can master tool-usage instructions better and exhibits stronger generalization capabilities, case studies comparing the reasoning trajectories between TP-LLaMA and LLaMA w/ SFT on different task scenarios is needed to prove the effectiveness of DPO.
questions: questions: 1. See weakness 1, I'd like the authors to provide some case studies on different task scenarios.

2. I'd like the authors to provide a more detailed explanation of their design choices for training hyperparameters:

(a) Could the authors elaborate on how they determine the optimal number of expert trajectories sampled for SFT training (11142 in the paper)?

(b) Similarly, how was the number of preference pairs used for DPO training decided (8202 in the paper)?

To strengthen the analysis, it would be valuable to include ablation studies investigating the impact of these two hyperparameters on model performance.

3. Aside from ToolBench, there are some other benchmarks that can test the tool reasoning capability, e.g. APIBench. Does TP-LLaMA still perform better than the other baselines?
limitations: limitations: None.

Comments: comment: Many thanks for your constructive feedback and support! Currently, given the complexity of multi-tool usage tasks, we adopt the common approach employed by many existing studies, using GPT as a surrogate for human experts to derive ground truth solutions.  

We fully understand your concerns about the dataset's alignment with human preferences. In our revised version, we plan to try several strategies to address this issue, including introducing a human validation and correction step during the construction of the ToolPreference dataset, and incorporating a human evaluation process for the model's solution paths in our experiments. 

Thanks again for your thoughtful review!
I didn't have any concern about method but still have some questions about dataset construction. 

If we want to apply this method other benchmarks, it also requires us to use DFSDT to get a trajectory tree and then find a solution. Such a solution is to find a way to execute API calls not from the perspective of human preference. And please note, ToolBench is also just a instruction dataset not from real-world scenarios or human data. So, it just represents a solution to find executable tools, not human perference. And it seems you did not conduct any human verification on ToolPreference data (line 187 - line 194). Therefore, I think the proposed method can help model to better find a solution to execute tools but not from human preference. As this dataset is not released and I cannot check this dataset, authors also don't provide any human evaluation in the response. So I will remain some concerns about the dataset and whether it can really align human preference. But I can increase score to 5 as I didn't have any issues about method.
Thank you for your positive feedback and for raising the score. We really appreciate your suggestion and will add the case studies in our revised version as you recommended.
Good work. My questions have been addressed. I hope the authors can add case studies in the appendix in the revised version. I therefore vote for acceptance of this paper.
Thanks so much for your thoughtful feedback and for raising the scores. We will incorporate your insights into the revised manuscript and address the concerns you've mentioned.
Thank you very much for your kind reply! We will improve our paper according to your constructive reviews.
Thank the authors for the detailed response. It addresses all my concerns. Looking forward to your update on A3.
Thank the authors for their detailed responses. The suggested revised content should be sufficient to address most of my questions. Therefore, I have raised the scores in anticipation of reading the updated manuscript.

According to A2, there are some "imputable" nodes in certain trajectories. It seems to me that removing them before pooling the preferred nodes could lead to a better preference dataset, as this would eliminate some avoidable errors from the failed paths that may be involved in your preference dataset.

It is interesting that the ToolLLaMA results are not reproducible. In this case, I think the writing in this part should clarify your replication efforts and include some explanations, as you gave to z4Vk, for future researchers' reference.

Solo re-SFT on the post-SFT model could address this concern to some extent, but not fully. Comparative experiments ideally should employ DPO and roughly the same amount of compute but on different datasets. I am looking forward to your revised section on this issue.