Title: Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Models

Tool Usage

Direct Preference Optimization

Tree of Thought
Abstract: Tool-augmented large language models (LLMs) leverage tools, often in the form of APIs, to improve their reasoning capabilities on complex tasks. This enables them to act as intelligent agents interacting with the real world. The recently introduced ToolLLaMA model by Qin et al. [2023] utilizes the depth-first search-based decision tree (DFSDT) mechanism for multi-step reasoning with $16000+$ real-world APIs, effectively enhancing the performance of tool-augmented LLMs compared to traditional chain reasoning mechanisms. However, their approach only employs successful paths from decision trees (also called inference trees) for supervised fine-tuning (SFT), missing out on the potential learning opportunities from failed paths. Inspired by this, we propose an inference trajectory optimization framework based on preference learning to address this limitation. We first introduce a novel method for constructing step-wise preference data from tree-like expert trajectories, which leverages the previously ignored failed explorations in the decision trees. In the subsequent training phase, we first fine-tune the LLM with successful tool-usage expert trajectories and then apply direct preference optimization (DPO) with the preference data to update the LLM's policy, resulting in our ToolPrefer-LLaMA (TP-LLaMA) model. This approach not only enhances the utilization of original expert data but also broadens the learning space of the model. Our experiments demonstrate that by obtaining insights from errors in inference trees, TP-LLaMA significantly outperforms the baselines across almost all test scenarios by a large margin and exhibits better generalization capabilities with unseen APIs. At the same time, TP-LLaMA has also demonstrated superior reasoning efficiency compared to the baselines, making it more suitable for complex tool-usage reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for improving tool-augmented large language models by incorporating insights from errors in inference trees. It constructs preference data from expert reasoning trajectories and finetune LLMs accordingly. Experiments demonstrate that the finetuned model significantly outperforms baselines in terms of pass rate and win rate, exhibiting better generalization capabilities with unseen APIs.\n\nSTRENGTHS:\n1. The proposed method is straightforward. The construction of a preference dataset makes full use of the failed attempts and can improve data utilization.\n2. The ablation experiments are sound. The paper conducts ablation experiments on multiple open source LLMs and the results all demonstrate the superiority of the proposed method.\n\nWEAKNESSES:\n1. The paper can benefit from some case studies. Though numbers suggest that TP-LLaMA can master tool-usage instructions better and exhibits stronger generalization capabilities, case studies comparing the reasoning trajectories between TP-LLaMA and LLaMA w/ SFT on different task scenarios is needed to prove the effectiveness of DPO.\n\nQUESTIONS:\n1. See weakness 1, I'd like the authors to provide some case studies on different task scenarios.\n\n2. I'd like the authors to provide a more detailed explanation of their design choices for training hyperparameters:\n\n(a) Could the authors elaborate on how they determine the optimal number of expert trajectories sampled for SFT training (11142 in the paper)?\n\n(b) Similarly, how was the number of preference pairs used for DPO training decided (8202 in the paper)?\n\nTo strengthen the analysis, it would be valuable to include ablation studies investigating the impact of these two hyperparameters on model performance.\n\n3. Aside from ToolBench, there are some other benchmarks that can test the tool reasoning capability, e.g. APIBench. Does TP-LLaMA still perform better than the other baselines?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEmpowering Large Language Models (LLMs) with external tools has been considered as an important research direction to extend LLMs' task scope. However, during the tool utilization of LLMs, it usually relied on successful tool execution path. Therefore, how to instruct LLMs to employ successful paths is an important element for using tools. To address this point, this paper introduces a ToolPreference dataset that collect successful and failure paths from ToolBench, and then deploy a preference-based fine-tuning based on the proposed ToolPreference dataset. Experimental results demonstrate that the proposed method can achieve better performance in using tools.\n\nSTRENGTHS:\n**Strengths**\n\nThe contribution of this paper can be summarized as two parts:\n1. This paper introduces a ToolPreference dataset.\n2. This paper introduces a fine-tuning method by using DPO algorithm based on the proposed ToolPreference dataset.\n\nWEAKNESSES:\n**Weaknesses**\n1. The contribution of this paper is limited. Its main contribution is based on DFSDT and then use DPO method for fine-tuning. Compared with ETO, it just replaces the feedback from environments by using ToolPreference. \n2. In the proposed ToolPreference dataset, it just uses successful path and failure path as preferred and dispreferred data. However, I think such a selection is too easy as model will never choose any dispreferred data. I think directly training on these preferrence data can also produce better performance. \n3. Besides, Tool using also require model to be generalized to other scenarios. Is the proposed model also suitable for other scenarios?\n4. The problems in tool usage usually involve multiple aspects, mentioned by ToolQA [a], like parameters incorrection, choose incorrect tools. I think besides to train tool preference, how to enable LLMs to successfully use Tools is more important. And authors can also analyze whether the proposed method can alleviate these errors when using Tools.\n\n\n**Minor issues**\n1. I didn't see any dataset about ToolPreference in supplemental materials. I think if this dataset is considered as a contribution, it should also be reviewed. \n\n[a] A Dataset for LLM Question Answering with External Tools\n\nQUESTIONS:\n1. Can the proposed method be generalized to other benchmarks? It seems that the proposed method is just applied to ToolBench.\n2. Did you involve any human evaluation? Pass rate and win rate are just two objective metrics to evaluate whether the model can execute the answer path and whether the solution path is preferred. It cannot guarantee the obtained answer is optimal. Therefore, human evaluation is also required in this task.\n\nLIMITATIONS:\nPlease see my above comments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors aimed to enhance the tool-augmented language model (LLM) by efficiently learning from the failed explorations that previous works had overlooked. Step-wise preference data were extracted by pairing the branch nodes with the corresponding child nodes from the successful path in the tool-using decision tree. A base model was fine-tuned using DPO on this new dataset, in addition to SFT using only successful trajectories, resulting in ToolPrefer-LLaMA. Evaluation metrics included pass rate and win rate, and assessments across six test scenarios demonstrated superior performance compared to baselines, along with a reduction in inference steps. Furthermore, the authors conducted an ablation study on several base models to validate the effectiveness and generalizability of their method.\n\nSTRENGTHS:\n- This paper proposes a method to leverage the inherent preferences embedded in failed trajectories from existing benchmark datasets, without imposing too much computational overhead. It improves data utilization.\n- The ablation study demonstrates the efficiency and portability of this method across different base models, indicating its potential for generalizability and immediate applicability to existing systems.\n- This method is practical and logically intuitive, making it likely to be adopted in future tool-augmented training processes.\n\nWEAKNESSES:\n- Handling of \"Error\" Nodes in the ToolBench Dataset\nThe proposed preference pair generation method does not account for \"error\" nodes in the original ToolBench dataset (see Figure 4. from https://arxiv.org/abs/2307.16789). This oversight could lead to the generation of incorrect preference pairs, such as `0-9-12(api1_failed) > 0-9-10(api1_succeed)`, potentially skewing the results.\n\n- Pass Rate Metric\nThe authors mentioned in Section 4.1 that the pass rate evaluation does not verify whether the given answer meets the requirements of the instruction. This could result in inflated scores. Additionally, the criteria for filtering out false positives and the details of the filtering process are not specified, which is a significant oversight.\n\n- Evaluation Settings\nThe paper introduces six test scenarios that closely resemble those described in ToolBench. However, the detailed testing dataset settings are not specified. Clarification is needed on whether these datasets are the same as those used by ToolBench or if they have been resampled.\nThe specific version of ChatGPT used in the baseline is not mentioned. Knowing the exact version (e.g., gpt-3.5-turbo-0125, gpt-4-0613, gpt-4-0125-preview) is essential for replicating the results and understanding the capabilities of the baseline model.\n\n- ToolLLaMA Results Report\nThe paper states that ToolLLaMA results were \"directly used,\" but it is unclear how this was done. If the scores were used directly, they do not match those in the referred publication. If the answers were used for score recalculation, the discrepancy in test performance, such as the pass rate for ToolLLaMA in G3-Ins. (0.17) versus I3-Inst. (66.0%), needs to be explained.\n\n- Data benefit or more post-SFT training\nThe DPO fine-tuned models on the ToolPreference dataset outperform both commercial models and open-source ToolLLaMA, as well as in-house SFT-trained LLaMA. This improvement might stem more from increased computational investment than from the dataset's inherent benefits. To clarify this, experiments should compare DPO fine-tuned SFT-trained models using different datasets: ToolPreference versus successful paths from the ToolBench dataset. This approach would maintain consistent compute resources and DPO methodology, allowing for a direct comparison of how dataset choice impacts performance.\n\nQUESTIONS:\n- Computation Time Clarification: Could you specify the computation time on the mentioned infrastructure? \n\n- Error Node Handling: How do you plan to address the issue of \"error\" nodes in the preference pair generation method? Could you provide examples or a detailed explanation?\n\n- Evaluation Criteria: Can you clarify the criteria for what is considered a false positive and the details of the filtering process? This would help in understanding the robustness of the evaluation.\n\n- Dataset Settings: Are the test scenarios and dataset settings identical to those used by ToolBench, or were they resampled, and what scale? This information is necessary for reproducibility and context.\n\n- ChatGPT Version Specification: Which specific version of ChatGPT was used in the baseline? This is important for replicating the results.\n\n- ToolLLaMA Results Usage: How were the ToolLLaMA results used in your comparison? Were the scores used directly, or were the answers recalculated? Clarifying this would help in understanding the validity of the comparison.\n\n- Same amount of post-SFT compute on different dataset: Would you consider conducting additional comparative experiments to address this? if not, could you provide specific reasons why you believe this is not a concern?\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}

comments: SUMMARY:
This paper proposes a method to collect preference data from decision trees, making use of the failed trajectories which are previously ignored.  Based on ToolBench (https://arxiv.org/abs/2307.16789) and the decision trees provided in the original dataset, a preference dataset, ToolPreference, is built with the proposed method. DPO is used to optimize LLaMA-2-7B on the new dataset, and the resulting model, TP-Llama, achieves state-of-the-art performance on ToolBench.

STRENGTHS:
1. Though the idea of the paper is not completely novel, it's the first to apply such ideas on this specific domain. 
2. The paper is well written. The results are sound and solid. 
3. The paper clearly expresses its own contribution. Yet, it still needs to clarify a few issues regarding the results of baseline methods.
4. The paper is of moderate significance. It provides a new preference dataset for LLM agents.

WEAKNESSES:
1. The paper is related to the works on process reward, e.g., https://arxiv.org/abs/2310.10080, https://arxiv.org/pdf/2312.08935, yet does not discuss about it in the related works section.
2. Two different methods of preference dataset construction are discussed in the paper (path-wise and step-wise), but there is no result supporting that discussion.

QUESTIONS:
The result of ToolLLaMA reported in this paper is different from the original paper. l feel like it's due to different definitions of "pass rate", but can you give an explanation on this?

LIMITATIONS:
The paper lacks a thorough discussion on its limitations. For example, it seems that the performance of the method highly depends on the quality of the decision trees, while "ETO" mentioned in the paper doesn't.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-22