ToolBridge: An Open-Source Dataset to Equip LLMs with External Tool Capabilities

Authors: Zhenchao Jin, Mengchen Liu, Dongdong Chen, Lingting Zhu, Yunsheng Li, Lequan Yu
Primary area: datasets and benchmarks
Keywords: LLM, datasets, tool use, open source

Abstract:
Through the integration of external tools, large language models (LLMs) such as GPT-4o and Llama 3.1 significantly expand their functional capabilities, evolving from elementary conversational agents to general-purpose assistants. We contend that the primary drivers of these advancements are the quality and diversity of the training data. However, the existing LLMs with external tool integration provide only limited transparency regarding their datasets and data collection approaches, which has led to the initiation of this study. Specifically, in this work, we endeavor to present a detailed exposition of the methodology for constructing datasets that facilitate LLMs in effectively learning how to utilize external tools and make this process available to the public through the introduction of ToolBridge. ToolBridge proposes to leverage a collection of general open-access datasets as its raw dataset pool and incorporates a series of strategies to identify the appropriate data entries for external tool API insertions. By supervised fine-tuning (SFT) on these curated data entries, LLMs can invoke external tools in appropriate contexts to boost their predictive accuracy, particularly for essential functions including factual retrieval, data processing and numerical computation. Our experiments meticulously isolate model architectures and training configurations, zeroing in exclusively on the role of data. The experimental results indicate that LLMs trained on ToolBridge exhibit consistent performance gains on both standard benchmarks and custom evaluation datasets. All associated code and data will be released as open source, promoting transparency and facilitating the broader community to explore methodologies for equipping LLMs with external tools capabilities.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730256237620

thread_type:
official_review

<thread_object>
creator: reviewer_qapt

<content>
summary:
This paper introduces ToolBridge, a collection of general open-source datasets designed to equip LLMs with effective external tool usage through a carefully structured pipeline. Experimental results demonstrate that fine-tuning on ToolBridge can enhance LLM performance across many benchmarks, including new benchmarks introduced in this paper for numerical calculation, factual retrieval, and data processing.

strengths:
- this paper curates a collection of datasets through a carefully designed pipeline, and I really appreciate that they made it open source. I believe it can help with the research community
- I believe RandomQA will also be useful for further research

weaknesses:
- The Figure 2 needs some revision, it's not informative, suggest including the processing details in each section
- Only llama models are used in experiments. Since llama is employed in the data curation pipeline, I suggest including other open or closed models with different sizes

questions:
- What are the source datasets? Could you provide a brief description for each?
- I think RandomQA is a really good benchmark for numerical computation, but less data processing?
- Regarding the FACT, why do you prefer the current approach over using existing factual datasets and systematically processing them, for example, into a Python dictionary? I believe those datasets are typically more carefully constructed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730307347232

thread_type:
official_review

<thread_object>
creator: reviewer_nxyv

<content>
summary:
This paper introduces ToolBridge, a dataset intended to enhance large language models' (LLMs) capacity for effectively employing external tools, and addressing gaps in model training. ToolBridge integrates various open-source SFT data, compiling an extensive dataset to train LLMs for some tasks that can be enhanced by using Python. The authors highlight the need for high-quality, transparent datasets specifically designed for tool integration and propose that they will release the ToolBridge dataset and data process pipeline.

strengths:
1. The paper describes a pipeline for generating tool-using data and presents a dataset (though direct access to the dataset is not provided here) to address gaps in large language model tool-usage capabilities.
2. The structured data-filtering approach introduced in this paper effectively enhances dataset quality and improves the ability of tool using(function calling).
3. The authors report that, following training on the ToolBridge dataset, the model shows improved performance in mathematical, fact and RandomQA benchmarks.

weaknesses:
1. This paper restricts external tool integration to Python APIs. Although the dataset includes 63 Python packages, this selection may be limited compared to other tool-using datasets, such as ToolBench.
2. A comparison with other tool-using datasets, like ToolBench, may further clarify ToolBridge's distinct contributions.
3. The paper could benefit from a more detailed explanation of the fact-checking process. While it states that the model can request external web pages using request packages, it remains unclear whether the URLs are generated by the model, predefined, or retrieved through other methods.
4. This paper does not evaluate the factual retrieval abilities of LLM against established benchmarks but uses a self-construct dataset. There are some existing benchmarks, such as TruthfulQA or HaluEval, to assess the model's accuracy and reliability in factual responses, which may be more convincing.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680854194

thread_type:
official_review

<thread_object>
creator: reviewer_5jrh

<content>
summary:
This paper introduces ToolBridge, a new dataset for finetuning models to run Python code to support the completion of data processing, numerical computation, and factual retrieval tasks. To construct this dataset, the authors begin with a pool of 13 supervised finetuning datasets. Then, they use Llama 3 70B to filter for dataset examples that can make use of external tool use, and use GPT-4o-mini to create the Python code that invokes external tools. They show that by finetuning on ToolBridge, models can improve task performance.

strengths:
The authors do a good job of motivating the purpose of their contribution, which emphasizes the open-source nature of their work juxtaposed against existing work.

weaknesses:
Some details for assessing the quality of this dataset and the quality of their data curation pipeline are missing. For instance, how well does Llama3-70B select “valuable” data entries (3.2) (e.g. false positive, false negative rates)? Did you experiment with other prompts for selecting data, and if so, how did you decide on your final prompt? Generally, it would be good to get some sense of what it means for a data entry to be “valuable” or suitable for tool use, and how you selected the few-shot examples in the prompts you provide. 

The authors’ data and experiments are essentially a case study of model distillation. That is, they use potentially stronger models (Llama-3-70B, GPT-4o-mini) to create data for weaker ones (Llama-3-8B, Llama-2-7B). It would be more interesting to see how data generated by strong models can improve their own performance. 

The authors could also provide more information on the composition of their dataset. They provide some examples of Python packages involved, but which ones are most common and what does the distribution look like? Approximately how many dataset examples pertain to the three key tasks of numerical calculations, factual retrieval, and data processing? It may also be useful to disaggregate RandomQA, which the authors create themselves, based on task type to show how ToolBridge may boost performance for some tasks more so than others. 

The qualitative results section in this paper is pretty lacking. It essentially just shows three examples of prompts to which Llama3-8B trained on ToolBridge answers correctly, along with other model’s responses (which are sometimes correct and sometimes not). There is no qualitative analysis present in this section; it’s just three input/output examples. These examples don’t really add much to the paper, and the models included in Figure 3 also seem a bit random, because these models are not mentioned at all among the paper’s quantitative results. To revise, I suggest that the authors do one of the following: 
- Actually conduct some qualitative analysis of results. For example, the authors could characterize what kinds of examples Llama3-8B trained on ToolBridge does best on and which ones it tends to struggle on (like the bullet list of deficiencies outlined in lines 472-481). To do this, I suggest the authors look up how to do “qualitative coding”. 
- Do not include a “qualitative results” section at all, and instead replace Figure 3 with a figure of example inputs/outputs from the four models experimented with in 4.3 and/or 4.4. 
- Keep Figure 3, but also include quantitative results in Tables 6-9 for Llama3.1-70B-IT, GPT-4, GPT-4o, and Gemma2-27B-IT. 

More minor comments: 
- Line 187: “Llama3” -> “Llama3-70B”
- Line 208: “using the Llama3-70B” -> “using Llama3-70B”
- Line 359: “four baseline models: the base model of Llama2-7B and Llama3-8B” This says “four” but names two models. 
- Line 484: “llama3-8B” -> “Llama3-8B” (to match model naming elsewhere in the paper)

questions:
Why do you run Llama 3 70B filtering twice (once in line 187, and once in line 206)? Why not just run it once over all datasets? 

How did you conduct your review of SFT datasets to finalize the ones you include in Table 1? That is, how did you ensure that it was sufficiently “comprehensive” as you say? 

For the random sampling done in line 187, how many examples did you sample to make $\mathcal{S}_i$ for each dataset? 

Line 196: What is $N$ and how did you determine what this value would be for each dataset?

Are there any substantive differences between the three different batches of the FACT benchmark? That is, why is performance reported for each batch instead of aggregating performance across them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719591798

thread_type:
official_review

<thread_object>
creator: reviewer_x8cw

<content>
summary:
The paper presents a novel post-training dataset, ToolBridge, aimed at enhancing the external tool usage capabilities of LLMs.

1. The authors consolidate a broad set of open-access datasets to form a comprehensive dataset pool. This pool undergoes a systematic process that includes sampling, code generation, and consistency validation to identify entries suitable for tool invocation.

2. The ToolBridge pipeline features three key phases—selection, conversion, and filtering. Entries are reformatted, enriched with Python code snippets by GPT-4 for tool calls, and subsequently validated to ensure the consistency of execution results with the corresponding responses.

3. The authors conducted experiments using both standard math and QA benchmarks as well as custom benchmarks. Results show consistent improvements over the base model after SFT with ToolBridge.

strengths:
1. The authors effectively assemble a large and diverse collection of existing datasets. Drawing from various sources, they create a robust dataset pool that enhances the relevance and generalizability of ToolBridge for training language models.

2. The pipeline for data conversion, validation, and tool invocation is meticulously designed, reflecting significant effort. The authors provide transparency in each step of the process, from dataset sampling to code generation and consistency checks, making it easier for the community to use.

weaknesses:
1. The authors rely entirely on ensembling existing datasets, limiting their contribution's novelty and scope. This also makes the pipeline depend on pre-existing collections of instruction following/post-training data, and cannot be derived from natural data on the web.

2. The pipeline primarily modifies existing data by inserting Python-based tool calls. it should consider including stronger baseline models that focus more on coding tasks, such as Code Llama. 

3. The evaluation raises questions, particularly in relation to benchmark selection. Since the dataset is targeted at tool use, it would be more appropriate to evaluate with dedicated tool-use benchmarks. Additionally, the performance gaps between ToolBridge and ToolBridge* (a variant without tool calls) on standard benchmarks are small, which brings concern of how much extra value is added through the pipeline rather than the underlying SFT data from existing works.

4. While the model shows greater improvement on custom datasets, these evaluations raise concerns. The RandomQA dataset is mostly template-generated, and the FACT dataset is generated by GPT-4. Given that much of ToolBridge data also originates from GPT-4, this could introduce potential biases.

questions:
1. Why not use more coding-focused base models?
2. Is it possible to include some dedicated tool use benchmarks?
3. Why use WebQA and FACT for evaluation? Is the purpose to test the ability to use search tools?
4. Is all methods evaluated use the same prompt and tool use support? Since the baselines are not tuned for the specific format, how often do they use the tools, or write code. Some more detail on the experiment setup could help understand the improvement on tool_use as separated from just encouraging tool use under the specific format.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work creates a tool use dataset by consolidating various non-too-use datasets and applying a systematic conversion process. Experiments on standard math, QA benchmarks, and custom benchmarks reveal consistent performance improvements after fine-tuning with ToolBridge.
The reviewers agree that the dataset coming from this work provides value, but all raised concern on how does this work distinguish from broader tool use scenarios and code generation. Particularly, the work lacks the experiments to demonstrate the benefits of ToolBridge on tool use benchmarks like GAIA, ToolBench and ToolCamp. As reviewer X8CW and NXYV pointed out, there is no clear distinction between tool use/function calling and ToolFormer-like approach.