Retrieval Models Aren’t Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models
Authors: Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, Zhaochun Ren
Accepted at: acl25

Retrieval Models Aren’t Tool-Savvy:
Benchmarking Tool Retrieval for Large Language Models

Zhengliang Shi1  Yuhuan Wang1  Lingyong Yan2
Pengjie Ren1  Shuaiqiang Wang2  Dawei Yin2  Zhaochun Ren3*
1Shandong University, Qingdao, China   2Baidu Inc., Beijing, China
3Leiden University, Leiden, The Netherlands

Tool-Retrieval-Benchmark
shizhl@mail.sdu.edu.cn   z.ren@liacs.leidenuniv.nl

Abstract

Tool learning aims to augment large language models (LLMs) with diverse tools, enabling them to act as agents for solving practical tasks. Due to the limited context length of tool-using LLMs, adopting information retrieval (IR) models to select useful tools from large toolsets is a critical initial step. However, the performance of IR models in tool retrieval tasks remains underexplored and unclear. Most tool-use benchmarks simplify this step by manually pre-annotating a small set of relevant tools for each task, which is far from the real-world scenarios. In this paper, we propose TOOLRET, a heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks, and a corpus of 43k tools, collected from existing datasets. We benchmark six types of models on TOOLRET. Surprisingly, even the models with strong performance in conventional IR benchmarks, exhibit poor performance on TOOLRET. This low retrieval quality degrades the task pass rate of tool-use LLMs. As a further step, we contribute a large-scale training dataset with over 200k instances, which substantially optimizes the tool retrieval ability of IR models.1

1 Introduction

Large language models (LLMs) have demonstrated remarkable progress across various natural language processing (NLP) tasks, such as text summarization (Chang et al., 2023). However, they suffer from inherent inabilities to interact with the physical world and access vast, up-to-date knowledge (Qin et al., 2024). To alleviate these drawbacks, tool learning is proposed to equip LLMs with external tools, augmenting them as agents to manipulate tools for practical task-solving (Qu et al., 2025b; Wang et al., 2024e).

In practical applications, retrieving useful tools from toolsets for LLM agents typically serves as the initial step (Wang et al., 2024c; Xu et al., 2024; Song et al., 2023). This step becomes particularly critical in real-world scenarios where the candidate tools are usually large-scale and many of them are similar in functionality (Qu et al., 2024a). However, most existing work (Guo et al., 2024; Qian et al., 2023) simplifies this retrieval process by manually pre-selecting a small set of 10-20 relevant tools for each evaluation task. For example, the ToolACE (Liu et al., 2024a) and ToolBench (Qin et al., 2023) annotate about 10 tools per task. While recent information retrieval (IR) techniques such as semantic matching (Qu et al., 2024a; Xu et al., 2024), can assist with tool retrieval, they are often trained on ad-hoc tool-use datasets, lacking comprehensive evaluation on diverse scenarios, especially for unseen tasks. To further explore the importance of tool retrieval, we conduct a pilot experiment on ToolBench (Qin et al., 2023). As shown in Figure 1, we observe that (i) the agent’s performance substantially drops when replacing the officially annotated toolset with the retrieved tools; and (ii) even strong retrievers like colbertv2 (Santhanam et al., 2021a), struggle to retrieve target tools effectively. These findings highlight the necessity to (i) systematically evaluate IR models on diverse tool retrieval tasks; and (ii) analyze the impact of retrieval on the end-to-end task pass rate.

In this work, we introduce TOOLRET, the first

![Correlation between the tool retrieval performance (e.g., Recall@10) of IR models and the end-to-end task pass rate of tool-use agents.](page_1012_384_393_246.png)

Figure 1: Correlation between the tool retrieval performance (e.g., Recall@10) of IR models and the end-to-end task pass rate of tool-use agents.

1Resource is available on Huggingface and Website.
*Corresponding author.
large-scale tool retrieval benchmark comprising 7.6k diverse retrieval tasks and a corpus of 43k tools, which comprehensively evaluates IR models across diverse retrieval scenarios. Specifically, we collect query-tool datasets from the following sources: (i) Tool-use agent benchmarks from published research papers in AI conferences, such as ACL and NeurIPS; (ii) Related conference resources such as AppBench in EMNLP and Tool-Lens in CIKM; and (iii) Other publicly available datasets from the open-source community, e.g., HuggingFace. The collected data is carefully curated to cover a wide range of practical tool requirements, comprising diverse types of tool documentation, domains, and varying query lengths. Then, we standardize the format of all the collected tasks, aligning them with retrieval tasks similar to the format in MTEB, where each retrieval task contains a query and target tools (e.g., labels). To support the instructional retrieval (Weller et al., 2024) setting of our benchmark, we also introduce a target-aware strategy to supplement each query with an instruction using the powerful LLMs (i.e., gpt-4o).

We systematically evaluate five types of IR models such as embedding models and LLM re-ranking, under various experimental settings. Our results reveal that even the best model (i.e., NV-embed-v1) that demonstrates strong performance in conventional IR benchmarks, achieves an nDCG@10 of only 33.83 in our benchmark. This highlights the challenges of the tool retrieval tasks. We identify two key factors contributing to this performance gap: (i) Lower term overlap between queries and target tools in tool retrieval tasks, which demands higher representation abilities for IR models to accurately match query intent with the correct tools; and (ii) Task shift from conventional information-seeking tasks (e.g., document retrieval) to tool retrieval, leading to suboptimal performance of IR models that are not explicitly optimized.

To enhance the retrieval performance and enable IR models to augment tool-use agents, we further propose the TOOLRET-train, a large-scale training dataset containing more than 200k retrieval tasks. We extend our data collection process from TOOLRET to include the training set of three mainstream tool-use datasets, including ToolACE (Liu et al., 2024a), APIGen (Liu et al., 2024b) and Tool-Bench (Qin et al., 2023). To enable the training, we pair each retrieval task with 10 negative tools retrieved by the NV-embed-v1. Finally, each training example contains the query, an generated instruction, the target tools, and the negative tools. Results show that the IR models trained over TOOLRET-train, exhibit significant improvements in the retrieval process, leading to a higher end-to-end task pass rate when integrated with tool-use LLMs.

Our contributions are summarized as follows: (i) We introduce TOOLRET, the first evaluation benchmark for tool retrieval tasks. (ii) We evaluate the tool retrieval performance of various IR models and analyze the impact of retrieval on the end-to-end task pass rate of tool-use LLMs; and (iii) We contribute to a large-scale training dataset that enhances the performance of IR models, improving their ability to augment tool-use LLMs effectively.

2 Related work

Tool learning with foundation models. Tool learning aims to equip LLMs with tools, such as web API (Song et al., 2023) and python packages (Wang et al., 2024d), expanding their utility (Qin et al., 2023). Existing work teaching LLMs to use tools can be broadly classified into tuning-free (Lu et al., 2023) and tuning-based methods (Gao et al., 2024). The former prepends the description of candidate tools in the LLMs’ context, prompting them to select and invoke tools (Huang et al., 2023). The latter enables LLMs to learn the usage of each tool through training on synthetic data (Liu et al., 2024a; Gao et al., 2024). However, both two paradigms struggle when facing the large-scale toolset in practice (Qu et al., 2024b; Liu et al., 2024b). First, real-world toolsets are typically massive, making it less possible to incorporate all tools within the limited context of LLMs. For example, the Rapi-dAPI platform contains more than 52k tools while the PyPI2 hosts over 600k frequently updated packages. Second, since tools are frequently updated, it is cost-intensive to re-train the LLMs to memorize all tools (Qu et al., 2025a). Although recent studies address this challenge using semantic retrievers (Qin et al., 2023; Wang et al., 2024c), these solutions are typically ad-hoc and lack systematic evaluation across diverse tool retrieval scenarios. To fill this gap, we present the first comprehensive tool retrieval benchmark with systematic analysis.

Information retrieval benchmark. Conventional information retrieval (IR) benchmarks are typically designed for information-seeking tasks, such as Nature Question (Kwiatkowski et al., 2019) for question answering and MS-MARCO (Nguyen

2https://pypi.org/
et al., 2016) for passage re-ranking. Recent work also explores the IR technique in various downstream tasks, such as table retrieval (Chen et al., 2024b; Zhang and Balog, 2020) and scientific retrieval (Ajith et al., 2024), which substantially augments the downstream models. However, tool retrieval, a crucial step for tool-use agents, remains underexplored. Compared with traditional IR tasks, retrieving useful tools is more challenging since solving a task typically requires the combination of multiple tools (Qu et al., 2024b). Most existing benchmarks simplify this retrieval process by manually annotating a small set of tools that fit the LLMs’ context, which is far from reality with a large toolset. In this work, we evaluate IR models on diverse tool retrieval tasks and contribute over 200k training data to facilitate future research.

3 Benchmark construction

3.1 Data collection
To build a comprehensive benchmark for tool retrieval evaluation, we collect data from the following well-known sources: (i) Tool-use LLM benchmarks: A wide range of benchmarks published in leading AI conferences such as ACL and NeurIPS; (ii) Conference Resources: Datasets from resource tracks in IR and NLP conferences (e.g., CIKM and EMNLP); and (iii) Other high-quality dataset: We identify related datasets released on open-source platforms like HuggingFace and their technique reports can be found in public submissions like arXiv. We include them to enrich TOOLRET.

Given the rapid development of benchmarks from these sources, we collect datasets released between the August 2023 to December 2024 in this version.3 We download these data from official channels based on their usage requirements and totally collect more than 30 datasets. Since the data sources are diverse and their original formats vary substantially, we perform necessary data cleaning operations such as deduplication and text normalization to ensure consistency and quality.

We observe that most of the collected datasets are originally designed to evaluate the tool-use capability of LLMs, where the LLM is required to correctly call a sequence of target tools given an input query. To facilitate retrieval evaluation in TOOLRET, we align the format of all collected tasks with the well-established IR benchmark like BEIR and MTEB. Specifically, each task consists of a query as input and target tools as label (a.k.a. ground truth), where a tool is identified by a unique identifier and paired with detailed documentation to describe its functionality. Endpoints of the collected datasets and concrete examples of our formatted dataset are provided in Appendix B.

3.2 Data sampling
After collecting the datasets, we observe data size imbalances across different datasets. Besides, some datasets are extremely large with substantial redundant content, making comprehensive model evaluation both inefficient and unnecessary. Therefore, we streamline them through effective data sampling while maintaining its evaluation integrity.

Task sampling. For each collected dataset, we encode the tasks using the embedding model, i.e., NV-embedd-v1, and apply the K-means clustering algorithm on the text embeddings. We set the number of clusters to the size of the corresponding toolset and randomly sample one task from each cluster. If the toolset size exceeds the number of queries, we retain all queries. For example, the original ToolEyes (Ye et al., 2024a) dataset contains 500 queries and 95 tools; Thus, we set the cluster number as \( \min(500, 95) = 95 \) for clustering.

Toolset sampling. To eliminate redundancy, we manually review the documentation of each raw dataset to identify and merge identical toolsets. For example, since the COLT (Qu et al., 2024a) toolset overlaps with the Toolbench (Qin et al., 2023) , we merge their intersecting tools. Ultimately, we merge all toolsets from the 34 datasets to form the final corpus, resulting in a total of 43k tools. Each tool is assigned a unique identifier.

After sampling, we obtain 7.6k retrieval tasks and a corpus of 43k tools.

3.3 Instruction construction
Instructional information retrieval (Sun et al., 2024; Weller et al., 2024) is an active research area, where an additional instruction is paired with the input query to guide IR models in retrieving target information. This instruction-following capability is especially critical in tool retrieval, as IR models are often used to augment LLM agents and receive additional context from the agents beyond the input query. To support this instructional IR scenario, we construct the instructions as part of TOOLRET.

Considering manually writing instructions is cost-intensive and challenging to scale, we introduce a target-aware strategy using powerful LLMs
<table>
  <tr>
    <th>Statistic</th>
    <th></th>
  </tr>
  <tr>
    <td># size of retrieval task</td>
    <td>7,615</td>
  </tr>
  <tr>
    <td>- # of web API retrieval task</td>
    <td>4,916</td>
  </tr>
  <tr>
    <td>- # of code function retrieval task</td>
    <td>950</td>
  </tr>
  <tr>
    <td>- # of customized app retrieval task</td>
    <td>1,749</td>
  </tr>
  <tr>
    <td># size of tool</td>
    <td>43,215</td>
  </tr>
  <tr>
    <td>- # of web API</td>
    <td>36,978</td>
  </tr>
  <tr>
    <td>- # of code function</td>
    <td>3,794</td>
  </tr>
  <tr>
    <td>- # of customized app</td>
    <td>2,443</td>
  </tr>
  <tr>
    <td>avg. query / instruction length (tokens)</td>
    <td>46.87 / 43.43</td>
  </tr>
  <tr>
    <td>avg. tool documentation length (token)</td>
    <td>174.56</td>
  </tr>
</table>

Table 1: Basic statistics of our benchmark TOOLRET.

<table>
  <tr>
    <th></th>
    <th>Ours</th>
    <th>NQ</th>
    <th>MSMARCO</th>
    <th>HotpotQA</th>
    <th>MTEB</th>
  </tr>
  <tr>
    <th># Average number of targets for an input query.</th>
    <td>2.17</td>
    <td>1.00</td>
    <td>1.00</td>
    <td>2.00</td>
    <td>2.57</td>
  </tr>
  <tr>
    <th># ROUGE-L overlap between query and targets.</th>
    <td>0.06</td>
    <td>0.31</td>
    <td>0.34</td>
    <td>0.11</td>
    <td>0.27</td>
  </tr>
</table>

Table 2: Comparison with conventional IR benchmarks.

to automate this process. Specifically, we first invite three human experts with strong NLP and IR backgrounds to manually craft 100 seed instructions. In line with the well-defined format from Asai et al., our instruction outlines the relevance criteria by bridging the query intent and the functionality of the target tools. For example, for the transcribing the audio to text task, the instruction is presented as “retrieve tools that process audio inputs to produce accurate textual transcriptions aligned with the user requirements”. Next, we employ a powerful LLM, i.e., GPT-4o, as an automatic instruction generator and guide it to generate instruction for each task through in-context learning. To enhance the diversity, we randomly sample in-context examples from the pool of both the generated and handcrafted instructions. A detailed pseudo algorithm is provided in Appendix B.

After the above three processes, we obtain TOOLRET, which consists of 7.6k tasks, each paired with an instruction, and a corpus of 43k diverse tools, providing a comprehensive testbed and supporting various evaluation settings.

4 Benchmark statistic

Table 1 provides the basic statics of TOOLRET. We observe that there are three mainstream formats of tool documentation: (i) Code, which is a function-level snippet in programming language; (ii) Web API, which elaborates the tool usage in structured JSON format following the Web OpenAPI specification; (iii) Customized application, which directly describes the tool functionality in free-form nature language. Based on these formats, we categorize TOOLRET into three subsets accordingly and divide the TOOLRET into Code Function, Web API, and Customized App subsets. Below, we report a more detailed analysis of TOOLRET.

![ROUGE-L overlap distribution](page_1012_120_420_312.png)
Figure 2: ROUGE-L overlap between the query (input) and the target tools (label).

![Length distributions](page_1012_480_420_312.png)
Figure 3: Length distribution of our benchmark.

4.1 Complexity

In tool learning, previous studies have highlighted the necessity of combining multiple tools for task solving (Shi et al., 2024). Thus, we analyze the complexity of our retrieval benchmark from two aspects. First, we calculate the average number of target tools for each retrieval task and compare it with well-known IR benchmarks such as HotpotQA (Yang et al., 2018) and MTEB (Muenni ghoff et al., 2022). As shown in Table 2, TOOLRET requires models to recall more targets, posing a challenge in comprehensive retrieval. Second, we compute the lexical overlap, i.e., ROUGE-L, between the input query and corresponding retrieval targets (tool documentation in TOOLRET and passage in IR benchmarks). We find that this overlap is substantially lower in TOOLRET. It indicates that, for neural IR models, TOOLRET requires more heavily on the semantic representation rather than simple lexical matching. Therefore, the retrieval task in TOOLRET is more challenging.
<table>
  <tr>
    <th>Quality Review Question</th>
    <th>Yes or No %</th>
  </tr>
  <tr>
    <td>Whether the instruction is relevant to the original input query?</td>
    <td>90.1% / 9.9%</td>
  </tr>
  <tr>
    <td>Whether the instruction describes the feature of target tools</td>
    <td>92.3% / 8.7%</td>
  </tr>
  <tr>
    <td>Whether the instruction comprehensively describe the feature of all target tools</td>
    <td>89.2% / 10.8%</td>
  </tr>
  <tr>
    <td>Whether the instruction contains hallucination about the target tools or input query?</td>
    <td>5.9% / 94.1%</td>
  </tr>
</table>

Table 3: The quality review for our generated instructions, which is conducted by five human experts with 0.743 Kappa statistics.

4.2 Length statistics

Figure 3 illustrates the length distribution of the query, instruction, and tool documentation in TOOLRET.\footnote{We use the tokenizer from gpt-3.5-turbo in this work.} We find that most queries are concise, typically containing fewer than 60 tokens (about 25 words), which aligns with real-world user behavior, as users tend to input brief queries with minimal effort. Additionally, most tool documentation is under 200 tokens, which is similar to the chunk length in standard IR document retrieval corpus, such as Wikipedia dump (Karpukhin et al., 2020).

4.3 Quality

So far, we have demonstrated the complexity and quantity of our benchmark while the quality of the LLM-generated instructions remains uncertain. To investigate this, we ask 5 human experts to label the quality based on four aspects listed in Table 3. Our evaluation reveals that 89.2% of the generated instructions correctly cover the feature of the target tools and are faithfully grounded on the original queries. For the remaining 10.8% instructions that mismatch the query or the target tools, we ask experts to revise them. This re-check mechanism ensures the high quality of instructions in TOOLRET, making it a reliable evaluation benchmark. To explain more intuitively, we list a number of seed instructions, high-quality and low-quality instructions in Table 8. Annotation guidance is also provided in Appendix B to promote our transparency.

4.4 Instruction diversity

We further analyze how the generated instructions differ from the seed instructions used to prompt the generation. For each generated instruction, we compute its highest ROUGE-L overlap with the 100 seed instructions. We plot the distribution of these ROUGE-L scores in Figure 4. The results indicate a decent number of new instructions are generated, which have low overlap with the seeds.

![ROUGE-L overlap between the handcrafted seed instructions and model-generated instructions.](page_1012_120_388_246.png)

Figure 4: ROUGE-L overlap between the handcrafted seed instructions and model-generated instructions.

5 Benchmark evaluation setup

5.1 Evaluation protocol

We use three widely used IR metrics to evaluate the retrieval performance: (i) \( NDCG@K \) (N@K): evaluates ranking quality based on the relevance of retrieved tools; (ii) \( Recall@K \) (R@K): evaluates the proportion of target tools successfully retrieved within the top-K results; and (iii) \( Precision@K \) (P@K): evaluates the accuracy of the retrieved tools within the top-K results. We also use \( Completeness@K \) (C@K) from COLT (Qu et al., 2024b), which specifically evaluates the retrieval completeness in tool retrieval tasks. The C@K is 1 if all target tools are included in the top-k retrieved tools; otherwise, it is 0.

We mainly evaluate IR models under two settings: (i) w/o inst.: The model take the query alone as input; and (ii) w/ inst.: The model takes the concatenation of the query and instruction as input to retrieve. This allows us to analyze the impact of instructions on retrieval performance.

5.2 Model to Evaluate

We comprehensively evaluate the following mainstream IR models on our benchmark.

Sparse retrieval. These methods measure the similarity between query and tool documentation based on lexical overlap. We evaluate BM25s (Lü, 2024).

Single-task dense retrieval. These methods use dual-encoder models trained on conventional IR datasets. We evaluate gtr (Ni et al., 2021a), contriever (Izard et al., 2021a), and colbertv2.0 (Santhanam et al., 2021a), all trained on MS-MARCO (Nguyen et al., 2016). We also evaluate COLT (Qu et al., 2024a), a recently proposed model trained on ad-hoc tool retrieval datasets.

Multi-task embedding Models. These methods
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">TOOLRET-Web</th>
    <th colspan="4">TOOLRET-Code</th>
    <th colspan="4">TOOLRET-Customized</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
  </tr>
  <tr>
    <td>BM25s</td><td>18.98</td><td>4.64</td><td>24.62</td><td>15.20</td><td>21.20</td><td>3.37</td><td>28.23</td><td>26.96</td><td>26.76</td><td>5.86</td><td>32.39</td><td>24.40</td><td>22.32</td><td>22.19</td>
  </tr>
  <tr>
    <td>COLT</td><td>15.43</td><td>2.63</td><td>21.11</td><td>20.04</td><td>20.69</td><td>5.12</td><td>28.07</td><td>18.53</td><td>21.63</td><td>4.72</td><td>29.19</td><td>22.40</td><td>19.25</td><td>20.32</td>
  </tr>
  <tr>
    <td>Colbert</td><td>22.40</td><td>5.37</td><td>27.41</td><td>15.45</td><td>16.43</td><td>2.65</td><td>22.54</td><td>21.65</td><td>19.54</td><td>3.65</td><td>23.72</td><td>18.97</td><td>19.46</td><td>18.69</td>
  </tr>
  <tr>
    <td>contriever-msmarco</td><td>21.15</td><td>5.83</td><td>27.19</td><td>14.70</td><td>14.56</td><td>2.40</td><td>19.28</td><td>17.71</td><td>17.72</td><td>3.31</td><td>22.77</td><td>18.31</td><td>17.81</td><td>16.91</td>
  </tr>
  <tr>
    <td>gtr-t5-base</td><td>17.36</td><td>4.25</td><td>24.17</td><td>15.95</td><td>16.47</td><td>2.71</td><td>22.27</td><td>21.16</td><td>23.47</td><td>5.09</td><td>28.93</td><td>22.49</td><td>19.10</td><td>18.87</td>
  </tr>
  <tr>
    <td>gtr-t5-large</td><td>22.45</td><td>5.42</td><td>29.75</td><td>18.72</td><td>18.25</td><td>2.89</td><td>24.12</td><td>23.08</td><td>26.30</td><td>5.76</td><td>31.86</td><td>24.45</td><td>22.34</td><td>22.09</td>
  </tr>
  <tr>
    <td>all-MiniLM-L6-v2</td><td>11.66</td><td>3.07</td><td>16.36</td><td>10.15</td><td>14.44</td><td>2.50</td><td>19.50</td><td>18.11</td><td>22.80</td><td>5.21</td><td>29.10</td><td>20.25</td><td>16.30</td><td>16.17</td>
  </tr>
  <tr>
    <td>e5-small-v2</td><td>19.89</td><td>5.08</td><td>26.46</td><td>16.26</td><td>15.48</td><td>2.39</td><td>19.26</td><td>18.05</td><td>24.60</td><td>5.56</td><td>29.67</td><td>20.76</td><td>19.99</td><td>18.36</td>
  </tr>
  <tr>
    <td>e5-base-v2</td><td>19.75</td><td>5.04</td><td>25.89</td><td>15.37</td><td>14.43</td><td>2.47</td><td>19.19</td><td>18.00</td><td>22.68</td><td>5.11</td><td>29.13</td><td>22.25</td><td>18.95</td><td>18.54</td>
  </tr>
  <tr>
    <td>e5-large-v2</td><td>18.99</td><td>4.90</td><td>25.97</td><td>16.27</td><td>17.09</td><td>2.68</td><td>21.87</td><td>20.70</td><td>26.42</td><td>6.07</td><td>32.19</td><td>23.17</td><td>20.83</td><td>20.05</td>
  </tr>
  <tr>
    <td>gte-base-en-v1.5</td><td>23.55</td><td>6.28</td><td>32.03</td><td>19.15</td><td>17.43</td><td>2.87</td><td>23.71</td><td>22.48</td><td>21.62</td><td>4.76</td><td>29.03</td><td>23.17</td><td>20.86</td><td>21.60</td>
  </tr>
  <tr>
    <td>gte-large-en-v1.5</td><td>22.41</td><td>5.91</td><td>30.14</td><td>18.44</td><td>16.66</td><td>2.87</td><td>23.64</td><td>22.39</td><td>20.62</td><td>5.19</td><td>26.75</td><td>17.67</td><td>19.90</td><td>19.50</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5</td><td>22.50</td><td>6.02</td><td>29.96</td><td>17.30</td><td>17.78</td><td>2.92</td><td>23.66</td><td>22.27</td><td>25.99</td><td>5.71</td><td>32.17</td><td>24.26</td><td>22.09</td><td>21.27</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5</td><td>24.45</td><td>6.66</td><td>32.93</td><td>19.30</td><td>18.90</td><td>3.12</td><td>25.76</td><td>24.47</td><td>25.72</td><td>5.54</td><td>32.18</td><td>24.79</td><td>23.02</td><td>22.85</td>
  </tr>
  <tr>
    <td>gte-Qwen2-1.5B-inst.</td><td>29.17</td><td>7.93</td><td>38.05</td><td>21.49</td><td>21.66</td><td>3.41</td><td>28.89</td><td>27.67</td><td>36.04</td><td>7.89</td><td>44.51</td><td>35.55</td><td>28.96</td><td>26.04</td>
  </tr>
  <tr>
    <td>e5-mistral-7b*</td><td>26.76</td><td>7.25</td><td>34.39</td><td>21.05</td><td>20.01</td><td>3.44</td><td>28.31</td><td>27.10</td><td>31.41</td><td>6.68</td><td>38.47</td><td>29.24</td><td>26.06</td><td>25.80</td>
  </tr>
  <tr>
    <td>GrimT-7B*</td><td>25.74</td><td>6.85</td><td>34.27</td><td>21.28</td><td>22.02</td><td>3.72</td><td>30.41</td><td>28.87</td><td>42.31</td><td>8.71</td><td>49.34</td><td>38.17</td><td>30.02</td><td>29.44</td>
  </tr>
  <tr>
    <td>NV-Embed-v1*</td><td>31.30</td><td>8.35</td><td>39.15</td><td>23.05</td><td>29.64</td><td>4.72</td><td>40.45</td><td>38.88</td><td>40.54</td><td>8.25</td><td>45.93</td><td>34.44</td><td>33.83</td><td>32.12</td>
  </tr>
  <tr>
    <td>mxbai-rerank-large-v1</td><td>22.99</td><td>5.61</td><td>30.32</td><td>18.38</td><td>24.76</td><td>3.88</td><td>34.86</td><td>33.22</td><td>26.76</td><td>5.91</td><td>34.53</td><td>26.03</td><td>24.84</td><td>25.88</td>
  </tr>
  <tr>
    <td>monot5-base-msmarco</td><td>28.92</td><td>7.70</td><td>36.44</td><td>19.97</td><td>21.61</td><td>3.62</td><td>30.06</td><td>27.88</td><td>36.22</td><td>7.54</td><td>45.11</td><td>36.41</td><td>28.92</td><td>28.09</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-m3</td><td>32.92</td><td>8.73</td><td>41.88</td><td>25.63</td><td>24.28</td><td>3.80</td><td>32.71</td><td>30.94</td><td>30.51</td><td>7.00</td><td>36.03</td><td>26.74</td><td>29.24</td><td>27.77</td>
  </tr>
  <tr>
    <td>jina-reranker-v2-base</td><td>35.38</td><td>9.25</td><td>44.65</td><td>26.98</td><td>26.47</td><td>4.15</td><td>35.20</td><td>33.94</td><td>38.94</td><td>8.14</td><td>46.06</td><td>35.42</td><td>33.60</td><td>32.11</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-gemma</td><td>36.72</td><td>9.69</td><td>45.94</td><td>27.85</td><td>29.89</td><td>4.42</td><td>38.23</td><td>36.82</td><td>39.93</td><td>9.06</td><td>49.43</td><td>37.75</td><td>35.51</td><td>34.14</td>
  </tr>
  <tr>
    <td>Mixtral-8x2B</td><td>28.21</td><td>8.31</td><td>34.13</td><td>25.42</td><td>27.41</td><td>3.14</td><td>34.13</td><td>36.98</td><td>30.76</td><td>5.40</td><td>34.12</td><td>28.65</td><td>28.80</td><td>30.35</td>
  </tr>
  <tr>
    <td>gpt-3.5-turbo-8125</td><td>30.29</td><td>8.01</td><td>36.00</td><td>24.22</td><td>28.69</td><td>4.27</td><td>36.25</td><td>35.64</td><td>29.80</td><td>6.39</td><td>35.01</td><td>28.70</td><td>29.60</td><td>29.52</td>
  </tr>
  <tr>
    <td>gpt-3.5-turbo-1106</td><td>31.01</td><td>7.86</td><td>35.82</td><td>23.76</td><td>28.95</td><td>4.44</td><td>38.16</td><td>38.45</td><td>32.30</td><td>6.89</td><td>38.31</td><td>30.84</td><td>30.75</td><td>31.01</td>
  </tr>
</table>

Table 4: Experiment results in w/o inst. setting (§ 5), where the model takes the query as input to retrieve. We mark the baselines pre-trained on instructional datasets with *. We highlight the best performance in each type of model.

utilize transformer encoders trained on various IR datasets. We evaluate all-MiniLM-L6-v2, gte (Li et al., 2023c), bge (Xiao et al., 2023a) and e5 (Wang et al., 2022), covering a wide range of sizes.

Cross-encoder re-rankers. These models re-rank the initially retrieved documents based on the query-passage relevance. We evaluate: MonoT5 (Nogueira et al., 2020), mxbai-rerank-large, jina-reranker-v2-base, and bge-reranker.

LLM agents. These methods leverage general-purpose LLM agents for re-ranking tasks in a zero-shot setting, simulating the tool selection process of tool-use agents. We evaluate the widely used LLM re-ranking framework, i.e., RankGPT (Sun et al., 2023), with various LLMs as backbone.

Initial tools for LLM agent and Re-ranking baselines are retrieved by NV-embedd-v1 model. Details of these baselines are provided in Appendix D.

6 Experiment result

6.1 Tool retrieval performance

Existing retrievers struggle. As shown in Table 9, the tool retrieval tasks in TOOLRET raise significant challenges for existing retriever models. Specifically, all retrievers in our experiments achieve less than 35% in Completeness@10 and under 52% in recall@10. Notably, retrieval methods that demonstrate strong performance in conventional information retrieval (IR) tasks, such as ColBERT, even underperform compared to simple lexical-based matching approaches like BM25. Similarly, other embedding-based models, even the NV-Embed-v1 with 7B parameter, achieve less than 45% in completeness@10, exhibiting limitations.

We identify two potential reasons for the above performance degradation: (i) Tool retrieval tasks require intensive reasoning over the input query to align user intentions with candidate tools, as the lexical overlap between the query and targets is low. (ii) There exists a domain shift between the conventional training corpora used for retrieval models and the specific tool retrieval tasks, which current models are not explicitly optimized for.

Re-ranking technique has limited improvement. As shown in Table 9, commonly used re-ranking

![Correlation between the score on our benchmark and MTEB (retrieval subset).](page_1012_1342_388_246.png)

Figure 5: Correlation between the score on our benchmark and MTEB (retrieval subset).
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">TOOLRET-Web</th>
    <th colspan="5">TOOLRET-Code</th>
    <th colspan="5">TOOLRET-Customized</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th>
    <th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th>
    <th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th>
    <th>P@10</th><th>R@10</th><th>C@10</th>
  </tr>
  <tr>
    <td>BM25s</td><td>26.33</td><td>6.10</td><td>34.22</td><td>22.79</td><td>41.90</td><td>6.20</td><td>56.49</td><td>55.39</td><td>41.16</td><td>8.39</td><td>48.60</td><td>38.90</td><td>36.46</td><td>39.03</td>
  </tr>
  <tr>
    <td>COLT</td><td>28.91</td><td>4.61</td><td>40.64</td><td>38.83</td><td>20.06</td><td>4.71</td><td>27.78</td><td>18.84</td><td>31.29</td><td>6.05</td><td>42.19</td><td>34.01</td><td>26.75</td><td>30.56</td>
  </tr>
  <tr>
    <td>Colbert</td><td>16.67</td><td>3.12</td><td>21.14</td><td>14.94</td><td>30.35</td><td>4.37</td><td>41.38</td><td>40.28</td><td>24.35</td><td>4.56</td><td>30.97</td><td>24.87</td><td>23.79</td><td>26.70</td>
  </tr>
  <tr>
    <td>contriever-msmarco</td><td>23.48</td><td>5.29</td><td>30.21</td><td>19.69</td><td>31.61</td><td>4.84</td><td>43.01</td><td>41.74</td><td>21.93</td><td>3.85</td><td>27.28</td><td>23.04</td><td>25.67</td><td>28.16</td>
  </tr>
  <tr>
    <td>gtr-t5-base</td><td>20.38</td><td>4.49</td><td>27.53</td><td>19.24</td><td>33.59</td><td>4.90</td><td>43.18</td><td>41.88</td><td>41.84</td><td>7.66</td><td>48.35</td><td>39.28</td><td>31.94</td><td>33.46</td>
  </tr>
  <tr>
    <td>gtr-t5-large</td><td>24.37</td><td>5.27</td><td>31.64</td><td>21.26</td><td>36.76</td><td>5.33</td><td>47.42</td><td>45.92</td><td>42.04</td><td>8.48</td><td>50.84</td><td>40.00</td><td>34.39</td><td>35.73</td>
  </tr>
  <tr>
    <td>all-MiniLM-L6-v2</td><td>12.77</td><td>3.26</td><td>19.38</td><td>13.33</td><td>31.59</td><td>5.06</td><td>43.86</td><td>42.25</td><td>32.24</td><td>7.14</td><td>43.53</td><td>32.34</td><td>25.53</td><td>29.41</td>
  </tr>
  <tr>
    <td>e5-small-v2</td><td>26.42</td><td>6.20</td><td>34.44</td><td>21.39</td><td>32.36</td><td>4.84</td><td>42.38</td><td>41.11</td><td>34.62</td><td>6.90</td><td>42.29</td><td>32.58</td><td>31.14</td><td>31.69</td>
  </tr>
  <tr>
    <td>e5-base-v2</td><td>24.71</td><td>5.78</td><td>33.45</td><td>21.94</td><td>31.40</td><td>5.01</td><td>42.83</td><td>41.38</td><td>38.06</td><td>7.54</td><td>46.84</td><td>36.43</td><td>31.39</td><td>33.25</td>
  </tr>
  <tr>
    <td>e5-large-v2</td><td>23.62</td><td>5.52</td><td>32.19</td><td>21.80</td><td>34.27</td><td>5.05</td><td>44.42</td><td>43.19</td><td>43.32</td><td>8.51</td><td>52.30</td><td>41.42</td><td>33.73</td><td>35.47</td>
  </tr>
  <tr>
    <td>gte-base-en-v1.5</td><td>30.75</td><td>7.00</td><td>39.44</td><td>25.88</td><td>41.68</td><td>6.20</td><td>53.96</td><td>51.64</td><td>37.95</td><td>6.96</td><td>46.57</td><td>38.10</td><td>36.79</td><td>38.54</td>
  </tr>
  <tr>
    <td>gte-large-en-v1.5</td><td>28.06</td><td>6.55</td><td>36.32</td><td>22.57</td><td>35.77</td><td>5.75</td><td>49.56</td><td>47.71</td><td>37.27</td><td>7.88</td><td>47.98</td><td>35.84</td><td>33.70</td><td>35.37</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5</td><td>25.95</td><td>6.16</td><td>35.12</td><td>23.40</td><td>35.15</td><td>5.22</td><td>47.44</td><td>43.20</td><td>43.20</td><td>8.82</td><td>53.54</td><td>42.29</td><td>34.77</td><td>36.67</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5</td><td>30.03</td><td>7.01</td><td>39.28</td><td>25.63</td><td>41.55</td><td>6.00</td><td>52.76</td><td>51.18</td><td>43.90</td><td>8.31</td><td>51.79</td><td>42.24</td><td>38.49</td><td>39.68</td>
  </tr>
  <tr>
    <td>e5-mistral-7b*</td><td>31.07</td><td>7.65</td><td>41.30</td><td>27.04</td><td>44.97</td><td>6.66</td><td>58.95</td><td>56.79</td><td>40.88</td><td>7.91</td><td>49.35</td><td>38.35</td><td>38.97</td><td>40.73</td>
  </tr>
  <tr>
    <td>NV-Embed-v1*</td><td>31.51</td><td>7.40</td><td>40.52</td><td>26.74</td><td>47.92</td><td>7.10</td><td>62.07</td><td>59.60</td><td>48.70</td><td>10.07</td><td>57.69</td><td>43.88</td><td>42.71</td><td>43.41</td>
  </tr>
  <tr>
    <td>qwen2-1.5B-inst.*</td><td>37.53</td><td>9.31</td><td>48.31</td><td>30.95</td><td>47.38</td><td>7.29</td><td>61.12</td><td>59.55</td><td>52.98</td><td>10.63</td><td>59.47</td><td>45.68</td><td>45.96</td><td>45.39</td>
  </tr>
  <tr>
    <td>GrimT-7B*</td><td>36.58</td><td>9.34</td><td>46.01</td><td>27.65</td><td>41.26</td><td>6.17</td><td>53.81</td><td>52.07</td><td>45.55</td><td>9.74</td><td>54.01</td><td>41.40</td><td>41.33</td><td>40.37</td>
  </tr>
  <tr>
    <td>mxbai-rerank-large-v1</td><td>17.53</td><td>4.05</td><td>25.82</td><td>17.95</td><td>33.86</td><td>5.05</td><td>47.84</td><td>46.47</td><td>26.83</td><td>6.71</td><td>37.61</td><td>28.60</td><td>31.01</td><td></td>
  </tr>
  <tr>
    <td>monoT5-base-msmarco</td><td>23.33</td><td>5.88</td><td>30.70</td><td>18.13</td><td>31.39</td><td>5.27</td><td>45.18</td><td>42.51</td><td>37.77</td><td>6.76</td><td>46.63</td><td>39.70</td><td>30.83</td><td>33.45</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-m3</td><td>34.83</td><td>8.54</td><td>45.23</td><td>31.73</td><td>50.86</td><td>7.64</td><td>67.26</td><td>64.78</td><td>42.35</td><td>9.52</td><td>53.75</td><td>39.90</td><td>42.68</td><td>45.47</td>
  </tr>
  <tr>
    <td>jina-reranker-v2-base</td><td>42.35</td><td>10.11</td><td>51.21</td><td>34.23</td><td>53.21</td><td>7.66</td><td>66.03</td><td>63.94</td><td>45.94</td><td>10.36</td><td>57.96</td><td>45.41</td><td>47.17</td><td>47.86</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-gemma</td><td>34.73</td><td>8.09</td><td>45.08</td><td>32.29</td><td>55.85</td><td>8.22</td><td>70.53</td><td>68.76</td><td>51.97</td><td>11.04</td><td>61.20</td><td>45.65</td><td>47.52</td><td>48.90</td>
  </tr>
  <tr>
    <td>Mixtral-8x2B</td><td>35.31</td><td>7.56</td><td>38.63</td><td>34.60</td><td>33.27</td><td>5.77</td><td>39.60</td><td>38.53</td><td>34.40</td><td>6.44</td><td>39.72</td><td>38.20</td><td>34.33</td><td>37.11</td>
  </tr>
  <tr>
    <td>gpt-3.5-turbo-0125</td><td>37.22</td><td>8.97</td><td>40.82</td><td>35.22</td><td>45.42</td><td>6.22</td><td>41.16</td><td>42.64</td><td>37.29</td><td>8.24</td><td>41.34</td><td>39.70</td><td>29.60</td><td>29.52</td>
  </tr>
  <tr>
    <td>gpt-3.5-turbo-1106</td><td>38.31</td><td>9.02</td><td>41.29</td><td>35.76</td><td>38.69</td><td>7.27</td><td>42.57</td><td>42.81</td><td>39.30</td><td>7.89</td><td>43.31</td><td>37.31</td><td>38.77</td><td>38.63</td>
  </tr>
</table>

Table 5: Experiment results in w/ inst. setting (§ 5), where the model takes the query and instruction as input to retrieval. * indicates the model is pre-trained on instructional datasets. We highlight the best performance.

methods provide limited and even negative improvements for the tool retrieval task. When using MonoT5 to re-rank the tools retrieved by NV-Embed-v1, the average NDCG@10 decreases from 33.83 to 28.92. A similar trend is observed with the mxbai-rerank. Other advanced models such as bge-reranker-v2-gemma only have 4.7% improvement in the Completeness@10 metric. These results further indicate the challenging nature of tool retrieval.

6.2 Substantial gains from instruction

Besides the evaluation results under w/o inst setting in Table 4, we also present the results under w/ inst setting in Table 5. We observe that all the IR model achieves better performance when an additional instruction is paired with the query as input. Notably, the instruction-tuned embedding model like NV-embed-v1 or e5-mistral has the most obvious improvement, which potentially benefits from its powerful instruction-following capability. These results illustrate the advantages of the instruction and instruction tuning in tool retrieval tasks.

6.3 Compare with conventional IR tasks

To further explore the complexity of tool retrieval tasks, we compare the models’ performance on TOOLRET and conventional IR task benchmark, i.e., MTEB, showing their relationship in Figure 5. First, we can see that the two benchmarks share a similar trend (Pearson’s coefficient \( \beta = 0.790 \)), but the score in TOOLRET is lower. This indicates that the task in TOOLRET has a correlation with conventional IR tasks but is more challenging. Second, we also observe that conventional IR models trained with relevance-oriented criteria such as contriever perform poorly on TOOLRET, which indicates that TOOLRET requires more target-aware reasoning ability. This is also illustrated in § 4.1.

7 Retrieval affect downstream task

In this section, we qualitatively analyze the impact of retrieval performance on downstream tool-use agents. We conduct end-to-end evaluations on ToolBench (Qin et al., 2023) dataset using the official Pass Rate metric that evaluates whether the model successfully calls target tools to complete a task.

7.1 Poor retrieval leads to poor tool-use agents

For each task in ToolBench, we replace the officially pre-annotated toolset (oracle) with tools retrieved by IR models. As shown in Figure 6, the tool-use LLMs, when equipped with the retrieved tools, exhibit substantially lower performance compared to their oracle counterparts. For example, in ToolBench-G1, GPT-3.5 achieves a pass rate of 50.60 using tools retrieved by bge–large, decreasing by 11.40. This indicates that tool retrieval is a crucial step to build better tool-use LLMs and
Figure 6: The horizontal axis indicates the retrieval performance of IR models, both before and after training. The vertical axis corresponds to the end-to-end pass rate of tool-use LLMs using the tools retrieved by these IR models.

improve their task-solving performance.

7.2 Towards better retrieval

The analysis in § 6.2 highlights the advantage of instruction-tuning in improving tool retrieval. However, to the best of our knowledge, there is no large-scale instructional IR dataset for tool retrieval tasks. We propose the TOOLRET-train to fill this gap.

Large-scale training data We extend the data collection process from TOOLRET to include the training sets of three mainstream tool-use datasets: ToolACE (Liu et al., 2024a), ToolBench (Qin et al., 2023) and APIGen (Liu et al., 2024b). Ultimately, we collect over 200k training instances, each comprising a query \( q \) and a set of target tools \( \mathcal{T} \). We also pair each query \( q \) with an instruction \( \mathcal{I} \) using our target-aware strategy (See Appendix D).

Learning objective To train a IR model (denoted as \( \theta \)), we first use it to retrieved top-K negative tools, denoted as \( \hat{\mathcal{T}} = \{ \hat{t}_j \mid j \in [K], \hat{t}_j \notin \mathcal{T} \} \). The model \( \theta \) is then optimized by maximizing the *log-likelihood of the target tools*. The loss function \( \mathcal{L} \) is formulated as:

\[
-\frac{1}{|\mathcal{T}|} \sum_{t_i \in \mathcal{T}} \log \frac{e^{\operatorname{sim}(\mathcal{I} \oplus q, t_i)}}{e^{\operatorname{sim}(\mathcal{I} \oplus q, t_i)} + \sum_{\hat{t}_j \in \hat{\mathcal{T}}} e^{\operatorname{sim}(\mathcal{I} \oplus q, \hat{t}_j)}}.
\]

The \( \mathcal{I} \oplus q \) indicates concatenation of instruction and query with a special token. During the training, we set the \( K \) to 10 and the learning rate to 5e-5.

Improvement from retrieval. As shown in Figure 6, all IR models trained on TOOLRET-train achieve substantial improvement in NDCG@10 metric. We further evaluate the task pass rate of two tool-use LLMs: GPT-3.5 and ToolLLama (Qin et al., 2023). When equipped with the improved IR models, both LLMs exhibit substantial gains in pass rate, confirming the critical role of retrieval in downstream tasks. As part of future work, we suggest adapting the IR models to better augment the tool-use LLMs, which offers a efficient plug-and-play solution compared with training LLMs.

We further conduct an ablation study by removing the instruction \( \mathcal{I} \) from the loss function \( \mathcal{L} \). The results show that this variant shows improvements compared with the non-tuned counterparts, but underperforms compared with their instruction-tuned counterparts (See Appendix D). These validate the effectiveness of our instructional training data in enhancing tool retrieval performance.

8 Discussion

In this section, we discuss the following three open questions related to the design and reliably of our benchmark construction, as well as potential extension.

Fine-grained functional differences of tools In this work, we construct the overall tool corpus by merging tools from various existing tool-use benchmarks. This heterogeneous construction strategy inevitably leads to overlapping functionality among different tools, which may raise concerns about unreliable evaluations, such as the one-to-many problem\footnote{The *one-to-many* problem arises because our dataset combines multiple existing datasets. For example, for a query from dataset A, the ground truth may not be limited to the single annotation provided in A. Similar tools in dataset B might also provide valid solutions to the same query. However, in the evaluation process, only the ground truth from dataset A is}. However, the following reasons
support the reliability of our approach: (i) In real-world scenarios, many different but functionally similar tools exist, but typically, only the most suitable one is chosen. In this work, we focus on such real-world scenarios, where the ground truth from the original dataset is considered the most appropriate. We argue that models should have the fine-grained discrimination ability to identify the most appropriate tool; (ii) Even if tools appear to have overlapping high-level functionality (e.g., Bing Search vs. Google Search), they often differ in important dimensions such as input parameters (e.g., support for language-specific filtering) and specific application scope (e.g., medical search vs. general news search). For example, for a query like search news articles in Chinese, a tool like bing_search_with_lang_param is a more precise match than a generic search API without language constraints.

In line with prior work (Gao et al., 2024; Qin et al., 2023; Qu et al., 2025a), our benchmark encourages models to retrieve semantically and functionally appropriate tools, not merely those with similar surface forms.

Reason for our format-based categorization
Unlike existing tool retrieval benchmarks (Qu et al., 2024a), we introduce a novel task categorization by dividing the overall TOOLRET dataset into three subsets: Web APIs, Code Functions, and Customized Applications. This division is motivated by two reasons: (i) It naturally reflects the structural features of tools collected from over 30 diverse datasets. This categorization aligns with conventions established in prior research on tool-augmented LLMs, where these three formats are commonly used as the primary categories for executable tools. (ii) This format-based taxonomy provides a clear, interpretable, and effective framework for analyzing model behavior across different tool representations.

Beyond format-based category
In addition to the format-based categorization presented in this work, we believe that other dimensions of categorization can also offer valuable insights into retrieval behavior. Therefore, we propose extending the category structure in future work to include three additional dimensions for more comprehensive and customized evaluations: (i) Query Length (Input Complexity): Tasks are grouped based on the query token length, with categories such as 0–25, 25–50, 50–75, and 75+ tokens. (ii) Number of Target Tools (Multi-Label Complexity): Queries are grouped by the number of relevant ground-truth tools, categorized as 1, 2–3, or \( \geq 4 \) tools. (iii) Lexical Overlap (ROUGE-L) Between Query and Tool Documentation: Tasks are grouped by the degree of lexical overlap, categorized as low (<0.05), medium (0.05–0.2), and high (\( \geq 0.2 \)). A lower overlap indicates a higher need for deep semantic matching between the query and the correct tools.

9 Conclusion
In this work, we introduce TOOLRET, the first diverse tool retrieval benchmark comprising 7.6k queries, each paired with an instruction, and a corpus of 43k tools. TOOLRET is a heterogeneous benchmark, constructed by aggregating existing tool-use datasets and aligning them into a unified format, similar to conventional IR benchmarks such as MTEB. We evaluate state-of-the-art IR models on TOOLRET and uncover a surprising finding: even models with strong performance on conventional IR benchmarks struggle in tool retrieval. This low retrieval quality significantly degrades the end-to-end task pass rate of tool-use LLMs. Inspired by this, we further propose TOOLRET-train, a large-scale training set containing over 200k retrieval tasks. Results show that IR models trained on TOOLRET-train exhibit substantial improvement and also enhance the pass rate of tool-use LLMs by 10%–20%. In the future, we plan to extend the TOOLRET into multimodal scenarios.

Acknowledgements
This work was supported by the Natural Science Foundation of China (Grant No. 62472261), the Shandong Province Key Research and Development Program (2024CXGC010108), and the Shandong Province Technology Innovation Guidance Program (YDZX2024088).
Limitation

The limitations of this work include the lack of exploration in multilingual retrieval settings. Currently, our benchmark is confined to the English language and focuses exclusively on text retrieval. To address this limitation, we plan to expand our research to encompass multilingual information retrieval (IR) scenarios in the future. Additionally, another limitation lies in the insufficient investigation of prompt sensitivity. Given that large language models (LLMs) are highly sensitive to prompt wording, we aim to annotate a broader range of instructions in the future to examine how variations in prompt phrasing influence LLM performance.

Building upon TOOLRET, we suggest the following directions for future work.

(i) Investigating sensitivity to instructions: Conduct a comprehensive study on how LLM performance varies with different prompt formulations and instruction styles.

(ii) In this work, we focus on the retrieval-then-calling method, where the tool retrieval step is triggered only once initially, and the top-k retrieved tools are then passed to the tool-use LLMs. In the future, we aim to benchmark IR models in interleaved retrieval-and-calling scenarios. Specifically, the LLM can generate long-form reasoning for task planning, and the IR model can receive this reasoning output, exploring step-by-step integration between the LLM and IR models.

(iii) Enhancing IR models for improved retrieval accuracy: Further optimize IR models to achieve higher retrieval precision, leveraging these improvements to augment tool-use LLMs and, consequently, enhance end-to-end task performance.

Ethics Statement

We acknowledge the importance of the ACM Code of Ethics and fully agree with it. We ensure that this work is compatible with the provided code in terms of publicly accessible datasets and models. Risks and harms associated with large language models include the generation of harmful, offensive, or biased content. The new benchmark is composed of various previous datasets and is therefore licensed under their respective data licenses.

In this research, we prioritize reproducibility by not only utilizing state-of-the-art commercial LLMs but also experimenting extensively with open-source LLMs. Throughout the study, we have strictly followed ethical standards to maintain the integrity and validity of our work. All tools and resources used in this research were obtained from publicly available platforms, ensuring transparency and reproducibility in our experimental procedures. Furthermore, we have made every effort to ensure that our research does not harm individuals or groups, nor does it involve any form of deception or potential misuse of information.
References

Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. Litsearch: A retrieval benchmark for scientific literature search. arXiv preprint arXiv:2407.18940.

Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2022. Task-aware retrieval with instructions. arXiv preprint arXiv:2211.09260.

Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology.

Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.

Si-An Chen, Lesly Miculicich, Julian Martin Eisen-schlos, Zifeng Wang, Zilong Wang, Yanfei Chen, Yasuhisa Fujii, Hsuan-Tien Lin, Chen-Yu Lee, and Tomas Pfister. 2024b. Tablerag: Million-token table understanding with language models. arXiv preprint arXiv:2410.04739.

Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. 2023. T-eval: Evaluating the tool utilization capability step by step. arXiv preprint arXiv:2312.14033.

Shen Gao, Zhengliang Shi, Minghang Zhu, Bowen Fang, Xin Xin, Pengjie Ren, Zhumin Chen, Jun Ma, and Zhaochun Ren. 2024. Confucius: Iterative tool learning from introspection feedback by easy-to-difficult curriculum. In Proceedings of the AAAI Conference on Artificial Intelligence: AAAI.

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86–92.

Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models. arXiv preprint arXiv:2403.07714.

Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Ji-ahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, et al. 2024. Planning, creation, usage: Benchmarking llms for comprehensive tool utilization in real-world complex scenarios. arXiv preprint arXiv:2401.17167.

Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. 2023. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021a. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021b. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. Association for Computational Linguistics.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics.

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428.

Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. 2023a. Making large language models a better foundation for dense retrieval. Preprint, arXiv:2312.15503.

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023b. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023c. Towards general text embeddings with multi-stage contrastive learning. ArXiv, abs/2308.03281.

Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al. 2024a. Toolace:
Winning the points of llm function calling. arXiv preprint arXiv:2409.00920.

Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. 2024b. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406.18518.

Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. In Neural Information Processing Systems: NeurIPS.

Xing Han Lü. 2024. Bm25s: Orders of magnitude faster lexical search via eager sparse scoring. ArXiv, abs/2407.03618.

Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, and Ranjay Krishna. 2024. m & m’s: A benchmark to evaluate tool-use for m ulti-step m ulti-modal tasks. In European Conference on Computer Vision, pages 18–34. Springer.

Niklas Muenni ghoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316.

Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. Generative representational instruction tuning. Preprint, arXiv:2402.09906.

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773. CEUR-WS.org.

Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2021a. Large dual encoders are generalizable retrievers. ArXiv, abs/2112.07899.

Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Abrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. 2021b. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899.

Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713.

Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334.

Cheng Qian, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2023. Toolink: Linking toolkit creation and using through chain-of-solving on open-source model. arXiv preprint arXiv:2310.05155.

Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, et al. 2024. Tool learning with foundation models. ACM Computing Surveys, 57(4):1–40.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789.

Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Xu Jun, and Ji-Rong Wen. 2025a. From exploration to mastery: Enabling llms to master tools via self-driven interactions. In The Thirteenth International Conference on Learning Representations.

Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2024a. Colt: Towards completeness-oriented tool retrieval for large language models. In CIKM.

Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2024b. Towards completeness-oriented tool retrieval for large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 1930–1940.

Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2025b. Tool learning with large language models: A survey. Frontiers of Computer Science.

Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2023. Identifying the risks of lm agents with an im- emulated sandbox. arXiv preprint arXiv:2309.15817.

Keshav Santhanam, O. Khattab, Jon Saad-Falcon, Christopher Potts, and Matei A. Zaharia. 2021a. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In North American Chapter of the Association for Computational Linguistics.

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2021b. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint arXiv:2112.01488.

Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. 2023. Taskbench: Benchmarking large language models for task automation. arXiv preprint arXiv:2311.18760.
Zhengliang Shi, Shen Gao, Xiuyi Chen, Yue Feng, Lingyong Yan, Haibo Shi, Dawei Yin, Zhumin Chen, Suzan Verberne, and Zhaochun Ren. 2024. Chain of tools: Large language model is an automatic multi-tool learner. arXiv preprint arXiv:2405.16533.

Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, et al. 2023. Restgpt: Connecting large language models with real-world restful apis. arXiv preprint arXiv:2306.06624.

Weiwei Sun, Zhengliang Shi, Jiulong Wu, Lingyong Yan, Xinyu Ma, Yiding Liu, Min Cao, Dawei Yin, and Zhaochun Ren. 2024. Mair: A massive benchmark for evaluating instructed retrieval. In EMNLP.

Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint arXiv:2304.09542.

Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301.

Hongru Wang, Rui Wang, Boyang Xue, Heming Xia, Jingtao Cao, Zeming Liu, Jeff Z Pan, and Kam-Fai Wong. 2024a. Appbench: Planning of multiple apis from various apps for complex user instruction. arXiv preprint arXiv:2410.19743.

Jize Wang, Zerun Ma, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. 2024b. Gta: a benchmark for general tool agents. arXiv preprint arXiv:2407.08713.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368.

Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, and Haonan Li. 2024c. Toolgen: Unified tool retrieval and calling via generation. arXiv preprint arXiv:2410.03439.

Xingyao Wang, Yangyi Chen, Lianf Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024d. Executable code actions elicit better llm agents. arXiv preprint arXiv:2402.01030.

Zhiruo Wang, Zhoujun Cheng, Hao Zhu, Daniel Fried, and Graham Neubig. 2024e. What are tools anyway? a survey from the language model perspective. arXiv preprint arXiv:2403.15452.

Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. 2024. Followir: Evaluating and teaching information retrieval models to follow instructions. arXiv preprint arXiv:2403.15246.

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023a. C-pack: Packaged resources to advance general chinese embedding. ArXiv, abs/2309.07597.

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023b. C-pack: Packaged resources to advance general chinese embedding. Preprint, arXiv:2309.07597.

Qiancheng Xu, Yongqi Li, Hening Xia, and Wenjie Li. 2024. Enhancing tool retrieval with iterative feedback from large language models. arXiv preprint arXiv:2406.17465.

Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504.

Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2024. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP).

Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, et al. 2024a. Tooleyes: Fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. arXiv preprint arXiv:2401.00741.

Junjie Ye, Yilong Wu, Songyang Gao, Caishuang Huang, Sixian Li, Guanyu Li, Xiaoran Fan, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024b. Rotbench: a multi-level benchmark for evaluating the robustness of large language models in tool learning. arXiv preprint arXiv:2401.08326.

Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R Fung, Hao Peng, and Heng Ji. 2023. Craft: Customizing llms by creating and retrieving from specialized toolsets. arXiv preprint arXiv:2309.17428.

Shuo Zhang and Krisztian Balog. 2020. Web table extraction, retrieval, and augmentation: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(2):1–35.

Yinger Zhang, Hui Cai, Xierui Song, Yicheng Chen, Rui Sun, and Jing Zheng. 2023. Reverse chain: A generic-rule for llms to master multi-api planning. arXiv preprint arXiv:2310.04474.
Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, and Hayato Yamana. 2024. Toolbehonest: A multi-level hallucination diagnostic benchmark for tool-augmented large language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP).

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.
A  Data Card

Following previous work (Bender and Friedman, 2018; Gebru et al., 2021; Zhuo et al., 2024), we provide the datacard for TOOLRET, where we tend to summarize and centralize all information that might be relevant for the benchmark analysis.

(i) The purpose of this benchmark: This benchmark is proposed to comprehensively evaluate the information retrieval (IR) models on tool retrieval tasks. On top of TOOLRET, we find that existing IR models, despite achieving strong performance in conventional IR benchmarks such as MTEB and BEIR, still suffer from substantial challenges in tool retrieval tasks. The poor retrieval quality further degrades the end-to-end task pass rate of tool-use LLMs. Thus, we believe that the TOOLRET reveals the importance of tool retrieval in building better tool-use LLMs, and can be used as a comprehensive and fair benchmark in facilitating the development of tool retrieval models.

(ii) How will the dataset be distributed (e.g., Tarball on Website or Github)? The proposed benchmark TOOLRET will be released to the public, and hosted on GitHub and Hugging Face. The TOOLRET will be managed and maintained by our research team.

(iii) Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?
Yes. If we include more tasks or find any errors, we will correct the dataset hosted on Hugging Face and GitHub and update the results in the leaderboard accordingly. It will be updated on our website.

(iv) Will the training dataset TOOLRET-train will be released publicly. Yes, the proposed training dataset TOOLRET-train will be released to the public, and hosted on GitHub and Hugging Face.

B  Details of Benchmark

B.1  Dataset collections

TOOLRET is a heterogeneous benchmark that integrates a wide range of well-established tool-use datasets and aligns them into a unified format, similar to standard information retrieval (IR) benchmarks such as BEIR and MTEB, to facilitate tool retrieval evaluation. In tool learning, we observe that previous work primarily focuses on three mainstream types of tools:

(i) Web APIs: These tools are encapsulated in the OpenAPI format (standard JSON documentation) and can be directly invoked via HTTP requests. Web APIs are typically used to access, manipulate (e.g., add, delete, edit, or query), or retrieve private data or information from specialized databases, covering a wide range of domains such as movies, music, and sports.

(ii) Code Functions: These tools are represented by source code containing function signatures and implementation details. Code functions primarily focus on low-level computations or atomic operations, such as tensor calculations, calling Hugging Face models, or utilizing PyTorch libraries.

(iii) Customized Apps: These tools are paired with free-form natural language descriptions. They are typically user-oriented or personalized, enabling tasks such as sending emails or other custom applications.

These tool types differ in functionality and documentation format, reflecting diverse scenarios for tool-use LLMs. For IR models, retrieving different types of tools may present varying levels of difficulty. Therefore, we categorize the collected datasets into these three types based on their paired toolset formats, resulting in three subsets of TOOLRET: TOOLRET-web, TOOLRET-code, and TOOLRET-customized. During evaluation, we report the performance of IR models on each subset to provide a fine-grained analysis. Below, we list the datasets included in each subset and provide detailed descriptions.
B.2 TOOLRET-Web
The TOOLRET-Web subset is constructed by integrating the following datasets, which contain tools in the form of Web APIs:

• AutoTools-Food (Shi et al., 2024): Contains APIs related to food recipes, where LLMs must retrieve specific food-related tools to answer user queries.

• RestGPT-TMDB (Song et al., 2023) and AutoTools-Movie (Shi et al., 2024): Includes web APIs from the TMDB platform, a movie database. Evaluation tasks require LLMs to retrieve tools to find relevant information about movies or celebrities and extract key evidence to answer given queries.

• AutoTools-Weather (Shi et al., 2024): Features web APIs from a weather database. LLMs must invoke these APIs and gather responses to answer weather-related queries.

• RestGPT-Spotify and AutoTools-Music (Shi et al., 2024): Contains web APIs from a music platform. Evaluation tasks require LLMs to retrieve tools for searching songs or albums based on user queries.

• ToolBench (Qin et al., 2023): Comprises over 16,000 web APIs crawled from RapidAPI. Queries are generated by LLMs, with ground truth tools labeled for each query.

• ToolLens (Qu et al., 2024a): A subset of ToolBench, where queries are annotated to evaluate tool functionality.

• APIbank (Li et al., 2023b): Contains web APIs for daily personalized applications, such as alarm booking and database login.

• MetaTool (ToolE) (Huang et al., 2023): Designed to evaluate whether LLMs are aware of tool usage and can correctly select tools.

• Mnms (Ma et al., 2024): Evaluates LLM-based agents’ tool-use abilities for multi-step, multi-modal tasks involving tools that process visual information. Since TOOLRET focuses on text-based IR models, images are represented using their URLs.

• Reverse-Chain (Zhang et al., 2023): Contains diverse multi-step tasks requiring LLMs to invoke relevant tools sequentially.

• ToolEyes (Ye et al., 2024a): Includes tools across various domains, such as advice, entertainment, and art, providing a broad evaluation of tool-use LLMs in practical scenarios.

• UltraTool (Huang et al., 2024): A benchmark designed to improve and evaluate LLMs’ tool utilization abilities in real-world scenarios, focusing on the entire process of planning, creating, and applying tools in complex tasks.

• T-Eval (Chen et al., 2023): A fine-grained benchmark assessing tool-use LLMs across multiple evaluation aspects, including instruction following, planning, reasoning, retrieval, understanding, and review.

B.3 TOOLRET-Code
The TOOLRET-code subset is constructed by integrating the following datasets, which contain tools in the form of code functions:

• Gorilla-PyTorch (Patil et al., 2023): Contains various PyTorch functions (code snippets) as tools, evaluating LLMs’ ability to correctly combine PyTorch functions for solving deep learning tasks. The functions in this dataset are collected from the Python Torch package.
<table>
  <tr>
    <th>Dataset</th>
    <th>Endpoint</th>
    <th>Query size</th>
    <th>Tool size</th>
    <th>Task type</th>
  </tr>
  <tr>
    <td colspan="5"><b>Web APIs</b></td>
  </tr>
  <tr>
    <td>GTA (Wang et al., 2024h)</td>
    <td>https://github.com/open-compass/GTA</td>
    <td>14</td>
    <td>14</td>
    <td></td>
  </tr>
  <tr>
    <td>Gorilla (Patil et al., 2023)</td>
    <td>https://github.com/ShishirPatil/gorilla</td>
    <td>598</td>
    <td>1,005</td>
    <td></td>
  </tr>
  <tr>
    <td>- gorilla-pytorch subset</td>
    <td>https://github.com/ShishirPatil/gorilla/tree/main/data</td>
    <td>43</td>
    <td>43</td>
    <td></td>
  </tr>
  <tr>
    <td>- gorilla-tensor subset</td>
    <td>https://github.com/ShishirPatil/gorilla/tree/main/data</td>
    <td>55</td>
    <td>55</td>
    <td></td>
  </tr>
  <tr>
    <td>- gorilla-huggingface subset</td>
    <td>https://github.com/ShishirPatil/gorilla/tree/main/data</td>
    <td>500</td>
    <td>907</td>
    <td></td>
  </tr>
  <tr>
    <td>CRAFT (Yuan et al., 2023)</td>
    <td>https://github.com/lifan-yuan/CRAFT</td>
    <td>654</td>
    <td>985</td>
    <td></td>
  </tr>
  <tr>
    <td>- craft-TabMWP subset</td>
    <td>https://github.com/lifan-yuan/CRAFT/tree/main/tab_and_math/TabMWP</td>
    <td>174</td>
    <td>180</td>
    <td></td>
  </tr>
  <tr>
    <td>- craft-Vqa subset</td>
    <td>https://github.com/lifan-yuan/CRAFT/tree/main/vqa</td>
    <td>200</td>
    <td>525</td>
    <td></td>
  </tr>
  <tr>
    <td>- craft-algebra subset</td>
    <td>https://github.com/lifan-yuan/CRAFT/tree/main/tab_and_math/MATH</td>
    <td>280</td>
    <td>280</td>
    <td></td>
  </tr>
  <tr>
    <td>- AutoTools-Food subset</td>
    <td>https://github.com/mangopy/AutoTools/tree/main/data</td>
    <td>159</td>
    <td>159</td>
    <td></td>
  </tr>
  <tr>
    <td>- AutoTools-Weather subset</td>
    <td>https://github.com/mangopy/AutoTools/tree/main/data</td>
    <td>11</td>
    <td>11</td>
    <td></td>
  </tr>
  <tr>
    <td>- AutoTools-Movie subset</td>
    <td>https://github.com/mangopy/AutoTools/tree/main/data</td>
    <td>54</td>
    <td>54</td>
    <td></td>
  </tr>
  <tr>
    <td>- AutoTools-music subset</td>
    <td>https://github.com/mangopy/AutoTools/tree/main/data</td>
    <td>72</td>
    <td>72</td>
    <td></td>
  </tr>
  <tr>
    <td>APIGen (Liu et al., 2024b)</td>
    <td>https://huggingface.co/datasets/Salesforce/xlai-function-calling-60k</td>
    <td>1,000</td>
    <td>3,605</td>
    <td></td>
  </tr>
  <tr>
    <td>APIBank (Li et al., 2023h)</td>
    <td>https://github.com/AlibabaResearch/DMD-ConvAI</td>
    <td>101</td>
    <td>101</td>
    <td></td>
  </tr>
  <tr>
    <td>AppPace (Zhang et al., 2024a)</td>
    <td>https://github.com/zhengapppace</td>
    <td>32</td>
    <td>32</td>
    <td></td>
  </tr>
  <tr>
    <td>Bian (Ma et al., 2024)</td>
    <td>https://github.com/BAYNvlab/bian</td>
    <td>33</td>
    <td>33</td>
    <td></td>
  </tr>
  <tr>
    <td>Metatool (k.e.a., ToolE) (Huang et al., 2023)</td>
    <td>https://github.com/Houweiwang/MetaTool</td>
    <td>200</td>
    <td>200</td>
    <td></td>
  </tr>
  <tr>
    <td>Reverse Chain (Zhang et al., 2023)</td>
    <td>https://github.com/ASK-83/Reverse-Chain</td>
    <td>200</td>
    <td>783</td>
    <td></td>
  </tr>
  <tr>
    <td>RestGPT (Song et al., 2023)</td>
    <td>https://github.com/Yfan-Song793/RestGPT</td>
    <td>94</td>
    <td>94</td>
    <td></td>
  </tr>
  <tr>
    <td>- RestGPT-TMDB subset</td>
    <td>https://github.com/Yfan-Song793/RestGPT/datasets/tmdb.json</td>
    <td>54</td>
    <td>54</td>
    <td></td>
  </tr>
  <tr>
    <td>- RestGPT-Spotify subset</td>
    <td>https://github.com/Yfan-Song793/RestGPT/datasets/spotify.json</td>
    <td>40</td>
    <td>40</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolLens (Qu et al., 2023)</td>
    <td>https://github.com/OpenBB-ToolBench</td>
    <td>1,100</td>
    <td>13,862</td>
    <td></td>
  </tr>
  <tr>
    <td>- GI-category subset</td>
    <td>https://drive.google.com/drive/folders/1yBUQ732mPu-KclJnuQELEhtKakdXfc3j</td>
    <td>200</td>
    <td>13,862</td>
    <td></td>
  </tr>
  <tr>
    <td>- GI-tool subset</td>
    <td>https://drive.google.com/drive/folders/1yBUQ732mPu-KclJnuQELEhtKakdXfc3j</td>
    <td>200</td>
    <td>13,862</td>
    <td></td>
  </tr>
  <tr>
    <td>- G2-instruction subset</td>
    <td>https://drive.google.com/drive/folders/1yBUQ732mPu-KclJnuQELEhtKakdXfc3j</td>
    <td>200</td>
    <td>13,862</td>
    <td></td>
  </tr>
  <tr>
    <td>- G3-instruction subset</td>
    <td>https://drive.google.com/drive/folders/1yBUQ732mPu-KclJnuQELEhtKakdXfc3j</td>
    <td>100</td>
    <td>13,862</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolACE (Liu et al., 2024a)</td>
    <td>https://github.com/junjie-Ye/ToolACE</td>
    <td>314</td>
    <td>314</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolACE (Liu et al., 2024a)</td>
    <td>https://github.com/junjie-Ye/ToolACE</td>
    <td>95</td>
    <td>95</td>
    <td></td>
  </tr>
  <tr>
    <td>GPT4tools (Yang et al., 2024)</td>
    <td>https://github.com/AllAB-CVC/GPT4tools</td>
    <td>1,000</td>
    <td>16,072</td>
    <td></td>
  </tr>
  <tr>
    <td>Rothenberg (Ye et al., 2024b)</td>
    <td>https://github.com/junjie-Ye/RoBench</td>
    <td>550</td>
    <td>919</td>
    <td></td>
  </tr>
  <tr>
    <td>T-eval (Chen et al., 2023)</td>
    <td>https://github.com/open-compass/T-Eval</td>
    <td>100</td>
    <td>100</td>
    <td></td>
  </tr>
  <tr>
    <td>- T-eval-step level subset</td>
    <td>https://huggingface.co/datasets/jovesnowbest/T-Eval</td>
    <td>50</td>
    <td>50</td>
    <td></td>
  </tr>
  <tr>
    <td>- T-eval-dialogue level subset</td>
    <td>https://huggingface.co/datasets/jovesnowbest/T-Eval</td>
    <td>50</td>
    <td>50</td>
    <td></td>
  </tr>
  <tr>
    <td>Taskbench (Shao et al., 2023)</td>
    <td>https://github.com/microsoft/JARVIS-taskbench</td>
    <td>103</td>
    <td>103</td>
    <td></td>
  </tr>
  <tr>
    <td>- Taskbench-multimedia subset</td>
    <td>https://github.com/microsoft/JARVIS-taskbench/multimedia</td>
    <td>40</td>
    <td>40</td>
    <td></td>
  </tr>
  <tr>
    <td>- Taskbench-daily subset</td>
    <td>https://github.com/microsoft/JARVIS-taskbench/dailylifeqs</td>
    <td>40</td>
    <td>40</td>
    <td></td>
  </tr>
  <tr>
    <td>- Taskbench-DL subset</td>
    <td>https://github.com/microsoft/JARVIS-taskbench/huggingface</td>
    <td>23</td>
    <td>23</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolApaca (Tang et al., 2023)</td>
    <td>https://github.com/tangqiaoyu/ToolApaca</td>
    <td>94</td>
    <td>1,937</td>
    <td></td>
  </tr>
  <tr>
    <td>Toolbench-sam (Xu et al., 2023)</td>
    <td>https://github.com/sanbanova/toolbench</td>
    <td>197</td>
    <td>197</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolEmu (Ruan et al., 2023)</td>
    <td>https://github.com/ryoung/ToolEmu</td>
    <td>38</td>
    <td>38</td>
    <td></td>
  </tr>
  <tr>
    <td>ToolLink (Qian et al., 2023)</td>
    <td>https://github.com/qiancheng/ToolLink</td>
    <td>497</td>
    <td>1,804</td>
    <td></td>
  </tr>
  <tr>
    <td>UltraTool (Huang et al., 2024)</td>
    <td>https://github.com/joeyting1910/ultraTool</td>
    <td>500</td>
    <td>1,171</td>
    <td></td>
  </tr>
  <tr>
    <td>Tool-be-honest (Zhang et al., 2024)</td>
    <td>https://github.com/ToolBeHonest/ToolBeHonest</td>
    <td>550</td>
    <td>892</td>
    <td></td>
  </tr>
</table>

Table 6: The detailed statistics about the each collected dataset in TOOLRET. We highlight that the subsets of ToolBench share a same toolsets containing 13,000+ tools. Besides, the TOOLRET combine and deduplicate the toolsets from the above datasets to build the final tool retrieval corpus.

• Gorilla-Tensor (Patil et al., 2023): Includes TensorFlow functions as tools, collected from TensorFlow Hub, to assess LLMs’ tool selection capabilities in deep learning scenarios.

• Gorilla-HuggingFace (Patil et al., 2023): Treats specific downstream models from the Hugging Face platform as tools. This dataset evaluates LLMs’ performance in correctly calling Hugging Face models based on user queries.

• CRAFT-TabMWP (Yuan et al., 2023): Evaluates LLMs’ ability to use functions for table processing. The functions in this dataset are first generated by GPT-4 and subsequently verified.

• CRAFT-VQA (Yuan et al., 2023): Provides evaluation cases for visual question answering (VQA), where LLMs must call image processing functions such as image capture and object detection.

• CRAFT-Math-Algebra (Yuan et al., 2023): Assesses LLMs’ ability to invoke algebra functions for solving complex mathematical problems.

B.4 TOOLRET-Customized

Besides Web APIs and code functions, we also collect datasets that contain customized apps. Unlike Web APIs and code functions, customized apps are described using free-form natural language documentation
Algorithm 1: The pseudo algorithm for our target-aware strategy in automatically constructing instructions for evaluation tasks.

Input: A set of \( N \) seed instructions \( S = \{s_i \mid i \in [N]\} \) manually crafted by human experts; A powerful LLM \( \mathcal{M} \) (e.g., GPT-4o); Collected tasks \( \mathcal{T} = \{t_i | i \in |\mathcal{T}| \} \)
Initialize an instruction pool \( I \leftarrow S \);
for \( i \in |\mathcal{T}| \) do
    Sample \( k \) examples \( \{s'_1, s'_2, ..., s'_k\} \) from \( I \);
    // Generate a new instruction \( s_i \) using \( \mathcal{M} \) through in-context learning:
    \( s_i \leftarrow \mathcal{M}(\text{prompt with } \{s'_1, s'_2, ..., s'_k\}) \);
    //Append new instruction to pool:
    \( \mathcal{I} = \mathcal{I} \cup \{s_i\} \);
Apply heuristic filtering to remove low-quality instructions from \( I \);
Output: A set of high-quality instructions \( \mathcal{I} = \{s_1, s_2, ..., s_{|\mathcal{T}|}\} \)

rather than structured formats. Specifically, we include the following datasets: ToolACE (Liu et al., 2024a), GPT4Tools (Yang et al., 2024), TaskBench (Shen et al., 2023), ToolAlpaca, ToolBench-sam (Xu et al., 2023), ToolEmu (Ruan et al., 2023), and ToolLink (Qian et al., 2023).

B.5 Task format

The final benchmark, TOOLRET, integrates the above datasets and reformats all test cases into a unified format, similar to conventional IR benchmarks such as BEIR and MTEB, to evaluate IR models in tool retrieval tasks. Each reformatted task consists of: an input query, an instruction, and the corresponding target tools (e.g., labels). Each tool is assigned a unique identifier and is paired with detailed documentation describing its functionality. Below, we present a concrete example from TOOLRET.

# An example of an evaluation task in our proposed benchmark

- Query: I need to find a grocery store near 123 Main Street, Downtown District that has a good selection of limes for my Easter celebration.
- ID: toolens_query_7
- Target tools (labels): toolLens_tool_20, toolLens_tool_50, toolLens_tool_2
- Instruction: Given a `local grocery search` task, retrieve tools that can locate grocery stores based on the user's specified location and criteria, such as the availability of specific items like limes, to meet the query's requirements.

# Examples of tool documentation

- toolLens_tool_20: {"category_name": "Food", "required_parameters": [{"name": "ingredient", "type": "STRING", "description": "", "default": "strawberry"}], "optional_parameters": [], "method": "GET", "template_response": {"name": "str", "ingredients": ["list of str with length 9"]}, "instructions": ["list of str with length 7"]}, "name": "pastry/ingredient", "description": "This API endpoint allows users to retrieve a random pastry recipe that contains a specific ingredient. Users can make a GET request to the endpoint with the name of the ingredient as a query parameter, and the API will return a JSON response with the given recipe, including the name, list of ingredients, and instructions."}

- toolLens_tool_50: {"category_name": "Health_and_Fitness", "required_parameters": [], "optional_parameters": [{"name": "limit", "type": "NUMBER", "description": "limit the length of response", "default": "10"}], "method": "GET", "template_response": {"count": "int", "food": [{"id": "str", "food_name": "str", "quantity": "str", "calories": "int", "uri": "str", "type_uri": "str", "core": "str", "core_uri": "str", "food_nutrition": [{"nutrient_name": "str", "value": "float", "unit": "str", "list_length": 3}], "list_length": 10}]}, "name": "View All Food Items", "description": "The request allows clients to retrieve a comprehensive list of all available food items.\n\nAPI request sent to [https://indnutrientsapi.tech/food](https://indnutrientsapi.tech/food)"}

- toolLens_tool_2: {"category_name": "Food", "required_parameters": [{"name": "
B.6 Details of instruction construction

In TOOLRET, each task is paired with an instruction using a target-aware strategy, where GPT-4o acts as an automatic expert through in-context learning. Specifically, we follow these steps. We first invite three human experts with strong backgrounds in NLP and IR to manually craft seed instructions. These expert-crafted instructions form an initial example pool. For each task, we randomly sample a set of instructions from this pool as in-context learning examples. Using these examples, GPT-4o generates a new instruction tailored to the given task. The newly generated instruction is then appended back to the instruction pool to enhance instruction diversity. The detailed pseudo algorithm is provided in Alg. 1.

Below, we provide a concrete example of the GPT-4o prompt used in our instruction construction process. The example of seed instructions and the generated instructions is provided in Table 8.

# The prompt for GPT-4o.

Given a query, you need to design an instruction about 20 words that clearly indicates this is a task to retrieve tools capable of solving the query based on its content. The instruction should emphasize the task requirements and target outcomes of the query while incorporating the functional characteristics of the tools to help the system accurately match the appropriate tools.

Below, I have provided the target tools (i.e., the labels for the query). Please analyze the key aspects of the query and the tool descriptions. Your instruction should implicitly highlight the task requirements and the characteristics of the target tools relevant to the query.

Here is an output template that your should follow. Please note that the instruction should be concise.

Query: I would like to generate a video presenting a text-based discussion on the topic of 'The Benefits of Exercise'
Labels: [1] {'id': 'taskbench_data_huggingface_tool_5', 'doc': {'input-type': ['text '], 'output-type': ['text '], 'name': 'Text Generation', 'description': 'Generating text is the task of producing new text. These models can, for example, fill in incomplete text or paraphrase.'}}
Instruction: Given a `text-to-video` task, retrieve tools that process text inputs to generate coherent textual outputs aligned with the query's topic and requirements .

Query: I have an audio file 'example.wav' which is difficult to understand. I would like you to help me transcribe the audio to text
Labels: [1] {'id': 'taskbench_data_huggingface_tool_19', 'doc': {'input-type': ['audio '], 'output-type': ['text '], 'name': 'Automatic Speech Recognition', 'description': 'Automatic Speech Recognition (ASR), also known as Speech to Text (STT ), is the task of transcribing a given audio to text. It has many applications, such as voice user interfaces.'}, 'relevance': 1}
Instruction: Given a `audio transcription` task, retrieve tools that process audio inputs to produce accurate textual transcriptions aligned with the query's requirements.

Query: Conduct a two-sample independent t-test with two samples, sample1=[1, 2, 3, 4, 5] and sample2=[6, 7, 8, 9, 10] and a significance level of 0.05.
Labels: [1] {'id': 'tool_id_693', 'doc': {'name': 'independent_samples_t_test', 'description': 'Conducts a two-sample independent t-test and returns the t-statistic, p-value, and conclusion.', 'parameters': {'sample1': {'description': 'The first sample of observations.', 'type': 'List[float]', 'default': '0.05'}, 'sample2': {'description': 'The second sample of observations.', 'type': 'List[float]', 'default': '0.05'}, 'alpha': {'description': 'The significance level of the test. Defaults to 0.05.', 'type': 'float, optional'}}}, 'relevance': 1}
Instruction: Given a `significance test` task, retrieve tools that perform statistical tests, specifically a two-sample independent t-test, by processing numerical inputs and returning the t-statistic, p-value.
Query: Can I get a list of all boards and their attributes on page number two with a page size of seven?
Labels: [1] {'id': 'ToolEyes_tool_34', 'doc': {'name': 'get_boards', 'description': 'A list of all boards and their attributes.', 'parameters': {'type': 'object', 'properties': {'page': {'type': 'string', 'description': 'Get the items on a specific page. 0(default) is the first page.'}, 'page_size': {'type': 'string', 'description': 'Get the number of boards on a specific page. Default: 5.'}}, 'required': []}}
Instruction: Given a `pagination query` task, retrieve tools that can list boards and their attributes by processing parameters such as page number and page size to return the requested information.

B.7 Human annotation

To ensure the quality of the generated instructions, we conduct human annotation to review them based on four key aspects listed in Table 3. For instructions deemed low-quality, human annotators manually revise them to improve accuracy and clarity. For a clear illustration, we provide concrete examples of handcrafted instructions, high-quality generated instructions, and low-quality generated instructions in Table 8. Below, we also provide the detailed human annotation guidelines used in our review process for reproducibility and transparency.

# Human guidance for instruction quality annotation

We ask you to evaluate the quality of the generated instructions based on the following four aspects. Please carefully assess each instruction and provide your judgment:

## Aspects for annotation

1. Hallucination Check: Does the instruction contain any incorrect or fabricated information about the target tools or the input query? (Are there any details in the instruction that do not align with the actual features of the target tools or the content of the input query?)

2. Comprehensiveness of Tool Features: Does the instruction fully and accurately describe the features of all target tools mentioned in the query? (Are there any important features of the target tools that are missing or inadequately described in the instruction?)

3. Accuracy of Tool Feature Description: Does the instruction correctly describe the features of the target tools? (Key question to ask: Are the descriptions of the target tools technically accurate and consistent with their actual functionality?)

4. Relevance to Input Query: Is the instruction directly relevant to the original input query? (Key question to ask: Does the instruction address the specific needs or context provided in the input query, or does it deviate from the query's intent?)

## Detailed annotation Process

For each instruction, evaluate it based on the four aspects above.
1. If the instruction meets all criteria (no hallucination, comprehensive, accurate, and relevant), mark it as correct.
2. If the instruction fails to meet any of the criteria, mark it as incorrect and provide a brief explanation of the issue (e.g., "contains hallucination," "missing key tool features," or "irrelevant to query").

For incorrect instructions, ``revise'' them to ensure they meet all quality criteria. The goal of this annotation process is to ensure that all instructions in our benchmark are of high quality and faithfully grounded in the original queries and target tools.

C Large-scaling training dataset: TOOLRET-train

We extend the data collection process from TOOLRET to incorporate the training sets of three mainstream tool-use datasets: ToolACE (Liu et al., 2024a), ToolBench (Qin et al., 2023), and APIGen (Liu et al.,
2024b). These datasets are selected for their diversity in task types, tool categories, and query complexity, ensuring a comprehensive representation of real-world tool-use scenarios.

After collecting and preprocessing the data, we ultimately collect over 200k training instances. Each instance consists of a query and a corresponding set of target tools. Next, we further pair each query with an instruction using our target-aware strategy (§ 3.3). This strategy generates instructions that explicitly guide the retrieval process by incorporating task-specific context and tool functionality descriptions. We report the basic statistics of TOOLRET-train in Table 7. We also show a training example of our TOOLRET-train.

# Query: Is 'https://www.apple.com' available in the Wayback Machine on September 9, 2015?

# Instruction: Given a `URL availability` task, retrieve tools that check if a given URL is archived and accessible on a specific date in the Wayback Machine.

# Target tools (labels): [{'name': 'availability', 'description': 'Checks if a given URL is archived and currently accessible in the Wayback Machine.', 'parameters': {'url': {'description': 'The URL to check for availability in the Wayback Machine .', 'type': 'str', 'de...}}}]]

# Negative tools:
    {'name': 'top_grossing_mac_apps', 'description': 'Fetches a list of the top-grossing Mac apps from the App Store.', 'parameters': {'category': {'description': 'The category ID for the apps to be fetched. Defaults to '6016' (general category).', 'type': 'str', 'default': '6016'}, 'country': {'descript...}, {'name': 'top_paid_mac_apps', 'description': 'Retrieves a list of the top paid Mac apps from the App Store.', 'parameters': {'category': {'description':...}, ...
    {'name': 'exact_url_non_english', 'description': 'Retrieves the backlinks of a specific non-English URL using the RapidAPI service...}'}

<table>
  <tr>
    <th>Statistic</th>
    <th></th>
  </tr>
  <tr>
    <td># size of retrieval task</td>
    <td>205,826</td>
  </tr>
  <tr>
    <td># Average token length of the input query</td>
    <td>52.87</td>
  </tr>
  <tr>
    <td># Average token length of the paired instruction</td>
    <td>46.72</td>
  </tr>
  <tr>
    <td># Average token length of the tool documentation</td>
    <td>163.52</td>
  </tr>
  <tr>
    <td># Number of negative tools per input query</td>
    <td>5</td>
  </tr>
  <tr>
    <td># Number of target tools (labels) per input query</td>
    <td>2.31</td>
  </tr>
</table>

Table 7: Basic statistics of the collected large-scaling training set TOOLRET-train. We use the tokenizer from gpt-3.5-turbo in this work.

D  More experiment details

D.1  Baselines

We comprehensively evaluate the following mainstream retrieval models on our benchmark, including:

• Sparse Retrieval. These methods measure the similarity between tasks and tool documentation based on lexical overlap. We evaluate BM25s (Lü, 2024).

• Single-task dense retrieval. These methods employ dual-encoder architecture models trained on conventional IR datasets. We evaluate gtr (Ni et al., 2021a), contriever (Izacard et al., 2021a), and colbertv2.0 (Santhanam et al., 2021a), all trained on MS-MARCO (Nguyen et al., 2016) with relevance criteria. We also evaluate the COLT (Qu et al., 2024a) which is a recently proposed model trained on an ad-hoc tool retrieval dataset.
<table>
  <tr>
    <th colspan="2">Example of our seed instructions (handcrafted instruction)</th>
  </tr>
  <tr>
    <td># Query: I would like to generate a video presenting a text-based discussion on the topic of 'The Benefits of Exercise'.</td>
    <td># Instruction: Given a "text-to-video" task, retrieve tools that process text inputs to generate coherent textual outputs aligned with the query’s topic and requirements.</td>
  </tr>
  <tr>
    <td># Query: I have an audio file ‘example.wav’ which is difficult to understand. I would like you to help me transcribe the audio to text.</td>
    <td># Instruction: Given a "audio transcription" task, retrieve tools that process audio inputs to produce accurate textual transcriptions aligned with the query’s requirements.</td>
  </tr>
  <tr>
    <td># Query: Conduct a two-sample independent t-test with two samples, sample1=[1, 2, 3, 4, 5] and sample2=[6, 7, 8, 9, 10], and a significance level of 0.05.</td>
    <td># Instruction: Given a "significance test" task, retrieve tools that perform statistical tests, specifically a two-sample independent t-test, by processing numerical inputs and returning the t-statistic, p-value.</td>
  </tr>
  <tr>
    <td># Query: Can you cancel a timer for my smart device?</td>
    <td># Instruction: Given a "timer cancellation" task, retrieve tools that handle smart device operations by processing device ID and switch time inputs to cancel a scheduled action and return the status of the operation.</td>
  </tr>
  <tr>
    <td># Query: Find cruise tickets from Fontana to Santa Rosa on date 2023-07-04.</td>
    <td># Instruction: Given a "ticket booking" task, retrieve tools that support booking cruise tickets by processing travel details such as departure location, destination, date, and time.</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">Example of our generated instructions (high-quality instructions)</th>
  </tr>
  <tr>
    <td># Query: Suppose that \( f(x) = 4x + 5 \). What is \( f^{-1}(f^{-1}(9)) \)?</td>
    <td># Instruction: Given a "inverse function calculation" task, retrieve tools that calculate the value of the repeated inverse for a linear function by processing coefficients, constants, and target values to determine the result.</td>
  </tr>
  <tr>
    <td># Query: Identify an function that can classify images and works with spiking neural networks.</td>
    <td># Instruction: Given an "image classification" task, retrieve tools that execute image classification by using spiking neural network models and processing image inputs.</td>
  </tr>
  <tr>
    <td># Query: I need to find a grocery store near 123 Main Street, Downtown District that has a good selection of limes for my Easter celebration.</td>
    <td># Instruction": "Given a "local grocery search" task, retrieve tools that can locate grocery stores based on the user’s specified location and criteria, such as the availability of specific items like limes, to meet the query’s requirements.</td>
  </tr>
  <tr>
    <td># Query: Please help me find a recipe with no more than 40 grams of carbohydrates per gram and at least 5 grams of protein per gram.</td>
    <td># Instruction: "Given a "nutritional recipe search" task, retrieve tools that can find recipes based on specific nutritional criteria such as carbohydrate and protein content.</td>
  </tr>
  <tr>
    <td># Query: What is 64277 times 38142?</td>
    <td># Instruction: Given a "multiplication" task, retrieve tools that compute the product of two numbers by processing numerical inputs and returning the result.</td>
  </tr>
  <tr>
    <td># Query: Can I get a list of all boards and their attributes on page number two with a page size of seven?</td>
    <td># Instruction: Given a "pagination query" task, retrieve tools that can list boards and their attributes by processing parameters such as page number and page size to return the requested information.</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">Example of our generated instructions (low-quality instructions)</th>
  </tr>
  <tr>
    <td># Query: I would like to generate a video presenting a text-based discussion on the topic of 'The Benefits of Exercise'</td>
    <td># Instruction: Given a text-to-video task, please retrieve relevant tools to generate video about exercise.<br><span style="color:red">// Too general to cover the key features.</span></td>
  </tr>
  <tr>
    <td># Query: Can you assist me in finding a 1-bedroom townhouse or condo in Little Rock with max rent 1541000? I want it on the sixth floor with 7 balconies.</td>
    <td># Revised version: Given a 'text-to-video' task, retrieve tools related to general video scripting, exercise video libraries or tools process text data. <span style="color:red">// More specific to the query and target tools.</span></td>
  </tr>
  <tr>
    <td># Query: Please retrieve tools find 1-bedroom or condo in Little Rock.</td>
    <td># Revised version: Given a property search task, retrieve tools that can find rental properties based on location, property type, rent budget, and specific features or requirements. <span style="color:red">// Cover all key points and related to tools functionality.</span></td>
  </tr>
</table>

Table 8: Example of the instruction in TOOLRet. We show the handcrafted instruction by human experts, the high-quality instruction and low-quality instruction generated by GPT-4o, respectively. We also show the revised version of the low-quality instruction paired with <span style="color:red">the reason for revision</span>.
<table>
  <tr>
    <th>Name</th>
    <th>Public Link of Endpoint</th>
  </tr>
  <tr>
    <th colspan="2">Conventional sparse and dense models</th>
  </tr>
  <tr>
    <td>BM25S (Li, 2024)</td>
    <td>https://github.com/xhluca/bm25s</td>
  </tr>
  <tr>
    <td>contriever (Izacard et al., 2021b)</td>
    <td>https://huggingface.co/facebook/contriever-msmarco</td>
  </tr>
  <tr>
    <td>ColBERTv2 (Santhanam et al., 2021b)</td>
    <td>https://github.com/stanford-futuredata/ColBERT</td>
  </tr>
  <tr>
    <td>gtr-t5-base (Ni et al., 2021b)</td>
    <td>https://huggingface.co/sentence-transformers/gtr-t5-base</td>
  </tr>
  <tr>
    <td>gtr-t5-large (Ni et al., 2021b)</td>
    <td>https://huggingface.co/sentence-transformers/gtr-t5-large</td>
  </tr>
  <tr>
    <th colspan="2">Multi-task embedding models</th>
  </tr>
  <tr>
    <td>all-MiniLM-L6-v2</td>
    <td>https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2</td>
  </tr>
  <tr>
    <td>e5-small-v2 (Wang et al., 2022)</td>
    <td>https://huggingface.co/intfloat/e5-small-v2</td>
  </tr>
  <tr>
    <td>e5-base-v2 (Wang et al., 2022)</td>
    <td>https://huggingface.co/intfloat/e5-base-v2</td>
  </tr>
  <tr>
    <td>e5-large-v2 (Wang et al., 2022)</td>
    <td>https://huggingface.co/intfloat/e5-large-v2</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5 (Xiao et al., 2023b)</td>
    <td>https://huggingface.co/BAAI/bge-base-en-v1.5</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5 (Xiao et al., 2023b)</td>
    <td>https://huggingface.co/BAAI/bge-large-en-v1.5</td>
  </tr>
  <tr>
    <td>gte-Qwen2-1.5B-inst. (Li et al., 2023c)</td>
    <td>https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct</td>
  </tr>
  <tr>
    <td>e5-mistral-7b (Wang et al., 2023)</td>
    <td>https://huggingface.co/intfloat/e5-mistral-7b-instruct</td>
  </tr>
  <tr>
    <td>GritLM-7B (Muenchhoff et al., 2024)</td>
    <td>https://huggingface.co/GritLM/GritLM-7B</td>
  </tr>
  <tr>
    <td>NV-Embed-v1 (Lee et al., 2024)</td>
    <td>https://huggingface.co/nvidia/NV-Embed-v1</td>
  </tr>
  <tr>
    <th colspan="2">Cross-encoder re-ranking</th>
  </tr>
  <tr>
    <td>mxbai-rerank-large-v1</td>
    <td>https://huggingface.co/mixedread-ai/mxbai-rerank-large-v1</td>
  </tr>
  <tr>
    <td>monoT5-base (Nogueira et al., 2020)</td>
    <td>https://huggingface.co/castorini/monoT5-base-med-msmarco</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-m3 (Li et al., 2023a; Chen et al., 2024a)</td>
    <td>https://huggingface.co/BAAI/bge-reranker-v2-m3</td>
  </tr>
  <tr>
    <td>jina-reranker-v2</td>
    <td>https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-gemma (Li et al., 2023a; Chen et al., 2024a)</td>
    <td>https://huggingface.co/BAAI/bge-reranker-v2-gemma</td>
  </tr>
  <tr>
    <th colspan="2">LLM agent</th>
  </tr>
  <tr>
    <td>RankGPT</td>
    <td>https://github.com/sunweilei/RankGPT</td>
  </tr>
  <tr>
    <td>- Mixtral-8x22B</td>
    <td>https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1</td>
  </tr>
  <tr>
    <td>- GPT-3.5-turbo-1106</td>
    <td>https://openai.com/chatgpt/overview/</td>
  </tr>
  <tr>
    <td>- GPT-3.5-turbo-0125</td>
    <td>https://openai.com/chatgpt/overview/</td>
  </tr>
</table>

Table 9: The public link or endpoint of the baselines in our experiments.

• Multi-task Embedding Models. These methods utilize transformer encoders trained on various annotated IR datasets. We evaluate gte (Li et al., 2023c), bge (Xiao et al., 2023a), and e5 (Wang et al., 2022), covering a wide range of parameter sizes. Additionally, we evaluate all-MiniLM-L6-v26 from the Sentence Transformers platform.

• Cross-encoder Re-rankers. These models re-rank the initially retrieved documents based on the query-passage relevance using bidirectional or unidirectional transformers. We evaluate MonoT5-Base and three re-rankers trained on diverse tasks: (i) mxbai-rerank-large-v17, (ii) jina-reranker-v2-base8, and (iii) BGE-reranker.

• LLM Agents. These methods leverage general-purpose LLM agents for re-ranking tasks in a zero-shot setting, simulating the tool selection process of tool-use agents. We evaluate the widely used LLM re-ranking framework, i.e., RankGPT (Sun et al., 2023), with various LLMs as backbone.

We highlight that the initial tools for LLM agent and Re-ranking baselines are retrieved by NV-embedd-v1 model. Details about these baselines are provided in Table 9.

D.2 Compare with conventional IR tasks

To further investigate the complexity of tool retrieval tasks, we conducted a comparative analysis of model performance between our proposed benchmark (TOOLRET) and the conventional Information Retrieval (IR) task benchmark, specifically the Massive Text Embedding Benchmark (MTEB). The relationship between these two benchmarks is visually presented in Figure 7. Our analysis reveals two significant findings.

First , we observe a strong positive correlation between the two benchmarks, as evidenced by a Pearson’s correlation coefficient of \( \beta = 0.790 \), indicating a similar performance trend across models. However, we observe that the absolute performance scores in TOOLRET are consistently lower than those in the

6 https://huggingface.co/all-MiniLM-L6-v2
7 https://huggingface.co/mxbai-rerank-large-v1
8 https://huggingface.co/jina-reranker-v2-base-multilingual
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">REST API</th>
    <th colspan="6">Code Function</th>
    <th colspan="6">Customized tool</th>
    <th colspan="3">Avg.</th>
  </tr>
  <tr>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
  </tr>
  <tr>
    <th colspan="21"><i>Conventional sparse and dense models</i></th>
  </tr>
  <tr>
    <td>BM25s</td>
    <td>62.09</td><td>15.68</td><td>72.98</td><td>58.06</td><td>56.98</td><td>8.24</td><td>73.95</td><td>72.81</td><td>68.48</td><td>14.78</td><td>80.51</td><td>69.55</td><td>62.51</td><td>66.81</td>
  </tr>
  <tr>
    <td>ColBERT</td>
    <td>56.73</td><td>14.59</td><td>67.86</td><td>47.87</td><td>53.56</td><td>7.82</td><td>71.66</td><td>70.09</td><td>64.49</td><td>13.05</td><td>76.35</td><td>64.85</td><td>58.26</td><td>60.94</td>
  </tr>
  <tr>
    <td>contriever-msmarco</td>
    <td>57.81</td><td>15.33</td><td>70.77</td><td>50.87</td><td>49.66</td><td>7.29</td><td>65.99</td><td>64.56</td><td>68.86</td><td>14.67</td><td>83.05</td><td>73.45</td><td>58.77</td><td>62.96</td>
  </tr>
  <tr>
    <td>gtr-t5-base</td>
    <td>57.06</td><td>14.54</td><td>68.26</td><td>49.75</td><td>49.38</td><td>7.29</td><td>65.76</td><td>64.09</td><td>70.48</td><td>14.29</td><td>81.60</td><td>70.96</td><td>58.97</td><td>61.60</td>
  </tr>
  <tr>
    <td>gtr-t5-large</td>
    <td>60.32</td><td>15.27</td><td>72.05</td><td>54.03</td><td>52.79</td><td>7.41</td><td>67.28</td><td>65.79</td><td>72.03</td><td>14.69</td><td>84.39</td><td>73.95</td><td>61.72</td><td>64.59</td>
  </tr>
  <tr>
    <th colspan="21"><i>Embedding models</i></th>
  </tr>
  <tr>
    <td>all-MiniLM-L6-v2</td>
    <td>53.92</td><td>14.47</td><td>66.77</td><td>48.64</td><td>50.14</td><td>7.58</td><td>68.39</td><td>66.74</td><td>68.31</td><td>14.20</td><td>81.57</td><td>71.14</td><td>57.46</td><td>62.17</td>
  </tr>
  <tr>
    <td>e5-small-v2</td>
    <td>61.95</td><td>15.84</td><td>72.91</td><td>53.34</td><td>51.45</td><td>7.76</td><td>68.16</td><td>65.46</td><td>69.13</td><td>14.24</td><td>79.69</td><td>68.33</td><td>60.85</td><td>62.38</td>
  </tr>
  <tr>
    <td>e5-base-v2</td>
    <td>62.90</td><td>15.90</td><td>73.98</td><td>53.83</td><td>55.81</td><td>8.44</td><td>74.17</td><td>72.53</td><td>69.96</td><td>14.98</td><td>83.63</td><td>73.94</td><td>62.89</td><td>66.77</td>
  </tr>
  <tr>
    <td>e5-large-v2</td>
    <td>61.72</td><td>15.90</td><td>73.27</td><td>52.84</td><td>56.21</td><td>8.42</td><td>75.25</td><td><b>73.14</b></td><td>69.88</td><td><b>15.01</b></td><td>81.13</td><td><b>71.30</b></td><td>62.60</td><td>65.76</td>
  </tr>
  <tr>
    <td>gte-large-en-v1.5</td>
    <td>64.35</td><td><b>16.55</b></td><td>75.80</td><td>57.38</td><td><b>59.18</b></td><td><b>8.77</b></td><td><b>76.95</b></td><td><b>74.45</b></td><td>71.79</td><td>14.53</td><td>81.90</td><td>70.07</td><td>65.11</td><td>67.30</td>
  </tr>
  <tr>
    <td>gte-large-en-v1.5</td>
    <td>60.67</td><td>15.46</td><td>72.30</td><td>52.41</td><td>54.11</td><td>8.22</td><td>73.35</td><td>71.37</td><td>68.59</td><td>14.36</td><td>80.41</td><td>69.82</td><td>61.12</td><td>64.53</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5</td>
    <td>65.05</td><td>16.37</td><td>75.72</td><td>57.30</td><td>54.55</td><td>7.72</td><td>69.22</td><td>67.48</td><td>71.21</td><td>14.71</td><td>83.13</td><td>72.53</td><td>63.60</td><td>65.77</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5</td>
    <td><b>66.25</b></td><td>16.48</td><td><b>75.84</b></td><td><b>57.75</b></td><td>58.61</td><td>8.41</td><td>74.91</td><td>72.74</td><td><b>71.19</b></td><td>14.20</td><td>80.44</td><td>69.27</td><td><b>65.35</b></td><td>66.59</td>
  </tr>
  <tr>
    <td>gte-Qwen-1.5B-inst.</td>
    <td>67.57</td><td>16.93</td><td>78.14</td><td>60.81</td><td>58.12</td><td>8.51</td><td>75.41</td><td>73.39</td><td>71.73</td><td>15.34</td><td>83.03</td><td>73.39</td><td>65.81</td><td>69.19</td>
  </tr>
  <tr>
    <td>e5-mistral-7b</td>
    <td><b>69.51</b></td><td><b>17.37</b></td><td><b>79.34</b></td><td><b>62.48</b></td><td>58.15</td><td>8.37</td><td>75.12</td><td>72.79</td><td>72.52</td><td>14.68</td><td>81.79</td><td>71.49</td><td>66.73</td><td>68.92</td>
  </tr>
  <tr>
    <td>GritLM-7B</td>
    <td>69.43</td><td>17.25</td><td>78.97</td><td>61.67</td><td>62.78</td><td>9.22</td><td>78.74</td><td>77.59</td><td><b>76.04</b></td><td>15.44</td><td>85.55</td><td>74.35</td><td><b>69.42</b></td><td>71.21</td>
  </tr>
  <tr>
    <td>NV-Embed-v1</td>
    <td>66.04</td><td>16.88</td><td>77.19</td><td>59.06</td><td><b>63.46</b></td><td><b>9.40</b></td><td><b>81.79</b></td><td><b>79.82</b></td><td>75.39</td><td><b>15.75</b></td><td><b>88.48</b></td><td><b>78.37</b></td><td>68.30</td><td><b>72.42</b></td>
  </tr>
  <tr>
    <th colspan="21"><i>Cross-encoder re-ranking models</i></th>
  </tr>
  <tr>
    <td>mxbai-rerank-large-v1</td>
    <td>57.48</td><td>14.60</td><td>68.65</td><td>49.54</td><td>50.37</td><td>7.75</td><td>69.59</td><td>67.88</td><td>62.24</td><td>13.32</td><td>73.61</td><td>62.14</td><td>56.70</td><td>59.55</td>
  </tr>
  <tr>
    <td>monot5-base-msmarco</td>
    <td>54.57</td><td>14.23</td><td>64.38</td><td>46.12</td><td>50.00</td><td>8.05</td><td>68.76</td><td>66.80</td><td>64.50</td><td>13.28</td><td>75.80</td><td>67.84</td><td>56.36</td><td>60.25</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-m3</td>
    <td>70.42</td><td>17.75</td><td>80.33</td><td>65.49</td><td>64.22</td><td>9.37</td><td>80.60</td><td>79.48</td><td>75.70</td><td>16.15</td><td>88.87</td><td>78.65</td><td>70.11</td><td>74.54</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-gemma</td>
    <td><b>75.67</b></td><td><b>18.63</b></td><td><b>84.07</b></td><td><b>71.00</b></td><td><b>69.59</b></td><td><b>9.78</b></td><td><b>84.18</b></td><td><b>83.47</b></td><td><b>77.17</b></td><td><b>16.55</b></td><td><b>88.34</b></td><td><b>79.80</b></td><td><b>74.14</b></td><td><b>78.09</b></td>
  </tr>
</table>

Table 10: Results of control experiment where each IR models is evaluated from the toolset of each integrated dataset in w/ inst. setting.

![Correlation between the score on our benchmark and MTEB (retrieval subset).](page_1012_1342_484_388.png)

Figure 7: Correlation between the score on our benchmark and MTEB (retrieval subset).

conventional IR benchmark. This discrepancy suggests that while our benchmark shares fundamental characteristics with conventional IR tasks, it presents additional challenges that make it more demanding for existing models.

Second , our experimental results demonstrate that state-of-the-art IR models, particularly those trained with relevance-oriented optimization criteria (e.g., Contriever), exhibit substantially degraded performance on TOOLRET. This performance gap underscores the necessity for target-aware reasoning capabilities in our benchmark, which goes beyond traditional relevance matching. The unique challenges of TOOLRET are further elaborated in § 4.1, where we identify two key distinguishing factors: (1) the presence of
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">TOOLRET-Web</th>
    <th colspan="6">TOOLRET-Code</th>
    <th colspan="6">TOOLRET-Customized</th>
    <th colspan="6">Avg.</th>
  </tr>
  <tr>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
  </tr>
  <tr>
    <td colspan="24"><i>Conventional sparse and dense models</i></td>
  </tr>
  <tr>
    <td>BM25S</td><td>51.79</td><td>13.78</td><td>63.35</td><td>45.46</td><td>38.74</td><td>5.87</td><td>52.65</td><td>51.39</td><td>59.72</td><td>13.55</td><td>71.83</td><td>60.38</td><td>50.08</td><td>52.41</td>
  </tr>
  <tr>
    <td>ColBERT</td><td>51.70</td><td>13.56</td><td>61.92</td><td>41.01</td><td>38.60</td><td>6.05</td><td>55.07</td><td>54.05</td><td>53.91</td><td>12.10</td><td>66.85</td><td>55.29</td><td>48.07</td><td>50.11</td>
  </tr>
  <tr>
    <td>contriever-msmarco</td><td>53.23</td><td>14.55</td><td>65.96</td><td>46.59</td><td>35.97</td><td>5.79</td><td>52.32</td><td>50.84</td><td>56.94</td><td>13.21</td><td>72.11</td><td>61.17</td><td>48.71</td><td>52.87</td>
  </tr>
  <tr>
    <td>gtr-t5-base</td><td>51.65</td><td>14.13</td><td>64.55</td><td>44.56</td><td>33.98</td><td>5.51</td><td>48.88</td><td>47.77</td><td>54.28</td><td>13.24</td><td>70.95</td><td>60.82</td><td>46.64</td><td>51.05</td>
  </tr>
  <tr>
    <td>gtr-t5-large</td><td><b>56.62</b></td><td><b>15.06</b></td><td><b>69.77</b></td><td><b>50.26</b></td><td>37.40</td><td>5.85</td><td>52.41</td><td>51.27</td><td>56.24</td><td>13.31</td><td>71.25</td><td><b>61.40</b></td><td><b>50.09</b></td><td><b>54.31</b></td>
  </tr>
  <tr>
    <td colspan="24"><i>Embedding models</i></td>
  </tr>
  <tr>
    <td>all-MiniLM-L6-v2</td><td>48.49</td><td>13.54</td><td>62.35</td><td>43.63</td><td>34.40</td><td>5.62</td><td>50.15</td><td>48.71</td><td>58.08</td><td>13.22</td><td>72.72</td><td>62.17</td><td>46.99</td><td>51.50</td>
  </tr>
  <tr>
    <td>e5-small-v2</td><td>54.40</td><td>14.60</td><td>66.47</td><td>46.76</td><td>35.18</td><td>5.67</td><td>50.70</td><td>49.19</td><td>56.85</td><td>13.45</td><td>73.43</td><td>60.86</td><td>48.81</td><td>52.27</td>
  </tr>
  <tr>
    <td>e5-base-v2</td><td>55.42</td><td>14.92</td><td>67.90</td><td>48.10</td><td>38.35</td><td>6.23</td><td>56.08</td><td>54.90</td><td>59.96</td><td>14.18</td><td>76.18</td><td>66.55</td><td>51.24</td><td>56.52</td>
  </tr>
  <tr>
    <td>e5-large-v2</td><td>54.32</td><td>14.81</td><td>67.69</td><td>47.89</td><td>40.24</td><td>6.23</td><td>56.33</td><td>54.85</td><td>59.40</td><td>13.79</td><td>72.96</td><td>60.37</td><td>51.32</td><td>54.37</td>
  </tr>
  <tr>
    <td>gte-base-en-v1.5</td><td>56.48</td><td><b>15.40</b></td><td><b>70.07</b></td><td><b>50.96</b></td><td>39.46</td><td><b>6.27</b></td><td><b>56.58</b></td><td><b>55.24</b></td><td><b>64.00</b></td><td><b>14.23</b></td><td><b>78.03</b></td><td><b>66.93</b></td><td><b>53.31</b></td><td><b>57.71</b></td>
  </tr>
  <tr>
    <td>gte-large-en-v1.5</td><td>55.39</td><td>14.89</td><td>68.33</td><td>48.98</td><td>38.23</td><td>6.22</td><td>56.16</td><td>54.95</td><td>57.88</td><td>13.86</td><td>75.12</td><td>65.15</td><td>50.50</td><td>56.36</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5</td><td>56.17</td><td>14.86</td><td>68.30</td><td>49.05</td><td>38.71</td><td>6.05</td><td>54.35</td><td>53.08</td><td>59.40</td><td>13.93</td><td>75.38</td><td>64.81</td><td>51.43</td><td>55.65</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5</td><td><b>58.20</b></td><td>15.33</td><td>69.83</td><td>50.20</td><td><b>40.39</b></td><td>6.13</td><td>55.03</td><td>53.67</td><td>61.40</td><td>13.83</td><td>77.27</td><td>66.71</td><td><b>53.33</b></td><td>56.86</td>
  </tr>
  <tr>
    <td>gte-Qwen2-1.5B-inst.▲</td><td>60.28</td><td>15.64</td><td>72.60</td><td>53.82</td><td>44.06</td><td>6.56</td><td>59.29</td><td>57.78</td><td>65.57</td><td>14.79</td><td><b>80.33</b></td><td><b>70.59</b></td><td>56.64</td><td>60.73</td>
  </tr>
  <tr>
    <td>e5-mistral-7b▲</td><td>60.78</td><td>15.93</td><td>73.12</td><td>55.60</td><td>44.20</td><td>6.77</td><td>61.18</td><td>59.79</td><td>60.56</td><td>13.80</td><td>74.47</td><td>63.38</td><td>55.18</td><td>59.59</td>
  </tr>
  <tr>
    <td>GritLM-7B▲</td><td><b>62.54</b></td><td><b>16.07</b></td><td>73.82</td><td>54.46</td><td>46.80</td><td>6.98</td><td>62.89</td><td>61.35</td><td><b>67.61</b></td><td><b>14.93</b></td><td>80.04</td><td>68.41</td><td>58.98</td><td>61.41</td>
  </tr>
  <tr>
    <td>NV-Embed-v1▲</td><td>61.76</td><td>16.02</td><td><b>73.86</b></td><td><b>55.96</b></td><td><b>50.38</b></td><td><b>7.54</b></td><td><b>67.93</b></td><td><b>66.03</b></td><td>67.01</td><td>14.61</td><td>79.70</td><td>69.74</td><td><b>59.72</b></td><td><b>63.91</b></td>
  </tr>
  <tr>
    <td colspan="24"><i>Cross-encoder re-ranking models</i></td>
  </tr>
  <tr>
    <td>mxbai-rerank-large-v1</td><td>56.45</td><td>14.51</td><td>67.51</td><td>49.02</td><td>42.55</td><td>6.28</td><td>57.85</td><td>55.96</td><td>54.63</td><td>13.03</td><td>72.68</td><td>60.20</td><td>51.21</td><td>55.06</td>
  </tr>
  <tr>
    <td>monot5-base-msmarco</td><td>56.10</td><td>14.82</td><td>65.29</td><td>47.12</td><td>41.05</td><td>6.31</td><td>57.20</td><td>55.14</td><td>64.60</td><td>13.91</td><td>75.79</td><td>65.76</td><td>53.92</td><td>56.01</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-m3</td><td>61.78</td><td>15.94</td><td>72.35</td><td>55.61</td><td>45.15</td><td>6.74</td><td>61.03</td><td>59.56</td><td>62.45</td><td>14.86</td><td>79.65</td><td>68.68</td><td>56.46</td><td>61.28</td>
  </tr>
  <tr>
    <td>jina-reranker-v2-base</td><td><b>65.86</b></td><td><b>17.14</b></td><td><b>77.54</b></td><td><b>62.33</b></td><td>47.23</td><td>7.01</td><td>63.50</td><td>62.24</td><td><b>69.10</b></td><td>15.38</td><td>81.29</td><td>69.63</td><td>60.73</td><td>64.73</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-gemma</td><td>65.80</td><td>16.87</td><td>76.85</td><td>61.40</td><td><b>52.49</b></td><td><b>7.60</b></td><td><b>68.14</b></td><td><b>65.94</b></td><td>67.87</td><td><b>15.63</b></td><b>81.98</b></td><td><b>72.24</b></td><td><b>62.05</b></td><td><b>66.53</b></td>
  </tr>
</table>

Table 11: Results of control experiment where each IR models is evaluated from the toolset of each integrated dataset in w/o inst. setting.

multiple potential target tools for each query, and (2) significantly lower term overlap between input queries and relevant tools compared to conventional IR scenarios. These characteristics collectively contribute to a more complex retrieval environment that requires advanced reasoning and understanding capabilities from retrieval models.

D.3 Results of controlled experiment

Since TOOLRET integrates multiple datasets, we also conduct controlled experiments where IR models retrieve tools exclusively within the toolset of each individual dataset instead of the overall tool corpus. Table 11 presents the results under the setting that the IR models only take the query to retrieve, i.e., the w/o inst setting. Table 10 presents the results under the setting that the IR models take the query and additional instruction to retrieve, i.e., the w/ inst setting.

D.4 Results of in-subset retrieval

TOOLRET contains three subsets, including TOOLRET-web, TOOLRET-code and TOOLRET-customized. The tool in each subset diverges by its documentation format, domain, and functionality. For a comprehensive evaluation, we also conduct an in-subset retrieval experiment, where IR models retrieve tools exclusively within the toolset of each subset instead of the overall tool corpus. Table 13 presents the results under the setting that the IR models only take the query to retrieve, i.e., the w/o inst setting. Table 12 presents the results under the setting that the IR models take the query and additional instruction to retrieve, i.e., the w/ inst setting.

D.5 Results of trained IR models

Experimental results on TOOLRET reveal that even IR models with strong performance on conventional IR benchmarks such as MTEB and BEIR struggle significantly in tool retrieval tasks. A key factor contributing to this performance degradation is the lack of a large-scale training dataset specifically tailored for tool retrieval. To address this gap, we introduce TOOLRET-train, a diverse training dataset comprising more than 200k tool retrieval tasks. Each example in TOOLRET-train consists of an input query, an instruction generated using our target-aware strategy, the corresponding target tools, and a set of
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">TOOLRET-Web</th>
    <th colspan="4">TOOLRET-Code</th>
    <th colspan="4">TOOLRET-Customized</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
  </tr>
  <tr>
    <td colspan="15"><b>Sparse and dense models</b></td>
  </tr>
  <tr>
    <td>bm25</td><td>49.01</td><td>7.17</td><td>64.96</td><td>63.68</td><td>28.92</td><td>6.78</td><td>37.09</td><td>24.44</td><td>51.28</td><td>10.66</td><td>60.70</td><td>48.40</td><td>43.07</td><td>45.51</td>
  </tr>
  <tr>
    <td>COLT</td><td>36.58</td><td>5.74</td><td>50.84</td><td>49.04</td><td>21.98</td><td>5.12</td><td>29.68</td><td>20.03</td><td>46.02</td><td>9.12</td><td>58.02</td><td>45.27</td><td>34.86</td><td>38.12</td>
  </tr>
  <tr>
    <td>Colbert</td><td>43.80</td><td>6.40</td><td>58.02</td><td>56.28</td><td>16.60</td><td>3.05</td><td>20.85</td><td>14.95</td><td>31.18</td><td>5.86</td><td>39.40</td><td>32.10</td><td>30.53</td><td>34.44</td>
  </tr>
  <tr>
    <td>contriever-msmarco</td><td>35.78</td><td>5.31</td><td>47.17</td><td>46.07</td><td>25.19</td><td>5.67</td><td>31.95</td><td>20.56</td><td>44.37</td><td>9.35</td><td>57.53</td><td>46.80</td><td>35.11</td><td>37.81</td>
  </tr>
  <tr>
    <td>gtr-t5-base</td><td>37.45</td><td>5.50</td><td>48.78</td><td>47.44</td><td>22.54</td><td>4.93</td><td>29.64</td><td>20.60</td><td>51.02</td><td>10.28</td><td>61.08</td><td>49.06</td><td>37.00</td><td>39.03</td>
  </tr>
  <tr>
    <td>gtr-t5-large</td><td>42.14</td><td>5.98</td><td>53.59</td><td>52.19</td><td>26.60</td><td>5.71</td><td>33.68</td><td>22.38</td><td>53.95</td><td>11.17</td><td>66.08</td><td>52.21</td><td>40.90</td><td>42.26</td>
  </tr>
  <tr>
    <td colspan="15"><b>Embedding models</b></td>
  </tr>
  <tr>
    <td>all-MiniLM-L6-v2</td><td>36.93</td><td>5.65</td><td>49.54</td><td>47.92</td><td>15.89</td><td>3.89</td><td>22.68</td><td>15.24</td><td>43.09</td><td>9.29</td><td>56.84</td><td>43.96</td><td>31.97</td><td>35.71</td>
  </tr>
  <tr>
    <td>e5-small-v2</td><td>38.22</td><td>5.55</td><td>49.34</td><td>48.07</td><td>28.97</td><td>6.81</td><td>36.96</td><td>23.45</td><td>47.59</td><td>9.78</td><td>58.19</td><td>45.31</td><td>38.26</td><td>38.94</td>
  </tr>
  <tr>
    <td>e5-base-v2</td><td>40.69</td><td>6.51</td><td>55.38</td><td>54.01</td><td>28.43</td><td>6.59</td><td>37.14</td><td>23.67</td><td>47.89</td><td>9.48</td><td>59.01</td><td>46.91</td><td>39.00</td><td>41.53</td>
  </tr>
  <tr>
    <td>e5-large-v2</td><td>40.14</td><td>5.87</td><td>52.23</td><td>50.80</td><td>26.88</td><td>6.16</td><td>35.65</td><td>24.31</td><td>51.40</td><td>10.65</td><td>61.45</td><td>48.28</td><td>39.47</td><td>41.13</td>
  </tr>
  <tr>
    <td>gte-base-en-v1.5*</td><td>48.25</td><td>7.16</td><td>61.96</td><td>59.17</td><td>33.28</td><td>7.57</td><td>41.99</td><td>27.20</td><td>50.33</td><td>9.70</td><td>62.05</td><td>50.09</td><td>43.95</td><td>45.49</td>
  </tr>
  <tr>
    <td>gte-large-en-v1.5*</td><td>40.48</td><td>6.46</td><td>56.34</td><td>54.52</td><td>30.58</td><td>7.00</td><td>38.79</td><td>24.78</td><td>49.24</td><td>9.98</td><td>59.13</td><td>47.20</td><td>40.10</td><td>42.17</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5</td><td>43.74</td><td>6.43</td><td>57.33</td><td>55.85</td><td>29.83</td><td>6.96</td><td>38.86</td><td>25.67</td><td>52.41</td><td>10.75</td><td>63.84</td><td>51.19</td><td>41.99</td><td>44.24</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5</td><td>44.07</td><td>6.44</td><td>56.78</td><td>55.22</td><td>33.88</td><td>7.90</td><td>43.11</td><td>28.62</td><td>53.48</td><td>10.53</td><td>63.66</td><td>52.00</td><td>43.81</td><td>45.28</td>
  </tr>
  <tr>
    <td>gte-Qwen2-1.5B-inst.*</td><td>47.29</td><td>7.10</td><td>61.89</td><td>59.98</td><td>39.30</td><td>9.90</td><td>48.58</td><td>29.51</td><td>55.56</td><td>11.67</td><td>65.55</td><td>51.75</td><td>47.38</td><td>47.08</td>
  </tr>
  <tr>
    <td>e5-mistral-7b*</td><td>48.76</td><td>7.16</td><td>63.57</td><td>61.40</td><td>33.06</td><td>8.14</td><td>43.56</td><td>28.49</td><td>57.16</td><td>11.43</td><td>67.28</td><td>52.62</td><td>46.32</td><td>47.51</td>
  </tr>
  <tr>
    <td>GrimT-7B</td><td>53.69</td><td>8.13</td><td>68.70</td><td>67.26</td><td>41.59</td><td>10.06</td><td>51.69</td><td>33.87</td><td>60.14</td><td>11.63</td><td>68.93</td><td>54.60</td><td>51.81</td><td>51.91</td>
  </tr>
  <tr>
    <td>NY-Embed-v1*</td><td>51.95</td><td>7.62</td><td>66.58</td><td>64.21</td><td>34.42</td><td>8.40</td><td>43.66</td><td>29.65</td><td>57.93</td><td>12.34</td><td>71.14</td><td>57.47</td><td>48.10</td><td>50.44</td>
  </tr>
  <tr>
    <td colspan="15"><b>Cross-encoder re-ranking models</b></td>
  </tr>
  <tr>
    <td>mxbai-rerank-large-v1</td><td>33.39</td><td>5.09</td><td>46.75</td><td>45.11</td><td>24.90</td><td>5.95</td><td>32.20</td><td>19.52</td><td>35.62</td><td>7.35</td><td>43.96</td><td>34.14</td><td>31.30</td><td>32.92</td>
  </tr>
  <tr>
    <td>monot5-base-msmarco</td><td>29.95</td><td>5.20</td><td>42.47</td><td>40.36</td><td>30.20</td><td>7.84</td><td>37.94</td><td>21.43</td><td>46.99</td><td>9.26</td><td>57.10</td><td>46.48</td><td>35.71</td><td>36.09</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-m3</td><td>56.49</td><td>8.42</td><td>72.26</td><td>70.67</td><td>38.09</td><td>9.25</td><td>48.14</td><td>33.48</td><td>54.66</td><td>12.46</td><td>70.79</td><td>56.17</td><td>49.75</td><td>53.44</td>
  </tr>
  <tr>
    <td>jina-reranker-v2-base</td><td>35.24</td><td>5.48</td><td>47.70</td><td>46.31</td><td>36.23</td><td>9.21</td><td>45.59</td><td>29.26</td><td>54.12</td><td>11.92</td><td>65.09</td><td>51.18</td><td>41.86</td><td>42.25</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-gemma</td><td>62.84</td><td>8.87</td><td>76.24</td><td>74.86</td><td>37.13</td><td>8.62</td><td>47.23</td><td>33.45</td><td>64.33</td><td>13.72</td><td>77.37</td><td>62.08</td><td>54.76</td><td>56.79</td>
  </tr>
</table>

Table 12: Experiments are conducted under the w/ inst. setting, with retrieval performed within each subset individually.

negative tools. IR models are trained to distinguish target tools from negative tools (§ 7). We evaluate these trained IR models on TOOLRET and present the results in Table 14.

D.6 Improved IR enhances tool-use LLMs

We further investigate the impact of improved IR models on the end-to-end performance of tool-use LLMs. Specifically, we evaluate tool-use LLMs on the ToolBench (Qin et al., 2023) dataset using the official Pass Rate metric, which measures whether the model successfully invokes the correct tools to complete a given task.

For each task in ToolBench, we replace the pre-annotated toolset (oracle) with tools retrieved by IR models from TOOLRET’ tool corpus, which contains 43,000 tools. Since TOOLRET integrates the ToolBench dataset, we can compute NDCG@10 for this retrieval step. For a comprehensive evaluation, we assess two widely used tool-use LLMs, including GPT-3.5 and ToolLLaMA (Qin et al., 2023).

Table 15 presents the retrieval NDCG@10 scores alongside the corresponding pass rates on ToolBench.9 Our results demonstrate that LLM agents equipped with improved IR models achieve substantial gains in pass rate, highlighting the critical role of accurate tool retrieval in downstream task performance. Furthermore, Figure 6 visually illustrates a positive correlation between improved IR performance and higher task pass rate, suggesting that better retrieval directly leads to improved downstream outcomes.

Based on this analysis, we propose that future work could explore the following two directions: (i) Further optimize IR models to enhance tool retrieval performance; or (ii) Adapt IR models by incorporating feedback from end-to-end task performance, allowing them to better support tool-use LLMs. These approaches provide a more efficient plug-and-play solution compared to fine-tuning LLMs, enabling flexible integration into diverse tool-use systems.

9ToolBench consists of three subsets: ToolBench-G1, ToolBench-G2, and ToolBench-G3.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">TOOLRET-Web</th>
    <th colspan="6">TOOLRET-Code</th>
    <th colspan="6">TOOLRET-Customized</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
  </tr>
  <tr>
    <th colspan="21">Sparse and dense models</th>
  </tr>
  <tr>
    <td>bm25</td><td>26.47</td><td>4.04</td><td>34.51</td><td>33.22</td><td>20.61</td><td>5.06</td><td>26.35</td><td>15.87</td><td>38.48</td><td>8.51</td><td>48.24</td><td>37.09</td><td>28.52</td><td>28.73</td>
  </tr>
  <tr>
    <td>COLT</td><td>22.23</td><td>3.75</td><td>31.66</td><td>30.15</td><td>21.65</td><td>5.36</td><td>29.15</td><td>19.12</td><td>36.12</td><td>7.99</td><td>47.63</td><td>37.64</td><td>26.67</td><td>28.97</td>
  </tr>
  <tr>
    <td>Colbert</td><td>23.58</td><td>3.84</td><td>32.64</td><td>31.19</td><td>23.50</td><td>5.70</td><td>28.82</td><td>16.03</td><td>33.71</td><td>6.64</td><td>40.85</td><td>32.01</td><td>26.93</td><td>26.41</td>
  </tr>
  <tr>
    <td>contriever-msmarco</td><td>19.11</td><td>3.16</td><td>26.52</td><td>24.97</td><td>21.84</td><td>5.93</td><td>27.83</td><td>15.19</td><td>35.01</td><td>7.57</td><td>44.07</td><td>34.71</td><td>25.32</td><td>24.96</td>
  </tr>
  <tr>
    <td>gtr-t5-base</td><td>21.19</td><td>3.48</td><td>29.05</td><td>27.90</td><td>18.18</td><td>4.45</td><td>25.09</td><td>16.32</td><td>35.95</td><td>8.10</td><td>45.77</td><td>36.50</td><td>25.11</td><td>26.91</td>
  </tr>
  <tr>
    <td>gtr-t5-large</td><td>24.48</td><td>3.91</td><td>33.55</td><td>32.44</td><td>23.49</td><td>5.64</td><td>30.79</td><td>19.31</td><td>38.88</td><td>8.89</td><td>49.55</td><td>38.64</td><td>28.95</td><td>30.13</td>
  </tr>
  <tr>
    <th colspan="21">Embedding models</th>
  </tr>
  <tr>
    <td>all-MiniLM-L6-v2</td><td>18.07</td><td>3.10</td><td>25.13</td><td>23.75</td><td>13.71</td><td>3.49</td><td>18.73</td><td>11.78</td><td>31.61</td><td>7.12</td><td>40.66</td><td>30.03</td><td>21.13</td><td>21.85</td>
  </tr>
  <tr>
    <td>e5-small-v2</td><td>20.10</td><td>3.14</td><td>26.47</td><td>25.12</td><td>21.02</td><td>5.33</td><td>27.70</td><td>16.88</td><td>32.58</td><td>7.35</td><td>39.85</td><td>30.53</td><td>24.57</td><td>24.11</td>
  </tr>
  <tr>
    <td>e5-base-v2</td><td>20.96</td><td>3.54</td><td>29.43</td><td>28.23</td><td>21.25</td><td>5.43</td><td>27.59</td><td>16.42</td><td>32.90</td><td>7.33</td><td>43.10</td><td>33.51</td><td>25.04</td><td>26.05</td>
  </tr>
  <tr>
    <td>e5-large-v2</td><td>22.93</td><td>3.49</td><td>29.53</td><td>28.09</td><td>20.16</td><td>5.19</td><td>27.26</td><td>16.68</td><td>39.45</td><td>8.81</td><td>49.07</td><td>38.19</td><td>27.51</td><td>27.65</td>
  </tr>
  <tr>
    <td>gte-base-en-v1.5</td><td>24.50</td><td>3.98</td><td>33.85</td><td>32.48</td><td>24.32</td><td>6.46</td><td>33.01</td><td>19.91</td><td>37.86</td><td>8.10</td><td>49.44</td><td>38.84</td><td>28.89</td><td>30.44</td>
  </tr>
  <tr>
    <td>gte-large-en-v1.5*</td><td>23.00</td><td>3.90</td><td>33.07</td><td>31.93</td><td>23.84</td><td>6.19</td><td>31.38</td><td>19.15</td><td>35.34</td><td>8.24</td><td>45.43</td><td>35.28</td><td>27.39</td><td>28.79</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5</td><td>23.44</td><td>3.86</td><td>32.79</td><td>31.47</td><td>24.17</td><td>6.37</td><td>31.80</td><td>18.68</td><td>36.59</td><td>8.47</td><td>47.44</td><td>36.87</td><td>28.07</td><td>29.01</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5</td><td>23.12</td><td>3.76</td><td>31.93</td><td>30.67</td><td>27.14</td><td>7.24</td><td>35.77</td><td>21.16</td><td>35.51</td><td>7.82</td><td>45.71</td><td>36.39</td><td>28.59</td><td>29.40</td>
  </tr>
  <tr>
    <td>gte-Qwen2-1.5B-inst.*</td><td>28.99</td><td>4.57</td><td>39.58</td><td>38.16</td><td>31.85</td><td>8.37</td><td>39.95</td><td>22.69</td><td>45.47</td><td>10.02</td><td>55.68</td><td>43.85</td><td>35.44</td><td>34.90</td>
  </tr>
  <tr>
    <td>e5-mistral-7b</td><td>25.38</td><td>4.13</td><td>34.81</td><td>33.47</td><td>28.22</td><td>7.60</td><td>35.99</td><td>22.28</td><td>42.31</td><td>9.03</td><td>51.34</td><td>40.24</td><td>31.97</td><td>32.00</td>
  </tr>
  <tr>
    <td>GrimL-7B*</td><td>29.67</td><td>4.82</td><td>41.30</td><td>39.77</td><td>30.86</td><td>8.18</td><td>39.99</td><td>25.25</td><td>48.92</td><td>10.31</td><td>59.01</td><td>46.55</td><td>36.48</td><td>37.19</td>
  </tr>
  <tr>
    <td>NV-Embed-v1*</td><td>35.50</td><td>5.54</td><td>48.36</td><td>46.72</td><td>33.08</td><td>8.77</td><td>41.08</td><td>24.83</td><td>51.28</td><td>11.13</td><td>61.49</td><td>48.05</td><td>39.95</td><td>39.87</td>
  </tr>
  <tr>
    <th colspan="21">Cross-encoder re-ranking models</th>
  </tr>
  <tr>
    <td>mbai-rerank-large-v1</td><td>31.75</td><td>4.79</td><td>43.82</td><td>42.20</td><td>24.84</td><td>5.96</td><td>32.13</td><td>19.53</td><td>35.65</td><td>7.31</td><td>43.38</td><td>33.43</td><td>30.75</td><td>31.72</td>
  </tr>
  <tr>
    <td>monot5-base-msmarco</td><td>26.91</td><td>4.32</td><td>37.16</td><td>35.03</td><td>30.43</td><td>8.11</td><td>38.04</td><td>20.45</td><td>46.32</td><td>9.15</td><td>56.53</td><td>46.36</td><td>34.56</td><td>33.95</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-m3</td><td>31.25</td><td>4.92</td><td>42.97</td><td>41.34</td><td>34.04</td><td>8.86</td><td>42.85</td><td>26.46</td><td>43.81</td><td>10.38</td><td>53.28</td><td>41.84</td><td>36.36</td><td>36.54</td>
  </tr>
  <tr>
    <td>jina-reranker-v2-base</td><td>33.31</td><td>5.06</td><td>44.20</td><td>42.93</td><td>36.79</td><td>9.49</td><td>46.20</td><td>28.55</td><td>53.18</td><td>11.56</td><td>63.22</td><td>50.70</td><td>41.09</td><td>40.72</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-gemma</td><td>38.59</td><td>5.67</td><td>50.14</td><td>48.67</td><td>38.08</td><td>10.05</td><td>47.65</td><td>28.98</td><td>50.39</td><td>11.54</td><td>61.69</td><td>49.48</td><td>42.36</td><td>42.38</td>
  </tr>
</table>

Table 13: Experiments are conducted under the w/ inst. setting, with retrieval performed within each subset individually.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">TOOLRET-Web</th>
    <th colspan="6">TOOLRET-Code</th>
    <th colspan="6">TOOLRET-Customized</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th><th>N@10</th><th>P@10</th><th>R@10</th><th>C@10</th>
  </tr>
  <tr>
    <td>bge-large-en-v1.5†</td><td>25.07</td><td>3.94</td><td>33.52</td><td>31.85</td><td>32.05</td><td>7.98</td><td>41.03</td><td>26.11</td><td>41.72</td><td>8.80</td><td>49.26</td><td>37.99</td><td>32.95</td><td>31.98</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5†</td><td>23.41</td><td>3.22</td><td>32.02</td><td>31.97</td><td>30.74</td><td>6.31</td><td>36.03</td><td>24.31</td><td>37.31</td><td>7.30</td><td>46.31</td><td>34.21</td><td>30.49</td><td>30.16</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5</td><td>18.90</td><td>3.13</td><td>25.80</td><td>24.50</td><td>24.49</td><td>6.67</td><td>32.95</td><td>19.30</td><td>25.72</td><td>5.54</td><td>32.18</td><td>24.79</td><td>23.03</td><td>22.86</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5†</td><td>20.35</td><td>3.41</td><td>28.19</td><td>26.77</td><td>30.69</td><td>7.70</td><td>39.16</td><td>24.95</td><td>37.01</td><td>8.53</td><td>47.21</td><td>36.40</td><td>29.35</td><td>29.37</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5†</td><td>19.05</td><td>2.98</td><td>25.13</td><td>23.70</td><td>24.90</td><td>6.41</td><td>35.44</td><td>22.50</td><td>33.51</td><td>6.40</td><td>44.20</td><td>35.71</td><td>25.82</td><td>27.30</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5</td><td>17.76</td><td>2.91</td><td>23.59</td><td>22.20</td><td>22.44</td><td>6.03</td><td>29.96</td><td>17.29</td><td>25.98</td><td>5.71</td><td>32.17</td><td>24.26</td><td>22.06</td><td>21.25</td>
  </tr>
  <tr>
    <td>e5-large-v2†</td><td>23.15</td><td>3.74</td><td>31.41</td><td>29.94</td><td>33.05</td><td>7.79</td><td>40.42</td><td>26.97</td><td>34.33</td><td>6.60</td><td>42.08</td><td>34.26</td><td>30.18</td><td>30.39</td>
  </tr>
  <tr>
    <td>e5-large-v2†</td><td>21.33</td><td>2.94</td><td>28.34</td><td>25.13</td><td>30.45</td><td>6.45</td><td>38.20</td><td>26.70</td><td>30.13</td><td>5.60</td><td>39.82</td><td>32.33</td><td>27.30</td><td>28.05</td>
  </tr>
  <tr>
    <td>e5-large-v2</td><td>17.03</td><td>2.67</td><td>21.77</td><td>20.63</td><td>18.94</td><td>4.90</td><td>25.95</td><td>16.26</td><td>26.37</td><td>6.07</td><td>32.19</td><td>23.17</td><td>20.78</td><td>20.02</td>
  </tr>
  <tr>
    <td>e5-base-v2†</td><td>19.97</td><td>3.33</td><td>27.67</td><td>26.36</td><td>26.45</td><td>5.92</td><td>32.71</td><td>22.24</td><td>31.03</td><td>6.06</td><td>38.42</td><td>30.92</td><td>25.81</td><td>26.51</td>
  </tr>
  <tr>
    <td>e5-base-v2†</td><td>15.44</td><td>2.78</td><td>25.37</td><td>23.61</td><td>24.50</td><td>5.20</td><td>30.12</td><td>19.37</td><td>28.03</td><td>5.47</td><td>40.21</td><td>31.92</td><td>22.66</td><td>24.97</td>
  </tr>
  <tr>
    <td>e5-base-v2</td><td>14.42</td><td>2.46</td><td>19.18</td><td>18.00</td><td>19.80</td><td>5.04</td><td>25.89</td><td>15.37</td><td>22.69</td><td>5.11</td><td>29.13</td><td>22.25</td><td>18.97</td><td>18.54</td>
  </tr>
</table>

Table 14: Experimental results of IR models before and after training on our datasets. Models trained with the concatenation of instruction and query are denoted by †. In contrast, the variants trained solely on the query as input are marked with ‡ (See the ablation study in § 7 for details).
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="2">ToolRet</th>
    <th colspan="2">ToolBench-G1</th>
    <th colspan="2">ToolBench-G2</th>
    <th colspan="2">ToolBench-G3</th>
  </tr>
  <tr>
    <th>NDCG@10</th>
    <th>NDCG@10</th>
    <th>Pass Rate</th>
    <th>NDCG@10</th>
    <th>Pass Rate</th>
    <th>NDCG@10</th>
    <th>Pass Rate</th>
  </tr>
  <tr>
    <th colspan="8"><i>gpt-3.5-turbo as tool-use LLM</i></th>
  </tr>
  <tr>
    <td>oracle</td>
    <td>-</td>
    <td>-</td>
    <td>62.00</td>
    <td>-</td>
    <td>57.20</td>
    <td>-</td>
    <td>67.40</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5</td>
    <td>23.03</td>
    <td>34.29</td>
    <td>50.60</td>
    <td>9.48</td>
    <td>49.00</td>
    <td>29.69</td>
    <td>56.90</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5<sup>♣</sup></td>
    <td>32.95<sub>+43.07%</sub></td>
    <td>71.11<sub>+107.38%</sub></td>
    <td>59.50<sub>+17.59%</sub></td>
    <td>18.11<sub>+91.03%</sub></td>
    <td>58.40<sub>+19.18%</sub></td>
    <td>67.87<sub>+128.60%</sub></td>
    <td>59.20<sub>+4.04%</sub></td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5</td>
    <td>22.06</td>
    <td>36.89</td>
    <td>50.60</td>
    <td>9.28</td>
    <td>51.20</td>
    <td>33.02</td>
    <td>57.70</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5<sup>♣</sup></td>
    <td>29.35<sub>+33.05%</sub></td>
    <td>67.52<sub>+83.03%</sub></td>
    <td>56.60<sub>+11.86%</sub></td>
    <td>16.01<sub>+72.52%</sub></td>
    <td>59.60<sub>+16.41%</sub></td>
    <td>60.75<sub>+83.98%</sub></td>
    <td>60.80<sub>+5.37%</sub></td>
  </tr>
  <tr>
    <td>e5-large-v2</td>
    <td>20.78</td>
    <td>44.91</td>
    <td>47.50</td>
    <td>11.57</td>
    <td>56.50</td>
    <td>43.43</td>
    <td>55.70</td>
  </tr>
  <tr>
    <td>e5-large-v2<sup>♣</sup></td>
    <td>30.18<sub>+45.24%</sub></td>
    <td>70.08<sub>+56.05%</sub></td>
    <td>57.00<sub>+20.0%</sub></td>
    <td>17.71<sub>+53.07%</sub></td>
    <td>62.10<sub>+9.91%</sub></td>
    <td>66.09<sub>+52.18%</sub></td>
    <td>58.00<sub>+3.99%</sub></td>
  </tr>
  <tr>
    <td>e5-base-v2</td>
    <td>18.97</td>
    <td>38.66</td>
    <td>49.60</td>
    <td>9.87</td>
    <td>54.10</td>
    <td>37.35</td>
    <td>54.20</td>
  </tr>
  <tr>
    <td>e5-base-v2<sup>♣</sup></td>
    <td>25.81<sub>+36.06%</sub></td>
    <td>65.79<sub>+70.18%</sub></td>
    <td>56.90<sub>+14.72%</sub></td>
    <td>17.45<sub>+76.80%</sub></td>
    <td>60.80<sub>+12.38%</sub></td>
    <td>62.74<sub>+67.98%</sub></td>
    <td>62.40<sub>+15.13%</sub></td>
  </tr>
  <tr>
    <th colspan="8"><i>ToolLlama as tool-use LLM</i></th>
  </tr>
  <tr>
    <td>oracle</td>
    <td>-</td>
    <td>-</td>
    <td>53.6</td>
    <td>-</td>
    <td>50.8</td>
    <td>-</td>
    <td>49.1</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5</td>
    <td>23.03</td>
    <td>34.29</td>
    <td>37.60</td>
    <td>9.48</td>
    <td>41.30</td>
    <td>29.69</td>
    <td>37.20</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5<sup>♣</sup></td>
    <td>32.95<sub>+43.07%</sub></td>
    <td>71.11<sub>+107.38%</sub></td>
    <td>45.10<sub>+19.95%</sub></td>
    <td>18.11<sub>+91.03%</sub></td>
    <td>47.30<sub>+14.53%</sub></td>
    <td>67.87<sub>+128.60%</sub></td>
    <td>39.60<sub>+6.45%</sub></td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5</td>
    <td>22.06</td>
    <td>36.89</td>
    <td>47.80</td>
    <td>9.28</td>
    <td>46.10</td>
    <td>33.02</td>
    <td>36.10</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5<sup>♣</sup></td>
    <td>29.35<sub>+33.05%</sub></td>
    <td>67.52<sub>+83.03%</sub></td>
    <td>50.60<sub>+15.86%</sub></td>
    <td>16.01<sub>+72.52%</sub></td>
    <td>49.80<sub>+8.03%</sub></td>
    <td>60.75<sub>+83.98%</sub></td>
    <td>45.70<sub>+26.60%</sub></td>
  </tr>
  <tr>
    <td>e5-large-v2</td>
    <td>20.78</td>
    <td>44.91</td>
    <td>41.50</td>
    <td>11.57</td>
    <td>46.60</td>
    <td>43.43</td>
    <td>40.20</td>
  </tr>
  <tr>
    <td>e5-large-v2<sup>♣</sup></td>
    <td>30.18<sub>+45.24%</sub></td>
    <td>70.08<sub>+56.05%</sub></td>
    <td>44.50<sub>+17.23%</sub></td>
    <td>17.71<sub>+53.07%</sub></td>
    <td>49.80<sub>+4.72%</sub></td>
    <td>66.09<sub>+52.18%</sub></td>
    <td>43.80<sub>+4.50%</sub></td>
  </tr>
  <tr>
    <td>e5-base-v2</td>
    <td>18.97</td>
    <td>38.66</td>
    <td>42.20</td>
    <td>9.87</td>
    <td>45.20</td>
    <td>37.35</td>
    <td>42.00</td>
  </tr>
  <tr>
    <td>e5-base-v2<sup>♣</sup></td>
    <td>25.81<sub>+36.06%</sub></td>
    <td>65.79<sub>+70.18%</sub></td>
    <td>49.10<sub>+16.35%</sub></td>
    <td>17.45<sub>+76.80%</sub></td>
    <td>48.30<sub>+6.86%</sub></td>
    <td>62.74<sub>+67.98%</sub></td>
    <td>44.70<sub>+16.60%</sub></td>
  </tr>
</table>

Table 15: Experiment results of IR models before and after training. We also show the end-to-end task pass rate of tool-use LLMs when equipped with the tools retrieved by the IR models on ToolBench dataset.

Metareview score: 3
Metareview: This paper introduces a tool retrieval benchmark called TOOLRET, and experimental results show that IR models wuth strong performance on conventional benchmarks struggle in tool retrieval.
Summary of reasons to publish: The paper constructs a diverse tool retrieval benchmark and conducts extensive evaluations.