Preview Tools before Using: Enhancing Tool Documentation with Multi-Tool Exploration
Authors: Zhen Zhang, Shen Gao, Zhengliang Shi, Jiabao Fang, Piji Li, Pengjie Ren, Zhumin Chen, Shuo Shang, Zhaochun Ren


Reviewer paper summaries: This paper presents ToolBFS+, a method to improve tool documentation for LLMs using a Breadth-First Search (BFS) strategy. This method enhances tool usage scenarios, corrects incomplete tool descriptions, and improves the model's ability to utilize tools accurately.
Summary of strengths: + Comprehensive methodology with a clear three-stage process for scenario generation, BFS exploration, and documentation revision.
+ The paper is well-structured, with clear sections and logical flow, making it easy to follow and understand.
Summary of weaknesses: -  **Shortcomings in evaluation**. For the direct evaluation of tool documentation, although the planning, calling, and parsing stages are taken into account, the self-evaluation using the model may be biased because the model may not be able to identify its limitations or errors. Although Correct Path Rate and Success Rate can measure the improvement effect of the method, these indicators may not fully reflect the impact of improved documentation quality on the actual tool usage experience.
- **Dependence on manual judgment**. During the exploration phase of Multi-Tool BFS (line 757), the correctness of the answer is judged by predefined rules and models, which may be subjective and inconsistent. Especially in the process of rule writing and model prompting, the human intervention component is high, which may lead to inconsistent evaluation results.
- **Reasonableness of selecting the least frequently used tool**. This paper selects tools based on the frequency of use, should the functionality and relevance of the tool also be considered? A scoring mechanism can be designed to comprehensively evaluate the tool's frequency of use, functionality, and relevance to the task. For example, tools with similar functions can be packaged in a toolkit for subsequent selection.
- **Comparison with other search strategies**. This paper does not provide a detailed discussion on how the BFS strategy compares to other search strategies (e.g., DFS, ToolLLM: Facilitating large language models to master 16000+ real-world APIs, ICLR 2024) in terms of efficiency and scalability.
Comments, suggestions and typos: Please see the weaknesses above.
Overall assessment scores: 2.5