CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions (2024)

Abstract: Assessing the capabilities of large language models (LLMs) as agents in decision making and operational tasks is crucial for the development of LLM-as-agent service. We propose CToolEval, a benchmark designed to evaluate LLMs in the context of Chinese societal applications, featuring 398 APIs across 27 widelyused Apps (e.g., Apps for shopping, map, music, travel, etc.) that cover 14 domains. We further present an evaluation framework that simulates real-life scenarios, to facilitate the assessment of tool invocation ability of LLMs for tool learning and task completion ability for user interation. Our extensive experiments with CToolEval evaluate 11 LLMs, revealing that while GPT-3.5-turbo excels in tool invocation, Chinese LLMs usually struggle with issues like hallucination and a lack of comprehensive tool understanding. Our findings highlight the need for further refinement in decision-making capabilities of LLMs, offering insights into bridging the gap between current functionalities and agent-level performance. To promote further research for LLMs to fully act as reliable agents in complex, real-world situations, we release our data and codes at https: //github.com/tjunlp-lab/CToolEval.

heading: 1 Introduction
text: Recent years have witnessed that large language models achieve remarkable progress in planning and decision making. This promotes the emergence of numerous LLM-as-agent applications, such as ChatGPT plugins1, AutoGPT2, and Generative Agents (Park et al., 2023), even enabling LLMbased plugin control with natural language interface and provoking the development of LLM-based
*Equal contribution, the order of authors is determined by the first letter of their surnames.
†Corresponding author 1https://openai.com/blog/chatgpt-plugins 2https://github.com/Significant-Gravitas/AutoGPT
OS. In view of such rapid progresses, pressing issues are urgently emerging and soliciting attention and solutions: (i) How should we assess the capability of LLMs as agents? (ii) Which LLMs have reached agent level, and which have not? (iii)What challenges do LLMs still face when used as agents?
To address these urgent and unresolved issues, we curate a benchmark, CToolEval, for tool learning in Chinese scenarios. This benchmark is of very high quality, characterized by that (i) all APIs come from a wide and rigorous collection of open APIs from widely-used real-world Apps, making the benchmark akin to a testbed of natural language driven mobile operation system. It assesses whether an LLM-powered agent can understand and accurately respond to various functionalities within Apps, offering a high degree of real-world interactions. (ii) The generation of instructions is meticulously designed, with the use of certain tools not being completely unrelated and abrupt, but rather closely simulating all-around scenarios in our daily lives. (iii) Each instruction is annotated with a detailed reference answer and has undergone a two-stage rigorous manual review to ensure its quality.
To facilitate the agent assessment with CToolEval, we futher propose a fine-grained evaluation framework. Initially, we categorize the data from single-tool and multi-tool scenarios into four types: fixed-answer, open-ended, real-time, and operational types. In the first stage of evaluation, we assess whether an LLM-powered agent has invoked an appropriate tool, including understanding and correctly calling the API in a single-tool scenario, or whether the agent can decompose a complex task into multiple subtasks, then plan to call APIs for each subtask and understand the use of API parameters to make the correct calls in a multi-tool scenario. In the second stage of evaluation, we start from the perspective of accuracy rather than having humans (Song et al., 2023; Tang et al., 2023; Shen
15711
et al., 2023) or GPT-4 (Qin et al., 2023a; Ye et al., 2024; Guo et al., 2023) judge the quality of almost all responses. Both approaches inevitably introduce bias and issues with reusability and scalability. Specifically, for real-time data, where answers are constantly changing, such as a user’s query about the highest temperature tomorrow, we aim for an accurate temperature but cannot know the answer in advance. This has always been a pain point in agent evaluation because neither humans nor GPT4 can judge whether the answers are correct. Thus, in our framework, we dynamically extract real-time answers in observations to address this problem.
With the proposed evaluation framework, we evaluate 11 LLMs with Chinese language capabilities. We find that OpenAI models generally have superior tool invocation capabilities compared to Chinese LLMs, but models with rich internal knowledge like GPT-3.5-turbo1, when faced with conflicts between internal and external knowledge, tend to rely on internal knowledge rather than toolinvoked knowledge to answer questions. For instance, in multi-tool scenarios requiring the acquisition of a location’s coordinates, it might provide coordinates it already knows internally, which are usually close but still have discrepancies, rather than those obtained through tool invocation. And we also find Chinese LLMs, like ChatGLM3-6b2, have severe hallucination issues. They can decompose instructions into subtasks and choose the correct tool names. However, due to a lack of complete understanding of how to use the API, they cannot pass in the correct parameters to obtain the correct output from the API. Nevertheless, they still pretend to have received output from the API server and reply with fabricated answers.
Overall, our contributions are threefold:
1https://platform.openai.com/docs/models/gpt-3-5-turbo 2https://huggingface.co/THUDM/chatglm3-6B
• We propose CToolEval, a benchmark for evaluating the planning and operational capabilities of large language models in real-world application scenarios, covering 398 APIs across 14 domains in 27 Apps (please see detailed comparison of CToolEval against existing tool benchmarks.) .
• We propose a fine-grained evaluation framework, meticulously designing evaluation methods and indicators for each type of interaction from an accuracy perspective, solving the evaluation problem for real-time dynamic answers.
• We evaluate 11 LLMs with Chinese capabilities and conduct a deep error analysis, providing profound insights into the improvements needed for LLMs, including GPT models, to act as decision-making agents.

heading: 2 Related Work
text: Research on evaluating LLMs for tool manipulation and API interaction has led to benchmarks like ToolBench (Qin et al., 2023a), with its extensive collection of 16,464 APIs across 3,451 tools and 12,657 instances. The emulation of real-world scenarios is a pivotal aspect of benchmark evaluation for LLMs, as it significantly affects the relevance and applicability of evaluation findings. API-Bank (Li et al., 2023), provides an interface with 53 APIs, asserting its utility with 274 instances drawn from real-world applications. ToolQA (Zhuang et al., 2023) highlights multi-tool interactions, emphasizing their significance in LLM evaluation.
The dimension of evaluation type in benchmarking LLMs is crucial as it directly influences the interpretability and applicability of evaluation results. This dimension encompasses the accuracy (Li et al., 2023; Patil et al., 2023; Tang et al., 2023; Ruan
et al., 2023) and execution success rate of API calls (Song et al., 2023; Xu et al., 2023; Zhuang et al., 2023; Qin et al., 2023a), the quality and relevance of the responses (Li et al., 2023; Tang et al., 2023; Zhuang et al., 2023; Huang and Xiong, 2023), and the model’s ability to avoid hallucination — generating responses based on false or nonexistent information (Patil et al., 2023). Specifically, the metrics used in RestBench (Song et al., 2023) and ToolAlpaca (Tang et al., 2023) require manual evaluation, while the win rate indicator in ToolBench1 (Qin et al., 2023a) relies on scoring by GPT-4, all of which inevitably introduce elements of subjective judgment and assessment unfairness caused by the biases of human and LLMs.
A concurent work has a motivation very similar to ours. ToolEyes (Ye et al., 2024) creates a finegrained evaluation of LLM’s tool learning capabilities. Particularly in the data dimension, both of our works include real-time and operational data types. However, ToolEyes (Ye et al., 2024) still relies mostly on scoring by advanced models like GPT-4 to deliver evaluation results, inevitably encountering issues such as model bias. Our contribution, CToolEval, seeks to address these shortcomings by prioritizing accuracy, solving for the dynamic evaluation of answers that may change over time, aiming to minimize reliance on model scoring for better reuse and scalability.

heading: 3 Benchmark Curation
text: Our data creation process, as illustrated in Figure 1, involves sourcing open APIs from real-world applications.

heading: 3.1 APP Pool
text: Against the backdrop of real interactions between users and Apps in Chinese society, we integrate APIs available from popular Apps across 14 domains. The selected Apps for integration satisify
the following conditions: (i) Open for research - These Apps are available for research purposes; (ii) Providing Detailed Parameter Descriptions and Usage instructions - Each App comes with comprehensive documentation. After selection, we finally choose 27 Apps from a wide range of domains, which are essential and commonly used in daily life, including but not limited to: travel apps such as Gaode Maps and Baidu Maps; workrelated apps like Dingding and Feishu; shopping apps including Taobao and Jingdong; music apps like Ximalaya; logistics apps such as Yunda and Express Bird and so on. We further sift through these app platforms to find APIs that meet our criteria. The division of all dimensions and apps can be seen in Figure 2. The shopping domain has the highest number of apps, including Taobao, Pinduoduo, Alibaba, Jingdong, and Red, with the Work and Meeting domains having the largest number of APIs, where the Feishu app alone includes 76 APIs.

heading: 3.2 API & LLM Integration
text: Utilizing the Langchain1 framework, our system successfully integrated a total of 398 APIs with LLMs. This integration involves meticulously designed incorporation of each API into a comprehensive toolset pool. The LLM, initially unaware of the available APIs within the pool, analyzes user queries to identify and choose the correct tool from the pool using vector storage for tool descriptions and query embeddings, facilitating a similarity search. It then translates queries into structured data matching the selected tool’s interface and executes API calls. If responses are inadequate, it iterates until achieving a satisfactory answer. This method enables easy API pool expansion. Specifically, for APIs related to database operations, we also establish essential databases, initialized with initial entries. This step is a pivotal element in constructing and utilizing the framework.

heading: 3.3 Instruction Generation
text: we designed queries reflecting real-life applications solvable through the detailed functionalities of these APIs. In this setup, GPT-4 is employed as an agent to orchestrate and execute API calls. Queries used for debugging the integrated code, are termed as our seed instructions. We then guide GPT-4 with meticulously crafted instructions for expan-
1https://www.langchain.com
sion. Further Recognizing complex user needs, we differentiated our evaluation into single-tool and multi-tool assessments, addressing the varying degrees of challenge.

heading: 3.3.1 Single-tool Instructions
text: For single-tool queries, we crafted diverse inquiries per API by: (i) changing query subjects like destinations in map and delivery scenarios; (ii) varying sentence structures; (iii) creating unique contexts such as weather suitability for barbecues or hikes. This generates 3,439 single-tool instructions.

heading: 3.3.2 Multi-tool Instructions
text: In the generation of multi-tool instructions, instead of forcibly combining unrelated tools, our focus is on the intrinsic connections between different tools, resulting in the creation of 3,370 multi-tool instructions. The generation encompasses both multiple APIs within the same App and across different Apps.
Multiple APIs within the same App. During API integration, we observe that many APIs are interdependent. For instance, to obtain a route plan from Gaode Maps, geographical encoding of the start and end points, obtained through Gaode Maps’ geocoding tool, is necessary. Typically, the number of APIs involved in these interactions is basically between 2 and 10.
Multiple APIs across different Apps. Beyond the same App, multi-API queries across different Apps also have numerous real-life applications. For instance, in the shopping category, comparing prices of the same item on Taobao and JD.com is a common practice. Based on this, we create similar cross-App multi-API queries to address more complex requirements.

heading: 3.4 Reference Answer Annotation
text: Upon the completion of designing each query, we deploy GPT-4 as an agent to plan and execute API calls, ensuring the feasibility of each query. The responses are then annotated based on the output of this execution. Employing the ReAct principle for the agent’s implementation, the final answers encompass the decision process, actions, observations, and final answers of the agent. The sequence of API calls and responses obtained from the final API execution are meticulously annotated in our responses. However, the design of our queries poses significant challenges to GPT-4. Therefore, a rigorous manual review is conducted, ocusing on output
format and logical consistency, these reviewed answers are annotated in our benchmark. Ultimately, we retained 6,816 queries in our benchmark.

heading: 3.5 Statistics
text: The data statistics of our dataset can be seen in Table 2. In the end, we construct a comprehensive benchmark that covers 27 tools, featuring 398 distinct APIs, comprising a total of 6,816 queries, 14,303 API calls. Within this benchmark, the distribution of queries is as follows: 3,439 queries utilize a single tool, while 3,377 queries employ multiple tools, showing a balanced distribution between single-tool and multi-tool instances. Moreover, single-tool instances involve 27 tools and 185 APIs, while multi-tool instances cover 24 tools and 231 APIs, indicating that our application scenarios are very rich. From the perspective of evaluation method classification, we can further categorize these instances as follows: 2,423 queries in the operational category, 371 queries in the open-ended category, 3,001 queries in the fixed-answer category, and 1,021 queries in the real-time response category. For more details about the division and instances of these four types of evaluation categories, please refer to Table 2.

heading: 3.6 Quality
text: To ensure the quality of the data in our benchmark, we also conduct a detailed review of each data instance. Every instance in the evaluation set has been reviewed by annotators, with the review process divided into two phases. During the data generation phase, we annotate the answers with great detail and strictness. We employ GPT-4 as the agent to execute all instances. However, given that our benchmark operates in real-world Chineselanguage scenarios, it also poses significant challenges to GPT-4. Throughout multiple debugging sessions, GPT-4’s responses are not always satis-
factory, sometimes presenting issues like incorrect tool planning or fabricated API results. Therefore, after obtaining GPT-4’s output, a rigorous manual review is conducted, including scrutiny of interface compatibility, the rationality of API planning, and the congruence of the final answer with the API’s actual return. We run multiple iterations until the correct answers are obtained, including manual adjustments for some responses. For instance, if after several attempts it still could not produce output in Chinese, we perform manual translations into English, and for responses containing “I am a text generation model, unable to invoke tools“, we manually remove them. In the evaluation phase, to enhance the robustness of our benchmark, we first test all the data using GPT-3.5. After this stage, we eliminate 2% of the data, including instances where the API itself has issues and could not respond correctly, as well as operational instances that could not meet our repeated evaluation requirements. These stringent quality controls further improve the quality of our benchmark, allowing us to primarily assess the quality of LLMs as proxy tools in a standard and fair setting.

heading: 4 Evaluation Framework
text: In a comprehensive process where an LLM acts as an agent to solve problems using tools, the model initially extracts the query and the list of available tools along with their descriptions from the prompts. It then decomposes the query into smaller sub-tasks and sequentially calls the corresponding tools, using the knowledge returned by each tool to progressively deduce the correct answers.
However, prior research (Ruan et al., 2023; Kong et al., 2023; Qin et al., 2023b) reveals the inadequacies of most LLMs in tool utilization, highlighting their failure in task decomposition, tool invocation, and knowledge application, with errors frequently emerging at various stages.
Thus, to thoroughly assess LLMs’ tool utilization proficiency, we scrutinize both tool invocation and task completion capabilities. Through an indepth analysis shown in Appendix A, we evaluate LLMs’ ability to apply tool-generated knowledge. These competencies are interrelated, marking ascending levels of tool usage mastery. Additionally, leveraging actual user inputs, CToolEval’s instructions are classified into four categories: fixed-answer, open-ended, operational, and realtime, enabling a nuanced, multi-faceted evaluation
of accuracy in these dimensions.

heading: 4.1 Data Type
text: Existing benchmarks predominantly feature open and fixed-answer queries, with their evaluation relying on advanced LLM-based scoring and accuracy metrics. Despite the sophisticated capabilities of such models, their assessments of answer accuracy can be unreliable, particularly for queries necessitating tool-derived knowledge.
To enhance the evaluation of large language models’ tool usage proficiency, we collect real user queries that demand tool invocation, and categorize them into four distinct types based on evaluation methods: fixed-answer, open-ended, operational, and real-time, as detailed in Table 7.
(i) Fixed-answer queries entail questions with a static, fill-in-the-blank response.
(ii) Open-ended questions require detailed responses without a predetermined format.
(iii) Operational questions necessitate the execution of tool-based APIs for specific tasks.
(iv) Real-time questions involve answers that are dynamic over time.
We apply different evaluation methodologies for these question types to accurately measure task completion efficacy.

heading: 4.2 Retriever
text: LLMs usually have limited context capacity. Overloading LLMs with excessive irrelevant tool information can hinder their tool invocation ability. Hence, employing a retriever to assist LLMs in selecting suitable tools is essential (Qin et al., 2023b). We use Sentence-BERT (Reimers and Gurevych, 2019), based on Chinese-bert-base1, to train a retriever that calculates the semantic similarity between the embeddings of CToolEval instructions and tool descriptions to identify the most relevant tool. The effectiveness of the retriever is measured using the NDCG metric (Järvelin and Kekäläinen, 2002). More details about the retriever are given in Appendix C.

heading: 4.3 Tool Invocation Capability
text: Tool invocation capability is the fundamental ability that determines whether an LLM can effectively complete tasks that involve the use of tools, which requires LLMs to decompose tasks into sub-tasks and appropriately match tools to each sub-task. The
1https://huggingface.co/google-bert/bert-base-chinese
evaluation metric we employ is the “Tool Matching Rate“, which is calculated as the number of tools correctly used divided by the total number of tools that should be used. Please note that, at this stage, we exclusively examine the correctness of tool usage.

heading: 4.4 Task Completion Capability
text: Task completion capability is a crucial competency that assesses LLMs’ ability to utilize knowledge obtained from tools to engage in reasoning and solve problems, which entails LLMs’ step-by-step integration of sub-task results to deduce the final answers.
For fixed-answer queries, we employ "accuracy" as the evaluation metric. For open-ended questions, we utilize advanced LLMs to score based on the reference answers, focusing on rationality, completeness, and richness of information. Regarding operational queries, where LLMs may potentially provide deceptive responses by falsely asserting task completion without performing the necessary operations, we thoroughly check the status codes of the responses from the tools invoked for each subtask. Typically, network responses returned by different APIs contain status codes that represent the result of the request. If the operation is successful, the status code is a specific value. The evaluation metric is the number of tools returning the correct response divided by the number of tools needed as per the reference. Furthermore, as CToolEval serves as an open-source evaluation dataset, it should ensure the consistency of the backend database objects across multiple evaluations. Therefore, we design a rollback function. Specifically, after each successful invocation of a tool, the system will automatically perform the "reverse operation" of this invocation. For example, after adding a friend using an instant messaging tool, the system will delete this friend. In the case of real-time queries, we treat them as real-time fillin-the-blank queries. When establishing reference answers for such queries, we store the API links and parameters to be called, ensuring that reference answers can be easily extracted or assembled from the returned information. This approach enables us to conduct "dynamic assessments" of real-time queries based on accuracy.

heading: 5 Experiments
text: We evaluated a variety of LLMs, including 1 closedsource LLMs and 10 open-source Chinese LLMs. As our evaluation requires LLMs to have a certain level of compliance with instructions, we selected those that have been fine-tuned with SFT and possess chat capabilities. Additionally, due to the large number of tools, and some multi-tool queries even containing more than 10 tools, the prompts fed into the LLMs can be very long, necessitating LLMs with a relatively large context window, at least above 8K. The evaluated closedsource LLMs is GPT-3.5-turbo.1 The open-source LLMs include Qwen-7B-chat,2 Qwen-14B-chat,3 Qwen-72B-chat,4 InternLM-chat-7b,5 InternLMchat-20b,6 Baichuan2-13B-chat,7 Baichuan2-13B-
1https://openai.com/docs/models/gpt-3-5 2https://huggingface.co/Qwen/Qwen-7B-Chat 3https://huggingface.co/Qwen/Qwen-14B-Chat 4https://huggingface.co/Qwen/Qwen-72B-Chat 5https://huggingface.co/internlm/internlm-chat-7b 6https://huggingface.co/internlm/internlm-chat-20b 7https://huggingface.co/Baichuan2-7B-Chat
chat,8 Chinese-alpaca-2-7b-16k,9 Chinese-alpaca2-7b-64k10 and Chinese-alpaca-2-13b-16k.11

heading: 5.1 Main Results
text: Table 3 presents the evaluation results of the tool invocation capability of different LLMs, while Table 4 shows the task completion capabilities of different LLMs. From these results, we can observe that: (i) The gap between open-source Chinese LLMs and the proprietary GPT-3.5-turbo remains significant. In terms of the comparison of different model capabilities, GPT-3.5-turbo far surpasses Chinese LLMs in both tool invocation and task completion capabilities, with InternLM-Chat20B scoring the highest among Chinese LLMs, reaching up to 53.47 points in tool invocation capability for single-tool fixed-answer data types; (ii) Task completion scores of LLMs are higher than tool invocation scores, especially in fixed-answer and open-ended types, indicating that many LLMs
8https://huggingface.co/Baichuan2-13B-Chat 9https://huggingface.co/chinese-alpaca-2-7b-16k
10https://huggingface.co/chinese-alpaca-2-7b-64k 11https://huggingface.co/chinese-alpaca-2-13b-16k
still lack the capability to invoke tools, yet they still use their internal knowledge to answer user questions. Despite demonstrating good interactivity, LLMs may deceitfully pretend to have obtained answers through tools, leading to significant issues with hallucination; (iii) Almost all LLMs score higher in single-tool scenarios than in multi-tool scenarios, particularly, for LLMs capable of tool invocation, the scores for operation types in singletool scenarios often exceed those for fixed-answers and open-ended responses. This may be due to the tool descriptions and parameters being relatively brief in single-tool operations, allowing LLMs to better understand and follow the context in shorter prompts, but multi-tool scenarios still pose a greater challenge, and real-time queries are difficult for LLMs in both single- and multi-tool contexts; (iv) Increasing model size can enhance the tool learning capability of LLMs to a certain extent, but not always. For InternLM and Qwen, increasing the model size has improved the capability in both tool invocation and task answering, especially for Qwen-72B-Chat, which has seen a qualitative improvement compared to the other two sizes. However, for Chinese-Alpaca-2, increasing the model size can actually impair model performance. Thus, to enhance the model’s tool learning capability, besides ensuring a sufficiently long context window, it is also necessary to further balance the model size and fine-tuning data.

heading: 5.2 Error Analysis
text: We conducted a detailed analysis on errors made by GPT-3.5-turbo in single-tool and multi-tool scenarios, and have categorized the errors in single-tool scenarios into 9 types and those in multi-tool scenarios into 10 types. The percentages of error types are presented in Table 5. In the single-tool scenario, the failure to correctly invoke an API can occur for various reasons, such as incorrect API names, incorrect or erroneous format of input parameters, the retriever failing to provide the required API or providing the needed API but making the wrong choice, and fabrications. The most common errors include false input parameters, Duplicate Response, and Redundant Response. In the multi-tool context, the failure to execute multiple tools is mainly due to the model not completing operations, only performing halfway and then outputting answers, as well as skipping the necessary preceding operations and directly relying on its internal knowledge or fabricating the required parameters’ inputs. In addition, it is common in multi-tool scenarios to insert multiple required tools into a single action, leading to persistent failure. These errors suggest that GPT-3.5-turbo’s learning and understanding of how to use tools, especially the ability to associate between tools, still needs improvement, and duplicate response failure indicates a deficiency in the ability to follow instructions. Moreover, we have also discovered that GPT-3.5-turbo has significant issues with temporal reasoning. Whenever our queries involve "tomorrow", it often reasons incorrectly about the input time parameters for APIs. This could be an area that GPT-3.5 needs to address in order to become a truly grounded agent.

heading: 6 Discussion

heading: 6.1 Impact of Retriever on Baseline Model Performance
text: To study how retrieval affects the tool invocation capabilities in LLMs, we provided gold APIs to InternLM-7B-chat, InternLM-20B-chat, and GPT3.5-turbo and assessed their performance. Results in Table 8 and 9 in Appendix D show that gold APIs significantly enhance invocation capabilities, particularly because the retriever presents ten tool options, making prompts longer and harder for models to process useful data. Although the retriever exhibits a high recall rate, we observe that in multi-tool tasks, the tools returned by the retriever sometimes do not include those that are
more critical in the earlier stages of the solution path. This absence of essential tools early in the tool invocation process imposes limitations on the LLMs.
Morever, InternLM-7B-chat and InternLM-20Bchat improve more than GPT-3.5-turbo. We speculate that providing gold APIs benefits LLMs with weaker reasoning ability in task planning, while stronger LLMs can effectively select APIs even without gold APIs.

heading: 6.2 Performance of LLMs Being Fine-tuned on CToolEval
text: We utilized the training set of CToolEval to finetune Baichuan2-7b-chat and Qwen-7b-chat models using two parameter-efficient methods, QLoRA and LoRA. Additionally, we benchmarked these models against GPT-4 in a zero-shot setting on the test set. Results, detailed in Table 10 and 11 in Appendix E, indicate significant improvements in both tool matching rate and task completion metrics for the fine-tuned Baichuan2-7B-chat and Qwen7B-chat models. Furthermore, the Qwen-7B-chat models, fine-tuned using both QLoRA and LoRA methods, exhibit comparable abilities.
However, fine-tuning also intensifies the tendency for tool invocation hallucination. This means that after fine-tuning, the models are more likely to simulate API responses rather than performing actual tool invocations, leading to higher tool matching scores but not proportionately higher task completion scores. This is because while selecting the correct tool contributes to the tool matching rate, effective task completion depends on the LLM’s ability to reason and analyze actual API responses. Since the nature of answers to real-time and openended queries fluctuates, without real tool invocations, task completion cannot improve. Therefore, for models to excel in both metrics, they must genuinely invoke tools similar to GPT-4.

heading: 6.3 Knowledge Utilization Capability of
text: GPT-3.5
Knowledge utilization capability is a crucial ability that determines whether LLMs can effectively harness the knowledge provided by tools, which requires LLMs to accurately utilize the knowledge obtained from each sub-task without hallucination. However, manually annotating correct results for each sub-task can be prohibitively expensive. Therefore, we introduce an innovative control group evaluation method.
For all the tools successfully invoked in a single command execution, we will set up corresponding control groups according to the order of tool invocation, where each tool’s control group will represent a command execution without using that particular tool. Specifically, if n tools are used, there will be n− 1 control groups. By calculating the similarity between the control group’s answers and the answers to be evaluated with the reference answer, we can determine whether the execution effectively utilizes the knowledge returned by each tool step. See Appendix A for results and analyses.

heading: 7 Conclusion
text: In this paper, we have presented CToolEval, a benchmark for assessing LLMs as agents to interact with Chinese Apps, which covers 398 realworld APIs. Through our fine-grained evaluation framework, we identify critical insights into the capabilities of LLMs, specifically in the domains of tool invocation and task completion. Our analysis indicates that while GPT-3.5-turbo demonstrates a higher proficiency in tool utilization, there remains a notable gap in the ability of Chinese LLMs to apply tools effectively. This gap is often marked by a reliance on internal knowledge and the occurrence of hallucinations, which leads to a discrepancy between the output provided and the information available through tool invocation. By dissecting the errors and pinpointing the nuances in the operational use of APIs, our work contributes to a more nuanced understanding and evaluation of the agent-level capabilities of LLMs and opens up avenues for future research and development of LLM-powered agents in real-world scenarios.
Limitations
Although we have established a Chinese benchmark that simulates real-world App interactions to conduct a comprehensive analysis of existing LLMs and outlined various existing issues, our work has two notable limitations. First, since the APIs we use are publicly accessible, there might be instances where an API interface becomes faulty over time, affecting our evaluation results, but this is an unavoidable aspect of simulating real-world tool interactions. Second, due to the token usage limitations of GPT-4, our ability to evaluate all existing LLMs is restricted. It is important to emphasize that we meticulously selected the most representative Chinese LLMs for evaluation and anal-
ysis, aiming to identify key problems with Chinese LLMs in agentization.
Ethics Statement
This paper establishes a new benchmark for evaluating the tool learning capabilities of Chinese LLMs, and herein we discuss some related ethical considerations. All App interfaces in our dataset come from open platforms accessible to researchers, without infringing upon any existing commercial software. Furthermore, to annotate data within the benchmark, annotators involved in the annotation were fully compensated for their efforts.

heading: Acknowledgements
text: The present research was supported by the National Key Research and Development Program of China (Grant No. 2023YFE0116400). We would like to thank the anonymous reviewers for their insightful comments. Moreover, we would also like to thank Yue Chen, Zhuowen Han, Hangyi Jia, Jianxiang Peng, Ling Shi, Juesi Xiao, Lei Yang, and Jiang Zhou for their invaluable assistance in data collection, processing, and annotation. Their efforts have greatly contributed to the success of this work.

heading: A Knowledge Utilization Capability of GPT-3.5-turbo
text: Figure 3 is a violin plot for GPT-3.5-turbo-1106 on multi-tool open-ended and single-tool open-ended queries, where each point represents a query execution, the horizontal axis represents the number of tools invoked, and the vertical axis represents the similarity between the answers and the reference answers. Surprisingly, the quality of LLMgenerated answers does not increase with the number of tools invoked. Most queries show improvement when the first tool is invoked compared to no tool invocation; however, after invoking the second tool, the quality of answers slightly decreases for some instructions, and after invoking three tools, the quality of LLM-generated answers decreases for most instructions. We speculate that the singleopen and multi-open commands in CToolEval require a large number of tokens from the returned information of the tools. After invoking a small number of tools, the LLM can obtain information to answer the question; when the number of invoked tools increases, the rapid increase in the number of tokens makes it difficult for the LLM to capture useful information, leading to a decrease in answer quality.

heading: B Four Examples of Question Types
text: CToolEval classifies queries into fixed-answer, open-ended, operational, and real-time types depending on how they are evaluated. Question examples and reference answers are shown in Table 7.

heading: C Retriever
text: We opt to fine-tune the Chinese-bert-base1 model based on Sentence-BERT (Reimers and Gurevych, 2019) using pairs of query-tool information data. Following ToolLLM (Qin et al., 2023b), we assess retrieval performance using NDCG (Normalized Discounted Cumulative Gain) (Järvelin and Kekäläinen, 2002), a widely used metric in information retrieval and recommendation systems to measure the quality of ranking results. NDCG accounts for the relevance of the results and assigns higher weights to more relevant results that appear earlier in the ranking, thus achieving better retrieval outcomes. The NDCG scores of the retriever used in CToolEval are as follows.
NDCG@K refers to the NDCG at a given rank position k. Average NDCG@1 is the average NDCG value for results ranked first in each query, while Average NDCG@3 and Average NDCG@5 are the average NDCG values for results ranked third and fifth, respectively. These values help evaluate the quality of a retrieval across different lengths of search. The retrieval scores indicate that the retriever used in CToolEval effectively returns the most matching tools.

heading: D Evaluation Results of Baseline Model Provided With Gold APIs
text: To explore the effect of retriever on tool invocation capabilities, we provided only the gold APIs to InternLM-7B, InternLM-20B, and GPT-3.5-turbo, and documented their performance on the test set. The experimental results are shown in Table 8 and 9.

heading: E Evaluation Results of LLMs Being Fine-tuned on CToolEval
text: We employed the training set of CToolEval for finetuning using two parameter-efficient fine-tuning methods, QLoRA and LoRA, on Baichuan2-7bchat and Qwen-7b-chat. The experimental results on the test set are presented in Table 10 and 11.
1https://huggingface.co/google-bert/bert-base-chinese

heading: Future Work
text: By dissecting the errors and pinpointing the nuances in the operational use of APIs, our work contributes to a more nuanced understanding and evaluation of the agent-level capabilities of LLMs and opens up avenues for future research and development of LLM-powered agents in real-world scenarios. Limitations
Although we have established a Chinese benchmark that simulates real-world App interactions to conduct a comprehensive analysis of existing LLMs and outlined various existing issues, our work has two notable limitations. First, since the APIs we use are publicly accessible, there might be instances where an API interface becomes faulty over time, affecting our evaluation results, but this is an unavoidable aspect of simulating real-world tool interactions. Second, due to the token usage limitations of GPT-4, our ability to evaluate all existing LLMs is restricted. It is important to emphasize that we meticulously selected the most representative Chinese LLMs for evaluation and anal-
ysis, aiming to identify key problems with Chinese LLMs in agentization