Venue: NAACL 2025

# Octopus: On-device language model for function calling of software APIs

Wei Chen\({}^{1}\), Zhiyuan Li\({}^{\dagger}\), Mingyuan Ma\({}^{\dagger}\)

Nexa AI

alexchen@nexa.ai, zack@nexa.ai, mingyua_ma@nexa.ai

Corresponding author, \({}^{\dagger}\) equal contribution

###### Abstract

Large Language Models (LLMs) are pivotal for advanced text processing and generation. This study presents a framework to train a series of on-device LLMs optimized for invoking software APIs. Using a curated dataset of 30,000 API function calls from software documentation, we fine-tune LLMs with 2B, 3B, and 7B parameters to enhance their proficiency in API interactions. Our approach improves the understanding of API structures and syntax, leading to significantly better accuracy in API function calls. We also propose a conditional masking technique to enforce correct output formats, significantly reducing generation format errors while maintaining inference speed. This technique is specifically tailored for API tasks. Our fine-tuned model, Octopus, outperforms GPT-4 in API calling tasks, showcasing advancements in automated software development and API integration. The model checkpoints are publicly available.

## 1 Introduction

The advent of Large Language Models (LLMs) has revolutionized artificial intelligence, enabling transformative applications in natural language processing and specialized domains such as mathematics [9, 8], healthcare [56, 103], and legal analysis [11, 12]. Despite their advancements, LLMs face challenges in adapting to real-time updates and performing domain-specific tasks like image/video editing [11] or complex tax filings. Integrating LLMs with external APIs offers a solution, enabling real-time access to specialized resources and fostering innovations such as code interpreters [11, 12]. Research on ToolAlpaca [104] and NexusRaven [11] demonstrates the potential of open-source LLMs in function-calling scenarios, extending their utility to IoT, edge computing, and automated software development.

Enhancing LLM integration with APIs requires balancing large-scale model capabilities and efficiency. While large models like GPT-4 [103, 104, 105] are powerful, they are computationally expensive for tasks using only a subset of APIs. Smaller, task-specific LLMs offer a cost-effective alternative [104, 105, 106] but risk increased errors or "hallucinations" [104, 105]. Precise output formatting is critical for software reliability [104], emphasizing the need for innovations that combine accuracy, efficiency, and reliability.

To address these challenges, we propose a framework for training and inference tailored to task-specific LLMs. Using a curated dataset of over 30,000 APIs from Rapid API Hub (rap, 2024), we employ curriculum learning [105] to improve precision in selecting appropriate API functions. Fine-tuning smaller models like Codellama7B [104], Google's Gemma (Gemma Team, Google DeepMind, 2023), and Stable Code 3B [104] demonstrates superior performance over GPT-4 on specific benchmarks. The framework also supports deployment on resource-constrained platforms such as mobile devices (team, 2023), ensuring broad applicability.

To ensure output consistency, we introduce a conditional masking technique tailored for API function calls. Unlike generic constrained decoding, this approach dynamically restricts token predictions to valid options based on the API schema, such as permissible parameter types and argument names. This guarantees syntactic and semantic

correctness, significantly reducing errors while preserving inference speed. Mathematical validation further demonstrates consistent improvements in accuracy, making this technique reliable for diverse real-world API interactions.

In summary, this paper makes the following key contributions:

* **Task-Specific Framework**: We introduce a training and data-cleaning framework, with a high-quality dataset of over 30,000 APIs from RapidAPI Hub, to fine-tune smaller, task-oriented LLMs for API function calls. This reduces operational costs while maintaining high accuracy, enabling on-device inference for resource-constrained environments like mobile devices and IoT systems.
* **Conditional Masking Technique**: A tailored technique ensuring syntactic and semantic correctness in API calls, addressing formatting errors and hallucinations. It dynamically enforces schema adherence without compromising inference speed.
* **Superior Performance and Model Checkpoint**: Leveraging curriculum learning and innovative dataset engineering, our models surpass GPT-4 in API function accuracy. Our Octopus series models are publicly available.

These contributions collectively advance the field of automated software development by addressing critical inefficiencies in LLM deployment for API interactions, providing open resources for the community, and setting a foundation for further research in task-specific LLM optimization and application.

## 2 Related Work

Enhancing LLMs with ToolsThe integration of external tools into Large Language Models (LLMs) like GPT-4, Alpaca, and Llama significantly enhances their capabilities. Early efforts focused on model-specific fine-tuning Lin et al. (2024); Hu et al. (2023); Schick et al. (2024); Zhang et al. (2023b), which faced challenges in flexibility. The adoption of prompt-based approaches broadened accessibility, enabling models to use code interpreters and retrieval frameworks Zhou et al. (2023); Zhang et al. (2023a). Developments in simulated tool environments Shen et al. (2024); Du et al. (2024); Xi et al. (2023) and API interaction frameworks Li et al. (2023) have further expanded tool capabilities. Additionally, advanced reasoning strategies Valmeekam et al. (2022); Hao et al. (2023); Lewkowycz et al. (2022) improve the efficiency of solving complex tasks. Some existing works demonstrate some solutions. For example, language models can teach themselves to use external tools via simple APIs and achieve the best of both worldsSchick et al. (2023).

Dataset FormatOptimizing datasets Zhuang et al. (2024); Kong et al. (2023) is critical for fine-tuning LLMs. Multi-stage refinements with models like GPT-4 and Alpaca iteratively improve prompts and develop advanced chain-of-thought processes Wang et al. (2023); Zhang et al. (2022); Shridhar et al. (2023); Zheng et al. (2023a); Wei et al. (2022). These refinements significantly enhance function-calling accuracy and establish benchmarks for dataset quality and model training, shifting the focus toward improved output precision.

Robustness in LLM GenerationUnlike article generation, software applications require strict adherence to structured output formats, such as JSON Zheng et al. (2023b). Format consistency issues in LLM outputs Vaswani et al. (2017); Ackerman and Cybenko (2023) have driven research into rigid format enforcement. Frameworks like LangChain Harrison (2022) introduce parsers for formats like YAML, JSON, CSV, but such tools often fail for complex cases like function call responses, where precise argument and schema adherence is critical.

Constrained DecodingThe use of constrained decoding techniques has been explored to address format consistency in LLM outputs. Grammar-constrained decoding Geng et al. (2023) enforces grammar rules, finite-state machines (FSM) Zhang et al. (2024) ensure syntax compliance, and monitor-guided decoding Agrawal et al. (2023) restricts vocabulary to predefined subsets. While effective for structured text generation, these methods struggle with API function calls due to their inability to capture nuanced API-specific requirements. Grammar-constrained decoding fails to adapt to diverse schemas, FSMs lack scalability for large argument spaces, and monitor-guided decoding cannot enforce structural or type-specific constraints.

Our proposed _conditional masking technique_ overcomes these limitations by dynamically adapta 

ing token predictions to API schemas. It integrates context-sensitive constraints at runtime, enforcing syntactic and semantic correctness to ensure outputs align with API specifications. This tailored approach addresses the gaps in existing methods, making it uniquely suited for reliable and accurate API function generation.

## 3 Methodology

In this section, we outline our approach to dataset collection, preparation, and model development, detailing the steps taken to optimize the training process for API function calling tasks. We introduce the workflow designed to curate, format, and refine the dataset to ensure its suitability for effective model fine-tuning. Furthermore, we describe the architecture and training process of our model, _Octopus_, including the innovative techniques applied to enhance inference accuracy and efficiency.

### Dataset Collection and Refinement

The initial dataset was sourced from RapidAPI Hub, a prominent repository with extensive and diverse API documentation, selected for its large developer base and relevance to real-world applications. We focused on approximately 30,000 frequently utilized APIs to ensure broad applicability.

The dataset preparation process involved two main stages. In the initial collection phase, we systematically gathered raw API documentation, capturing function names, descriptions, argument types, and return formats. This provided an unprocessed view of widely used APIs. The refinement phase focused on optimizing the dataset for training through standardization, validation, and error correction. Formats across APIs were standardized for consistency in naming conventions and schema representations. Large language models such as GPT-3.5 and CodeLlama 70B were employed to fill in missing details, validate accuracy, and align descriptions with Google Python Style guidelines. Errors, duplicates, and overly verbose descriptions were corrected to create a concise and informative dataset.

This structured approach ensured high-quality data inputs, critical for the effective fine-tuning of the Octopus model.

### Single API Data Preprocess

From our detailed exploration of RapidHub's API documentation, we derived a comprehensive understanding of how API usage examples are structured and utilized. The preprocessing approach involves meticulously extracting API usage examples, which include the API's name, description, argument names, and their respective descriptions, and formatting this information in JSON. This data is then reorganized using OPENAI GPT-3.5 and CodeLlama 70B models to align with standardized organizational guidelines.

Function names are refined based on their descriptions to ensure they are concise and informative, and arguments' names and descriptions are carefully captured. To mitigate potential inaccuracies ("hallucinations") from smaller LLMs, we adopt the Python coding format. This strategic decision leverages the inherent code reasoning capabilities of models such as CodeLlama7B and StableCode3B, which are pretrained on extensive code datasets. This process streamlines API information for enhanced usability while leveraging advanced AI models to present the information in a structured and accessible manner. By prioritizing function descriptions for renaming and thoroughly detailing argument names and descriptions, we ensure that essential elements of API usage are conveyed effectively, enabling developers to integrate these APIs seamlessly into their projects.

**Example Converted Function:**

```
defget_flight_details(flight_id):""" Getdetailedinformationon specificflights,includingrealtimetracking, departure/arrivaltimes,flightpath,andstatusinsightsArgs: flight_id(string):Theflight_id representstheIDofaflight."""
```

In our methodology, we deliberately excluded the function body from the final dataset compilation. Through a meticulous selection process, we aggregated approximately 30,000 APIs, employing OPENAI GPT-4 for a comprehensive examination to identify and remove APIs with deficiencies, such as missing arguments or inconsistencies between function descriptions and their parameters 

This stringent selection criterion was pivotal in assuring the dataset's quality. Each API underwent this rigorous scrutiny, culminating in the compilation of Dataset A, which serves as the foundation for subsequent data processing.

### Dataset Refinement

Dataset RefinementTo enhance the decision-making capabilities of Large Language Models (LLMs) for real-world API usage, we propose a sophisticated dataset construction approach. This process is central to our study, as it ensures the model's ability to effectively handle diverse and challenging scenarios. Our methodology begins by integrating various functions, intentionally incorporating irrelevant ones to create a complex training environment for the LLM. Inspired by curriculum learning, we gradually introduce hard negative samples, incrementally increasing the difficulty of selecting the most relevant function. Figure 1 illustrates the detailed pipeline for compiling the dataset. Below, we outline the key techniques employed in this process.

1. **Negative Samples:** To improve the model's reasoning capabilities and applicability, we incorporate both positive and negative examples into the dataset. The ratio of positive to negative samples is represented as \(\frac{M}{N}\) in Figure 1, where we set \(M\) and \(N\) both equal to 1. This balance ensures a robust training setup, enabling the model to distinguish between correct and incorrect API calls effectively.
2. **Similar Function Clustering:** To further challenge the model, we introduce semantically similar functions into the training data. For each data point, three similar functions are selected based on their vector embeddings, computed from function descriptions. Milvus is used to facilitate this similarity search, and functions ranked between 5 and 10 by similarity scores are chosen to avoid redundancy while maintaining diversity. This approach cultivates a model capable of differentiating between closely related functions in real-world applications.
3. **GPT-4 Generated Queries:** High-quality queries are essential for effective training. Positive queries are generated using GPT-4, ensuring each query is solvable by a single API. To further enhance training, we include Chain of Thought (CoT) reasoning for these queries. CoT annotations have been shown to significantly improve model reasoning abilities and performance Srinivasan et al. (2023). This step ensures that the training data not only covers diverse scenarios but also supports advanced reasoning.
4. **GPT-4 Verification:** While GPT-4 is highly capable, its outputs are not immune to errors. To address this, we implemented a self-verification workflow using GPT-4 to identify and rectify inaccuracies. After compiling the initial dataset (Dataset A), GPT-4 was employed to meticulously verify the data, eliminating approximately 1,000 data points that failed to meet our stringent quality standards. This rigorous process resulted in Dataset B, a highly optimized dataset for training.

Using this methodology, we compiled a robust training dataset consisting of approximately 150,000 data points. Each API is associated with five positive queries it can resolve. To provide a comprehensive understanding of the dataset, a sample of the complete dataset is included in the Appendix (B.1), showcasing its detailed structure and composition.

### Octopus

To validate the efficacy of our framework, we fine-tuned four open-source models: CodeLlama7B,

Figure 1: Refining Dataset A into Dataset B through a rigorous workflow. This process involves three critical steps: generating positive queries solvable by specific APIs and corresponding Chain of Thoughts (CoT); introducing unsolvable queries and augmenting them with irrelevant function bodies; and incorporating semantically similar functions using vector embeddings. Following GPT-4’s verification, Dataset B emerges as an optimized dataset for training, designed to enhance model performance significantly.

 

Google Gemma 2B & 7B, and Stable Code LM 3B. A standardized training template, detailed in Appendix (B.1), was applied across all models. We employed LoRA with 8-bit quantization and allocated GPU hours on A100 80GB as follows: 90h for CodeLlama7B and Google Gamma 7B, 30h for Google Gemma 2B, and 60h for Stable Code LM 3B. The learning rate was set at \(5\times 10^{-5}\) with a linear scheduler for optimization. During inference, user queries trigger function retrieval and execution by mapping generated functions and arguments to corresponding APIs, ensuring accurate responses.

Experiments with different LoRA setups revealed that the optimal configuration uses a LoRA rank of 16 applied to the layers "q_proj", "v_proj", "o_proj", "up_proj", "down_proj". Training followed a curriculum learning strategy, progressively introducing data points with more similar examples. Training and validation losses for selected models are shown in Figure (2).

### Inference using conditional mask

The utilization of smaller-parameter Large Language Models (LLMs) has a pivotal challenge: a noticeable decrement in robustness when generating outputs. This challenge is also observed in our model, which necessitates the need to enforce the response with precise function names along with their corresponding arguments. The expected output format demands that arguments be encapsulated within parentheses, function names align with a pre-defined repository, and argument values conform to their designated types. Discrepancies, such as typographical errors in function names or misalignment in argument types, critically undermine the integrity of the output, rendering it susceptible to errors. For instance, both in GPT-4 and our model, deviations in the function name--whether through misspelling or elongated expressions--can lead to unintended corrections that fail to map back to the original function names, thereby distorting the intended output. The original LLM, denoted as \(\pi\), generation process to sample the next token is

\[P(x_{t+1}|x_{1:t}) =P(x_{t+1}|x_{1:t};\pi), \tag{1}\] \[x_{t+1} =\operatorname*{argmax}P(x_{t+1}|x_{1:t};\pi)\]

where \(x_{1:t}\) is all the current tokens, with the sequence length as \(t\), and \(x_{t+1}\) is the next token to be sampled. What we do here is to introduce another dynamic mask dependent on \(x_{1:t}\) so that

\[x_{t+1}=\operatorname*{argmax}\left[P(x_{t+1}|x_{1:t};\pi)\odot\mathrm{mask }(x_{1:t})\right]. \tag{2}\]

In constructing the dynamic mask, we designate all tokens, which are not aligned with correct format, to be masked by assigning a value of 0 to their respective positions, and a value of 1 to all other positions. For example, if we already know the next token represents integers, we will only unmask the tokens that are used for integers. Therefore, the formulation of an accurate _mask_ is paramount for achieving the desired outcome. In this context, we delineate several methodologies that were investigated for the derivation of the _mask_.

* **enum data type** Function names are usually already known, and will not change during inference. We can treat them as enumerable data variables. To efficiently manage these names, a Trie tree can be constructed, facilitating the retrieval of the _mask_ with a time complexity of \(O(D)\), where \(D\) denotes the Trie tree's depth, equivalent to the maximum length of a function name, which in our case is approximately 20. This result in the constant time complexity. As an alternative approach, storing all prefixes of potential function names within a dictionary could further reduce the complexity to \(O(1)\). The implementation of the Trie class is provided in the Appendix (B.2).
* **string, float, dict, int type** Regular expressions can be employed to analyze subsequent tokens and generate the conditional mask.

Therefore, we can confirm that the output result is free from formatting errors. Our experimental

Figure 2: The training and validation loss for selected pretrained models



findings indicate that the application of the conditional mask significantly enhances the robustness of the Large Language Model (LLM) in the context of function calls.

## 4 LLM Evaluation for Function Calling

We evaluated the Octopus model's ability to interpret and execute API function calls, comparing its performance to GPT-4 and GPT-3.5-turbo. The evaluation focused on function name recognition and parameter generation, with and without the use of conditional masking. The test set contains a vast diversity of APIs in the real world.

### Evaluation Dataset and Benchmark

To benchmark function calls for commonly used APIs, we constructed an evaluation dataset and sampling queries tailored to these APIs. Queries were generated using the same prompt template as training (Appendix B.1). Solvable queries, requiring a single API to resolve, were balanced with unsolvable queries in a 1:1 ratio to test model robustness against ambiguous inputs. Human annotations ensured accurate ground truth, and minor format discrepancies (e.g., JSON issues) were overlooked for models not fine-tuned on this dataset to focus on semantic correctness.

### Without Conditional Masking

In the initial evaluation, responses were generated without conditional masking. Greedy decoding was used across all models to prioritize precision in function name and argument selection. As shown in Figure 3, GPT-4 achieved the highest accuracy among pre-trained models. However, it exhibited common issues such as correcting typos in function names (e.g., send_emil to send_email), which deviated from input queries, and generating invalid parameters like Australian instead of a valid country name. While GPT-3.5 and GPT-4 performed well in function name recognition, their accuracy declined when generating contextually appropriate parameters.

### With Conditional Masking

To address these challenges, we applied conditional masking during inference for Octopus models. This technique constrained token predictions to align with API schema requirements, such as valid parameter types and enumerations. As illustrated in Figure 4, conditional masking significantly improved parameter generation accuracy, particularly for structured inputs like country names. By enforcing schema adherence, the Octopus models avoided errors observed in pre-trained models. However, since GPT-3.5 and GPT-4 APIs do not expose logits, conditional masking could not be applied, leaving their metrics unchanged. With this enhancement, Octopus variants matched or surpassed GPT-4's accuracy, demonstrating the efficacy of conditional masking in improving model reliability.

### Discussion and Key Insights

GPT-4 demonstrated high accuracy in function name recognition but lacked schema constraints, leading to frequent parameter errors. Conditional masking significantly enhanced Octopus models,

Figure 4: Accuracy comparison between GPT-3.5, GPT-4, and Octopus models with conditional masking.

Figure 3: Accuracy comparison between GPT-3.5, GPT-4, and Octopus models without conditional masking.

 

ensuring robust parameter generation for real-world API tasks. Without masking, parameter errors were prevalent, particularly for ambiguous or complex queries. These findings underscore the importance of schema-aware mechanisms like conditional masking for improving LLM performance in structured tasks.

## 5 Conclusion

This study introduces a novel framework for training large language models on practical software APIs and evaluates their performance in API calling tasks, surpassing GPT-4 in specific scenarios. Our approach includes a refined dataset preparation methodology, leveraging negative sampling and curriculum learning to enhance model performance. Additionally, we propose a conditional masking technique to address mismatched output formats, significantly improving accuracy and robustness in API function generation.

## References

* R. Abadi et al. (2019)DBLAS: A survey of machine translation techniques. In Proceedings of the 2nd International Eurographics Conference, pp. 378-386. Cited by: SS1.
* J. Ackerman and G. Cybenko (2023)Large language models for fuzzing parsers (registered report). In Proceedings of the 2nd International Fuzzing Workshop, pp. 31-38. Cited by: SS1.
* L. A. Agrawal, A. Kanade, N. Goyal, S. K. Lahiri, and S. K. Rajamani (2023)Guiding language models of code with global context using monitors. Preprint arXiv:2306.10763. Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, Arvind Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 1573-0221, Link, Document Cited by: SS1.
* M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Pande de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, E. Khalat 3 GrishSastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. Setroski Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. Hebgen Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlisi, I. Sutskever, and W. Zaremba (2021)Evaluating large language models trained on code. arXiv:2107.03374. Cited by: SS1.
* W. Chen, Z. Li, and S. Xin (2024)Omnivlm: a token-compressed, sub-billion-parameter vision-language model for efficient on-device inference. preprint arXiv:2412.11475. Cited by: SS1.
* J. Cui, Z. Li, Y. Yan, B. Chen, and L. Yuan (2023)Chatlaw: open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092. Cited by: SS1.
* Y. Du, F. Wei, and H. Zhang (2024)Anytool: self-reflective, hierarchical agents for large-scale api calls. arXiv preprint arXiv:2402.04253. Cited by: SS1.
* Z. Fei, X. Shen, D. Zhu, F. Zhou, Z. Han, S. Zhang, K. Chen, Z. Shen, and J. Ge (2023)Lawbench: benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289. Cited by: SS1.
* T. Fu, W. Hu, X. Du, W. Yang Wang, Y. Yang, and Z. Gan (2023)Guiding instruction-based image editing via multi-modal large language models. arXiv preprint arXiv:2309.17102. Cited by: SS1.
* G. Team, Google DeepMind (2023)GEMA: open models based on gemini research and technology. Cited by: SS1.
* S. Geng, M. Josifoski, M. Peyrard, and R. West (2023)Grammar-constrained decoding for structured NLP tasks without finetuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 10932-10952. Cited by: SS1.
* S. Hao, Y. Gu, H. Ma, J. Jiahua Hong, Z. Wang, D. Zhe Wang, and Zhiting Hu (2023)Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992. Cited by: SS1.
* C. Harrison (2022)Langchain. Note: Accessed on February 29, 2024 Cited by: SS1.
* J. He-Yueya, G. Poesia, R. E. Wang, and N. D Goodman (2023)Solving math word problems by combining language models with symbolic solvers. arXiv preprint arXiv:2304.09102. Cited by: SS1.
* Z. Hu, Y. Lan, L. Wang, W. Xu, E. Lim, R. Ka-Wei Lee, L. Bing, and S. Poria (2023)LIm-adapters: an adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933. Cited by: SS1.
* S. Imani, L. Du, and H. Shrivastava (2023)MathPromper: mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398.

 

Ziwei Ji, YU Tiezheng, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards mitigating llm hallucination via self reflection. In The _2023 Conference on Empirical Methods in Natural Language Processing_.
* Jiang et al. (2023) Zhihan Jiang, Jinyang Liu, Zhuangbin Chen, Yichen Li, Junjie Huang, Yintong Huo, Pinjia He, Jiazhen Gu, and Michael R Lyu. 2023. LLmparser: A llm-based log parsing framework. _arXiv preprint arXiv:2310.01796_.
* Jo et al. (2023) Eunkyung Jo, Daniel A Epstein, Hyunhoon Jung, and Young-Ho Kim. 2023. Understanding the benefits and challenges of deploying conversational ai leveraging large language models for public health intervention. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_, pages 1-16.
* Kong et al. (2023) Yilun Kong, Jingqing Ruan, Yihong Chen, Bin Zhang, Tianpeng Bao, Shiwei Shi, Guoqing Du, Xiaoru Hu, Hangyu Mao, Ziyue Li, et al. 2023. Tptu-v2: Boosting task planning and tool usage of large language model-based agents in real-world systems. _arXiv preprint arXiv:2311.11315_.
* Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. _Preprint_, arXiv:2206.14858.
* Li et al. (2023) Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llvms. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
* Lin et al. (2024) Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua. 2024. Data-efficient fine-tuning for llm-based recommendation. _arXiv preprint arXiv:2401.17197_.
* Liu et al. (2024) Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, and Wei Lu. 2024. Let's learn step by step: Enhancing in-context learning ability with curriculum learning. _arXiv preprint arXiv:2402.10738_.
* Pallagani et al. (2024) Vishal Pallagani, Kaushik Roy, Bharath Muppassani, Francesco Fabiano, Andrea Loreggia, Keerthiram Murugesan, Biplav Srivastava, Francesca Rossi, Lior Horesh, and Amit Sheth. 2024. On the prospects of incorporating large language models (llms) in automated planning and scheduling (aps). _arXiv preprint arXiv:2401.02500_.
* Pinnaparaju et al. (2023) Nikhil Pinnaparaju, Reshinthith Adityan, Duy Phung, Jonathan Tow, James Baicoianu, and Nathan Cooper. 2023. Stable code 3b.
* Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yoossi Adi, Jingyu Liu, Tal Remez, Jeremy Rapin, et al. 2023. Code llama: Open foundation models for code. _arXiv preprint arXiv:2308.12950_.
* Schick et al. (2024) Timo Schick, Jane Dwivedi-Yu, Roberto Dessl, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. _Advances in Neural Information Processing Systems_, 36.
* Schick et al. (2022a) Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. _Preprint_, arXiv:2302.04761.
* Shen et al. (2024a) Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. 2024a. Small llms are weak tool learners: A multi-llm agent. _arXiv preprint arXiv:2401.07324_.
* Shen et al. (2024b) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2024b. Hugging gpt: Solving ai tasks with chatgpt and its friends in hugging face. _Advances in Neural Information Processing Systems_, 36.
* Shridhar et al. (2023) Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ram Pasunuru, Mrnimaya Sachan, Jason Weston, and Asli Celikyilmaz. 2023. The art of llm refinement: Ask, refine, and trust. _arXiv preprint arXiv:2311.07961_.
* Srinivasan et al. (2023) Venkat Krishna Srinivasan, Zhen Dong, Banghua Zhu, Brian Yu, Damon Mosk-Aoyama, Kurt Keutzer,iantao Jiao, and Jian Zhang. 2023. Nexusraven: a commercially-permissive language model for function calling. In _NeurIPS 2023 Foundation Models for Decision Making Workshop_.
* Tang et al. (2023) Qiayou Tang, Ziliang Deng, Hongyu Lin, Xianguei Han, Qiao Liang, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. _arXiv preprint arXiv:2306.05301_.
* team (2023) MLC team. 2023. MLC-LLM.
* Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. _Nature medicine_, 29(8):1930-1940.
* Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In _Chi conference on human factors in computing systems extended abstracts_, pages 1-7.
* Valmeekam et al. (2022) Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can't plan (a benchmark for llms on planning and reasoning about change). _arXiv preprint arXiv:2206.10498_ 

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 Attention is all you need. _Advances in neural information processing systems_, 30.
* Wang et al. (2023) Hongru Wang, Rui Wang, Fei Mi, Zezhong Wang, Ruifeng Xu, and Kam-Fai Wong. 2023. Chain-of-thought prompting for responding to in-depth dialogue questions with llm. _arXiv preprint arXiv:2305.11792_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in Neural Information Processing Systems_, 35:24824-24837.
* Wu et al. (2023) Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, and Chi Wang. 2023. An empirical study on challenging math problem solving with gpt-4. _Preprint_, arXiv:2306.01337.
* Xi et al. (2023) Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. _arXiv preprint arXiv:2309.07864_.
* Xu et al. (2024) Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, and Ziyuan Ling. 2024. On-device language models: A comprehensive review. _arXiv preprint arXiv:2409.00088_.
* Yao et al. (2023) Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, and Li Yuan. 2023. Llm lies: Hallucinations are not bugs, but features as adversarial examples. _arXiv preprint arXiv:2310.01469_.
* Zhang et al. (2023a) Kechi Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. 2023a. Toolcoder: Teach code generation models to use API search tools. _CoRR_, abs/2305.04032.
* Zhang et al. (2023b) Kechi Zhang, Huangzhao Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. 2023b. Toolcoder: Teach code generation models to use api search tools. _arXiv preprint arXiv:2305.04032_.
* Zhang et al. (2024)Kexun Zhang, Hongqiao Chen, Lei Li, and William Yang Wang. 2024. Tooldec: Syntax error-free and generalizable tool use for LLMs via finite-state decoding.
* Zhang et al. (2022) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. _arXiv preprint arXiv:2210.03493_.
* Zheng et al. (2023a) Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023a. Progressive hint prompting improves reasoning in large language models. _Preprint_, arXiv:2304.09797.
* Zheng et al. (2023b) Linamin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyzrakis, Ion Stoica, Joseph E Gonzalez, et al. 2023b. Efficiently programming large language models using sglang. _arXiv preprint arXiv:2312.07104_.
* Zhou et al. (2023) Xuanhe Zhou, Guoliang Li, and Zhiyuan Liu. 2023. Llm as dba. _arXiv preprint arXiv:2308.05481_.
* Zhuang et al. (2024) Yuchen Zhuang, Yue Yu, Kuan Wang, Haotiian Sun, and Chao Zhang. 2024. Toolqa: A dataset for llvm question answering with external tools. _Advances in Neural Information Processing Systems_, 36.

## Appendix A Mathematical Derivation

### Impact of conditional masking on inference performance

In this appendix, we examine the effect of applying a conditional mask during inference on a causal language model's accuracy and validation loss. Consider the validation loss without masking defined as:

\[L^{\text{non-mask}}_{\text{val}}=\sum_{i\in V}-y_{i}\log(\hat{y}_{i}), \tag{3}\]

where \(V\) denotes the vocabulary set, and \(y_{i}\) is a binary indicator (0 or 1) if class label \(i\) is the correct classification for the current observation.

Introducing a conditional mask allows us to partition the vocabulary \(V\) into two subsets: \(V_{1}\), containing indices not masked, and \(V_{2}\), containing indices that are masked. Given that the true label \(y_{i}\) belongs to \(V_{1}\) during inference, and considering that for all \(i\),

\[-y_{i}\log(\hat{y}_{i})>0, \tag{4}\]

the validation loss with masking can be expressed as:

\[L^{\text{mask}}_{\text{val}}=\sum_{i\in V_{1}}-y_{i}\log(\hat{y}_{i})<L^{\text {non-mask}}_{\text{val}}, \tag{5}\]

indicating that the validation loss is reduced when a conditional mask is applied during inference.

Accuracy, particularly precision in this context, for the non-masked scenario is determined by the alignment between the ground truth label's index and the index of the maximum value in the predicted distribution:

\[\text{Precision}^{\text{non-mask}}=\mathds{I}[\text{argmax}_{i}(y_{i})=\text{ argmax}_{i}(\hat{y}_{i})],\] (6) 

where \(\mathds{1}[\cdot]\) is the indicator function, returning 1 if the condition is true, and 0 otherwise.

With conditional masking, the prediction \(\hat{y}_{i}\) is constrained to \(V_{1}\), effectively reducing the search space for \(\text{argmax}_{i}(\hat{y}_{i})\) and increasing the likelihood of matching \(\text{argmax}_{i}(y_{i})\), given that \(y_{i}\in V_{1}\). Hence,

\[\text{Precision}^{\text{mask}}\geq\text{Precision}^{\text{non-mask}}, \tag{7}\]

demonstrating that conditional masking during inference not only reduces validation loss but also enhances the model's precision by focusing on a more relevant subset of the vocabulary.

## Appendix B Dataset and code illustration

### Dataset template

""" You are an assistant, and you need to call find appropriate functions according to the query of the users. Firstly, find the relevant functions, then get the function arguments by understanding the user's query. The following functions are available for you to fetch further data to answer user questions: Function: def no_relevant_function(user_query): ... Call this when no other provided function can be called to answer the user query. Args: user_query (str): The user_query that cannot be answered by any other function calls. ...

def youtube_downloader(videurl): ''' Get direct video URL for youtube to download and save for offline viewing or sharing. Args: videurl (string): The URL of the video being accessed as a string. ...

def facebook_dl_link(url): ... Get downloadable link for facebook, allowing convenient offline viewing and sharing. Args: url (string): The URL string for the function argument. ...

def pinterest_video_dl_api(url): ... Get download feature for videos from Pinterest enabling users to save videos for offline viewing. Args: url (string): The URL string represents the web address of the resource being accessed. ...

def insta_download_url(url): "" Get download access to Instagram content by inputting the URL, enabling users to save and view content offline. Args: url (string): The URL string. ...

Oblation download access for viewing a recent Instagram post offline using the URL https://www. instagram.com/p/CODEinstantiate123/Response:insta_download_url('https:// www.instagram.com/p/CODEinstantiate123/')<im_end> Thought:To acquire download access for Instagram content for offline 

viewing, 'insta_download_url' is called with the post's URL as the argument, ensuring the content specified by the URL is fetched for download.

### Trie class to process the enum variable

class TrieNode: def__init__(self) -> None: self.children: Dict[str, TrieNode] = {} self.isEndOfWord: bool = False

class Trie: def__init__(self) -> None: self.root: TrieNode = TrieNode() def insert(self, word: str) -> None: node = self.root for char in word: ifchar not in node. children: node.children[Char] = TrieNode() node = node.children[Char ] node.isEndOfWord = True def is_prefix(self, prefix: str) -> bool: node = self.root for char in prefix: ifchar not in node. children: return False node = node.children[Char ] return True def get_all_prefixes(self) -> List[Str]: prefixes: List[Str] = prefix: str, prefixes: list[Str] -> None: ifnode != self.root: prefixes.append(prefix) for char, next_node in node. children.items(): self.dfs(next_node, prefix + char, prefixes) def search(self, prefix: str, include_prefix: bool = True) -> List[Str]: node = self.root for char in prefix: ifchar not in node. children: return [] nder node = node.children[Char ] initial_string: str = prefix ifinclude_prefix else "" returnself. find_words_from_node(node, initial_string) def _find_words_from_node(self, node: TrieNode, current_string: str) -> List[Str]: words: List[Str] = [] ifnode.isEndOfWord: words.append( current_string) for char, next_node in node. children.items(): words.extend(self. find_words_from_node(next_node, current_string + char)) return words