Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning
Authors: Mingyang Chen
sunhaoze
Tianpeng Li
Fan Yang
Hao Liang
KeerLu
Bin CUI
Wentao Zhang
Zenan Zhou
weipeng chen
Venue: ICLR 2025 Poster (2025)
Keywords: large language model
function calling
instruction tuning
synthetic data
Primary area: datasets and benchmarks
TL;DR: We introduce a "bottom-up then top-down" method to enhance LLMs' multi-turn function calling by generating synthetic compositional instruction data.

Abstract: Large Language Models (LLMs) have exhibited significant potential in performing diverse tasks, including the ability to call functions or use external tools to enhance their performance. While current research on function calling by LLMs primarily focuses on single-turn interactions, this paper addresses the overlooked necessity for LLMs to engage in multi-turn function calling—critical for handling compositional, real-world queries that require planning with functions but not only use functions. To facilitate this, we introduce an approach, BUTTON, which generates synthetic compositional instruction tuning data via bottom-up instruction construction and top-down trajectory generation. In the bottom-up phase, we generate simple atomic tasks based on real-world scenarios and build compositional tasks using heuristic strategies based on atomic tasks. Corresponding function definitions are then synthesized for these compositional tasks. The top-down phase features a multi-agent environment where interactions among simulated humans, assistants, and tools are utilized to gather multi-turn function calling trajectories. This approach ensures task compositionality and allows for effective function and trajectory generation by examining atomic tasks within compositional tasks. We produce a dataset BUTTONInstruct comprising 8k data points and demonstrate its effectiveness through extensive experiments across various LLMs.

Published as a conference paper at ICLR 2025
FACILITATING MULTI-TURN FUNCTION CALLING FOR LLMs VIA COMPOSITIONAL INSTRUCTION TUNING
Mingyang Chen' " Haoze Sun ' Tianpeng Li Fan Yang', Hao Liang?, Keer Lu?, Bin Cui?, Wentao Zhang"t, Zenan Zhou't, Weipeng Chen' Baichuan Inc., 2 Peking University
fchenmingyang, sunhaoze, litianpeng, zhouzenan| | @ baichuan-inca.o wentao.zhang @ pku.edu.cn
ABSTRACT
Large Language Models (LLMs) have exhibited significant potential in perform- ing diverse tasks, including the ability to call functions use external tools enhance their performance. While current research on function calling by LLMs primarily focuses on single-turn interactions, this paper addresses the overlooked necessity for LLMs to engage in multi-turn function calling calling-critical critical for handling compositional, real-world queries that require planning with functions but not only use functions. To facilitate this, we introduce an approach, BUTTON, which gen- erates synthetic compositional instruction tuning data via bottom-up instruction construction and top-down trajectory generation. In the bottom-up phase, gen- crate simple atomic tasks based on real-world scenarios and build compositional tasks using heuristic strategies based on atomic tasks. Corresponding function def- initions are then synthesized for these compositional tasks. The top-down phase features a multi-agent environment where interactions among simulated humans, assistants, and tools are utilized to gather multi-turn function calling trajectories. This approach ensures task compositionality and allows for effective function and trajectory generation by examining atomic tasks within compositional tasks. We produce dataset BUTTONInst rucct comprising 8k data points and demonstrate its effectiveness through extensive experiments across various L.MMs]
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks (OpenAl 2023 Dubey et al. 2024). Beyond generating human-like text, recent studies have shown that LLMs can also call functions (i.e., use external tools) to perform specific actions or op- erations (Qin et al. 2023 Patil et al. 2023 Mu et al. 2024 Liu et al. 2024b). This ability enhances LLM performance, such as retrieving information from external knowledge bases to reduce hallu- cinations (Schick et al.) 2023 Gao et al. 2023 Zhao et al. 2024 Wang et al., (024c). Moreover, LLM-based agents can interact with numerous external APIs through function calls, providing stan- dardized interfaces that increase their utility and versatility in real-world applications (Chen & Li
2024
Wang et al. 2024a Zeng et al. 2024).
Existing research aligning LLMs for function calling predominantly focuses single-turn approach, primarily constructing instruction data to teach and evaluate them on selecting appropriate functions and providing the correct arguments (Patil et al. 2023 Liu et al. 2024b). While it is important for LLMS to learn how to understand and use functions. these studies often overlook the crucial ability to plan with functions. Many real-world user queries are complex and cannot be resolved in a single step. For example, "List the flight schedule from London to Edinburgh" may be a single-step task since simply retrieving exact information can complete it, while "Book me the first flight from London to Edinburgh" requires calling multiple functions sequentially: first retrieving the flight schedule and finding the first one, then booking ticket for that flight. Real-world complex
"Equal Contribution
1 'Corresponding Author
'The data is available at httpps://// httttp/::/ggthub..coo/oo/ chuan- BaaonnMi..ystellllbee www.sssttel....s.tt...

---

Published as a conference paper at ICLR 2025
tasks are inherently compositional (Press et al. LLMs not only to invoke functions but also to decompose these tasks into manageable steps and plan the sequence of function calls. Therefore, in this study, we focus constructing an instruction- tuning (Ouyang et al., 2022, Longpre et al. 2023 Lin et al.) 2024) dataset where the inputs are complex compositional queries and the outputs are their decompositions into multi-turn function calls, addressing real-world scenarios of function calling for LLMS via Compositional Instruction Taming.
2023 Hayati et al. 2024 Hu et al. 2024 requiring
However, acquiring such data from existing sources is unrealistic. First, selecting and identifying instructions in a compositional manner is challenging (Hayati al., [024] Hu et al. (2044) and finding instructions paired with their corresponding functions is even more difficult (Shen et al.
2023
Schick et al. 2023 Patil et al. 2023). Additionally, we need a "solution" in the form of labeled multi-turn function calls that align with the compositional instructions based on the given functions.
All these factors make difficult to obtain such data without extensive manual annotation. Recently, synthetic data has emerged as promising solution to the lack of manually curated data, with data being created through advanced generative LLMs using tailored processes and simulations (Liu
et al. 2024a). Compared to synthetic data in general domains (Xu et al., 2023 Lou et al.) 2023 Xu et 2024a), data in our scenarios must consider the following challenges: I) How to ensure the compositionality of generated instructions that they are complex, reasonable and solvable: 2) How to ensure the compatibility of an instruction with its functions; 3) How to simulate high-quality multi-turn function calling trajectories without human supervision.
To address these challenges, we propose BUTTON, a "Bottom-Up Then Top-dOwN" pipeline for generating synthetic compositiona instruction tuning data enhance the multi-turn function call- ing abilities of LLMs. In the "Bottom-Up" Up" phase, we begin by generating atomic tasks from general real-world scenarios. These tasks are designed to be simple, clear, and executable in a single step without the need for planning. Compositional tasks are constructed based on atomic tasks using two heuristic strategies: Sequential Composition and paralll-then-SSqueeiie straightforward, these two composition strategies, combined with the entire pipeline, can generate diverse compositional instructions. Then, corresponding functions are generated based on com- positional tasks with the conscious of their atomic tasks. During "Top-Down" phase, we set up multi-agent environment where the human, assistant and tools are simulated by generative LLMs steered by specifically curated system prompts, where tool agents simulated according to pre- vious generated function definitions. The trajectory of multi-turn function calling, initiated by a user and involving interactions between assistant and tools, is collected based on this simulated environment. Finally, the collected trajectories, along with their corresponding function definitions, are filled into predefined prompt template serve as instruction tuning data for LLMs. The bottom-up procedure, rather than generating complex tasks directly, ensures compositionality of instructions (challenge 1). Generating functions with an awareness of the atomic tasks within com- positional tasks makes these functions more general and suitable for fine-grained sub-tasks, rather than being monolithic (challenge 2). Using multi-agents to simulate the trajectories enhances their quality, and examining the sub-tasks for compositional tasks also guides the agents toward effective decomposition and planning with functions (challenge 3). Based on BUTTON, we collected a com- positional instruction tuning dataset called BUTTONInstruCe consisting of 8k high-quality data points labeled with multi-tun turn function call trajectories. We demonstrate that LLMs fine-tuned with BUTTON I nstruct show improved performance multi-turn function calling benchmarks.
Composition. Although
2
METHOD
In this section, we introduce the details of our method for constructing compositional instruction- tuning data for multi-turn function calling tasks. This framework consists of two stages: 1) bottom- up instruction construction, and 2) top-down trajectory generation. We refer to this framework as the "Bottom-Up Then Top-dOwN" pipeline, denoted as BUTTON.
During the bottom-up instruction construction phase, we begin by gathering variety of real-world scenarios. Drawing from these scenarios, we proceed to construct a series of atomic tasks, each as simple as possible and capable of being completed in one step. Compositional tasks will be evolved from such atomic tasks. Lastly, for each compositional task, we further generate function definitions that are likely to be called in this task. The term "bottom-up" signifies approach of constructing
2

---

Published as a conference paper at ICLR 2025
Function Rosulto
Functions
Mutti-Agent
Sortheas" "davcripeinn: Tresseees"
Agent
Setup
Query
Furct
Hi. when the "The Gotsay" born.
Compositional
"The Great Gatsby" bom??
First, need know the author book, <callaget authorCThe Gorshy)
Fitagerold
Trajectory Generation
Thi written Scote Fitageratd.
Shi.rrrrchhitthhhyCCttttt Filagerate)</catt.
Atomic Task
Who the outhor book: "The Great: latsly"?
Ahen was author of this book born?
Sept.
Gonortte aubsciuont
The of Gotaby" Septeeber Sttoal-Sepieeber
1856-/Tinat-
Bottom-Up Instruction Construction
Top-Down Trajectory Generation
Figure 1: Overview of our bottom-up then top-down pipeline.
comprehensive compositional tasks with their corresponding functions, starting from the foundation of simple atomic tasks, which can be shown in the left part of Figure
During the top-down trajectory generation phase, we collect multi-turn function calling interaction trajectories for compositional tasks and functions that were constructed earlier. This step simulates the usage process of functions in each task, providing supervision data that guides the L.MMs in learning how to call functions in multi-turn manners. The term "top-down" indicates that from compositional task with corresponding functions, simulate and gather the interaction trajectories that demonstrate the breakdown of complex tasks and the invocation of corresponding functions in multi-ture manner, as depicted in the right part of Figure In essence, the bottom-up instruction construction is process of composition, while the top-down trajectory generation is process of decomposition.
Definitions We begin by collecting real-world scenarios C (c,} and transforming them into atomic tasks A -aajh For compositional tasks requiring multiple function calls, we generate related atomic tasks from defined atomic task a, forming sub-task set S,. This set is used to compose compositional task C; represented as C = (,)). Functions are then constructed based on these tasks, forming instruction tuples = (.).F. where F, are a set of functions for task c;. These functions are defined by their descriptions, allowing to simulate interactions without actual implementations. The task breakdown S, is retained enhance multi-agent inter- action for trajectory generation, without being disclosed in the final data. After obtaining I, generate user, assistant, and tool interactions, resulting in data D = {(Fi, t.), where t; is the col- lected trajectory. These function definitions and trajectories are integrated into prompt template for instruction-tumn tunning data.
2.1 BOTTOM-UE UP INSTRUCTION CONSTRUCTION
In this section, we detail the procedures for constructing instructions using a bottom-up approach. The process begins with simple scenarios, from which we generate tasks at the atomic level. These atom tasks are then evolved to create more complex, composite tasks along with their corresponding functions.
Scenario Collection To ensure our generated tasks are grounded in everyday experiences and not meaningless, we first extract a series of real-world scenarios from existing datasets that focus on function calling for LLMs (glaiveai, 2023 Qin et al., 2023). Such scenarios can be a concise overview, like "book a flight" or "ordering meals". We then conduct a deduplication operation on the collected scenarios. This involves using sentence embeddings to calculate scenario similarities, and setting a specific threshold to filter out similar ones (Xiao et al., 2023). Furthermore, to expand scenario collection, we also attempt to generate new scenarios from existing ones by altering
3

---

Published as a conference paper at ICLR 2025
Table 1: Examples of compositional task construction.
Sequential Composition
Initial Atomic Task:
Who is author of the book "The Great Gatsby"?
Subsequent Task:
When the author of this book born?
Composition Task:
When was the author of the book "The Great Gatsby" born?
Garallelthen--eqqeeee Composition
Initial Atomic Task:
Give me the flight schedule from London Edinburgh today.
Parallel Task:
Find the every hour weather forecast Edinburgh today.
Subsequent Task:
What is the weather condition when the first flight arrives?
Composition Task:
I am in London, and want to know the weather condition when the first flight arrives in Edinburgh today.
their actions or subjects. Details of seed data for scenario extraction, prompts for extracting and expanding scenarios can be found in Appendix A.I
Atomic Task Construction Based on the collected scenarios, we construct atomic tasks, each of which can be considered as straightforward problem, query instruction. We anticipate that these atomic tasks should be simple, clear, and don't need complex planning for human to solve. Such atomic tasks are able to serve as atomic units for constructing complex compositional tasks. In designing the prompts for transforming collected scenarios into atomic tasks, we focus on the following three aspects:
The atomic task should be realistic, reasonable, and representative of tasks fre- quently encountered in the real world.
Reasonable:
Self-contained: The atomic task should be solvable based on the information it provides. It must contain sufficient details and information necessary for calling functions.
Function -ggnosic:: During the articulation of the atomic task, we do not consider the specific functions that can be employed to solve this task. The task should not mention any specific function or solution.
Note that despite having numerous descriptions of our anticipated atomic tasks, do not pro- vide a strict definition of atomic tasks, nor of the following compositional tasks. We use specific prompts and the powerful instruction-fllloinng capabilities of cutting-edge LLMs ensure that the generated data aligns with our general expectations. The carefully crafted prompt can be found in Appendix A.2
Compositional Task Construction To enhance the capability of multi-turn function calling for LLMs. is essential to construct compositional tasks that require multiple interactions to be re- solved. Starting with the atomic tasks created in the previous step, we develop compositional tasks using two heuristic strategies: "Sequential Composition" and "Parallel-then--eqeeer tion". The principle behind sequential composition is start with atomic task and generate subsequent task that needs to be solved based on the result of the first, combining them into new compositional task. For parallel-then-see -te---euunnt-allll composition, can be solved in parallel with the atomic task. Then, a subsequent task is generated based on the results of the first two tasks, and they are all composed together. Examples can be found in Table I Prompts for implementing these two strategies can be found in Appendix A.3 Although these two heuristic strategies seem simple, they do not compromise the diversity of the dataset collected using our method. For instance, parallel function calls are not limited to the first turn; multiple functions may be generated for single atomic task subsequent function generations. Details on the data diversity can be found in Sec. 2.3 To ensure the quality of the generated compositional tasks, we filter out any that are inconsistent with their original atomic tasks. The filtering criterion is as follows: since the quality of atomic tasks is more controllable, by checking whether each one can be completed by its atomic sub-tasks. This allows us to filter out low-quality compositional tasks. The task filtering prompt be found in Appendix A.3
Composi-
begin by generating task that
filter compositional tasks
4

---

Published as a conference paper at ICLR 2025
Function Generation After constructing the compositional tasks, generate functions that are likely to be called in these tasks. This differs from most previous works. which first collect func- tions and then generate tasks based these collected functions (Patil et al. 2023 Liu et al. 2024b). Our fask-generation procedure is function-agnostie, which we believe allows for the construction of more realistic tasks, rather than those based solely on specific functions. Using the aforementioned methods, we have constructed series of compositional tasks for which we know the corresponding sub-tasks (i.e., the breakdown of the complex compositional task). These task breakdowns can be used as hints for function generation. This represents the advantage of our bottom-up instruction construction method, where we can examine the decomposition of the compositional task and gen- erate corresponding functions. During the generation of function definitions, we mainly focus on the following points:
The name and description of the function should be illustrative to aid in distinguish- ing different functions. The input arguments and output returns should also be clear since we not only use these definitions to but also use them to simulate corresponding functions.
Descriptive:
General:
The function should possess level of generality that enables its use for future tasks as well. In the real world, function is more likely to be constructed for a frequently encountered atomic task rather than a highly specific one. For instance, a function get weather (city) is more likely to be utilized than get. www.www www.iinnoooooo london ()
Consistency: As need to generate multi-turn function calling interaction trajectories in later steps, the input arguments and output results of these functions should maintain consistency. For example, if two functions will be called sequentially, the output of the first should either align with or constitute part of the input for the second function, regardless of the varying parameter terminologies.
The generated function definitions include five main fields: name, description, parameters, re- sponses, and required. The name indicates the function name, while description details its usage and capabilities. Parameters and responses cover the input and output, including the type and de- scription of each argument. The required field lists necessary input parameters. We allow flexibility in mapping sub-tasks to functions; sub-task may require zero, one, or multiple functions. If a sub- task involves logic, comparison, set operations, or calculations manageable by language models, no function is needed. For more details on the format and prompts, Appendix A.4
2.2 Top-Dow TRAJECTORY GENERATION
After obtaining the compositional tasks and their corresponding functions through the bottom-up instruction construction method, create multi-turn function calling interaction trajectories. These trajectories simulate how L.LMs use these functions and serve as supervision data, teaching L.MSs to perform multi-turn function calls.
Multi-agent Setup How to effectively simulate the multi-turn function calling interaction pro- cess is the key to collecting high-quality interaction trajectories. In our framework, we set up a multi-agent environment in which each agent simulates specific role during the multi-turn func- tion calling interactions, as shown in the right part of Figure [ We design three types of agents to simulate the interaction process: the user, the assistant, and the tool. The user agent initiates the interaction and provides the query to the assistant agent based on a specific compositional task. The assistant agent decomposes the task into sub-tasks and calls the corresponding functions to address these sub-tasks. The tool agent simulates the specific implementations of a function. It's important to note that tool agent simulates a specific function based on its definition. We do not implement the actual functionality of the function as we merely require reasonable feedback from the tool agents advance the interaction. Specifically, the assistant agent is aware of the available tools, compositional tasks, and the task breakdown. Since our compositional tasks are constructed in bottom-up manner, the breakdown of these tasks naturally comprises their atomic components. For each function, we establish tool agent based on its definitions. The behavior of the agents is steered by their system prompts, and the details of system prompts are listed in Appendix A.5
Interaction Trajectory Collection. After setting up the multi-agent environments, given spe- cific compositional task and corresponding functions, we tailor prompts for each agent. The inter- action trajectory begins with the user agent. The assistant agent reviews the task, decides which
5

---

Published as a conference paper at ICLR 2025
Frequency
Frequency
(c) Average per
3000
1.30
2500
2500
1.25
Prequency
2000
Requenccy
2000
Average 8FC
1.0
2500
1500
3.19
1000
1000
1.10
500
500
1.00
1.00
2
6
3
5
2
3
6
#Tum One Trajectory
#FC One Tajectory
Tum of Assistant
Figure 2: Statistic investigation on our collected data BUTTONInstruct.
functions call, and determines the function parameters. The assistant's response includes: first, observations and thoughts in free text, similar to the ReAct (Yao et al. 2023) format; second, a specified function call. This function call parsed as an action to invoke the corresponding tool agent and obtain simulated function call results. Finally, when the assistant has the final answer to the user's question, it invokes the function that provides the final response the user.
2.3 DATASET COLLECTION VIA BUTTON
BUTTONInstruct Based on the aforementioned pipeline, we leverage the cutting-edge LLM, GPT-46 to generate data at each step of bottom-up instruction construction, progressing from sim- ple scenarios compositional tasks and functions, and conducting trajectory generation as agents. We ultimately collect 8,000 multi-turn function calling data points, BUTTONInst each con- taining several entries, including content with roles such as 'system', 'user', 'assistant', and 'tool'. The available functions for each user question are listed in corresponding system prompt. More examples of finally generated data can be found in Appendix B.I
Parallel Function Calling Furthermore, in our collected data, we consider scenarios involving parallel function calling. If multiple functions can be called independently, they can be executed in Section[333 To control the behavior
parallel. The effectiveness of this parallel calling is discussed different system prompts to guide whether parallel calling
of conducting parallel calling, should be executed. Details on how to construct data for aligning [.LMs with the corresponding calling behaviors be found in Appendix [..2]
Data Diversity To demonstrate the diversity of the collected BUTTONInst we analyzed the distribution of the total number of assistant turns (#Turn), as well as the number of function calls (#FC) throughout the entire trajectory and each step. Figure presents key statisties of dataset. In Figure ((a), the frequency distribution of the total number of turns shows that most data points involve three or more turns of assistant responses. We plotted the distribution of the total number of function calls for each trajectory in Figure (b) and found that most data points contain more than two function calls. By plotting the average number of function calls per turn Figure Z(.,, we observed that every turn averages more than one function call. These findings indicate the diversity our collected data and demonstrate that, even with simple heuristic strategies to generate compositional tasks from atomic tasks, the final collected data are diverse across different aspects. Furthermore, more details about the distribution of the functions in the BUTTONInstruct are provided in Appendix B.3
EXPERIMENTS
To evaluate the effectiveness of our multi-turn function calling data BUTTONInstruct collected via proposed BUTTON pipeline, train two series of open-source LLMs of different sizes: Llama3-8B, Llama3-70B (Dubey et al. 2024), Qwen2-7B, and Qwen2-72B (Yang et al. 2024a). We denote models that have been instruction-tuned using BUTTONInstruet with the suffix BUTTON', In our experiments, we primarily focus on the following questions: 1) Ql: Can our
052-440-0024-0--1 from https::/platform..ggr.
a/ppt/ 8/

---

Published as a conference paper at ICLR 2025
proposed BUTTON approach enhance multi-turn function calling abilities compared existing instruction-tuned models designed for general purposes? 2) Q2: Are the bottom-up and top-down procedures effective and necessary? 3) Q3: What is the influence of data size and parallel calling on model performance?
3.1 EXPERIMENTS SETUP
Benchmarks. We evaluate performance using two benchmarks, GTA and Tool-Query.
GTA (Wang et al. 2024b). a benchmark for General Tool Agents, consists of 229 human-crafted queries designed to reflect real-world tasks. The queries span a total of 14 real-world deployed tools (i.e.. functions) across the perception, operation, logic, and creation categories. Each query is ac- companied by one or two authentic image files and the LLM is tasked with solving the queries based on the multimodal context and user queries. The tasks in this benchmark require multiple steps to solve and necessitate LLMs to reason about suitable tools and plan the solution steps. More details and examples about this benchmark can be found in Appendix B of Wang et al. (0024)) Tool-Query (Ma et al. (024) is a tool-using environment in the domains weather, movies, and academia. consists of 60 tasks requiring complex multi-round interactions with corresponding tools. There 18, 14, and tools (i.e., functions) in the weather, movie, and academia environments respectively, which are developed by corresponding real-world APIs and databases. This benchmark not only contains annotated final answers but also intermediate subgoals, which makes it easier to evaluate the process of function calling during multi-turn interactions. Tasks are also labeled hard easy based on the number of subgoals, with a count of 4 in this benchmark. More details and examples of Tool- hool-Query can be found Appendix F.4.1 Ma et al. (2024).
Evaluation Metries. For each benchmark, we follow their original evaluation strategies and met- rics. For GTA, there are two modes and nine metrics. In the Step-by-Step Mode, the model is provided with steps 1 to n from set of human-labeled function calling chains, and it is tasked with predicting the function call in the n + step. This demonstrates performance in a fine-grained way. In the End-to-End Mode, the model initiates its function calling process based solely on the user's question and proceeds until it arrives at the answer. This reflects the performance of the model in real-world applications. Four metries are used during the step-by-step mode evaluation: Instruction Accuracy (Inst.) is the accuracy of executing without errors, which indicates that the model knows how to follow the instruction to conduct a function call; Tool-selection Accuracy (Tool.) and Argu- ment Accuracy (Arg.) denote the accuracy of selecting tools and predicting arguments respectively; Summary Accuracy (Summ.) denotes the model's 's ability to summarize and derive the answer based on all previous steps. During end-to-end mode, we show the Fl scores of tool selection on percep- tion (P.), operation (O.), logic (L.), and creativity (C.) tasks, and the final answer accuracy (Ans.). The final answer accuracy is only calculated solely queries with pure text answers, excluding image generation queries. For Tool-Query, Grounding Accuracy (G.A.) is used to denote the accu- racy of generating valid, executable function calls. Process Rate is used evaluate the completion proportion of subgoals during the handling of complex user queries and the subgoals are labeled in this dataset. Furthermore, Success Rate is the accuracy of the final answer. More details of metric calculation can be found in Appendix C.I
Baselines. We showcase the performance on cutting-edge API-based LLMs, including GPT-4o, GPT-4-Turbo, and GPT-3.5-Turbo. Furthermore, since our collected data BUTTONInStruce is adapted for tuning the Llama3 and Qwen2 base models, we also use their original instruction-tun versions as baselines, including Llama3-8B ---------Innccc Llama3-70B Instructt
IB-structt and
Swen2-72BBInnsrruss (Dubey et al. 2024 Yang et al. 2024a).
Implementation. We conduct full hallmeareettrr supervised fine-tuning for all base models, with a maximum sequence length of 8.192. The models are trained on 4x8 NVIDIA H800 GPUs. In addi- tion, we randomly select 100,000 general instruction tuning data from OpenHerrmes-2.5 (Teknium) 2023) and mix them with BUTToNInstruet to align the models' basic cestruction-folloiing pabilities. For implementation details, please refer to Appendix C.2
3.2 MAIN RESULTS
As shown in Table and by our proposed BUTTON method perform better than their corresponding original instructed ver- sions. For smaller models, the improvements significant. For example, we improved the
and Table 3 LLMs tuned with compositional instruction tuning data constructed
7

---

Published as a conference paper at ICLR 2025
accuracy of Llama3-8B 8B from 1.4% 30.5%. The low performance of Llama3-8B 8--trrttuct this metric consistent with the findings the original benchmark paper. Notably, the results of comparable to GPT-4o, demonstrating the
-----------TTONNNN BUTTON and Owen2-72B-BUTTH effectiveness of the data collected through our proposed data collection pipeline.
Llama3-80B-B
Table 2: Main results(%) on GTA. Accuracy of Inst., Tool., Arg., and Summ.. Fl score of P., O., L, and C.. Ans. is the final answer accuracy. Bold numbers highlight better performance between the original instruction model and our tuned versions, while underlined numbers denote the best performance across all models.
Step-by-Step Mode
End-to-Eie tn--nd Mode
Model
Inst.
Tool.
Arg.
Summ.
0.
L
C
Ans.
GPT-4o
90.0
70.3
38.6
72.9
76.4
88.2
84.8
90.0
46.0
GPT-4-Turbo
84.6
60.6
34.3
73.5
51.8
87.7
61.8
86.1
30.6
GPT-3.5-Turbo
64.8
33.1
22.4
66.2
60.2
48.4
69.0
95.8
18.3
Llama3-8B-Inntruct
70.9
23.6
2.2
42.9
23.5
18.5
23.6
24.6
1.4
Llama3-BB-BUTTON
90.7
63.4
32.3
65.6
84.2
76.5
73.5
88.9
30.5
Qwen2-7B C
59.1
28.5
3.9
54.8
39.1
65.4
56.1
72.7
13.1
89.4
62.5
30.7
63.0
80.3
83.5
82.6
89.2
27.3
75.2
46.7
22.2
68.4
67.9
83.8
71.0
95.9
40.1
70B-BBTTON
96.4
73.6
38.1
70.5
84.9
96.2
89.5
96.1
43.5
Owen2-72B-Instreen Qwen2-72B-BUTTON
73.4
49.2
17.9
73.9
46.5
67.5
46.7
64.3
27.3
91.9
69.3
38.1
71.5
85.0
87.4
86.7
91.4
45.7
For Tool-Query. LLMs tuned with our data BUTTONInstruct demonstrate improved perfor- mance across various metrics. We observe that even though the grounding accuracy for both the original instruction models and those tuned with our dataset is quite high (near or above 95%), our models achieve better process and success rates. This suggests that the challenge lies not only in using functions correctly but also in effectively planning with them. Overall, from the main results from GTA and Tool-Query. we can conclude that BUT BUTTONInSt ruct can align LLMs with better ability on multi-turn function calling, and BUTTON is effective on such tasks.
Table 3: Main results(%) on Tool-Query. G.A. indicate grounding accuracy. Process Rate and Suc- cess Rate are presented for Easy, Hard and ALJ. test samples. Bold numbers highlight better per- formance between the original instruction model and tuned versions, while underlined numbers denote the best performance across all models.
Process Rate
Success Rate
Model
G.A.
Easy
Hard
All
Easy
Hard
All
GPT-4o
92.3
83.2
70.6
76.5
50.0
31.3
40.0
GPT-4-Turbo
95.4
80.7
78.9
79.8
50.0
34.4
41.7
GPT-3.5-Turbo
93.6
54.9
43.2
48.7
3.6
9.4
6.7
Llama3-8B-Instruce
96.7
55.1
42.0
48.1
10.7
0.0
5.0
Llama3-88 -8--BBBTOTOO
97.4
72.9
54.8
63.2
50.0
21.9
35.0
Qwen2-7B-Instee
97.0
66.7
46.8
56.0
32.1
15.6
23.3
Qwen2-7B-BUTTON
95.5
69.5
59.0
63.9
42.9
15.6
28.3
Jlama3-70B-Instrun
95.6
80.8
61.5
70.5
42,9
21.9
31.7
Mlama3-70B-UTTT0
94.0
85.2
77.2
80.9
71.4
46.9
58.3
Mroen-72B-Inntruun
95.8
83.7
72.2
77.6
50.0
34.4
41.7
Qwen2-72B-BUTTe
98.4
85.5
77.0
81.0
71.4
46.9
58.3
8

---

Published as a conference paper at ICLR 2025
3.3 FURTHER ANALYSIS
Ablation Study To evaluate the effectiveness of the "bottom-up instruction construction" and "top-down trajectory generation" procedures in BUTTON respectively, we conduct an ablation study by simplifying these two procedures into single direct generation steps using one prompt with generative LLMs, instead of our curated prompts and procedures. Specifically, to sim- plify the bottom-up procedure, we instruct gen- erative LLMs to directly generate a composi- tional complex task based on a given scenario. As a result, no sub-tasks are revealed for the subsequent function generation and trajectory generation steps. To reduce the top-down pro- cedure, we do not set up the multi-agent envi- ronment. Instead, we use a monolithic prompt that instructs the LLMs to act as the user, as- sistant, and tools, generating trajectories based on previously constructed tasks with functions. Refer to Appendix C.3 for details of prompts. The results of the ablation study on Tool-Query shown in Figure 3 We perform a compari- son using the aforementioned two ablation set- tings (i.e., w/o Buttom-Up and w/o Top-Down), and the default setting (i.e., w/ BUTTON) on Llama3-8B and Llama3-70B. By calculating the relative performance degradation, we found that the smaller model was influenced more, with average relative degradations on process rate and success rate of 16.1% 6611, 16, and that decline for Mamma3-0BB 6.4%. The average relative success rate decline across both model sizes and ablation settings is 15.0%. while the process rate decline is 7.5%. This suggests that models tuned without BUTTON can partially execute correct functions contributing to the process rate, but their limited planning ability prevents them from effectively achieving the final answer, leading greater decline in the success rate.
80
Ablation Settings
w/o Buttom-Up
wo Top-Down
70
BUTTON
Performance (%)
50
40
20
Success (Llame3 (Liama3-80)
Success Rate (178ma3-708)
Process Rabe [Liama (-tama3-7888)
Process
Metrics on Tool-Query
Figure 3:
Performance comparison in ablation set- tings for metrics Tool-Query.
Data Scaling In this section, investigate the influence of data size on compositional in- struction tuning data constructed by BUTTON. We vary the data size from 2,000 to 8,000 and tune Llama3-SB with the corresponding data. As shown in Table we present the results on Tool-Query with varying data sizes. By cal- culating the average growth rate (AGR), we found that as the data size increases, the process
Table 4: Results (%) on Tool-Query with Llama3- 8B tuned with varying data sizes.
Process Rate
Success Rate
#Data
Easy
Hard
All
Easy
Hard
All
2,000
67.5
46.9
56.5
35.7
9.4
21.7
4,000
73.1
45.8
58.5
50.0
12.5
30.0
6,000
74.6
62.3
46.4
15.6
30.0
rate and success rate increase accordingly, with AGRs of 13.5% and 2.7%. respectively. Fur- thermore, found that the improvement hard samples is greater than on easy samples, indicating that our constructed data can effectively enhance performance multi-turn function calling as the data size increases, particularly for tasks that require more turns of function calling.
8,000
72.9
54.8
63.2
50.0
21.9
35.0
AGR
13.5
32.8
18.4
2.7
5.5
3.8
Efficiency on Parallel Calling As described in Section 2.3 we have mentioned that our tuned model can conduct parallel function calling when multiple functions can be called independently within single turn. This capability can significantly enhance performance when operating un- der restricted step conditions. By turning off the parallel calling ability by changing the system prompt, we found that performance on different metrics affected. For example, the success rate of Llama3-8B. Beee23-BBBBTTE BU-BBUTOO creases from 63.2% to 58.7%. This indicates the effectiveness of parallel calling and our proposed BUTTON method. More cases about comparisons of parallel calling be found in Appendix C.4
Tool-Query decreases from 35.0% to 28.3%. and the process rate de-

---

Published as a conference paper at ICLR 2025
4 RELATED WORK
Synthetic Data Data has been a critical driving force behind the success of large language models (LLMs). Recent advancements in these models owe much the availability of large-scale, diverse, and high-quality datasets (Dubey et al. 2024). However, obtaining such datasets is both resource- intensive and expensive, posing significant challenges to researchers and practitioners (Xu et al.] 2023 [024a]] Luo et al.] 2023 Yang et al., 2224b)) To address these limitations, generating syn- thetic data has emerged as a promising alternative for creating scalable and high-quality datasets. Researchers have explored various techniques to improve the quality, diversity, and utility of syn- thetic data across different domains (Liu et et. 2024a| Wang et al.. 2024d] Wei et al. 2023 [ou et al.) 2023 Yang et al., 2023; 2024c). Recent studies have also focused on synthetic data for complex and compositional instructions. Chain-of-Instruct Instruct (Hayati et al. 2024) proposed a pipeline for building instructions composed of subtasks, allowing LLMs to solve compositional tasks diep-by-step in an explainable manner. Sequential Instruction Tuning (Hu et al. 2024) introduced a data construction pipeline for automatic instruction augmentation, where intermediate tasks are seeded from a single- task instruction. However, existing works do not focus on constructing compositional instructions for multi-turn function calling tasks, where identifying, invoking, and planning with functions all necessary.
Function Calling The ability to call functions or use tools effectively has become a significant aspect of instruction-tuning large language models (LLMs) (Wang et al. 2024a] Zeng et al. 2024
Shen et al. 2023]_ Xu et al. 2024b)). This functionalityy is crucial as it enhances the models' ability to
tackle complex tasks by enabling modular, structured, and interactive workflows (Qiao et al. 2024
Shen et al. 2024). Such capabilities expand the potential of LLMs beyond language generation, em- powering them to dynamically address practical challenges that require integrating computational or domain-specific tools. Recent efforts have focused on creating and curating datasets to enhance LLMs' function-calling competencies. Patil et al. (2023) collected APIs from repositories such as TorchHub, TensorHub, and Hugging Face, subsequently generating synthetic user prompts for each API using the Self-Instruct framework (Wang al. 2023). Similarly, Qin et al. (023) expanded upon this work by incorporating additional API data from RapidAPI. They generated diverse in- struction datasets involving a variety of cnnction-calling contexts, covering numerous interaction scenarios. However, these approaches face significant challenges: many API descriptions are un- clear, and considerable number of API calls fail due to availability issues or runtime errors (Guo) eet al., 2024], complicating the generation of accurate solution path annotations using LLMs. To address some of these challenges, Liu et al. (2024b) proposed APIGen, a multi-stage pipeline for generating diverse datasets tailored to function-calline tasks. This method employs rigorous verifi- cation process to improve data quality but is limited in scope, focusing predominantly on single-turn function-calling scenarios.
CONCLUSION
In this work, we address the importance of multi-turn function calling in the field of LLMs by fo- cusing on their ability to plan with functions, rather than merely use them. We introduce BUTTON, novel "bottom-up then top-down" pipeline for generating synthetic compositional instruction tun- ing data. This approach effectively tackles the challenges of ensuring compositionaliiy of tasks, generating compatible function, and high-quality multi-turn function calling trajectories without hu- man supervision via the curated prompts and procedures during the pipeline. Our methodology, which includes the generation of atomic tasks, compositional task construction, function generation, and a multi-agent simulation environment, has resulted in the creation of BUTTONI, nst ruct, dataset of 8,000 high-quality data points labeled with multi-turn function call trajectories. The ef- fectiveness of this approach is demonstrated by the improved performance of LLMs fine-tuned with BuTToNInst ruct on multi-turn function calling benchmarks. However, although our method has been empirically proven effective through experiments and analysis, the quality of the data cur- rently relies on our prompts and procedures, with no additional verification steps applied. In future work, will implement more curated data verification filtering stages to enhance the quality of the synthetic data. Furthermore, we may focus on extending the pipeline to align LLMs with more real-world applications, including embodied AI, where tool use, complex reasoning, and planning need to be integrated to complete more complex tasks.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Wei Chen and Zhiyuan Li. Octopus v4: Graph of language models. CoRR, 0tas/2404.192999 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178: 2024.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. CoRR, 12.19997.. 2023.
glaiveai. glaive-funetion--cllinn-2- https: httt::///nggioofffo.coo//tttaase g glaive-function-call1
2023.
Zhicheng Guo, Sijie Cheng. Hao Wang. Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking tool learning of large language models. In ACL (Findings), pp. 11143-11156. Association for Computational Linguistics, 2024.
Shirley Anugrah Hayati, Taehee Jung, Tristan Bodding-Long, Sudipta Kar, Abhinav Sethy, Joo- Kyung Kim, and Dongyeop Kang. large language models. CoRR, abs/2402. 11532, 2024.
Compositional instruction tuning on
Hanxu Hu, Pinzhen Chen, and Edoardo M. Ponti. Fine-tuning large language models with sequential instructions. CoRR, abs/2403.07794, 2024.
Mingan Lin, Fan Yang. Yanjun Shen. Haoze Sun, Tianpeng Li, Tao Zhang, Chenzheng Zhu, Tao Zhang, Miao Zheng, Xu Li, Yijie Zhou, Mingyang Chen, Yanzhao Qin, Youquan Li, Hao Liang, Fei Li, Yadong Li, Mang Wang, Guosheng Dong, Kun Fang, Jianhua Xu, Bin Cui, Wentao Zhang, Zenan Zhou, and Weipeng Chen. Baichuan alignment technical report. CoRR, abs/2410. 2024.
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang. Denny Zhou, and Andrew M. Dai. Best practices and lessons learned on syn- thetic data for language models. CoRR, abs/2404.07503, 2024a.
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406. 18518, 2024b.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 3688, 2023.
Renze Lou, Kai Zhang, Jian Xie, Yuxuan Sun, Janice Ahn, Hanzi Xu, Yu Su, and Wenpeng Yin. MUFFIN: curating multi-faceted instructions for improving cetruuctio-folloiinn CoRR, abs/2312.02436, 2023.
Haipeng Luo. Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. CoRR, abs/2308.0955 095883, 2023.
Chang Ma, Junlei Zhang. Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Ling- peng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn LLM agents. CoRR, abs/2401.13177. 2024.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang. Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36, 2024.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
Shishir Patil, Tianjun Zhang, Xin Wang, and Joseph Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305. 15334, 2023.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In EMNLP (Findings), pp. 5687- 5711. Association for Computational Linguistics, 2023.
Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Benchmarking agentic workflow generation. CoRR, abs/2410.07869, 2024.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toollim: Facilitating large language models master 16000+ real-world apis. arXiv preprint arXiv:2307. 2023.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools. In NeurlPS, 2023.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- gpt: Solving AI tasks with chatgpt and its friends in hugging face. In NeurIPS, 2023.
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. In NeurlPS, 2024.
Teknium.
Openhermees 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023. URL URLLittppss///ggiigsaa...//// http://aww...etaatatteee//eeee/a
2.5
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Trans. Mach. Learn. Res., 2024, 2024a.
Jize Wang, Zerun Ma, Yining Li. Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. GTA: A benchmark for general tool agents. CoRR, abs/2407.08711 2024b.
Junjie Wang, Mingyang Chen, Binbin Hu, Dan Yang. Ziqi Liu, Yue Shen, Peng Wei, Zhiqiang Zhang, Jinjie Gu, Jun Zhou, Jeff Z. Pan, Wen Zhang, and Huajun Chen. Learning to plan for aetrenval-uggmeneed large language models from knowledge graphs. In EMNLP (Findings), pp- 7813-7835. Association for Computational Linguistics, 2024c.
Yifei Wang, Jizhe Zhang, and Yisen Wang. Do generated data always help contrastive learning? CoRR, abs/2403. 1024/4403.1248,, 2024d.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In ACL (1), pp. 13484-13508. Association for Computational Linguistics, 2023.
Yuxiang Wei, Zhe Wang. Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. CoRR, abs/2312.02212) 2023.
Thomas Wolf, Lysandre Debut. Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's trans- formers: State-of-the-at natural language processing. CORR, abs/1910.03771. 2019.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
12

---

Published as a conference paper at ICLR 2025
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In ICLR. OpenReview.net, 2024a.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore. December 6-10, 2023, pp. 6268-6278, 2023.
Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang. Caiming Xiong, and Tao Yu. Lemur: Harmonizing natural language and code for language agents. In ICLR. OpenReview.net, 2024b.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024a.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024b.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical ex- pert model via self-improvement. CoRR, abs/2409. 2024c.
Dongjie Yang, Ruifeng Yuan, Yuantao Fan, Yifei Yang, Zili Wang, Shusen Wang, and Hai Zhao Refgpt: Dialogue generation of gpt, by gpt, and for GPT. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 2511-2535. Association for Computational Linguistics, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR. OpenReview.net, 2023.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttun- ing: Enabling generalized agent abilities for llms. In ACL (Findings), pp. 3053-3077.. Association for Computational Linguistics, 2024.
Penghao Zhao, Hailin Zhang. Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. aatrieval-augmente generation for ai-generated content: A survey. arXiv preprini 24Xi 19473, 2024.
13

---

Published as a conference paper at ICLR 2025
A PROMPT DETAILS
A.1 SCENARIO COLLECTION
In our work. the seed data for scenario extraction is derived from glaive-fumction-calling glaiveai 2023) and Toma datasets (Qin et al. 2023).
Here are prompts for extracting and expanding scenarios. The placeholders {conversation)
and (scenar i o are used to fill in conversation for extracting a scenario and a scenario for being modified, respectively.
Prompt for Extracting Scenarios
Please analyze the conversation below between a user and an assistant bot and identify the general life scenario it represents. Provide concise overview of the scenario type, such as 'booking flights' or 'ordering meals'. Avoid mentioning specific details like numbers or items. Your response should be a description of the scenario without additional commentary, and should not exceed 10 words.
Conversation: (conversation)
Concise Overview of the Scenario:
Prompt for Expanding Scenarios
Based on the provided daily scenario, creatively generate a new and entirely different scenario. The new scenario must meet the following requirements:
1. You may alter the action subject of the original scenario.
2. The new scenario should differ substantially from the original.
3. Ensure the new scenario is realistic and feasible within a daily life context.
4. Retain the same format as the original scenario.
5. Limit your response to 10 words and present the new scenario in single sentence.
Original Scenario:
(scen)
Modi fied Scenario:
A.2 ATOMIC TASK CONSTRUCTION
Here is the prompt for generating atomic task from scenario. The placeholder (scenario) will be substituted with a collected scenario when generating specific task.
Prompt for Atomie Task Construction
You are training model that can take user's task description or query, and available functions as input, and generate sequence of function calls to accomplish the task. Currently, you are generating basic atom tasks. Given general life
14

---

Published as a conference paper at ICLR 2025
scenario as the context, please generate a basic atom task that can be accomplished in one step.
Requirements of the task:
1. The task should be reasonable real life task based on the given scenario, and can be accomplished in one step.
2. If you mention some information, criteria or constraints in the task, please give the details of these information, criteria or constraints. Do not assume the model has access to your personal information or prior knowledge, and it does not have chance
ask you for clarification.
3. Please give enough details and make the task description as specific as possible, so the model can make deterministic function calls with deterministic arguments. Do not include any ambiguous or vague information.
4. Do not mention specific tools or functions in the task description, and not propose solutions, hints, or project outcomes.
5. Limit the task description to 30 words, and avoid using adjectives and ambiguous words.
Given Scenario: (scenaric)
Please give your response in one line directly, without any extra notation format:
A.3 COMPOSITIONAL TASK CONSTRUCTION
Following two prompts used for constructing compositional tasks from atomic tasks, including sequential composition and parallel-thenn-sqqeentil composition strategies.
Prompt for Sequential Composition
You are training a model that can take user's s task description or query, and available functions as input, and generate sequence of function calls to accomplish the task. Currently, you are generating complex tasks for model training. Given task, you need to add a subsequent task for this given task to make a more complex task.
The requirements for the subsequent task are as follows: 1. The subsequent task should use the output of the given task as input.
2. The subsequent can only be conducted after the given task has been completed.
3. The subsequent task and the given task can form new compositional task, and composing them can make more complex multi-step task.
## Examples:
### Given Task: Give list of all the pets.
### Subsequent Task: What is the most common kind of pet in the list?
### Composition Task: Check the most common kind of pet in the list of all the pets.
15

---

Published as a conference paper at ICLR 2025
### Given Task: Who is author of the book "The Great Gatssby??
###
Subsequent Task: When was the author of this book born?
### Composition Task: When was the author of the book "The Great Gatsby" born.
#*# Given Task: Give me the flight schedule from London to Edinburgh today.
Н#* Subsequent Task: Which fight has the shortest duration?
### Composition Task: Give me the flight from London to Edinburgh with the shortest duration according to the flight schedule today.
### Given Task: Retrieve the headlines of the news today from BBC.
#4# Subsequent Task: What is the sentiment of the news respectively?
**# Composition Task: What is the sentiment of each headline in today's 8 news from BBC?
### Given Task: Which team won the World Cup in 2018?
### Subsequent Task: What is the team's captain?
### Composition Task: Who is the captain of the team that won the World Cup in 2018.
## Here is the given task, please give your response following the above format:
### Given Task: [task]
Prompt for Parallel-then Sequennnna Composition
You are training a model that can take user's task description or query, and available functions as input, and generate sequence of function calls to accomplish the task. Currently, you are generating complex tasks for model training. Given task, you need add a paralle task and subsequent task for this given task to make more complex task.
The requirements for the parallel task are follows:
1. The parallel task should be related to the given task, and the input should independent of the output of the given task.
2. The parallel task can conduct at the same time as the given task, and they can be independent of each other.
3. The output of the given task and the parallel task can be used together conduct subsequent task.
The requierments for the subsequent task are as follows:
1.
The subsequent task should use the output of the given task and generate parallel task as input.
2. The subsequent can only be conducted after the given task and the parallel task have been completed.
3. The subsequent task, the given task and the parallel task can form a new compositional task, and composing them can make a more complex multi-step task.
## Examples:
### Given Task: Give me list of all the pets.
16

---

Published as a conference paper at ICLR 2025
### Parallel Task: Find available pet food currently in the store.
### Subsequent Task: Check if the pet food is suitable for the pets in the list.
### Composition Task: Check if the pet food is suitable for the pets in the list of all the pets.
*** Given Task: When was the author of the book "The Great Gatsby" born.
**# Parallel Task: Find the publication date of the book "The Great Gatsby".
**# Subsequent Task: When the book was published, how long had it been since the author was born?
### Composition Task: How old was the author of the book "The Great Gatsby" when the book was published?
*** Given Task: Give me the flight schedule from London Edinburgh today.
### Parallel Task: Find the every hour weather forecast Edinburgh today.
**# Subsequent Task: What is the weather condition when the first flight arrives?
### Composition Task: am in London, and I want to know the weather condition when the first flight arrives in Edinburgh today.
### Given Task: What is the sentiment of each headline in today' news from BBC?
### Parallel Task: Find the sentiment of each headline in today' s news from CNN.
#*# Subsequent Task: Which news scurce has more positive news today?
#** Composition Task: Compare the sentiment of each headline in today's s news from BBC and CNN, and check which news source has more positive news.
### Given Task: Who is the captain of the team that the World Cup in 2018?
**# Parallel Task: Who is the coach of the team that won the World Cup in 2018?
#44 Subsequent Task: Are the captain and the coach from the same country?
### Composition Task: Check if the captain and the coach of the team that won the World Cup in 2018 are from the same country
## Here is the given task, please give your response following the above format: ### Given Task: [task)
The following is the prompt for filtering compositional tasks, and the key idea is to verify the sistency between a compositional task and its atomic sub-tasks.
Prompt for Filtering Compositional Tasks
You expert in task decomposition. Currently, you are given a compositional task and its potential task breakdown.
17

---

Published as a conference paper at ICLR 2025
Please check if the sub-tasks can be used to complete the compositional task.
Compositional task: (task)
Potential task breakdown:
(sub_tasks)
Please check if the sub-tasks can be used to complete the compositional task. You should first give your analysis and thinking, and finally give your conclusion (yes or no) enclosed in <ans>, for example, <ans>yes</ans: or <ans>no</ ans?:
A.4 FUNCTION GENERATION
Prompt for Function Generation
You are training a model that can take user's s task description or query, and available functions as input, and generate sequence of function calls to accomplish the task. Currently, you are generating the training data for this model.
Given compositional task and its task breakdown, please generate corresponding aviliable functions that can be used to accomplist ish each sub-task, and finally the composition task can be accomplished by calling these functions sequentially.
## Requirements for the functions:
1. The functions must possess a succinct, comprehensible name and description.
2. The functions should not tailored for current task, are to be used for other future tasks as well, hence the design of APIs should sufficiently generalized.
3. Avoid the recurrence of the task or its components in the function description and name, offering a generic perspective that can be employed across different contexts.
4. Make every function sufficiently granular and independent, avoiding the conflation of multiple tasks within single function and avert creating monolithic APIs.
5. Consistency in terms of parameters and returns from each function on is critical. For instance, if two functions are called sequentially, the output of the first should either align with or constitute a part of the input for the second function, irrespective of varying parameter terminologies.
## Requirement for the number of functions:
1. One sub-task may need zero, one or multiple functions to complete it.
2. If sub-task is about logic, comparision, set operation or calculation, which can be solved by large language models, then no function is needed for this sub-task, just leave the func_list of this sub-task empty.
## Compositional task:
18

---

Published as a conference paper at ICLR 2025
(task)
## Task breakdown:
(sub_task)
## Response format: "jsoon
"sub_task"; "a sub task from the task breakdown", [func_list")
"name": "<function name>",
"description": "<function usage description>", "parameters":
"<paraml>":
"type": "<ean string, number, boolean, object, array, enum and anyOf>", "description": "<parami description>", <more keys if needed>
... <more parameters if needed>
"required"; "<array of required parameters, maybe not all parameters above are required>" "responses":
"<resl>"
"type": "<valuel type>", "description": "<valuel description>"
"<res>>";
" "type": "<value2 type>", "description": "<value2 description>"
<more functions if needed>
<more sub tasks and corresponding functions if needed>
## Please respond following the format above:
A.5 MULTI-AGENT
System Prompt for User Agent
Assume that you are human interacting with an AI assistant. You need to engage in a meaningful conversation while always remembering to demonstrate human-like behaviour. Avoid inquiring if the AI assistant requires assistance, as this contradicts your human role. Your main objective is to sustain a conversation as a typical user would.
19

---

Published as a conference paper at ICLR 2025
Currently, your goal is complete predefined task, and you are seeking the AI assistant for this purpose.
**Task** (task)
During this conversation, you should take on an active role and explore the AI assistant's capability to solve problems \ within the **Task** using a series of function (tool) calls. You should adhere to the following guidelines:
1. Your task involves complex task requiring multiple steps to complete. In your initial question to the AI assistant, you should provide detailed explanation of the task, including necessary information (such as potential data) that might be needed to solve the problem. However, you should withhold specific solution steps (e.g., avoid sequential terms like firstly," "secondly") and not dictate which functions (tools)
the AI should use that is for the AI to determine.
2. Remember, during this multi-ture dialogue, you are portraying the role of human user. Your questions and responses should reflect this human aspect. All your outputs should enclose within "<human>" tag, for example, "<human> ... </ human>".
System Prompt for Assistant Agent
You simulating the role of expert in using functions (1.e tools) to solve users' tasks. You already possess knowledge on how decompose the task into subtasks and understand which tools to use for their resolution.
*ubtasks** (sub_task,
**vailable Functions for Subtasks** (subtask_func)
Please the tools provided above to answer the question posed by "<human>", You must try as much as possible to use these tools, instead of directly answering the question using your prior knowledge.
Your response must obey the following format:
Observation: Carefully observe the user "<human>"' question as well as the output of the function call (often enclosed within the dxfunc_return>> tag). Be sure to check for any errors in previous outputs, as they may not always be accurate. Enclose your observation within the "<observation"" tag.
Thought: After observing and combining the previously listed
steps, give detailed and clear thoughts, reasonings, reflections, and according to the plan decide the next step. Note: When you believe the task to be complete, you may use final_answer' to provide a detailed sunnary of the results to
20

---

Published as a conference paper at ICLR 2025
give to the user. Enclose your thoughts within the "<thought tag.
Function Call: Name and arguments of the function call. The function name must be same as its name in above function list and the arguments must obey the format required by the function. Enclose the function call within the "<func_call>" tag. If possible, you can call multiple functions in parallel be sure the functions called paralleily are independent of each other.
Example (regular function call):
<observation> User has provided two numbers 15 and 25. </
observation>
<thought> Based on user's request, we need to find the greatest common divisor these two numbers. We can use the function geweeeeeeeeeaeeeeeeeee common_divisor"" divisor' to solve this problem. </ thought> <func_call>[
"name": "find_ ceanddgrretttt_toooe ceaattss__cmnon_coe "oomonn_iivsor"". "arguments": ""numl:: 15, "num2": 25
j</func_call>
Example 2 (parallel function call):
<observation> User wants know the weather in two cities New York and London. </observation> <thought> can use the function 'get_weather' to find the weather in New York and London. And the call to this function can be done in parallel. </thought> <func_call>[
"name": "get_weather "arguments": ("city": "New York")
"name": "get_weather", "arguments"; ("city": "London")
|</func_call>
Example (call final_ final_answer):
<observation> find_greatest__commn_ooo "5" </observation>
returns the result
<thought> The result returned by the function call, along with the information collected previously, is sufficient to answer the user's question, therefore we now use [Ginal_answer to provide the user with the answer. </thought> <funation_call>[
"name": ffinal_answer
"arguments"; ("final answer"; "5")
|</function_cal>>
Furthermore, when the user "<human>" raises a question, you need to provide structured plan to solve the question (' structured' means that the plan needs to include steps in
21

---

Published as a conference paper at ICLR 2025
sequential order, such as Step 1, 2, 3, etc., or logic processes that include loops and decision branches), The contents of the plan can be placed in the first round response's <thought>, and try as much as possible to follow this plan in every subsequent function call. However, as necessary, you may also modify the relevant plans according to the result of the function call.
System Prompt for Tool Agent
You are simulating computer system with powerful computational capabilities and a complete setup. You possess ample external prior knowledge, allowing you to run any arbitrary function and execute calls to produce results, and you never make errors. Give following function, you should simulate the operation of a computer system program as closely as possible.
**Funtion** [function]
Given function call, you should execute the function and provide the results JSON format. Your response should directly provide the results in JSON format, should not contain irrelevant information, and must enclose within func_return>" tag.
### Example of function return: <func_call>
"name": "get_weather", "arguments"; ("city": "New York")
<func_rette reunc_return>
"temperature": "25C",
</func_return>
B DATA COLLECTION
B. EXAMPLE OF COLLECTED DATA
Collected Data Example
System:
You are an expert in using functions (i.e., tools) solve users' tasks. The functions available for you to use are detailed below:
<tool>[
{
"name":
"description": "Fetches the current timestamp from the device.",
"parameters"; (),
22

---

Published as a conference paper at ICLR 2025
"requires red": []
"name": "get_ hwuuidity_readiitte
"description": "Fetches the current humidity reading from a device.",
"parameters":
"device_id" :
"type": "string", "description": "The ID of the device.
"required"; [ "device_id" id"
"name": _to___dat__t_ddatbbbeee
"description": "Logs data to server's s database.", "parameters";
"server_id";
"type": "string",
"description": "The ID of the server.
"data": {
"type": "object", "description": "The data to be logged. "properties";
"message":
"type": "string",
"description": "The status update message."
"
"timestamp": "zype": "string", "description"; "The current timestamp."
"temperature":
"type": "number",
"description"; "The current temperature reading."
" hhumi "humidity", {
"type": "number",
"description": "The current humidity reading."
required":
"message",
"t "timestamp"
" "temperature", "humidity"
"required"; [ "server_ice id",
23

---

Published as a conference paper at ICLR 2025
"data"
}
"name": tarree__tt_sreerree
"description"; "Sends a message from a device to a server.",
"parameters";
"device_id" _
"type": "string",
"description": "The ID of the device sending the message."
"server_id";
"type": "string", "description": "The ID of the server receiving the message."
"message": {
"type": "string", "description": "The message to be sent."
"rquuire""; [ idevice_ion "e"rver_id", "message"
,
"name": "yge__emmpraturreeedddd
"description". "Fetches the current temperature reading from device.",
"parameters"; "device_id". "type": "string", "description": "The ID of the device."
b, "required"; [ "device_io" id"
></tool>
In your response, you need first provide your observation and thought on the user's s task, the current situation, and what you plan to do next. After your thinking, you can do following two things:
**Funcioon Call*x: For fountion calling, you need provide the function name and its argument.s. The function name must be same as its name above function list, and the arguments must obey the format required by the function. Enclose the possible,
function call within the tag "<call></cal>>" you can call multiple functions in parallel, be sure the functions called parallelly are independent of each other. **Final Answer**: When you believe the task is complete, you may use
to provide detailed summary of the
24

---

Published as a conference paper at ICLR 2025
results to give to the user, enclose the final answer within the tag "<fimal></final>",
User:
Hi, I need to send a status update from DeviceA to ServerB. This update should include the current timestamp, Additionally, want to log this status update along with the current temperature and humidity readings in the database on ServerB. Could you help with this?
Assistant:
The user wants to send a status update from DeviceA to ServerB including the current timestamp, and then log this status along with the current temperature and humidity readings on ServerB's s database. To achieve this task, we will follow these steps:
1. Send status update message from DeviceA to ServerB including the device's a current timestamp.
-Fetch the current timestamp from DeviceA.
-end the status update message with the timestamp to ServerB.
2. Get the current temperature and humidity readings from DeviceA.
3. Log the status update message along with the temperature and humidity readings on Server5' s database.
We can perform steps and 2 in parallel since they are independent of each other. Once we have the required data, we will proceed to step 3.
<call>[
{
"name": "get_current__iimstaa"" "azguments": {}
"name": "get_temperee "gel_tempertuure_rraddnnn"" "arguments":
"device_id", id": "DeviceA"
}
{
"name": "arguments":
"device_id"" id"; "DeviceA"
<</cal>>
Tool: :
"name": gyet_cuurent_timsttaag "arguments": (),
"results"; {
"timestamp"; "2023-10-03314422:000
{
25

---

Published as a conference paper at ICLR 2025
"name": "get_temparature_rrrdinn""" "arguments", "device_id", id": "DeviceA"
"results": {
"temperature" 22.5
"name": "get_humiee" www.hhhmiiiii...............iiit geet__hmiditty_eddiiie "argumentss:
"device_id", "DeviceA"
} "results": { "humidity"; 45.3
Assistant:
We have successfully fetched the current timestamp, temperature, and humidity readings from DeviceA. The details are
follows:
Timestamp: 2023-10-03T14:22:00
Temperature: 22.5C
Humidity: 45.38 Now, need to send status update message from DeviceA to ServerB including the timestamp. After that, will log the status update along with the temperature and humidity readings in Server3's database.
<call>[
(
"name": "send_ "azguments":
"device_iae id": "DeviceA",
"server_id"" id"; "ServerB",
"message": "Status update at 2023-10-0311:22:0022
}
|</call>
Tool:
"name": mosssmssssggeee_ereeeeeee "arguments": {
"device_id", "DeviceA",
; "ServerB",
"message": "Szatus update at 2023-10-03114:22:00
"results";
"status": "Message del ivered successfully"
Assistant:
The status update message was successfully delivered from DeviceA to ServerB. The message content wast "Status update
26

---

Published as a conference paper at ICLR 2025
at 2033-10-03714:22:000 Now, we need to log this status update message along with the temperature and humidity readings in ServerB's database. We will use the following
data:
Message: "Status update at 2023-10-0311:22:0027
Timestamp: 2023-10-03T14:222
Temperature: 22.5C
Humidity: 45.3%
<call>[
"name": "io___att__t_ddttaaes"" "arguments": _eever__dd; "ServerB",
"data": {
"message": "Status update at 2033-10-0311::22:00 2",
"timestamp": 22033-00-03114:22000
"temperature": 22.5,
"humidity": 45.3
<</cal>>
Tool: :
"name": "Log__atta_to_dttbbsss "arguments": "server_id"; "ServerB", "data":
"message": "Status update at 2023-10 1023-10-03T14:220 03:14:220012 2"
"timestamp": "2023-10-0311::22:000
"temperature": 22.5, "humidity": 45.3
b.
"results": {
"log_status" "Data logged successfully on server ServerB"
Assistant:
The status update message along with the temperature and humidity readings was successfully logged in ServerB's database. The task has been completed successfully. A status update message was sent from DeviceA to ServerB including the current timestamp, and the data was logged on ServerR's database along with the current temperature and humidity readings.
<final>The status update message including the current timestamp was successfully sent from DeviceA to ServerB, and the data was logged in ServerB's s database along with the temperature (22.5C) and humidity (45.38) r </iina>>
27

---

Published as a conference paper at ICLR 2025
B.2 PARALLEL FUNCTION CALLING
We use different system prompts to guide the behavior of whether to conduct parallel calling or not. We demonstrate both the system prompt allowing parallel calling and the prohibiting it. The only difference is the instruction at the end of x FFFncti calling, we use: "If possible, you can call multiple functions parallel. Be sure the functions called in parallel are independent of each other". For disabling parallel calling, we use: "You should call one function a time and wait for the response before calling the next function". These instructions help distinguish between turning parallel calling or off. { {func list) } is the placeholder for the function definition list in ISON format.
Call**. For enabling parallel
To align LLMs with the correct behavior regarding parallel calling, we implement 50% probability of turning off parallel calling during trajectory generation for compositional tasks where sub-tasks can be executed in parallel. Since we generate compositional tasks from the bottom up, we in- herently know the sub-tasks involved. We these trajectories into prompts that prohibit parallel calling. For the other 50% of compositional tasks, where sub-tasks be executed in parallel, use system prompts that encourage parallel calling. We also configure trajectories conducted by compositional instructions, where sub-tasks cannot be done in parallel, use the system prompt that enables parallel calling with a probability signed prompts can effectively align LLMs either conduct parallel calling or not, according the corresponding system prompts. In our work, the accuracy of the parallel calling behavior is not the focus, and plan address it in future work.
50%. Finally, the constructed data with the de-
System Prompt Enabling Parallel Calling
You are expert in using functions (i.e., tools) to solve users' tasks. The functions available for you to use are detailed below: (tool>((unn_listt)<(s
In your response, you need first provide your observation and thought on the user's task, the current situation, and what you plan to do next After your thinking, you can do following three things:
**Function Call+*: For fountion calling, you need to provide the function name and its arguments. The function name must same as its name in above function list, and the arguments must obey the format required by the function. Enclose the function call within the tag "<call></call>" If possible, you can call multiple functions in parallel, be sure the functions called in parallel are independent of each other. **Final Answer<*: When you believe the task is complete, you may use 'final_answer' to provide detailed summary of the results to give to the user, enclose the final answer within the tag "<final><<Einal>>"
System Prompt Disabling Parallel Calling
You are an expert in using functions (i.e., tools) solve users' tasks. The functions available for you to use are detailed below:
In your response, you need first provide your observation and thought on the user's task, the current situation, and what you plan to do next. After your thinking, you can do following three things:
28

---

Published as a conference paper at ICLR 2025
**Function on Call**: For fountion calling, you need to provide the function name and its arguments. The function name must same as its name in above function list, and the arguments must obey the format required by the function. Enclose the function call within the tag "<call></call>", You should call one function at time, and wait for the response before calling the next function.
**Final Answer**: When you believe the task is complete, you may to provide detailed summary of the
use ' results to give to the user, enclose the final answer within the tag "<final></finall"
B.3 FUNCTION DISTRIBUTION
We show sunburst chart of the distribution of generated functions in proposed BUTTONInst in Figure where the inner circle and outer circle represent the first and sec- ond words in function name, respectively. It shows the diversity of our synthesized data, and the distribution of these functions is also consistent with our daily tasks.
event
latest
item
top
product
Freq.
vvailable
nutritional al
recipe
total
0.06
weather
average
get
calculate
current
0.04
send
set
email
create
search
measure
check
oven
notification
ppe
calandar
Find
event
recipe
reci scipess
Ingredient
Aam
0
neress
nearest
Len
soii
-0.02
Figure 4: Distribution of the generated functions in BUTTONInst ruct.
C EXPERIMENTS
C.I EVALUATION METRICS
For metric calculation for GTA and Tool-Query, we follow the methodologies outlined in their orig- inal papers with slight modifications. Specifically, during the end-to-end evaluation mode in GTA, the MathOCR functionality is implemented using the Mathpix API (https://mattpixx.com comm.. How- ever, since this API requires a subscription and we needed to conduct numerous pilot experiments and analyses, did not evaluate questions end-to-end mode that would invoke this API. The
29

---

Published as a conference paper at ICLR 2025
number of test samples excluding these questions is 209. For Tool-Query, refined the metric calculation by comparing the final answer with the ground truth. The original implementation was based on exact match; however, such strategies can overlook many successful answers. For example, consider the question "Which paper has received more citations: "Stability and Risk Bounds of Iter- ative Hard Thresholding" "Compressive Widehand Spectrum Sensing and Signal Recovery With Unknown Multipath Channels"?" models may finished with "The paper "Stability and Risk Bounds of Iterative Hard Thresholding' has received more citations (5) compared *Compressive Wide- band Spectrum Sensing and Signal Recovery With Unknown Multipath Channels' (2)". . However, the ground truth answer labeled as "Stability and Risk Bounds of Iterative Hard Thresholding", and exact match strategies may incorrectly label this answer as unsuccessful. Thus, we refined metric strategy to check if the final answer is present the model output. Note that all these modifications are applied to all models, including baselines, ensuring that the comparison between baselines and models fair.
C.2 IMPLEMENTATION
All instruction-tune tunnnng training is performed on 4 NVIDIA H800 GPUs, using the training frame- work based HuggingFace Transformers (Wolf et al. 2019) We use the corresponding instruction format for Llama and Qwen models. To enhance training efficiency, we pack short instances into longer ones and apply flash attention. During model training, we optimize the loss only on the sponse content from assistant roles. We use a learning rate of 2e-5 with cosine decay and a batch size of 64 for all models. For Llama3-8B and Qwen2-7B, we train for five epochs, and for Llama3-70B and Qwen2-72B, we train for two epochs.
C.3 PROMPTS FOR ABLATION STUDY
Here is the monolithic prompt for the 'w/o Bottom-Up Setting' in the ablation study. The place- holders {scen} represents the given scenario. This prompt is used to generate compositional tasks directly, without our proposed bottom-up process.
Monolithic Task Construction Prompt for the w/o Bottom- Up Setting
You are training a model that can take user's task description or query, and available functions as input, and generate sequence of function calls to accomplish the task. Currently, you are generating the training data for this model. Given general life scenario as the context, please first generate task.
## Requirements for each task: 1. The task should be accomplishable by calling multiple functions with multiple and no more than 7 steps (i.e., turns
).
2. If you mention some information, criteria
constraints in the task, please give the details of these information, constraints. Do not assume the model has access to your personal information or prior knowledge, and it does not have chance
criteria
ask you for clarification.
3, Please give enough details and make the task description as specific as possible, so the model can make deterministic function calls with deterministic arguments. Do not include any ambiguous or vague information.
4. Do not mention specific tools or functions in the task description, and do not propose solutions, hints, or project outcomes.
5. Limit the task description to 30 words, and avoid using adjectives and ambiguous words.
30

---

Published as a conference paper at ICLR 2025
## viven scenario: (scen)
## Please respond a task directly following the requirements above in one line:
Here are the monolithic prompts for the 'w/o Top-Down Setting' in the ablation study. The two prompts are used for tasks with without parallel function calling, respectively. The placeholders (task), }, (sub_task), and Srubtask_fund represent a specific generated task, its sub-tasks, and the corresponding generated functions for the sub-tasks.
Monolithic Trajectory Generation Prompt for the w/o Top-Down Setting (Non-parallel)
You are labeling data for training an AI assistant that can solve a complex compositional task by using tools in a multi- turn manner. Given complex compositional task, its potential subtasks and the available tools (i.e., functions) to solve these subtasks, you should generate synthetic data about the trajectory of solving the task by using tools in multi-turn manner.
**Task**
(task)
*ubbtasks**
(sub_task)
*xvailabbe Functions for Sub-tasks** foubtask_func]
During trajectory generation, you should simulate three roles: human: ask questions to the assistant
1.
2.
assistant: answer the questions of human by leveraging the tools
3. tool: execute the functions and return the results the assistant
The requiremente for the human are:
The human need give the initial question to the AI assistant based the given task, and should provide detailed explanation of the task, including necessary information ( such as potential data) that might be needed to solve the problem. However, you should withhold specific solution steps (e.g., avoid sequential terms like "firstly," "secondly") and not dictate which functions (tools) the AI should use that is for the AT to determine.
The requirements for the assistant are: 1. The assistant should use the tools provided above to answer the question posed by human. to use these tools, instead of directly answering the question using your prior knowledge.
must try as much as possible
2. The format must obey the following format:
a) Observation. Carefully observe the human's question as well as the output of the tool call. Be sure to check for any errors in previous outputs, as they may not always be accurate. Enclose your observation within the "< observation>" tag.
31

---

Published as a conference paper at ICLR 2025
b) Thought. Give detailed and clear thoughts, reasoning, or reflections, and according to the plan decide the next step. Enclose your thoughts within the "<thought>" tag. c) Function call. Give the name and arguments of the function call. The function name must be same as its name in above function list, and the arguments must obey the format required by the function. The function call should enclosed within the "<func "<func_call>" tag. Note: When you believe the task to be complete, you may use final_answer' to provide a detailed summary of the results to give to the user.
The requirements for the tool are:
1. You should simulate the operation of computer system program as closely as possible. Your response should directly provide the results in JSON format, should not contain irrelevant information.
2. You should also provide the response in the format of JSON including name, arguments and results.
### Example:
<role>Human</rol>>
Hi, I want to know when was the author of "The Great Gat was born.
<role>/assistant</rolee Lobservation>Tee
has provided book name "The Great Gatsby and wants to know the author's birthdate. birthdate.<oobbervtttte </observation>> <thought>We can use the function "get_book_authoxx author of the book and the sit_aathor_iifo to find the birthdate of the author. </thought> <func_call>[
find the
"name": "get_book book author" "arguments";
"The Great Gatsby"))
j</func_call> <role>Took<rrole> II
"name": "get_book_author",
"arguments": ("book_nad :bbook_name"; "The Great Gatsby"),
"results"; ("name": "F. Scott Fitzgerald")
}]
<role>eesssstatt</rlee
<observation>thee function bret_bock_authox returns the author of the book "The Great Gatsby" is F. Scott Fitzgerald.</ observation>
<thought>Now we can use the function 'get_author_ine info' to find the birthdate of the author. //thorr//thouhht>> <func_call>[
"name": "get_author_infa"
"arguments"; { ("name": "F. Scott Fitzgerald")
j</func_call>
<role>Tool</ooe>>
II
"name": "get_author_info",
"arguments": ("name": "F. Scott Fitzgerald")
32

---

Published as a conference paper at ICLR 2025
"results": ("birthaate" "September 24, 1896")
}]
<role>/ssisstant</zole>>>>>>
<observation> function "get_author_____ _nnfor returns the birthdate of the author "F. Scott Fitzgerald" is September 24, 1896. </observattionn
<thought>The result returned by the function call, along with the information collected previously, is sufficient to answer the user's question, therefore now use @aiallanssee. to provide the user with the answer. </thoughtt <func_call>[
"name": "final_answer "arguments": { ["final_answer" "F. Scott Fitzgerald was born on September 24, 1896.") ")
j</func_call call>
You need only generate the trajectory the above format, without any other explanation comments.
Monolithic Trajectory Generation Prompt for the w/o Top-Down Setting (Parallel)
You are labeling data for training an AI assistant that can solve a complex compositional task by using tools in a multi- turn manner. Given complex compositional task, its potential subtasks and the available tools (i.e., functions) to solve these subtasks, you should generate synthetic data about the trajectory of solving the task by using tools in multi-turn manner.
**Task** (task)
*ubbassks** (sub_task,
**vailable Functions for Subtasks** (subtask_func)
During trajectory generation, you should simulate three roles: human: ask questions to the assistant
1.
2. assistant: answer the questions of human by leveraging the tools
3. tool: execute the functions and return the results the assistant
The requirements for the human are:
The human need give the initial question to the AI assistant based on the given task, and should provide detailed explanation of the task, including necessary information ( such as potential data) that might be needed to solve the problem. However, you should withhold specific solution steps (e.g., avoid sequential terms like "firstly," "secondly") and not dictate which functions (tools) the AI should use that is for the AI to determine.
33

---

Published as a conference paper at ICLR 2025
The requiremente for the assistant are:
1. The assistant should use the tools provided above answer the question posed by human. It must try as much as possible use these tools, instead question using your prior knowledge.
directly answering the
2. The format must obey the following format:
a) Observation. Carefully observe the human's question as well as the output of the tool call. Be sure to check for any errors in previous outputs, as they may not always be accurate. Enclose your observation within the "< observation>" tag.
b) Thought. Give detailed and clear thoughts, reasonings, or reflections, and according the plan decide the next step. Enclose your thoughts within the "<thought>" tag. c) Function call. Give the name and arguments function call. The function name must be same as its name in above function list, and the arguments must obey the format required by the function. The function call should enclosed within the "<func_call>" tag. possible, you can call multiple functions in parallel, be sure the functions called parallelly are independent of each other Note: When you believe the task to be complete, you may use 'final_answer to provide a detailed summary of the results to give to the user.
the
The requiremente for the tool are:
1. You should simulate the operation of computer system program as closely as possible. Your response should directly provide the results in JSON format, should not contain irrelevant information.
2. You should also provide the response in the format of JSON including name, arguments and results.
### Example 1:
<tole>Human</rol>>
Hi, I want. to know when was the author of "The Great Gatsby" was born.
<role>issisttant</rolee <observation>The user has provided book name "The Great Gatsby and wants to know the author's birthdate </observation> <thought>We can use the function "get_book_autee author' find the author of the book and the ret__autho_infoo to find the birthdate of the author. author.</thoue </thought> <func_call>[
"name": "arguments": "book_name"; "The Great Gatsby")
|</func_call> <role>Took</role [[
"name": "get_book hiet_book_author".
"arguments"; ["book_name"" "The Great Gatsby"), , "results": ("name": "F. Scott Fitzgerald")
]]
<tole>Assi.santt<rolle
34

---

Published as a conference paper at ICLR 2025
<observatiion>the function gget_boskaauthop returns the author the book "The Great Gat is F. Scott Fitzgerald, observation>
<thought>Now we can use the function 'get_author_inne info' to find the birthdate of the author. <aobor.//thught>>> <func_call>[
"name": @get_author__ _ane_"nfo,,,, "arguments": ("name"; "F. Scott Fitzgerald")
|</func_call> <role>Tool</rol>>
II
"name": "get_author_info", "arquments": ("name": "F. Scott Fitzgerald") "results": ("birthdate"; "September 24, 1896")
}]
<tole>Assiistant//roll
<observationn>the function 'get_author_ing _inf' returns the birthdate of the author "F. Scott Fitzgerald" is September 24, 1896. </observation>>
<thought>The result returned by the function call, along with the information collected previously, \ is sufficient to answer the user's question, therefore now use "final_answel to provide the user with the answer.</ thought> <func_call>[
"name": "final_answe
"arguments": { "'final_answer" "F. Scott Fitzgerald was born on September 24, 1896.") "}}
ssffunc_calls
### Example 2:
<role>Human</role>
What is the age when the author of "The Great Gatsby" published
the book.
<roll>>lssittant<</rle>>> <observation> user mention the book "The Great Gatsby" and wants to know the how old the author was when he published the book. </observations <thought>We can use the function gget_book_autiie to find the author of the book and the the publish date of the book, such two function can be used in parallel.</thought. <fune_call>[
to find
"name": "gst_book_auther", "arguments"; ("book_name": "The Great Gatsby")
b,
"name": "arguments"; ("book_name:; "The Great Gatsby")
j</func_calla crole>Tool</role: II
35

---

Published as a conference paper at ICLR 2025
"name": "get_book_auther
"arguments": "pook_name":, "The Great Gat "results"; ("name": "F. Scott Fitzgerald")
}
"name": "get_book_publist.h..
"arguments": ("book_name", "The Great Gatsby"), "results": ("publish_dater "April 10, 1925")
H]
<role>Alssistaant<zzllo>>
<observation>The function "get_book jos__bock_authrrr author' returns the author of the book "The Great Gatsby" is F. Scott Fitzgerald, and the function pget_book_publish_aar _atte returns the publish date of the book "The Great Gatsby" is April 10, 1925.</ observation>
<thought>Io determine the age of the author when the book was published, we need caleulate the difference between the publish date and the author's birthdate, so we need to use the function "get_author_ime author, author,</thoie </thought>>>> <func_call>[
to get the birthdate of the
"name": "get_author_ina"
"arguments": ("name": "F. Scott Fitzgerald")
}</func_call>
<role>Tool</role> I
"name": wwt.aauthor_inof""" "arguments": ("name": "F. Scott Fitzgerald"), "results": ("birthdate"; "September 24, 1896")
H]
<role>>ssistant<<rzleee
<observatioon-The function 'get_ aet_author_innee info' returns the birthdate of the author "F. Scott Fitzgerald" is September 24, <
<thoughttPrevvoossiyy have got the publish date of the book The Great Gatsby" is April 10, 1925, so we can infer that the author was 28 years old when he published the book. </thought
<fune_call>[
"name": "final "iinal_answer'
"arguments"; ("final_answer"; "F. Scott Fitzgerald was 28 years old when he published "The Great Gatsby".""
f</func_car call>
You need only generate the trajectory in the above format, without any other explanation or comments.
C.4 CASES ON PARALLEL CALLING
We present two cases of conducting Llama3-708 DAL---7OB--BUTTT BUB-BNTTON on Tool-Query. In both cases, the func- tions authorEdgeCheck and paperNodeCheck parallel calling effectively reduces the number turns during the multi-turn function calling pro- cedure. For example, in case 2, with parallel calling, the language model only requires 6 steps
called in parallel where appropriate. This
36

---

Published as a conference paper at ICLR 2025
(including providing the final answer) to complete the task. Without parallel calling, 2 of these steps would expand steps each, resulting in a total of 12 steps.
Parallel Calling Case |
User:
Who has the most collaborations with Lazhar Labiod in the DBLP citation network?
Assistant: answer this question, need to follow these
steps:
1. Load the AuthorNet.
2. List the first-order neighbors (collaborators) of Lazhar Labiod.
3. Determine which collaborator has the most collaborations with Lazhar Labiod.
Let's s start by loading the AuthorNet. <call>[
"name": "loadAuthorNet", "arguments": ()
] <</cal>>
Tool: AuthorNet is loaded.
Assistant:
AuthorNet is loaded successfully, Next, need to list the first-order neighbors (collaborators) of Lazhar Labiod.
<call>[
"name": "neighbourCheck", "arguments": "graph": "AuthorNet", "node": "Lazhar Labiod"
}
]</call>
Tool: ['Mohamed Nadif', 'Chakib Fettal']
Assistant:
The collaborators of Lazhar Labiod are Mohamed Nadif and Chakib Fettal. Now, we need to determine which of these collaborators has the most collaborations with Lazhar Labiod. We can do this by checking the number of co-authored papers between Lazhar Labiod and each of his collaborators. <call>[
"name": "authorEdgeCheck", "arguments":
"nodel"; "Lazhar Labiod",
"node2": "Mohamed Nadif"
},
"name": "authorEdgeCheck", "arguments":
"nodel": "Lazhar Labiod",
37

---

Published as a conference paper at ICLR 2025
"node2": "Chakib Fettal"
}</call>
Tool: [o ('TensorClus: A python library for tensor (Co) (Co)-clustering' 'Bfficient Graph Convolution for Joint Node Representation Learning and clustering' ))" })", "( collaborative sllaborative_papers'' ('Efficient Graph Convolution for Joint Node Representation Learning and Clustering' [1]
Assistant: Lazhar Labiod has co-authored 2 papers with Mohamed Nadif and paper with Chakib Fettal. Based on the number of co-authored papers, Mohamed Nadif has the most collaborations with Lazhar Labiod. Now, will provide the final answer the user.
<final>Mchamed Nadif has the most collaborations with Lazhar Labiod, with total of 2 co-authored papers.</finall
Parallel Calling Case 2
User: How many citations de papers au- by Florian Kirchbuchrer and Fadi Boutros have in the DBLP citation network? Please give me a numerical value as an answer.
Assistant: To answer this question, need to follow these
steps:
1. Load the AuthorNet.
2. Check the collaboration relationship between Florian Kirchbuchner and Fadi Boutros.
3. Count the total number of citations of the papers they co- authored.
Let's s start by loading the AuthorNet <call>[
"name": "loadRuthorNet",
"arguments": ()
<</call>
Tool: AuthorNet is loaded.
Assistant:
AuthorNet is loaded successfully. Next, we need to check the collaboration relationship between Florian Kirchbuchrer and Fadi Boutros to get the list of papers they co-authored.
<call>[
"name": "authorEdgeCheck", "arguments":
"node1". "; "Florian Kirchbuchner",
"node2": "Fadi "outros"
J</call>
38

---

Published as a conference paper at ICLR 2025
Tool:
['PocketNet: Extreme Lightweight Face Recognition Network Using Neural Architecture Search and Multistep Knowledge Distillation', 'ElasticFace: Elastic Margin Loss for Deep Face Recognition.', Template-Drrivn Knowledge Distillation for Compact and Accurate Periocular Biometrics Learingg Models', "self-restraines triplet loss for accurate masked face recognition']
('col coollaborative_papere
Assistant:
The papers co-authored by Florian Kirchbuchner and Fadi Boutros are:
1. PocketNet: Extreme Lightweight Face Recognition Network Using Neural Architecture Search and Multistep Knowledge Distillation
2. ElasticFace: Elastic Margin Loss for Deep Face Recognition.
3. Template-Driven Knowledge Distillation for Compact and Accurate Periocular Biometrics Deep-Learning Models Self-restrained triplet loss for accurate masked face recognition
We need to find the total number of citations for these papers. To find the total number of citations for these papers, we will check the citation count for each paper individually and then sum them up.
<call>[
"name": "paperNodeCheck",
"arguments":
"node": *PocketNet: Extreme Lightweight Face Recognition Network Using Neural Architecture Search and Multistep Knowledge Distillation"
b
"name": "paperNodeCheck",
"azguments": "node": "ElasticFace: Elastic Margin Loss for Deep Face Recognition."
{
"name": "paperNodeCheck", "arguments":
"node": "Template-Drive Knowledge Distillation for Compact and Accurate Periocular Riometrics Deep- Learning Models"
),
"name": "paperNodeCheckk,
" "arguments":
"node": "Self-ree feelf-restrained triplet loss for accurate masked face recognition"
J</call>
39

---

Published as a conference paper at ICLR 2025
Tool: ['Please load the PaperNet first.', 'Please load the PaperNet first.', 'Please load the PaperNet first.', 'Please load the PaperNet first.']
Assistant:
The system requires PaperNet to be loaded before checking the details of individual papers. We need to load PaperNet first before proceeding to check the citation counts of the papers.
<call>[
{
"name": "loadFaperNet", "arguments": ()
] <</all>>
Tool: PaperNet is loaded.
Assistande PaperNet is loaded successfully. Now that PaperNet is loaded, we can proceed to check the citation counts for each of the papers co-auteered by Florian Kirchbuchner and Fadi Boutros.
<call>[
"name": "paperNodeCheck",
"a "arguments":
"node": "PocketNet: Extreme Tightweight Face Recognition Network Using Neural Architecture Search and Multistep Knowledge Distillation"
),
(
"name": "paperNodeCheck",
"azguments": "node": "ElasticFace: Elastic Margin Loss for Deep Face Recognition."
{
"name": "paperNodeCheck",
"arquments":
"node": "Template-Driven Knowledge Distillation for Compact and Accurate Periocular Biometrics Deep- Learning Models"
}
{
"name": "paperNodeCheck", "arquments":
"node": "Self-restrain triplet loss for accurate masked face recognition"
<</cal>>
Tool: ('authors'; ['Fadi Boutros', 'Naser Damer', 'Florian
Kirchbuchner', 'Arjan Kuijper'), 'year': 2022, 'venue' 'IEEE ACCESS', 'n_citation': 0, keywords'; [' ('Compater
architecture', MMicroprocessoss", 'Training', "Task analysis
40

---

Published as a conference paper at ICLR 2025
'Network architecture', 'Computational modeling', Architecture', "Cmputer vision', 'deep learning', 'face recognition'), ], 'doc_type': "Jouznal')", "f'authors': ['Fadi Boutros', 'Naser Damer', 'Florian Kirchbuchner', 'Arjan Kui Kuijper'), yyear' 2022, 'venue': 'IEEE Conference on Computer Vision and Pattern Recognition', 'n_citation's 43, keywords' [], 'doc_type' Conference' ,", "l'authors': [' Fadi Boutros', 'Naser Damer', 'Florian Kirchbuchner', 'Arjan Kui Kuijper'), 'year': 2022, 'venue' SENSORS', 'n_citaaion' 0, 'keywords' ['biometrics', 'knowledge distillation', periocular verification'). 'doc_type': 'Journal'"", "(' authors': ['radi Boutros', 'Naser Damer', 'Florian Kirchbuchrer', 'Arjan Kuijper'), 'year': 2022, 'venue': Pattern Recognition', 'n_citation's 8, 'keywords' COCDD -19', 'Biometric recognition', 'Identity verification', Masked face zecognition') ), 'doc_type' 'Journal')
Assistant:
The citation counts for the papers co-authored by Florian Kirchbuchner and Fadi Boutros are as follows: Extreme Lightweight Face Recognition Network Using Neural Architecture Search and Multistep Knowledge Distillation citations
1. PocketNet:
2. ElasticFace:
Elastic Margin Loss for Deep Face Recognition. 43 citations
3. Template-Drive Knowledge Distillation for Compact and Accurate Periocular Biometrics Deep-Tearning Models 0 citations
4. self-restrained triplet loss for accurate masked face recognition - 8 citations
The total number of citations is the of these counts. The total number of citations for the papers co-authored by Florian Kirchbuchner and Fadi Boutros is \O0 43 + 0 8 511). I will now provide the final answer.
<final>The total number of citations for the papers co-authored by Florian Kirchbuchner and Fadi Boutros is 51.</final>
41


Reviews:
[{"review_id": "PG0ei7oim6", "reviewer": "Reviewer_y93N", "summary": "This paper introduces a new approach, BUTTON, to generate synthetic compositional instruction tuning data, which could further serve as instruction tuning data for LLMs. Compared to existing methods, this approach can engage in multi-turn function calling, which enables planning with functions in handling real-world compositional tasks.", "questions": "1.\tDoes the designed heuristic strategy for compositional task construction only contain 2 or 3 sub-tasks? For example, can the sequential composition heuristic allow for an arbitrary length of sub-tasks? How is the performance according to different length of sub-tasks?\n2.\tIt is a confusing on how function generation is conducted in Bottom-Up. The authors state that they allow for the construction of more realistic tasks, and generate functions that are likely to be called in these tasks. Is there any limitation or alignment on the function call generation? Since the function tools are specified and given in GTA and Tool-Query for evaluation, it seems that the Bottom-Up should be limited in using these given functions, rather construct new ones. Is there a conflict?\n3.\tThe authors are encouraged to provide how prompts are designed for ablation study, i.e., how to call the single direct generation steps using one prompt. \n4.\tIn Introduction, the bullet-in points in the last paragraph of how to solve challenges do not tackle with the challenges very well. The authors may better summarize how “compatibility” in challenge 2 and “without human supervision” in challenge 3 are tackled.\n5.\tTypo and grammar check, such as ‘we first extract a series of real-world scenarios from existing datasets that using external tools.' in Scenario Collection in Section 2.1.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper first addresses the problem of planning with functions but not only use them when handling compositional tasks. The originality sounds good. \n2.\tThis paper is well structured with good clarity.\n3.\tThe experiment results are mostly convincing and significant compared to existing methods.", "weaknesses": "1.\tThe authors state that they differ from existing papers that ‘use functions’ by ‘plan with functions’. They are encouraged to compare with these methods. A simple way can be conducting chain-of-thought methods to decompose a compositional tasks, and then use these reference methods in each step. Can the performance still be significant enough?\n2.\tMore experiment designs could make it more convincing. How is the performance when the task composition length, i.e. the amount of sub-tasks in one task increases? See also Question 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pDpqhnmG5t", "reviewer": "Reviewer_UdcM", "summary": "The paper discussed an approach, BUTTON, which generates synthetic compositional instruction tuning data via bottom-up instruction construction and top-down trajectory generation.", "questions": "1.  It might be better to explain “function calling” a bit ahead in the abstract part to avoid confusion as the pipeline is not essentially invoking a function.\n2. I would recommend to move the “related work” after the introduction to give the readers more background before introducing the methodology.\n3. I would recommend to start section 3.1 experiments with a brief introduction and split them into two subsections under which you can explain the description and evaluation metrics with some bullet points.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research topic itself has a wide range of practical use cases. \n2. The model training approach is innovative and offers a new perspective on training large language models to make multi-turn function calls. \n3. The explanation of the methodology is clear and detailed. The authors trained multiple popular large language models to compare performance and validated accuracy on two benchmark datasets, providing a more comprehensive analysis.", "weaknesses": "In the paper, the author mentioned about “multi-turn function calling”,  “function calling” and “invoke functions” many times. Although the solution purposed could potentially solve the function calls with other additional infrastructure setups, it could be a bit misleading to the readers since the pipeline is still mocking the function call with simulated responses.\n\nIn the section 3.1 experiments setup, the author evaluated the pipeline with two different benchmark datasets. This part could be better structured.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JbIl5uzVaR", "reviewer": "Reviewer_piq6", "summary": "The authors found that when large language models are called external functions, there is another important but neglected problem: how should the model plan and use these functions when dealing with complex tasks that require multiple rounds of dialogue?\n\nTo solve this problem, they proposed a method called **BUTTON**. This method is divided into two steps:\n\n- The first step is to build training data from the bottom up. They first designed some simple basic tasks and then combined them into more complex ones. Then, they developed corresponding functional interfaces for these tasks.\n\n- The second step is to generate dialogues from the top down. They built a multi-agent environment to allow simulated users, assistants, and tools to interact and generate data for multiple rounds of dialogue.\n\nIn this way, they generated 8,000 high-quality training data. Experimental results show that models trained with this data perform better when dealing with complex tasks.\n\nThis research is efficient because tasks in real life often require multiple steps to complete, and models must learn to plan what functions to use and when to use them.", "questions": "How the 8,000 data points mentioned in the experiment were selected was not mentioned, which lacks the rigor of experimental design. It is recommended that the author describe the data mining process in detail. In addition, the specific distribution of these data for different roles is not explained. It is better to provide a chart to show it.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**Originality**\n\nCurrently, large prediction models are very popular, and most researchers are studying how to make the model give higher-quality human-like text in dialogue. The authors found two difficulties: 1. The instructions of multi-round dialogues are too complex to be recognized by the model, and 2. There are problems with the compatibility of instructions and functions.\n\n**Quality**\n\nThe author proposed a new method, the BUTTON method, which has a clear technical route and includes two stages: 1) bottom-up instruction construction and 2) top-down trajectory generation. Two professional benchmarks (GTA and Tool-Query) were used to evaluate performance during the experiment. The scale of 8,000 data points is medium-sized in current similar studies, but it includes multiple roles such as system, user, assistant, tool, etc., which enriches the interaction scenario. The experimental results compare several currently very popular large language models, which is convincing.\n\n**Clarity**\n\nThe paragraphs of the article are very clear and easy to understand. The technical terms are also explained.\n\n**Significance**\n\nThis study solves the data acquisition problem through synthetic data and function call mechanism, which not only reduces the development cost but also expands the practical scenarios of LLM, which is of great significance in promoting the implementation of AI technology in practical applications.", "weaknesses": "The sample size of the benchmark GTA is small, with only 229 queries, and the benchmark Tool-Query has only 60 tasks, which may not fully cover the actual application scenarios. At the same time, the benchmark may have a subjective bias because the manually written queries may carry the subjective judgment of the writer, and the difficulty classification criteria may not be objective enough. Suggestions can be combined with questionnaires for evaluation, such as collecting feedback from actual users and obtaining more real scenario requirements through questionnaires.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CSzZkdrbzr", "reviewer": "Reviewer_5MrE", "summary": "This paper introduces BUTTON, a generation pipeline to improve multi-turn function calling for LLMs through compositional instruction tuning, to handle complex real-world tasks requiring sequential or parallel function calls.  BUTTON uses a combination of bottom-up task creation and top-down trajectory generation in a multi-agent environment. This pipeline also produces BUTTONInstruct, a dataset with 8,000 multi-turn function call trajectories, and demonstrates improved model performance across various benchmarks.", "questions": "1. The pipeline effectively generates parallel-then-sequential compositions, but how does BUTTON determine which tasks are best suited for parallel execution? Are there specific criteria or heuristics that guide the system in identifying parallelizable tasks?\n\n2. What is the seed data to generate the atomic tasks?\n\n3. How many atomic tasks can be used to generate a composed task at most? Does it affect the data complexity and fine-tuned performance?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper offers a scalable solution for creating synthetic multi-turn datasets, enhancing LLM capabilities in function calling. This is an important feature for AI agents in many applications.\n\n2. The BUTTON pipeline is innovative in combining both bottom-up and top-down data generation processes to address the need for getting complex, multi-turn interactions data  from real-world.\n\n3. The performance seems good. After finetuned with the generated dataset, all models get improved in terms of accuracy.", "weaknesses": "1. While the BUTTON pipeline effectively generates compositional tasks and interaction trajectories, the paper lacks a rigorous quality control process. For example, it does not sufficiently address how to ensure that two atomic tasks are logically compatible for creating a meaningful complex task. Furthermore, the paper mentions filtering compositional tasks by “checking whether each one can be completed by its atomic sub-tasks.” However, it remains unclear whether this condition alone is adequate to ensure that the composed tasks are realistic and applicable to real-world scenarios.\n\n2. The generated function calls are conceptual, lacking actual implementations, which may limit their practical usability. Without concrete implementations, there is a risk that these generated function calls might not be feasible in real-world applications. Additionally, it’s unclear whether the pipeline checks for duplicated function calls or if previously generated functions can be reused in later processes, both of which could enhance efficiency and coherence in the generated dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)