ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction
Authors: Xingshan Zeng
Weiwen Liu
Lingzhi Wang
Liangyou Li
Fei Mi
Yasheng Wang
Lifeng Shang
Xin Jiang
Qun Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Agentic Data Synthesis
Non-Autoregressive Generation
Primary area: generative models

Abstract: Agentic task-solving with Large Language Models (LLMs) requires multi-turn, multi-step interactions, often involving complex function calls and dynamic user-agent exchanges. Existing simulation-based data generation methods for such scenarios rely heavily on costly autoregressive interactions between multiple LLM agents, thereby compromising the practical efficiency of agentic data generation. In this paper, we propose ToolACE-MT, a novel Non-Autoregressive Iterative Generation framework for constructing high-quality multi-turn agentic dialogues. ToolACE-MT generates full conversational trajectories through three stages: coarse-grained initialization, iterative refinement, and offline verification. The initialization phase builds a structurally complete yet semantically coarse dialogue skeleton; the iterative refinement phase introduces realistic complexities and continued refinement via mask-and-fill operations; and the offline verification phase ensures correctness and coherence via rule- and model-based checks. Experiments demonstrate that ToolACE-MT enables efficient, effective and generalizable agentic data generation, offering a new paradigm for high-quality data construction in tool-augmented LLM scenarios.

Published as a conference paper at ICLR 2026
TOOLACE-MT:
GENERA- TION FOR AGENTIC MULTI-TURE INTERACTION
Xingshan Zeng', Weiwen Liu Lingzhi Wang', Liangyou Li', Fei Mi', Yasheng Wang, Lifeng Shang', Xin Jiang', Qun Liu'
Huawei Technologies Co., Ltd 2Shanghai Jiao Tong University HHarbin Institute of Technology, Shenzhen zeng. Sanngshan@huwwer com, wwiiu@sjtu.edu en
ABSTRACT
Agentic task-solving with Large Language Models (LLMs) requires multi-turn. multi-step interactions, often involving complex function calls and dynamic user- agent exchanges. Existing iimulation-based data generation methods for such sce- narios rely heavily on costly autoregressive interactions between multiple LLM agents, thereby compromising the practical efficiency of agentic data genera- tion. In this paper, we propose ToolACE-MT, a novel Non-Autoregressive It- erative Generation framework for constructing high-quality multi-turn agentic di- alogues. ToolACE-MT generates full conversational trajectories through three stages: coarse coarse-grained initialization, iterative refinement, and offline verification. The initialization phase builds structurally complete yet semantically coarse di- alogue skeleton; the iterative refinement phase introduces realistic complexities and continued refinement via mask-and-fil operations; and the offline verifica- tion phase ensures correctness and coherence via rule- and model-based checks. Experiments demonstrate that ToolACE-MT enables efficient, effective and gen- eralizable agentic data generation, offering a new paradigm for high- data construction in augmeneddt LLM scenarios.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable abilities in open-ended generation, reasoning, and instruction following Guo et al. (2025); Wang et al. (2024); : Jiang et al. (2024). Beyond passive language understanding, growing frontier in LLM research involves agentic task- solving, where models take on the role of autonomous agents interacting with users and environ- ments multi-turn dialogues Wang et al. (2023): Luo et al. (2025). These settings often involve multiple function calling" and adaptive decision-making, significantly broadening the applicability of LLMs in real-world scenarios.
To enable such agentic capabilities, high-quality multi-turn multi-step interaction data is essential. Multi-turn refers to multiple exchanges between the user and the assistant, while multi-step denotes task completion that requires executing a sequence of dependent actions, often through function calls. Together, they reflect the complexity of real-world agentic scenarios where task states are par- tially observable. However, constructing such data is inherently challenging: it requires generating complex but solvable tasks, maintaining coherent user-agent exchanges, and accurately simulating tool behaviors. A promising direction lies in multi-agent simulations, where multiple LLMs are as- signed roles including user, assistant and tool to collaboratively generate full conversations through autoregressive interactions Liu et al. (2025); Prabhakar et al. (2025). While effective at generating natural conversations, these approaches have several drawbacks: 1) They are computationally costly due extended Pack-and-forth interactions, where each turn must be generated one-by-one based on all previous context; 2) Task complexity and dialogue length are implicitly determined by
"Corresponding Author. In this paper, function calling, tool calling and tool use used interchangeably.

---

Published as a conference paper at ICLR 2026
model interactions and are difficult to constrain explicitly, which poses challenges for fine-graine tne-graiddd data design: 3) Most critically, since assistant behaviors are generated autoregressively without ac- cess to global context, ie. the overall task and dependencies between steps, it is difficult for the assistant to optimize the overall output structure and ensure consistency at each step. This lack of holistic awareness hinders factual accuracy, tool-use consistency, and task solvability, especially in scenarios requiring long-term planning. As a result, the quality of generated data largely depends on the capability of the LLMs playing the assistant role, often resembling a form of knowledge distillation from larger (assistant) models.
In this paper, we propose Too,ACE-MT, a novel framework for constructing multi-tur turn dialogues involving agentic tool-use behav- iors, inspired by Non-Autoregreessiv Trans- lation (NAT) and masked diffusion language models Gu et al. (2018); [Sahoo et al. (2024), which have been shown to be more effi- cient in language generation. Unlike tra- ditional autoregressive multi-agent simula- tions (MAS), ACE-MT generates full conversational trajectories through a non- autoregressive pipeline consisting of three stages (see Figure [): 1) Coarse-Grained Ini- tialization: A coarse but structurally com- plete dialogue skeleton is generated, speci- fying overall tasks and action trajectory. 2)
selstant
Initialization
Query
More
Call
Madc & Exlennd
Oniput
ReCinement
TeuAleraaii Ope
More Cafls
Summary
Malk & Fill
Check Finalization (a) M Multi-Agent Simulation
Check and Finalization
(b) Non-Autoregresiie G Generation
Figure I:
Multi- Agent Simulation v.s. our proposed Non-Autoregressive Generation.
Iterative Refinement: Through carefully de- signed mask-and-fill procedures, the skeleton is progressively enriched with complexity injection and improved with reasonability refinement. 3) Offline Verification: Rule- and model- checks are applied, filtering out invalid samples.
TooIACE-MT offers notable improvements in generation efficiency and complexity control, while preserving functions to generate high-quality agentic data. Through the iterative refinement opera- tions, Too also provides flexible scaling, enabling budget-constrained data generation.
Exerimental results on several agentic multi-turn benchmarks, including BFCL-v3 (Berkeley Func- tion Calling Leaderboard) Yan et al. (2024), 7-Bench Yao al. (2025) and ACEBench Chen et al. 2025), show that models trained with ToolACE-MT generated data outperform those with autore- gressive MAS. Data analysis and ablation studies confirm the efficiency and effectiveness of our generation pipeline, and more experiments show the generalizability to different backbones.
In summary, our contributions are as follows:
We propose ToolACE-MT, a non-autoregressive iterative generation framework for agentic multi- turn interaction.
The iterative refinement strategy in ToolACE-MT enables flexible complexity enhancement and quality improvement, which can be further scaled based budgets.
We provide extensive empirical evidence showing that ToolACE-MT enables efficient, high- quality generation of agentic dialogue data suitable for tool-use LLM training.
2 RELATED WORK
Agentic Data Synthesis. LLM agents equipped with external tools have shown realistic problem- solving capabilities Qin et al. (2024): : Gou al, (2024): Lu et (025). As current LLMS still face challenges with complex tasks Mialon et al. (2024); Yao et al. (0255. learning from synthesized
agentic data offers promising direction. Early work focuses on single-turn synthesis, where the agent receives one-time query and responds accordingly Patil et al. (2023): Zeng et al. (2023); Qin et al. (2024): Liu et al. (2024). However, real-world tasks often involve multi-turn, multi- step interactions, prompting recent studies explore conversational data synthesis via multi-agent simulations Tang et al. (2023): Liu et al. (2025); WWang et al. (2025). Closest to our approach is Prabhakar et. 2025), which adopts two-stage synthesis process. While their first stage resembles
2

---

Published as a conference paper at ICLR 2026
Initialization
Iterative R Refinement
A
Initial Trajectory
T
A
Tool Pool
Inittal Tasks
Complecity Injection
Reasonability Refinement
Task Examples
Initial Trajectory
A
Offiine Verification
(Mask & Extend with Complex Interaction)
(Mask Fill with Improved Content)
Checker
Model C Checker
Refined Trajectory
User Turn
ssistant Turn
Tool Turn
Figure 2: Overall workflow for our ToolACE-MT framework.
ours which generates task configurations and ground-truth answers, they still rely on multi-agent simulations the second stage collect full interaction trajectories.
Agentic Model Training. Fine-tuning on synthesized data remains central to agentic model train- ing Qin al. (2024); ; [Liu al. (2025); Prabhakar al. (2025). With reinforcement learning (RL) proving effective in enhancing LLM reasoning Shao et al. (2024); Guo et al. (2025), agentic RL has emerged as promising alternative for developing agentic capabilities recent research Feng et al.
(205 Qian et al. (2025) Zhang et al. (2025) y Jin et al. (2025). While agentic RI. may reduce the complexity of data synthesis by enabling learning from sparse or indirect supervision, high-quality synthesized data remains essential to guide and stabilize training.
Non-Autoregressive Generation. To overcome the inefficiencies and quality limitations in cer- tain scenarios of traditional autoregressive generation, where tokens are produced one-by-one, non-autoregressivee approaches have been proposed Gu et al. (2018); Xiao et al. (2023). These methods include CTC-based objectives Libovicky & Helel (2018), iterative refinement with Mask- Predict Ghazvininejad et al. (2019), and insertion and deletion strategies Gu et al.) (2019). Inspired by this line of work, extend the non-autoreeeessive paradigm to the turn level, enabling more efficient and coherent agentic dialogue data synthesis.
3 METHOD
3.1 PROBLEM FORMULATION
Solving complex agentic tasks requires multi-turn interaction between the Al assistant and the user/environment. During this process, the assistant may ask clarification questions or interact with the environment to accomplish the user's tasks. The interaction with the environment can also be multi-step. involving multiple function calls either at a single turn (called parallel function calls) or one after another (dependent function calls). This task-solving process tially observable Markov decision process (POMDP), defined as (S.U, A, O, R), where S is the state space. 24 the task space, A the action space, O the observation space, T Sxx SxO is the transition function, and R is the reward function evaluating the overall process.
be formulated
par-
We define one interaction between the assistant and the user/environment as single turn. The assistant executes a sequence of actions (al .a. ........." ..., where each a' E .A, to accomplish the user's tasks (u', u2 u ... ) where each u' E U. A single conversation may involve multiple tasks issued incrementally. Each action a' can be either function call or a natural language response to the user. The corresponding observation of either the tool's output or the user's 's follow-up message. Importantly, the environment state s after executing a' remains latent both the assistant and the user. The interaction concludes when all user tasks u' are completed or the maximum number of turns is reached. The final reward R is computed based on the cumulative state changes and, optionally, the action sequence, depending on the level of granularity desired.
This interaction results in sequence of alternating observations and actions (o", a' ,o', ..... _n-1, .a), where o is the user's 's initial message and of is the tool output or user reply following at. C' constitutes the target for data generation, i.e., multi-turn conversational data.
trajectory, C'
3

---

Published as a conference paper at ICLR 2026
3.2 OVERVIEW
To construct high-quality conversational sequences C, prior work applies multi-agent simula- tions Liu al. ( Prabhakar et al. (2025), where LLMs simulate the (producing obser- vations after assistant responses), the tools (producing outputs after function calls. can be either ac- tual tools simulated ones), and the assistant (producing actions). While this approach is shown to work, it is costly, hard to verify and complexity control. We propose more efficient and controllable method: Mon-Autoregressin Iterative Generation (ToolACE-MT), inspired by non-autoregressive translation (NAT) and diffusion models Gu et al. (2018); Sahoo et al. (2024).
Our pipeline consists of three stages: Initialization 2) Iterative Refinement, and 3) Offline Verifi- cation. Figure illustrates the overall workflow. We describe each stage in detail below.
3.3
COARSE-GRAINED INITIALIZATION
The goal of the initialization stage is to generate a coarse but structurally complete skeleton of multi-turn conversation. Both the user tasks and the conversational trajectory are initialized loosely coupled fashion, enabling later stages to enhance coherence and inject complexity. This stage lays the groundwork for efficient and controllable non-autoregressive generation.
3.3.1 TASK INITIALIZATION.
We begin by sampling or specifying a candidate tool list from a predefined tool pool Liu et al. (025); ; Wang et al. (2025). ) The overall task is then generated with the following components: 1) set of subtasks (u ..... u²") (with m predefined per instance), 2) the required tools for each subtask, 3) and the number of steps required for tool usage for each subtask.
This step serves as high-level planning, outlining the overall trajectory without finalizing all the details. To ensure coverage across domains and promote task diversity, we curate both actual and simulated tools from prior work Qin al. (2024): Liu et al. (2025) and handeraft initial task exam- ples. The examples will be further enriched during the data generation process.
3.3.2 TRAIECTORY INITIALIZATION.
Given the tool list and the generated user tasks, we generate initial conversational trajectory skeleton (o", at,00
a, o², .'.,....................... ....") by composing subtask trajectories sequentially. For each sub-
task u', generate sub-trajectory C, (of: ...... .......) based on the generated subtask metadata (i.c., tool requirements and number of steps) and previously generated sub-trajectories (Co..... .._...,,_11). The final initial traeectory is obtained by concatenating all subtask trajectories: C C= C.U U.................... U....C C.-
Notably, each sub- nub-trajectory is generated with tool calls and outputs generating in parallel to ensure consistency. Also, to simplify downstream refinement, we enforce that the user query of contains all necessary information (e.g., parameter values for function calls), and all subsequent observations of (s * 0) are tool outputs.
This structure ensures proper alternation between action types (e.g., function calls follow by tool outputs, and natural language responses follow by user replies) and facilitates easier post-processing. Note that this stage prioritizes structural completeness over semantic correctness. The generated content may be shallow or partially inconsistent and need to be refined later.
3.4 ITERATIVE REFINEMENT
In this stage, we enhance the initial trajectory through multiple refinement passes, improving both complexity and semantic coherence. Inspired by Masked-Predict Ghazvinineeja et al.) (2019),
iteratively apply mask-and-extenddfill to progressively improve the trajectory C' (see Figure 3
3.4.1 COMPLEXITY INJECTION.
To better simulate roal-world dialogues, we inject complexity into the initialized trajectories. The injection types include:
4

---

Published as a conference paper at ICLR 2026
Clarttication Turne
Tool Awarnnss
Erroc Simulation
Noa Function-
Selex
Call
Manday
Igel currest dateO)
"2025 Friday
booo 201 1........ ...))
Mack
book hobeljdato eutzattteersssss Hiz5--n...
("ressle" "basked"
&
W Whrch Biee
booking done
Fill
booking is
Complecity Injection
Reasonability Retinement
Theration
"Aceggt or not!"
Figure 3: Illustration figure for Iterative Refinement process.
Clarification: user gives incomplete information following by assistant's clarification question.
Tool awareness: assistant recognizes unsupported tasks and user updates the tool list.
Error simulation: tool call fails, resulting assistant reflecting and adjusting actions.
Non-function-calling requirements: c.g., chitchat or open-ended user inputs to increase diversity.
Each kind of injection is implemented via specific eration refers to replacing the whole content of one turn with a placeholder, and "extend" means to fill with revised content and add additional turns. For instance, if masking turn o', we generate: (o", a² a', o', a', o", a++1 .a") ) = fLLM(O, (o", a., """...",''_ X, a ,a)")"))) where o is the selected injection type and X indicates the masked turn.
operation. The "mask" op-
Since the trajectory is clean by initialization, we can easily maintain an injection log to record which turns have been modified and avoid redundant modifications.
3.4.2 REASONABILITY REFINEMENT.
Apart from injecting complexity, perform another refinement pass to enhance logical consis- tency and coherence. This includes checking whether tool calls have appropriate parameters, en- suring natural language responses contextually relevant, verifying dialogue flow and resolving inconsistencies.
We adopt mask-and-ff fill strategy that randomly masks several non-adjacent turns and regenerates them using an LLM. Initially, all turns have equal selection probability, but each time a turn is chosen, its probability is reduced, encouraging diverse turns be refined across iterations. To prevent problematic refinement, an LLM-based judger is used to determine whether to adopt the newly generated content or retain the original ones.
For each trajectory, complexity injection and reasonability refinement are both applied alternatively in iterative manner, until all turns have been refined the predefine refinement count for each type is reached.
3.5 OFFLINE VERIFICATION
Given the extensive use LLMs in the aforementioned stages, hallucination remains a critical issue, particularly in long multi-turn dialogues involving large tool lists Liu et al. (2023). To address this, we conduct offline verification on the refined trajectories using a hybrid approach combines rule-based and model-based methods et al. 2024 2025).
For rule-based, we evaluate several aspects, including dialogue and tool-calling format compliance, exccutability (when real tools are available), repetition, and identifiable hallucinations that can be detected with rules, such as references to special IDs that do not appear in the history.
decompose the evaluation into multiple sub- questions. Each sub-question is handled independently by an LLM-based checking expert, ensuring modular and focused assessment. The final decision is made based the aggregation of the indi- vidual outputs. We focus on semantic coherence and the detection of complex hallucinations that rule-based methods may miss in this step.
For model-based, inspired by Liu et al. 2025
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset Construction. We construct in total 8000 training instances using ToolACE-MT for periments. For comparison, we also construct 8000 instances with multi-agent simulation (MAS) method introduced in Wang al. 2025). For fair comparison, we leverage the same candidate tool pool and LLM 20-24-4-2024-11 I- 20) for generation, the same offline verification is also applied.
For each instance, the number of subtasks sampled from [2.5], and each subtask contains [L. 6] steps. During iterative refinement, we randomly inject 1 to 3 different types of complexity to avoid redundant patterns (such as repeatedly asking clarification questions for the same subtask) which could harm dialogue naturalness. Each instance undergoes reasonability refinement up to 5 times (More refinement can be applied, while this empirically cost-effective balance).
Models. We use LLaMA3.1 1-BB--88B---st InSB-lntttuct A Meta
(2024) as the base model in our main experi- ments. Other models, including (0we2.55-nnsrruct--e
Yang er al. (2024) (0.5B, 1.5B, and 7B)
and Qwen3-8B Yang al. (2025), are also tested to validate the generalizability of our method. To verify the effectiveness our proposed three stages, we also train models with data without offline verification and iterative refinement for ablation study.
Benchmarks and Evaluation. We conduct experiments on several representative benchmarks tar- geting on the multi-turn capabilities of tool togl-augeented LLMs, including the Berkeley Function Call Leaderboard (BFCL-v3) Yan et al.] (2024). т r-Bench Yao et al.] (2025). and ACEBench Chen et al. (2025). As focus on the realistic multi-turn capabilities, we mainly present and analyze results on the categories related to Multi-Turn categories. Results in single turn are also listed (for BFCL, while those for ACEBench listed in Appendix Table 7 to show the robustness.
Training Details. Given resource constraints, we employ the parameter-efficient fine-tuning method LoRA Hu et al. (2022) for model training. All model modules are configured for LoRA fine-tuning, with rank of 16 and an alpha value of 32. Training is performed with global batch size of 64 and learning rate of 10 following a cosine learning rate schedule with a warmup ratio of 0.1. tuning details.
Please refer to Appendix Table 6 for
Table I: Data Statistics and corresponding quality assessment comparing MAS and ToolACE-MT. "D" is short for "Dialogue", "UT" is short for "User Turn", and "CT" is short for "Call Turn". "SS" (Semantic Similarity) and "Enk" (Entailment Rate) evaluate coherence, and "H" (Entropy) and "Dist-3" (Distinct-3) assess diversity.
Statistics
Quality
Method
Tool /D
Turn #/D
UT #/D
CT #/D
Call #/UT
SS
EnR
H
Dist-3
MAS
8.8
28.0
5.8
8.7
2.3
65.23
43.60
7.92
0.319
TooIACE-MT
8.4
23.7
3.4
8.5
3.7
68.34
50.71
9.28
0.357
4.2 DATA QUALITY ASSESSMENT
For clarity, Table reports the core data statistics and quality assessment results comparing MAS and ToolACE-MT In terms of data statistics, ToolACE-MT produces fewer user turns per dialogue but more tool calls per turn. This pattern indicates that we achieve tasks with fewer conversational steps while maintaining stronger focus on tool invocation, reflecting improved efficiency multi- step task completion. For quality assessment, following Wang et al. (2025), we evaluate coherence and diversity. Coherence measures the logical consistency and contextual alignment across turns, where we calculate the rate of entailment relation (EnR) and semantic similarity (SS) between two consecutive turns. Diversity captures the variety of linguistic expressions and interaction patterns, where Shannon entropy (H) based on the word frequency and Distinct-N Score with N 3 are reported (please refer to Wang et al.] (2025) for details). The results show that our method yields dialogues with higher coherence and greater diversity.

---

Published as a conference paper at ICLR 2026
4.3
MAIN RESULTS
4.3.1 RESULTS ON BFCL
Table Accuracy comparison (%) on BFCL-v3 (Last updated on 2025-08-26). The table is divided into three parts: Proprictary Models, Open-Source Models, and our experimental models trained based on 1-88 Inst. The best results for the last part in each category are marked in bold. The second best results are underlined.
Multi-Turn
Single-Turn
Hallucination Overall
Miss
Miss
Long Context
Models
Overall Base
Non-Live Live
Rel
Irrel
Overall
Func
Puram
GPT-44-2024-11-20
50.00
61.00
45,50 35,50
58.00
86.81
78,85
83.33
81.31
71.71
1rainii-.--Pro-PPevii-------
34.62
39.50 29.50 31.50
38.00
65.35
74.59
33.33
90.67
59.94
DeepSeek-V3-0333
29.87
41.00
21.00
23.00
34.50
88.54
77.34
83.33
76.49
64.71
Llama3.--0BB-nntt
12.50
17.00
13.00
10.50
9.50
89.98
62.24
100
54.78
53.57
LJama3.1-B-IIstt
9.25
12.00
10.00
7.00
8.00
84.21
61.08
77.78
48.82
49.57
Multi-Agent Simulation
31.38
46.50
19.00
31.00
29.00
80,29
78.05
72.22
90.11
64.17
ToolACE-MT
40.25
57.50
31.50
34.00
38.00
84.94
71.52 77.78
72.83
65.41
Offline Verification
32.50
48.00
25.50
25.50
31.00
79.71
75.52
83.33
80.65
63.01
Iterative Refinement
20.88
39.00
12.00
10.50
22.00
75.92
61.57
72.22
46.25
52.10
Table shows the experimental results on BFCL-v3. The results demonstrate that ToolACE-MT significantly improves multi-turn function calling accuracy, outperforming strong open-source and even some proprietary models (e.g. Gemini-2.5-Pro
Specifically, ToolACE-MT achieves a 40.25% multi-turn accuracy, a 31% absolute improvement over the base model Llama3. 1araa...-..0 1-8B-Inst (9.25%), and even higher than models with larger sizes like 1lama3.1-708 (12.50%) and DeepSeek- (29.87%). Compared to the Multi-Agent Simulation (MAS) (31 31.38)), ToolACE-MTT also achieves consistently better results across all multi-turn subcategories. These findings highlight the effee- tiveness of our ToolACE-MT framework in constructing coherent, contextually grounded dialogues with accurate tool usage.
Beyond multi-turn performance, ToolACE-MT also demonstrates strong generalization to single- turn and hallucination evaluation settings. achieves 84.94% accuracy in the non-live single-turn category, par with the base model Llama3. 1-8B-Insh) while MAS fails to preserve (80.29%). An interesting finding is that performance on the live single-turn category achieves less improvement compared MAS, which attribute to the nature of real queries in live category: they often ambiguous. Models trained with richer multi-turn supervision tend to favor asking clarifica- tion questions before executing tool calls for ambiguous queries. This behavior reflects trade-off' between cautious multi-tum planning and aggressive single-tury execution.
Ablation studies further validate the effectiveness of our proposed three-stage stage framework. Removing the Offline Verification stage results in 2.4% absolute drop in overall performance, underscoring its importance in filtering out problematic or inconsistent instances. Further removing the Iterative Refinement stage leads to substantial performance decline across all evaluation categories. Upon manual inspection of the generated initial dialogues, we observe that many are either overly simplis- tic or contain semantic flaws. This highlights the critical role of Iterative Refinement in improving dialogue coherence and increasing complexity.
Interestingly, the Reasonability Refinement part in Iterative Refinement also provides partial func- tionality similar to that of Offline Verification, such as identifying and correcting inconsistencies during generation. The complementary relationship between these two stages and their overlapping effects will be further discussed in later subsection.
4.3.2 RESULTS ON MORE BENCHMARKS
We further conduct experiments ACEBench and Benchh, which involve more realistic multi-turn interaction settings. In the Agent category ACEBench and both the Airline and Retail domains in BeBcch, LLM simulates the user interact with the assistant model. Unlike BFCL-v3, they do

---

Published as a conference paper at ICLR 2026
Table Accuracy (%) on Multi-turn (MT) and
Table 4: Pass @ (%) comparison on 7-Bench. Results marked with indicate anomalies due to known evaluation limitations r-Bench.
Agent ("EA": End-to-End Accuracy: "PA": Pro- cess Accuracy) categories of ACEBench (En).
Models
MT
EA
PA
Models
Retail
Airline
Avg.
GPT-44-2024-11129
68.0
56.0
77.8
GPT-44-2024-11-20
60.4
42.0
51.2
Alama3.1-70B-Im
61.0
41.0
62.5
Llama3.1-70B-Inst
50.4
26.0
38.2
L-ama3.--BBInnt
24.0
6.7
18.3
Llama3.1-8B-Inst
6.1
26.0*
16.1
Multi-Agent Simulation
48.0
6.7
15.0
Multi-Agent Simulation
21.7
10.0
15.9
ToolACE-MT
51.0
ToolACE-MT
25.2
16.0
20.6
OOffline Verification
44.0
1.7
28.5
-Offline Verification
22.6
6.0
14.3
-tterative Refinement
34.0
1.7
22.8
-Iterative Refinement
9.5
6.0
7.8
not provide fixed ground-truth trajectories. Instead, a dialogue is considered successful and rewards are assigned accordingly only if the assistant achieves correct states.
We present the results on ACEBench in Table including results on Multi-Tune and Agent cate- gories. For Agent category, we report both End-to-End Accuracy (EA) and Process Accuracy (PA), where PA assesses the consistency between predicted trajectories and ground-truths. As can be seen, TooIACE-MT outperforms MAS baseline across all three metrics, with particularly strong gains in Agent PA, indicating better planning and execution consistency. Ablation studies further confirm the contributions the Offine Verification and Iterative Refinement stages, each contributing to perfor- mance improvement. Notably, the Agent EA remains low for all 8B-scale models, highlighting the significant challenge this setting poses for smaller L.MMs.
The results r-Bench (shown in Table show consistent trend, with TooLACE-MT outperform- ing MAS baseline and the ablation models. Interestingly, the base model Llama3. 1-8B-Inst obtains higher score of 26% in the Airline domain, surpassing all trained models. This counterintuitive outcome can be attributed to a known evaluation limitation in r-Bench Zhu et al. (2025): several instances define empty actions as the correct responses, assessing the assistant's 's ability to recognize unsolvable user requirements. When a model lacks sufficient capability and consistently fails to produce valid function calls, it may coincidentally align with these empty actions and receive posi- tive rewards, despite not demonstrating actual understanding. However, this phenomenon does not persist after training, which ultimately leads to lower evaluation scores.
4.4 DATA EFFICIENCY
Table 5: Cost and quality comparisons for the two generation methods. "with GPT-40/GPT-40- mini" indicates the used LLMs. "Cost" refers the corresponding cost (total API call times, input and output tokens and final pricing) for generating 8000 samples, and "Quality" is the overall pass rate (%) when applying Offline Verification. "Performance" is the average accuracy in BFCL-v3.
Cost
Method
Quality Performance
API Call times Input (#) Output (#) Pricing (USD)
MAS with GPT-4o
275k
652,145k
10,712k
1,737
61.1
64.17
ToolACE-MT with GPT-4o
188k
478, 115k
18,480k 33,062k
1,380
72.3
65.41
ToolACE-MT with GPT-4o-mini
394k
855,391k
148
48.7
60.13
Data Generation Efficiency. We compare the cost and quality of generating agentic dialogue data using TooLACE-MT versus MAS. As shown in Table[5] MAS yields a lower Offline Verification pass rate (61.1% vs. 72.3%), therefore requiring a larger initial dataset and in total 275k API calls, to obtain 8 8,000 valid samples, significantly more than ToolACE-MT (188k). To clarify token usage, report total input/output tokens and the corresponding cost (GPT-4o pricing: $2.5/M input, $10/M output). The main efficiency gain comes from reduced input tokens, since fewer iterations needed to generate longer multi-tum dialogues. Models trained on ToolACE- MT data also perform better, demonstrating both higher efficiency and effectiveness.
8

---

Published as a conference paper at ICLR 2026
65
Accurncy
W/ Offinee Verification With Ofline Verifiiatiion
15
30
Trsh Data
Train Data (Our))
T-llanch
T-Blanch
Refinement Time
Figure 4: Statistics of assistant turn counts for MAS and our method, measured the training data and successful inference cases in r-Bench.
Figure 5: The accuracy results of our method BFCL-v3 when scaling Iterative Refinement times, with and without offline verification.
Model Choices. We further ToolACE-MT with GPT-4o-mini (results also shown in Table 5 which results in a much lower pass rate and increased API calls, due to more frequent formatting errors and hallucinations. This reinforces that generating long, tool-intensive dialogues demands strong long-con long-context handling, which smaller models like GPT-4o-mini and LLaMA3.1 1-8B- Inst (in our attempt, failed to produce valid instances in most of time thus cannot generate sufficient usable data for training) struggle with.
Finally, even after filtering, the model trained on GPT-4o-mini generated data still show a notable performance gap compared to that trained on GPT-4o generated data (60.13% vs. 65.41%), high- lighting that generation quality before verification remains crucial despite post-processing.
Task Completion Efficiency. In addition to generation efficiency. we further examine how our non-autoregressive pipeline influences task completion efficiency. Here we define task completion efficiency as how effectively the assistant completes user task, where turn numbers per task serve as a signal. We hypothesize that this generation paradigm supports more effective overall task plamning, therehy reducing the number of interaction turns required complete task. In contrast, MAS often involves trial-and-error behavior from the assistant model to identify correct actions. This hypothesis first supported by statistics from the training data: as shown in the left part of Figure 4 instances generated by our method ToolACE-MT have fewer assistant turns on average than those generated by MAS. Evaluation on 7-Bench (right part) further validates this advantage, where our model completes tasks successfully with an average of 13.7 assistant turns, compared to 15.4 turns for MAS. These findings suggest that ToolACE-MT leads better task structuring and more efficient interaction patterns.
4.5 DATA EFFECTIVENESS AND GENERALIZABILITY
Iterative Refinement Time Scaling, In the previous subsection, we mentioned the complementary roles of Iterative Refinement and Offline Verification, both of which contribute to enhancing final data quality. To further investigate their interaction, we conduct an experiment where we vary the number of Iterative Refinement steps, specifically by applying more Reasonability Refinement oper- ations (as Complexity Injection is not well-suited for repeated application within single dialogue). For each refinement level, we train two models: one using data that has passed Offline Verification and one without. The performance trends are illustrated in Figure 5
As shown, when the number of refinement steps is low, the performance gap between models trained with and without Offline Verification is large (around 5%), indicating that Offline Verification is cru- cial for filtering low-quality data in the pipeline. As refinement iterations increase, this gap narrows (dropping below 2% after 15 iterations), showing that additional refinement improves data quality and reduces the need for further filtering. However, the gap never fully disappears even after 30 re- finement steps, highlighting the distinct but complementary roles of the two stages. While Iterative Refinement primarily improves semantic coherence and function call accuracy, Offline Verification excels catching issues like long-range inconsistencies overall structural flaws that are harder to correct through refinement alone.
Different Backbones. To evaluate the generalizability of our generated data across different back- bone models, we conduct experiments using base models similar sizes, including Qwen2.5-7B- Inst and Qwen3-8B (no-thinking). The results, shown in Figure include comparisons between raw

---

Published as a conference paper at ICLR 2026
Fine duned with our data
50
65
Agsuracy
Accuracy
(Fine Fine-tured]
60
(Raw)
30
Muiti-Ton Turn (Fine-tuned)
55
Muitt: Multi-Turn (Raw)
50
45
40
0.5B
1.5B
78
Llama3.1-88
Qwen2.5-78
Qwen3-88
Modol Size
Figure The accuracy results on BFCL-v3
Figure 7: The accuracy results on BFCL-v3 for Qwen2.5 series models.
when training based on different backbones.
models (without training), models trained with MAS data, and models trained with TooIACE-MT. As observed, both backbones benefit from training with MAS data, and training with ToolACE-MT leads to further consistent gains.
Interestingly, although the initial (raw) performance of Qwen2.5-7B-Ins' and Qwen3-8B is higher than that of Llama3.1-8PB I-st-Inst, the performance gain after fine-tuning is smaller. We attribute this to the training strategies of more recent models. Both Qwen2.5 and Qwen3 were released after Llama3.1 1 and likely to have incorporated improved agentic capabilities during their training. As a result, further fine-tuning on similar task formats may yield diminishing returns, reflecting a saturation effect from repeated exposure related domains.
Model Size Scaling. Scaling laws suggest a strong correlation between model size and performance. generated data, series across a range of model sizes (0.5B, 1.5B, 3B, and 7B). Both the raw and fine-tuned versions (trained on our generated 8000 instances) are assessed on BFCL-v3, with results (Multi Turn and overall) shown in Figure 7 As expected, larger models consistently outperform smaller ones. The smaller raw models (0.5B and 1.5B) exhibit little no multi-turn capabilities, but fine-tuning with our dataset can enhance the corresponding performance. Notably, the improvements are more pronounced in the 3B and 7B models, suggesting that multi-turn function calling remains relatively advanced ability that small models struggle acquire. Overall, the fine- tuned models demonstrate a clear scaling trend, reinforcing the effectiveness our data in equipping larger LLMs with complex function calling skills.
To explore the scalability of function calling capabilities after training evaluate the Qwen-2.5-xB-Bn
CONCLUSION
This paper introduces ToolACE-MT, a non-autoregressive framework for generating multi-turn function-calling dialogues. Inspired by non-autoregressive generation, ToolACE-MT combines it- erative refinement and offline verification to ensure semantic coherence, contextual consistency, and tool executability. It achieves substantial improvements in multi-turn function-calling accuracy, outperforming strong baselines while being efficient in both data generation and task completion. Further analysis demonstrates the complementary effects of refinement and verification, as well as the generalizability of ToolACE-MT across various model sizes and backbones.
REPRODUCIBILITY STATEMENT
We have taken several measures ensure the reproducibility of our work. The proposed methods and experimental settings (including data amount, models used for data generation, configuration during data generation, trained models, evaluation details, and hyper-parameters for training) are described in detail in the main text (Sections and 4.1), with additional implementation details provided in the appendix, including parammta tuning strategies, prompts used for data gen- eration, and training protocols. We also provide representative data example in the appendix to illustrate the structure and format of the data. Together, these efforts are intended to allow other researchers to replicate experimental setup and verify findings with minimal ambiguity.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The research described in this paper is partially supported by The National Natural Science Foun- dation of China (Grant No. 62502118, 62502310).
REFERENCES
AetMe
Llama model card. 2024. URL https://github.com htt.:/hhuu.c.mmmmmtmte.et ammaaa
Sima3/blobmmain/Me
CARD md
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, et al. Acebench: Who wins the match point in tool learning?
arXiv preprint arXiv:2501.1285. 2025.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms. arXiv preprint arXiv:2504.1 11536, 2025.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel de- coding of conditional masked language models. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 6112-6121, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. URL https::/acclaatchoo /aaaacclannoolooooooooooooooo hoy.o.y....ooooooooooooooooooo.. org/ D19 6337
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
ew net. forum? ? dd=
EpCTt-Joag
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non- autoregressive neural machine translation. In 6th International Conference on Learning Represen- tations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceed- ings. OpenReview.net, 2018. URLhttps:: http:///////preeeeeeeeeeee.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer.
Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11179-11189, 2019. URL http::/proccedingg www.ggsneeriss..m Abstraaattttttbbooboooeaaatttttt
Abstract. cc/paper/2019/ oetps/so...rrrrppo//////////0011
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501.12941 2025.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https: http::::///peeeevieee net / forum?
id=nZev/eeFFyfg
Yuxin Jiang, Yufei Wang, Xingshan Zeng. Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. FollowBench: A multi-level fine -nn-gnai constraints fol- lowing benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4667-4688, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: //aclanthoooge org 2024. acl - long.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXivv2503.095 2025.

---

Published as a conference paper at ICLR 2026
Jindrich Libovický and Jindřich Helel. End-to-end non-autoregressive neural machine trans- lation with connectionist temporal classification. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. pp. 3016-3021, Brussels, Bel- gium, Octoher-November 2018. Association for Computational Linguistics. URL https:
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilaequa, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts, July 2023. URI.
httt:://axiie orgo abs 2307 .03122 UtXiv:2307.03172 [es].
Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang. Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong Wang. Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang. Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang, Ruiming Tang. Defu Lian, Qun Liu, and Enhong Chen. Toolace: Winning the points of LLM function calling. In The Thirteenth International Conference on Learning Repre- sentations, ICLR 2025, Singapore, April 24-28, 2025, 2025. URL http:::tt////tt //ppnrevie/p 2tt.:runnni--BP8E5555500
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh R. N., Liangwei Yang. Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. Apigen: Auto- mated pipeline for generating verifiable and diverse function-calling datasets. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurlPS 2024. Vancouver, BC, Canada. December 10 nips wwwips.cc/paee cc/paper files/papez/2022 15, 2024, 2024. URL F0 Datasssss,
hash/6 www.ccc66600 (01.ce8861180111455550 Benchmarks, Track. html
Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. Octotools: An agentic framework with extensible tools for complex reasoning. In ICLR 2025 Workshop on Foundation Models in the Wild, 2025.
Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, al. Large language model agent: survey on methodology, applications and challenges. arXiv preprint arXiv:2.533.21400 2460, 2025.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general Al assistants. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httts::/oppenrevie
net / forum? -im.didddfiiaassss.smmmm
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305. 15334, 2023.
Akshara Prabhakar, Zuxin Liu, Weiran Yao. Jianguo Zhang, Ming Zhu, Shiyu Wang, Zhiwei Liu, Tulika Awalgaonkar, Haolin Chen, Thai Hoang, et al. Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay. arXiv preprint arXiv:2504.03600 2025.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- Gokhan Tur, and Heng Ji. Toolrl: Reward all tool learning needs. arXiv preprint urXiv:2504.1390 2025.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xi- angru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-worle apis. 2024. URL htttp:://opereeeeeee
net forum??mm did=
dHH:g000Jir
Subham S. Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrIPS 2024, Vancouver, BC, Canada, December 10-
12

---

Published as a conference paper at ICLR 2026
15, 2024. http://///ppppess.......ccc/aaa/
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, April 2024. URL hRtttttp//aaee arXiv:2402.03300 [es].
orq/
abs/ 2402. 03300
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.053001 2023.
Lei Wang, Chen Ma. Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang. Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A Survey on Large Language Model based Autonomous Agents, September 2023. URL http://arxiv.orgg arXiv:2308. 1432 [es].
abs/ /20//230.... 11432
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 09955266-95290, 2024.
Zezhong Wang, Xingshan Zeng, Weiwen Liu, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang. Qun Liu, and Kam-Fai Wong. ToolFlow: Boosting LLM tool-calling through natural and coherent dialogue synthesis. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pp. 2246-42635 Albuquerque, New Mexico, April 2025. Association for Computational Linguisties. URL https: http:///aaaattthaa http:///aaaaathoooy..ooooo org.loog...g//22...... Tsacl--long211
Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie-yan Liu. A survey on non-autoregressive generation for neural machine translation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10): 1407-111411 2023.
Fanjia Yan, Huanzhi Mao, Charlic Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https: berkeley. edw edu/blogs/8 2024.
Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint afX1w:241..15115, 2024.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R. Narasimhan. 7-bench: benchmark for tool-agent-user interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, 2025. URL https:
//openzeview.netee nttp: roNSSXpUUU
Aohan Zeng. Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. AgentTun- ing: Enabling Generalized Agent Abilities for LLMs, October 2023. URL ptetp://arxx.
org/ abs//bb//// /abs/2310. 12823 arXiv:2310.12823 [cs].
Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. cemootoonressearc--coooom Tool-using language models with rein- forced reasoning. arXiv preprint 1.XXi::200500024 2025.
Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng. Rebecca Weiss, et al. Establishing best practices for building rigorous agentic benchmarks. arXiv preprint arXiv:2507.02825, 2025.
13

---

Published as a conference paper at ICLR 2026
A
MORE TRAINING DETAILS
Table 6 displays the selections of nyper--aramette, where all are selected based on performance of development set. All the models are trained based on LLaMA-Factory"
Table 6: Hyper-paraneen
experiments.
Hlype-paaaamete
Selection Ranges
Final Seletion
Learning rate
[5e-5, le-4, 2e-4]
le-4
Batch size
[32, 64]
64
Warmup ratio
[0.1]
Lora rank
[8, 16]
Lora alpha
116. 32]
32
B USE OF LLMS
We used large language models (LLMs) solely for writing assistance, specifically polish the lan- guage of the manuscript. The models were employed to improve readability, grammar, and style, without altering the technical content, analysis, or results. All ideas, experimental designs, and conclusions were conceived and developed by the authors.
C FULL RESULTS ON ACEBENCH
Full results on ACEBench are shown in Table where additional results for single-turn categories are added ("S-Turn" is short for Single-Turn, "M-Turn" is short for Multi-Turn). The results reflect that TooLACE-MT still outperforms baselines those single-turn categories, with a higer overall result.
Table 7: Accuracy (%) comparison on ACEBench (En) full set.
Models
Atom S- S-Turn M-Turn Similar Preference Special Agent (EA) Agent (PA) Overall
GPT-44-2024-11-20
90.0
78.0
68.0
80.0
78.0
92.7
56.0
77.8
81.1
Llama3.1-7BB-Int
83.7
71.5
61.0
74.0
66.0
29.3
41.0
62.5
57.9
Llama3.-SB-IIst
52.7
45.0
24.0
54.0
50.0
9.3
6.7
18.3
30.4
Multi-Agent Simulation
81.3
63.5
48.0
70.0
64.0
S.3
6.7
15.0
43.8
ToolACE-MT
83.0
64.0
51.0
68.0
68.0
8.7
8.4
34.0
45.2
Offline Verification
77.1
61.0
44.0
60.0
64.0
8.7
1.7
28.5
41.8
Iterative Refinement
61.7
56.0
34,0
56.0
50.0
5.3
1.7
22.8
34.5
D ADDITIONAL EXPERIMENTS
To further highlight the value of our generated data, we conducted additional experiments three representative open-source datasets (ToolACE-100F) Toucan-1.5M" ToolDial-9117 (91)). where we train several models using the same training setting and the corresponding results on the three benchmarks are summarized Table As shown, except for the 7-Airline dataset (for which we previously noted reliability issues), TooLACE-MT consistently achieves superior performance across all multi-turn evaluations. Although lags slightly behind ToolACE (which primarily contains single- turn samples) in the single-turn setting, ToolACE-MT still delivers substantial improvements over
htter///wugppreeeee///tttttt/aap
14

---

Published as a conference paper at ICLR 2026
the base Llama3.1 1-BB model. Overall, the results indicate that the data generated by TMT matches or surpasses that produced by existing methods, with particularly strong gains on multi-turn scenarios, underscoring the effectiveness and superiority of our ToolACE-MT approach.
Table 8: Accuracy comparison (%) on the three representative benchmarks.
BFCL
ACEBench
r-Bench
Models
Single-Tune
Multi-Turn
Multi-Turn
Agent-PA
T-Retail
T-Airline
Llama3.1-8B-
72.7
9.3
24.0
18.3
6.1
26.0*
ToolAAE-10k
80.0
8.6
51.0
13.9
22.0
+ TTuucan-1.5M (sampled 8k)
75.2
15.0
41.0
22.6
18.3
18.0
ToolDial-9k
70.1
12.3
30.0
11.4
10.4
20.0
MAS (this work)
79.2
31.4
48.0
15.0
21.7
10.0
ToolACE-MT (this work)
78.2
40.3
51.0
34.0
25.2
16.0
E FAILURE CASES ANALYSIS
This section analyzes the most frequent failure modes encountered during Offline Verification. Ta- ble [ summarizes the top three failure types for both rule-based and model-based verification. As shown, most errors stem from parameter-related issues, including incorrect parameter values, mis- matches between parameters and function signatures, and improper incomplete parameter format- ting. These findings highlight that parameter handling remains the primary bottleneck in achieving fully reliable tool-calling data generation.
Table 9: Top-3 failure problems for Offline Verification.
Rule-based Problems
Ratio
Model-based Problems
Ratio
Parameter Hallucination
Parameter Extraction
23%
No Tool Calling Conversation Role Misordering
18%
Tool Chosen/Ordering
21%
11%
Parameter Format Compliance
16%
DATA EXAMPLE
Below show data example for reference.
[system] You are an expert in composing functions. You are given a question and a set of pos- sible functions. Based the question, you will need to make one or more function/ tool calls to achieve the purpose. tion lacks the parameters required by the function, also point it out. You should only return the function calls in your response. If you decide to invoke any of the function(s), you MUST put it in the format of [func_ name (params (paramssnameelleeeee namel=params valuel, params, name2=params valuee...), v func (paammparams)), ...] You SHOULD NOT include any other text in the response. Here is a list of functions in JSON format that you can invoke:
none the functions can be used, point it out. the given ques-
("name": "get_ccur.cce date", "description": "Retrieves the current date in YYYY-MM-DD format.",
"parameters":
I"name": "get curr weekday", "description": "Retrieves the current day of the week.", "parame- ters": ...)
("name": "book_flight "description": "Books flight between two cities a given date.", "pa- rameters":...
15

---

Published as a conference paper at ICLR 2026
[user] I'm planning trip and need to fly Shanghai. Could you help me book flight for next Tuesday?
[assistant] Could you provide your departure city?
[user] Singapore.
[assistant] [get_curr_date(] get.curr..eeekday)))
[tool] [["curent date": "2023-10-05"), ("current weekday": "Thursday")]
[assistant] [book bbookk//iggiidpppaauuur..ttsssse
Siwval.ctt='Shannhaa
depar-
sre_ddae==2023-1000
[tool] [ "Flight successfully booked from Singapore to Shanghai on 2023-10- 10."]]
[assistant] Great news! Your flight from Singapore Shanghai for next Tuesday, October 10th, has been successfully booked. Feel free to let me know if there's 's anything else you need assistance with!
[...] ... (More Turns)
The data highlights several essential abilities that an agent must possess. For example:
1) Asking clarification questions when user input is ambiguous or incomplete.
2) Multi-step function calling by leveraging implicit cues from historical context. Here the as- sistant needs to book flight, but the book_flight tool requires specific date in YYYY-MM-DD format. As the date is not provided explicitly, the agent should infer using supporting tools includ- ing get. date and get_eeur_eekkde
G PROMPTS FOR DATA GENERATION
Figure 8 to Figure 15 display the prompts for our TooLACE-MT data generation.
16

---

Published as a conference paper at ICLR 2026
You are task generation expert. Your responsibility is to generate multi-step, tool-usage-related task description English, based the given inputs following the requirements.
You will be provided with:
Several examples for your reference;
A list of available tool candidates;
One completed task descriptions (may also be empty);
A target number of steps N, indicating that the new task should contain N sequential tool calling steps.
## Task Structure Requirements
1. Write concise paragraph in English that describes complete objective consisting of mul- tiple logically related subtasks.
2. The task should contain N steps that can be executed sequentially, with each step triggering one more tool callings.
3. Parallel tool callings (c.g., processing multiple unrelated callings independently at the same time) are counted as a single step.
The steps should exhibit contextual dependency or natural progression, forming a coherent task flow.
5. Each step be deseribed abstract level (no need for detailed parameters), but the exccutable intent must be clear.
## Continuation Requirements
If the "Completed Task" input is not empty, your newly generated task should serve as a natural continuation of those tasks, such as further processing, analysis, or expansion within the same context based on the existing results.
If "Completed Task" input is empty, you are invent a reasonable new task flow.
## Language Requirements
The output should be an English task description.
The description should be concise and fit the context of multi- turn tool usage.
## Given Inputs
#i#l Task Examples (examples)
#### Available Tool Candidates (candidate tools}
#ill Completed Task (completed task)
#### Target Step Number (step_numbery
## Output Format
<aak Start>... (English task dascription)<<Task End>>
Figure 8: The prompt for task initialization.
17

---

Published as a conference paper at ICLR 2026
You are multi-tumn tool-calling dialogue completion expert. Your responsibility to simulate complete trajectory for given task description, hased on the given inputs following the requirements.
You will be provided with:
One example trajectory for your reference;
A list of available tool candidates;
Current task description;
History trajectory that about the previous task completion.
## Completion Requirements
The trajectory should start with the user role raising request, followed by the assistant role completing the task interacting with the tool role. The final turn should be the assistant role, summarizing all results to the user role.
The user role should avoid direct deseriptions of operation steps. Instead, the requests should embodded in context with appropriate discourse markers, interjections, and connecting language to better resemble real human interaction.
3. The input should provide complete parameter information required for tool invocation.
4. The format for the assistant role to call the tools is: [func namel (params namel = params valuel, params name2 params value2...), func name2(parama)]] fol- lowed by tool turn returning results.
5. Tool return results must be in dictionary format, based on the calling parameters the preceding assistant turn and the tool's functionality introduced in tool description.
## Language Requirements
The output should be in English.
The whole trajectory should be reasonable and fit the context of multi-tum tool usage.
## Given Inputs #illi Example Trajectory (cxample)
### Available Tool Candidates (candidate tools}
###f Current Task {current.task)
###l History Trajectory (history_traa trajectory)
## Output Format
"role": "user", "content"; "..."), "),
"role": "assistant", "content"; "... "),
"rooe": "tool", "content": "..."),
Figure 9: The prompt for trajectory initialization.
18

---

Published as a conference paper at ICLR 2026
You are data transformation expert. Your responsibility is modify and extend one specific user turn in given conversation, following the requirements.
You will provided with:
One example for your reference;
A list available tool candidates;
A conversation need be modified;
The specific user turn to be modified and extended.
## Specific requirements
1. First, modify the user's content in this turn to make it vague question or omit necessary information, so that the assistant cannot determine which tool to use or lacks the required parameters needed invoke the tool (avoid using 'this', but 'a' 'some').
2. Then, extend the conversation by adding an assistant turn that asks questions (the assistant cannot assume prior knowledge of the user's intent; the question should naturally match the context) gather sufficient information for invoking the tool.
3. After that, extend with user turn that provides complete and accurate answer with the required parameters.
Ensure that modified extended conversation remains smooth, natural, reasonable.
## Given Inputs #illl Example Modification (cxample)
#### Available Tool Candidates (candidate_
#### Given Conversation {conversation)
#iHl Target User Turn [user_turn)
## Output Format
["role": "user", "content": "..."), "), ["role": "assistant", "content"; " "),
("role": "user", "content": "...")
Figure 10: The prompt for adding clarification turns in complexity injection.
19

---

Published as a conference paper at ICLR 2026
You are a data transformation expert. Your responsibility is to extend one specific user turn in given conversation, following the requirements.
You will be provided with:
One example for your reference;
A list available tool candidates;
A conversation need to be modified;
The specific user be extended.
The specific candidate tool to be removed.
## Specific requirements
1. Keep the user turn entircly unchanged, but adding two additional turns.
2. The first added turn should be an assistant turn, expressing that the current candidate tools cannot meet the user's needs.
3. The second added should he turn, directly providing the description of the moved tool the assistant to call.
4. Ensure that the extended conversation remains smooth, natural, and reasonable.
## Given Inputs ### Example Extension (example)
#illi Available Tool Candidates (candidate_tooly
### Given Conversation {conversation)
### Target User Turn [user_turn]
###l The Tool to be Removed (removed.tool)
## Output Format
"role": "user", "content"; "...",, "role": "assistant", "content"; " "),
["role": "user", "content": "...")
Figure 11: The prompt for tool awareness complexity injection.
20

---

Published as a conference paper at ICLR 2026
You are a data transformation expert. Your responsibility is to extend one specific assistant turn in given conversation, following the requirements.
You will be provided with:
One example for your reference;
A list available tool candidates;
A conversation need be modified;
The specific assistant turn to be extended.
## Specific requirements
1. Modify the tool calling part of the assistant turn, injecting one error parameter value.
Add tool turn returning error messages and showing possible solutions.
3. Then add another assistant turn that corrects the tool calling statement.
Ensure that the modified and extended conversation remains smooth, natural, reasonable.
#i Given Inputs ### Example Modification (example)
#ill Available Tool Candidates (candidate tools)
#iff Given Conversation {conversation)
### Target Assistant Turn (ussistant_turm)
##i Output Format
""role": "assistant", "content"; "..."), ["role": "tool", "content": "..."),
["role"; "assistant", "content"; " ")
Figure 12: The prompt for error simulation in complexity injection.
21

---

Published as a conference paper at ICLR 2026
You data transformation expert. Your responsibility is modify and extend specifie turn in given conversation, following the requirements.
You will be provided with:
example for your reference;
A list of available tool candidates;
conversation need be modified:
The specific user turn to be modified and extended.
## Specific requirements
Add two turns before the specific
2. The first added turn should be a user turn. Its content may be casual chit-chat or request that does not require function calling (e.g.. asking for recommendations, translation, or open- ended writing). The topic should be related to the original user turn.
3. The second added turn should be an assistant response directly addressing the first added user turn.
Keep the content of the original (specified) user turn unchanged. and append it as the next tur..
5. Ensure that the modified and extended conversation remains smooth, natural, and reasonable.
## Given Inputs ### Example Modification (example)
#illi Available Tool Candidates (candidate tools}
### Given Conversation {conversation)
### Target User Turn (user.turn)
##i Output Format
"role": "user", "content": "... ",,
("role"; "assistant", "content"; ",.."), ["role": "user", "content": "..."}
Figure 13: The prompt for non-function-calling in complexity injection.
22

---

Published as a conference paper at ICLR 2026
You data completion expert. Given conversation between user and an assistant, where the assistant perform tool calling complete the user's task, your responsibility is fill in the missing content following the requirements.
You will be provided with:
A list available tool candidates;
A partially completed conversation, with some content missing and replaced by placeholders such as "xxx", ", "yyy", etc.
## Completion Requirements
should try your to recover the missing content, by replacing the placeholders with actual content.
2. If the recovered content is in user turn, the content should avoid direct descriptions of oper- ation steps. Instead, the requests should embedded context with appropriate discourse markers, interjections,
connecting language to better resemble real human interaction.
3. If the recovered content is in assistant turn and need calling tools, the for- namel
mat for the assistant role call the tools is: [func mamel(parenn params valuel, params name? params value2...), f fune vame2(parans)]
4. If the recovered content is in tool turn, should simulate reasonable tool output that coherent with its adjacent turns' actions.
5. Ensure that the recovered whole conversation is smooth, natural, and reasonable.
#i Given Inputs #illi Available Tool Candidates (candidate_ton) [tolss
### Given Conversation (conversation)
## Output Format
"xxx" "...", "yyy": : "...",
Figure 14: The prompt for mask-and-fill in reasonability refinement.
23

---

Published as a conference paper at ICLR 2026
You are data quality evaluation expert. Given a conversation history and two possible continued trajectories, your responsibility is to determine which continued trajectory is of higher quality.
You will be provided with:
A list of available tool candidates;
A conversation history;
Two continued trajectories.
## Evaluation Criteria
1. Coherence: Choose the trajectory that exhibits smooth and natural progression.
2. Correctness:
Tool calling statements must be strictly correct, consistent with the dialogue history, and must not assume any values that have not previously appeared.
3. Consistency: Pay close attention to aspects such user-assistant consistency, the plausibility parallel function calls, tool output formatting, and overall structure.
4. Deep thinking:
Before providing your final judgment, first present your reasoning process.
## Given Inputs ###f Available Tool Candidates {candidate.
Given Conversation History (conversation)
### Continued Trajectory A (trajectory a}
#ilfi Continued Trajectory B (trajectory.b)
## Output Format
"think": "...", "judgement": "A/B",
Figure 15: The prompt for judger in reasonability refinement.
24


Reviews:
[{"review_id": "dAIeHiux2c", "reviewer": "Reviewer_HKC7", "summary": "This paper introduces NAIG (Non-Autoregressive Iterative Generation), a novel framework for generating multi-turn dialogues involving function-calling and agentic behavior in large language models (LLMs). Inspired by Non-Autoregressive Translation (NAT) and masked diffusion language models, NAIG generates full conversational trajectories through a non-autoregressive pipeline consisting of three stages. First, a coarse-grained initialization step produces a structurally complete but semantically shallow dialogue skeleton. Second, an iterative refinement stage enhances complexity and semantic coherence using a mask-and-fill strategy inspired by masked language models. Third, an offline verification step applies a combination of rule-based and model-based checks to filter out inconsistent or hallucinated samples. The authors evaluate NAIG on several benchmarks, demonstrating significant improvements over multi-agent simulation (MAS) approaches in multi-turn function-calling accuracy, data generation efficiency, and model generalizability.", "questions": "1. Please provide detailed statistics of the dataset, such as the number of tools, average number of turns per dialogue, and the average number of tool-calling events per dialogue.\n2. Is NAIG extendable to multimodal agentic tasks, such as those involving visual inputs? \n3. It would be helpful to include additional comparative analysis against other state-of-the-art tool-based dialogue generation methods to better contextualize NAIG's performance. (e.g. ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models, ICLR 2025)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. NAIG produces more coherent, context-aware, and complex multi-turn dialogues by combination of structured initialization and iterative refinement.\n2. Their method reduces reliance on expensive multi-agent autoregressive interactions, lowering API costs\n3. Offline verification enhances data reliability by catching structural and semantic inconsistencies that are hard to detect during generation.\n4. Models trained on NAIG-generated data significantly outperform both MAS-based on multi-turn benchmarks.", "weaknesses": "1. NAIG generation quality significantly drops when using smaller models (e.g., GPT-4o-mini), limiting its applicability in low-resource settings.\n2. Models trained on NAIG-generated data may over-optimize for multi-turn planning, potentially at the expense of performance on single-turn or one-shot queries.\n3. The experiments only compare NAIG against a single model despite the diversity of baselines available for tool-calling dialogue generation, raising concerns about the sufficiency and generalizability of the evaluation. \\", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5JNkg7wq8y", "reviewer": "Reviewer_TB5g", "summary": "This paper addresses the challenge of generating high-quality multi-turn, multi-step dialogues for agentic task-solving with LLMs, where existing simulation-based methods rely on costly autoregressive interactions and often limit task performance. To tackle this, the authors propose NAIG (Non-Autoregressive Iterative Generation), a three-stage framework that constructs complete conversational trajectories through coarse-grained initialization, iterative refinement via mask-and-fill operations, and offline verification to ensure semantic coherence, contextual consistency, and correctness. Rich experiments were conducted to prove the method's effectivenes.", "questions": "1. The authors mention three key issues in back-and-forth interaction generation. If a single LLM generates the entire interaction via a well-designed prompt and then is refined interactively, would these issues persist, or would new ones emerge?\n2. The abbreviation \"MAS\" is commonly used to refer to multi-agent systems. To avoid potential confusion, I suggest using a different short name for multi-agent simulations.\n3. In Figure 4, why is the maximum number of assistant turns in the NAIG training data larger than that in the MAS training data? Could the authors clarify this observation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This work identifies three important challenges in simulation-based data generation for agentic multi-turn interactions.\n2. The dataset has the potential to significantly contribute to research on agentic LLMs, provided it can be publicly released.\n3. The paper is clearly written, and the methodology is described in sufficient detail.", "weaknesses": "1. The multi-agent simulation process does not incorporate offline verification, and such verification is not an inherent requirement for non-autoregressive methods. This raises concerns about the fairness of comparing NAGI with methods that include offline verification.\n2. The improvement of NAIG over Multi-Agent Simulation appears to be marginal. Considering Weakness 1, the core component of the non-autoregressive framework (i.e., NAIG without offline verification) does not seem to provide a meaningful improvement.\n3. The reported computational cost may be misleading, as it appears to consider only API calls without accounting for the tokens required for model training and inference. In particular, NAGI likely requires more tokens for both inputs and outputs per API call, which could further increase the actual computational cost.\n4. The conclusion that the NAIG generation paradigm supports more effective overall task planning may not be fully supported by the assistant turn counts. Notably, the maximum number of assistant turns in the NAIG training data is higher than in the MAS training data, and the analysis is limited to models trained on τ-Bench. Additional analyses would help strengthen this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ylEYam7tmX", "reviewer": "Reviewer_9xQr", "summary": "This paper proposes Non-Autoregressive Iterative Generation for Agentic Multi-Turn Interaction. The authors address the need for high-quality multi-turn, multi-step dialogues involving function calls and user-agent interaction in tool-augmented LLM settings. They argue that existing multi-agent simulation approaches rely on costly autoregressive generation across multiple agents. To improve efficiency and control, they introduce a three-stage data construction framework: coarse-grained initialization, iterative refinement, and offline verification. The initialization produces a structurally complete but semantically rough dialogue skeleton; the refinement injects complexity and uses mask-and-fill operations to improve the trajectories; the verification filters for correctness and coherence via rule-based and model-based checks. Experiments on multiple benchmarks show that models fine-tuned on data generated by their framework outperform MAS-generated data in downstream function-calling tasks while reducing API-call cost.", "questions": "1. Can you provide quantitative metrics on the generated dataset itself: e.g., number of unique skeleton templates used, duplication rate of structures, Distinct-n diversity scores, or human preference judgments?\n2. Could you report token counts, compute hours and/or monetary cost estimates for the data generation process (initialization + refinement + verification) to strengthen the cost-reduction claim?\n3. Do you have statistics on how many samples were discarded during offline verification, and what the major categories of failure were? Could you share sample failure cases?\n4. How do you ensure that sub-task sequences in each dialogue reflect realistic correlations rather than random concatenations? Did you compare your generated task‐structure distribution with that of real agent dialogues?\n5. Are there template or skeleton reuse effects in the initialization phase? How many skeleton variants were used, and how did you guard against structure homogenization?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses an important problem: generating realistic, tool-augmented agentic dialogues for training LLMs with function-calling and multi-turn behaviour.\n- The framework is clearly presented. The three-stage design offers a practical pipeline that balances structure, refinement and quality control.\n- The experimental evaluation is reasonably broad. Multiple benchmarks, different model backbones, cost vs quality trade-off are all considered.\n- The authors attempt to quantify cost savings (via API-call count) and show downstream performance improvements, which strengthens the applied relevance of the work.", "weaknesses": "1. Lack of human evaluation and direct quantitative assessment of the generated data itself. The paper focuses on downstream model performance improvements, but fails to report metrics such as diversity (Distinct-n), structural repetition (how many dialogue skeleton templates were reused), naturalness/human preference, or other data-intrinsic quality measures.\n2. Cost evaluation metric is limited. While the paper reports that API calls drop from ≈ 275k (MAS) to ≈ 188k (their method), there is no breakdown of token consumption, compute hours, or monetary cost. Thus the claim of “cost reduction” lacks transparency and might not reflect real-world resource savings.\n3. Failure case analysis could be provided. The paper does not include a detailed look at failed/generated invalid samples: which trajectories failed verification, what error modes occurred (parameter mismatch, tool-call misordering, semantic incoherence), or how many were discarded. Without this, it is hard to judge the robustness or failure modes of the proposed pipeline.\n4. The initialization stage lacks transparency and poses risks of structural homogenization or unrealistic data distribution. The “coarse-grained initialization” phase is described at a high level but lacks detail on how skeletons are generated, how varied they are, whether templates are reused, or whether sub-tasks are meaningfully correlated as in real distributions. This raises concerns that many samples might share the same skeleton, reducing diversity, or that sub-tasks may be arbitrarily concatenated in ways rare in real user-agent dialogues.\n5. The novelty is somewhat incremental. The combination of non-autoregressive generation + iterative mask-and-fill + simulation for data generation is practically useful, but from a model-research perspective, it is more of a pipeline engineering effort than a conceptual leap in generation modelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nNHvt2RbrN", "reviewer": "Reviewer_JVVr", "summary": "This paper proposes a non-autoregressive iterative generative framework (NAIG) for constructing high-quality multi-agent dialogues. NAIG consists of three main phases: coarse-grained dialogue structure initialization, iterative refinement, and offline verification. NAIG first globally initializes a fully structured multi-agent dialogue. Iterative refinement uses masking and padding to continuously refine dialogue details. The offline verification phase verifies the correctness and consistency of the dialogue. Experiments show that, using the same base model, training on data generated by NAIG achieves better accuracy and efficiency than traditional multi-agent simulations.", "questions": "(1) Is there comparative data on the total token consumption for NAIG and MAS?  \n(2) I would like to know the specific steps of the MAS experiment in the article. How is it different from NIAG, which only uses the initialization phase and offline verification phase?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This work makes the following contributions to synthesizing complex agent task data:  \n(1) It proposes a framework for generating complex agent task data, focusing on refining conversation details while maintaining the overall task through masking and padding.  \n(2) It splits initialization and refinement into two phases, allowing for the definition and selection of task complexity to be completed at a low cost in the initialization phase, while controlling complexity and budget by the number of iterations in the iterative refinement phase.  \n(3) The NAIG framework offers a straightforward approach that addresses the following motivations: (a) When using autoregressive methods, trajectories often exhibit undesirable shifts (especially in long-range scenarios). Employing the mask-fill method allows LLMs to perceive the full context of trajectories, mitigating inconsistencies arising from modifications to some extent. (b) During initial trajectory generation, NAIG enables explicit control over length and complexity, thereby facilitating more controllable data design.", "weaknesses": "(1) The article mentions (Table 4) that NAIG is less expensive than traditional MAS. I understand this refers to the number of API calls. Although the number of API calls is less than MAS, since operations such as iterative optimization may consume a lot of tokens, is the total token cost still lower than MAS?\n(2) In Section 4.3, the authors mention that there is a significant performance gap between GPT-4o-mini and GPT-4o. This highlights that the quality of initial generation is still crucial. I don't understand why only the quality of initial generation is mentioned here.\n(3) I would like to know the specific steps of the MAS experiment in the article. How is it different from NIAG, which only uses the initialization phase and offline verification phase?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)