Failure makes the agent stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactions

## Abstract

Abstract Tool-augmented large language models (LLMs) are typically trained via supervised imitation learning or coarse-grained reinforcement learning, approaches that primarily optimize one-shot tool calls. Existing practices of self-reflection largely rely on heuristic prompting or unidirectional reasoning traces: the model is encouraged to “think more,” rather than to treat error diagnosis and correction as a learnable capability. This makes them fragile in multi-turn interaction settings—once a call fails, the model tends to repeat the same mistake instead of recovering.
To address this issue, we propose structured reflection, which transforms the “from error to repair” process into a first-class, controllable, and trainable action. The agent produces a concise yet precise reflection process: specifically, the model diagnoses the error based on evidence from the previous step and then proposes a correct and executable follow-up call. During training, we combine DAPO and GSPO’s objective functions and design a more principled reward mechanism tailored to tool calling, optimizing the stepwise strategy Reflect → \to Call → \to Final.
To evaluate this capability, we introduce Tool-Reflection-Bench, a lightweight benchmark dataset that programmatically verifies structural validity, executability, parameter correctness, and result consistency. Tasks in the benchmark are constructed as miniature trajectories of Erroneous Call → \to Reflection → \to Corrected Call and are split into disjoint training and testing sets.
Experiments on BFCL v3 and Tool-Reflection-Bench show that our method achieves significant improvements in multi-turn tool-call success rates and error recovery, while also reducing redundant calls. These results demonstrate that making reflection explicit and treating it as an optimization objective can substantially enhance the reliability of tool interaction, providing a reproducible pathway for agents to grow stronger by learning from failure. We will release all the code and datasets as open source once the paper is accepted by the community.

## 1 Introduction

The integration of external tools with large language models through tool calling represents a significant breakthrough in the development of agents. It transforms large language models from mere text generators into highly practical tools for interacting with humans [1, 2], significantly enhancing the ability of AI agents to solve complex real-world tasks [3, 4, 5]. Tool calling bridges the gap between the vast internal knowledge of LLMs and external resources, enabling LLMs to access up-to-date information, perform delicate computations, and more, thereby unlocking their broad potential for applications across multiple domains [6, 7, 8].

Currently, the training of tool-call capabilities in large language models typically relies on supervised fine-tuning and reinforcement learning [9, 10], where these methods optimize the ability for single-turn tool calls through carefully designed reward mechanisms. However, these approaches face several challenges in the context of tool calling. First, the issue of rewards in tool calling is particularly prominent—small errors in parameter selection or formatting often render the entire function call invalid, thus limiting the effective learning signal [11]. Second, existing methods generally rely on unidirectional reasoning, which, while sufficient for simpler scenarios, has clear limitations: when LLMs make mistakes during tool calls, they often struggle to locate the root cause of the error [12]. While generating correct function calls is crucial, it is even more important for LLMs to learn how to identify and correct their own mistakes [13].

To address the above-mentioned issues, we propose an innovative reflection process aimed at error localization and correction through explicit reflection steps, which differs from existing forward reasoning methods. Specifically, we design a process in which the LLM intentionally makes mistakes during tool calls, carefully crafts reflection content based on the errors, and then generates the correct call. Through this approach, we transform the self-correction ability of large models from a heuristic process [14] into a clear, trainable capability.
Our training approach is primarily reinforcement learning–based. During the reinforcement learning process, we specifically design a customized reward mechanism tailored for tool-calling scenarios, with a particular emphasis on multi-turn interactions. Concretely, the reward design encompasses multiple dimensions, including format reward, tool-name reward, parameter reward, and semantic reward of reflection, which together provide the model with multi-dimensional feedback and effectively guide its learning, and we further combine DAPO’s decoupled clipping range and dynamic sampling—expanding exploration while skipping near-zero-advantage rollouts—with GSPO’s sequence-level importance sampling and same-granularity clipping, which avoids token/sequence mismatch and stabilizes optimization. With this training methodology, our approach equips LLMs with genuine self-reflection and error-correction capabilities. On the BFCL v3 benchmark, our method yields significant improvements in LLM accuracy for multi-turn tool calling, thereby demonstrating its effectiveness in real-world applications.

We construct a Tool-Reflection-Bench based on the BUTTON dataset [15] style. First, we collected tool-call failure cases from real-world scenarios and various benchmarks, analyzing and summarizing several common failure patterns. Next, We selected several existing tool-call datasets [4, 16] and randomly combined them according to the call style of the BUTTON dataset and introduced these failure patterns into the data, disrupting the originally correct call processes to generate failure cases. Finally, we meticulously designed a reflection process to repair these failures, resulting in successful tool calls.
The training set includes the complete process described above to train LLMs to achieve true self-correction capabilities, while the test set only contains the first two steps, used to evaluate the self-correction abilities of the LLMs.
By constructing the Tool-Reflection-Bench in this manner, combined with our custom reward mechanism for tool calling, we have made breakthroughs in LLMs’ self-correction abilities during training. Particularly in multi-turn tool-calling scenarios, we observed significant improvements in accuracy. Through the reasoning process from failure to correction, LLMs can more effectively identify and learn from potential mistakes, thus enhancing the model’s stability and robustness in interactions. This makes the agent’s behavior more robust and powerful.

In summary, our contributions are as follows:

- •
We introduce an explicit, trainable reflection process that diagnoses the cause of a failed tool call using prior evidence and proposes a corrected, executable call. This transforms the "from failure to repair" process from a heuristic method into a learnable action strategy, enabling LLMs to genuinely possess self-reflection and error-correction capabilities, thereby enhancing the agent’s multi-turn interactions with users.
- •
We design a more effective reward mechanism for tool call, tailored for RL training, using a GRPO-style objective function. This approach employs multi-dimensional rewards—format executability, tool name accuracy, parameter correctness, and semantic consistency—to mitigate sparse rewards and propagate signals across multi-turn trajectories.
- •
We propose Tool-Reflection-Bench, which collects failure patterns from real interaction scenarios and benchmark datasets, injects perturbations into correct calls, and attaches a reflection process to repair the calls. This allows for training LLMs in their Self-Correction ability in tool-calling scenarios.
- •
Our method significantly improves the accuracy of multi-turn tool calls and the ability to recover from tool call errors, while maintaining competitive single-turn tool call performance. We validate this by experiments on BFCL v3 [17] and Tool-Reflection-Bench.

## 2 Related Works

### 2.1 Tool-augmented Large Language Models

Integrating external tools into large language models has become a key approach to enhancing their functionality, surpassing the simple task of text generation. Traditional LLMs are limited by static knowledge, constrained to the data they were trained on. However, tool-augmented models extend the capabilities of LLMs by enabling them to interact with external resources [18, 19] (such as APIs [20], databases, and computational engines) through tool calls. This extension allows LLMs to access real-time data, perform external computations, and even interface with external hardware, making them more practical for solving complex real-world tasks that require dynamic information or specific external operations [21].
ToolBench [4] demonstrates the feasibility of integrating external tool calls into LLMs. Through such systems, LLMs can handle more specialized tasks. However, one major challenge of tool augmentation is how to effectively train LLMs to use these tools. Existing training methods, such as supervised fine-tuning and reinforcement learning, typically focus on optimizing single tool calls. This type of interaction often does not involve multi-turn tool calls or responses, which makes the limitations of current methods particularly apparent when errors occur during tool usage. In such cases, the model’s ability to recover from errors becomes crucial.

### 2.2 Self-Correction in LLMs

Self-correction in large language models refers to the model’s ability to diagnose its own errors and correct them based on previous actions [22, 23]. However, this area has not been fully explored. Existing self-correction techniques mostly rely on heuristic methods or unidirectional reasoning processes [24].

Self-Refine framework [25], which involves having LLMs provide an initial response, followed by a reflection process where the model identifies flaws and makes improvements. Specifically, the same LLM acts as both the responder and the evaluator: the model first generates an initial response, then self-reflects and iteratively revises the output. This approach has been shown to enhance the performance of LLMs in certain domains. However, subsequent studies [26, 27] have found that relying solely on the model itself often fails to detect subtle errors. Some research [28, 29] has introduced auxiliary verifiers (such as additional models or mechanisms [30, 31]) to help check the correctness of the initial response. This external self-checking assistance avoids unnecessary repeated revisions, improving efficiency and enhancing the model’s reasoning and verification capabilities. However, this approach remains highly sensitive to the specific phrasing of the prompts, with different prompt wordings leading to varying results [23].

However, even though these methods have somewhat improved the capabilities of LLMs, their essence still relies on external feedback to assist in correcting their own errors. In contrast, our work introduces a reflection method that transforms self-correction into a trainable and controllable capability. The reflection process is an explicit part of the task, where the model actively evaluates its previous actions, identifies errors, and generates explicit corrections. This process is achieved through error localization, diagnosis, and correction, which goes beyond simple unidirectional reasoning and can be integrated into the training process. By providing supervisory signals during training, our approach enables LLMs to truly possess self-correction capabilities, which are then reflected in tool calling tasks.

## 3 Method

### 3.1 Tool-Reflection-Bench

The construction of Tool-Reflection-Bench consists of the following steps: perturbation-based disruptions, positive samples transformations, and the reflection repair process. The original positive samples are derived from BUTTON [15] transformations and self-constructed based on few-shot prompts. The entire benchmark is divided into a training set and a test set, with approximately 5,000 samples in the training set, in addition to the reflection-augmented data constructed as described above, the training set also contains a very small portion of original data drawn from BUTTON [15] and XLAM [18]. And around 1,000 samples in the test set, the test set is exclusively composed of perturbation-derived items and does not include raw, unperturbed positives from BUTTON or XLAM.

Figure: Figure 2: We illustrate the effectiveness of our method with an example. As shown in the figure, the left side presents the tool panel, while the upper-right part depicts industry-standard self-correction approaches, where models attempt to fix errors through heuristic trial-and-error reasoning or by relying on external feedback. In contrast, the lower-right part shows our approach: we introduce an explicit forced reflection process $<$r$>$, enabling the model to truly master the ability to repair errors based on its own failures.
Refer to caption: x3.png

#### 3.1.1 Perturbation-based Disruptions

Let the initial correct message sequence be

$$ $D^{+}=\Big(m^{\mathrm{sys}}_{0},\,m^{\mathrm{usr}}_{1},\,m^{\mathrm{ast}}_{2},\,m^{\mathrm{tool}}_{3},\,m^{\mathrm{ast}}_{4},\,m^{\mathrm{tool}}_{5},\,\dots,\,m^{\mathrm{ast}}_{2k},\,m^{\mathrm{tool}}_{2k+1},\,\dots,\,m^{\mathrm{final}}_{n}\Big),$ (1) $$

where $m^{\mathrm{sys}}_{0}$ is the system prompt, $m^{\mathrm{usr}}_{1}$ the user query, $m^{\mathrm{ast}}_{2i}$ the assistant’s $i$-th tool call in structured form (e.g., <call>[{…},{…},...]</call>), $m^{\mathrm{tool}}_{2i+1}$ the tool return (JSON), and $m^{\mathrm{final}}_{n}$ the final answer.

We define a set of disruption operators

$$ $\mathcal{P}=\{P_{1},P_{2},P_{3},P_{4}\},$ (2) $$

each operating on an assistant call $m^{\mathrm{ast}}_{2k}$ and instantiating a common failure mode:

- 1.
$P_{1}$ call-order swap: replace the current tool call with the next-round tool call dialogue and force an error.
- 2.
$P_{2}$ redundant call: repeat the same tool at the step (unchanged/irrelevant arguments) and force an error.
- 3.
$P_{3}$ missing call: replace the intended tool by another tool and force an error.
- 4.
$P_{4}$ argument error: randomly corrupt the arguments of a call (missing/typed/alias/boundary) and force an error.

These operators specify how a correct tool call can be broken.

#### 3.1.2 Positive Samples Transformations

Given a clean trajectory $D^{+}$ and a chosen operator $P_{j}\in\mathcal{P}$ acting on step $2k$, we produce the negative (erroneous) context; no repair is performed in this step.
We construct the erroneous call

$$ $\tilde{m}^{\mathrm{ast}}_{2k}=\mathrm{ApplyPerturbation}\!\left(m^{\mathrm{ast}}_{2k},\,P_{j}\right),$ (3) $$

and simulate the tool’s error feedback with a LLM $\mathcal{L}$:

$$ $\tilde{m}^{\mathrm{tool}}_{2k+1}=\mathcal{L}\!\left(\tilde{m}^{\mathrm{ast}}_{2k};\,\mathcal{L}\right).$ (4) $$

This yields the negative trajectory prefix

$$ $D^{-}=\mathrm{Perturb}\!\left(D^{+},P_{j}\right)=\Big(m^{\mathrm{sys}}_{0},\,m^{\mathrm{usr}}_{1},\,\ldots,\,\tilde{m}^{\mathrm{ast}}_{2k},\,\tilde{m}^{\mathrm{tool}}_{2k+1}\Big),$ (5) $$

which will later serve as evidence of failure. At this stage, the item consists only of the broken call and its error signal.

#### 3.1.3 Reflection Repair Process

Given a clean trajectory $D^{+}$ and its perturbed prefix $D^{-}$, we present the LLM with a paired view of the step-$2k$ evidence:

$$ $\text{clean: }(m^{\mathrm{ast}}_{2k},\,m^{\mathrm{tool}}_{2k+1})\quad\text{vs.}\quad\text{broken: }(\tilde{m}^{\mathrm{ast}}_{2k},\,\tilde{m}^{\mathrm{tool}}_{2k+1}).$ (6) $$

The model outputs a response.

$$ $\langle\texttt{reflect}\rangle r\langle/\texttt{reflect}\rangle,$ (7) $$

where $r$ briefly diagnoses the discrepancy, and $c$ proposes the fixed tool call. We then apply human supervision to obtain
$(r^{\star},c^{\star})$, with $c^{\star}$ set to the original correct call:

$$ $(r,\,c)\;\xRightarrow[\text{human supervision}]{\text{post–editing}}\;(r^{\star},\,c^{\star}),$ (8) $$

$$ $\mathcal{L}_{\Sigma}(c^{\star})=\texttt{Success Call}.$ (9) $$

The finalized item is packaged as

$$ $x\;=\;\big(D^{-},\,r^{\star},\,c^{\star},\,D^{+}_{>2k+1}\big),$ (10) $$

where $D^{+}_{>2k+1}$ is the untouched suffix of $D^{+}$ (including $m^{\mathrm{final}}_{n}$). We retain $x$ only if:
(i) tags/JSON are well-formed; (ii) $c^{\star}$ is executable; (iii) $r^{\star}$ correctly cites the clean–broken contrast.

### 3.2 Reward Design

##### Preliminary.

Given a model completion $C$ and a ground truth $G$, we decompose both into three (possibly empty) parts:

$$ $C\mapsto\big(c_{\text{ref}},\;C_{\text{calls}}=\{c_{i}\}_{i=1}^{m},\;c_{\text{final}}\big),\qquad G\mapsto\big(g_{\text{ref}},\;G_{\text{calls}}=\{g_{j}\}_{j=1}^{n},\;g_{\text{final}}\big).$ (11) $$

Here $c_{\text{ref}}$ (*reflection*) is the diagnosis text wrapped in <reflect></reflect>, $C_{\text{calls}}$ is the multiset of tool calls wrapped in <call></call>s produced by the model, and $c_{\text{final}}$ is the message wrapped in <final></final>. The ground truth can alse be decomposed into three parts mentioned above.

##### Component scores.

We compute three component scores:

$$ $s_{\text{ref}}=\mathrm{Sim}(c_{\text{ref}},g_{\text{ref}}),\qquad s_{\text{call}}=\mathbb{I}\!\left[\mathrm{EqualCalls}(C_{\text{calls}},G_{\text{calls}})\right],\qquad s_{\text{final}}=\mathrm{Sim}(c_{\text{final}},g_{\text{final}}),$ (12) $$

where $\mathrm{Sim}\!\in[0,1]$ is a semantic similarity function, and
$\mathbb{I}[\cdot]$ is the indicator:

$$ $\mathbb{I}[P]\;=\;\begin{cases}1,&\text{if $P$ is true},\\ 0,&\text{otherwise}.\end{cases}$ (13) $$

We say $\mathrm{EqualCalls}(C_{\text{calls}},G_{\text{calls}})$ holds iff the two sets of produced calls can be put in a one-to-one correspondence such that for every matched pair the tool name is identical and the argument is identical.

##### Normalization with presence masks.

Our goal is to keep the aggregated score in $[0,1]$ even when an instance specifies only a subset of targets
(e.g., only <call> without <reflect> or <final>). To this end we use normalization
to renormalize over the parts that actually appear in the ground truth, so the maximum remains $1$ regardless of how
many parts are present.

We define

$$ $I_{\text{r}}=\mathbb{I}[\,g_{\text{ref}}\neq\varnothing\,],\qquad I_{\text{c}}=\mathbb{I}[\,|G_{\text{calls}}|>0\,],\qquad I_{\text{f}}=\mathbb{I}[\,g_{\text{final}}\neq\varnothing\,].$ (14) $$

Let $(w_{\text{r}},w_{\text{c}},w_{\text{f}})\!\geq\!0$ be *normalized* base weights (e.g., $w_{\text{r}}+w_{\text{c}}+w_{\text{f}}=1$).
We renormalize over the active parts via

$$ $W_{\text{act}}\;=\;w_{\text{r}}I_{\text{r}}+w_{\text{c}}I_{\text{c}}+w_{\text{f}}I_{\text{f}}.$ (15) $$

The aggregated structure/semantics score is then

$$ $S\;=\;\dfrac{w_{\text{r}}I_{\text{r}}\,s_{\text{ref}}\;+\;w_{\text{c}}I_{\text{c}}\,s_{\text{call}}\;+\;w_{\text{f}}I_{\text{f}}\,s_{\text{final}}}{W_{\text{act}}}.$ (16) $$

This normalization yields a consistent scoring standard across fully and partially supervised instances, avoiding artificial deflation of scores when some targets are absent.

##### Format/penalty factor.

We designed structural penalties tailored for tool-call data formats. Specifically, $P_{miss}$ accounts for cases where the tool is not invoked at all, while $P_{extra}$ and $P_{count}$ penalize redundant calls and mismatches in the total number of calls, respectively. Let

$$ $n=|G_{\text{calls}}|,\qquad m=|C_{\text{calls}}|,$ (17) $$

Here $n$ and $m$ denote the number of tools invoked in the ground truth and completion calls.
Define the three components:

$$ $P_{\text{miss}}\;=\;w_{\text{ref}}\;\mathbb{I}[\,g_{\text{ref}}\!\neq\!\varnothing\wedge c_{\text{ref}}\!=\!\varnothing\,]\;+\;w_{\text{final}}\;\mathbb{I}[\,g_{\text{final}}\!\neq\!\varnothing\wedge c_{\text{final}}\!=\!\varnothing\,]\;+\;w_{\text{calls}}\;\mathbb{I}[\,n>0\wedge m=0\,],$ (18) $$

$$ $P_{\text{extra}}\;=\;w_{\text{ref}}\;\mathbb{I}[\,c_{\text{ref}}\!\neq\!\varnothing\wedge g_{\text{ref}}\!=\!\varnothing\,]\;+\;w_{\text{final}}\;\mathbb{I}[\,c_{\text{final}}\!\neq\!\varnothing\wedge g_{\text{final}}\!=\!\varnothing\,]\;+\;w_{\text{calls}}\;\mathbb{I}[\,m>0\wedge n=0\,],$ (19) $$

$$ $P_{\text{count}}\;=\;w_{\text{calls}}\;\mathbb{I}[\,n>0\wedge m>0\wedge n\!\neq\!m\,]\;\frac{|n-m|}{\max(n,m)}.$ (20) $$

Let $\mathrm{EqualCalls}$ be the schema-strict equality on bags of calls.
We use a reduction factor

$$ $r\;=\;\begin{cases}r_{\text{reduce}},&\text{if }\mathrm{EqualCalls}(C_{\text{calls}},G_{\text{calls}}),\\ 1,&\text{otherwise,}\end{cases}\qquad r_{\text{reduce}}\in(0,1].$ (21) $$

Aggregate the penalty as

$$ $P_{\text{total}}\;=\;P_{\text{miss}}\;+\;\beta_{\text{extra}}\,P_{\text{extra}}\;+\;\gamma_{\text{count}}\,P_{\text{count}},$ (22) $$

and define the instance-wise format factor

$$ $\mathrm{FormatFactor}(C,G)=\begin{cases}1,\text{if }P_{\text{miss}}=0\ \wedge\ P_{\text{extra}}=0\ \wedge\ P_{\text{count}}=0,\[4.0pt] \max(0,\min(1,\,1-\lambda_{m}P_{\text{total}}r)),\text{otherwise.}\end{cases}$ (23) $$

Here $\beta_{\text{extra}},\gamma_{\text{count}},\lambda_{m}\!\geq\!0$ control the strength of extra-part, count-mismatch,
and overall scaling penalties, respectively; $(w_{\text{ref}},w_{\text{calls}},w_{\text{final}})\!\geq\!0$ are part weights.

Figure: (a) The reward curve of llama-3.1-8b-Instruct during RL training
Refer to caption: picture/rl_llama.png

##### Core reward and backoff.

The core reward is

$$ $R_{\text{core}}\;=\;S\cdot F.$ (24) $$

Early in training, $S$ contains a binary component ($s_{\text{call}}\!\in\!\{0,1\}$) and
$F$ applies hard penalties; small formatting or argument errors can drive $R_{\text{core}}$ close to zero.
This yields sparse or unstable gradients and large variance across samples. To stabilize learning and provide a
dense shaping signal when the exact-match objective is not yet achieved, we introduce a similarity backoff:

$$ $R_{\text{total}}\;=\;\begin{cases}\mathrm{clip}_{[0,1]}(R_{\text{core}}),&R_{\text{core}}\geq\varepsilon,\[4.0pt] \mathrm{clip}_{[0,1]}\!\left(w_{\text{b}}\cdot\mathrm{Sim}\!\big(\mathrm{concat}(C),\,\mathrm{concat}(G)\big)\right),&\text{otherwise,}\end{cases}$ (25) $$

where $w_{\text{b}}\in(0,1]$ and $\mathrm{concat}(\cdot)$ linearizes the messages. We use
$\mathrm{clip}_{[0,1]}(x)=\max(0,\min(1,x))$ to keep rewards bounded.

### 3.3 RL for Tool-Reflection-Bench

We adopt a reinforcement-learning objective for tool calling that combines two complementary ideas:
(i) DAPO-style decoupled clipping [32]: we use a decoupled clipping range with different lower/upper bounds $(\varepsilon_{\text{low}},\varepsilon_{\text{high}})$ and a clip-higher policy (a looser upper bound when $r>1$ for positive advantages), and we skip uninformative prompt groups whose rollouts carry negligible learning signal;
(ii) GSPO-style sequence-level importance sampling [33]: we compute the importance ratio at the sequence level and apply clipping at the same granularity as the sequence-level reward, which avoids the mismatch between token-wise importance sampling and sequence-level rewards and stabilizes optimization.

##### Objective.

Let $(q,a)$ denote the dialog context and the ground-truth targets, and let $\{o_{i}\}_{i=1}^{G}$ be $G$ candidates sampled from the behavior policy $\pi_{\theta_{\text{old}}}(\cdot\mid q)$.
Each completion $o_{i}$ is scored by the reward in Sec. §[3.2](https://arxiv.org/html/2509.18847v2#S3.SS2), yielding $R_{i}\in[0,1]$.
We maximize a *sequence-level, asymmetrically clipped* objective and minimize its negative as the loss:

$$ $\mathcal{J}_{\text{RL}}(\theta)=\mathbb{E}_{(q,a)\sim\mathcal{D},\,\{o_{i}\}\sim\pi_{\theta_{\text{old}}}(\cdot\,|\,q)}\left[\frac{1}{G}\sum_{i=1}^{G}\min\!\Big(r_{i}(\theta)\,\hat{A}_{i},\;\mathrm{clip}\!\big(r_{i}(\theta),\,1-\varepsilon_{\text{low}},\,1+\varepsilon_{\text{high}}\big)\,\hat{A}_{i}\Big)\right],$ (26) $$

where $\mathrm{clip}(x,a,b)=\min\{b,\max\{a,x\}\}$ and typically $\varepsilon_{\text{high}}>\varepsilon_{\text{low}}$ (“clip-higher”).

##### Prompt-group dynamic filtering.

DAPO skips prompt groups whose candidates provide almost no learning signal (e.g., all-correct or all-wrong).
Concretely, define batch-normalized advantages and a group-level acceptance criterion:

$$ $\hat{A}_{i}=\frac{R_{i}-\mathrm{mean}\!\big(\{R_{j}\}_{j=1}^{G}\big)}{\mathrm{std}\!\big(\{R_{j}\}_{j=1}^{G}\big)},\qquad\mathcal{S}(q,a)=\Big\{\,i\in\{1,\ldots,G\}\;:\;|\hat{A}_{i}|>\tau_{\text{adv}}\Big\},$ (27) $$

and require sufficient reward dispersion within the group:

$$ $\mathrm{Var}\!\big(\{R_{i}\}_{i=1}^{G}\big)>\tau_{\text{var}}\quad\text{and}\quad 0<|\mathcal{S}(q,a)|<G.$ (28) $$

If ([28](https://arxiv.org/html/2509.18847v2#S3.E28)) fails, we drop the zero-information rollouts and (optionally) draw up to $K$ additional candidates from $\pi_{\theta_{\text{old}}}$, then re-apply the filter. Only indices in $\mathcal{S}(q,a)$ contribute to the expectation in ([26](https://arxiv.org/html/2509.18847v2#S3.E26)).

##### Sequence-level importance ratio.

For a completion $o_{i}=(o_{i,1},\ldots,o_{i,|o_{i}|})$, we use the geometric-mean, length-normalized importance ratio:

$$ $r_{i}(\theta)=\left(\prod_{t=1}^{|o_{i}|}\frac{\pi_{\theta}\!\left(o_{i,t}\mid q,\,o_{i,<t}\right)}{\pi_{\theta_{\text{old}}}\!\left(o_{i,t}\mid q,\,o_{i,<t}\right)}\right)^{\!1/|o_{i}|},$ (29) $$

and perform clipping at the same sequence granularity as the reward (see ([26](https://arxiv.org/html/2509.18847v2#S3.E26))), thereby avoiding token/sequence granularity mismatch.

## 4 Experiments

**Table 1: Comparison across dimensions (Base, Miss_Func, Miss_Param, Long_Context, Multi-turn Overall) on BFCL v3.**
| Models | Method | Base | Miss_Func | Miss_Param | Long_Context | Multi-turn Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Llama-3.1-8B-Instruct-FC | Origin | 5.0 | 6.5 | 4.5 | 4.5 | 5.12 |
| \cellcolorManBlueOurs | \cellcolorManBlue9.5 ($\uparrow$95%) | \cellcolorManBlue7.0 ($\uparrow$8%) | \cellcolorManBlue5.0 ($\uparrow$11%) | \cellcolorManBlue7.0 ($\uparrow$56%) | \cellcolorManBlue7.12 ($\uparrow$39%) |  |
| Qwen2.5-7B-Instruct-FC | Origin | 16.5 | 11.0 | 9.0 | 7.5 | 11.00 |
| \cellcolorManBlueOurs | \cellcolorManBlue22.0 ($\uparrow$33%) | \cellcolorManBlue13.0 ($\uparrow$18%) | \cellcolorManBlue13.5 ($\uparrow$50%) | \cellcolorManBlue11.0 ($\uparrow$47%) | \cellcolorManBlue14.88 ($\uparrow$35%) |  |
| Qwen3-4B-Instruct | Origin | 18.0 | 19.0 | 13.5 | 14.5 | 16.25 |
| \cellcolorManBlueOurs | \cellcolorManBlue25.0 ($\uparrow$39%) | \cellcolorManBlue19.5 ($\uparrow$3%) | \cellcolorManBlue17.0 ($\uparrow$26%) | \cellcolorManBlue21.5 ($\uparrow$48%) | \cellcolorManBlue20.75 ($\uparrow$28%) |  |

### 4.1 Experiment Settings

In this part, we will detail the experimental setup, including datasets, hyperparameters, base models, and evaluation metrics.

##### Datasets.

We conduct training on our self-constructed Tool-Reflection-Bench. After human supervision and post-editing, we retained approximately 5k samples in JSONL format to ensure compatibility with RL training under the Swift [34] framework.

##### Implementation Details.

We train models for 1 epoch (a total of 1,000 steps) on 5,000 training samples, using the reward function defined in Sec.[3.2](https://arxiv.org/html/2509.18847v2#S3.SS2). For each training instance, 4 completions were sampled to form a group. The training parameters were set as follows: temperature = 0.85, repetition penalty = 1.1, epsilon = 0.2, epsilon-high = 0.28, with a dynamic sampling strategy adopted.

##### Base Models.

To verify the generalizability of Tool-Reflection-Bench and our training methodology, we conducted experiments using Llama3.1-8B [35], Qwen2.5-7B-Instruct [36], and Qwen3-4B [37] as base models.

##### Evaluation Metrics.

We evaluated multi-turn tool-calling performance using the Berkeley Function Calling Leaderboard (BFCL) v3 [17], with evaluation dimensions covering multi-turn-base, multi-turn-long-context, multi-turn-miss-func, and multi-turn-miss-param, and the evaluation metric being accuracy. To assess the model’s repair capability when tool calls fail, we used Tool-Reflection-Bench, with the evaluation metric being repair rate, Repair@n denotes that for the same data instance, if at least one out of n trials succeeds, the metric is recorded as 1; otherwise, it is 0.

### 4.2 Experiment Results

#### 4.2.1 Result on BFCL v3

##### Comparison with base models.

We conduct performance evaluation on the multi-turn category of BFCL v3 to assess the benefits of enhancing the model’s self-reflection capability in multi-turn tool calling, the detailed results are showed in Table. [1](https://arxiv.org/html/2509.18847v2#S4.T1). Compared the results against the corresponding base models. The most striking lift appears on Llama-3.1-8B: Base rises from 5.0 to 9.5 (+95%) and Long_Context from 4.5 to 7.0 (+56%). Qwen2.5-7B shows the largest Miss_Param gain (9.0 $\to$ 13.5, +50%), evidencing stronger parameter repair. Qwen3-4B attains an amazing absolute Multi-turn Overall (20.75, +28%) with a sizable Long_Context improvement (+48%). In contrast, its Miss_Func gain is modest (19.0 $\to$ 19.5, +3%), indicating tool selection remains comparatively harder—consistent with our method’s emphasis on reflection-driven parameter correction and long-context recovery.

#### 4.2.2 Result on Tool-Reflection-Bench

As shown in Table. [2](https://arxiv.org/html/2509.18847v2#S4.T2), across open-source baselines, repair rates are low at one try (Repair@1 $\leq$ 9.6%) and only mildly improve with more tries.
Training with our method yields consistent gains for all bases:
Llama-3.1-8B-Instruct jumps from $0.7/5.1/6.8$ to 4.7/20.5/26.4 (Repair@1/3/5), a large improvement especially at higher $n$;
Qwen2.5-7B-Instruct improves from $2.4/6.1/8.0$ to 9.3/10.3/11.4;
Qwen3-4B-Instruct rises from $9.6/10.6/10.6$ to 14.9/18.5/19.5 (best Repair@1 among our models).
All finetuned models surpass the closed-source LongCat-Lite-8K-Chat across $n\!\in\!\{1,3,5\}$, indicating that our reflection-aware reward and RL objective substantially enhance repairability and yield more reliable multi-try recovery. It is also worth noting that when tool calls fail and require repair, our method achieves superior performance compared to closed-sourced models of the same scale such as LongCat-Lite-8K-Chat [38], GPT-4o-mini [39, 40], GPT-4.1-mini [41].

**Table 2: Experimental Results of Open-Source and Closed-Source Models on the Tool-Reflection-Bench Test Set.**
| Models | Repair@1 (%) | Repair@3 (%) | Repair@5 (%) |
| --- | --- | --- | --- |
| Close-Sourced Models |  |  |  |
| LongCat-Lite-8K-Chat | 2.3 | 3.4 | 4.9 |
| GPT-4o-mini | 6.1 | 8.7 | 9.0 |
| GPT-4.1-mini | 3.1 | 4.3 | 5.1 |
| Open-Sourced Models |  |  |  |
| Llama-3.1-8B-Instruct | 0.7 | 5.1 | 6.8 |
| Qwen2.5-7B-Instruct | 2.4 | 6.1 | 8.0 |
| Qwen3-4B-Instruct | 9.6 | 10.6 | 10.6 |
| Open-Sourced Models Trained on Our Method |  |  |  |
| \cellcolorManBlueLlama-3.1-8B-Instruct | \cellcolorManBlue4.7 | \cellcolorManBlue20.5 | \cellcolorManBlue26.4 |
| \cellcolorManBlueQwen2.5-7B-Instruct | \cellcolorManBlue9.3 | \cellcolorManBlue10.3 | \cellcolorManBlue11.4 |
| \cellcolorManBlueQwen3-4B-Instruct | \cellcolorManBlue14.9 | \cellcolorManBlue18.5 | \cellcolorManBlue19.5 |

## 5 Conclusion

This paper proposes a structured reflection method for handling tool call failures, transforming the “from error to repair” process into an explicit, controllable, and trainable action. Our approach overcomes the limitations of previous heuristic, feedback-based self-correction methods in terms of controllability and stability.
We further construct Tool-Reflection-Bench for both training and evaluation, and design a task-specific reward function tailored to the tool-calling scenario. In the reinforcement learning stage, we combine the strengths of DAPO and GSPO to enhance training effectiveness.
Experimental results show that the proposed method significantly improves multi-turn tool call accuracy on BFCL v3 as well as error repair performance on Tool-Reflection-Bench. Overall, our method and dataset effectively enhance the reliability of tool interactions and offer a new perspective on enabling agents to acquire new capabilities by learning from failure.

## 6 Prompt for Perturbation-based Disruptions

In this section, we provide simplified prompts for generating the four types of tool call perturbations, enabling the community to reproduce our setting. The full prompts and implementation code will be released upon the paper’s acceptance.

### 6.1 Prompt for Call-Order Swap

##### How to construct an error tool call example

##### How to generate a reflection

##### An Example

### 6.2 Prompt for Redundant Call

##### How to construct a redundant tool call example

##### How to generate a reflection

##### An Example

### 6.3 Prompt for Missing Call

##### How to construct a missing-call perturbation example

##### How to generate a reflection

##### An Example

### 6.4 Prompt for Argument Error

##### How to construct an argument–error perturbation example

##### How to generate a reflection

##### An Example

## 7 Training Data Case Study

In this section, we extract one complete sample from each of the four perturbation modes for analysis.

### 7.1 Case Study of Call-Order Swap

##### Setup.

The user requests end–to–end logistics for a 10–person business trip (NYC $\rightarrow$ MIA): search and *book* round–trip flights, search and *book* hotel rooms, and arrange airport–hotel ground transportation. The toolset exposes
search_flights, book_flight, search_hotels, book_hotel, and arrange_transportation, where the last function *depends* on confirmed itinerary details (arrival time and hotel).

##### Failure trace (order dependency violated).

The agent correctly issues two discovery calls, then prematurely attempts to arrange transportation *before* booking:

The tool returns a dependency error because dropoff_location cannot be finalized prior to book_hotel and pickup timing should be aligned to book_flight:

##### Model reflection and repair.

The agent emits a concise <reflect> that identifies the *order–dependency* (transport must follow booking), then repairs by booking first and only then arranging transport:

##### Why the reward favors the fix.

Under our reward in Sec. §[3.2](https://arxiv.org/html/2509.18847v2#S3.SS2):
(i) the reflection receives a high semantic score $s_{\text{ref}}$ for explicitly diagnosing the *order* constraint;
(ii) the repaired calls exactly match schema and parameters, yielding $s_{\text{call}}=1$ (order–*invariant* matching across the multiset of calls), while the *format/penalty factor* $F$ upweights the repaired trajectory because executability and dependency checks now pass;
(iii) the premature arrange_transportation in the failed rollout triggers penalties in $F$ (dependency/executability), so $R_{\text{core}}$ is suppressed and the dynamic sampling filter (Sec. §[3.3](https://arxiv.org/html/2509.18847v2#S3.SS3)) down–weights such uninformative rollouts.
Overall, the corrected plan yields a strictly larger $R_{\text{total}}$ than the erroneous one.

##### Summary.

This example shows that explicit reflection converts a latent, long–range constraint (“book before transport”) into a short textual diagnosis that the policy can optimize against. Our order–invariant call equality avoids over–penalizing benign output permutations, while dependency–aware format shaping in $F$ enforces *inter–call* preconditions. Empirically, such cases account for a substantial share of multi–turn gains, whereas single–turn metrics remain stable.

### 7.2 Case Study of Redundant Call

##### Setup and failure mode.

The user requests the list of services for all NYC banks operating for $>10$ years. The agent first queries the bank roster with
get_operational_banks, then correctly queries services for B001, but erroneously *repeats* the same service lookup for B001 instead of proceeding to unseen banks (B002--B004). This is a canonical *redundant call* error: wasted tool budget without adding new information.

##### Abbreviated trajectory.

##### Reflection and repair (ground truth).

The reflection diagnoses duplication and prescribes de-duplication; the corrected plan proceeds with the remaining, unseen IDs:

##### Why the model failed.

The failure arises from (i) insufficient state tracking over the set of already-seen entities (here, bank IDs), and (ii) weak inductive bias against issuing calls whose *marginal information gain* is near zero. In multi-turn settings, local myopic policies often re-issue the last successful pattern without cross-step deduplication.

##### How the reward steers recovery.

Our scoring treats call sets as order-invariant but schema-strict; redundant calls trigger the count-mismatch component in the format factor $F$ (penalizing $|C_{\text{calls}}|\neq|G_{\text{calls}}|$) while EqualCalls fails due to multiset mismatch. The reflection text receives a positive semantic score if it explicitly identifies the duplication and prescribes the missing IDs, encouraging concise, actionable self-correction. Together, the structure score $S$ and format factor $F$ downweight redundant completions and upweight the repaired sequence.

##### Summary.

This case shows that explicit reflection converts a silent efficiency bug into a supervised correction step: the agent (1) cites the duplicated identifier, (2) enumerates the remaining targets, and (3) completes them exactly once. Empirically, such reflection-shaped supervision reduces redundant tool usage and improves multi-turn success without harming single-turn accuracy.

### 7.3 Case Study of Missing Call

##### Setup.

The user asks to register *four* tax documents: (i) W-2 (ABC Corp), (ii) 1099-INT (First National Bank), (iii) property tax statement (county assessor), and (iv) Form 1098 (mortgage lender).
The tool schema exposes a single function add_tax_documents(name, value, category, priority) with name,value required.

##### Baseline failure ( missing calls ).

The baseline assistant emits only two <tool_call>s (W-2, 1099-INT) and then stops, yielding a 50% recall on required calls. Formally, let $G_{\text{calls}}$ contain the four intended calls and $C_{\text{calls}}$ the two produced calls. Then $|G_{\text{calls}}|=4$, $|C_{\text{calls}}|=2$, and the call–set equality test fails: $\mathrm{EqualCalls}(C_{\text{calls}},G_{\text{calls}})=0$. This is a typical *missing-call* error in multi-item requests: the model recognizes the pattern “one item $\rightarrow$ one call” but truncates the sequence, leaving later items unprocessed.

##### Structured reflection ( diagnosis ).

Our method takes the partially executed trajectory as *negative evidence* and the original request as *positive intent* and generates an explicit reflection:

> <reflect> “I missed 2 tool call(s). The user listed multiple items, and each item requires a separate call. I should enumerate all items and complete the remaining calls.” </reflect>

The reflection correctly localizes the failure (under-counting of required calls), quantifies the deficit (missed$=2$), and states the repair rule (enumerate all items $\Rightarrow$ one call per item).

##### Repairs ( corrective calls ).

Conditioned on the reflection, the agent appends the missing tool calls for the remaining items:

- •
name: Property tax statement; value: county assessor record; category: personal;
- •
name: Form 1098; value: mortgage interest statement; category: personal.

The assignments work$\rightarrow$W-2,1099-INT and personal$\rightarrow$property tax, 1098 are semantically consistent: the former are employment/bank income records; the latter are household liabilities/taxes.*(Any schema-compatible categorization would pass executability; ours also preserves natural semantics.)*

##### Why this matters.

This case highlights a frequent multi-turn brittleness: once the agent produces a plausible prefix of calls, it prematurely concludes and fails to cover all requested items. By making missingness an explicit, trainable concept, structured reflection converts a sparse binary signal (success/failure) into actionable supervision:

- 1.
Detection: Compare item cardinalities and arguments; compute $\mathbb{I}[|C_{\text{calls}}|<|G_{\text{calls}}|]$ and list uncovered entities.
- 2.
Diagnosis: Attribute the error to *enumeration/coverage* rather than formatting or parameters.
- 3.
Repair: Synthesize the exact missing calls with schema-valid arguments; preserve already-correct calls.

##### Summary.

Empirically, such instances improve the model’s *coverage discipline*: after training, we observe higher multi-item completion rates with negligible increase in redundant calls, indicating that the model learned “one-mentioned-item $\Rightarrow$ one-call” as a robust policy rather than overcalling.

### 7.4 Case Study of Argument Error

##### Setup.

The tool schema exposes multiple functions with *schema–strict* parameters (e.g.,
check_plant_water_level(plant_location:string),
start_watering(plant_location:string, duration:number),
start_trimming(hedge_location:string), …).
The user requests two primary actions in the backyard: (i) trim hedges and (ii) water all potted plants for about 10 minutes; afterwards ensure plants have enough water and dispose clippings.

##### Baseline failure ( argument error ).

The assistant issues

$$ <call>[{"name":"check_plant_water_level","arguments":{}}]</call> $$

omitting the *required* key plant_location. The tool returns a schema warning that the
arguments “did not match expected schema.” Under our reward, the call-level indicator
$s_{\text{call}}$ is 0 because the produced call fails schema equality (tool name matches, but the
argument map does not).

##### Structured reflection ( diagnosis ).

The reflection generated by our process states that the call “failed because it did not include the required
arguments needed by the function’s schema,” and prescribes: “ensure all necessary parameters are provided according
to the function’s documentation.” This localizes the error to parameter mis-specification (not tool
selection or ordering), and points to the concrete fix—satisfy the schema.

##### Repairs ( efficient plan consistent with the request ).

Given the user’s 10-minute target and the backyard scope, the corrected action set executes the two core operations
with schema-valid arguments:

- •
start_watering(plant_location="backyard", duration=10)
- •
start_trimming(hedge_location="backyard")

These can be dispatched in parallel (independent resources), achieving the requested time budget while ensuring
plants receive sufficient water and hedges are trimmed. This replaces the invalid pre-check with a direct,
time-bounded watering call that already satisfies the user’s constraint.

##### Why this matters.

Argument errors are common in tool use and typically yield *sparse* feedback (“schema mismatch”). By forcing the
model to (i) recognize the missing required field and (ii) restate the schema-conformant fix, the reflection step
converts a low-information error into actionable supervision. In our benchmark, such instances consistently improve:

- 1.
Schema adherence: higher exact-match rate on name/arguments.
- 2.
Planning under constraints: selection of parameterized calls (duration=10) aligned with user constraints instead of brittle pre-checks with empty arguments.
- 3.
Stability: fewer retries and warnings downstream because calls are executable on the first attempt.

##### Summary.

This case illustrates how reflection-guided repair turns a malformed <call> into a compact, correct, and time-efficient action plan.

## 8 Test Data Case Study

In this section, we present two representative test cases and their corresponding evaluation results as a case study, providing an intuitive demonstration of the effectiveness of our method and the model’s self-reflection capability for tool-call repair.

### 8.1 Case I

##### Setting.

The tool set exposes three functions:
getRecipes(max_time, meal_type), getSmoothieIngredients(max_time), and
findComplementaryRecipes(recipes, ingredients).
The user asks for *breakfast* recipes under 15 minutes and smoothie pairings under 5 minutes.

##### Failure mode (pre-training).

The baseline model immediately issues

which violates the function schema (both recipes and ingredients are required).
The tool returns a schema-warning. Under our reward, this yields $s_{\text{call}}=0$ and triggers format penalties $F<1$
due to missing required parameters.

##### Reflection-driven repair (post-training).

After RL on Tool-Reflection-Bench, the model first *reflects* that the failure arises from absent inputs,
then correctly decomposes the task into *produce inputs $\rightarrow$ compose*:

This satisfies the schema strictly (tool names and parameter maps match), making the call set correct and executable.

##### Why our method helps.

(i) Reward shaping: The instance accrues a hard penalty when required fields are absent; after repair,
$s_{\text{call}}$ flips to $1$ and $F\!\to\!1$, raising $R_{\text{core}}=S\cdot F$ substantially.
(ii) Sequence-level RL: The GSPO-style sequence-ratio with dual clipping aligns the optimization granularity with the sequence reward,
while DAPO-style dynamic filtering removes near-zero-advantage rollouts (all-wrong/all-correct), sharpening learning signals for this failure mode.

##### Takeaway.

Compared to the baseline that *jumps* to composition with empty inputs, the trained policy learns to (a) diagnose the schema error,
(b) *stage* prerequisite calls to produce the missing inputs, and (c) complete the composition with a schema-valid call set.
This precisely matches our benchmark’s objective: enable robust, multi-turn tool use via reflection and repair.

### 8.2 Case II

##### Setting.

Available tools include get_current_season(), get_seeds_by_season(season),
filter_seeds_by_availability(seeds), purchase_seeds(seeds, quantity), and
calculate_total_cost(purchased_seeds).
The user asks to *buy 10 packets of seasonal vegetable seeds* and *report the total cost*.

##### Failure mode (pre-training).

The baseline calls the aggregator first, with no inputs:

This violates the required schema (purchased_seeds missing), producing a warning and yielding
$s_{\text{call}}=0$ and a strong format penalty $F<1$ in our reward.

##### Reflection-driven repair (post-training).

After RL on Tool-Reflection-Bench, the model first *reflects* that costing requires purchased items, then
executes a staged pipeline to materialize prerequisites before aggregation:

Each call now matches tool name and parameter map exactly (schema-strict), so $s_{\text{call}}=1$ and $F\to 1$.

##### Why it works.

*Reward design* penalizes missing required fields and redundant structure, while granting full credit only when the
<call> set exactly matches the ground truth (schema-strict, order-invariant).
The *sequence-level RL objective* (GSPO-style ratio, dual clipping) aligns optimization with sequence rewards, and
*DAPO-style dynamic filtering* removes near-zero-advantage groups, concentrating updates on informative failures.
Together these guide the policy to diagnose schema errors, stage prerequisite calls, and complete the costing correctly.

##### Takeaway.

The trained policy no longer “guesses” totals from empty inputs. Instead, it *plans $\rightarrow$ acquires data
$\rightarrow$ purchases $\rightarrow$ aggregates*, a behavior precisely targeted by our reflection-and-repair rewards.

## 9 Theoretical Analysis

We analyze the main design choices of our reward in Sec. §[3.2](https://arxiv.org/html/2509.18847v2#S3.SS2) and the RL objective in Sec. §[3.3](https://arxiv.org/html/2509.18847v2#S3.SS3). Throughout, $\mathrm{Sim}\in[0,1]$, all
weights are nonnegative, presence masks are indicators, and $\mathrm{clip}(x,a,b)=\min\{b,\max\{a,x\}\}$.
*To avoid symbol overloading, we denote by $r_{\mathrm{fmt}}$ the format-penalty attenuation scalar used in Sec. §[3.2](https://arxiv.org/html/2509.18847v2#S3.SS2) (called $r$ there), and by $r_{\mathrm{seq}}$ the sequence-level importance ratio in Sec. §[3.3](https://arxiv.org/html/2509.18847v2#S3.SS3).*

### 9.1 Consistency of Presence-Mask Normalization

Recall

$$ $S\;=\;\frac{w_{\text{r}}I_{\text{r}}\,s_{\text{ref}}+w_{\text{c}}I_{\text{c}}\,s_{\text{call}}+w_{\text{f}}I_{\text{f}}\,s_{\text{final}}}{W_{\text{act}}},\qquad W_{\text{act}}\;=\;w_{\text{r}}I_{\text{r}}+w_{\text{c}}I_{\text{c}}+w_{\text{f}}I_{\text{f}},$ (30) $$

where $w_{\bullet}\!\geq 0$, $I_{\bullet}\!\in\{0,1\}$, at least one $I_{\bullet}=1$, $s_{\text{ref}},s_{\text{final}}\!\in[0,1]$, and $s_{\text{call}}\!\in\{0,1\}$.

##### Lemma 1 (Convex-combination form).

Let $\mathcal{A}=\{k\in\{\text{r,c,f}\}:I_{k}=1\}$ and define

$$ $\alpha_{k}\;=\;\frac{w_{k}}{\sum_{j\in\mathcal{A}}w_{j}}\quad\text{for }k\in\mathcal{A}.$ (31) $$

Then $\alpha_{k}\!\geq 0$, $\sum_{k\in\mathcal{A}}\alpha_{k}=1$, and

$$ $S\;=\;\sum_{k\in\mathcal{A}}\alpha_{k}\,s_{k}\quad\text{with }s_{\text{r}}=s_{\text{ref}},\ s_{\text{c}}=s_{\text{call}},\ s_{\text{f}}=s_{\text{final}}.$ (32) $$

*Proof.*
Since $I_{k}=1$ iff $k\in\mathcal{A}$, the numerator equals $\sum_{k\in\mathcal{A}}w_{k}s_{k}$ and
$W_{\text{act}}=\sum_{k\in\mathcal{A}}w_{k}>0$. Divide both to obtain the stated form.

##### Proposition 1 (Boundedness, stability, and scale invariance).

With $W_{\text{act}}>0$:

- *(a)*
$S\in[0,1]$ and, more sharply, $S\in[\min_{k\in\mathcal{A}}s_{k},\ \max_{k\in\mathcal{A}}s_{k}]$.
- *(b)*
If one only toggles *absent* parts (keeps $\mathcal{A}$ and $\{w_{k}\}_{k\in\mathcal{A}}$ unchanged), then $S$ is unchanged.
- *(c)*
For any $\lambda>0$, replacing each active weight by $\lambda w_{k}$ leaves $S$ unchanged.

*Proof.*
(a) By Lemma 1, $S$ is a convex combination of $\{s_{k}\}_{k\in\mathcal{A}}$; the interval bound follows from $s_{k}\in[0,1]$.
(b) Absent-part toggles do not change $\mathcal{A}$ nor the active $w_{k}$. (c) Common scaling cancels in numerator/denominator.

##### Corollary 1 (Continuity and Lipschitzness).

Fix $\mathcal{A}$ and $w_{k}$ for $k\in\mathcal{A}$. Then $S$ is an affine (hence continuous) map of $(s_{k})_{k\in\mathcal{A}}$ with

$$ $|S-S^{\prime}|\ \leq\ \sum_{k\in\mathcal{A}}\alpha_{k}\,|s_{k}-s^{\prime}_{k}|\ \leq\ \max_{k\in\mathcal{A}}|s_{k}-s^{\prime}_{k}|,$ (33) $$

so $S$ is $1$-Lipschitz w.r.t. the $\ell_{\infty}$-norm on the active scores.

##### Remark.

The definition via $\mathrm{clip}_{[0,1]}(\cdot)$ in ([25](https://arxiv.org/html/2509.18847v2#S3.E25)) is not needed for $S$ since the convex-combination form already implies $S\in[0,1]$.

### 9.2 Format Factor: Boundedness, Monotonicity, and EqualCalls Attenuation

Let

$$ $P_{\text{total}}\;=\;P_{\text{miss}}\;+\;\beta_{\text{extra}}\,P_{\text{extra}}\;+\;\gamma_{\text{count}}\,P_{\text{count}},\qquad\beta_{\text{extra}},\gamma_{\text{count}}\geq 0,\quad P_{\bullet}\geq 0,$ (34) $$

and define the attenuation scalar

$$ $r_{\mathrm{fmt}}\;=\;\begin{cases}r_{\text{reduce}},&\mathrm{EqualCalls}\!\big(C_{\text{calls}},\,G_{\text{calls}}\big),\\ 1,&\text{otherwise,}\end{cases}\qquad r_{\text{reduce}}\in(0,1].$ (35) $$

Consider

$$ $F\;=\;\mathrm{clip}_{[0,1]}\!\left(1-\lambda_{m}\,P_{\text{total}}\,r_{\mathrm{fmt}}\right),\qquad\lambda_{m}\geq 0.$ (36) $$

This is equivalent to the piecewise definition in ([23](https://arxiv.org/html/2509.18847v2#S3.E23)) since $P_{\text{miss}}{=}{P_{\text{extra}}}{=}{P_{\text{count}}}{=}0$ implies the inner value equals $1$.

##### Proposition 2 (Core properties of F F ).

- *(a)*
*Boundedness and regularity.*
$F\in[0,1]$ for all inputs; $F$ is continuous, piecewise affine in $(P_{\text{miss}},P_{\text{extra}},P_{\text{count}})$ and $1$-Lipschitz w.r.t. its scalar argument before clipping.
- *(b)*
*Monotonicity.*
For fixed $(\lambda_{m},r_{\mathrm{fmt}})$, $F$ is nonincreasing in $P_{\text{miss}},P_{\text{extra}},P_{\text{count}}$ and nonincreasing in $\lambda_{m}$ and in $r_{\mathrm{fmt}}$.
- *(c)*
*EqualCalls attenuation improves $F$.*
If $\mathrm{EqualCalls}$ holds so that $r_{\mathrm{fmt}}$ is replaced by $r_{\text{reduce}}\leq 1$, then $F$ weakly increases.
- *(d)*
*Plateau characterization.*
$F=1$ iff $\lambda_{m}P_{\text{total}}r_{\mathrm{fmt}}=0$ (e.g., $P_{\text{total}}=0$ or $\lambda_{m}=0$). If $\lambda_{m}>0$ and $r_{\mathrm{fmt}}>0$, then $F=0$ iff $P_{\text{total}}\geq 1/(\lambda_{m}r_{\mathrm{fmt}})$.

##### Corollary 2 (Sensitivity bound).

Off the plateaus ($1-\lambda_{m}P_{\text{total}}r_{\mathrm{fmt}}\in(0,1)$),

$$ $|\,\Delta F\,|\ \leq\ \lambda_{m}r_{\mathrm{fmt}}\,\bigl(\,|\Delta P_{\text{miss}}|+\beta_{\text{extra}}|\Delta P_{\text{extra}}|+\gamma_{\text{count}}|\Delta P_{\text{count}}|\,\bigr).$ (37) $$

### 9.3 Core Reward with Similarity Backoff: Signal and Variance Control

Let $R_{\text{core}}=S\cdot F$ as in ([24](https://arxiv.org/html/2509.18847v2#S3.E24)). The total reward uses a backoff when $R_{\text{core}}$ is very small:

$$ $R_{\text{total}}\;=\;\begin{cases}\mathrm{clip}_{[0,1]}(R_{\text{core}}),&R_{\text{core}}\geq\varepsilon,\[4.0pt] \mathrm{clip}_{[0,1]}\!\left(w_{\text{b}}\cdot\mathrm{Sim}\!\big(\mathrm{concat}(C),\,\mathrm{concat}(G)\big)\right),&\text{otherwise,}\end{cases}$ (38) $$

with $w_{\text{b}}\in(0,1]$ and $\varepsilon>0$. Note $R_{\text{core}}\in[0,1]$ already, hence clipping is redundant but harmless and keeps the two branches notationally symmetric.

We analyze its effect under a standard policy-gradient estimator
$\nabla_{\theta}\mathbb{E}[R_{\text{total}}]=\mathbb{E}\!\left[R_{\text{total}}\;\nabla_{\theta}\log\pi_{\theta}(\cdot)\right]$.

##### Lemma 3 (Uniform bounded variance of the reward).

Since $R_{\text{total}}\in[0,1]$, we have $\mathrm{Var}(R_{\text{total}})\leq\tfrac{1}{4}$ for any data distribution.

##### Lemma 4 (Non-degenerate gradient second moment on the backoff branch).

Let $\mathcal{B}=\{R_{\text{core}}<\varepsilon\}$ with $\mathbb{P}(\mathcal{B})=p>0$.
Assume $\mathrm{Sim}(\mathrm{concat}(C),\mathrm{concat}(G))\geq\sigma$ a.s. on $\mathcal{B}$ for some $\sigma>0$, and $\mathbb{E}\big[\|\nabla_{\theta}\log\pi_{\theta}(\cdot)\|^{2}\,\mathbf{1}_{\mathcal{B}}\big]>0$.
Then

$$ $\mathbb{E}\!\left[\ \big\|R_{\text{total}}\,\nabla_{\theta}\log\pi_{\theta}(\cdot)\big\|^{2}\ \right]\ \geq\ (w_{\text{b}}\sigma)^{2}\ \mathbb{E}\!\left[\ \big\|\nabla_{\theta}\log\pi_{\theta}(\cdot)\big\|^{2}\,\mathbf{1}_{\mathcal{B}}\ \right]\ >\ 0.$ (39) $$

*Implication.* When $R_{\text{core}}$ requently approaches 0 (in the early stages of training), the backoff branch ensures that the second moment of the gradients does not degenerate; combined with the variance upper bound from Lemma 3, this helps stabilize the optimization updates.

### 9.4 Sequence-Level Importance Sampling and Clipping

Let the sampled completion be $o=(o_{1},\dots,o_{T})$, and define the sequence-level (geometric-mean, length-normalized) ratio

$$ $r_{\mathrm{seq}}(\theta)=\left(\prod_{t=1}^{T}\frac{\pi_{\theta}\!\left(o_{t}\mid q,\,o_{<t}\right)}{\pi_{\theta_{\text{old}}}\!\left(o_{t}\mid q,\,o_{<t}\right)}\right)^{\!1/T}\;=\;\exp\!\Bigl(\tfrac{1}{T}\sum_{t=1}^{T}\log\rho_{t}\Bigr),\quad\rho_{t}=\frac{\pi_{\theta}(o_{t}\mid\cdot)}{\pi_{\theta_{\text{old}}}(o_{t}\mid\cdot)}.$ (40) $$

##### Proposition 3 (Length-independent ratio range under bounded log-ratios).

If $\log\rho_{t}\in[-L,L]$ a.s. for some $L>0$, then

$$ $e^{-L}\;\leq\;r_{\mathrm{seq}}(\theta)\;\leq\;e^{L}\qquad\text{for all }T\geq 1,$ (41) $$

whereas the unnormalized product ratio ranges in $[e^{-LT},e^{LT}]$.

*Implication.*
The geometric mean aligns the *ratio granularity* with the *sequence-level reward* in ([26](https://arxiv.org/html/2509.18847v2#S3.E26)), prevents exponential blow-up with $T$, and—together with dual clipping—reduces variance at the sequence level.

### 9.5 Dynamic Filtering of Prompt Groups (DAPO-style)

Let a prompt group produce $G$ rollouts $\{o_{i}\}_{i=1}^{G}$ with rewards $R_{i}\!\in[0,1]$ and batch $z$-scored
advantages

$$ $\hat{A}_{i}\;=\;\frac{R_{i}-\bar{R}}{s_{R}},\qquad\bar{R}=\tfrac{1}{G}\sum_{j=1}^{G}R_{j},\ \ s_{R}=\sqrt{\tfrac{1}{G}\sum_{j=1}^{G}(R_{j}-\bar{R})^{2}}\;>\;0.$ (42) $$

Define the *accepted* set

$$ $\mathcal{S}\;=\;\big\{\,i:\ |\hat{A}_{i}|>\tau_{\text{adv}}\,\big\},\qquad 0<|\mathcal{S}|<G,\qquad\mathrm{Var}(\{R_{i}\}_{i=1}^{G})>\tau_{\text{var}}>0.$ (43) $$

Write the per-sample (sequence-level, dual-clipped) PPO-like term as

$$ $\ell_{i}(\theta)\;=\;\min\Big\{\,r_{\mathrm{seq},i}(\theta)\,\hat{A}_{i},\;\mathrm{clip}\!\big(r_{\mathrm{seq},i}(\theta),1-\varepsilon_{\text{low}},1+\varepsilon_{\text{high}}\big)\,\hat{A}_{i}\,\Big\},$ (44) $$

and denote its gradient by $g_{i}(\theta)=\nabla_{\theta}\ell_{i}(\theta)$.
Assume the usual score-function bound and clipped ratio range:

$$ $\big\|\nabla_{\theta}\log\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})\big\|\;\leq\;B_{\pi},\quad r_{\mathrm{seq},i}(\theta)\in[\,1-\varepsilon_{\text{low}},\,1+\varepsilon_{\text{high}}\,].$ ( $\star$ ) $$

##### A uniform bound on per-rollout gradients.

Since $r_{\mathrm{seq},i}(\theta)$ is the geometric mean of token ratios,

$$ $\nabla_{\theta}r_{\mathrm{seq},i}(\theta)\;=\;r_{\mathrm{seq},i}(\theta)\,\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\nabla_{\theta}\log\pi_{\theta}(o_{i,t}\mid q,o_{i,<t}).$ (45) $$

Using $(\star)$ and that the clipped branch is constant on plateaus, there exists a finite
$C_{\psi}=(1+\varepsilon_{\text{high}})\,B_{\pi}$ such that

$$ $\|g_{i}(\theta)\|\;\leq\;C_{\psi}\,|\hat{A}_{i}|\qquad\text{for all }i,\ \theta.$ (46) $$

##### Lemma 5 (Zero or near-zero advantages).

(a) If $\hat{A}_{i}=0$, removing $o_{i}$ leaves the group-wise expected gradient unchanged.
(b) If $|\hat{A}_{i}|\leq\tau_{\text{adv}}$, then, for any $\theta$,

$$ $\big\|\ \mathbb{E}\,[\,g_{i}(\theta)\,]\ \big\|\;\leq\;C_{\psi}\,\tau_{\text{adv}},\qquad\mathbb{E}\!\left[\ \|g_{i}(\theta)\|^{2}\ \right]\;\leq\;C_{\psi}^{2}\,\tau_{\text{adv}}^{\,2}.$ (47) $$

*Proof.*
(a) The contribution is proportional to $\hat{A}_{i}$. (b) Apply ([46](https://arxiv.org/html/2509.18847v2#S9.E46)) and take expectations.

##### Bias and variance effects with 1 G \frac{1}{G} normalization.

Let the *filtered* group gradient be

$$ $\tilde{g}(\theta)\;=\;\frac{1}{G}\sum_{i\in\mathcal{S}}g_{i}(\theta),\qquad g(\theta)\;=\;\frac{1}{G}\sum_{i=1}^{G}g_{i}(\theta)\ \ \text{(unfiltered)}.$ (48) $$

Define the discarded set $\mathcal{S}^{c}=\{1,\dots,G\}\setminus\mathcal{S}$.
Then

$$ $\displaystyle\mathbb{E}[\tilde{g}(\theta)]-\mathbb{E}[g(\theta)]$ $\displaystyle=\;-\,\frac{1}{G}\sum_{i\in\mathcal{S}^{c}}\mathbb{E}[\,g_{i}(\theta)\,],$ (49) $\displaystyle\Big\|\ \mathbb{E}[\tilde{g}(\theta)]-\mathbb{E}[g(\theta)]\ \Big\|$ $\displaystyle\leq\;\frac{|\mathcal{S}^{c}|}{G}\,C_{\psi}\,\tau_{\text{adv}}\;\leq\;C_{\psi}\,\tau_{\text{adv}},$ $$

using Lemma 5(b). Moreover,

$$ $\mathbb{E}\!\left[\ \Big\|\frac{1}{G}\sum_{i\in\mathcal{S}^{c}}g_{i}(\theta)\Big\|^{2}\ \right]\;\leq\;\frac{|\mathcal{S}^{c}|}{G^{2}}\,C_{\psi}^{\,2}\,\tau_{\text{adv}}^{\,2},$ (50) $$

thus, discarding near-zero advantageous terms induces at most an $O(\tau_{\text{adv}}^{\,2})$-level change in the second moment; with respect to the $\frac{1}{G}$ normalization, it does not introduce any additional scaling bias.

##### Acceptance constraints avoid degeneracy.

The constraints $0<|\mathcal{S}|<G$ and $\mathrm{Var}(\{R_{i}\})>\tau_{\text{var}}$ ensure: (i) the batch standardization
$s_{R}$ is well-defined; (ii) both positive and negative (or at least non-identical) signals are present, preventing the trivial zero-gradient case where all $\hat{A}_{i}$ are identical. Consequently, $\tilde{g}(\theta)$ is a non-degenerate direction whenever useful learning signal exists.

##### Asymptotic unbiasedness with vanishing threshold.

If the threshold decays $\tau_{\text{adv}}^{(t)}\downarrow 0$ and the law of $\hat{A}_{i}$ has a continuous density at 0,
then the discard probability $\mathbb{P}(|\hat{A}_{i}|\leq\tau_{\text{adv}}^{(t)})\to 0$, and

$$ $\lim_{t\to\infty}\Big\|\ \mathbb{E}[\tilde{g}_{\,t}(\theta)]-\mathbb{E}[g(\theta)]\ \Big\|\;=\;0,$ (51) $$

i.e., the dynamic filtering becomes asymptotically unbiased while retaining finite-time variance-reduction benefits.

##### Summary.

Dynamic filtering deletes rollouts whose contributions are provably negligible (zero or $O(\tau_{\text{adv}})$),
thereby reducing variance and compute without altering the expected update in the limit $\tau_{\text{adv}}\!\to\!0$; using the same $1/G$ normalization as ([26](https://arxiv.org/html/2509.18847v2#S3.E26)) avoids spurious scaling bias.

### 9.6 Convergence Considerations for the Clipped Sequence-Level Objective

Consider the surrogate objective $\mathcal{J}_{\text{RL}}(\theta)$ in ([26](https://arxiv.org/html/2509.18847v2#S3.E26)), where rewards are bounded in $[0,1]$ and the sequence-level importance ratios are dual-clipped to $[\,1-\varepsilon_{\text{low}},\,1+\varepsilon_{\text{high}}\,]$.

##### Assumptions.

- (A1)
Bounded scores. There exists $B_{\pi}<\infty$ such that for all histories $(q,o_{<t})$ and tokens $o_{t}$,
$\big\|\nabla_{\theta}\log\pi_{\theta}(o_{t}\!\mid\!q,o_{<t})\big\|\leq B_{\pi}$.
- (A2)
Bounded rewards & finite clipping. For each rollout $o_{i}$, $R_{i}\in[0,1]$ and
$r_{\mathrm{seq},i}(\theta)\in[\,1-\varepsilon_{\text{low}},\,1+\varepsilon_{\text{high}}\,]$ with $0<\varepsilon_{\text{low}},\varepsilon_{\text{high}}<\infty$.
- (A3)
Non-degenerate batch dispersion. On accepted groups, $\mathrm{Var}(\{R_{i}\}_{i=1}^{G})\geq\tau_{\text{var}}>0$, so $\hat{A}_{i}=(R_{i}-\bar{R})/\mathrm{std}(R)$ are well-defined.
- (A4)
Vanishing filtering. $\tau_{\text{adv}}^{(t)}\downarrow 0$ and the law of $\hat{A}_{i}$ has a continuous density at 0, so $\mathbb{P}(|\hat{A}_{i}|\leq\tau_{\text{adv}}^{(t)})\to 0$.
- (A5)
Stepsizes. Robbins–Monro conditions: $\sum_{t}\eta_{t}=\infty$ and $\sum_{t}\eta_{t}^{2}<\infty$.

##### Lemma 6 (Bounds on per-sample gradients and second moments).

Let $o=(o_{1},\dots,o_{|o|})$ and $r_{\mathrm{seq}}(\theta)$ denote the (clipped) sequence ratio. Then

$$ $\nabla_{\theta}r_{\mathrm{seq}}(\theta)\;=\;r_{\mathrm{seq}}(\theta)\,\frac{1}{|o|}\sum_{t=1}^{|o|}\nabla_{\theta}\log\pi_{\theta}(o_{t}\!\mid\!q,o_{<t}),\qquad\big\|\nabla_{\theta}r_{\mathrm{seq}}(\theta)\big\|\leq(1+\varepsilon_{\text{high}})\,B_{\pi}.$ (52) $$

Moreover, the PPO-style term is piecewise smooth and its gradient magnitude is bounded by $C_{1}:=(1+\varepsilon_{\text{high}})B_{\pi}\,|\hat{A}|$; together with (A3), $|\hat{A}|\leq\frac{1}{\sqrt{\tau_{\text{var}}}}$ yields a uniform second-moment bound $\mathbb{E}\big[\|\nabla_{\theta}\ell_{i}(\theta)\|^{2}\big]\leq C_{2}<\infty$.

##### Lemma 7 (Asymptotic unbiasedness under vanishing filtering).

Let $g(\theta)$ denote the full (unfiltered) stochastic gradient and
$\tilde{g}_{\tau}(\theta)=\frac{1}{G}\sum_{i:\,|\hat{A}_{i}|>\tau}g_{i}(\theta)$ the filtered version with $\frac{1}{G}$ normalization.
Under (A4) and the bounded second moments above,

$$ $\lim_{\tau\downarrow 0}\ \big\|\,\mathbb{E}[\tilde{g}_{\tau}(\theta)]-\mathbb{E}[g(\theta)]\,\big\|\;=\;0\quad\text{for all }\theta.$ (53) $$

##### Theorem 1 (Convergence to a stationary point of the surrogate).

Suppose (A1)–(A5) hold. Then the iterates of stochastic gradient ascent on $\mathcal{J}_{\text{RL}}(\theta)$ with the dynamic filtering scheme converge almost surely to the set of stationary points of the surrogate objective.

*Proof sketch.*
By Lemma 6 and the reward boundedness (Lemma 3), the stochastic gradients have uniformly bounded second moments; the objective is bounded and piecewise smooth (kinks of measure zero). Lemma 7 guarantees that the bias due to filtering vanishes as $\tau_{\text{adv}}^{(t)}\to 0$. Therefore the noisy gradient process forms a Robbins–Monro stochastic approximation with asymptotically unbiased gradients and square-summable noise, yielding a.s. convergence to stationary points of $\mathcal{J}_{\text{RL}}$ (e.g., Kushner–Yin/Bottou).

##### Remarks.

(i) The min-with-clipping introduces bias w.r.t. the *true* off-policy objective, but ensures variance control and stability; the theorem concerns the surrogate we optimize. (ii) Sequence-level ratios and sequence-level clipping align the gradient scale with the sequence reward, avoiding token/sequence granularity mismatch and contributing to the boundedness needed above. (iii) In practice, we keep $\tau_{\text{var}}$ and the clip window fixed and decay $\tau_{\text{adv}}$, which satisfies the lemmas’ conditions and matches our training protocol.

## References

- [1]
MAOLIN WANG, YINGYI ZHANG, CUNYIN PENG, YICHENG CHEN, WEI ZHOU, JINJIE GU, CHENYI ZHUANG, RUOCHENG GUO, BOWEN YU, WANYU WANG, et al.
Function calling in large language models: Industrial practices, challenges, and future directions.
2025.
- [2]
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and J Wen.
Tool learning with large language models: A survey. corr abs/2405.17935(2024).
arXiv preprint arXiv:2405.17935, 2024.
- [3]
Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, et al.
Planning, creation, usage: Benchmarking llms for comprehensive tool utilization in real-world complex scenarios.
arXiv preprint arXiv:2401.17167, 2024.
- [4]
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.
Toolllm: Facilitating large language models to master 16000+ real-world apis.
arXiv preprint arXiv:2307.16789, 2023.
- [5]
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and J Wen.
Tool learning with large language models: A survey. corr abs/2405.17935(2024).
arXiv preprint arXiv:2405.17935, 2024.
- [6]
Ruizhe Zhong, Xingbo Du, Shixiong Kai, Zhentao Tang, Siyuan Xu, Hui-Ling Zhen, Jianye Hao, Qiang Xu, Mingxuan Yuan, and Junchi Yan.
Llm4eda: Emerging progress in large language models for electronic design automation.
arXiv preprint arXiv:2401.12224, 2023.
- [7]
Adrian Theuma and Ehsan Shareghi.
Equipping language models with tool use capability for tabular data analysis in finance.
arXiv preprint arXiv:2401.15328, 2024.
- [8]
Yilun Hao, Yongchao Chen, Yang Zhang, and Chuchu Fan.
Large language models can plan your travels rigorously with formal verification tools.
CoRR, 2024.
- [9]
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie.
Sft or rl? an early investigation into training r1-like reasoning large vision-language models.
arXiv preprint arXiv:2504.11468, 2025.
- [10]
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji.
Toolrl: Reward is all tool learning needs.
arXiv preprint arXiv:2504.13958, 2025.
- [11]
Barrett Martin Lattimer, Varun Gangal, Ryan McDonald, and Yi Yang.
Sparse rewards can self-train dialogue agents.
arXiv preprint arXiv:2409.04617, 2024.
- [12]
Xuefeng Li, Haoyang Zou, and Pengfei Liu.
Torl: Scaling tool-integrated rl.
arXiv preprint arXiv:2503.23383, 2025.
- [13]
Junjie Ye, Yilong Wu, Sixian Li, Yuming Yang, Tao Gui, Qi Zhang, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan, et al.
Tl-training: A task-feature-based framework for training large language models in tool use.
arXiv preprint arXiv:2412.15495, 2024.
- [14]
Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E Gonzalez, Bin Cui, and Shuicheng Yan.
Supercorrect: Supervising and correcting language models with error-driven insights.
arXiv preprint arXiv:2410.09008, 9, 2024.
- [15]
Mingyang Chen, Haoze Sun, Tianpeng Li, Fan Yang, Hao Liang, Keer Lu, Bin Cui, Wentao Zhang, Zenan Zhou, and Weipeng Chen.
Facilitating multi-turn function calling for llms via compositional instruction tuning.
arXiv preprint arXiv:2410.12952, 2024.
- [16]
Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al.
Toolace: Winning the points of llm function calling.
arXiv preprint arXiv:2409.00920, 2024.
- [17]
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez.
The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models.
In Forty-second International Conference on Machine Learning.
- [18]
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al.
xlam: A family of large action models to empower ai agent systems.
arXiv preprint arXiv:2409.03215, 2024.
- [19]
Bingguang Hao, Maolin Wang, Zengzhuang Xu, Cunyin Peng, Yicheng Chen, Xiangyu Zhao, Jinjie Gu, and Chenyi Zhuang.
Funreason: Enhancing large language models’ function calling via self-refinement multiscale loss and automated data refinement.
arXiv preprint arXiv:2505.20192, 2025.
- [20]
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li.
Api-bank: A comprehensive benchmark for tool-augmented llms.
arXiv preprint arXiv:2304.08244, 2023.
- [21]
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, et al.
Acebench: Who wins the match point in tool learning?
arXiv e-prints, pages arXiv–2501, 2025.
- [22]
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou.
Large language models cannot self-correct reasoning yet.
arXiv preprint arXiv:2310.01798, 2023.
- [23]
Fengyuan Liu, Nouar AlDahoul, Gregory Eady, Yasir Zaki, and Talal Rahwan.
Self-reflection makes large language models safer, less biased, and ideologically neutral.
arXiv preprint arXiv:2406.10400, 2024.
- [24]
Matthew Renze and Erhan Guven.
Self-reflection in llm agents: Effects on problem-solving performance.
arXiv preprint arXiv:2405.06682, 2024.
- [25]
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.
Self-refine: Iterative refinement with self-feedback.
Advances in Neural Information Processing Systems, 36:46534–46594, 2023.
- [26]
Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang.
Large language models can self-correct with key condition verification.
arXiv preprint arXiv:2405.14092, 2024.
- [27]
Juraj Vladika, Ihsan Soydemir, and Florian Matthes.
Correcting hallucinations in news summaries: Exploration of self-correcting llm methods with external knowledge.
arXiv preprint arXiv:2506.19607, 2025.
- [28]
Yuhua Jiang, Yuwen Xiong, Yufeng Yuan, Chao Xin, Wenyuan Xu, Yu Yue, Qianchuan Zhao, and Lin Yan.
Pag: Multi-turn reinforced llm self-correction with policy as generative verifier.
arXiv preprint arXiv:2506.10406, 2025.
- [29]
Xutong Zhao, Tengyu Xu, Xuewei Wang, Zhengxing Chen, Di Jin, Liang Tan, Zishun Yu, Zhuokai Zhao, Yun He, Sinong Wang, et al.
Boosting llm reasoning via spontaneous self-correction.
arXiv preprint arXiv:2506.06923, 2025.
- [30]
RI Saveliev and MV Dendiuk.
Self-reflective retrieval-augmented generation (self-rag) in analytical systems.
In Forestry Education and Science: Current Challenges and Development Prospects. International Science-Practical Conference, October 23-25, 2024, Lviv, Ukraine, 2024.
- [31]
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong.
Retool: Reinforcement learning for strategic tool use in llms.
arXiv preprint arXiv:2504.11536, 2025.
- [32]
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
arXiv preprint arXiv:2503.14476, 2025.
- [33]
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al.
Group sequence policy optimization.
arXiv preprint arXiv:2507.18071, 2025.
- [34]
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, et al.
Swift: a scalable lightweight infrastructure for fine-tuning.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 29733–29735, 2025.
- [35]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
arXiv e-prints, pages arXiv–2407, 2024.
- [36]
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al.
Qwen2. 5-coder technical report.
arXiv preprint arXiv:2409.12186, 2024.
- [37]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
arXiv preprint arXiv:2505.09388, 2025.
- [38]
Meituan LongCat Team, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al.
Longcat-flash technical report.
arXiv preprint arXiv:2509.01322, 2025.
- [39]
OpenAI.
Hello gpt-4o.
[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/), May 2024.
Accessed: 2025-09-25.
- [40]
OpenAI.
Gpt-4o system card, 2024.
Accessed: 2025-09-25.
- [41]
OpenAI.
Introducing gpt-4.1 in the api.
[https://openai.com/index/gpt-4-1/](https://openai.com/index/gpt-4-1/), April 2025.
Accessed: 2025-09-25.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: **Related work.** The paper seems at least loosely related to [Wang et al. (2024)](https://aclanthology.org/2024.acl-long.570/), [Xu et al. (2024)](https://arxiv.org/abs/2406.17465), and [Qu et al. (2025)](https://arxiv.org/abs/2410.08197), none of which are cited.  However, those papers actually try executions, and I think this paper doesn't.  Can you discuss the submitted work in the context of that line of prior work?

**Reflection generation.** The prompt at line 691 doesn't seem able to generate a good context-dependent reflection+correction, because it doesn't give any documentation for the tool API or any context about what this particular call was trying to achieve!  Can you explain?  

**Cost of human supervision.** Human supervision is used to correct (r,c) to (r*,c*) at (8).  Who provided the supervision?  How expensive was it?  Apparently (line 415) you produced at least 5k examples, though I'm not sure whether that means steps or multi-step trajectories.

**STaR workflow.** The point of this is to get a good reflection r*, since c* is known already.  But after obtaining a few human examples of this to illustrate the style (e.g., for few-shot prompting), perhaps one could use STaR ([Zelikman et al. 2022](https://arxiv.org/abs/2203.14465)) to have the LLM itself fill in r* (given c*) that is achievable by the model and tends to lead to c*?

**Imitation learning.**  Since supervised trajectories $m_0,m_1,\dots$ are available to the learner, why not try imitation learning?  For example, instead of generating errors by artificial corruption, why not train to correct the learner's *currently predicted* action $\tilde{m}\_i$ in the context $m_0,\dots,m_{i-1}$ to the supervised action $m\_i$?  

* Apologies that my notation here is a little sloppy.  The currently predicted action may be the result of preceding predictions that are self-corrected; here I am imagining that those preceding predictions and their reasoning steps fall between $m_{i-1}$ and $\tilde{m}\_i$.  If the currently predicted action is wrong, then we want to learn an additional self-correction action, otherwise an "accept" action.  

* This standard supervised imitation learning method is imperfect because of exposure bias: the distribution of contexts $m_0,\dots,m_{i-1}$ at test time is produced by the learned policy and probably won't match the training distribution ([Ross et al. 2010](https://arxiv.org/abs/1011.0686)).  But it may still be better than the method proposed in the submission.

* There are certainly *non*-sequential settings where a post-correction model is learned to correct system output to the reference output.  The machine translation community has a whole shared task series around "automatic post-editing" (APE).  Here, exposure bias is not an issue.

**Supervised training.** How important is it to train with the reward of section 3.2 rather than just log-loss (SFT to match the demonstrated (r*, c*) pair)?

**Task reward.** Do you only have rewards from (24)-(25), which ask about matching the demonstration?  (That's what line 419 implies.)  Would it make sense to also reward trajectories where the tool calls manage to achieve the goal in some other way?

**Reward design.** Can you give an example of (11)?  I'm not sure I understand what C\_calls and G\_calls look like.  Are they each a sequence of revisions to a call, with c\_final or g\_final being the final accepted revision in that sequence?  But if so, I have several confusions:
* should the ground truth have $j=0$ in almost all cases?
* such a sequence would be ordered, but you say "multiset," which implies that they are unordered (and presumably ordered arbitrarily in the prompt).
* why doesn't each $c_i$ or $g_j$ have its own reflection 
* if c\_ref and g\_ref are reflections, why isn't their main symbol $r$? (perhaps these should be called $r$ and $r*$ as in the previous section?)

**Learning how many times to revise.** Section 3.1.3 creates supervised examples of tool calls that need to be corrected, but don't you also need supervised examples of tool calls that _don't_ need correction (hence, 0 corrections)?  Also, does your system learn how to iteratively correct until convergence?  I'm not sure.

**Long-context recovery.** Mentioned at line 443: what do you mean?

**Table 1 results.** Table 1 evaluates on BFCL, but almost nothing is said about this setup.  What is BFCL?  How many turns are there per trajectory?  What metric is being reported in Table 1, and is it per-turn or per-trajectory?  Do the columns correspond to error categories that are provided by the BFCL dataset?  Are the improvements statistically significant?  Can you say more about whether corrections are applied at the right times (precision/recall), and whether they are the correct type of correction and whether they are successful?  If they are unsuccessful, is a further correction then applied?

**Table 2 results.** This is the test split of your own synthetic dataset, right?  Since your model was trained on the training split of the same dataset, you would expect it to do well, right?  So is this table just a sanity check that your training worked?
soundness: 3
strengths: Quality: The basic idea makes sense and the execution seems competent.  I agree that self-correction by LLMs should be trainable, and that supervised trajectories can be modified to provide examples that need to be corrected back to the supervised trajectory.

Significance: The improvements on the existing BFCL benchmark (Table 1) seem quite strong -- although I am not familiar with the benchmark and may not be interpreting the results appropriately.

The approach might not be too hard to adapt to training some other kinds of LLM self-correction (this paper just focuses on tool calls).

Clarity: I was mostly able to follow the paper, although I have some confusions (see questions below).  Detailed appendices are provided; the "case studies" in section A.3 are helpful for making the method more concrete.
summary: This paper aims to "transform self-correction [by LLMs] into a trainable and controllable capability" (line 186).
The basic idea is to convert self-correction into a supervised training problem x ↦ y.  Each synthetic training example starts with a correct y and synthesizes a plausible erroneous version x that must be corrected into y (lines 100-102, 112-113).

(Cf. a denoising autoencoder, where a supervised model is trained to undo a corruption step, or similarly BART and its ilk.  However, those are merely artificial tasks used for representation learning, whereas this paper is interested in the real task of correcting errors that actually arise during LLM message sequences.  Such real-task settings in prior work include grammatical error correction, e.g., [Felice 2016](https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-895.pdf), [Kiyono et al. 2019](https://aclanthology.org/D19-1119/), and code correction, e.g., [Gupta et al. 2017](https://ojs.aaai.org/index.php/AAAI/article/view/10742).)

This is really a general recipe for training self-correction, and a sensible idea if not a new one.
The difficulty is in figuring out how to create "plausible" corruptions.

This paper specifically focuses on tool calling.  Some points worth noting:

1. It uses a few simple hand-designed corruption operators (lines 220-227) based on manual analysis of error patterns (line 98).  

1. The corruptions are not *learned* to match the actual distribution of errors, as I had imagined at first. 

1. While the corrections are observed, the chains of thought needed to achieve them ("reflections") are provided by humans.

1. The method is trained by RL and not just by SFT, which introduces more complexity since a finicky loss function must be specified (section 3.2).
weaknesses: The general direction is not exactly new.  The detailed method is somewhat complicated, and is not compared to simpler alternatives.

The results are presented very briefly, and are not really discussed in the text (just recapitulated from the tables).  I don't have a clear understanding of the qualitative patterns of the results.

There may have been some missed opportunities.  See questions below.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. How does the learned reflection capability generalize to failures beyond the four specific perturbation types? Additionally, how do you expect the method's effectiveness to scale with much larger models (e.g., 70B+)?

2. Could you analysis the trade-off between the complexity of the RL pipeline and the performance gains, particularly when compared to simpler, less costly alternatives?

3. Have you analyzed instances of "reflection failure," where the model generates an incorrect reflection or its subsequent action is inconsistent with its own reflection?
soundness: 3
strengths: 1. This work introduces a novelty training mechanism about reflection, making contributions on models' tool-call abilities. And the results show great improvement for models.  

2. This work proposes 4 pertubations and constructs Tool-Reflection-Bench for evaluation.

3. The paper is well-written and easy to follow, with enough case studies.
summary: This paper introduces structured reflection in which transforming the "from error to repair" process into a trainable action, to solve the problem that models  often fail to recover from errors and tend to repeat the same mistakes. The authors combine objective functions of DAPO and GSPO in training and desigh a reward  mechanism for tool calling. The authors also introduce Tool-Reflection-Bench for evaluation. The results on BFCL v3 and Tool-Reflection-Bench show great improvements in multi-turn tool-call success rates and error recovery.
weaknesses: 1. Limited Discussion on generalization and model scale: Although  the paper discusses about performance on 4 perturbations and BFCL v3, it still lacks analysis about how well the learned capability generalizes to more failures in real world. Also, the models in experiments are 4B to 8B. The results remains unclear on much larger models.  

2. Comparison with other RL methods : The paper chooses a complex RL pipeline with DAPO+GSPO and increase the complexity and cost without doubt. However, the paper lacks a discussion on the cost-benefit trade-off with other traditional RL methods. 

3. Failure of reflection itself: This paper focuses on using reflection to fix tool-call errors. However, the failure of reflection itself also makes sense. I think authors should also pay attention to the error reflection or the inconsistency between reflection and execution in the process.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: How does your method perform on naturally-occurring tool-calling errors from real user interactions or other benchmarks, and what percentage of real-world failures fall into your four perturbation categories versus other failure modes not covered by your taxonomy?
soundness: 3
strengths: - The paper addresses a genuine problem in tool-augmented LLMs by making error recovery an explicit, trainable capability rather than relying on heuristic prompting, which represents a meaningful shift from prior self-correction approaches.

- The reward design is well-motivated with multiple components (format, tool name, parameters, semantic consistency) that provide granular signals for tool-calling scenarios.

- The experimental results show consistent improvements across multiple base models on both benchmarks, with particularly strong gains in repair rates that surpass closed-source models of similar scale.
summary: This paper proposes a structured reflection method to improve tool-calling capabilities of large language models through explicit error diagnosis and correction. The authors introduce Tool-Reflection-Bench, a benchmark constructed by systematically perturbing correct tool calls to create failure cases, and design a multi-dimensional reward mechanism for reinforcement learning. The method combines DAPO and GSPO objectives to train models to reflect on failed tool calls and generate corrected ones. Experiments on BFCL v3 and Tool-Reflection-Bench demonstrate improvements in multi-turn tool-call accuracy and error recovery rates.
weaknesses: - The benchmark construction methodology raises concerns about generalization, as all four perturbation types (call-order swap, redundant call, missing call, argument error) are synthetic and may not capture the full distribution of real-world tool-calling failures that occur naturally in deployed systems.

- The paper lacks critical ablation studies to validate design choices—it seems there is no analysis of individual reward components, no comparison between DAPO+GSPO versus alternatives (standard PPO, pure DAPO, pure GSPO), and no investigation of how much improvement comes from the reflection mechanism versus the enhanced reward structure.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 1
questions: See above weakness.
soundness: 1
strengths: 1. Enabling "reflection ability" of LLMs is an urgent and important topic for tool-use agents. This paper builds a new dataset, Tool-Reflection-Bench, and uses reinforcement learning (e.g., GRPO) to incentivize LLMs' reflection abilities.

2. The data synthesis approach in this work is well-organized and easy to follow. This simple yet efficient approach can automatically generate reflection-oriented training data (trajectories and prefixes).
summary: This paper adopts reinforcement learning (such as GRPO) to incentivize the reflective capabilities of large language models (LLMs). It first identifies four types of mistakes that frequently occur during LLMs' task-solving process. Based on these mistake types, the author proposes an automatic data synthesis approach to generate a prefix that includes predefined mistakes, forcing LLMs to reflect on the context and generate a new action. Experiments on several datasets validate the effectiveness of the proposed method.
weaknesses: 1. I am not clear about the training data. Does each example in the training data contain a prefix $x$, which is then fed into LLMs for continuous action generation until reaching the final answer?

2. What is the main difference between existing papers like [1], especially considering that "reflection" has been a common technique? It seems like the main contribution is enabling reflection using RL rather than supervised fine-tuning.
 
3. Experiments in this work lack the necessary comparison with existing baselines, such as ToolLLama and ToolACE, other advanced baselines in the BFCL leaderboard.

4. The main figure of this paper should be polished. The current version is not clear. I suggest that the author highlight the main methodology design and the differences with existing work.

5. I am a bit confused about the experiment setup. Does the author put all candidate tools into the context of LLMs, and then prompt the LLMs to call appropriate tools on demand? If so, what if the tool scale is large and the tool description exceeds the model context length?

---
### Reference

[1] Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning

[2] ToolACE: Winning the Points of LLM Function Calling