ResT: Reshaping Token-Level Policy Gradients for Tool-Use Large Language Models
Authors: Zihan Lin
Xiaohan Wang
Jie Cao
Jiajun Chai
Guojun Yin
Wei Lin
Ran He
Venue: ICLR 2026 Poster (2026)
Keywords: Token-level Policy Gradients Reshape;Tool-use Large Language Model;  Entropy-aware; Reinforcement Learning; Reasoning Model
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) transcend passive generation and act as goal-directed agents by invoking external tools. Reinforcement learning (RL) offers a principled framework for optimizing these emergent tool-use policies, yet the prevailing paradigm relies exclusively on sparse outcome rewards and lacks consideration of the particularity of tool-use tasks, inflating policy-gradient variance and resulting in inefficient training.
To better understand and address these challenges, we first establish a theoretical link between policy entropy and training stability of tool-use tasks, which reveals that structured, low-entropy tokens are primary determinants of rewards. Motivated by this insight, we propose Reshaped Token-level policy gradients (ResT) for tool-use tasks. ResT reshapes the policy gradient through entropy-informed token reweighting, progressively upweighting reasoning tokens as training proceeds. This scheme enables a smooth shift from structural correctness to semantic reasoning and stabilizes convergence in multi-turn tool-use tasks. Evaluation on BFCL and API-Bank shows that ResT outperforms other strong baselines, outperforming prior methods by up to 8.76%. When fine-tuned on a 4B base LLM, ResT further surpasses GPT-4o by 4.11% on single-turn tasks and 1.50% on multi-turn base tasks. Code is available at https://github.com/1229095296/ResT_Tool_use_LLM.

Published as a conference paper at ICLR 2026
REST: RESHAPING TOKEN-LEVEL POLICY GRADI- ENTS FOR TOOL-USE LARGE LANGUAGE MODELS
Zihan Lin 12,3+t, Xiaohan Wang", Jie Cao', " Chai²,
Guojun Yin Wei Lin?, Ran He
MMISS&NLPR, Institute of Automation, Chinese Academy of Sciences
2 Meituan
3 Schooo of Advanced Interdisciplinary Sciences, University Chinese Academy of Sciences
ABSTRACT
Large language models (LLMs) transcend passive generation and act as goal- directed agents by invoking external tools. Reinforcement learning (RL) offers a principled framework for optimizing these emergent tool-use policies, yet the prevailing paradigm relies exclusively on sparse outcome rewards and lacks con- sideration of the particularity of tool-use tasks, inflating volicy-gradien variance and resulting in inefficient training. To better understand and address these chal- lenges, we first establish theoretical link between policy entropy and training stability of tool-use tasks, which reveals that structured, low-entropy tokens are primary determinants of rewards. Motivated by this insight, propose Reshaped Token-level policy gradients (ResT) for tool-use tasks. ResT reshapes the policy gradient through entropy-informee token reweighting, progressively upweighting reasoning tokens as training proceeds. This scheme enables a smooth shift from structural correctness to semantic reasoning and stabilizes convergence in multi- turn tool-use tasks. Evaluation on BFCL and API-Bank shows that ResT out- performs other strong baselines, outperforming prior methods by up to 8.76%. When fine-tuned on a 4B base LLM, ResT further surpasses GPT-4o by 4.11% on single-turn tasks and 1.50% on multi-turn base tasks. Code available at fttps://githun wtt.ccccccmm..
ResTT Tool __su____il LLM
INTRODUCTION
The recent emergence of large language models (LLMss) has precipitated a paradigm shift from passive, data-driven models to proactive, goal-oriented autonomous agents with complex tool-use abilities (Cheng et al. 2024 Xi et 2025). Consequently, considerable research has focused on post-training LLMs to enhance their tool-use capabilities for domain-spechie
applications such as deep search and travel assistance. (Wang et al., 2025a Feng et al.) (025a). Post-training for agentic LLMs centers supervised fine-tuning (SFT) and Reinforcement learning (RL) with distinct trade- offs (Chu al., 2025 Wang al., 2025b). tool-use tasks, RI. demonstrates strong generalization and robust performance (Song al. 2025 Qian et al. 2025 Feng et al.) 2025b). Most approaches, however, assign a single outcome reward after multi-turn tool calls (Jin et al. 2025); despite gains in multi-hop scarch (Liu et al. 2025a), this paradigm introduces two challenges for general tool use: First, reward design is intrinsically noisy. Unlike multi-hop search with near-unique references, many practical tool-use tasks admit multiple valid outputs (e.g., recommendations). As result, outcome-only rewards induce high-variance gradients and provide weak incentives for reasoning, even when augmented with LLM-as-a-judge or learned reward models (Wang et al. 2025c Gu et al. (025). Second, multi-turn tool-use tasks are infffme-nnffficient: per-sample interactions, con- currency control, and variable horizons depress throughput and inflate infrastructure cost, making tool-use RL substantially heavier than SFT and single-turn RL (Team et al. (025a).
"Both authors contributed equally this research.
I Work was done during an internship at Meituan.
1 "oorrsponning authors: y ingu hngojunn22@me tuan com, ran. he@i cn

---

Published as a conference paper at ICLR 2026
The challenges motivate a central question: Could tool-use agent be trained by rewarding each turn with specific and law-variance rewards? Inspired by Deepseek-RI use of GRPO with sule-based rewards, compute rule-based matching scores between generated responses and ground truth. Recent work, such as ToolRI. (Qian et al., 2025). has validated this approach, achieving promising results. Specifically, these studies decompose multi-turn tool calls into series of single-turn tasks and establish rule-based matching rewards for efficient RL training. However, prior work often overlooks a crucial aspect: in tool-call settings, rule-based rewards are inherently concentrated on format tags, tool names, and key parameters, while reasoning and general- tokens contribute little during the early stages of training. Uniformly treating all tokens dilutes the RL signal and is particularly detrimental for approaches lacking token-level critics (e.g., GRPO).
Optimization Objective
Region T Tokenizer
Entropy Culculator
(0) En of mint Feladie. vie)) clip Soli Verl No. (o) Mac) Ta..d
+
He Pru log
Chain- of-hh-ugghhhs
Step N
Kay Parameters Uurput Format
Varis! ( 1.S. (1 (o0)
Weighted Policy Grudients Update
Curriculum Learning
Reweighting
ResT
User
Step
Assistant
Step Reward
cliylff + C2 v)
Hary
Context History
Weighted Policy Gradients Update
ResT
Step
Step
Step
Step
User
Assistant
Assistant
Assistant
Outcome Reward
ResT decomposes multi-turn tool-use tasks into single-turn tasks and further reshapes the policy gradient according to the average entropy in different regions, enabling dense and effective reward signals.
Figure 1:
To effectively and efficiently train tool-use agents, this paper proposes to Reshape Token-level pol- icy gradients based on single-turn RL training (ResT). As introduced in Figure ResT factorizes multi-turn tool use into single-turn steps and reshapes policy gradients with regron-level average entropy. This design emphasizes structured, low-entropy tokens such as tool names and parameters, yielding dense and effective reward signals. While RI. for tool-use agents should ideally progress from format compliance to parameter accuracy and ultimately complex reasoning, such curricula are seldom adopted. In ResT, this progression is guided by adaptive weighting across token regions, providing lightweight curriculum that further stabilizes training and achieves superior performance compared to strong baselines on established tool-use benchmarks. The contributions are summarized as follows:
1) We establish a theoretical and empirical link between policy entropy and training stability. Specif- ically, demonstrate that lower average entropy correlates with reduced variance in policy- gradient updates. This insight reveals that structured, low-entropy tokens such as tool names and parameters- are primary determinants of the resulting rewards. By uncovering this connection, we provide a principled explanation for why emphasizing these tokens during training leads to more stable optimization and improved performance on tool-use tasks.
2) We introduce an entropy- token-level reshaping mechanism that incorporates curriculum learning. Within this framework, tool-use and reasoning tokens are weighted by their region-level average entropy, which directly influences training stability and regulates how different token cat- egories contribute to reward signals. As training progresses and entropy decreases, the weights for reasoning tokens gradually increased. This currieulum-based approach stabilizes convergence and systematically enhances the model's reasoning capabilities in complex, multi-turn tool-calling scenarios.
3) Experimental results across multiple benchmark datasets show that our proposed method im- proves upon existing approaches by up to 8. 3.76%, outperforming other strong baselines. Notably, when fine-tuned on the Qwen3-4B-22 2507 base model, our method outperforms GPT-4o (Hurst et al.) 2024 by 1.50% multi-turn tool-use base tasks and 4.11% on single-turn tool-use tasks. Further-
2

---

Published as a conference paper at ICLR 2026
more, comprchensive ablation studies reveal that our curriculum based reshaping outperforms static reward weighting by up to 4.86%.
RELATED WORK
2.1 REINFORCEMENT LEARNING FOR LLM REASONING
Research on reinforcement learning (RL) for large language models (LLMs) progressed from early paradigms such as reinforcement learning from human feedback (RLHF) (Kaufmann et al. 2024)
and proximal policy optimization (PPO) (Schulman et al.] 2017) to class of methods based on preferences and policies, including direct preference optimization (DPO) (Rafiilov et al. 2023
SimPO (Meng et al.) 2024), and group relative policy optimization (GRPO) (Shao et al. 2024
Subsequent variants, such as dynamic sampling policy optimization (DAPO) (Yu et al. (025) and vallee-ugmeeted PPO (VAPO) (Yue et al. (025). emphasize training stability and sample efficiency. Taken together, these advances indicate that RL can substantively enhance LLM reasoning, moving beyond surface-level pattern matching toward competencies such as logical inference and strategic planning (DeepSeek-Al AI et al. 2025 Team et al., 2025b).
Beyond sequence-level feedback, a line of work studies token-level feedback and eredit assignment for fine-grained RLHF TLCR (Yoon et al.) (024) uses GPT-4 to generate token-level preference labels and trains discriminator to output continuous token rewards, thereby providing dense su- pervision at each position. TLDR (Fu et al.) constructs token-level reward models for multi-modal LLMs detect hallucinations and support self-correction, highlighting the interpretability and prac- ticality of token-wise rewards. Discriminative policy optimization with token-level Q-RMs (Chen al. (025) learns token-level Q-function-styy style reward models from preference data and applies them to RLHF scenarios such as mathematical reasoning. These methods explicitly construct token- level rewards or credit and then optimize policies under dense token- wise signals.
Curriculum RL provides another perspective on structuring learning signals. Classical work sur- veys curriculum design in RL and categorizes how task difficulty can be staged over time (Narvekar eet al. 2020). Opsimal-tanssoortbasse curriculum RL (Huang et al., (022) formalizes curricula as Wasserstein geodesics between a source and target task distribution, generating intermediate tasks via gradual domain adaptation and analyzing the benefits of multi-stage task transitions. More cently, E2H Reasoner (Parashar et al.] 2025) proposes an easy-to-hard reasoning curriculum for LLMs,, and theoretically and empirically studies how combining RL. with curricula improves sample efficiency on complex reasoning tasks. These approaches operate primarily at the task, environment, or prompt level by organizing training samples from easy hard.
2.2 TOOL-USE AGENTS
Tool use extends LLMs from passive text generators interactive systems capable of invoking external tools and APIs (Zhang et al. 2025| Hao et al.] (025). Existing approaches span prompt en-
gineering and supervised fine-tuning (SFT) (Chen et al. 2023) on curated corpora (Qin al. 2025 Hao et al., 2025 Zhang et al.) 2024), alongside specialized architectures and retrieval-augmenen
pipelines that select tools conditioned on the query (Nguyen et al. 2024). Recent agentic frame- works further support multi-step planning and autonomous tool orchestration (Team et al. (2025a).
However, most methods remain predominantly supervised, limiting exploration of diverse reasoning strategies and generalization to novel tool-use scenarios (Liu et al. 2025b).
To mitigate these limitations, RL formulations have been explored for tool-using agents. ToolRL (Qian et al., 2025) systematically compares SFT with GRPO (Shao et al., 2024): GIGPO (Feng al. 2025b) adapts GRPO to multi-step tool use via multi-trajectory sampling and two-level re- ward scheme; PVPO (Feng et al.] 2025c) further reduces rollout dependence and alleviates local optima through a reference-anchor mechanism and data pre-sampling; meanwhile, Agent Lightning (Luo et al., 2025) decouples training from execution and employs hierarchical RL to convert arbi- trary agent trajectories into transition-level learning signals. Despite these advances, practical RL methods still largely optimize at the sequence level, where coarse -raiined rewards induce high vari- ance in policy gradients, constraining reward granularity and impeding efficient credit assignment and optimization.
3

---

Published as a conference paper at ICLR 2026
ResT-4B GFT4o
03-mini
DeepSeak Gemini 2 Pro
DeepSeak-V3
SFT
TSFT
RSFT
GRPO
ResT
75
70.00%
70
60%
Overall Are (%)
65
64. 99%
60-
55
50
LST
MTB
45
Qwen3-1.78
Qwen3-4B
Qwen3-8B
Qwen3-14B
MTMP
Figure 2: Left: Overall accuracy on the API-Bank test set. Right: Overall accuracy on the BFCL test set. Axes are: NLST: None-Live Single Turn, MTLC: Multi-Tun Long Context, MTB: Multi- Turn Base, MTMF: Multi-Tur w/ Missing Functions, LST: Live Single Tur,, MTMP: Multi-Turn w/ Missing Parameters.
3 METHOD
In this section, we make two contributions. First, we develop a formal variance analysis that links token-level policy entropy the stability of policy-gradient updates, and show that reward mass in tool-use tasks concentrates on structured, low-entropy tokens such as tool names, arguments, and format tags. Second, we introduce ResT, an antropy-aware token-level gradient reshaping scheme with a lightweight curriculum that initially upweights these reward-critical structural tokens and gradually shifts emphasis toward higher-entropy reasoning tokens, thereby reducing gradient vari- ance, improving sample efficiency, and stabilizing training on complex tool-use tasks.
3.1 THEORETICAL MOTIVATION
Effective tool use often hinges on correctly generating just a handful of key tokens- -hose governing tool names, arguments, and output formats. Standard uniform reward schemes dilute these sparse yet decisive signals, inducing high-variance gradients and hampering stable convergence. To mitigate this, we develop theoretical framework for reducing volicy-gradient variance through entropy- aware token reweighting. Our analysis decomposes the gradient variance, establishes its connection to leken-level entropy, and derives an optimal reweighting scheme that simultaneously enhances training efficiency and promotes stable convergence on high-precision tool-use tasks.
Lemma (Policy Gradient Variance Decomposition). Let J(A) E R[[)) denote the expected return. For a single trajectory T; (Ji,orr-1). define the trajectory-level gradient
Vo log (V I (hrce) A.,
9:
where À, is advantage function. Given i...d. trajectories i from Tg. the mini-batch estimator is JJ (1/G)E& Di. Then for each coordinate I the variance satisfies
Var Var(V)(()
Var(g; (K)
If A, = u//a with = Err and o² = Var(r), then E[[g] = (o) and VJ is unbiased. Proof deferred to Appendix B.
Lemma establishes the variance scaling law under mini-batch sampling. Building on this decom- position, next characterize the second-order moment of g, and connect token-level entropy. Lemma (Second-Oe -Order Moment and Entropy Connection). For the single-trajectryy estimator the variance satisfies
T-1
Var(g: S E
A
Σ
JT
(3)
1=0
4

---

Published as a conference paper at ICLR 2026
where Cy. Pt is the logit-space pt softmax(z1) is the token distribution with 24 being the logits, and Jr = d2/ 09 is the Jacobian. Moreover, the per-step score satisfies
B||| =1 Σ Fh. S 1 1-e
(4)
where H₁ - Pt.v log Pi. u is the Shannon entropy, with indexing over the token vocabulary. Proof is deferred to Appendix B.2
Lemma 2 shows that variance fundamentally depends both squared advantage and token entropy. This result provides a natural handle for reweighting gradient contributions across timesteps. For a sequence of length T, we denote the reweighted policy gradient estimator as:
(rw) 9i
Σ t=
wi Vo log mo I, Vi,ee) A..
subject to
Σ 1=0
1 = T,
(5)
Theorem (Variance Upper Bound for Entropy-Aware Reweighting). The variance of the reweighted estimator y; (rw) is upper-bounded using Lemma
T
Vangg ((w)) S E[A
Σ Be (w,)²
(6)
where Be = E[l - c-H quantifies the variance contribution of step t by combining the Jacobian norm and the token entropy.
Building this upper bound, variance. This leads to a closed-form characterization of the weights.
seek the optimal reweighting scheme that minimizes the
Theorem 2 (Optimal Entropy-Aware Reweighting). Minimizing the variance bound in Theorem with respect to the weights Wh yields a flosed-form solution: (Proof is deferred to Appendix B.3
T
1
Eu=0
3
which effectively down-weights sequence positions with larger intrinsic variance contributions. Sub- stituting the optimal weights út into Theorem gives the minimized variance bound:
T2
Var(o, (()) I E[A E1 5
In practice, computing the exact value of 3, depends several variance terms that are difficult to estimate reliably at scale. Our variance decomposition shows that the leading structure of these terms governed by token-level entropy. We therefore use entropy-based surrogate approximate B.. which yields simple and scalable reweighting rules such as
W e Havy
or
O
Havg
Hagg denotes the average token-level entropy computed separately for each region in tool-use tra- jectories (format tags, tool names, key parameters, and chain-of-thoughh tokens). These region-wise entropy summaries preserve the qualitative inverse relationship between entropy and variance con- tribution implied by analysis, and therefore provide efficient, entropy-based surrogates for the optimal weights in Equation
These findings motivate principled variance-reduction scheme for policy-gradient training tool- use trajectories. In our method, open-ended tokens (c.g., chain-of- oftthoggghh which exhibit higher en- tropy (Besta al. 2025) (see Figure D.I are assigned lower weights than structured, low- low-entropy control tokens. This entropy-aware reweighting tightens variance bounds and improves sample effi- ciency while preserving the unbiasedness of the policy-gradient estimator.
5

---

Published as a conference paper at ICLR 2026
3.2
ALGORITHM DESIGN
3.2.1
REWARD SCORE
Rule-based reward mechanisms have been empirically validated to achieve robust performance across diverse settings. line with prior work (Qian et al. 2025] Jin al. 2025 Liao et al, 2024 Xie et al. 2025), our training objective incorporates components for both output format and execution correctness. Accordingly, we formulate the total reward as a weighted sum of a format matching score and tool-calling correctness score. Format score of the i-th sample is determined by exact-match:
1, if all required fields are complete in order otherwise
forrrat
(10)
0,
The tool-calling correctness score is based on three components: tool name, parameter names, and parameter values. We measure name accuracy using the Jaccard similarity coefficient, defined as the intersection over union of the predicted P and ground-truth G sets:
NG Npl NGU E [0, I],
Po Ppl Σ GLEG
E IG]],
(11)
name
Para
P Ppl
where NG and Np are the of ground-truth and predicted tool names, while PG and Pp the cor- responding sets of parameter names for a given tool. Due to the precision requirements of parameter values, we use an exact-match criterion for their evaluation:
Tvalue
Σ
P E
0, Σ |vGG.))
(12)
GGEG
where v(.) denotes the function that maps graph G, to its corresponding parameter value set. The normalized correctness score then computed as:
I' Tname + Tpara
Tvaluc
St,000
(13)
G Ea,ea ((G.)
The final dynamic scaling reward Rinal is then calculated weighted sum:
Ritiiall = Stace (1 - 2) B
- D),
(14)
where / denotes the relative importance and i E (0, 1) denotes the training progress.
3.2.2 RESHAPING TOKEN-LEVEL POLICY GRADIENTS
ResT enables a more stable and effective use of rule-based rewards during model fine-tuning. Given prompt containing user requirements and context, policy me generates G multi-turn response sequences composed of plain text and tool invocation segments. For the i-th sample in a group G, the advantage function A, are then calculated:
R final à EG-1 R Ry
A
(15)
(Ri,inal 1 LG-1 Ri,fnaa)²
where 8 > 0 is a numerical stability constant.
We first tokenize the response and then partition it into four weight fm (format tags), Wt,name (tool names), Wtrpara (tool parameters), and Wr.0kk (chain-of-thoughn) Reweighting factor w; initialized according to the entropy- reweighting equation
Our curriculum dynamically adjusts weights based on progress v, enforcing transition from mas- tering structural syntax refining semantic strategy. While tool names retain constant high impor- tance, we initially prioritize format tokens to ensure validity, subsequently shifting focus to semantic accuracy by annealing format weights while increasing parameter and thain-of-thougee weights:
(ee(( max Wiinn font ... vV), Wr,para (v) min(Wmax, Wara Op v), (16)

---

Published as a conference paper at ICLR 2026
where [Wmins Wmax] denotes the clipping interval and denotes the magnitude of the update. To reduce policy-gradeee gradient variance, encourage step-by-step reasoning, and improve parameter accuracy, the weights for chain-of-thought and tool parameter tokens are synchronized:
Ür (v) (nmax thith + v).
(17)
We obtain the normalized reweighting factor by its mean over the sequence:
Wt
t max (Wmins min (W,, Wmax)), 0
W
Wr
(18)
This normalization mechanism redistributes learning signals across tokens within each sample, gov- erned by their relative policy entropy. By incorporating this reweighting factor with the advantage function, we derive the final optimization objective:
CRE (8)
1 3ΣΣ
wi min (rar4,
(19)
T
t=
To(3) F, V1,
where Ti. Thaa (3i.t x, Vi,ce))
and limits the magnitude of gradient updates. Our approach
omits the KL loss term from the GRPO objective, which typically penalizes excessive policy devia- tions. In its place, ResT intricately modulates the exploration/eeppoiiaaion trade-off via a combina- tion of entropy-aware reweighting, clipping mechanisms, and curriculum learning. We validate this design choice through ablation studies in Figure A and Table A..I which show that these compo- nents together yield stable and performant training dynamics even in the absence of an explicit KL penalty. The objective guides the policy toward structurally consistent and semantically accurate tool calls, while gradient reshaping alleviates variance in policy gradients, resulting in more stable and efficient alignment with task requirements. The complete algorithmic procedure is provided in Appendix C
EXPERIMENTS
4.1 TRAINING DETAILS
We conducted comprehensive experiments using the verl 0.5.0 framework. We curate mixed cor- pus to facilitate robust tool learning in RL: ToolACE (Liu et al.] (0255)) targets the invoke-vs-answer decision in multi-step interactions; Hammer (masked) (Lin et al. (025) randomizes tool and argu- ment names to enforce description description-groundee grounded generalization; and XLAM (Zhang et al.) 2024) sup- plies compositional tasks that require one or multiple tool calls per turn. Together, these sources supervise invocation timing, robust grounding, and compositional planning (Qian et al. 2025).
We convert multi-un tultii-turmm dialogue samples into single-step step instances provide denser and more fine- grained reward signals. Concretely, dialogue with K interaction steps is decomposed into K sub-instances, each conditioned on the full preceding conversation as input and supervised with the model's 's current action (e.g., tool invocation or response) as the target. This decomposition strategy, adopted from the SWiRL framework (Goldie et al., 2025), substantially increases the density of training signals. Consequently, the model receives process-level supervision at each step, rather than only for the final outcome. This step-wise supervision fosters more stable learning of multi- step reasoning and tool use.
4.2 MAIN RESULTS
We evaluate six baselines spanning supervised and reinforcement-leamiig regimes: Base Model: instruction-tuned without task-specific fine-tuning, SFT: supervised fine-tuning on the task training set. TSFT: SFT with up-weighted loss (Hucrta-Enoohian & Ko 2024) tool-call tokens to pri- oritize invocation accuracy. RSFT: SET with up- ue-weighted loss on reasoning tokens to strengthen intermediate reasoning. GRPO: A critic-free reinforcement learning algorithm that leverages group- relative rewards to efficiently optimize uoel-use policies. SFT+GRPO: initialize from an SET check- point, then apply GRPO isolate RL's marginal gains. Dr.GRPO (Liu et al. 2025c) removes
7

---

Published as a conference paper at ICLR 2026
Table 1: BFCL multi-turn results (updated June 14, 2025). Metrics computed with official
scripts. TSFT scales loss on tool-call tokens. RSFT scales loss on reasoning tokens. Best in bold. Model
Overall Ace
Base
Miss Func
Miss Param
Long Context
Qwen3-1.7B (Base)
8,62%
12.00%
5,50%
9,50%
7.50%
Qwen3-1.7B (SFT)
14.25%
17.50%
14.50%
14.50%
10.50%
Qwen3-1.7B (TSFT)
13.50%
18.00%
14.50%
11.50%
10.00%
Qwen3-1.7B (RSFT)
15.00%
17.00%
16.50%
14.50%
12.00%
Qwen3-1.7B (GRPO)
14.50%
17.00%
15.50%
12.50%
13.00%
Qwen3-1.7B (SFT+GRPO)
14.88%
20.50%
18.00%
9.00%
12.00%
Qwen3-1.7B (Dr.GRPO)
15.25%
16.50 %
15.00%
17.00%
12.50%
Qwen3-1.7B (Ours, ResT)
16.00%
20.00%
16.00%
15.00%
13.00%
Qwen3-4B-2507 (Base)
48.00%
57.00%
53.00%
33.50%
48.50%
Qwcn3-4B-2507 (SFT)
43.00%
52.50%
46.50%
33.00%
40.00%
Qwen3-4B-2507 (TSFT)
48,25%
58.00%
52.00%
32.50%
50.50%
Qwen3-4B--207 (RSFT)
41.25%
52.00%
47.50%
31.50%
34.00%
Qwen3-4B-2507 (GRPO)
41.62%
48.50%
45.50%
27.00%
45.50%
Qwen3-4B-2507 (SFT+GRPO)
40.38%
46.00%
42.00%
29.50%
44.00%
Qwen3-4B-2507 (DrGRRPO
48.62%
58.00 %
54.00%
30.50%
52.00%
Qwen3-4B-2507 (Ours. ResT)
50.38%
62.50%
56.00%
33.00%
50.00%
Qwen3-8B (Base)
33.00%
38.50%
32.00%
31.50%
30.00%
Owen3-8B (SFT)
28.75%
34.50%
33.50%
22.00%
25.00%
Qwen3-8B (TSFT)
29.00%
32.50%
34.00%
24.00%
25.50%
Qwen3-8B (RSFT)
25.12%
33.00%
29.50%
19.00%
19.00%
Qwen3-8B (GRPO)
36.00%
42.00%
42.00%
28.50%
31.50%
Qwen3-8B (SFT+GRPO)
32.38%
42.50%
35.00%
27.50%
24.50%
Qwen3-8B GDPO)
38.12%
39.50 %
44.50%
36.50%
32.00%
Qwen3-8B (Ours, ResT)
40.13%
50.50%
45.00%
32.00%
33.00%
Qwen3-14B (Base)
36.62%
48.00%
33.00%
33.50%
32.00%
Qwen3-14B (SFT)
28.12%
33.50%
30.00%
23.00%
26.00%
Qwen3-14B (TSFT)
27.62%
34.50%
34.00%
17.50%
24.50%
Qwen3-14B (RSFT)
27.00%
33.50%
27.50%
20.00%
27.00%
Qwen3-14B (GRPO)
38.88%
50.00%
43.50%
26.50%
35.50%
Qwen3-14B (SFT+GRPO)
34.12%
45.00%
37.50%
22.00%
32.00%
Qwen3-14B (Dr.GRPO)
37.00%
46.00 %
33.50%
41.00%
27.50%
Qwen3-14B (Ours, ResT)
44.25%
57.00%
45.00%
36.50%
38.50%
GPT-46-2024-11-22
50.00%
61.00%
45.50%
35.50%
58.00%
GPT-4-5-Pteveew22255.2222
45.38%
58.00%
37.50%
43.00%
43.00%
GPT-4.1-2025-04-14
40.50%
48.00%
34.00%
35.00%
45.00%
03-mini-2025-01-31
28.75%
32.00%
26.00%
23.00%
34.00%
Grok-3-beta
45.50%
54.00%
39.00%
42.00%
47.00%
DeepSeck V3-0324
DecpScck-V3-0324
29.87%
41.00%
21.00%
23.00%
34.50%
SeepSeek-R1-0522
38.88%
49.50%
31.50%
32.00%
42.50%
Geminn-2.5-Pro-Peevie-----
34.62%
39.50%
29.50%
31.50%
38.00%
standard-deviation normalization in the group-relative advantage and instead uses a mean-centered estimator equivalent up a constant rescaling RL.0O-tyle advantage, reducing bias toward low-variance prompts and length- distortions while preserving the GRPO update structure.
Benchmarks and Setup. We rigorously evaluate our method on two authoritative tool-use bench- marks. The Berkeley Function Calling Leaderboard (BFCL) (Patil et al. 2023) provides a standard- ized assessment of multi-step function calling and related capabilities, while API-Bank (Li et al. 2023 is a comprehensive three-tiered suite for evaluating tool invocation in multi-turn dialogues. To isolate the contribution of our algorithmic design, we fine-tune several open-source models and compare them against strong, Domain-specializen baselines under matched training conditions.
Performance Results. As reported Figure Table D and Table 2 fine-tuning the Qwen3 model family with ResT establishes new state-of-the-art on both benchmarks. The method yields sig- nificant performance gains over the GRPO baseline, with improvements of up to 8.76% on BFCL and 3.02% on API-Bank under an identical data budget. Notably, the Qwen3-4B-250 model fine- tuned with ResT even surpasses GPT-4o by 0.38% on multi-turn overall tool-use use tasks, especially by 1.50% multi-tun base tasks. On the single-turn tool-use tasks, leads by 4.11%. These behav- iors indicate emergent forms of proactivity and metacognition, which enhance efficiency, mitigate hallucinations, and signal the emergence of foundational agentic intelligence.
8

---

Published as a conference paper at ICLR 2026
Table 2: API-Bank Test Results. The results presented correspond the highest score achieved by each method with its optimal hyperparameter settings. The evaluation dataset consists of 399 samples for Level 67 for Level 2, and 131 for Level 3. Best in bold.
Model
Overall Acc
Level 1 Acc
Level Acc
Level 3 Acc
Qwen3-1.7B (Base)
47.57%
53.38%
28.36%
39.69%
Qwen3-1.7B (SFT)
58.96%
66.92%
52.24%
38.17%
Qwen3-1.7B (TSFT)
54.77%
61.4%
43.28%
40.46%
Qwen3-1.7B (RSFT)
61.81% %
67.92%
59.7%
44.27%
Qwen3-1.7B (GRPO)
63.65%
70.68%
61.19%
41.22%
Qwen3-1.7B (SFT+GRPO)
59.46%
65.91%
58.21%
40.46%
Qwen3-1.7B (DrGRPO)
57.55%
84.06%
73.66%
15.25%
Qwen3-1.7B (Ours, ResT)
64.99%
70.18%
65.67%
48.85%
Qwen3-4B-250 (Base)
66.33%
72.68%
64.18%
48.09%
Qwen3-4B-2500 (SFT)
60.30%
67.92%
59.70%
37.40%
Qwen-44B-2507 (TSFT)
66.83%
72.93%
64.18%
49.62%
Qwen3-4B-2507 (RSFT)
64.49%
69.92%
59.7%
50.38%
Qwen3-4B-2507 (GRPO)
65.33%
71.93%
62.69%
46.56%
Qwen3-4B-2507 (SFT+GRPO)
68.51%
74.44%
65.67%
51.91%
Qwen3-4B-250 (DrGRPPO
65.66%
71.68%
61.19%
49.62%
Qwen3-4B-2500 (Ours, ResT)
68.68% %
73.68%
64.18%
55.73%
Qwen3-8B (Base)
63.32%
70.68%
53.73%
45.80%
Wwe3-8BB (SFT)
60.64%
69.17%
56.72%
36.64%
Qwen-88B (TSFT)
59.80%
66.42%
50.75%
44.27%
Qwen3-8B (RSFT)
60.47%
67.42%
53.73%
42.75%
Qwen3-8B (GRPO)
66.50%
75.44%
64.18%
40.46%
Qwen3-8B (SFT+GRPO)
63.82%
72.18%
52.22%
42.75%
Qwen3-8B (Dr.GRPO)
68. 15%
88.00%
79.25%
38.12%
Qwen3-8B (Ours, ResT)
70.69 %
74.94%
65.67%
60.31%
Qwe3-11BB (Base)
65.49%
71.68%
56.72%
51.15%
Qwen3-14B (SFT)
67.34%
76.44%
62.69%
41.98%
Qwen3-14B (TSFT)
64.32%
68.67%
59.70%
53.44%
Qwen3-14B (RSFT)
64.99%
70.68%
55.22%
52.67%
Qwen3-14B (GRPO)
66.33%
73.18%
59.70%
48.85%
Qwen3-14B (SFT+GRPO)
67.34%
73.43%
58.21%
53.44%
Qwen3-14B (Dr.GRPO)
69.01% %
74.44%
61.19%
56.49%
Qwen3-14B (Ours, ResT)
69.35% %
73.43%
62.69%
60.31%
Table 3: API-Bank Test Results for Relaxations. No dynamic reward denotes no dynamic scaling reward value. No gradicnts for CoT denotes no gradient update for chain-of- of-thought.. No curriculum learning assigns constant weight each segment. Best bold.
Model
Overall Acc
Level Acc
Level 2 Ace
Level Acc
Qwen3- 1.0m3-11.7B I.7B (No dynamic reward)
64.15%
69.92%
58.21%
49.62%
Qwen3-1.7B, (No gradients for CoT)
62.98%
68.92%
62.69%
45.04%
Qwen3- 1.7B (No curriculum learning)
61.14%
67.92%
58.21%
41.98%
Qwen3-1.77 (Ours, ResT)
64.99 %
70.18%
65.67%
48.85%
Qwen3-4B -Wen3-4B-2507 (No dynamic reward)
64.15%
71.18%
64.18%
42.75%
Qwen3-4B 2507 (No gradients for CoT)
64.49%
71.18%
59.70%
46.56%
Qwen3-4B-255 2507 (No curriculum learning)
67. 07.84%
73.18%
64.18%
53.44%
Qwen3-4B
2507 ---05--
(Ours, ResT)
68.68 %
73.68%
64.18%
55.73%
Qwen3-8B (No dynamic reward)
64.15%
71.68%
64.18%
41.22%
Qwen3-8B (No gradients for CoT)
66.33%
74.44%
65.67%
41.98%
Qwen3-8B (No curriculum learning)
65.83%
73.68%
62.69%
43.51
Qwen3-8B (Ours, ResT)
70.69 %
74.94%
65.67%
60.31%
4.3 ABLATION STUDY
Ablations across base models (Table 3) show that each ResT component is necessary for peak per- formance. Disabling dynamic rewards, CoT policy gradicnts, or the curriculum consistently reduces accuracy. On Qwen3-8B, removing dynamic rewards lowers accuracy 6.54%, and dropping CoT

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study on ResT initialization strategies and curriculum alignment on the
BFCL benchmark with Qwen3-1.7B. SFT warm-starting combined with Tuned Curriculum (Tuned Curr.) achieves the best performance, demonstrating a synergistic effect. Average performance is calculated using the official scripts.
Model
Overall Acc
Base
Miss Fune
Miss Param
Long Context
Base
8.62%
12.00%
5.50%
9.50%
7.50%
SFT
14.25%
17.50%
14.50%
14.50%
10.50%
ResT(Coll Start)
15.75%
22.50%
16.50%
13.50%
10.50%
(erriinal Cum:)
15.25%
19.50%
16.50%
13.00%
12.00%
SFT+ResT(Tuned Curr.)
17.62% %
23.50%
19.00%
14.00%
14.00%
policy gradients and the curriculum yields declines of 4.36% and 4.86%, respectively. These results indicate complementary contributions and explain ResTT's robustness.
The ResT-trained model demonstrates robust generalization beyond its multi-turn training focus. As shown in Table D.I] method performs comparably the baseline single-turn tasks and exhibits superior performance on live (real-time updated) tasks, highlighting its ability to generalize across different interaction regimes and execution environments. Considering that SFT cold-start alters the entropy distribution, we conduct controlled comparison on the 4B model using both the original weights and variant with upweighted reasoning and tool-parameter terms in Table 4 demonstrating that our entropy-awan extend our evaluation to the Llama-3.22 -.---3.--Bssstuueee fine-tuned with ResT in Table D.2 and Table D.3 to verify that the effectiveness of our method is not confined to specific model family. This confirms that the benefits of our approach are model-agnostic.
weighting effectively bridges the post-SFT model. We also
CONCLUSION
We study tool-use in LLM agents through the lens of RL and establish a theoretical link between token-level policy entropy and training stability: lowering entropy tightens variance bounds pol- icy gradients, and reward mass concentrates on structured, low-entropy tokens such as tool names, arguments, and output-format markers. Building this insight, we propose ResT, entropy- aware, token-level RI. algorithm that reshapes policy gradients via entropy-informed reweighting and lightweight curriculum that progressively upweights reasoning tokens. This enables smooth shift from structural correctness to semantic reasoning and stabilizes convergence in long-horizon, multi-turn tool use. Empirically, ResT attains state-of-the-art results on BFCL and API-Bank, im- proving prior methods by up to 8. 76%; when fine-tuned on a 4B base LLM, it further surpasses GPT-4o by 4.11%% 11% on single-tum tasks and 1.50% on multi-turn base tasks.
Our experiments concentrate on schema-conse constrained tool-use benchmarks. Within this setting, ResT implements principled, entropy-aware reshaping of token-level policy gradients, translating our variance analysis into practical curriculum over structural and reasoning tokens. A natural next step to earry the same design philosophy to more open-ended language-agent tasks with learned or preference-basee rewards, replacing rule-based execution scores with scalar feedback from reward models or LLM-as-a-judge evaluators applied to the final answer or the quality of the reasoning trace, while preserving the mechanism of entropy-guider token reweighting. We expect these extensions to further broaden the applicability of ResT.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by National Natural Science Foundation of China (Grant No. 62550062 and 62576338), the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDA0480302), Beijing Natural Science Foundation (Grant No. L252145), and Young Sci- entists Fund of the State Key Laboratory of Multimodal Artificial Intelligence Systems (Grant No. (S22100116).
ETHICS STATEMENT
We propose ResT to improve the training efficiency and stability of large language models (LLMs) acting tool-use agents. We recognize that methods which increase agentic capability may entail dual-use risks. In particular, more capable tool-using model, if misapplied, could facilitate ma- licious tasks, including automating harmful behaviors attempting to access information without authorization. Our design choices and evaluation protocols therefore emphasize safety, bounded generalization, and auditability.
All experiments were conducted in a controlled academic setting on public benchmarks (BFCL and API-Bank) whose tools and APIs are simulated; no real-world services were invoked, and no sensitive or private data were accessed. We document data provenance, filtering, and preprocessing; exclude any potentially identifying content; and release scripts that regenerate the exact evaluation inputs. To reduce misuse risk, we enforce strict rate limits in evaluation, log all tool calls, and disable any actions that could affect external systems. Model selection and reporting follow a pre-specified protocol, and we provide seeds, prompts, and run logs to enable independent verification.
Our goal is to contribute a robust (Duan et al. 2026) and efficient training paradigm for reinforce- ment learning with LLM agents while advancing understanding of safe and controllable agent behavior. We encourage subsequent researchers and practitioners to assess societal and ethical impacts incluudnn distributional harms, failure modes under domain shift, and cascading error risks-and to adopt responsible release practices. Upon acceptance, we intend to release model checkpoints and inference artifacts under research-only license, accompanied by detailed model card describing intended use, limitations, and safety considerations, as well as red-teaming guidance and an abuse-reporting channel.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility, release the complete implementation of ResT ResT-including train- ing and evaluation code, experiment scripts, and configuration files-on on an anonymized GitHub seposiioy/httsss///.ttttttt httpp///sggggtooooo///////////// /tt
The repository specifies exact dependency versions, provides an environment requirement file, and pins all results tagged commits. We include turnkey scripts for every table and figure, along with hardware specifications, random seeds, and expected wall-clock times. Logs, intermediate artifacts, and checkpoints necessary to reproduce the reported numbers are stored with checksums; unit tests cover core components of the training loop and evaluation harness. Our experiments build on several open-source LLMs, and the repository documents model choices, preprocessing steps, and all implementation and training details.
We evaluate on two widely used public benchmarks, BFCL and API-Bank, both of which provide simulated tools/APIs rather than real services. For robustness, we report the best run for baselines and for ResT, using identical evaluation protocols. We provide full hyperparameter sweeps, ablation studies, and sensitivity analyses (c.g., s.. to reward scaling, rollout length, and optimizer settings). All hyperparameters, data preprocessing pipelines, and experimental configurations appear in the ap- pendix and in the repository documentation. Any deviations from benchmark defaults explicitly noted, and we include scripts to validate dataset integrity and to regenerate all results from scratch.
To further support open science, upon acceptance we plan release the trained model checkpoints and inference-time artifacts under a research license, accompanied by model cards that describe intended use, limitations, and safety considerations. We will also provide minimal, dependency-

---

Published as a conference paper at ICLR 2026
pinned inference package and reproducible notebooks to facilitate downstream evaluation and com- parison.
REFERENCES
Maciej Besta, Julia Barth, Eric Schreiber, Ales Kubicek, Afonso Catarino, Robert Gerstenberger, Piotr Nyezyk, Patrick Iff, Yueling Li, Sam Houliston, Tomasz Sternal, Marein Copik, Grzegorz Kwasniewski, Jürgen Müller, Lukasz Flis, Hannes Eberhard, Zixuan Chen, Hubert Niewiadomski, and Torsten Hoefler. Reasoning language models: A blueprint, 2025. URL https:
org/abs 2501. 11223
Hongzhan Chen, Tao Yang, Shiping Gao, Ruijun Chen, Xiaojun Quan, Hongtao Tian, and Ting Yao. Discriminative policy optimization for token-level reward models. arXiv preprint arXiv:2505.23363, 2025.
Nuo Chen, Hongguang Li, Baoyuan Wang, and Jia Li. From good to great: Improving math rea- soning with lool-augmented interleaf prompting, 2023. URL https:
24C1. 05384
Yuheng Cheng, Ceyao Zhang, Zhengwen Zhang, Xiangrui Meng, Sirui Hong, Wenhao Li, Zihao Wang, Zekai Wang, Feng Yin, Junhua Zhao, et al. Exploring large language model based intelli- gent agents: Definitions, methods, and prospects. arXiv preprint arXiv:2401.03428, 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-trate teaintriiiingg.... arXiv preprint 2r1thv2501.1716.1 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li. Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforce- ment learning, 2025. htR.htt.s: / //trxivvvvv////////v//e/////.vvl org/ abs//////// 2501 12948
Junxian Duan, Siyu Liu, Yiming Hao, Huaibo Huang, and Ran He. Dual frequency-guide spa- tiotemporal feature learning for face forgery detection. IEEE Transactions on Biometrics, Behav- ior, and Identity Science, 1(2):179-191, 2026. doi: 10. 00.1099/BBIOMM.20555666
12

---

Published as a conference paper at ICLR 2026
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536 2025a.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for Ilm agent training. arXiv preprint 1xXv:2505..0978, 2025b.
Wenfeng Feng, Penghong Zhao, Guochao Jiang, Chuzhan Hao, Yuewei Zhang, and Hao Wang. Pvpo: Pre-estimated value-based policy optimization for agentic reasoning. arXiv preprint arXiv:2508.21 1044, 2025c.
Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. Tidr: Token-level detective reward model for large vision language models, 2024b.
Anna Goldie, Azalia Mirhoseini, Hao Zhou, Irene Cai, and Christopher D. Manning, Synthetic data generation & multi-step rl for reasoning & tool use, 2025. URL htltttsss///xiii/ ww..vvrrggaaa abs/ 2504. 04736
Jiawei Gu, Xuhui Jiang, Zhichao Shi. Hexiang Tan, Xuchao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang. Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge, 2025. URL
abs/
2411..1592
Bingguang Hao, Maolin Wang, Zengzhuang Xu, Cunyin Peng. Yicheng Chen, Xiangyu Zhao, Jinjie Gu, and Chenyi Zhuang. Funreason: Enhancing large language models' function calling via self- refinement multiscale loss and automated data refinement, 2025. /tt.httss://arxiv
abs/2505 2505.20197
Peide Huang. Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum rein- forcement learning using optimal transport via gradual domain adaptation. Advances in neural information processing systems, 1e.10666-16670 2022.
Mathew Huerta-Fnochian and Seung Yong Ko. Instruction fine-tuning: Does prompt loss matter? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2272222772222 1-22795, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.11863////2244cmmm.. 1267. URL htt.aatt///claaatoollotooo
wwg. www.mmmmmmamm.mmmmm 1267/
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training 11ms to reason and leverage search engines with reinforcement learning, arXiv preprint arXiv:2503.09516 2025.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. 2024.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Aai-bank: A comprehensive benchmark for tool- aug-enteenteed lims, 2023. URL https:: /tap:::/aaaae w/wrxiiivorrre abs/ 2304. 08244
Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output a reproducible pipeline, 2024. URL htttpss//aaxx...or///bbb//////// 08190]
Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Ji- amu Zhou, Cheng Cheng, Yin Zhao, Jun Wang, and Weinan Zhang. Robust function function-calling for on-device language model via function masking. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL yV0cr4qip6
wwt/forum?? forum?id
13

---

Published as a conference paper at ICLR 2026
Boyuan Liu, Feng Ji, Jiayan Nan, Han Zhao, Weiling Chen, Shihao Xu, and Xing Zhou. Omne-rl: Learning to reason with memory for multi-hop question answering. arXiv preprint arXiv:2508. 17330, 2025a.
Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong Wang, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang. Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang. Xin Jiang, Ruiming Tang, Defu Lian, Qun Liu, and Enhong Chen. Toolace: Winning the points of IIm function calling, 2025b. URL https: / http//aww..oggg/aoeee/o wtw.org//as///000//0 200bb./244900000 00920
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025c.
Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, and Yuqing Yang. Agent lightning: Train any ai agents with reinforcement learning, 2025. URL
/tttp::/aarivv org/ abs/ 2508. 03680
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122235, 2024.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1-50, 2020.
Xuan-Phi Phi Nguyen, Shrey Pandit, Senthil Purushwalkam, Austin Xu, Hailin Chen, Yifei Ming, Zix- uan Ke, Silvio Savarese, Caiming Xong, and Shafiq Joty. Sfr-rag: Towards contextually faithful Ilms, 2024. https: arxiv. org/ www.b//92... 2409.4999999
Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, et al. Curriculum reinforcement learning from easy to hard tasks improves Ilm reasoning. arXiv preprint arXiv:2506.06632) 2025.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. 2023. URLEttpp /RL/ttts:://aaat http::///axxvvvvo.aa wbs/ 2tabb/22300 2305 15334
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tie Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint htti:2504.135.. 2025.
Shengqian Qin, Yakun Zhu, Linjie Mu, Shaoting Zhang, and Xiaofan Zhang. Meta-tool: Un- leash open-world function calling capabilities of general-purpose large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume Long Papers). pp. 30653 30677, Vienna, Austria, July 2025, Association for Compu- tational Linguistics. ISBN 979-8-89176-2511 https://allantoolos org/ wttpoo/gg//////// ac1-oooo 1481/]
doi: 10. 12118653/112222.allll
1481. URL
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 01555728--37444 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024.
wwwp:/iwwrrga///////a/a 2402
03300
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-searcher: Incentivizing the search capability in 1mms via reinforcement learning. arXiv preprint 2rXiv:2503.05592, 2025.
14

---

Published as a conference paper at ICLR 2026
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: preprint arXiv:2507.20534 2025a.
Open agentic intelligence. arXiv
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai. Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng. Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang. Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengman Dong. Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang. Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Weixin Xu, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang. Zaida Zhou, Zhaoji Wang. Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, Zonghan Yang, and Zongyu Lin. Kimi k1.5: Scaling reinforcement learning with Ilms, 2025b. URL https: /ttpss////xiv..// org/ abs / 2501. 12599
Hongru Wang, Cheng Qian, Manling Li, Jiahao Qiu, Boyang Xue, Mengdi Wang, Heng Ji, and Kam-Fai Wong. Toward a theory of agents as tool-use decision-makers. arXiv preprint N0Nv:25006008886, 2025a.
Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, and Fei Yang. Parameter- efficient fine-tune tuning in large language models: a survey of methodologies. Artificial Intelligence Review, 58(8):227, 2025b.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025c.
Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang. A survey of llm-based deep search agents: Paradigm, optimization, evaluation, and challenges. arXiv preprint arXiv:2508. 05668, 2025.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing I1m reasoning with rule-based reinforcement learning, 2025. WRL/https: //axivv.// org/ abs 2502. 117 14768
Eunscop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo. Kyoung- Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. Tler: Token-level continuous reward for tne-grained reinforcement learning from human feedback. In Findings of the Associ- ation for Computational Linguisties: ACL 2024, pp. 14969- 14981, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guang- ming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang. Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingx- uan Wang. Dapo: An open-source 11 reinforcement learning system at scale, 2025. URL
https: /tt::///xxivi www.rvii.gg/..o.ee.o als///b////// awss//53.33. 14476
Yu Yue, Yufeng Yuan, Qiying Yu. Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05//18 2025.
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng. Tulika Awalgaonkar, Rithesh Murthy, Eric Hu, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong. xlam: A family of large action models to empower ai agent systems, 2024. URL httpt:tt////aat
www.//og/////oooooooo/o/./..eee/ abs /209 03215
15

---

Published as a conference paper at ICLR 2026
Yixian Zhang, Huaze Tang, Chao Wang, and Wenbo Ding. Policy newton algorithm in reproducing kernel hilbert space. arXiv preprint arXiv:2506.01597 2025.
16

---

Published as a conference paper at ICLR 2026
A SUPPLEMENTARY TRIAL
KL Loss Curve
actor_Ki Kd_loss_Luu KL001
0.020
W
actor kl [__loss__LL__ KL003
actor kl le_loss_Ket KLO05
0.015
Actor/KL Loss
0.005
Training Step
Figure A.1: Ablation Study on ResT: KI. Loss Dynamics under Varying Coefficients.
Mean Training Reward Curve
2.5
2.0
Critic/Mean Reward
1.5
reward _ea__meaa_______e ResT
rewards dewards_mean htlmmmeaaannnnnae
Entropy/eeaard
rewards mean KL001
rewards meaiis_mea____ KL003
rewards meannmmeaann_aaaaaaa KL005
20
Training Step
Figure A.2: Ablation Study on ResT: Impact of Regularization on Mean Training Reward.
17

---

Published as a conference paper at ICLR 2026
Table A.1: Impact of Regularization Components on ResT Performance on the BFCL Multi-Tun Benchmark. Metries computed with official scripts.
Model
Overall Acc
Base
Míss Fune
Miss Param
Long Context
Qwen3-4B-22 Dwen3-4B-25077 (ResT with loss. = 0.001)
48.00%
57.00
52.50%
34.00%
48.50%
Qwen3-4B-2507 (ResT with KL loss, 0.003)
50.25%
59.50
53.00%
34.00%
48.50%
Qwen3-4B-2507 (ResT with KL loss. 5 0.005)
49.62%
62.50 %
53.00%
37.00%
51.50%
Qwen3-4B-22 4B-n3-4--22077-- (ResT with entropy reward)
47.88%
58,50 %
46.50%
36,00%
50.50%
Qwen3-4B-2507 ResT)
50.38%
62.50%
56.00%
33.00%
50.00%
Table A.2: Statistical Validation with Multiple Seed Runs on Qwen3-4B-2507 and Qwen3-8B
trained with ResT and GRPO.
Model
Overall Acc
Base
Miss Fune
Miss Param
Long Context
ResT-4B-seedl
50.38 %
62.50 %
56.00%
33.00%
50.00
ResT-4B-seed2
49.00 %
58.50 %
54.50%
34.00%
49.00
ResT-4B-seee3
50.50 %
61.50 %
56.50%
34.50%
49.50
GRPO)4[B-sced]
40.88 %
47.00 %
43.50%
31.50%
41.50 %
GRPO-4B-sced2
41.12 %
48.50 %
43.50%
31.50%
41.00
GRPO-4B-seed3
41.62 %
48.50 %
45.50%
27.00%
45,50
ResT-8B-socd]
40.00 %
52.50 %
43.50%
28.50%
35,50 %
ResT-8B-socd2
40.13 %
50.50 %
45.00%
32.00%
33.00 %
ResT-8B-soed3
41.88 %
52.00 %
45.50%
33.50%
36.50
GRPO-8B-secd
37.62 %
45.00 %
39.50%
34.50%
31.50
GRPO-8B-scae -cced2
36.00 %
42.00 %
42.00%
28.50%
31.50 %
ERPO--B-scedd:
36.62 %
47.00 %
39.00%
32.00%
28.50 %
Table A.3: BFCL results with different methods on Qwen3-32B. Non-Live and Live represent the single-turn settings, Conversely, the multi-turn evaluation comprises four distinct subsets: Base, Miss Function, Miss Parameter, and Long Context.
Model
Overall
Non live
live
Base
Miss Fune
Miss Param
Long Context
Base 69.25%
88.90%
77.83%
54.50%
41.00%
30.50%
46.50%
GRPO 70.32%
89.19%
77.70%
59.50%
47.00%
32.00%
45.50%
ResT
71.69%
89.58%
78.85%
60.50%
56.50%
33.50%
44.00%
Table A.4: Abalation study on no gradients for COT. BFCL multi-turn test results on different mod-
els.
Model
Overall
Base
Miss Func
Miss Param
Long Context
1.7B (ResT)
16.00%
20.00%
16.00%
15.00%
13.00%
1.7B (No gradients for CoT)
15.62%
22.50%
13.50%
12.00%
14.50%
4B (ResT)
50.38%
62.50%
56.00%
33.00%
50.00%
4B (No gradients for CoT)
48.75%
62.00%
48.00%
32.50%
52.50%
8B (ResT)
40.13%
50.50%
45.00%
32.00%
33.00%
8B (No gradients for CoT)
37.62%
45.00%
43.00%
31.50%
31.50%
18

---

Published as a conference paper at ICLR 2026
B PROOFS
This appendix provides detailed proofs of the theoretical results presented in Section 3.1
B.1
PROOF OF LEMMA 1 (POLICY GRADIENT VARIANCE DECOMPOSITION)
Proof. Setup. Let trajectory be (z, 30:T sampled from ae. with return R(T). For trajectory index i, write T,, F2 == R(T.), and Yi.e. The per-time-step score is V log TT (Vi, I, si,))))
Step Policy gradient with a baseline. By the policy gradient theorem,
VaJ(0) ET
Σ 1=0
Vo log (ve x. vCC) R(T)
Let by = (x. be any baseline that does not depend on the sampled action Ye given the state/history. Then, conditioning on (z, v<t).
Byiaa! Ve log (M6 Mee) bi] = be Σ Tolue Mee) V log log vee)
be Vo TV(Ve x. I., = 0.
Hence one may replace R(r) by the advantage A(r) R(T) E, by inside the expectation without changing value. what follows we set by for simplicity (equivalently, redefine μ4, a as the population moments of A(T) if a baseline is used).
Step Population standardization and unbiased direction. standardized advantage
Consider the population-
Á.
Elr), o² == Var(r),
where expectations are under the current policy ne. Define the trajectory-leved gradient
gb A
v log ne (y x. vi.
Taking expectations and using B/r; 14] 0.
Egg.
E
(Tr p.)
Σ V log (v x, vi.e) = (o)
Thus gi is an unbiased direction estimator for (0) up to scalar factor 1/a. instead one uses the centered (unnormalized) advantage A; ri μ4, then E(gi] = VoJ(O) exactly. In practice the factor 1/ can be absorbed into the learning rate.
Step 3. Mini-batch variance decomposition. Let g1,..., 9G be i.i.d. copies gi with finite second moments, and define the mini-batee estimator
G
g
=
G
1=1
Its covariance is
G
Cov(g) = [g Elg Elg
Gov(g.ggj).
G2
1=1
Independence gives Cov(gi, 0j) = 0 for + j.
hence
G
Cov(g)
Σ
Cov(gi. gi) = Cov(g1).
G
G
T=1l
19

---

Published as a conference paper at ICLR 2026
Taking the k-th diagonal entry yields, for any coordinate k,
Var(g(h) = T )
proving equation 2
Step 4. Consistency checks. All symbols are trajectory-indexed (f) time-indexed (t) but not conflated. The expectation and variance are with respect to the mata-generating policy Tg. Population moments (H, o)) refer to returns under Te at the current parameters. The i... assumption holds when trajectories are drawn independently from ng.
Remark (batch normalization of advantages). If one instead uses the batch-normalicnn statistic
A,
r:
1 1
Σ r)
og
0g
Ng
G
G
j=1
1=1
then {gr } are longer independent because each gi is function of the full batch. direct expan- sion gives the exact decomposition
G-1 G
Cov (g) Var(gi)
Cov 92),
where Cov Nov.g199 92) denotes the cross-covariance induced solely by the shared batch statistics. Under mild moment and smoothness conditions, a delta-method (influence function) expansion of ((g,,a)) around (1, o)) yields Cov (gv. 92) = OL1/G), so
(
Cov(g) = G
Var(g1) +
and the bias of E[gl relative to VoJ(O)/o is O(1/G). Both corrections vanish as G - oc, but for small G they can be non-neglig in practice.
20

---

Published as a conference paper at ICLR 2026
B.2 PROOF OF LEMMA 2 (SECOND-ORE ORDER NOMENT-ENTRONE CONNECTION)
Proof. Setup and notation. Fix a single trajectory. Let 2t E RY denote the logits, Pr softmax(21) A V-1 the token distribution, and Ey. the one-hot vector of Yt Pc- Define the score vector 5t = Cy. - Pr RV and the Jacobian 0z/000 R Rxd Then
T V log To (Ve 1x = J 80. = A
Jr 88-
t=
All expectations below are taken over the joint randomness of the trajectory and the policy unless explicitly conditioned.
Step 1. Second-order moment control. By the definition of vector variance,
T-1 Σ
Var(g:) |1..11
||
s E [1:01
A
Jr
St
Condition on the trajectory prefix up time t. Since
E[st x, Mer] 0,
the sequence (J s forms a vector-valued martingale difference, and thus is I -2-orthog..al. Therefore
| - t=0
E
Σ
J. st
BIJT 13
t=0
Applying the inequality
||J sell = |e [e||e||sse] gives
T-1 Σ
T-1 E[A ||
E
A
Σ J
St
B
Σ
8, B 1.].
where we set
B = 2 more tightly
B1 20,
depending whether insert the entropy factor this stage. Either choice preserves the quadratie-in-weights structure used later; we adopt the tighter definition when convenient.
Step Exact expression for the expected score norm. Condition on Pt. Since 8, Ey, Pt with Pt.
V
| || = ||e 2(ee pt) + l|pell = 1 - 2pt.y
+
P
1=1
Taking expectation over ~ Pt removes the index:
E Eyg [| ps]
Σ p 1 1-11pt
Step 3. Entropy connection via the ordering H2 < H. Since the Rényi-2 entropy Hz(pt)
log E, př.. is always upper-bounded by the Shannon entropy He, we obtain
V
H. = Emmpul log Pt.0 2 - log Euwgd = -log
Σ )
p=
Equivalently,
V
Σ
Pt. 2 c Hr,
($)
21

---

Published as a conference paper at ICLR 2026
Combining equation i and equation [ gives the entropy upper bound
E sel
2 pel 1Ip.12
Hr
Taking expectation over the randomness in yields
||||s|||]
Ele #.] S 1 BlHel
where the last inequality uses convexity of exp(-) (optional if one keeps - Ble equation
Mo)). Equality in
@ hoolds iff pt is uniform on its support, certifying tightness.
Step 4. Conclusion. Substituting the entropy bound for E[[[ [12] into equation gives
T-1 Σ
Var(g:) E[AP]
E |
E[A
-
which is precisely the claimed second-order moment-entropy relation.
Remarks assumptions and tightness. (i) The step factoring E[A,"] is exact if A, is independent of (Je, 8t) or after conditioning on the trajectory prefix; without independence one may apply Hölder to absorb the mismatch into constant, leaving the functional dependence entropy unchanged. (ii) The bound [ = 1 - ||p:112 < e - d. becomes tight when is uniform over its support. reflecting the ordering H/2 S // between Rényi-2 and Shannon entropies.
22

---

Published as a conference paper at ICLR 2026
B.3 PROOF OF THEOREM 2 (OPTIMAL ENTROPY-AWARS REWEIGHTING)
Proof. Step 1. Starting point. By Lemma and the inequality || 8el12 S ||
Var(o:) E A Σ
Jell [|se1
t=
Applying the entropy bound [[[[ || E 1 Mr gives
T-1
Var(g:) S E[AA"]
Σ E8
=[[A]] 1=0
Σ
3cr
t=
where we set
B: = (1- 20000000 200
Remark. Factoring out [[[A") tacitly assumes that A; is independent of (Je, He) or that the bound is taken conditionally on the trajectory; without independence one may invoke Hölder/Cauchy- Schwarz to obtain an equivalent quadratic objective in the weights, which preserves the optimal proportionality derived below.
Step 2. Effect of reweighting, Introduce per-step nonnegative weights V, 2 0 with normalization E Wi = T, and consider the reweighted estimator
9; (rw)
A
Σ wr Je sc.
Variance scales quadratically in the weights, hence
Varlg (
S A Σ Bg (1²)
Step Convex program. We minimize the separable quadratic
T-1
min
Σ Be (w))
s.t.
Σ +
TT
We 2.
t=
If 3e > for all t, the objective is strictly convex and the feasible set is simplex, so the minimizer is unique.
Step 4. Lagrangian and stationarity, We first solve the couslity-constrained problem (the inequal- ity will be verified inactive). The Lagrangian reads
T-1
C(w, A)
Σ Be (w))² +
Wi -T).
Taking the derivative with respect to w; gives the stationarity condition
ac
A
23 th + x O w
with c:= -2/2.
twi
23
3
Step 5. Normalization and interiority. Enforcing
Σ 1=
Wi = T yields
T
27
Σ =T
c
x
2um 3 < 0.
3-
Hence, for all t,
T 3-11
1
> 0,
Σ T-1
4== 3-1
23

---

Published as a conference paper at ICLR 2026
which a posteriori verifies that the nonnegativity constraints are inactive at the optimum (the KKT multipliers for the inequalities vanish). Strict convexity then guarantees uniqueness.
Step 6. Degenerate case. If some 3t, = 0, any feasible allocation supported on {t 3, = 0} attains objective value O, which is minimal. Otherwise, when all 3, > O, the interior solution above is the unique minimizer.
Step 7. Optimal value and variance bound. Substituting xh into the quadratic gives
T
B
T
Σ (wxz)
Σ
3
t=0
-==0
1 1-1
u==
A
Therefore,
T
Var(g, m))) 1 [[A"]
Pe
Step Dual certificate via Cauchy-Schward For any feasible {
T-1
T-1
T-1
T-1
1
v B. wj Br 12) -
Be (1x))
t=
t=
t=
which implies
T Be (w)) 2
T
Σ 3,
with equality if and only if úh xc 8, matching मł and certifying optimality.
24

---

Published as a conference paper at ICLR 2026
C TRAINING ALGORITHM: REST WITH RULE-BASED REWARDS
Algorithm 1: Rased Reward Computation
Input: Prediction (Np, Pp. vp); ground truth (Ng. Pa. vc) for each tool; format template T; training progress (0, 1); weights (Bn, By)
Output: Riai
Format score:
format < preddcction matches T exactly)
Name score:
NohNNl
Yname
NGUNp
Parameter-name score:
para + EGGEG Pc(C))UPP(C.]
Parameter-value score:
Vvalue EG,eG Evevo(G,) [P = [r(
Normalization:
Z + 1 + |G| + Ec (v((G))
S Tname + para + Vvalue
Z Dynamic scaling:
Ri,fiall f- (1 - D) (Ba Sccc + 3g Si,format)
return Ri,final
25

---

Published as a conference paper at ICLR 2026
Algorithm 2: ResT: Reshaping Token-level Policy Gradients
Input:
Prompt z: policy me with frozen reference Te group size G; PPO clip e: stability constant > 0; curriculum proportion E [0. 1 clipping bounds (Wmins Wmax).: step sizes (at,, Op, )e entropy-baser initializer (Eq.
Output: Updated parameters 0.
Rollout:
for to G
Sample response y/ (T( |x) containing plain-text and tool-call segments
Compute reward Ri,tmaal via Alg. I
Group-normalized advantage:
+- C Ei= Ri,fmal, of v ④ Di RRi,inal 11)2 + 8
for i 1 to G
A, + Ri,final -
Token partition and initialization:
// Initialize global token weights (shared across samples)
Let T be representative sequence length or maximum length Initialize Wy for all 1,,... T using entropy-based rule (Eq.
Curriculum reweighting (global schedule):
// Apply curriculum updates once, shared by all sequences for t to T do L W24- t- Clip( Wi Umin; Wmaxx
for 1 Timt do L w; + Clip( w; p Winin Wmxx)
for 1 Iname do W.=- + Wmax
for t E Ipara do ú Clip( wi + apy, Wming Wmax)
for t Irnk do
L w; + Clip(W; + a,y, Wmins Wmax)
// Normalize globally
T 1=1 wu @ + 1 Σ1=1
for Wr
W 5
PPO-style token-level objective and update: for i to G do
for 1 to 7 do
Fia + ne (Vi. x, <I)
Tua (Ui,e
x. Vi,
Ir.t min Á,, 1
CReST(O) + - E ΣT 4 Ii.
Update +- 0 - V (oRRss(())
Clip(u, a, b) min(max(u, (u, b). Sets Iname Ipera Iink obtained via deter- ministic token tagging of the generated sequence. Eq. 9 provides the entropy-based initializer for
Helper:
26

---

Published as a conference paper at ICLR 2026
D COMPLETE EXPERIMENTS RESULTS
Table D.I: BFCL Benchmark Results on Single Turn Scenarios. All metrics are calculated using
the official scripts. Rel. and Irrel. denote the gllvance--detcction and inclevance-dettectoon subsets of BFCL. respectively.
Model
Overall Acc
Non-live Acc
Live Acc
Rel.
Irrel.
Qwen3-1.78 (Base)
77.70%
83.27%
72.95%
83.33%
85.49%
Qwen3-1.7B (SFT)
78.13%
82.90%
74.06%
72.22%
78.02%
Qwen3-1.7B (TSFT)
78.76%
83.67%
74.59%
77.78%
78.04%
Qwen3-1.7B (RSFT)
77.99%
82.46%
74.19%
66.67%
76.53%
Qwen3-1.7B (GRPO)
77.57%
83.00%
72.95%
72.22%
73.42%
Qwen3-1.7B (SFT+GRPO)
78.17%
85.31%
72.10%
66.67%
76.94%
Qwen3-1,7B (Ours, ResT)
78.59 %
84.65%
73.43%
66.67%
76.30%
Qwen3-4B-2507 (Base)
83.41%
87.98%
79.52%
77.78%
79.18%
Qwen3-4B-2507 (SFT)
84.66%
89.04%
80.94%
83.33%
81.90%
Qwen3-4B-2507 (TSFT)
83.70%
87.31%
80.63%
83.33%
81.39%
Qwcn3-4B-2507 (RSFT)
83.85%
87.69%
80.59%
88.89%
82.02%
Qwen3-4B-2507 (GRPO)
83.82%
89.40%
79.08%
83.33%
81.52%
Owen3-4R-2507 (SFT+GRPO)
84.54%
89.46%
80.36%
77.78%
80.22%
Qwen3-4B-2507 (Ours, ResT)
84.58
88.65%
81.12%
77.78%
79.67%
Qwen3-8B (Base)
83.26%
88.81% %
78.54%
71.78%
79.08%
Qwen3-8B (SFT)
77.78%
82.58%
73.70%
72.22%
75.62%
Qwen3-8B (TSFT)
78.30%
81.88%
75.26%
83.33%
76.72%
Qwen3-8B (RSFT)
77.16%
81.08%
73.83%
77.78%
77.38%
Qwen3-8B (GRPO)
84.37 %
88.50%
80.85%
83.33%
82.24%
Qwen3-8B (SFT+GRPO)
80.39%
85.90%
75.70%
77.78%
75.96%
Qwen3-8B (Ours, ResT)
84.11%
90.08%
79.03%
94.44%
77.62%
Qwen3-14B (Base)
83.74%
88.17%
79.96%
72.22%
81.01%
Qwen3-14B (SFT)
74.35%
77.31
71.83%
77.78%
69.95%
Qwen3-14B (TSFT)
75.62%
79.90%
71.97%
83.33%
72.48%
Qwen3-14B (RSFT)
72.66%
71.81%
73.39%
55.56%
85.44%
Qwen3-14B (GRPO)
84.04%
89.10%
79.74%
77.78%
79.31%
Qwen3-14B (SFT+GRPO)
81.17%
86.56%
76.59%
77.78%
75.41%
Qwen3-14B (Ours, ResT)
84.63
89.10%
80.81 %
77.78%
80.56%
Entropy by segment (mean over 10 samples)
Entropy by segment (mean over 10 samples)
0.58
0.10
Average token entropy (bits)
(D-406)
0.49 (n-195 194)
Average token entropy (bits)
0.10
(n=1714)
0.09 (n=236)
0.5
0.08
0.3
0.04 (n-491)
0.02
0.00
(n=331)
0.00
think
tool_coll call
response
think
tool call
response
Figure D.1: Average region entropy on Llama-3.2-30 2.3 (left) and Qwen3-1.n (right). For each model, we randomly sample 10 instances and compute the entropy distribution over gener- ated tokens. Tokens are partitioned into functional regions- rraioonnr,s,,,ng, tool invocation, and final response.
27

---

Published as a conference paper at ICLR 2026
Table D.2: API-Bank Test Results on Llama3.2-3BIInstre model. We report the highest aceu-
racy achieved for each method after hyper-parameter tuning, with a breakdown by difficulty level. Model
Overall Acc
Level Ace
Level Ace
Level 3 Acc
Baama-3,2-3B-Inste (Base)
37.35%
43.61%
32.84%
20.61 %
46.4%
52.88%
29.85%
35.11% %
49.25%
57.14%
31.34%
34.35%
51.26%
57.89%
29.85%
41.98%
59.13%
65.66%
52.24%
42.75%
52.76%
61.15%
43.28%
32.06%
Llama-3,2-3B 1:3B-Instruct(Ouuss ResT)
62.81%
69.42%
53.73%
47.33%
Table D.3: BFCL Results on Llama3.2-3B- Instruct model. All metrics are calculated using the
official scripts. Overall accuracies for the three different tasks are provided separately. Best bold.
Model
Non-Live AST Acc
Live Ace
Multi Turn Ace
Relevance Detection
Irrelevance Detection
17.44%
43.85%
0.00%
77.78%
66.07%
Llamm- 3.2m.-3.22.11 Insiruct (SIT)
65.42%
63.04%
1.38%
77.78%
78.25%
Llama-3.2-33 Instructt (TSFT)
57.06%
45.45%
0,12%
38.89%
59.17%
Llama-3.2-1B-Insiie (RSFT)
35.35%
60,20%
1.25%
94.44%
71.00%
Llama-3.2. 3B Instruct (GRPO)
74.38%
$6.86%
1.37%
94.44%
62.23%
Llama-3.2-3B-Insstuue (SFT-GRPO)
23.92%
89.99%
Llama-3.2-38-- 100m--3---1BInnttueet (Ours, ResT)
83.19%
61.53%
3,50%
88.89%
66.36%
28

---

Published as a conference paper at ICLR 2026
E
GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
Group Relative Policy Optimization (GRPO) serves as critic-free, on-policy optimization algo- rithm designed for Large Language Models (LLMs). presents an efficient alternative Proximal Policy Optimization (PPO) by constructing a relative baseline for advantage estimation. This ap- proach circumvents the need to learn separate value network, thereby avoiding dependencies on methods like Generalized Advantage Estimation (GAE) which require value function.
The core idea of GRPO is dynamically construct baseline for each sampled output using the reward distribution of set of candidate outputs generated from the same input prompt. Specifically, for a given prompt q, we first sample set of complete output sequences O {01, ......: 0o} from the current policy Te.i A scalar reward = R(g. 0,) is then computed for each output o,.
The empirical mean of these rewards serves as the baseline for evaluating each output, i.e., b(g) = deviation of the rewards obtain the normalized advantage estimate A,
). The advantage for each rollout or is then normalized using the standard
mean( .r. rs})
A, std (ri: ................................ rs)) +
(20)
where 5 is small constant added for numerical stability to prevent division by zero. This "intra- group relative" advantage estimation effectively reduces reward variance.
During the policy update phase, this sequence-level advantage value A, is applied to every timestep (token) within that sequence. GRPO performs policy updates by maximizing a clipped surrogate objective, analogous to PPO:
GRROO (0) Dg anotoald
Σ min min(p.(0)) (c) clip(p. 1-e, 1))
KL[ (=)(-1)
(21)
Toets (oxlg,0cr) the likelihood ratio of the new and old policies at timestep t. Á corresponds to the normalized advantage of the current sequence The hyperparameters e and B control the clipping range and the strength of the KL-divergence penalty against a reference policy Thff respectively.
where Pe(0) (e(oclg.
By adopting this formulation, GRPO retains the stability of PPO's trust-region updates while elim- inating the complexities of training a value network, offering simple yet effective framework for fine-tuning LLMs with reinforcement learning.
29

---

Published as a conference paper at ICLR 2026
F TRAINING DETAILS
We conducted comprehensive experiments using the verl 0.5.0 framework, and the detailed training
hyperparameters are provided in Table F.1 and F.2 The System prompt is shown in Figure G.1
Training Dynamics Analysis. As illustrated in the training dynamics in Figure F.] ResT exhibits smoother and consistently lower policy entropy compared to GRPO. Regarding the mean response length, ResT shows a relatively small amplitude of change, whereas GRPO exhibits significant vari- ation. Furthermore. ResT demonstrates stable convergence; the reward curve converges starting from step 30 and maintains this level through step 80, showing no signs of reward collapse.
3.2
0.10
Woatn Resporun Leogth
350
3.0
Pollyy Entroge
0.08
300
Roward
2.8
250
0.06
200
2.6
150
20
40
60
80
20
40
60
80
20
40
60
80
Training Step
Trnining Step
Tramiig Step
Figure F.1: Learning curves for ResT and GRPO during training steps. The training dynamics show that ResT achieves a significantly lower and smoother policy entropy compared to GRPO, while maintaining comparable reward performance and longer responses.
30

---

Published as a conference paper at ICLR 2026
Table F.1: Configuration for ResT training Qwen3 models.
Data Configuration
Train Batch Size
512
Validation Batch Size
128
Max Prompt Length
2048
Max Response Length
2048
Optimization
Learning Rate
1 x 10-6
PPO Mini Batch Size
64
KL Loss Used
False
Rollout Configuration
Rollout Name GPUs
vllm 0.8.5
8x 141G H20 or 8x 80G A100
GPU Memory Utilization
0.8
Number of Rollouts
4 or 8
Training & Logging
Save Frequency (Steps)
15
Test Frequency (Steps)
15
Total Epochs
15
Initial Reweighting Factor
Initial Factor of Chain of Thoughts
0.016
Initial Factor of Tool Names
1.2
Initial Factor Tool Parameters
0.8
Initial Factor of Output Format
0.95
Curriculum Learning Configuration
Øy (update magnitude of output format)
0.003
Opp (update magnitude of output tool parameters)
0.003
Og (update magnitude of reasoning)
6 x 10 -5
31

---

Published as a conference paper at ICLR 2026
Table F:2: Configuration for ResT training
Data Configuration
Train Batch Size
512
Validation Batch Size
128
Max Prompt Length
2048
Max Response Length
2048
Optimization
Learning Rate
1 x 10-6
PPO Mini Batch Size
64
KL Loss Used
False
Rollout Configuration
Rollout Name GPUs
vllm 0.8.5
4x 80G A100 0.8
GPU Memory Utilization
Number of Rollouts
8
Training & Logging
Save Frequency (Steps)
15
Test Frequency (Steps)
15
Total Epochs
15
Initial Reweighting Factor
Initial Factor of Chain of Thoughts
0.6
Initial Factor of Tool Names
1.2
Initial Factor of Tool Parameters
0.6
Initial Factor of Output Format
1.0
Curriculum Learning Configuration
a (update magnitude of output format)
0,002
Op (update magnitude of output tool parameters)
0.002
(update magnitude of reasoning)
0.002
32

---

Published as a conference paper at ICLR 2026
G
PROMPT DETAILS
Important Notes
System Prompt Design
field your reasening Provide at least one of <ocol, or "crpppnse" Decide whether use "shool cull>" (possibly mulipple times), "crmpposse"" or both.
always
You multi-tann dislogue
felveraggnng calls user tokk provide stractured respooses.
Available Tools
"<tool call>" "peramctens" ficid cuniiiing distionury of punmoccess If pommsters are nesied, laave the "pumensters fiekl an cmpty dietionse..
You invoke mutiple tool calls sinultanconsly call should
your resporse, you (Namest Deseripliune.... (Parancterss...
following
Output Format
thoughts ressoning
dislogue records histoy. including user's queries, previous "stool "tesponse"" any feedback "aaheo" eniss))
Refer
<tool cilliin ("mmmfht Toooo nome!", "pammonien"" ('Pammmeer namelt: ("Parameler contant" th with Vi.. "f Pi.. ("pummeiens"" 1" if if.. ..f 1h. .u "...."/.....ooooo... ealle- dinal
<hesponsee>
Answer
<thinl-Y Ressozing progress<tiinl>
Steps for Each Turn
stoel sall--The The tool name and punmenun-choll,cco sulls> Snepume--hhe The you to the uerrstrspomee
1. Think: Rocall relevan context and entlyas curnent goal.
Decide Tool
toel nedded, specify
Respood Appropriately: response needed, generate maintanning cocsistency acrees user quectes.
Figure G.1: The following Q&A template-commprsingg predefined instructions, guidelines, and constrains--standarddiee standardizes model behavior and output formatting across training and evaluation. ensures that, during training, the model learns consistently in line with target objectives, and that, during BFCL evaluation, performance produced under a unified rule set.
33

---

Published as a conference paper at ICLR 2026
H
OUTPUT CASES
Table H.l: System Instruction Case of ResT Models generalizing to irrelevant tool detection goals.
User Prompt:
** Dialogue Records History**
< user
Can you please modify my appointment scheduled for March 25th with Dr. Kim March 26th with Dr. Lee?
< //user >
<
response >
Sure, I can help you with that. Please provide me with the appointment and the new appointment date and doctor's name.
/response >
user The appointment ID is 34567890 and the new date is March 26th with Dr. Lee.
//user
<
response >
Alright. I'll modify your appointment now.
<.
</response
< user
Based on conversation above, please only make tool call to solve my need.
< /user> >
Tools Available:
In your response, you can use the following tools:
QueryHealthData
This API is used to query recorded health data for given user within a specified time span. Parameters:
user (string): the identifier the user, case-insensitive.
start time (string): the start of the time span, formatted YYYY-MM-DD HH.MM:SS.
end didm (string): the end of the time span, formatted as YYYY-MM-DD HH.MM:SS.
Ceiseloegittration
This API cancels the registration of a patient given an appointment ID. Parameters: appointment.iig id (string): the identifier of the appointment. Modify Registration
This API modifies the registration of patient given an appointment ID. Parameters: appointment id (string): the identifier of the appointment. new wew.appointmentt.oo (string): the new appointment date, formatted as YYYY-MM-DD. new.appointment.doete (string): the new appointment doctor.
34

---

Published as a conference paper at ICLR 2026
Table H.2:
Response case of Qwen3-1.7B (ResT) generalizing irrelevant tool detection goals.
<think>>> Okay, let's see. The user wants to modify their appointment from March 25th with Dr.
Kim to March 26th with Lee. The appointment ID provided 34567890. First, I need to check which tool to use. The available tools are QueryHealthDataa, Can- celRegistration, and ModityRegissration. The user is asking to modify an appointment, so ModiffRegisttrtion is the right choice here. Looking at the parameters for ModityRegistration:: new www.aaannneeenddaae and www.appointment_ddeeee The user provided the appointment ID as 34567890, the new date as March 26th (which needs to be in Y-9%m- format, so 2023-03-26). and the doctor Dr. Lee.
appointment id,
I need to make sure the date format is correct. The user said March 26th, so the format should be 2023-03-26. The appointment ID is given string, so that's fine. The doctor's name is Dr. Lee. which is a string.
So the parameters would be: appointment id: "34567890" new appointment date: "2023-03-26" new new_appoinne appointment__ooctnn "Dr. Lee" I should call Modify with these parameters. No need for CancelRegistration since the user isn't 't canceling the old appointment. Just modifying it.
< </think>
< tool_call >
"name": MMgiityeeiistration, "parameters": "aapointment.idl idd": "34567890",
"new appointment date": "2023-03-26", "new wwwpoinmntment doctor": "Dr. Lee"
< </lool call >
35

---

Published as a conference paper at ICLR 2026
Table H.3: Response case of Qwen334B-Thhininn2255 (ResT) generalizing to irrelevant tool de- tection goals.
< think >
Okay, let's see. The user wants modify their appointment from March 25th with Dr. Kim to March 26th with Dr. Lee. They provided the appointment II) 34567890 and the new date is March 26th with Dr. Lee.
Wait, the previous response from the assistant said they'd d modify the appointment But the user is now asking to only make one tool call. The available tools include Modi- fyRegistration. Let me check the parameters for that.
requires appointment id, new appointment date (format Y- %d), and appointment doctor. The user specified the new date as March 26th, which I need to convert to YYYY-MM-DD. March 26th would be 2024-03-26 assuming the cur- rent year, but maybe the system handles the date correctly. Wait, the user didn 't specify the year, but the tool expects specific format. Hmm, but the user said "March 26th" " so the assistant should format as 2024-03-26? Or maybe the API just takes the month and day without the year? Wait, the parameter description says the format is % Y-%mm-% %,d so it needs the year. But the user didn't 't mention the year. Wait, in the conversation history, the original appointment was March 25th, so the date is March 26th of the same year. Maybe the system assumes the current year, but since the user talking about a specific date, the assistant should use the correct format.
ModifyRegistratinn doneet
Wait, the user's ' message says "the new date is March 26th with Dr. Lee." So the date March 26th. The tool requires the date in YYYY-MM-DD. Let me think: in the context of the problem, the user is probably in the same year as the original appointment. Since the original was March 25th, maybe the current year is 2024. So March 26th would be 2024-03-26.
But the tool call needs have the exact date string. Wait, the user hasn't specified the year, but in the problem context, maybe the system handles it. Alternatively, the assistant should just use the month and day as given. Hmm, but the tool requires the full date. Wait, the problem says the user is asking to make one tool call. The previous assistant said they'd modify the appointment now, so the next step is to call ModifyRegistration with the provided parameters.
The appointment be formatted 2024-03-26 (assuming current year 2024). The new appointment doctor Dr. Lee.
34567890. The new www.apppintment.coo date is March 26th, which should
Wait, the user said "the new date is March 26th with Dr. Lee." So the doctor's name is is string, so it's "Dr. Lee" or
Dr. Lee. The tool's parameter for new appointment_ just "Lee"? The description says "the new appointment doctor". probably the given. The user said "Dr. Lee", so the parameter should be "Dr. Lee".
So the tool call would be ModittRegisttation with appointment.ie id "34567890", new appointment date "2024-03-26", and new appointment doctor "Dr. Lee". Wait, but the user's previous message says "the new date is March 26th with Dr. Lee." So the date is March 26th, but the year might be different. However, since the original appointment was March 25th (probably in the same year), the year is the same. Let's assume 2024 for the example.
So the parameters are:
appointment.id: "34567890"
new appointment date: "2024-03-26"
appointment doctor: "Dr. Lee"
Therefore, the tool call should be to ModifyRegistration with those parameters.
/think
< ttoo_calll "name": "ModiiyRegiisrattonn.. "parameters": "appointment_id"" "34567890",
"new new.aappontment.date "2024-03-26", hwwwaappintmeen.doorrr. "Dr. Lee"
/tooo call >
36

---

Published as a conference paper at ICLR 2026
Table H.4:
Response case of Qwen3-8B (ResT) generalizing to irrelevant tool detection goals.
<think>>>
Okay, the user wants to modify their appointment. Let me check the previous messages. They provided the appointment ID as 34567890, the new date is March 26th, and the new doctor is Dr. Lee. The tool required is ModifyRegistration The parameters needed are appointment.id. new www.appointment.dee date, and www.appointment.dem doctor. The date format should be YYYY-MM-DD. So March 26th becomes 2023-03-26. The doctor's name is Dr. Lee. I'll structure the tool call with these details.
/thinkk>
<tool_call>
"name": "ModifyRegistrationn "parameters": "appointment.id" "34567890",
nhewaappontmeentddte"" "2023-03-26", pnow.appointmenn.docot.."" "Dr. Lee" //lool >
Table H.5: Response case of L-ama3.2-3B-Instree (ResT) generalizing to irrelevant tool detection goals.
think >
Based on the previous conversation, we have the appointment ID and the new appointment date and doctor's name. We the ModifyRegistration tool to update the appointment details.
</think
< <tool_call>>
"name": Registtyegisttrationn.... "parameters": "appointment.id" _dd'' "34567890",
"new www.appointment.aat "2023-03-26", "new www.aappintment..doo doctorr" "Dr. Lee"
/tool /tool_call >
37

---

Published as a conference paper at ICLR 2026
Table H.6: Response case of Qwen3-14B (ResT) generalizing to irrelevant tool detection goals.
<think>>>
Okay, let's see. The user wants to modify their appointment. They mentioned the appoint- ment ID is 34567890, the new date March 26th, and the new doctor is Dr. Lee. Looking at the tools available, the Modify tool is the right one. It requires the appointment.id. new www.appointment.dee The date needs to be in Y-%%m- %d format. March 26th would be 2023-03-26, but since the year isn't specified, mayhe it's safe to assume the current year unless stated otherwise. However, the user didn't mention the year, so perhaps the system expects just the date without the year. Wait, the parameter specifies the format as % --%-m-%d. so the year is required. Hmm, but the user only said "March 26th". Maybe the system allows omitting the year, but according to the tool's parameters, it's required. Wait, the user's original request was to change from March 25th March 26th. The original appointment date wasn't 't specified with a year, but the tool requires full date. This might be an issue. How- ever, since the user hasn't provided the year, maybe the system expects to be the same year the original appointment. But since the original appointment date wasn't t given with a year, perhaps the assistant should use the same year as the current date. Alterna- tively, maybe the system is designed to handle dates without years, but according to the tool's ' parameters, the format is strict. Wait, the tool's parameter for new appointment date says the format is 4-%%m- So the user must provide the year. But the user only said "March 26th". This could be a problem. However, in the previous response, the assis- tant said they would modify the appointment, maybe they assumed the year is correct. Maybe the user is in the same year as the original appointment. Since the original ap- pointment was on March 25th (without year), perhaps the system is designed use the current year. But since the user hasn't i specified, maybe the assistant should proceed with the date as given, even if the year is missing. Wait, but the tool requires the full date. This might be an error. However, given the user's s input, the assistant has to work with what's 's provided. Maybe the system lenient, or the user expects the assistant to proceed with the given date. So, the assistant will format the date 2023-03-26, assuming the current year. Alternatively, maybe the system allows just the month and day, but according to the tool's s parameters, it's required to have the year. This is a bit conflicting, but since the user hasn't 't provided the year, perhaps the assistant should proceed with the information given, even if it's incomplete. But that might cause error. However, the user's previous message said "March 26th with Dr. Lee," so the assistant has to use that. Maybe the system is designed accept the date without the year, but the tool's parameter says it's required. Hmm, this is a problem. But given the context, the assistant has to proceed with the available infor- mation. So, the parameters would be appointment id: 34567890, new wew.appintteen date: 2023-03-26 (assuming current year), new www.appointment doctor: Dr. Lee. The tool call would be ModifyRegistrrtion with those parameters.
and new www.appointment.com doctorr
< </lhink>
<. tool call >
"name": "Modifyyegistration". "parameters": "appointment_id" "34567890",
"new_apper wwwwaappointmen.datee "2023-03-26", www.aappontment..oocr "Dr. Lee"
< </tool_co calll> >
38

---

Published as a conference paper at ICLR 2026
Table H.7: System Instruction Case of ResT Models generalizing to irrelevant tool detection goals.
User Prompt
**Dialogue Records History**
< userr >
Could you get the current levels of nitrous oxide in the atmosphere for me?
< /user >
< lhink >
should use the appropriate tool with proper parameters to respond to the user's need.
/think
< tool coll >
"name": "Nitrous Ozone Levels API", "parameters": "nitrous": "current"
< //too_ccal
Lbss >
I"name": "Nitrous Ozone Levels API", "results": "nitrousOxideLLvee's "currentLevel"":
"330 ppb". "dateMeasured": "2020- 10-099T0::000000000000""" "location": "Global", "data-
Sources": ["NASA Ozone Monitoring Instrument", "NOAA Global Monitoring Labora-
tory"]. "error": null]
</obs>
< think >
should directly respond to the user's need.
< /think >
response >>
The current nitrous oxide levels in the atmosphere are as follows:
Current Level: 330 parts per billion (ppb)
Date Measured: 2020-10-99900::000000000
Location: Global
Data Sources
NASA Ozone Monitoring Instrument
NOAA Global Monitoring Laboratory
<
/response
Tools Available
In your response, you can use the following tools:
Nitrous Ozone Levels API
Description: Retrieves nitrous oxide levels in the atmosphere, providing current and his- torical data. Parameters:
"nitrous\(" ( ("description": "Optional filter for nitrous oxide levels (e.g., 'current', 'his- torical', or a specific value)", "type": "string", "default": """)
Country Information API
Description: Retrieve information about countries, including their capitals, currencies, and flags. Parameters: {
M4.5+ Earthquakes, Past 7 Days
Description:
Retrieve list of earthquakes with magnitude of 4.5 or greater that have occurred in the past days. Parameters:
"start_string"" { ("description": "The start string of the 7-day period (in ISO for- mat, c.g., 023-03-01100:000000.00000
"type": "string", "default": """, "end string\." ("description":: "The end string of the 7-day period (in ISO format, e.g., 2023-03- 08T00:000000000007,,, "type": "string", "default": ""]
39

---

Published as a conference paper at ICLR 2026
Table H.8: System Instruction Case 2 of ResT Models generalizing irrelevant tool detection goals (Continued).
User Prompt
< user
Thank you for the information. It's alarming to see such high levels nitrous oxide.
Could you explain the impact of nitrous oxide on the environment and human health in
more detail?
/user
< think >
T should directly respond to the user's need.
< /ttiink >
response >
Certainly. Nitrous oxide, often known as laughing gas, has several significant impacts on both the environment and human health.
Environmental Impact:
1. Greenhouse Gas: Nitrous oxide is over 300 times more potent as greenhouse gas than carbon dioxide. It can trap significant amounts of heat in the atmosphere, thereby contributing global warming and climate change.
2. Ozone Layer Depletion: Nitrous oxide is also the most significant ozone-depleting substance being emitted today. It breaks down and destroys the ozone layer, increasing the Earth's 's exposure to harmful ultraviolet (UV) radiation.
Impact on Human Health:
1. Respiratory Issues: Inhalation of nitrous oxide can lead to oxygen deprivation, which can affect both the brain and the heart. Reduced oxygen can lead an array respi- ratory issues, including shortness of breath, wheezing, and potential long-term merg-term harm to overall lung health.
2. Neurological Issues:
Nitrous oxide known to affect the nervous system. It can cause harmful neurological impacts, including dizziness, loss of motor control, and in severe cases, a risk of seizures or coma.
3. Vitamin B12 Deficiency: Prolonged exposure to inhalation of nitrous oxide can inhibit the body's 's ability to use vitamin B12, leading to vitamin B12 deficiency. This can result in anemia and potential neurological damage.
Always be aware of your environment and attempt reduce nitrous oxide emissions where possible. This can include practices like reducing motor vehicle emissions and being mindful of agricultural practices. /response
< user >>
Thank you for the detailed explanation. It's quite concerning how much nitrous oxide affects both our planet and our health. Could you also provide some information on which countries are the largest emitters of nitrous oxide?
s/user
40

---

Published as a conference paper at ICLR 2026
Table H.9: Response case 2 of Qwen3-1.7B (ResT) generalizing irrelevant tool detection goals.
<think>>>
Okay, the user asked for information the largest emitters of nitrous oxide. Let me think about the tools available. The Country Information API can retrieve country details, including capitals, currencies, and flags. But the user wants specific data emissions, which the API might not provide directly. The API's description says it retrieves country information, but the parameters data. The user might need data that's not covered by the available tools. Since the tools don't include specific emissions API, I can't fetch that information. should inform the user that the current tools can't 't provide that data and suggest they check official sources like the EPA WHO. But wait, the user might expect the Country Information API have some data. Let me check the API's description again. It says "Retrieve information about countries, including their capitals, currencies, and flags." There's no mention of emissions. So, the answer is that the tools can't 't provide emission data. I need to respond that accordingly.
empty. Hmm, maybe the API doesn't 't have emission
/think
response
The available tools do not provide specific data on nitrous oxide emissions by country. For detailed information the largest emitters, you would need to consult official sources such as the Environmental Protection Agency (EPA) the World Health Organization (WHO), which track greenhouse gas emissions globally. These organizations publish comprehensive reports on emissions by country, sector, and year. /response >
41

---

Published as a conference paper at ICLR 2026
Table H.10: Response case 2 of Qwen3-4BB-Thinkinn2507 (ResT) generalizing irrelevant tool detection goals.
< think > Okay, the user is asking about the largest emitters of nitrous oxide. Let me think.
First, need to check which tool can provide country information. The available tools are Nitrous Ozone Levels API, Country Information API, and Earthquakes API. The Country Information API retrieves info like capitals, currencies, flags, but does it include emissions data?
Wait, the Country Information API's description says it's for country info including capi- tals, currencies, flags. It doesn't i mention emissions. The Nitrous Ozone Levels API gives current levels but country-specific data. The earthquakes tool is unrelated.
Hmm, the user is asking for the largest emitters of nitrous oxide. The tools provided don't have specific one for emissions by country. The Country Information API might have that data either.
Wait, maybe the user expects me use the Country Information API but that's not the right tool here. The problem is that the available tools don't 't include nitrous oxide emissions by country API. The user's question is beyond the scope of the provided tools. In the previous interactions, the assistant used the Nitrous Ozone Levels API to get current levels, but that's global data. The user now wants country-specific emissions data, which isn't covered by the existing tools.
So, since none of the tools can retrieve country emissions data, should inform the user that can't t provide that information with the current tools. need to respond that don't have access to that data and maybe suggest they check other sources. /hhnkk
< response
I don't have access real-time country-specific nitrous oxide emissions data through the available tools. The tools can use (Nitrous Ozone Levels API, Country Information API, and Earthquake API) don't 't' provide emissions statisties by country. For this type of environmental data, you might want to consult sources like the Environmental Protection Agency (EPA), the Intergovernmental Panel Climate Change (IPCC), the United Nations Framework Convention on Climate Change (UNFCCC). Would you like me to help with something else that *can address with the available tools?
/response >>
42

---

Published as a conference paper at ICLR 2026
Table H.11: Response case 2 of Qwen3-8B (ResT) generalizing to irrelevant tool detection goals.
< think Okay, the user is asking about the largest emitters of nitrous oxide. Let me think. I remember that nitrous oxide emissions are influenced by factors like agriculture, industrial activities, and energy production. But wait, the tools available here are the Nitrous Ozone Levels API, Country Information API, and the Earthquakes API. Hmm, none of these tools directly provide emission data by country.
The Nitrous Ozone Levels API gives current levels but not emission sources. The Country Information API provides general country data but not specific emissions. The Earth- quakes APT is unrelated. So, I can't the provided tools to get the exact countries with the highest emissions. need to inform the user that I can't 't retrieve that specific informa- tion with the current tools. But maybe user might be expecting an answer, so should clarify the limitation and provide available alternatives or sources where they can find that info. Let me check if there's any way to use the existing tools indirectly. No, I don't i think so. So the best approach is to apolo- gize the limitation and suggest reliable sources like the IPCC reports or environmental agencies.
offer general knowledge instead. Wait, the
/think
< response >
Unfortunately, don't have access real-time emission data country-specific nitrous oxide emission statistics through the current toolset. However, can share that agricultural activities (e.g., fertilizer use), industrial processes, and combustion of fossil fuels major global sources of nitrous oxide emissions. For detailed country-level data, you might refer to reports from organizations like the Intergovernmental Panel on Climate Change (IPCC) or the United Nations Environment Programme (UNEP), which track greenhouse gas emissions globally. //response
43

---

Published as a conference paper at ICLR 2026
Table H.12: Response case of Qwen3-14B (ResT) generalizing irrelevant tool detection goals.
<think
Okay, the user is asking which countries are the largest emitters of nitrous oxide. Let me think about the tools have. The available tools are the Nitrous Ozone Levels API, Country Information API, and the M4.5+ Earthquakes API. None of these tools directly provide data on countries' emissions of nitrous oxide. The Nitrous Ozone Levels API gives current and historical levels but not by country. The Country Information API provides general country data but not emissions. The Earthquake API is unrelated.
Since of the tools answer the user's question, I need inform them that I can't retrieve that specific information. I should apologize and explain that my current capabil- ities don't include accessing emission data by country. Maybe suggest they check sources like the IPCC or EPA for detailed reports. need to make sure the response is helpful and guides them to reliable external resources. /thin
response
appreciate your concern, and you're absolutely right that nitrous oxide emissions are a eritical issue. However, I currently don't have access to tools data sources that directly retrieve country-specific nitrous oxide emission rankings. For detailed informa- tion on national emissions, I recommend consulting authoritative sources like: *#C Reports** (Intergovermmental Panel on Climate Change) **EPA's Greenhouse Gas In- ventory** (U.S. Environmental Protection Agency) **UNFCCC National Reports** (United Nations Framework Convention on Climate Change)
These organizations track emissions by country and provide breakdowns of sources like agriculture, industry, and fossil fuels. Would you like help finding links to these resources? /response
Table H.13: Response case 2 of Llama3.2-3 2-3B-Instruct (ResT) generalizing to irrelevant tool detec- tion goals.
< think >
To provide information the largest emitters of nitrous oxide, will the Country Information API to gather data on countries that emit significant amounts of nitrous oxide. //hinkk
< response
Unfortunately, the Country Information API does not provide specific information ni- trous oxide emissions. It primarily focuses on providing general information about coun- tries. including their capitals, currencies, and flags. If you re looking for information on greenhouse gas emissions, recommend consulting the United Nations Framework Con- vention Climate Change (UNFCCC) other reputable sources that track and report greenhouse gas emissions by country. </response
44

---

Published as a conference paper at ICLR 2026
1 USE OF LARGE LANGUAGE MODELS
We used a general-purpose large language model solely to polish the English writing of this manuscriptt-speeifcallly grammar, wording, and stylistic clarity-and only in the Introduction and Conclusion sections, after all research design, experiments, analyses, and writing were completed. The model was not used generate ideas, datasets, code, algorithms,, proofs, or empirical results; it was not used for code synthesis or editing, image creation or editing, figure design, LaTeX equa- tion formula derivation, mathematical reasoning steps. data analysis, statistical testing, literature search, citation generation, surface-level copy-editing of author-written text. No proprietary. sensitive, or personal data were provided beyond our own draft sentences for editing. All suggested edits were reviewed, verified, and, when appropriate, further revised by the authors, who take responsibility for the accuracy, originality, and integrity of the final manuscript.
reference formatting. Interactions with the model were limited to
45


Reviews:
[{"review_id": "H7g4Fomxmu", "reviewer": "Reviewer_m8vW", "summary": "This paper introduces an entropy-aware RL (ResT) method to improve tool-use in large language models. Traditional approaches rely on sparse, outcome-based rewards that cause instable and inefficient training. ResT tackles this by linking token entropy to policy-gradient variance, showing that emphasizing low-entropy tokens (like tool names and parameters) stabilizes learning. It reshapes gradients to weight these structured tokens more early on, then progressively increases focus on reasoning tokens through a curriculum-learning schedule. Experiments on benchmarks like BFCL and API-Bank show ResT outperforms prior methods (including GRPO and GPT-4o).", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Strong theoretical foundations: This paper provides a thorough theoretical motivations and proofs to the method, rigorously deriving the relationship between token-level entropy and policy-gradient variance,\n* Extensive experiments: This paper provides extensive experiments, including different sizes of models and training methods, which provides a well-established evidence to the effectiveness of the method.\n* Innovation methods by considering different types of tokens, providing a different view of Tool RL.", "weaknesses": "Limitations:\n* The paper does not compare SFT+ResT against ResT alone. Since SFT can produce a different token distribution, it may alter the entropy of certain token types. For instance, some function words might exhibit lower entropy after SFT, potentially requiring adjustments to the training curriculum.\n* It remains unclear whether larger models inherently generate lower-entropy outputs—particularly for chain-of-thought tokens—which could reduce the benefit of grouping tokens by entropy type.\n* Regarding the partition mechanism, the theoretical framework does not constrain how tokens are grouped, as long as distinct entropy characteristics exist. Other grouping strategies, such as those based on word frequency, might also be effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7GxFs3US19", "reviewer": "Reviewer_fPpy", "summary": "This paper introduces ResT (Reshaped Token-level policy gradients), a reinforcement learning approach to improve LLM tool-use capabilities by addressing the high-variance credit assignment problem in multi-turn tasks. The authors observe that in tool-using scenarios, success hinges on generating a few critical structured tokens. ResT decomposes a multi-turn interaction into single-turn steps and redistributes reward feedback at the token level, with an emphasis on low-entropy tokens that determine task success. ResT therefore reweights the policy gradient to focus on these high-signal tokens, and employs a lightweight curriculum: early training strongly rewards structural correctness (format, tool invocation tokens), then gradually up-weights semantic reasoning tokens as the agent improves. Experiments on the BFCL and API-Bank benchmarks demonstrate state-of-the-art results.", "questions": "1. By design, ResT upweights low-entropy tokens (tool names, parameters) which makes the policy more deterministic on those critical parts early on. While this improves short-term stability, is there a risk of reduced exploration or diversity in the model’s behavior? For instance, could focusing too much on format and tool tokens lead the agent to ignore alternative valid strategies or outputs (especially if the initial reference is suboptimal)? The current experiments show longer outputs and no collapse, which is encouraging. Still, could the authors clarify how ResT balances exploration vs. exploitation? Was any entropy bonus or regularization used alongside the token reweighting (common in PPO-based methods), or any schedule to anneal the focus to avoid overly greedy policies?\n2. The method uniquely includes the model’s chain-of-thought (reasoning) tokens in the training signal, gradually increasing their weight later in training. This is an interesting design choice – essentially encouraging the model to not only get the final tool API call correct, but also to produce the intermediate reasoning. Could the authors provide more intuition or evidence on why weighting the reasoning tokens helps? The ablation, turning off CoT gradients, shows a performance drop, confirming its utility. Is the idea that by training the model to articulate its reasoning, even though those tokens don’t directly affect the environment, we guide it to a better policy for the tool calls? If so, might this relate to known benefits of chain-of-thought prompting for decomposing tasks?\n3. This is practical for tasks like API calls where an exact answer is known, but how would ResT apply to more open-ended tasks or learned rewards?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. A clear identification of the high-variance reward problem in tool-use and a principled solution grounded in theory.\n2. A method that is relatively simple to implement, no additional model modules, just weighting of gradient contributions, yet effective, which is a practical advantage.", "weaknesses": "1. Missing discussion of token-level feedback and curriculum RL literature, which raises concerns that the contribution might be incremental relative to those.\n2. The approach is somewhat domain-specific in its current form: it relies on known ground truth outputs to compute rewards for each token category, which may not directly extend to tasks without a deterministic reference solution.\n3. While the theoretical results are solid, the practical benefit of the optimal weighting vs. simpler heuristics (e.g. just heavily reward tool tokens) could be discussed more; the authors did compare to a static weighting baseline and showed improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ieLSK3uVQ1", "reviewer": "Reviewer_Nagq", "summary": "The work is a modification on GRPO that reshapes token-level policy gradients using entropy-aware weights computed over regions of the response that are of interest to a tool call, such as format tags, tool names, parameters, chain-of-thought, so that so lower-entropy, reward-critical tokens get higher weight. This is combined with a curriculum that increases weights on reasoning/parameter tokens as training progresses, smoothly shifting focus from structural correctness to sophistication of reasoning. Building on top of veRL, this technique achieves achieves state-of-the-art on BFCL and API-Bank.", "questions": "veRL is a complex codebase. I would appreciate the authors highlighting the changes they have made branching off of v0.5 to facilitate future replication and adoption efforts.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The work is reasonably well written with no major grammatical or formatting errors. The schematics are easy to understand and the paper is easy to read. \n* The work theoretically establishes a principled connection between token-level entropy and gradient variance which grounds the approach well. I've checked the proofs roughly and they appear correct\n* The problem is important (tool-call agents are of great interest in the community, and sparse reward is an on-going problem) and the essential ideas intuitively make sense (we need to get the calling conventions right before focusing on argument semantics)", "weaknesses": "The experimental results, though extensive, does not unequivocally establish ResT as a state of the art. It's unclear whether the accuracy gap falls within the variance or not (https://www.anthropic.com/research/statistical-approach-to-model-evals), and I cannot find an authoritative benchmark on either BFCL multi-turn or API-Bank that can support the SOTA claim. The results do look strong, just not conclusively strong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)