Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learning
Authors: Yifei Chen
Guanting Dong
Zhicheng Dou
Venue: ICLR 2026 Poster (2026)
Keywords: reasoning model
tool-integrated reasoning
self-evolved training
information entropy
Primary area: foundation or frontier models, including LLMs

Abstract: Tool-Integrated Reasoning (TIR) enables large language models (LLMs) to enhance their internal reasoning ability by integrating external tools. However, models with TIR often exhibit suboptimal behaviors, including insufficient tool calls, excessive tool calls, and overthinking after receiving tool call results. How to empower LLMs to perform TIR efficiently and accurately, while stabilizing the reasoning process, remains an open challenge.
In this paper, we first analyze the impact of tool calls on model reasoning from the perspective of information entropy. We find that when tool call results are provided, the information entropy of subsequent reasoning content will show a clear trend of change, and the overall information entropy of the reasoning chain will vary depending on the number of tool calls. Based on these observations, we propose Tool-Light, a framework designed to encourage LLMs to perform TIR efficiently and accurately. Our framework consists of dataset construction and multi-stage fine-tuning. For dataset construction, we use the trained model for continuous self-evolved sampling, integrating two methods: vanilla sampling and entropy-guided sampling. At the same time, during the sampling process, we design strict criteria for selecting positive-negative pairs. For the training process, we introduce a two-stage method, which includes a Supervised Fine-Tuning (SFT), and Self-Evolved Direct Preference Optimization (DPO).
Test results on 10 datasets reveal the effectiveness of Tool-Light, significantly improving the efficiency and accuracy of the model in completing TIR tasks.

Published as a conference paper at ICLR 2026
TOWARD EFFECTIVE TOOL-INTEGRATES REASONING VIA SELF-EVOLVED PREFERENCE LEARNING
Yifei Chen, Guanting Dong, Zhicheng Dou" Renmin University of China
(zhangboguodong, dou)@ruc. cefeeruc.duu.com edu.cn
ABSTRACT
Tool-Integrated Reasoning (TIR) enables large language models (LLMs) en- hance their internal reasoning ability by integrating external tools. However, mod- els with TIR often exhibit suboptimal behaviors, including insufficient tool calls, excessive tool calls, and overthinking after receiving tool call results. How to empower LLMs perform TIR efficiently and accurately, while stabilizing the reasoning process, remains open challenge. In this paper, we first analyze the impact of tool calls on model reasoning from the perspective of information entropy. We find that when tool call results are provided, the information en- tropy subsequent reasoning content will show a clear trend of change, and the overall information entropy of the reasoning chain will vary depending the number of tool calls. Based on these observations, we propose Tool-Light, framework designed to encourage LLMs to perform TIR efficiently and ac- curately. Our framework consists of dataset construction and multi-stage fine- use the trained model for continuous self-
tuning. For dataset construction, evolved sampling, integrating two methods: vanilla sampling and setropy-guudedd sampling. At the same time, during the sampling process, we design strict cri- teria for selecting positive-negative pairs. For the training process, we introduce a two-stage method, which includes a Supervised Fine-Tuning (SFT), and Self- Evolved Direct Preference Optimization (DPO). Test results on 10 datasets re- veal the effectiveness of Tool-Light, significantly improving the efficiency and accuracy of the model in completing TIR tasks. Our codes are released at http://hhbbb.ccm/m//c////////r/-
INTRODUCTION
Recently, large language models (LLMs) have demonstrated strong reasoning abilities after train-
ing (Team et al. 2025 Li al. 20251 Huang & Chang 2023 Sun et al. (023). They excel in tasks
such mathematical reasoning (Shao et al. al. 2024 Yuan al. [2023] Li al. 2024b), logical puz-
zles (Xie et al., (205), and code generation (Zhong et al. 2024 Li & Wang 2025). However, when faced with more challenging tasks (such as deep information retrieval and precise computation (Min) et al. 2024 Qin et al. 2024)), those trained L. often struggle when relying solely on their in- ternal reasoning capabilitics. To solve those problems, Tool- Integraaee Reasoning (TIR) (Paranjape) al. 2023 Gou et al. 2024 Li al. 2025b Wang et al. (202dd) has emerged. Compared to general reasoning models, TIR models can autonomously utilize external tools during the reasoning process to compensate for the deficiencies in their internal knowledge abilities (Chen et al. 2025a Song
jet al. 2025 Li et al. 2024a 2025b Dong et al.) b).
While introducing external tools enables models enhance performance reasoning tasks, there still exist challenges, such as unreasonable tool calls or overthinking. For example, during the TIR process, the model often exhibits suboptimal tool call patterns, such excessive insufficient tool calls (Wu et al.] 2025bb Bai et al.) (2025). Additionally, when low-quality tool call results are provided, models may experience overthinking or even analysis paralysis (Zheng al.) 2025 Sui et al. 2025 Chen et al. 2025b). We collectively refer to them as incorrect tool calls. Recent studies have focused on optimizing tool calls with the help of reinforcement learning (RL). These
"Corresponding author.

---

Published as a conference paper at ICLR 2026
studies include scenarios of using a single tool and scenarios of using multiple tools (Bai et al. 2023 Feng et al. 2025a Wu et al.] 2025b] Wang et al., 2025e; Dong et al. 2025a Wang et al.
2025ap. Nevertheless, existing studies typically focus the issue of tool-overuse, neglecting both tool underuse and the impact of tool call results on subsequent reasoning processes (Qian et al.
2025 We consider that these works do not comprehensively address incorrect tool call issues.
At present, many works have analyzed reasoning tasks from the perspective of information en- tropy (Cui et al. 2025] Wang et al. 2025c). They find that the high-entropy part of the reasoning chain often determines reasoning direction. Inspired by these works, we conduct preliminary exper- iments and analyze the TIR task's information entropy. We find that after receiving tool call results, the information entropy of the model's 's subsequent outputs will fluctuate. We also observe that when multiple correct reasoning paths exist, the path with fewer tool calls tends to have lower overall entropy distribution compared to those with more. Based on the limitations of existing work and the TIR task's s entropy characteristics, we attempt to explore the efficiency of the TIR task from the following two perspectives:
From the training perspective: Can we improve the effectiveness of tool call in TIR tasks from the perspective of post-training (including the algorithm side and the data side)?
From the inference perspective: How can we apply the entropy distribution characteristics of the model's TIR process to guide data sampling?
In this paper, we propose Tooh-Light, framework that optimizes the reasoning capability and tool call effectiveness of TIR models from the data construction and algorithmic perspectives, From the data construction perspective, we introduce an innovative entropy-guided sampling method. This approach first generates main chain, then branches from the highest-entropy positions to create multiple paths. We then apply sampling criteria to derive high-quality positive-negative pairs. To ensure data diversity and balance, integrate this approach with direct sampling, producing hy- brid method that sustains performance while improving tool-call effectiiveness. From the algorithmic perspective, propose the "Two-stage TIR Training" pipeline. First, we perform supervised fine- tuning (SFT) on the original model, and then conduct self-evolved DPO training. In self-evolved DPO stage, we divide the training into two parts: Pre-Aligned DPO Training and Self-Evolved DPO Alignment. The former is used to improve the model's 's reasoning ability while reducing the re- dundant tool calls. The latter alternates sampling and training process, enabling the model to learn necessary tool call. At the same time, the difficulty of the training data and the model's 's capability can be adapted, thereby maintaining its original reasoning ability.
To comprehensively evaluate the capabilities of Tigh-Light, we use ten challenging reasoning tasks, including knowledge-intensive and mathematiiaal-ressoiing tasks. Tool-Light not only ensures sta- ble reasoning performance but also significantly improves the efficiency and accuracy of tool calls. In summary, our contributions are as follows:
We explore and analyze the TIR paradigm from the perspective of information entropy, demon- strating the connection between TIR effectiveness and entropy change.
We propose an innovative entropy-guided -uudddd sampling strategy, which is combined with two-stage training method incorporating a self-evolution mechanism, thereby enhancing the effectiveness of the TIR process.
Experiment results across 10 challenging reasoning datasets prove the effectiveness of Tool-Light. Further quantitative analyses offer practical guidance for efficient tool-integrated reasoning.
2 RELATED WORK
Effective Tool-integrated Reasoning. Tool--Itegrated Reasoning (TIR) refers to guiding models
to leverage external tools during reasoning (Yang et al. 2025). Teaching models to use tools cor-
rectly and rationally is core challenge in TIR (Su et al. 2025a Fang et al. 2025 Li et al. 2025d c
Tao et al. 2025 Li et al., 2025j] Qiao et al. 2025 Qin et al. 2023 Schick et al. 2023
Lu et al.
2023). Currently, fine-tuning the main method to guide the model in efficiently completing the
TIR task (Li et al. 2025e). For example, IKEA (Huang et al., 2025) and SMART (Qian et al.) 2025)
design training methods based metacognitive theory (Kontostavlou & Drigas 2021 focusing on
2

---

Published as a conference paper at ICLR 2026
Entropy Distribution Comparison Step 2
Step
1
Step 3
Step 4
Marr Zess
Max
Mire
1.4
Moor
Entropy Value
10
3--
1.2-
0.0
0.4
0.6
0.4
0.4
011
15
70
145
140
175
45
50
125
180
225
43
85
129
172
215
30 66 96
132
Token Index
Token Index
Token Index
Token Index
Figure 1: The entropy distribution of tool-integrated reasoning inference. "Step" refers to the rea- soning steps-free foroffrmmm the model starting inference or receiving the result of the previous tool call, up until the next tool call is triggered or the final answer is reached-including the process of triggering the tool call (but not including the result of the tool call). Step 1-4 thus represents the process of intermediate reasoning in the first, second, third, and fourth segments. "Token Index" represents the position of a token within that step. For example, indez, represents the subscript index of the i-th output token in certain step.
the model's knowledge boundaries. Self-DC framework (Wang et al. 2025b) controls the model's self-behavior by leveraging internal signals. ToolPAefer-LLaMA (Chen et al. 2025c) uses the DPO algorithm to update LLM strategies. Additionally, some works efficiently complete the TIR task by optimizing the reinforeement learning training process. For instance, Search Wisely (Wu al. 20256) and OTC (Wang et al., 2025a) carefully designed reward functions during training, while CoRT (Li et al. 2025a) meticulously optimized the training process. However, most existing work focuses solely on reducing the excessive tool calls and does not explore the accuracy of tool calls and the reasoning process within TIR. We consider that the effectiveness of TIR includes not only reducing redundant tool calls, but also invoking necessary tools. Moreover, it also involves mini- mizing overthinking during the reasoning process and avoiding analysis paralysis. In this paper, we do not limit our discussion to reducing redundant tool calls, but explore the effectiveness of TIR in a broader range of scenarios.
Self-Evolved Methods in LLMs. Due to the variability and complexity of world knowledge, mod- els need to continuously evolve to better adapt to the real-world tasks (Wu et al. [2025a] Qiao et al.) 2025 Dong et al., 2025c et al., 2025dd] Tao et al., 2025). The self-evolution strategy refers to paradigm for dynamically improving model output performance through continuous learning and adaptation (ang Gao et al. 2025). can divided into two methods: parameter-updatiig and in- context learning. The former involves having the model autonomously generate training data to fine- tune weights, or using the experience from interacting with the environment to drive model train- ing(Zhou al., 2025 Simonds al. 2025 Zeng al.) 2025 Su al. 2025b). The latter achieves model self-evolution through the optimization of prompt words or memory mechanisms Chhikara al. 2025; Yükkekgönül al., (0044). Referring to existing research, the of Tool-Light lies
in enabling the model to learn from data generated by itself(Li et al. 2025g). At the same time, it continuously generates better training data to perform parameter-updating self-evolution.
PRELIMINARIES
(1) Problem Definition. Multi-TIR aims to enable the model to autonomously call multiple tools, thus completing different complex tasks. Specifically, given the input instruction I and the model 0, the final output answer y of multi-TIR& can be expressed in the following form:
N y ~ Poly Rx) Po(R Tees R),,
where represents the reasoning step, N represents the total number of reasoning steps, Rgú Tost and R; represent the reasoning content before the i-th step, the tool call result before the i-th step,
3

---

Published as a conference paper at ICLR 2026
and the reasoning content of the i-th step, respectively. For the tool T, it may contain variety of different tools in multi-TIR tasks.
(2) Fre-Experiment. In this section, explore the relationship between the TIR path and the distribution of information entropy. First, we present the formula for calculating the information entropy distribution:
N 111) - - PlrnieeiNooooolllllleeee
where N represents the vocabulary length the i-th position, yęi and Yje represent the sequence before position i and the j-th token at position i, respectively. We use the Search-R1 (Jin et al. 2025a) model perform TIR on multiple QA datasets. For each sample, we rollout ten chains and classify them into two groups based on the tool calls: more tool calls (more) and fewer tool calls (less). We record the average entropy distribution for each step. Our results can be found in Figure
From the results of this experiment, we can draw some conclusions:
When the model receives tool call result, its output information entropy initially rises, then fluctuates, and finally drops sharply before the next tool call arrives.
For the same sample, low-entropy chains tend to involve fewer tool calls, and as reasoning pro- gresses, tool calls" difference between high and low entropy chains becomes increasingly evident.
These conclusions guide our subsequent data sampling.
(3) Tool Design Inspired by the existing work, choose the most mainstream tools, the code interpreter tool and the search tool, for exploration (Dong et al. 2025aa] Wang et al. 2025a Li et al., 2025h Jin et al. 2025a). The code interpreter tool receives a piece of code, compiles it, and returns the compilation result or error message. The search tool receives a query, and then retrieves the content most relevant to the query from the local knowledge base or web page and returns it.
METHOD
Overview. We propose Tool-Light, multi-stage training pipeline aiming improve the effective- ness of model tool calls. As shown in Figures [2n Tool-Light consists of two key components: (1) Dataset construction, which includes carefully designed sampling strategies screen out train- ing data. (2) Two-stage TIR training paradigm. which trains the model successively with SFT and self-evolved DPO training. In the self-evolved DPO training stage, we design pre-aligned DPO training and self-evolved DPO alignment stages to gradually improve the model's capabilities.
4.1 DATASET CONSTRUCTION
In this section, a detailed introduction to the data construction and sampling strategies is provided.
Source Data Construction. In this section, we first obtain the source data for subsequent sampling. To improve the model's ability to ex- ecute TIR tasks, we adopt the SFT data from existing work, defined 2025a). We use Den to train the instruct model, resulting the model Mst. Then we use the in- struct model to perform inference on Dan- Dur- ing this process, we do not offer the model ex- ternal tools. After that, we only retain the data where the model's ''' inference answers are incor- rect, resulting in Dsouce- This process can be modeled as:
Continue TIR...
Answer
Continue TIR...
Answer
Dan (Dong
al.
Question
Contizue IIR...
Answer
Centine TIR...
Answer
Figure 2: The overall process of entropy-guidee sampling. Gray and red positions represent tool calls and the branching positions, respectively.
D Dan + Vinference].
where Yaaell and Yinference are the golden answer and the result obtained from direct inference on that sample, respectively. Ylabel o Jinference means that only retain data where the model's inference
4

---

Published as a conference paper at ICLR 2026
Stepl: Supervised Fine-Tuning
Step3: Self-Evolved DPO Alignment
M Mutrrcc
Supervised Fine Tuning
M
Iterative DPO
Dinect Inference
Daource
sft
M dpo1
Mapp
Step2: Pre-Aligned DPO Training
Mer
Direct Preference Optimization
Sumpling Strilegies
sample DPD pairs
Vanilla
Sampling Stritegies
dpos
Entropy guided
Criteria2
Vanilla
Encropy guided
Dápo
Cri Criterial
Dap
Figure 3: The overall structure of Tigh-Light's training pipeline. Among them, the Self-Evolved DPO Alignment stage will conduct multiple rounds of training.
result is incorrect. We consider these data to be more challenging, and training on them can help improve the model's reasoning capabilities.
Sampling Strategy Design. After obtaining Doouree use Mst to perform TIR inference on it. For each question, we sample multiple TIR paths. The final set of all inference paths is denoted Dapo- For this step, carefully design two sampling strategies, and mix the data from both strategies to construct high-quality training data. The specific strategies are described as follows:
(1) Vanilla TIR Sampling. We guide the model to generate multiple paths for each sample. This helps us construct positive-negee negative pairs. This process can be expressed as follows:
Dioo tuly Maa [f(g] q soouree
where D represents the sampling results generated under vanilla sampling, I, and y represent the prompt used, each question D source. and the output result from Man. respectively.
(2) Eutropy-Guidedd Sampling, Vanilla sampling strategy incurs high inference costs and inference uncertainty. In this section, as shown in Figure 2
we propose an cntropy-guided sampling strategy.
Entropy Distribution Calculation. We begin by generating primary reasoning chain, denoted as Cmmin- Each segment of the chain is considered a step. According to existing research (Liao let al. (025). the initial inference step has a significant impact on the subsequent outputs. There- fore, for each step, we compute the entropy distribution at each token position in the first 10, 20, 30, 40, and tokens. The average information entropy distribution for a subsequence is:
Havg(1) (i) = Σ H()),
(5)
where Hivg()) represents top-i tokens' average entropy distribution. We retain the maximum Hav(() (1) and its corresponding sequence length at each step.
Branch Sampling Execution. We pick the top-k steps with the highest entropy, along with their sampling positions. At each position, we guide the model to continue generating several branches. Since those subsequences have higher information entropy. based on previous research (Wang et al. 2225cc Cui et al. 2025 Cheng et al.) 2025), it is more likely to produce branch outputs with higher diversity. This process can be modeled as follows:
Dipo
{ul (g) + veil. )soure),
where D2 dpo and Ysi represent the dataset guided by entropy sampling and the subsequent sam- pling output, respectively. [(q) @ yes represents concatenating the prompt and the existing se- quence together as the input. y represents concatenating y>i and yev together. Assuming m
5

---

Published as a conference paper at ICLR 2026
rollout times and average sequence length of n, this method, in the ideal scenario, reduces computational complexity from (mm) to O( logm). m). We collect all the main chains and branch chains, and construct the set of outputs based on tree-structured sampling.
Finally, Pap and 28 mixed in certain ratio obtain the final dataset Dapo for the positive- negative pairs construction.
4.2 TWO-STAGE TIR TRAINING PARADIGM
Based on existing research (i et al. 2025g Dong et al.) 2025a Song et al.] 2025), we propose two-stage self-evolved training pipeline to gradually boost the effectiveness and stability of the model's TIR process. The specific pipeline is shown in Figure
Supervised Fine- Tuning First, we obtain the SFT model Maft with the same setup as in Tool- Star (Dong et al. 2025a). The SFT's loss function is: LSrr(0) "(a))) log (u( As shown In the first step of Figure 3 this step aims help the model quickly acquire the ability to complete TIR tasks.
Self-Evolved DPO. Inspired by existing works (Li et al. 2025g Wu at. we further fine-tune Man through a self-evolved DPO strategy, using a loss function as follows:
2023 Tao al. 2024).
log a log To (yo (r)
Teer ( ]
Ty (yn (x)
(Tp; Teet)
B Blog
(7)
where rer is the original strategy model, and 3 is the hyperparameter and is the sigmoid function. The primary objective of DPO is to maximize the relative probability of the model generating compared to y! This step aims endow the model with the capability finish TIR tasks accurately and efficiently. First, we establish a specialized data sampling criterion Cri,. It is used to obtain training data Dapo from Man using both vanilla and untropy-guided strategies. We then train Man on Dapo to yield Mapo1 Subsequently. a new criterion Cri, is defined, guiding Mapot to sample the next iteration of training data Dapo² We train Mapo1 on this dataset and get Mapaz- After that, we update Mápol + Mapo2 and iteratively repeat the sampling and training steps until performance converges. Details these two criteria follow in the subsequent sections.
(I) Pre-Aligned DPO Training. In each sample, we first categorize correct or incorrect trajec- tories based on the Fl score. Only trajectories with F1 score considered correct, and only trajectories with Fl score are considered incorrect. Then, we divide the samples into a hard set and easy set. After that, referring the conclusions in Section 3 design criteria for sampling positive and negative pairs in the entropy-guided strategy:
Hard set: Samples with < 50% correct trajectories.
Easy Set: Samples with 2 50% correct trajectories.
Positive Example: Correct trajectory with the minimal number of tool calls and lowest entropy. If no correct trajectory exists, the corresponding SFT trajectory from Doorre used.
Negative Example: An incorrect trajectory with more tool calls than the positive example.
For vanilla sampling criteria, we choose samples with < 40% correct trajectories as the hard set, and those with 2 70% correct trajectories as the easy set. For each set, the shortest correct trajectory is the positive example, and the incorrect trajectory longer than the positive one is the negative example.
We set the hard set and easy set's data ratio 2:1. The goal of training in this round is to teach the model to reduce unnecessary tool calls while avoiding excessive reasoning.
(2) Self-Evolved DPO Alignment. We use Mapul to resample from Dapo 1. At the same time, we introduce the second criterion to filter out Dapo? 2 In Criy, samples whose number correct tra- jectories is less than half of the incorrect ones are classified into the hard set, while the criteria for the easy set remain unchanged. For each sample generated by Ma if is classified into the easy set, it means that after pre-aligned DPO training, Mapo1 has mastered the knowledge solve this

---

Published as a conference paper at ICLR 2026
problem. it is classified into the hard set, it means that this problem is still difficult for Mdyol For these two different situations, we have also designed the sampling criteria for positive and negative examples:
Negative Example in Easy Set: Incorrect trajectory with the most tool calls.
Positive Example in Easy Set: Correct trajectory with fewer tool calls and lowest entropy.
Positive Example in Hard Set: Correct trajectory with the longest reasoning chain.
Negative Example in Hard Set: Incorrect trajectory with the shortest reasoning chain.
Based on preliminary experiments, low-entropy outputs' tool calls are often fewer. Therefore, we also incorporate the consideration of low entropy into positive examples' selection. We use the sampled Dapo" 2 to continue training Mapot, and continuously sample with the trained model. Self- evolved DPO alignment will continue for several loops. This step is intended to equip the model with the ability to make necessary tool calls while maintaining its original efficient TIR capability. After several rounds of training, we obtain the final model Mapo2.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Dataset. We carefully select datasets and categorize them into two types: (1) mathematical-
reasoning tasks, including AIME24, AIME25 II AMC23 F MATH (Hendrycks et al. 2021)
MATH500 (Lightman et al. 2024), and GSM8K (Cobbe et al. (021). and (2) knowledge- intensive tasks, including HotporQA (Yang al., 2018), 2WikiMultiHopQA (Ho al. 2020 MuSiQue (Trivedi et al. 2022). and Bamboogle (Press et al. 2023).
Metrics. For mathemmatical-resoniii 2024]. utilizing the Seeen..572B-Insiiie model to evaluate answer correctness. For knowledge- intensive tasks, we use Fl score to evaluate answers. Additionally, we introduce two metrics: Effi- ciency and Necessity. (1) Efficiency is defined as Effi 1
tasks, we adopt the LLM-as-Judge approach (Gu et al.
Mr n Ei=1 T. where n is the total number of samples, M; the performance on the i-th sample, and Ti the number of tool calls. It captures the model's tendency to overuse tools. (2) Necessity is defined as Nece = M (1 2
Noo)), where N in is the count of methods for the i-th sample that make more tool calls than the current method but yield incorrect answers, N the count of methods with fewer tool calls yet correct answers, and M denotes Min-Max Scaling. It reflects the model's 's tendency to underuse tools.
Baselines. We classify baseline methods into two types: (1) Songle-TToo--nteraatee Reasoning, including Search-ol (Li et al. 2025t). Search-RI (Jin et al. 2025a), DotaMath (Li et al. 2024a), ToRI. (Li et al. 2025h), and Rooo (Feng et al. 2025b) ) and (2) monti-Tool-nntegratee Reasoning. including prompt engineering, ReCall (Chen et al. 2025a), and ToolStar (Dong al. 2025a
5.2
MAIN RESULTS
Our main results are presented in Table I and more details can be found in Appendix draw the following insights:
We can
(1) Proper use of external tools can be of great help to model reasoning. For example, when ex- method on the Qwen model has significant advan- tages over the direct inference method in mathematical reasoning tasks. However, for knowledge- intensive tasks, the effect of introducing external tools is reduced (only improved on MuSiQue dataset). This also indicates that untrained models cannot use external tools well.
ternal tools are introduced, the prompting
(2) While training enhances the model's TIR capabilities, single-tool training hinders the de- velopment of generalization abilities. Compared to using prompt engincering methods, fine-tuning
https::/huggiingfacct http:///nwwaaane/aaaaaaaaaa/aaaa
https://huggingt httpdaaasets//we////aaaa
More implementation details can be found in Appendix
7

---

Published as a conference paper at ICLR 2026
Table I: Results on 10 reasoning tasks, with top two results highlighted bold and underlined. Unless noted, Qwen2.5-7B- Innntteet-- is used as the backbone. For ToRL and ReTool, their RL mod- els generate training data inferences for SFT, which serves as the baseline. Abbreviation: 2Wiki. (2WikiMuliiHopOAA HQA. (HotpotQA). MSQ. (MuSiQue). Bamb. (Bamboogle).
Mathematical Shathemattial-Reesonnnng Tasks
Bhowledge-Intensigg Tasks
Method
Avg.
AIME24 AIME25 AMC23 MATH MATH500 GSM8K HQA 2Wiki. MSQ
Bamh
Direct Inference
Qwen2.5-7B.
0.0
6.7
30.0
68.6
57.2
71.4
26.1
25.6
7.9
36.5
33.0
Instruct
Llama3.1-80
0.0
3.3
15.0
52.8
33.4
75.0
16.2
13.7
7.4
23.2
24.0
Instruct
Single-TIR Methods
Search-ol
6.7
10.0
37.5
73.6
61.8
80.2
41.1
35.4
13.2
39.8
39.9
Search-R1
16.7
6.7
45.0
48.7
40.0
24.1
47.4
DotaMath
16.7
10.0
50.0
74.6
62.2
82.6
26.2
21.7
6.5
28.6
37.9
ToRL
30.0
26.7
67.5
87.0
80.2
89.2
41.3
35.4
9.5
36.9
50.4
ReTool
23.3
30.0
62.5
84.8
78.4
86.2
31.5
29.0
11.1
35.8
47.3
Multi-TIR Methods
Prompting- -Basid
6.7
13.3
47.5
73.8
62.2
69.4
21.1
23.8
9.9
25.5
35.3
ReCall
3.3
6.7
27.5
73.2
79.8
51.9
54.0
25.0
55.5
43.2
Tool-Star
30.0
26.7
65.0
85.6
77.2
89.4
54.7
55.7
22.8
56.6
Ours
Tool-Light (Llama)
10.0
6.7
30.0
59.4
56.8
76.6
41.3
33.5
12.2
41..3
36.8
Tool-Light (Qwen)
33.3
23.3
67.5
87.4
79.0
92.0
57.7
56.1
25.0
58.7
58.0
Efficiency
Necessity
Sequence Length Distribution Comparison
0.5
1.0
350
Took Star
Counts Ontpats
Eifelency Score
Necesolty Scoree
250
0.3
0.6
200
0.2
0.4
100
0.1
0.2
Search-ot Seardh-Ri R1
ReCall DataMath
Toll
grompting
Tool
Tool-s Star Teal-
Search-ot Search-Ri Ri
ReCall DataMath
Toll. Prompting
Teal
Teal Star
Light
1000
1500
2000
2500
3000
3500
4000
Sequence Length Distribution
Figure 4: Efficiency, Necessity, and Sequence Length differences between Tool Light and baselines.
can effectively teach the model use tools. For example, Search-RI outperforms Search-ol by 5.7 on average, while Tool-Star outperforms prompting-based methods by 21.3 on average. It is worth noting that models trained for single-TIR tend to have poor generalization performance. For exam- ple, Search-RI performs well on inowledge-intenivee intensive tasks but poorly on tasks, while ToRL shows the opposite pattem. The Tool-Star and Tool-Light frameworks perform well on both types of tasks, highlighting the generalization advantages of multi-TIR training.
(3) Multi-round DPO training allows for further enhancement the model's performance. Compared with the simple SFT method, Tool-Light framework additionally introduces two oper- ations: Pre-Aligned DPO Training and Self-Evolved DPO Alignment. This leads to an overall improvement in the model's performance. For example, in paathematical-resonnii Light achieves optimal performance on four datasets, and ranks among the top two on all datasets for Knowledge- Intensive tasks. Notably, by solely adopting the DPO method, we outperform most baselines that are trained using GRPO when comes to average performance. This fully demon- strates the effectiveness of training pipeline design.
tasks, Tool-
-etteegg--ntensive
8

---

Published as a conference paper at ICLR 2026
Step
Step
Step
Step d
1.0
1.2
1.4
0.8
0.6
0.8
0.0
0.8
2.8
0.4
0.2
0.2
0.0
0.0
0.0
0.0
35
70
105
140
175
33
59
132
186
51
162
155
204
255
0
120
172
215
Talora
Tolken Iodex
Tokem
Tabro badee
Figure 5: Output sequences' entropy distribution under different methods.
5.3 QUANTITATIVE ANALYSIS
Tool-Use Effectiveness Analysis In this section, we analyze the differences in TIR effectiveness between Tigh-Light and other baseline methods. As shown in Figure Tool-Light achieves the best performance in Efficiency and Necessity metrics, which indicates the superiority of our designed training process. Regarding the output sequence length metric, Tool-Light has shorter length dis- tribution than Tool-Star, and outperforms Tool-Star in both result correctness and reasoning effee- tiveness. shows that our training method can effectively reduce the phenomenon of overthinking. Detailed cases are list in Appendix D
Entropy Distribution Analysis As described in Section 3 in this section, we explore the impact of introducing the entropy-guided sampling strategy on final output results' information entropy distribution. The results are shown in Figure 5 The existing TIR models tend to produce output sequences with relatively high entropy. In contrast, Tool-Light results in lower entropy distributions for its output sequences. This might be due to the consideration of information entropy during sampling. Our training method introduces the learning low-entropy path outputs. Since high- entropy tokens can sometimes trigger excessive reasoning in the model, our method effectively mitigates the phenomenon of overthinking. This also aligns with the results in Figure
Impact of DPO Loop Number In this section, we investigate how the number of training loops in the self-evolved DPO alignment stage influences the model's s performance. For this stage, we increase the loop count from to 5 and record the Performance, Efficiency, and Neces- sity metries. The results are shown in Ta- ble Is 1-5 rows. As the number train- ing loop increases, all metrics reach op- timal values in the second training round and then decline. This may be because the self-evolved method is initially able obtain sufficient positive-negative pairs. However, as the evolution proceeds, there are fewer sample pairs beneficial for train- ing, and the model may overfit to the dis- tribution of the training set, leading to lower performance on the test set.
Ablation experiment results for various as- pects: 1/1 strategy ratio indicates 1:1 data ratio for two sampling strategies, p-r. and n-r denote random positive and negative example selection.
Table 2:
Method
Performance Efficiency Necessity
Tool-Light (2 loop)
58.0
0.44
0.75
Ablation for self-evolved Loops
w. loop
57 .9 (o.))
0.42 (-0.2)) 0.71 (-004)
loop
56.1 61.9)
0.39 (-0.5) (..0))
0.73 (-002)
loop
56 (-16)
0.37 (-007) 0.71 (0004)
w. loop
54.. (339)
0.36 (0.08) 0.72 (c003)
Ablation for Sampling Criteria
w. 1/1 data ratio
56.9GLD
0.44 0.76
W. p-r
53.6 (-.6
0.42 (4002)
0.63 (0.12)
53.9 (4.1)
0.41 (0.03) 0.74 (0.00)
Impact of Sampling Criteria In this section, explore the impact of sampling criteria on the training effect. We modify the origi- nal criteria from the following aspects: the data ratio of two strategies is 1:1, positive examples are
'We also explore the impact of different data ratios the final training results during sampling. The experimental results and analyses are provided in Appendix C

---

Published as a conference paper at ICLR 2026
randomly selected, and negative examples are randomly selected. The results are shown in rows 6 Table 2 We can find that the different data ratio has the least impact on the training effect, while the change in the positive-negative example selection criteria has greater impact the sults. This indicates that for the DPO training in Tool-Light, is very important to fully distinguish between positive and negative examples.
CONCLUSION
In this paper, propose Tool-Light, framework empowering models to efficiently and accurately complete TIR tasks. We first analyze the TIR process from the perspective of information entropy. Based on this analysis, introduce Tool-Light, framework improving model's TIR capabilities from both data construction and training aspects. Within this framework, we design a new entropy- guided sampling strategy. Based this method, we construct two-stage TIR training paradigm. The trained model achieves excellent results in terms of overall performance, tool calls effectiveness, and simplicity of the inference process. We confirm that our work can provide more assistance for future research on TIR.
7 ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China No.62272467, The work was partially done the Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE.
8 ETHICS STATEMENT
All authors of this work have strictly adhered to the ICLR Code of Ethics. It does not involve any research with human subjects, practices data set releases, potentially harmful insights, method- ologies and applications, potential conflicts of interest and sponsorship, discrininationbbbsfalieesss concerns, privacy and security issues, legal compliance, and research integrity issues.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of Tool-Light's experiments, detail all the experimental settings in Appendix B Among them, the data ratio based on the difficulty level of data sampling, as well as the data ratio between the vanilla sampling method and the entropy-guided sampling method, respectively provided in Section 4.2 and Appendix C
To further facilitate reproducibility, we provide detailed code in the Supplementary Material, includ- ing the entropy outropy-guided sampling, SFT and DPO. Upon acceptance, we will make our experimental data and model checkpoints publicly available.
REFERENCES
Huan ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi. Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Qihan Ren, Cheng Qian, Zhenhailong Wang, Minda Hu, Huazheng Wang, Qingyun Wu, Heng Ji, and Mengdi Wang. A survey of self-evolving agents: On path to artificial super intelligence, 2025. URL /htpps:/ arxiv. abss 258.2114667.6
Fei Bai, Yingqian Min, Beichen Zhang. Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang. and Ji-Rong Wen. Towards effective code-integrated reasoning. CoRR, abs/2505.24488 24.400, 2025, doi: 10.48550/ARXI..2555.222 24400. URL ptttp:://doi.org// 48550/ 2rxiv..205.24444 24480
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Pan, Wen Zhang, Huajun Chen, Fan Yang. Zenan Zhou, and Weipeng Chen. Research: Learning
10

---

Published as a conference paper at ICLR 2026
to reason with search for Iims via reinforcement learning. CoRR. abs/2503.19470, 2025a. doi: 00.48550/ARXIV.50....20
URL https://p/oiii org/10. 0005//...555//1 48550/arXiv www..i.ii.............. 19476
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. CoRR, abs/2503.095677, 2025b. doi: 10.48550/
ARXIIV2503.09567. URL htt.:htp:///// //doii: org/10. wttp 2503. 09567
Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang. Advancing tool-augmented large language models: Integrating insights from errors in inference trees, 2025c.
07115
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective, 2025. URL httpp:://axiiv. org/abs/2506. 14758
Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready Al agents with scalable long-term memory. CoRR, abs/2504. 19413, 2025. doi: 10.48550/ARXIV2550.11M
URL (tthhtts://doi.. http:://66o..rr///...48885/////0 2504. 19413
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1X.v:2110.14168. 2021.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism ing language models. CoRR, abs/2505.22617 2025. doi: 10.48500ARXXI.2255.28111 URL
reinforcement learning for reason-
https://doi.org: wtw.g//...85.////
48550/arXiiv 2505. 2561.22617
Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. Tool-star: Empowering llm-brained multi-tool reasoner via reinforcement learning. CoRR, abs/2505. 16410, 2025a. doi: 10.48550/ARXXV2555..1.44
URL https://dol. orwoorgg///1 10. 48550/arXiv. 2505. 16410
Guanting Dong, Hangyu Mao, Kai Ma. Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. Agentic reinforced policy optimization. CoRR, abs/2557..19845 2025b. doi: 10-48500/AXX1V2557.19944 URL https:://oii wrg/ 48w.0/a05500....... 2507. 2587.19849
Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Ji-Rong Wen, and Zhicheng Dou. Understand what LLM needs: Dual preference alignment for retrieval-auggennte generation. In Guodong Long, Michale Blumestein, Yi Chang. Liane sewin-Eytan,, Zi Helen Huang, and Elad Yom-Tov (eds.). Proceedings of the ACM Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025- 2 May 2025, pp. 3714717. URL https://doi. org 10. 1111/115541111 3714717
ACM, 2025c. doi: 10.1 10.1145/3696400.
Runnan Fang, Shihao Cai, Baixuan Li, Jialong Wu, Guangyu Li, Wenbiao Yin, Xinyu Wang, Xi- aobin Wang, Liangcai Su, Zhen Zhang. Shibin Wu, Zhengwei Tao, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Towards general agentic intelligence via environment scaling, 2025.
URL https://axxx wttp://wwww...///../.// 13311
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use Ilms,
2025a. UtL/htts:://axxx httt:p///aarxi..og/aa////////// org/ 11536
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. CoRR, abx/2504. 2025b. doi: 1044550/ARXXI.2504.1555 URL hottps://don org/
10. 48550/arXiv 2504 11536

---

Published as a conference paper at ICLR 2026
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang. Minlie Huang. Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: htttps:///ppneevieww http://wwww.eetttooomiii/d--oooo
Jiawei Gu, Xuhui Jiang. Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang. and Jian Guo. A survey llm-as-a-judee. CoRR, abs/2411.1559) 2024, doi: 1044550/ARXI.2411.55555 URL httthtts///oi.org////
48550/arXiv. 2411 15594!
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.). Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, Decem- ber 2021, virtual, 2021. URL oote
neur ips.rips.. cc/ html
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Núria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computa- tional Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pp. 6609- 6625. International Committee on Computational Linguistics, 2020. doi: 10. 1/53//60/ COLING-MAIN.555 URL https: org 10. 18653/v1/2020 htt.:22220.ccloooo coling main.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. The Tenth Inter- national Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A sur- vey. In Anna Rogers and Jordan L. Boyd-Graber and (eds.). Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9414, 2023, pp. 1049-106; 1065. Asso- URL
ciation for Computational Linguistics, 2023. doi: 10. https://doi.orgg 18653/v1/2023. findingsnninnn..cccc
Ziyang Huang. Xiaowei Yuan, Yiming Ju. Jun Zhao, and Kang Liu. Reinforced internal-external knowledge synergistic reasoning for efficient adaptive search agent. CoRR, fbs/25005.0756. 2025. doi: 10.4A550/AXXI..555500777 07596
URI. https://doi org/10. 48550/arXiv. 2505.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- rl: Training Ilms reason and leverage search engines with reinforcement learning. CoRR, abs/2503. 09516, 2025a. doi: 10448550AARXI.2550.00555
URL
48550/arXiv. 2503. 09516.
Jajje Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang. Chenghao Zhang, Tong Zhao, Zhao Yang, and Ji-Rong Wen. Flashrag: A modular toolkit for efficient retrieval-augmeened gen- eration research. In Guodong Long, Michale Blumestein, Yi Chang, Liane Lewin-Eytan, Zi Helen Huang. and Elad Yom-To Tom-Tov (eds.). Companion Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025 May 2025, pp. 737-740. ACM, 2025b. doi: 10.1145/3701111.33113.3
URL htw.ht /ttp://diii.o. org / 18. /1144//77711166 3715313
Eirini-Zoi Kontostavlou and Athanasios Drigas. How metacognition supports giftedness lead- ership A review of contemporary literature. Int. Adv. Corp. Learn., 14(2):416,, 2021. doi: 1.1991
htthhttss:/ddd org/10.0.... 3991/....9........ v1 v1412.23233
Chengpeng Li, Guanting Dong. Mingfeng Xue, Ru Peng, Xiang Wang, and Dayiheng Liu. Dota- math: Decomposition of thought with code assistance and self-correction for mathematical rea- soning. CoRR, abs/2407, .04078, 2024a. doi: 10448550/ARXI..240..000
URL https:
40050/00.855///r// 2407. 04078
12

---

JRL/http:://do..rr/// Published as a conference paper at ICLR 2026
Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. Mugglemath: Assessing the impact of query and response aug- mentation on math reasoning. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), ACI. 2024, Bangkok, Thailand, August 11-16, 2024, pp- 10230- 10258. Asso- ciation for Computational Linguistics, 2024b. doi: 10. https://dof org/10. 18653/v1/2024 acl-long.
URL
Chengpeng Li, Zhengyang Tang, Ziniu Li, Mingfeng Xue, keqin Bao, Tian Ding, Ruoyu Sun, Benyou Wang, Xiang Wang, Junyang Lin, and Dayiheng Liu. Cort: Code-integrated reasoning within thinking, 2025a. URL fttt::://arxiv.or//bbbsss
09820
Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. START: self-taught reasoner with tools. CoRR, abs/2503.004255, 2025b. doi: 1044550/ARXIV.2500.000 04625. URL https://dii.orgg 10. 11.45555/arxv..222 2503. 04625
Kuan Li, Zhongwang Zhang, Huifeng Yin, Rui Ye, Yida Zhao, Liwen Zhang, Litu Ou, Dingchu Zhang, Xixi Wu, Jialong Wu, Xinyu Wang, Zile Qiao, Zhen Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Websailor-v2: Bridging the chasm to proprietary agents via synthetic data and scalable reinforcement learning, 2025c. URL htttts://aaaiiiii .wwlvvorg//eooe www.oogg///////.e haggg/bb//50..... 13305
Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baix- uan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang. Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou. Websailor: Navigating super- human reasoning for web agent. CoRR, hitp2507702552.. 2025d. doi: 10.48550/ARXIV.5507 02592. URL https: org/10. 48550/arXiv.... 2507. 02592.
Weizhen Li, Jianbo Lin, Zhuosong Jiang. Jingyi Cao, Xinpeng Liu, Jiayu Zhang, Zhenqiang Huang, Qianben Chen, Weichen Sun, Qiexiang Wang, Hongxuan Lu, Tianrui Qin, Chenghao Zhu, Yi Yao, Shuying Fan, Xiaowan Li, Tiannan Wang, Pai Liu, King Zhu, He Zhu, Dingfeng Shi, Piaohong Wang. Yeyi Guan, Xiangru Tang, Minghao Liu, Yuchen Eleanor Jiang, Jian Yang. Jiaheng Liu, Ge Zhang, and Wangchunshu Zhou. Chain-of-agents: End-to-end agent foundation models via multi-agent distillation and agentic RL. CoRR, abs/2508. 13sp/2258811166.,. 13167, 2025c. doi: 10.48550/ARXIV. 2508.13167 URL httpp: org/ 48550/ 2508. 13167
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. CoRR,
abs/2501.05366 2025f. doi: 20-4550/ARXXV.2501.05553 arXiv.
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. CoRR, abs/2504.21776, 2025g. doi: 10445500AARXIV2250022777 URL https: /rg/
10. 48550//arrxiv. 2504.21776
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated RL. CORR, 0tn/2503.23383,, 2025h. doi: 10.45550ARXIV.2503.2555 URL /ittps://doi.org
48550/arXiva 2503. 23383
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhijiang Guo, Le Song, and Cheng-Lin Liu. From system to system 2: A survey of reasoning large doi: 10445550/ARXIV.2502..2A 17419. URL
language models. CoRR, abs/2502. 17419, 2025i.
https://do:: org/10. 18850/arXiing 250......202.111. 17419
Zijian Li, Xin Guan, Bo Zhang, Shen Huang, Houquan Zhou, Shaopeng Lai, Ming Yan, Yong Jiang, Pengjun Xie, Fei Huang, Jun Zhang, and Jingren Zhou. Webweaver: Structuring web-scale evidence with dynamic outlines for open-ended deep research, 2025j. URL https:///rxiv.. org/abs/2509) 1
Zongjie Li and Shuai Wang. Reasoning as a resource: Optimizing fast and slow thinking in code generation models, 2025. htttps://arxxv.orgg.....
09396
13

---

Published as a conference paper at ICLR 2026
Baohao Liao, Xinyi Chen, Sara Rajace, Yuhui Xu, Christian Herold, Anders Søgaard. Maarten de Rijke, and Christof Monz. Lost at the beginning of reasoning. CoRR, abs/2506.22058 2025. doi: 10.48550ARRXIV2500.2222 22058.. httphhtts//ooioi... orwoorg//00 185500/axivv 2506 22058
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL.https://ppenrviiiee wtt.e//rreeeee..eefffoonno/odaaa 1e
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Wei Chang. Ying Nian Wu, Song-Chur Chng-Chun&& Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language mod- els, 2023. URL httphhttss///ri....oot 0wtp./.0000//bb//22000044494
Yingqian Min, Zhipeng Chen, Jinhao Jiang. Jie Chen, Jia Deng. Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji- Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. CoRR, abs/2412.09413. 2024. doi: 10.4850/AAXXIV2442.00111 URL URL https://doi.
485500.4855//rx/11 2412. 09413
Bhargavi Paranjape, Scott M. Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Túlio Ribeiro. ART: automatic multi-step reasoning and tool-use for large language models. CoRR, abs/2303.0900 2023. doi: 10.48550AARIIV.333.0000 09014. URL httpp::////o
org/10. 48550/ 2303. 09014
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10. 2023, pp. 5687-5711 Association for Computational Linguistics, 2023. doi: 10. 11118665//1/0223F11DDDA FNNDINS URL https:://oii 18653/ v1/2023.Cv wwndingsnnnnnnnnpp 378
Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. SMART: self-aware agent for tool overuse mitigation. CoRR, 1435, 2025, doi: 10.48550/RXXV.25022.110 2502.
httht:ss:////6..rr///
Zile Qiao, Guoxin Chen, Xuanzhong Chen, Donglei Yu, Wenbiao Yin, Xinyu Wang. Zhen Zhang, Baixuan Li, Huifeng Yin, Kuan Li, Rui Min, Minpeng Liao, Yong Jiang, Pengjun Xie, Fei Huang. and Jingren Zhou Webresearcher: Unleashing unbounded reasoning capability in long-horizon agents, 2025. URL htt.httpss//aaww.rr//bbsas
13309
Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. O1 replication journey: A strategic progress report part 1. CoRR, abs/2410.18982. 2024. doi: 10.45500ARRXI.2410..888 org/10 48..48855///rii 2410. 18982.
URL httpps://oi.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong. Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toollim: Facilitating large language models to master 16000+ real-world apis, 2023. URL htttt:://///att orgo absaabs////// 16789
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. /ttps:::s////xiivv.o. wrw.:/..g./bss.../ 2302. 04761
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu. and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. doi: I0.4850AARXI224020030000 URL
https://doi. org/10 (01550/arXivv 248.2.332233
Toby Simonds, Kevin Lopez, Akira Yoshiyama, and Dominique Garmier. Self rewarding self improving. CoRR, abs/2505.0883 08882 2025. doi: 10.4855/AARX1.22555.887.7 URL https:
7/doi.oog/11 http//10..8550///0
2505. 08827
14

---

Published as a conference paper at ICLR 2026
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-searcher: Incentivizing the search capability in Ilms via reinforeement learning. CoRR, https/2253300552... 2025. doi: 1044550/ARXIV2250300055 435/10.48550/a0
URLhttpp:///dg
2503. 05592
Liangcai Su, Zhen Zhang, Guangyu Li, Zhuo Chen, Chenxi Wang, Maojia Song, Xinyu Wang, Kuan Li, Jialong Wu, Xuanzhong Chen, Zile Qiao, Zhongwang Zhang, Huifeng Yin, Shihao Cai, Runnan Fang, Zhengwei Tao, Wenbiao Yin, Chenxiong Qian, Yong Jiang, Pengjun Xie, Fei Huang. and Jingren Zhou, Scaling agents via continual pre-training, 2025a. URL https:
waaxxiv.og/abll.... 2569. 13310
Xuerui Su. Shufang Xie, Guoqing Liu, Yingee Xia, Renqian Luo, Peiran Jin, Zhiming Ma, Yue Wang, Zun Wang, and Yuting Liu. Trust region preference approximation: A simple and stable reinforcement learning algorithm for LLM reasoning. CoRR, 005544 2025b. doi: 10.
48550/ARXIV25004.00 044224. httpst/ts:///oo org/10. 48550/ 2504. 04524
Yang Sui, Yu-Neng Chuang, Guanchu Wang. Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Ben Hu. Stop overthinking: A survey on efficient reasoning for large language models. CoRR, abs/2503. 16419, 2025. doi: 10.48550/ ARXIV.2503.16419 URL https: org/ 10. 48550/ 2503. 16419
Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, Yue Wu, Wenhai Wang. Junsong Chen, Zhangyue Yin, Xiaozhe Ren, Jie Fu, Junxian He. Wu Yuan. Qi Liu, Xihui Liu, Yu Li, Hao Dong, Yu Cheng, Ming Zhang, Pheng-Ann Heng, Jifeng Dai, Ping Luo, Jingdong Wang, Ji-Rong Wen, Xipeng Qiu, Yike Guo, Hui Xiong, Qun Liu, and Zhenguo Li. A survey of reasoning with foundation models. CORR, abs/2312. 562,, 2023. doi: 10.48550/ARXIV.23122. 11562. URL httphhtts:///oi..rgg. 48550/arXiv. 2312. 11562
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. A survey on self-evolution of large language models. CoRR, abs/2404. 14387, 2024. doi: 10.48550/ARXI..400..220 48550/arXiv 2404. 14387.
URL httphtts://doo.rrg////
Zhengwei Tao. Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Webshaper: Agen- tically data synthesizing via information-seekinng formalization. CoRR, abs/2507. 15061, 2025. doi: 10.48550/ARXV..507..555 15061. URL httphhttt///.... .rgo 485588/855iii 2507. 15061
Kimi Team, Angang Du, Bofei Gao, Bowei Xing. Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang. Dehao Zhang. Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding. Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu. Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li. Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang. Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang. Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with 1lms. CORR, abs/2501. 12599, 2025. doi: 10.48550/ARXI12550..2225
httLhttts://dd orgo /0/ 48550/arxiv. 2501. 12599
Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-5:: 554,,,55,,44 2022.
Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Acting less is reasoning more! teaching model to act efficiently, 2025a. httttpp://aaxii.or//aaaaaa//
14870
15

---

Published as a conference paper at ICLR 2026
Hongru Wang, Boyang Xue, Baohang Zhou, Tianhua Zhang, Cunxiang Wang, Huimin Wang, Guan- hua Chen, and Kam-Fai Wong. Self-dc: When to reason and when to act? self divide-and- conquer for compositional unknown questions. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associ- ation for Computational Linguisties: Human Language Technologies, NAACI 2025 Volume I: Long Pupers, Albuquerque, New Mexico, USA, April 29- May 4, 2025, pp. 6510-6525, Associ- ation for Computational Linguistics, 2025b. doi: 10. 18.118666////// 1/252//112025..22 1AAA1220..NAALLLN........ URL
bttpss///ont org/ 1ttp///..11115111111111 18653/v1/2025 naacl long 331
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. CoRR, abs/2212.035333, 2022. doi: 10445500/ARX11:2222.333. URLhttps:://oii org/10 48550/
arXiv.
Shenzhi Wang, Le Yu. Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang. Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. CORR, abs/2506.01939, 2025c.
doi: 10.
48550/ARXXV2200600100 URL httph://p//oii org/10. 48550/arXiv 2506.
Zihan Wang, Kangrui Wang, Qineng Wang. Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. RAGEN: understanding self-evolution in LLM agents via multi-turn reinforcement learning. CoRR, a0s/25504.0007,, 2025d. doi: 10. https:/s::///oo org/ 18550//rXiiv www.ariwiv.......... 20073
48550/ARXIV.250..0007
Ziliang Wang, Xuhui Zheng. Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. Stepsearch: Igniting Ilms search ability via step-wise proximal policy optimization. CoRR, abs/2505. 15107, 2025c. doi: 10448500AAXXIV2225.55100 URL https:///ol..rr//16 48550/arXiv. 2505 15107
Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingehu Zhang, Zekun Xi, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Webdancer: Towards au- tonomous information seeking agency. CoRR, 0abo/2505.22645 2025a. doi: 10.48550/ARXIV. 2505.22648. URL httpp:://doi.rgg/// w1w.r///......////
2505 22648
Peilin Wu, Mian Zhang, Xinlu Zhang, Xinya Du and Zhiyu Zoey Chen. Search wisely: Mitigating sub-optimal agentic searches by reducing uncertainty. CoRR, abs/2505. 17281, 2025b. doi: 10.
1S550AARXIV.2505.12220
http::/tt:://do
org/10. 48w.0.88555araa 005550/arXiv.0 www.a.iwii..........m... 17281
Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, and Chang Zhou Self-evolved diverse data sampling for efficient instruction tuning. CoRR, abs/2311.08182. 2023, doi: 10.
48550/ARXIV.2311.081 82. URL https://doi org/10 48550/arXiv 2311. .001822
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing LLM reasoning with rule-based reinforcement learning. CoRR, abs/2502, 14768, 2025. doi: 10448550/ARXIV.255001 14762 14768. URL.https://do 4///10.4855//r//// 2502. 14768
Dayu Yang, Tianyang Liu, Daoan Zhang, Antoine Simoulin, Xiaoyi Liu, Yuwei Cao, Zhaopu Teng, Xin Qian, Grey Yang, Jiebo Luo, and Julian J. McAuley, Code to think, think to code: A survey on code-enhanced reasoning and reasoning-driven code intelligence in llms. CORR, abs/2502. 19411, 2025. doi: 10.48550AARXI..2522.1944 19411
URL https://doi http://doo.orgg//// 40t. 2502.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hor question answering. In EMNLP, pp. 2369-2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DIs. 1259. URL https: h/tacaataaannooy.....
1rg/
16

---

Published as a conference paper at ICLR 2026
Zheng Yuan, Hongyi Yuan, Chengpeng Li. Guanting Dong. Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. CoRR, abs/2308.01 01888, 2023, aarXiv. 2wt.233888...1
doi: 10.48500AARXIV.2300001222 URL https:://ddi.orr/11 48550/
Mert Yüksekgönül, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic "differentiiation" via text. CoRR, fbs/2406.00499 2024. doi: 10:48550/RRXIV224600774499 URL https:://ooi.rrg/ 4r5.0//r.55///xi....... 2406 07499
Yongcheng Zeng. Xinyu Cui, Xuanfa Jin, Qirui Mi, Guoqing Liu, Zexu Sun, Mengyue Yang, Dong
Li, Weiyu Ma, Ning Yang. Jian Zhao, Jianye Hao, Haifeng Zhang, and Jun Wang. Evolving Ilms' self-refinement capability via synergistic training -nffnencefeeee optimization, 2025. URL https:
05605
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world envi-
ronments. CoRR, abs/2504.03166 160, 2025. doi: 1044550/ARXIY25000311..0. URL https:
//doi.org/11 48550/
2504. 03160
Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model de- bugger via verifying runtime execution step by step. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 851-870. Association for Computational Linguistics, 2024. doi: 10. //doi.org/1e 8653/v1/2024 findings- ac1. 49
URL https:
Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, and Sainbayar Sukhbaatar. Self-challenging language model agents. CoRR, abs/2506.01716, 2025. doi: 10.4850/ARXX1:25560077666 URL https://doi. www.://r.iv..506.....
17

---

Published as a conference paper at ICLR 2026
Saatteemmatial-Reassmmte Tasks
Knowledge-Intensiee Tasks
Method
Avg.
AIME24 AIME25 AMC23 MATH MATH500 GSM8K 2Wiki. MSQ
Bamb
Frompting-Based
6.7
13.3
47.5
73.8
62.2
69.4
21.1
23.8
9.9
25.5
35.3
Tool-Light (w. SFT)
30.0
26.7
65.0
85.6
77.2
89,4
54.7
55.7
22.8
58.8
56,6
Tool- Rogl-Light (w. PA.)
30.0
26.7
67.5
85.8
80.4
90.8
55.5
56.5
24.3
56.0
57.4
Tool- Light (w. SE.)
w: SE. 1 Loop
30.0
26.7
67.5
86.4
78.2
92.2
56.7
55.6
26.4
59,5
57.9
SE. 2 Loop
33.3
23.3
67.5
87.4
79.0
92.0
57.7
56.1
25.0
58.7
58.0
Table 3: Full results of Tool-Light on 10 reasoning tasks, with the best two results highlighted in bold and underline. We use Qwen2.5-7B-Insstee IIstruct as the backbone model. Abbreviation: 2Wiki. (2WikiMultHHopQA)) HQA. (HotporQA), MSQ. (MuSiQue). Bamb. (Bamboogle). SE. (only con- ducting pre- are-aligmeent DPO training after SFT). PA. (conducting pre-alignment training and several self-evolved DPO alignment loops after SFT).
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
In this study, use LLM assist in literature reading, and we do not use any large language models during other research ideation and/or writing phase.
B MORE DETAILS OF MAIN RESULTS
Implementation Details. We use LoRA training method (Hu et al. 2022). During training, the LoRA rank is set to 8, the epoch set to 3, the learning rate is 7c-6, and the batch size is 8. For the self-evolved DPO alignment, to 1 and limit the maximum call count for each tool to 4. For knowledge-intensive tasks, we use E5-base-v2 (Wang et al., 2022) with Wikipedia retrieval based FlashRAG's settings (Jin al., 20256). For mathematical reasoning tasks, we use Bing Web Search for retrieval. To avoid overly lengthy during web search, we introduce Web Browser mechanism following Tool-Star.
train for two rounds. During testing, set the temperature
Full Results at Tool-Light's Different Stages We present the full results of Qwen2.5-7B-Inse model Tool-Light framework's 's different stages. The results be found in Table 3 From these results, we can draw the following insights: (1) As each training phase of Tool-Light progresses, the performance of the model gradually improves. For example, the performance of the model has significantly improved after SFT. After multiple rounds of DPO training. its performance further in- creases. (2) As the self-evolved DPO alignment loops proceed, the model's performance gradually converges. The performance of the model with the self-evolved DPO alignment is better than with only pre-aligned DPO training. However, with the increase of self-evolution rounds, the model's performance improves slowly. This may be because it is difficult to further generate samples ben- eficial for model training, resulting in only a small improvement in the model's performance after training.
C IMPACT OF DIFFERENT DATA RATIOS
In this section, we explore different data ratios' impact of two sampling methods on training perfor- mance. We sample data using the vanilla method and the entropy-guide guided method separately. Then, we mix the data in ratios ranging from 1:7 to 7:1. Meanwhile, we ensure that other settings remain unchanged. The model is then retrained in the first round of DPO, and the results shown in Figure
Results show that the model's performance in the TIR task is strongly related to the mixing ratio of the two data types. As the proportion of data with the vanilla method increases, both the model's performance and the number of tool calls improve. This is likely because training this type of data makes the model more inclined to use tools to solve problems. However, the Efficiency metric does not continuously improve with the increasing data ratio, likely due to excessive tool call reducing
18

---

Published as a conference paper at ICLR 2026
Relationship between Various Metrics and Dataset Ratio
LLM-as-Judge Performance
Avg. Tsel-use Counts
Efficiency
65.0
0.47
2.4
0.46
60.0
2.2
0.45
55.0
1.8
52.5
0.41
50.0
0.40
3/5
7/1
3/5
6/2
7/1
2/6
6/2
Dataset Ratio
Dataset Ratio
Dataset Ratio
Figure 6: The variation of metries with changes in the data ratio.
the efficiency of the model's reasoning. Based on the above analysis, we set the data ratio used in the experiment 13:7.
D CASE STUDY
To better demonstrate the superiority of Light over other methods in efficiency and necessity metrics, we conduct case study. We compare the reasoning trajectories of Tool-Light with those of Tool-Star. As shown Table 4 when addressing the same problem, even the final answer correct, Tool-Light's result achieves fewer tool calls compared to that of Tool-Star. And as shown in Table S when faced with a more difficult problem, the Togh-Light method will proactively call external tools to assist in answering questions when necessary.
19

---

Published as a conference paper at ICLR 2026
Table 4: A comparison example between Tool-Light and Tool-Star on MATH500 dataset.
Example #1 from MATH150 Dataset
Question:
The square root of is greater than and less than 3.5. How many integer values of satisfy this condition?
Golden Answer:
8
Tool-Light's TIR Stragety
<think>> To determine how many integer values of satisfy the condition 2 < vi < 3.5, we can follow
these steps:
Square all parts of the inequality to eliminate the square root:
2 < l < (3.5)²
This simplifies to:
4 < ( < 12.25
2. Identify the integer values of / that fall within this range. The integers between 4 and 12.25 are 5,
8, 9, 10, 11, and
3. Count the number of these integers.
Let's implement this in Python to ensure accuracy. </think><python
Define range Lower_bound = 4 upper_bound = 12. 25
Find the integer values the range integer_values [t for t range (int (lower_bounde 1, int upper_bound) 1)]
Count the number of integer values nun_integer_values (innteger__vllue pr int (iinn.inntteerrr
</python>><result </result> <answer><hh number of integer values of / that satisfy the condition
2 < vi is </answer>
Tool-Star's TIR Stragety
<think>>kka so need to figure out how many integer values t satisfy condition that square root of between and 3.5. Hmm, let me start by understanding what exactly the question asking. First, the square of But since we're dealing with real numbers, maybe even irrational ones, but question specifies that must he an integer. So f has to be perfect square between 4 (since 2 squared is 4) and... well, 3.5 squared. Let me compute that. </thin>><yython
greater
and than 3.5. That means /(t) > and v(t) 3.5.
lower 2 ** 2
upper 3.5 ** 2
print Lower bound: (lower), Upper bound: (upper}')
</python>-<resuhe Lower bound: Upper bound: 12.25 </result>So /(t) needs he strietly greater than 2, which gives f > 1, and strictly less than 3.5, which gives t (3.5)² R 12.25. Therefore, t must be integers from up to 12 inclusive. To confirm: <ythoo><ython>>>>>>>>>
start int (2**2)
end int (3.5xx2) count end start +
(start), End: (end), Count: (count)')
</ython>><result>>> Start: 5, End: 12, Count: 8 </result> <think> Thereffor,, the number of integers
satisfying the condition is indeed 8, </think> <answerr>
8 </answer>>
20

---

Published as a conference paper at ICLR 2026
Table 5: A comparison example between Tool-Light and Tool-Star on AIME24 dataset.
Example #2 from AIME24 Dataset
Question:
Find the largest possible real part of
96 144i (75 + 1174)=
2
where z is a complex number with |x| 4. Golden Answer:
540
Tool-Light's TIR Stragety
<think> To solve the problem, we need to find the maximum value of the expression (75 + 1171)= + 964-144; where z is complex number with |a| 4.
Let's represent 2 4e where is real number. Then we can write:
(75 1171)= + 96 + 144i
96 + 144
(75 + 117i)4e
+
4e
This can be simplified to:
24 36
4(75 + 117i)e
e:0
(#**More reasoning contentare omitted here"**)
import numpy np
# Define the complex numbers
a 75 117j
b 96
144
Define the tagnitude of z ==
Initialize the maximum real part max_real_part -np. inf
max_theta =
# Evaluate the expression for range of theta values for theta in np. linspace (0, pi. 1000):
z = np. exp expr = a +
theta)
real_part real (expr)
if real_part max_real_parts
max_real_part real_part
max_theta theta
(rant(max_eeaa_pprr))
</python><<reuul>>>>>> 539.9979693595761 <resul>><<sswerr> largest possible real part of the
pression (75 117f)= + 96+144i where is complex number with |a] 4 is 540 </answer>>
Tool-Star's TIR Stragety
<think> To solve the problem, we need to find the maximum value of the expression (75 + 1171)z + 96+144; where 2 a complex number with |=| = 4. Let's represent z as 29 where 0 real number.
This way, we can express z in terms of its polar form. Mo**More reasoning contentare omitted here***)
However, should confirm this by calculating the critical points. The derivative of 324 cos 0- 504 sin 0 --2244ii sin 0 504 cos Setting this zero gives the critical points. But since we know the maximum value of the expression cos 8 b sin 0 is / V + b2, we can directly use this formula. Thus, the largest possible real part is:
<
594 </answer>
21


Reviews:
[{"review_id": "Wut71EO4AL", "reviewer": "Reviewer_2Xcg", "summary": "This paper presents Tool-Light, a framework designed to enhance the efficiency of tool-integrated reasoning (TIR) in large language models. The core empirical finding demonstrates that invoking external tools causes significant shifts in downstream token-level uncertainty, as measured by entropy. Based on this observation, the authors propose: (i) an entropy-guided sampling procedure for constructing training data, and (ii) a two-stage self-evolved Direct Preference Optimization (DPO) pipeline comprising Pre-Aligned and On-Policy phases. Experimental evaluation across ten benchmarks covering mathematical reasoning and multi-hop question answering shows comparable or superior accuracy while reducing redundant reasoning and improving both efficiency and necessity of tool usage.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured with clear writing that facilitates comprehension of the proposed methodology.\n2. The investigation of information entropy changes during tool invocation processes is particularly interesting. The authors effectively leverage these insights to guide their methodological design, providing a principled foundation for their approach.\n3. While some components of the proposed data construction and self-evolved DPO framework draw upon established techniques, the overall approach remains intuitive and theoretically sound.\n4. The authors conduct thorough experiments demonstrating the effectiveness of their method. The entropy distribution analysis particularly convincingly shows that the approach achieves lower entropy distributions, validating the theoretical motivation.", "weaknesses": "1. The analysis of entropy in tool invocation has been explored in prior work, and the conclusions drawn are not particularly surprising or groundbreaking.\n2. The proposed techniques, including entropy-based sampling and evolved DPO, represent relatively incremental advances rather than significant methodological innovations.\n3. While the core idea is interesting, the evaluation is restricted to DPO. The work would be significantly strengthened by demonstrating applicability to other preference learning methods such as GRPO or PPO, which would provide stronger evidence for the generalizability and robustness of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tloPB47yvO", "reviewer": "Reviewer_WYvp", "summary": "This work addresses inefficiencies in Tool-Integrated Reasoning (TIR), where LLMs overuse, underuse, or mismanage tool calls. The authors analyze tool-usage dynamics through information entropy, finding that tool call results significantly influence reasoning entropy patterns. Building on this insight, they propose Tool-Light, a framework for more efficient and accurate TIR. Tool-Light combines self-evolved dataset construction—using vanilla and entropy-guided sampling with strict positive-negative selection—with a two-stage training scheme of Supervised Fine-Tuning and Self-Evolved DPO. Experiments on 10 datasets (e.g., AIME24, MATH, HotpotQA, etc) show that Tool-Light notably enhances both the efficiency and accuracy of tool-integrated reasoning.", "questions": "Please see the above Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Tool-Integrated /Agentical reasoning is a important topic with many potential practical applications.\n2. The information-entropy based analysing after calling tools is novel and insightful. \n3. The experiment evaluation is comprehensive and the performance boost is obvious.", "weaknesses": "1. Discuss more related work from the Self-Evolved Preference Learning perspective, such as Zeng et al., Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization, and Su et al., Trust Region Preference Approximation: A Simple and Stable Reinforcement Learning Algorithm for LLM Reasoning.\n2. Clarify whether the information entropy observation generalizes across all tools, and list the specific tools used in the experiments for completeness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LOwvAw2Mby", "reviewer": "Reviewer_mPkz", "summary": "The paper studies “tool-intelligence” in LLM agents and targets two failure modes: overthinking (too many tool calls/long chains) and underuse (skipping necessary tools). It first runs a pre-experiment that tracks token-level entropy over reasoning traces and observes patterns around tool-call moments. Building on this, it proposes Tool-Light, a two-stage pipeline: (1) SFT to establish a reasonable tool-using policy, then (2) self-evolved DPO using curated preference pairs to favor accurate, tool-efficient traces. In inference, it adds entropy-guided branching that expands alternatives at high-uncertainty steps while keeping compute controlled. Experiments span math and knowledge-intensive QA with web search and a code interpreter, reporting accuracy alongside two bespoke metrics—Efficiency (tool economy) and Necessity (using tools when needed). Results show competitive or better accuracy with fewer tool calls and shorter outputs compared to strong baselines (e.g., Tool-Star, Search-R1), plus initial ablations on sampling knobs. The paper positions entropy as a practical signal for both training supervision and test-time exploration control.", "questions": "(1) Entropy dips after tool calls.\nCould the “drop before the next tool call” be driven by inserting long, deterministic tool outputs (i.e., context length/format effects) rather than better reasoning? What happens if you replace tool results with semantically equivalent, length-matched paraphrases? Does the pattern remain?\n\n(2) Scope to multi-tool.\nDo the pre-experiment findings hold in true multi-tool settings, not just single-tool or search-heavy cases? Please show the same entropy analysis when multiple heterogeneous tools are available.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Actionable diagnostic. Using token-level entropy to profile traces is simple, implementation-light, and yields intuitive visualizations to reason about when/why tools are called.\n\n(2) Modular training recipe. The SFT → self-evolved DPO pipeline is straightforward to adopt on top of popular backbones and existing tool-use frameworks; no exotic infra required.\n\n(3) Competitive results. Shows strong performance relative to recognized baselines while reducing tool calls—evidence that “lighter” tool use need not sacrifice accuracy.", "weaknesses": "**Weaknesses**\n\n(1) Missing details. The pre-experiment omits decoding settings. No temperature, logit scaling, or sampling vs. greedy are reported. These choices change entropy levels and trends. Please specify the exact decoding config used to measure entropy and justify it. You say the entropy study spans “multiple QA datasets,” but the figure does not list them. It’s also unclear whether the same model/datasets are used later to train Tool-Light. Please enumerate the datasets in Figure 1 and clarify any reuse. For the method, entropy decides branching at top-k steps and uses top-i prefix averages. The values of k and i and their stability are not reported.\n\n(2) Missing ablations. Table 2 varies loop count and a few sampling knobs only. Missing: (i) no-entropy (vanilla) baseline, (ii) entropy-only,  (iii) β sensitivity in DPO, (iv) reference-policy choice, and (v) branch-width sensitivity.\n\n(3) “Multi-tool” over-claim. Most experiments use only two tools (web search + code interpreter), and several evaluations appear to be single-tool setups. There’s no test on unseen tool types or cross-tool composition tasks. As a result, the paper does not yet support broad “multi-tool generalization” claims.\n\n**Suggestions**\n\n(1) Control for length. Lower entropy often comes with shorter or more templated outputs. Your own results show Tool-Light reduces sequence length (Fig. 4). The same factor could explain both “lower entropy” and “fewer tool calls.” Please report entropy at fixed token positions or use length-normalized entropy.\n\n(2) Link entropy to correctness. “Low-entropy chains use fewer tool calls” may reflect early commitment, not better answers. Please test whether entropy predicts correctness. Report AUROC for path-level mean or area-under-entropy, controlling for tool-call count.\n\n(3) Stress test with noisy tools. Overthinking and underuse show up when tools are imperfect. Add controlled corruptions: vary retrieval precision/recall, inject code stderr/noise. Show how Tool-Light adapts tool frequency and preserves accuracy vs. Tool-Star/Search-R1. This directly probes the “analysis paralysis” claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vnvv9TULZe", "reviewer": "Reviewer_nxVk", "summary": "The paper studies the Tool-Integrated Reasoning for LLMs, and proposes a new framework namely Tool-Light to resolve the problem of excessive tool calls during the reasoning. The key designs of the proposed framework contain dataset construction and multi-stage fine-tuning. Specifically, for dataset construction, Tool-Light employs two sampling strategies and for multi-stage fine-tuning, it introduces a two-stage training method. To support the effectiveness of Tool-Light, extensive empirical studies are conducted.", "questions": "Answer questions in Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors provide extensive experiments to demonstrate the effectiveness of **Tool-Light**, particularly through the experiments in **Figure 1**, which clearly illustrate the motivation behind the work.\n- The paper is clearly written and well organized, making it easy to follow.", "weaknesses": "- **Figure 1** lacks sufficient explanation. For instance, the meaning of *Token Index* and the specific roles of *Step 1–4* are unclear.\n- In **Figure 4(c)**, it is not specified whether the response length includes the tool-calling part. The figure shows that **Tool-Light** produces shorter responses than **Tool-Star**, yet the examples in the appendix (Examples 1 and 2) suggest otherwise.\n- In **Line 257**, the computational complexity is claimed to be $O(n\\log m)$, but the derivation of this result is not provided.\n- The authors adopt two sampling strategies but do not include an ablation study to compare their effects.\n- The experiments are conducted only on the 7B model. To further validate the effectiveness of **Tool-Light**, the reviewer encourages testing on both smaller (e.g., 3B) and larger (e.g., 72B) models.\n- In **Line 80**, the authors state that *Pre-Aligned DPO Training* can reduce redundant tool calls. However, it is unclear why this is the case, as *Pre-Aligned DPO Training* does not appear to include mechanisms that explicitly control the number of tool calls.\n- It is recommended to report the standard deviation across multiple runs to demonstrate the robustness of the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)