T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Models
Authors: Minki Kang
Jongwon Jeong
Jaewoong Cho
Venue: ICLR 2026 Poster (2026)
Keywords: Large language model
math reasoning
tool use
test-time scaling
small language model
Primary area: foundation or frontier models, including LLMs

Abstract: Recent studies have demonstrated that test-time compute scaling effectively improves the performance of small language models (sLMs).
However, prior research has mainly examined test-time compute scaling with an additional larger model as a verifier, leaving verification by sLMs underexplored.
In this work, we investigate whether sLMs can reliably verify the output candidates under test-time scaling.
We find that even with knowledge distillation from larger verifiers, sLMs struggle with verification tasks requiring memorization, such as numerical calculations and fact-checking.
To address this limitation, we propose Tool-integrated verification (T1), a two-stage framework that first filters candidates with external tools and then uses an sLM for final verification, offloading memorization-heavy steps to tools such as a code interpreter.
Within T1 we prove that offloading to external tools reduces the memorization burden on sLMs and improves test-time scaling performance.
Experiments on the MATH benchmark demonstrate that, with T1, a Llama-3.2 1B model under test-time scaling outperforms the significantly larger Llama-3.1 8B model. 
Moreover, T1 improves the verification accuracy of both process reward models (PRMs) and critic models.
Our findings highlight the potential of tool integration to substantially improve the verification abilities of sLMs.

Published as a conference paper at ICLR 2026
T1: TOOL-INTEGRATED VERIFICATION FOR TEST-TIME COMPUTE SCALING IN SMALL LANGUAGE MODELS
Minki Kang Jongwon Jeong2* Jaewoong Cho
KAIST University of Wisconsin-Madison SKRAFTON
ABSTRACT
Recent studies have demonstrated that test-time compute scaling effectively im- proves the performance of small language models (sLMs). However, prior research has mainly examined test-time compute scaling with an additional larger model as verifier, leaving verification by sLMs underexplored. In this work, investigate whether sLMs can reliably verify the output candidates under test-time scaling. We find that even with knowledge distillation from larger verifiers, sLMs struggle with verification tasks requiring memorization, such as numerical calculations and fact-checking. To address this limitation, we propose Tool-integrated verification (T1), two-stage framework that first filters candidates with external tools and then uses an sLM for final verification, offloading memorization-heay steps tools such code interpreter. Within T1, we prove that offloading to external tools reduces the memorization burden on sL and improves test-time scaling performance. Experiments the MATH benchmark demonstrate that, with TI, Llama-3.2 IB model under test-time scaling outperforms the significantly larger Llama-3.1 8B model. Moreover, T1 improves the verification accuracy of both pro- cess reward models (PRMs) and critic models. Our findings highlight the potential of tool integration substantially improve the verification abilities of sLMs.
INTRODUCTION
Recent advances in large language models (LLMs) have demonstrated strong emergent abilities through large-scale pretraining (Brown al., 2020; Hurst et al., 2024; Reid et al., 2024), enabling them to tackle complex reasoning tasks such as mathematical problem-solving and competitive coding (Wei et al., 2022b; Lightman et al., 2024). While small language models (sLMs) offer advantages in deployment efficiency and cost (Liu et al., 2024; Lu et al., 2024), they struggle significantly with high-complexity tasks (Wei et al., 2022a).
Test-time compute scaling has emerged as promising approach to enhance sLMs by dynamically allocating additional computation during inference (Wu al., 2024). Prior works suggest that test-time scaling can surpass pretraining- based scaling (Snell et al., 2024), allowing 3B LM to outperform 405B LLM on mathematical benchmarks such as MATH and AIME (Liu et al., 2025). This success depends on reliable verification of generated solutions.
To enable reliable verification, existing approaches have leveraged process reward models (PRMs) and critic models (Wang et al., 2024a; Zhang et al., 2024), but these typically require LLMs (7B+ parameters). Relying on large verifiers counteracts the efficiency benefits of sLMs. Therefore, remains unclear whether verification, where sLMs verify the generated solutions, can enable strong reasoning capabilities without relying on larger models. This raises key research question:
Can small language models reliably perform verification for test-time scaling?
While verification is often easier than generation, prior work has shown that sLMs still struggle to verify the solutions (Song et al., 2024). Our concept-prood experiment in Figure (b) confirms this finding that larger models can reliably verify with chain-of-thowe oh-thhttht reasoning alone. In contrast,
"Equll contribution. Work done KRAFTON, Contact: kang@kai st.

---

Published as a conference paper at ICLR 2026
Q: There were 237 apples. Someone brought 321 more apples. How many apples there now?
Venfication
aod 301
sé6
Large Verifier
Reliable
2534 10 58k
257+321- 556
Small Verifier
dedlic (it
Not
NNussom CrCodaattonn (N)
All 2ltand gnaenssmm
Reliable
Code VeriEsation
257-321- 558
100
M 100
237-321- 356
Mixill
Small Verifier
Reliable
237+521 558
Camos
Concept Figure
(b) Concept-Proof Results
Figure 1: (a) Concept. Small language models (sLMs) often fail due to their limited capacity. How- ever, when sLMs utilize external tools, their reliability improves. (b) Concept-
Experimental Results. We evaluate Llama IB and 3B models verifying calculations of N three-digit numbers. The IB model's performance drops significantly as N increases, while the 3B model remains stable. Enabling code generation and execution largely mitigates this drop for the 1B model.
sLMs fail to verify even simple calculations, particularly as the complexity of calculations N increases. We hypothesize that this gap is due the limited capacity of sLMs to memorize all calculation facts required for verification (Kandpal et al., 2023).
However, as shown in Figure 1 (b), code generation and execution substantially improves sLM's verification accuracy, narrowing the gap with larger models even as N increases. This result suggests that integrating external tools with sLMs reduce the need to memorize arithmetic facts. Therefore, tool integration is not merely beneficial but necessary to enable successful verification in sLMs.
Motivated by these findings, we introduce Tool-integrated Verification (T1), a two-stage framework in which external tools first verify candidate solutions and ss then verify the filtered solutions. By offloading memorization-hent heaay steps, such as numerical calculations and fack-checking, to external tools, T1 enables sLMs to achieve verification accuracy comparable to much larger models without increasing parameters. Crucially, this two-stage design applies seamlessly to both generative verifiers and process reward models (PRMs), offering single framework for test-time scaling, Moreover, the two-stage design theoretically guarantees that tool-based verification reduces the memorization burden and improves test-time scaling performance.
Our experiments demonstrate that enables both generative verifiers (Zhang et al., 2024) and PRMs for accurate verification. This leads significant performance improvements widely-used mathematical and multi-domain reasoning benchmarks, with notable gains on GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021). These results underscore the effectiveness of TI in improving the performance of sLMs under test-time compute scaling.
Our contributions are as follows:
We conduct systematic study of sLMs' verification under test-time scaling, identify memorization- heavy steps in verification as key bottleneck, and motivate addressing them with external tools.
We propose Tool-integrated Verification (TI), a two-stage framework that leverages external tools to offload memorization-heau steps before sLM verification.
We provide a theoretical analysis showing that external tools reduces the memorization burden and that two-stage design in T1 improves test-time scaling performance.
We show that Tl integrates seamlessly with both generative verifiers and PRMs, achieving strong results on math reasoning benchmarks GSM8K and MATH.
RELATED WORKS
2.1 TEST-TIME COMPUTE SCALING
Test-time compute scaling has emerged as a promising approach for improving the reasoning ca- pabilities of large language models (L.LMs) (Wu et al., 2024). It can be broadly categorized into
2

---

Published as a conference paper at ICLR 2026
sequential and parallel methods (Snell et al., 2024). Sequential scaling iteratively refines solutions by leveraging post-training, enabling the model to perform self-reflection and verification (Muennighoff et al., 2025; DeepSeek-Al et al., 2025). Parallel scaling, in contrast, generates multiple candidate solutions simultaneously and selects the hest one using a verifier model (Cobhe et al., 2021; Lightman 2024; Brown et al., 2024). A common strategy is the best-of-N method, which produces N parallel outputs and ranks them based on verification scores (Cobbe et al., 2021; ;; Lightman et al., 2024). Increasing N has been shown to enhance LLM on challenging benchmarks (Brown al., 2024; Snell et al., 2024).
In this work, we focus on the parallel sealing paradigm due to its simplicity and popularity. Prior research shows that even small models can achieve strong results when paired with large verifier in parallel scaling (Liu et al., 2025). We further investigate whether small language models (sI.Ms) can verify, enabling test-time scaling without large models.
2.2 VERIFIER IN TEST-TIME COMPUTE SCALING
The verifier plays a crucial role in parallel scaling. One approach, the Process Reward Model (PRM), scoring each reasoning step individually using regression head, enables fine-grained feedback (Lightman et al., 2024; Wang et al., 2024a; Zeng et al., 2025). An alternative approach leverages LLM itself Critic model, prompting it to evaluate reasoning steps (Zheng et al., 2023). Zheng et al. (2024) have shown that powerful LLM-based critic models can outperform PRM, particularly in mathematical reasoning tasks. Recent works (Zhang al., 2024; Mahan al., 2024) proposed the Generative Reward Model (GenRM), formulating verification as a next-token prediction problem with chain-of of-thoughtt (Wei et al., 2022b) improving interpretability in step-wise verification.
Despite these advances, ensuring consistent and high-quality step-wise verification remains an open problem for both PRM and GenRM. Additionally, prior works have not thoroughly explored the change in verification performance depending on the size of L.s..
2.3 TOOL-INTEGRATER LANGUAGE MODEL
The integration of external tools has significantly enhanced LLM capabilities. Program-aides language models (Gao al., 2023) introduced delegating computations to interpreters via synthesized code. Subsequent works expanded this by using tools like search engines and calculators for fact retrieval and arithmetic (Schick al., 2023; Qin et al., 2024). Recent methods further integrate tools into multi-step reasoning (Gou et al., 2024b; Zhu et al., 2024), reward modeling (Li et al., 2024), and self-correction (Gou al., 2024a).
Our work extends this line of research by studying how to design tool integration that benefits sLM verification across both process reward models (PRMs) and critic models. We formulate tool use as an additional dimension of test-time scaling, emphasizing its effectiveness in eemmorization-haay verification tasks.
PRELIMINARIES
Test-time scaling Following Snell et al. (2024), we view test-time scaling as modifying the model's proposal distribution. Given problem æ E X, we sample a solution from the policy It æ, Ip: 0) in the set V, where I is the generator-spese specific instruction prompt, and the policy is parameterized by 0 which refers to the pre-trained language models.
Several algorithms can be used to scale test-time computation, including those that adjust the input level (e.g., self-reflection (Kumar et al., 2024)) and those that modify the output level (e.g., best-of- (Cobbe et al., 2021), beam search (Yao al., 2023)). Among them, the best-of-N algorithm is simple yet powerful approach. It samples multiple candidate solutions from the policy and selects the one with the highest score, as determined by the verifier, as the final prediction. Formally, the Best-of-N policy T
(y x. 0p:
8) is defined
arg max r(æ, y), W (
S.1. y: ~ T x. Ip: 0),
where r(x, v) X x y + R is verifier that assigns scalar score
3

---

Published as a conference paper at ICLR 2026
Problem
RM-based Verifier
Charlettoo Cardeston bus shorp Seattle How de Touivuse Chadlestzz, Svattle here tugeter of Seatle lot shrep?
twice
X Tool-based Verifier
Verification
Charleston
Step by-Step
ahor por nortectia,co chopp sattte seepp
E-o cale
Score Verification
Generator
Step- Step-by-step Solutions
42 Step 1: Seattl his sherp, 44 Step
Final verification
Clarlestzo 4 20 = stoep. (...) The final answve'ts is 250ss
Interpreter
>_
True
False
X Solution is Wrong!
Figure 2: Tool-integrated self-verification for mathematical reasoning. (a) Generator: A small language model (sLM) may produce incorrect solutions due to calculation errors. (b) Tool-based Verifier (ToolV): The sLM generates executable code based on its reasoning; the output the code is used verify the solution's correctness. (c) Reward Model (RM)-based Verifier: The reward model (GenRM/ PRM) still evaluates the solution as before, but its verdict only contributes to the final decision if the solution passes the tool-assisted filter. Concrete examples are in Appendix F,
Verifier The verifier can be modeled using the following models: (1) process reward model (PRM), which assigns the score of each step of reasoning (Wang et al., 2024a), (2) critique model, which generates a rationale for the verifiation (Zheng al., 2023). For both cases, the sequence of verification scores or tokens z are sampled from x( y. Io: 0) where I, is the verifier-specific instruction prompt. In general, we use the last score token of the sequence as the final score for the solution (Snell et al., 2024; Zhang et al., 2024). For instance, in generative verifier (Zhang et al., 2024), the verification score can be obtained as follows:
r( 3) (22 'Yes' <; y, Lus -1:T-1:0), s.t. 3I-T1 ~ x( x. y. Lo;
(2)
where -1:T-1 is Chain-of-thhught (Wei al., 2022b) and last token zT ('Yes', 'No').
4 METHOD
4.1
TOOL-INTEGRATED VERIFICATION
Test-time scaling can improve a base policy model that generates valid solutions from its proposal distribution, but the effectiveness of scaling at test-time heavily relies on the performance of the veri- fier. However, sLMs often struggle to reliably verify the correctness of their generated outputs (Song et al., 2024). Specifically, sLMs exhibit limitations in precisely validating numerical computation or detecting incorrect or outdated knowledge information, due to their limited parameter size and insufficient memorization capacity.
To address these limitations, we propose a Tool-integrated verification (T1) approach for parallel test-time scaling in sLMs. As shown in Figure 2, our verification approach involves two stages: 1) filtering stage with Tool-based Verifier (ToolV) and 2) scoring stage with reward model (RM))based verifier, and formally can be expressed as:
y arg max [xx, y: TTT, 0) 0), s.t. Y4 ( ar, Ip: 0), Vi (1, (U,...,,.. N}. (3) VE(WI....NN
where f(x, y: T, 0) E {0, 1} indicate binary tool-based verification function (with 0 indicating filtered-out response). T denotes the utilized tool (e.g., code interpreter, retriever), and r(æ, y 0)
denotes the verifier score defined in Equation 1.
Tool-based verifier stage In this stage, sLM utilizes the external tool T, such as a code interpreter or knowledge retriever, to verify generated outputs. Here, we assume the scenario in which the utilized tool T is explicitly known. Specifically, sLM uses these tools to verify numerical accuracy and validate the knowledge in generated solutions. One specific example is that multiple generated responses are filtered based on tool-based verifiers, discarding those with incorrect calculations or inaccurate knowledge information. Specifically, the tool-based verification function, f(x y; T, 0), consists of three parts: 1) generating the tool-calling query (i.e., c1), 2) the execution of the tool (i.c.. T(-)), and 3) extraction of the verification (i.e., c2). Therefore, f(x y: T, 00)) can be represented as:
f(xr, y; T,0) c2 क (cl T (c1). , y, 15;0), where ei n <cc x. y, Ic;0),
(4)
4

---

Published as a conference paper at ICLR 2026
and y and Le are task-specific instruction prompts. Detailed formulation for mathematical reasoning and www.ledge-intensiie tasks are represented in Appendix B.
RM-based verifier stage Following ToolV stage, the remaining generated responses are scored using a reward model, the same model used for generation and filtering. This reward model assesses the overall logical consistency, coherence, and correctness of each response. The final output is chosen as the response with the highest reward score.
4.2 VERIFIER DISTILLATION
To further enhance the performance of both verification stages, we employ knowledge distillation (Hin- ton et al., 2015; Kim & Rush, 2016) from LLMs. Specifically, we fine-tune sLM using tool-based and RM-basedd verifications generated by a larger teacher model Oz- To efficiently manage multiple distinct tasks during the distillation process, adopt multi-LoRA, (Hu et al., 2022) approach, assigning separate LoRA adapters, Afrool. and Arreardd for each verifier.
The distillation for ToolV is formulated as:
Ctoot Crool (Airool) = -Bgg Xiain g-::()1x;I;:00), com(-1t.a.g.1:00) log [ a, y, 1:0 A0tool), (5)
where Xerain is training dataset, C2), ) I (Ie, } and empty á or the output T (e)). Note that we generate tool-based verifications using the teacher model by applying corresponding instruction I in a zero-shot manner (Ouyang et al., 2022; Chung et al., 2024).
Similarly, distillation for the R.M-baed verifier is expressed as:
Creward (eewaard) = -eg g-<r(-la.I,;0); x~ wwwp(-/...,,.....0 log T( x, y, I,; 0+ AO Arwward); (6)
where I, is RM-based verifier-specife instruction. In Equation 6, responses are first sampled from the student model's proposal distribution. Each sampled response is then verified by the teacher model, and finally, the student model is fine-tuned based on these verifications.
5 THEORETICAL ANALYSIS
In this section, we present theoretical analyses of two aspects with conceptual illustrations: (1) how external tools reduce memorization requirements and improve verification performance (subsec- tion 5.1), and (2) how our two-stage approach guarantees improved performance under test-time scaling (subsection 5.2).
5.1 MEMORIZATION BOUND WITH WITHOUT TOOL
Let us consider a simple verification task in which verifier assesses whether the given equation a + = c is true or not. Let this task's data distribution is q and P ((a,b,c),r) q. Assume
have |X] number of training samples such that X (((a, b, r), a, (0...., M - 1}, c (0....,2M 2M. - 2}, r -==1=+a latbme), where ((a, b, c),r) is independently sampled according to q. Then, let X ~ q be the random variable representing the distribution of the training set X. A learning algorithm A receives X to produce 0 such that 0 - (X). Then, we call that A is s-close-tor s-close-to-optimad if (A) < eITg, ( + where err, xx], (.A) Prx~q0 x ((((a_bc).)))aggff. forlea t... (f + r) and AOPT is the optimal learning algorithm. Then, Lemma 5.1 shows how much information of X should be memorized within 0 to satisfy almost zero error.
Lemma 5.1 (Memorization without Tool (Brown et al., 2021)). Any learning algorithm A that is e-close-to-optimal with sufficiently small > 0 also satisfies I (X;0) P) = 2(M²), where I is the mutual information.
Proof sketch. Theorem 1.1 in Brown et al. (2021) said that T(X;0 P) is proportional to at least dataset size. Since |X| 2 (M - 1)3, we can get ((M)). Refer to subsection C.2 for the detailed proof.
O
On the other hand, using an external tool that verifies whether a + b b c allows the model to avoid memorizing the full table of sums. Specifically, define tool T. Suppose 0 is generated by learning
5

---

Published as a conference paper at ICLR 2026
algorithm A that has access to T, and that C,,,,, T) latb=e holds. Then obtain the following result:
Theorem 5.2 (Memorization with Tool). Suppose 0 is generated by learning algorithu A that has access to T, and that f(a,b, b, c; (, = a+bb= holds. Then, any learning algorithm A that is e-close-to-optimal with sufficiently small 0 also satisfies (X; 0 P) 0, where is the mutual information.
Proof sketch. As the learning algorithm can access external tool T, then 0 = A(X) such that f(a, b, c; 0, T = latb= f makes ,xT,,,i(( = 0. Also, is independently determined regardless of X, resulting T(X; 0 P) = See subsection C.3 for the detailed proof.
I (X; 0 P) quantifies the amount of information about X, drawn from P. that must be memorized in 0 learned by to achieve near-zero error. By comparing (X; P) from Lemma 5.1 and Theorem 5.2, we demonstrate that a tool drastically reduces the required memorization of X within 6, lowering [ (X; 0 P) from Dt 2) to Consequently, this result implies that with the tool, small
models become reliable for the verification task.
5.2 EFFECT OF BASE- VERIFIER ON TEST-TIME SCALING
We employ the toy setting introduced in Beirami et al. (2024). Specifically, for given input æ. the ground-truth label produced by this generator is set as 1, and the generator и produces binary outputs, ie., y = {0, 1}. Furthermore, we consider an imperfect verifier inducing noise. Then, we can show Theorem 5.3 that increasing Q1 directly increases the probability of obtaining correct output from the best-of-N.
Theorem 5.3 (Monotonicity of Imperfect Verifier). Let the generator output 0 or 1 with equal probability, i..., x (0)z) = π (1|z²) = 2. and the verifier r with noise level p, q be defined as follows:
0. w.p. p.
1, w.p. 9,
(xx, 0)
r (ar, 1)
1, w.p. 1 - p,
10, w.p.
with the condition that p 1 p and q 9. Assume p and p be the noise level of two verifiers
with p > P. Then, for any N 2 2,
x)
(1 x))
Proof sketch. By the law of total probability, we get N (1 æ). Then, we can get the monotonicity of n (1 æ). Refer to subsection C.4 for the detailed proof.
tool-based verification function subsection 4.1 effectively acts as a filter for incorrect solutions, we can say that using f increases thus improving the verifier's capability to choose the correct label as shown Theorem 5.3.
EXPERIMENTS
6.1
SETUP
Datasets We mainly focus on mathematical reasoning task due to its widespread adoption and strong effectiveness assessing the reasoning capabilities of language models (Snell al., 2024; Liu et al., 2025). We use (1) MATH500 (Hendryeks et al., 2021; Lightman et al., 2024), dataset containing college-level math problems.
(2) GSMSK (Cobbe et al., 2021), which consists of grade- school math problems. We use the training set of each dataset for distillation. We also include additional experimental results on the subset of MMLU-Pro (Wang et al., 2024b) in Appendix E, which contain a multi-domain Knowledge-intensiv problems.
Evaluation setting Following previous works (Cobbe et al., 2021; Lightman al., 2024; Snell et al., 2024; Liu et al., 2025), we evaluate weighted oest-of-N performance, where we aggregate the score of the solutions ending with the same final answer, to assess test-time compute scaling. We generate 64 solutions using a fixed generator and measure the percentage of correctly solved problems after verifications. As verifier, we use both PRM (Wang et al., 2024a) and GenRM-CoT (Zhang et al., 2024) (we refer to it as GenRM).

---

Published as a conference paper at ICLR 2026
Majority
Distilled PRM
TooTV + Distilled PRM (Ours)
SmooLMM2-60M-Instuuct
Qwen2.5-0.50-mmsss
Llama-3.2 2-1---1-1100 B-Instruct
10
50
55
Lama' I-8f-In..
Accuracy (%)
Qwen2 5-1.50-ms.
50
45
40
35
30
30
25
25
2
2'
2
2
26
20
2-
2
2
24
21
2
2
2
2
2
2
Number of Solutions per Problem
Number of Solutions per Problem
Number of Solutions per Problem
Figure 3: MATH500 with PRM. Weighted Best-of-N performance of three small language models, emphasizing the benefits of ToolV on college-level math problems. TooIV significantly enhances PRM, enabling small models to outperform or match much larger models. Qwen2.5-1.5p and Llama3.1 1-BB performances are reported as N greedy decoding.
Majority
Zero-shot GenRM
Distilled GenRM
ToolV + Distilled GenRM (Ours)
SmoLLLM23300M-nnnte
Owen2.5-0.50IInsttee
Accuracy (%)
45
45
40
35
30
25
2-
2
26
2
24
2
27
2
Number of Solutions per Problem
Number Solutions per Problem
Number of Solutions per Problem
Figure 4: MATH500 with GenRM. Weighted Best-of-N performance of three small language models, showcasing the effectiveness of ToolV with GenRM, where even generative verification cannot supplement the calculation error which can be easily filtered out by using a tool.
Baselines We compare ours, Tool-integrated verification, that utilizes both fine-tuned reward model and tool-based verifiers (ToolV), against the following baselines: (I) Majority Voting (Wang et al., 2023) (without using verifier), (2) Zero-shot GenRM (Zheng et al., 2023; Song et al., 2024) (without any fine-tuning), (3) Distilled PRM//enRRM (with fine-tuning), (4) Themis (Li al., 2024).
Models & training We experiment with the smallest eestruction-tuned models from widely used families: 2et.2.......--stttteee (Yang et al., 2024) and -l.ma-3.2-1B-Inms. (Dubey et al., 2024). (Allal et al., 2025) for evaluation in extremely small model. As the teacher model, we employ 20t-44-mini-2224-077 model is prompted to generate outputs used to fine-tune student models (Hu et al., 2022). For PRM's teacher, we use Qwen2.5-Maan -athh PRM-7B (Zhang et al., 2025).
In addition, we test M2-oLLM--360M-B10
(Hurst et al., 2024). The teacher
We include more implementation details in Appendix D.
6.2 EXPERIMENTAL RESULTS
ToolV improves PRM in small LMs As shown in Figure 3, ToolV improves performance when combined with the distilled Process Reward Model (PRM) on the MATH500 benchmark. Our results show that adding ToolV provides substantial gains in test-time scaling, suggesting that distilled PRM alone is still prone numerical errors. Notably, with TooIV, only using Llama 1B models outperforms the performance of the 8B model demonstrating that extra test-time computation can meaningfully boost smaller models, where distilled PRM alone cannot enable the IB model to reach that performance until generating 64 solutions. Similarly, ToolV enables Qwen2.5 0.5B to match the performance of the 1.5B model by generating just 16 solutions, showing impressive effectiveness.
ToolV improves GenRM in sLMs As shown in Figure 4, ToolV boosts test-time scaling for three small language models on MATH500 when combined with the distilled GenRM (Zhang et al., 2024). While GenRM struggles alone, ToolV compensates- at the cost of code generation. Similar gains appear GSM8K in Figure 5, especially for 100--M2---0M--nttmaaa the weakest model. This
7

---

Published as a conference paper at ICLR 2026
Majority Zero-shot GenRM
Distilled GenRM
ToolV + Distilled CenRM (Ours)
MmmolLM2-360M-nnsttee
Owen2.5-0.50-Instti
Llama-3.2-1B Gamma-.2-IB-Instrucc
40
70
75
Accuracy (%)
35
70
30
60
65
25
55
60
20
45
50
2
2'
2 2
26
20
2'
2
2
2
2
2
2
2
2
Number of Solutions per Problem
Number of Solutions per Problem
Number of Solutions per Problem
Figure 5: GSM8K with GenRM. Weighted Best-of-N performance comparison across three small language models. The results show that ToolV also improves model performance on graduate- arithmetic problems. However, the gains smaller this simpler task, where existing verifiers already perform reliably compared to more challenging tasks.
Majority
Distilled PRM
ToolV + Distilled PRM (Ours)
Performance by Problem Category
Performance by Problem Level
Accuracy (%)
75
75
50
50
25
25
P'recaluulus Inlernediial
Algebra
Number
Pieslgebra Goometry Counting
2
3
5
Algeerra
Theorry
Privbiiliity
Category
Level
Figure 6: Analysis with problem types and levels. We perform analysis on the effect of tool-based verifier with problem types and levels MATH500 dataset. The results are from Llama-3.2- 1-B Instruct with PRM using weighted Best-of-N (N 64). This analysis shows ToolV is most effective on mid-level problems and calculational domains.
supports our analysis in subsection 5.1 that ToolV enables even small models to memorize key information. Zero-shot GenRM ablations confirm that without distillation, small models struggle to verify solutions (Song et al., 2024).
Our two-stage method outperforms prior tool- integrated verification. Themis (Li et al., 2024) demon- strates that integrating external tools can enhance verifi- cation performance of 7B models across tasks requiring tools such as calculator, weather, or calendar. In contrast, focus is mathematical reasoning benchmarks (e.g., GSM8K, MATH500) and much smaller models that were addressed in Li et al. (2024). However, it notable that our two-stage tool-integrated verification approach proves effective for small models on math reasoning, whereas Li et al. (2024) explored a unified tool-augmented reward modeling framework.
Majentt Themss
GenRM
PBM TexiV
Teetv GenilM
Accuracy (5)
40
35
30
22
23
Number of Solutions por Problem
Figure MATH with Themis base- line. Weighted Best-of-N performance 1--ma-3.2-1BB-nnstrre model. ToolV outperforms Themis (Li et al., 2024).
Similar to our method, we first generate tool-integrated ver- ification trajectories using gpt-4o and distill them into the Llama-3.2- -- I-Istrucct model. These trajectories are sim- ilar to GenRM but explicitly include intermediate Python code generation (Gou et al., 2024b). As shown in Figure 9, Themis (Li et al., 2024) surpasses other distilled GenRM and PRM baselines without tool usage. However, both ToolV + GenRM and ToolV + PRM outperform Themis, indicating that our two-stage approach is better suited for test-time scaling on math reasoning with small models, as it can combined with distilled PRM and ensures performance improvements as analyzed in subsection 5.2.
8

---

Published as a conference paper at ICLR 2026
GenRM
ToolV GenRM (Ours)
Distribution Successful Ratio
GSM8K
MATH
GenR
Accuracy n=
TooTV GenRM (Ours)
Frequency
40
75
45
0.00
0.25
0.50
0.75
1.00
Model Size
Model Size
Coorect/ / All Solutions
Figure 7: Effects of ToolV on sizes of GenRM. Weighted Best- of-N (N- -64) performance GenRM based on different sizes
Figure 8: Correct solutions ra- tio among N 64 generations to show how the tool-baser ver- ifier works.
of Llama 3 (Dubey et al., 2024) MATH500. For ToolV, we use IB and only scale up the GenRM.
Zero-shot GenRM
Distilled GenRM
ToolV Distilled GenRM (Ours)
SmolLM2-330M-instaee
Llama-3.2-1 3.2--BBInsstruct
50
Accuracy (%)
40
35
25
29
21
23
24
2
2
20
2
2
2
2
2
2
2
2
24
2
Number of Solutions per Problem (scaled with compute budget)
Figure 10: MATH500 with GenRM under scaled x-axis reflecting compute budget. Our method remains the best even when compute budget is taken into account.
6.3 ANALYSIS
Effects of ToolV on category and difficulty In Figure 6, we analyze the N = 64 weighted Best-of-N performance on MATH500 using Llama-3.2- Baaima-..2-I-BB-- 1.B--IBBInsttuctt On the left, category-wise results show ToolV brings clear gains, especially in Algebra, Number Theory, and Counting & Probability. Geometry sees drop, likely due ToolV being less effective in that domain. On the right, performance by problem level shows consistent improvements with ToolV for Levels 2-4. However, results dip at Level 5, suggesting TooIV struggles with the most challenging problems. Overall, ToolV works best on mid-level problems and math areas requiring accurate calculation, but improvements are needed for higher-difficlly cases.
ToolV benefits larger verifiers Figure shows how performance varies with distilled GenRM size, keeping ToolV fixed at 1B. As GenRM scales, the gap with and without ToolV narrows but remains. Notably, on MATH500, 1B GenRM ToolV outperforms GenRM, suggesting ToolV can be more effective than scaling the verifier-especialll on harder tasks.
Effects of ToolV as filter Figure 8 shows how ToolV acts as an effective filter for mathematical solutions. Using 1.a--1..-------uU- with GenRM on MATH500 (N 64 candidates per sample), we recalculated accuracy after applying ToolV to remove wrong outputs. The results support our analysis in subsection 5.2, showing ToolV reliably filters out incorrect solutions and significantly improves accuracy.
6.4 DISCUSSION: COMPUTATIONAL OVERHEAD OF TOOLV
ToolV requires generating executable code in addition to pro- ducing solutions from the generator and verifier outputs such as GenRM. Understanding the compute budget is therefore important to assess the effectiveness of each method and to determine which approach is preferable under constrained com- pute resources (Singhi et al., 2025). We begin by analyzing the number of generated tokens required by the generator, the
Stage
Tokens
Solution (Generator)
574.39
Verification (GenRM)
4431.11
Code (ToolV)
610.84
Table 1: Token usage per stage.

---

Published as a conference paper at ICLR 2026
verifier (GenRM), and ToolV on the MATH benchmark using 1-t---....----I---uee Table 1 reports the average token count per solution.
As shown, ToolV introduces extra code generation tokens. This raises two natural questions: (1) How can we ensure that the performance improvement comes from TooIV itself, rather than from increased compute? If additional budget is available, why not simply use larger verifier? (2) Under equal compute budget, does ToolV still provide benefits compared to using GenRM or PRM alone? We address these questions below.
Using ToolV is more beneficial than increasing verifier size. ToolV enables the effective use of small models, which offers a practical advantage larger verifiers in terms of GPU memory requirements. Our framework allows small models to function as strong verifiers, which is particularly environments such as on-device or limited GPU setups.
valuable in
Even when accounting for the extra compute, the cost of ToolV corresponds to using only a 1.14 times larger verifier. Using Table 1, let k denote the relative scale factor. Then, k (5616.34 / the performance gain from ToolV more than justifies this small additional cost, especially when compared with simply scaling up the verifier.
574.39) /4431.11 = 1.14. Since ToolV IB GenRM IB surpasses GenRM 8B on MATH (Figure 7),
ToolV still improves performance under the same compute budget. To account for the overhead of ToolV, we shift the x axis in performance plots such as Figure 4. Applying the scaling factor of 1.14, we shift the x-axis of our method with GenRM accordingly. The resulting comparison in Figure 10 shows that even after budget normalization, ToolV continues to provide meaningful gains, particularly for the smallest model, 220.--MM---000--tttn-t and settings with the large number of generated solutions.
Additional discussion and extended experimental results are provided in Appendix E.
CONCLUSION
In this work, we introduced Tool-integrated Verification (T1), which delegates memorization- intensive tasks verification to external tools for sLMs. Our method involves a tool-based verification stage and a reward-model-basee scoring stage, both enhanced by knowledge distillation from large verifiers. Theoretical analysis confirmed that tool use substantially reduces the memorization burden sLMs and improves test-time scaling accuracy. Empirical experiments demonstrated that T1 significantly improves the test-time scaling performance of sLM on mathematical reasoning and knowledge-intensive tasks. A key conclusion of our work is that tool integration is essential for enhancing sLM performance, even under test-time scaling, by reducing the memorization burden.
Limitations & Future Works While TI shows strong improvements, some limitations remain. (1) ToolV acts only as a rejection filter and cannot recover from false negatives- -orrect solutions mistakenly rejected by the verifier. As possible implementation of T1, this limitation could be mitigated by integrating tool-based reasoning into the verification step, allowing the verifier to leverage correctness guarantees from tool outputs (Gou al., 2024b; Li et al.. 2024), which we do not explore in this work. (2) Our work focuses on best-of-N (parallel) test-time scaling, which lacks information sharing between generations. However, tools can also benefit other test-time scaling strategies, such as step-level search (Yao et al., 2023) in sLMs or long reasoning chains in sequential test-time scaling as demonstrated by Li et al. (2025). Exploring these directions presents a promising avenue for future work.
ETHICS STATEMENT
This work uses only public datasets for math and knowledge tasks and does not involve human subjects personal data. Tool based verification runs a code interpreter for numeric checks and a retriever over Wikipedia abstracts for factual checks, and these tools do not store user data. The main risks are retrieval errors and code execution failures; simple evaluation code, and we discard unverifiable claims. We follow all dataset and model licenses, disclose model families, teacher models, and training compute, and report the resources used.
bound both with conservative rules,
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We specify datasets, baselines, metrics, and the evaluation protocol for weighted Best of N with both process reward model and generative verifier. The appendix provides implementation details, training setup, key hyperparameters, and the exact prompts for code based math checks and document based fact checks. We state the retriever, source, and document count per query, and we define the rules for code execution and success signals. We will release an anonymous package with scripts, prompts, configuration. Plus, we will provide instructions and seeds to recreate all tables and figures.
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfček, Agustín Piqueres Lajarin, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of a small language model, 2025. URL https:://// /ttp///aavvvv///a///a///e.../.aa org/ abs / 2502 02737.
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alex D' Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv, 2401.01879, 2024.
URL ht //to: org/ 10. 4 55 w.taaaa.vvvv.... 01879.
Bradley C. A. Brown, Jordan Juravsky, Ryan Saul Ehrlich. Ronald Clark, Quoe V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv, 2407.217877 2024. URL. www.thttt////..... //doi worg/orgg///oo 4 11/88/5500iii0 wtw.xi..4000 2407. 21787.
Gavin Brown, Mark Bun, Vitaly Feldman, Adam M. Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, New York, NY, USA. Association for 3451131.
Computing Machinery, 2021. URL https: /tl:://dooh /ooioii.oo./ org/loggg/oooooo /10 11e11114444333
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpps:/pprcceedingg mewrdddiiiinnnnnnnsooeeeeoeuerrr wwtpccpppp/pe//ecc 2a002//hhhh//// hash// 1457c0d6bfcbr 9 6741 1144441fbacc1222642222 Absttalrratataaaaaaaaataae html.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter. Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai. Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling iestruction-finetunn language models. J. Mach. Learn. Res., 25:70:1 2024. URL https: 3ml r org/ /apprr//p////////// v25/23-0870 html.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv, 14166, 2021. URL
https:/ /ttp///axvvv.vvvv///v org/ wtwg/bbs//////oooo////////////// 2210 14168.
DeepSeck-AL, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu. Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao
11

---

Published as a conference paper at ICLR 2026
Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen. Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen. Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L.. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong. Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang. Meng Li, Miaojun Wang. Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen I Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong. Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He. Yunfan Xiong. Yuxiang Luo. Yuxiang You, Yuxuan Liu, Yuyang Zhou. Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li. Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https: / org/ abs/ 2501. 12948.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Authony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson. Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. arXiv, 2407.21788 2024. URL https: doi. org/10 0. 4 8 85500 aawivv.vvvvvv...iva....aaaaaaav. 2407 0407.21783... 21783.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: program-aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10764-10799. PMLR, 2023. URL https:/ / http///poocedeeeee/ee/ee ml www.:r/eessseee. gao22f html.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: large language models can self-correct with tool-interactive critiquing. In The Twelfth In- lernational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-1 1, 2024.
OpenReview.net, 2024a. URL https: /oppnreviii...eeeeeoeeeee. net
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlic Huang. Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. In
12

---

Published as a conference paper at ICLR 2026
The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL https: / net forum? d EpCTt iVoap.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sii-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
https: hasth//683a33ecc0b7772222222222
www.eeeedinnn.eeuiiiiieeeeee neurips cc cc/ppppe//021.111
-attraaat-tsttttt--aaaaaaaa- round... html.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. arXiv, 1503.02531, 2015. URL
httpg//bb//15000 02531.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OvenReview.net, 2022. URL https: http///enneeiieeeeeeeeeee net/
Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codis- poti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph. Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng. Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll L. Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Chamning Con- ger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang. Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeave, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, and Dane Sherburn. Gpt-40 system card. arXiv, 2410.21276, 2024. URL https://doi. .rgg 1 48550/ arxi 2410 21276.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge, In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt. Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 15797717707 PMLR, 2023. URL https: //proceedings. ml .press v202 22202/kancae wwn.kanndaaaaa 1atoppallla..a html.
Minki Kang, Seanie Lee, Jinheon Back, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge- augmented reasoning distillation for small language models in powleege-intensive tasks. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: pwwppsppppppeeeeeee nips cc wccpppper_filleee files http//paer////// 22222/aahhhh 97faedc 92 000.e00000000000000 www.ec4000022200111e0 Abst Conference..tenneeeeeee html..
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1317-1327. The Association for Computational Linguisties, 2016. URL https:://oi.. org/ 10. 18658 30/53/vl/dll 6- 1139.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, Lei M. Zhang, Kay McKinney, Disha Shrivastava,
13

---

Published as a conference paper at ICLR 2026
Cosmin Paduraru, George Tucker, Doina Precup, Feryal M. P. Behbahani, and Aleksandra Faust. Training language models to self-correct via reinforcement learning. CoRR, abs/2409. 12917, 2024. URL https:
48550/ arXiv 2409 12917.
Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang. Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. Start: Self-taught reasoner with tools, 2025. URL
petps://arxiie org/ abs 2503. 04625.
Lei Li, Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Ningyu Zhang, and Hua Wu. Tool-augmented reward modeling. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: / openrevie..eeeeeeeeeeeeee net/ / /trumm/rum///////// dd=d9 gWTUX.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OvenRevie 2024. URL https:/ /ttp//ppppp..w..e///.e net forum?id- v8L0pN6E0i.
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pp. 2356-2362, 2021.
Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can 1b IIm surpass 405b 1lm? rethinking compute-optimal test-time scaling, 2025. URL
https: aarivviiv.. org/ abs 2502 06703.
Zechun Liu, Changsheng Zhao, Forrest N. landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: /ttp//ppppppew.eeee/ee/ee net forum?rrmmi EIGbXbxcUQ.
Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu. Small language models: Survey, measurements, and insights. arXiv, 2409.15790. 2024. URL https:://// /oi. /10 wtt.d//..........///////.///////
2409:.........1...... 15790.
Dakota Mahan, Duy Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan- Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv, 2410. 2024. URL https:
004448055////i 2410. 12832.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL hetps: http/arxvvvvooo/// abs 2501. 19393.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022. 2022. URL http: http:://aapppess nips . oc hwpppappee__ae fiwef/ http//ape////eeeeeeeeeeeeeeeeeae / 2022 hash// htt.://a--....aaaaaaaattt---eeee
html.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:///penree
net www.terummt///// forum? H0HHng
14

---

Published as a conference paper at ICLR 2026
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean- Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou. Orhan Firat, Julian Schrittwieser, loannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mcllroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang. Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring. Stephen Spencer, Eren Sezener, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv, 2403.05530, 2024. URL https: //o////oii//// org/10 48550/aa 11550/arxiv 2403. 05530.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Bric Ham- bro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Lan- guage models can teach themselves Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: /ttp:://ppperssee nips wwtppaper.fileeeeee /paper/pppep2/0/ /2033/ 20223/haahr hash// d842425e 4bf wwwpeee9ba0002a0eae5ee02 ottp:/000000000000000----aaaarra ceterraat--ooffeencc html.
use tools.
Alice Oh, Tristan Naumann,
Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wet Chang, Marcus Rohrbach, and Anna Rohrbach. When to solve, when to verify: Compute-optimal problem problem solving and generative verification for Ilm reasoning, 2025. URL //tpiis:///axx...o abb/ / 2504. 01005.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv, 2408.03314, 2024. URL https: //doi. org/10 10. 3550//500iii wwe.xxii.v...... 03314.
Yuda Song, Hanlin Zhang, Carson Eisenach, Sham M. Kakade, Dean P. Foster, and Udaya Ghai. Mind the gap: Examining the self-improvement capabilities of large language models. arXiv, 2412.02674, 2024. URL https: /ttp::/doooo.... 10 1.00..8555 48550/ arxiv. 2 2412. 02674.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2024, Bangkok, Thailand, August 11-16, 2024, pp. 9426-9439) Association for Computational Linguistics, 2024a.
URL https::///// / / doii org / www.s//66.vve/.
acl 1all--onn long. .50..
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: httpss//e//eeeeee.ee..ee..tt forum? 1itmm1dd-1PPNIIMM---
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang. Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024b. URL https: http:///xxvvvvxeeeeee
2406. C1574.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022a. URL https: opwprevee...eeeeeooo//e/.. net forum?? ? iid= yzkSU5zdwD.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe V Le. Denny Zhou, et al. ofain-of-thouuh prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022b.
15

---

Published as a conference paper at ICLR 2026
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang. Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoe V. Le. Long-form factual- ity in large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: / httpppappprssss/eppppeeeepeeee nips wwtppppe__iiles http://i//le/pap/////222.. / 937ae0e8 3eboe
html.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Scaling inference computation: Compute-optimal inference for problem-solving with language models. In The 4th Workshop on Mathematical Reasoning and Al at NeurlPS'24, 2024. URL https:
copenreview.net... forum? id= 7D2WSc8qu.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue. Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren. Yang Fan. Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv, 2412.15115, 2024. doi: 10. 48550/ARRXV.2442.5555 URL https: /ooii 1rg/10. 48554/8550/00 wwt.v..ivvvvv../. 2412 211112111111 15115.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023. URL http:: http::::/ppppsssssssee nips oc wwcpppapr_fftee files hpapss/appep/////0 h 271 92 2b8 (tw.9992222d111dd7eeeeeeee2 -tw.r//aeaaaaaea1aaaaaaaaaaaaaee
html.
Thomas Zeng. Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee, Heeju Kim, Wonjun Kang, Jackson Kunde, Ying Fan, Jungtaek Kim, Hyung ll Koo, Kannan Ramchandran, Dimitris Papailiopoulos, and Kangwook Lee. Versaprm: Multi-domain process reward model via synthetic reasoning data, 2025. URL https: htt///rxivv..//////ee//ea/ee abs/ 06737.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv, 2400.11520, 2024. URL https: //oii org/10 /8550/ arXiv 2408 15240.
Zhennu Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning, 2025. URL https: arwii.iiivv..../ wwwp:iiwwrrrab///.aeaao.o.. attaaa////1... 07301.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow, 16(12):3848-3866 2023. URL https: /wwwwwww. vldb. wwwg/vvlllee pwvldb/vlll vttpvvll1/////// w005///35555/0 0555//3848-aaannn huann......... pdf..
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jin- gren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning. arXiv, 2412.06559, 2024. URL tittps://dont org/10 /10. 4854044554000 /tt//aaiii../..///vvv//eaeeaaai. 2412.444 06559.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans. LA, USA, December 10 16, 2023, 2023. URL
.iieersn.pspss. mips cc/paper__pp.....eeeee fi wwlss//pper///222
16

---

Published as a conference paper at ICLR 2026
Benchmarks html.
Xuekai Zhu, Biqing Qi, Kaiyan Zhang. Xinwei Long, Zhouhan Lin, and Bowen Zhou. Pad: Program- aided distillation can teach small models reasoning better than chain-of-thought fine-tuning. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. Association for Computational Linguistics, 2024.
17

---

Published as a conference paper at ICLR 2026
A CONCEPT-PROOF EXPERIMENT DETAILS
We provide additional details about the proof-of-concept experiment shown in Figure (b) of the main paper. The verification task focuses on arithmetic calculations involving randomly selected N three-digit numbers, using both addition and subtraction, with ranging from 3 to 10.
For each value of N, we generate 500 equations with correct answers, along with another 500 equations where the output is slightly wnchhrect-within a 5% margin of error.
We then prompt Llama-3.2 1----32-----n-nste B-Instruct (Dubey al., 2024) verify these calculations. Specifically, we use Prompt A.1 to make the language model to generate a step-by-step explanation in natural language.
Prompt A.1: Check Calculation
Evaluate the below calculation. Is this calculation correct? If correct, return True, Return False otherwise.
# Calculation: (exp) = {ans)
the caleulation is correct, return True. If not, return False.
Think step-by-step, and MUST output True False at the end of your verification.
To use the tool, we prompt LM to generate code instead of verification in natural language using Prompt A.2.
Prompt A.2: Check Calculation with Code
Generate simple Python script that evaluates the correctness a given mathematical calculation.
# Calculation: (exp) = [ans)
The script should print 'The calculation is correct' the calculation correct, otherwise print 'The calculation is incorrect'.
### Constraints:
The output must be single Python code block without any function definition.
The script should evaluate the expression as boolean comparison.
the evaluated result 'exp' matches 'ans', print "The calculation is correct', otherwise print "The calculation is incorrect'.
B DETAILS OF OuR METHOD
The choice of the component of the tool-based verification function depends on the task:
1. Mathematical reasoning task: sLM generates executable programming code C1 then the code interpreter executes the code and validates the correctness of computations (Schick al., 2023; Gou et al., 2024b). Since the code interpreter executes the code as well as outputs the verification score, regard the extraction of the verification
part the identity function, i.e., C2 = T(c). Therefore, f(x, y: T. 0) for numerical reasoning tasks can be represented as:
f(x, y: T, 0) C2 T(c1), where C1 TTCc æ, 0):
(8)
Knowledge-intessive task: The tool T acts as a retriever that returns a set of relevant knowledge passages & based the input and candidate response y. Subsequently, sLM verifies the consistency between the retrieved knowledge & and the claims within Since the retriever utilizes and y directly query, can regard the tool-calling query part the identity function. i.c. nhoweedgeintensivee tasks can be represented as: Ci = (). Therefore, f (xx y: T, 0) for knowledge intensive
f(x, y: C2 (c T (c1) y. 0);0), where e₁ (a. v),
18

---

Published as a conference paper at ICLR 2026
C PROOF FOR THEORETICAL ANALYSIS
C.I FROM THEORETICAL ANALYSIS TO PRACTICE.
Our theoretical analysis illustrates two key ideas: (1) tool integration reduces the memorization burden of small language models, and (2) the two stage design improves test time scaling by enabling a more reliable filtering function. These results are based on simplified and idealized settings and not intended capture full practical behavior. Instead, they provide conceptual illustrations grounded in existing theoretical frameworks (Brown et al., 2021; Beirami et al., 2024). The empirical trends observed in our experiments are consistent with these theoretical intuitions, suggesting that the underlying principles extend to more complex real-world scenarios.
C.2 PROOF OF LEMMA 5.1
Proof. X' has cardinality [ = 2 - -M ((M3). Also, Theorem 1.1 in Brown et al. (2021) says that any learning algorithms A that is e-close-to-optimal with sufficiently small E > also satisfies that the mutual information between data samples and the model learned by A given the data distribution is proportional to at least the number of data samples multiplying dimension of data dimension. Since the number of data samples' cardinality in subsection 5.1 is ((M)) and the data dimension is 1, we can state that 7 (X; 0| P) = ((M)). This proof says that if a model directly memorizes which (a, b, c) pairs map to each c = b with near-zero error, & must encode on the order of 3 bits of information about X.
C.3 PROOF OF THEOREM 5.2
Proof. In this case, has access to the tool function as follows:
f (a. = latb=e.
Then, 0 = A() such that f(a, b, c; 0 T) = latb=a Regardless of X, f can perfectly get label r, resulting in err,, 0.
In addition, since is determined regardless of X through A, we get
I(X;0 P) = (0) P) H(O X,
H (0 P) H(0 P)
= 0,
where is the entropy. Therefore, we can state that / (x; 01 P) = 0.
C.4 PROOF OF THEOREM 5.3
We first show Best-of-N N accuracy first in Lemma C.I. Then we show the monotonicity with respect
to the p.
Lemma C.1 (Best-of-N Accuracy with Imperfect Verifier). Let the generator output or 1 with equal probability, i.e.,
1 क (0)x) = π (1|x) 2
and let the verifier with noise level p.g be defined as follows:
0, w.p. p.
w.p. q.
Tp.g (x,, 0)
Tp.q (x, 1) 0, w.p. 1-
1, w.p. 1
with the condition that 1 and 1 p. Then, for N 2 1, the probability that the best-of-N output is ground truth label, i.e. 1, is given by
N7
1 1-
1- 1-q+29 +
π (1|#) = q 9+1
1
+
(10)
2
1
19

---

Published as a conference paper at ICLR 2026
Proof. A single sample from the generator is labeled with probability and with probability 1. Given the verifier r with noise level p and g. the joint probability are:
- 1 1 x) 40
P 0,r 1 x) = (1 - p). 2
(11)
P = 1, 0 x) = (1 - 1) 2
P(y 0- 0 x) 2 P.
Then, the probability that single sample yields a verifier score of and 0 are
p(r=1<=) x)) Σ p
Ø) 2
(1-1 p) + 2
(4)
=E(1,1}
1+p-o
-(r=0 p 1
2
respectively.
Define the event
A = {at least one Best-of-N sample is r(x, v) }
Then, from Equation 11, the probability that all N candidates yield 0 is
N
1 +
P(A)) = P(r = 0)
(12)
2
and consequently,
P(A) 1- 1 1 (1+p-
(13)
Consider cases:
Case 1. At least one candidate yields r = (event A occurs). In this case, the final output chosen uniformly among the candidates with 1. For any candidate with the probability that it originated from = 1 is given by
Plu = 1| 1,
P(y = 1,r = 1 x) P(r
29
q
x
( + p)
q +
Case 2. All candidates yield r = 0 (event A occurs). In this case, the output chosen uniformly among all N candidates. For a candidate with r = 0, the probability that it is 1 is
Ply 1, r 0 x)
(1
1 q
P(y = 1 r 0,
P(r = x
(p
p+1
By the law of total probability, the overall probability that the Best-of-N output is is
T (1 x) P(A) Pvy = r 1, 1, P(A") P(y 1 r= 0,
1 1-
= P(A) q +11-2 + P(A))
p+1- 1-1-9
From Equation 12 and Equation 13, we have
N x)
p
9
+
1 + p- ạ
1- q
(14)
2
p
2
p 1 -q
20

---

Published as a conference paper at ICLR 2026
Using Lemma C.I, Theorem 5.3 is proven follows:
Proof. Define the difference A = so that
1 p
1+4 A
2
and note that
q 1+1-p=1-A and p+1-9=11
Then, from Lemma C.I, Eqqation 14 is rewritten as
f(A) = (1 (11-22) )
1 + A
1 + A
1-
+
(15)
2
2
1+4 +
Since y is held fixed, an increase in corresponds to an increase in A. Define
1 [+4
A(A)
so that
S(A) = [1 - A(A)]
4
+
A(A)
1-Q
1- A
1+A
Differentiating equation 15 with respect to A yields
((A) = - 1-A + A(A)]
9
A'A
1 1- q
A(A)
1-4 4
(16)
A)
1 A
(1+A)2
with
N 1+
N-1
A((A) 2
2
Reformulating Equation 16 results in
((())
9
-
A(A
(1 - AAA)
+
1-
(1+ A'AA) A(A)
(1+
(a)
(6)
(a) Define
1+A Then,
2
- A(D) (1 - A) A'(A) = 11 - N xx) x)x N-1
Set g()) 1- N N N x) x. N-1 Then, g(0) =1. 1.) 0. and g(xx) S 0 induces
((x) >= 0 for æ E [0, 1]. Since q1 92 E 1,,, E [0,1] is satisfied. Therefore, we have
1 A(A) (1 A) A'AA) 0.
N-1
N
(b) Since A'(A) = 2
14A
and
A(A) =
1+4 2
we have
N
1+
A
N-1
(1 + (+)))A(A) = (1 A)
2
2
=N( 1 1+A
N
A(A).
Hence, we have
(1 (+A))((A) - A(A) = N A(A) - A(A) -NN 1) A(A).
For N > 2 and A > -1, both N -1> 0 and A(D) > 0. Therefore,
(1 + A) A' A(A) 0.
21

---

Published as a conference paper at ICLR 2026
Since (a) and (b) non non-negative, conclude that f'( 0 for all A and N 2 2. Thus, for 2 2, p > & (i.e. A > A follows that
S( s(a2),
equivalently,
(111x)
x x)
p=B
p=p
D IMPLEMENTATION DETAILS
D.1 MODEL
We use LLaMA-3.2-11 1------ttt-IIIIn (Dubey et al., 2024), 200m-2.5-0.5B-Inttrrr (Yang al., 2024), SmolLLM22360M--nssrnnud (Allal et al., 2025) as base models for our experiments.
D.2 TRAINING
Hyperparameters & Setting As mentioned in subsection 4.2, we fine-tune small language models (sLMs) for each module using LoRA (Hu et al., 2022). However, for PRM, we fine-tune the full model including the classifier head following Wang et al. (2024a). Only for we fine-tune the model on generation as achieves under 10% accuracy on both GSM8K and MATH. We organize the hyperparameter details in Table 2. We use A100 40GB GPUs with FSDP (Zhao et al., 2023) for training.
Table 2: Hyperparameters used in fine-tuning sLM for each component.
Hyperparameter
Verifier
PRM
ToolV
Learning rate
10
10-5
10
Batch size
16
16
16
Max length
2048
2048
2048
LoRA rank
64
64
LoRA
128
128
Optimizer
Adam W
AdamW 3
AdamW
Training epochs
3
Scheduler
Linear
Linear
Linear
Dataset for Distillation We perform distillation using the training split of each dataset. For MMLU- Pro, we adopt the train-test split provided by Zeng al. (2025). Training dataset sizes are 7473 for GSM8K, 7500 for MATH, 1284 for MMLU-Pro.
During distillation, we prompt the teacher model 100-4o-mmii2222-00---0 in our experiments to generate sequences used supervision for training. For the generative verifier, we follow the prompt design from Zhang et al. (2024). Specifically, we generate 8 completions per problem using temperature of 0.6, and treat these outputs the training data.
For code generation tasks, we apply the Prompt D. Using this prompt, we generate 4 completions problem at a temperature of 0.6, which are then used training samples. For fact-checking in MMLU-Pro, we similarly generate completions per problem with temperature of 0.6, using the teacher model to construct the training dataset. We use the Prompt D.2. In addition, retrieve documents for fact-checking from wikipedia abstracts using BM25 implemtented in Pyserini (Lin al., 2021).
22

---

Published as a conference paper at ICLR 2026
Prompt D.1: Code Generation
SYSTEM_PROMPT:
Write a Python code block that verifies whether a given solution correct based on the provided question, following these guidelines:
The code should be single Python block, formatted as: "'python CODE
The code should only print True if the solution is verified as correct. Otherwise, it should only print False the solution incorrect.
Use only the following built-in modules where necessary:
'math' (for floating-poin comparisons using math.isclose()
"sympy" (for symbolic calculations, including и and fractions)
"emath" (for complex number operations)
For floating-point comparisons, use math. mach.isclosse) instead ==,
Use 'sympy.pi' for and 'sympy. Rational for fractions,
Siiplify all fractions and square roots without converting them decimal values.
USER_ PROMPPRCC::::
### Input
Quesstion: [question]
-Solution: [solution]
### Output: Return python code only.
Prompt D.2: Fact-checking generation
SYSTEM_PROMME
You are domain expert.
PREE__:
Check the factual correctness of each statement in the provided solution to the question, using only the information available in the given document.
Evaluate only the explicit factual claims made in the solution. Do not verify or evaluate
the final conclusion or answer itself (e.g., The answer is ...).
If statement is factually incorrect based on the document, mark it as incorrect. If a statement cannot be verified using the document (i.e., the document does not confirm or deny it), treat it as not verifiable, and assume is correct for the purpose of final verification.
(question) (question) (/question)
(document) (document] (/document)
(solution) (solution) (/solution)
At the end of the fact check, provide final summary in the following format: Verification:
Are all statements correct? (Yes/No)? X (where X is either Yes No). If any verifiably false statement is found, output: Verification: Are all statements correct?
(Yes/No)? No
If no false statements found (i.e., all either correct or unverifiable), output: Verification: Are all statements correct? (Yes/No)? Yes
23

---

Published as a conference paper at ICLR 2026
Table Performance comparison between GenRM and ToolV GenRM. Results are from experi- ments with Llama-3.2-1 2-2-BB 1--1B-Innstinn Instrtruct on MATH500 benchmark.
Method
Accuracy
Precision
Recall
F1 Score
GenRM
80.91%
0.6153
0.7759
0.6863
GenRM ToolV
86.99
0.7666
0.7427
0.7545
Table 4: Performance of LLama-3 -B- Instrrcttt on the MATH500 benchmark for Python code generation, using teacher model outputs as reference (gold). We set rejection as positive label for computing precision, recall, and fl score.
Model Size
Accuracy
Precision
Recall
Fl Score
1B
0.7687
0.8720
0.6946
0.7733
3B
0.7973
0.8949
0.7286
0.8033
8B
0.7906
0.9207
0.6908
0.7893
D.3 EVALUATION
Hyperparameters & Setting We generate N 64 solutions using a temperature of 0.8. In the case GenRM, we follow the chain-of-thought variant proposed by Zhang et al. (2024). As in their setup, generate 8 rationales and average the correctness scores across them, following the self-consistency method (Wang et al., 2023), using a temperature of 0.6. For PRM, we apply the final score aggregation approach, consistent with previous studies (Wang al., 2024a; Snell et al., 2024). When using ToolV for mathematical reasoning, we generate code completions with a temperature of 0.6 and consider the result correct if least of the generated codes passes. In knowledge-ittensive reasoning with ToolV, we generate 4 rationales at the same temperature and consider the result correct only if all of them pass. In the case of MMLU-Pro, we retrieve three documents following the training setup. Gold documents are generated from each question using GPT-4o.
E ADDITIONAL EXPERIMENTAL RESULTS
E. TOOLV MULTI-DOMAIN KNOWLEDGE-INTENG TASKS
We demonstrate that ToolV is effective in verifying solutions across range of knowledge-intensivy reasoning tasks from the subset of MMLU-Pro benchmark (Health, Economics, History do- mains) (Wang et al., 2024b). We adapt ToolV to function fact-checker, verifying claims in solutions without other components such as query transformation and reranker (Wei et al., 2024; Kang al., 2023). We provide experimental results on the MMLU-Pro benchmark with minimal framework in this work (Wang et al.. 2024b).
As shown in Figure 12, TooIV outperforms the distilled PRM baseline derived from the Ver- saPRM (Zeng et al., 2025). For the tool, we retrieve three documents from Wikipedia using BM25. Due to variability in document quality, performance somewhat unstable in some cases. explore ToolV's upper bound, we also evaluate it using gold documents generated by GPT-4o. Results show that TooIV performance improves significantly with higher-quality documents, demonstrating its potential for multi-domain knowledge-intensive intensive reasoning.
In Figure 13, we plot the best-of-N results for all N values used in the experiments from Figure 12. Compared to math reasoning tasks, the plot is less clearly separated. However, ToolV + PRM generally outperforms the other baselines and clearly surpasses them even when using gold documents.
E.2 ACCURACY OF GENERATIVE REWARD MODEL VERIFICATION GENERATION
In Table 3, report the accuracy of GenRM with and without ToolV. The results show that ToolV significantly improves accuracy, precision, and Fl score, indicating that effectively removes false positive cases among the solutions. The confusion matrix in Figure 14 further illustrates this trend.
24

---

Published as a conference paper at ICLR 2026
Distilled PRM
ToolV Distilled PRM (Ours)
SmoLLM2-360ML 360M-instrcct
Qwen225-0058-Instruuc
Llama-3.2-1B 3.2-1B-Instrucc
10
55
Liama 3.1-88-Ins.
Accuracy (%6
Qwen2.5 1.1m2..-...Bii.
50
45
40
35
35
25
29
2
22
2
24
2
2
27
20
2
2
2
24
2
2
2
20
2
2
2
24
2
2
27
Number of Solutions per Problem (scaled with compute budget)
Figure 11: MATH500 with PRM under a scaled x-axis reflecting compute budget. PRM only performs better at small budgets, but our method surpasses as test-time scaling increases.
However, it also reveals that ToolV occasionally removes true positives, primarily due to incorrectly generated Python code.
E.3 ACCURACY OF TOOL-BASED VERIFIER IN CODE GENERATION
In Table 4, we present the accuracy of Python code generation, treating the teacher-generated code as the ground truth. The results show that precision is quite high-even for the 1B model, the distilled IB ToolV is able to filter out more than 85% of incorrect solutions among the incorrect solutions that teacher model predicted. However, the recall is low, indicating that the generated code sometimes mistakenly filters out correct solutions.
E.4 DATA EFFICIENCY OF RM AND TOOL-BASED VERIFIER IN DISTILLATION
In Figure 15, we conduct experiments to evaluate how much data is needed for each R.M-based and nool-based verifier to achieve satisfactory performance. In each plot, we reduce the distillation data to 10% and 1% for one verifier, while keeping the other verifier fully distilled using 100% of the dataset. The results show that ToolV maintains competitive performance even with only 10% data, demonstrating its data efficiency during distillation.
E.5 NECESSITY OF
DESIGN
A natural question is whether ToolV alone is sufficient for verification with small models. To answer this, we conduct ablation studies that isolate the contribution of each stage. We report results in Table and Table 7.
ToolV alone a strong verifier. It already surpasses PRM on MATH500 and GenRM on GSM8K. This shows that executable consistency checks provide an effective signal for filtering incorrect solutions. However, ToolV focuses primarily on execution- based correctness such as calculation. It does not capture higher-level reasoning errors that are not executable, such as misinterpreting problem structure or producing logically inconsistent intermediate steps.
Our ablation results show that this complementary relationship leads to consistent performance gains. ToolV alone improves over PRM and GenRM, but the two stage design (ToolV plus PRM or GenRM) achieves the highest accuracy across both MATH500 and GSM8K. These findings confirm that the two stages provide different verification signals and that both are necessary for strong verification under test time scaling with small models.
E.6 COMPUTE BUDGET DISCUSSION WITH PRM
We extend the discussion in Section 6.4. Since PRMs require no generation, adding ToolV can be interpreted as allocating roughly twice the compute budget compared using PRM alone. Figure 11 presents the compute scaled comparison. PRM without ToolV performs better when the number of generated solutions small, where the additional verification cost is not yet leveraged. As the compute budget increases, ToolV combined with PRM consistently achieves higher accuracy than PRM alone. This shows that ToolV remains effective under test-time scaling, especially for small model settings.
25

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison of GenRM variants across different numbers of generated solutions N.
Method
N = 1
N = 2
N
4
N = 8
N
16
N = 32
N
64
One-stage GenRM
27.0
31.6
36.0
40.0
42.0
44.8
45,4
Two-stage GenRM
27.0
29.6
35.0
39.8
42.0
44.4
46.0
GenRM ToolV
27.0
33.8
38.8
43.8
47.2
49.4
50.6
Health
Economics
History
43.0%
44.7%
44.7%
40
38.7%
Accuracy N=
44
40
35
42 0%
32.7%
34.1%
30
31.1%
31.1%
40.0%
28 28.0%
27.3%
Majority
Distilled
PRM (Retrieved Docs.)
ToolV PRM (Gold Decs.)
Figure 12: MMLU-Pro with PRM. Weighted Best-of-N (N 64) performance of Llama-3.22 3.2t- 2-1B- Instruct on three wwwllegge-nntnsiiie domains, illustrating the effect of different document sources in ToolV Distilled PRM (retrieved and gold documents). ToolV extends beyond math, improving PRM on multi-domain knowledge-intensiee reasoning tasks.
Majortty
Distilled FRM
TealV PRM (Retrieved Does.)
TaolV PRM (Gold Docs.)
Health
Economics
History
40
35
Accuracy
40
30
25
35
20
20 2-
26
22
20
22
2
Number of Solutions per Problem
Number Solutions per Problem
Number of Solutions per Problem
Figure 13: MMLU-Pro with PRM (Line Plot). Weighted Best-of-N performance of Llama-3.22 2BB-BB Instruct three knowledge-intensiv domains from MMLU-Pro.
E.7 TWO-STAGE GENRM WITHOUT TOOL INTEGRATION
Table 5 reports the performance two stage verification using GenRM without any tool integration. The results show that simply adding additional verification stage does not provide the gains observed with ToolV. This confirms that the improvements come from tool integration itself rather than from increased compute through multi-stage verification.
E.8 EXACT PERFORMANCE OF THE PLOOT
To provide clear measurements corresponding to Figure 3, Figure 4, and Figure 5, we include tables that report the exact performance values for each model.
For Figure 3, the exact scores are shown in Table 8, Table 9, and Table 10 for SmoILMM-360MM respectively.
Instruct, Qwen2.5-0.57 5..b---to.B-nnsttuuct and
For Figure 4, the corresponding tables are provided Table 11, Table 12, and Table 13. For Figure 5, detailed results appear in Table 14, Table 15, and Table 16.
26

---

Published as a conference paper at ICLR 2026
GenRM
GenRM + ToolV
True
6684
1930
True
6398
2216
Gold Label
Gold Label
False 4179
19207
False
1948
21438
True
False
True
False
Predicted Label
Predicted Label
Figure 14: Confusion matrix of verification results from GenRM and GenRM TooIV, where True denotes the correct solution. This result indicates ToolV improves the performance removing false positive cases. Results are from experiments with Llama-3.2 I--nutruct on MATH500 benchmark.
0.52
0.50
0:48
0.46
0.41
Tooly Scaling
GenRM Scaling
Data Scale (%)
Figure 15: Data-scale experiment. Performance comparison with varying distillation data sizes. In each plot, one verifier is distilled with 10% 1% of data, while the other uses the full dataset. ToolV remains competitive even with only 10% of data. highlighting its data efficiency. Results are from experiments with Llama-3.2- I-nstrcct on MATH500.
CASE ANALYSIS
In this section, we present cases where ToolV either enables or fails the self-verification process. All examples are taken from level 4 problems in MATH500.
In Example G.I, the solution is incorrect-it computes x as 374, which is wrong. However, the verifier (GenRM-CoT)) fails to detect this error and incorrectly marks the step as correct, ultimately concluding that the entire solution valid. In this case, ToolV implements program that checks the correctness of the solution by solving the equation using the SymPy library. As expected, the result of the check is False, successfully identifying and removing the incorrect solution.
Occasionally, ToolV may produce incorrect Python code, as discussed in subsection E.3. To il- lustrate such a case, we present a failure example in Example G.2. Here, the solution is actually correct, and GenRM correctly verifies it. However, ToolV generates code that leads to incorrect conclusion. Although the code appears reasonable, the comparison sp s simpl fy (resull 1 t sp. s mp fy (expected resdd_rrsut might return Fa se, since symbolic expressions differ in form even when they are mathematically equivalent. To properly compare equations, the code should instead use sp. Eq (...) which is more reliable for symbolic equality checks.
G LLM USAGE
We used large language models (LLMs) solely as a writing assistant, for improving grammar and clarity of the paper. No part of the research ideation, experimental design, or analysis relied on LLMs.
27

---

Published as a conference paper at ICLR 2026
MATH500
Accuracy
GSM8K
Accuracy
PRM
47.4
GenRM
71.87
ToolV
52.4
ToolV
73.62
ToolV + PRM
53.2
TooIV + GenRM
74.60
Table 6: Ablation on ToolV with PRM verifier with Alama-3.2-IB-Instmus
Table 7: Ablation on ToolV with GenRM verifier with Llama-3.2-IB-Instee B-Instruct.
Table Weighted Best-of-N accuracy on MATH500 for the Smoll M2-360M-Innstuu with PRM verification. ToolV provides clear improvements over PRM and majority voting.
Method
4
8
16
32
64
Majority
3.4
3.4
3.4
3.8
5.0
5.8
6.2
PRM Verifier
3,4
3,8
3.4
3.8
4.8
5.8
5.6
ToolV PRM Verifier (Ours)
3.4
4.6
4.2
4.6
6.0
8.0
9.6
Table 9: Weighted Best-of-N accuracy on MATH500 for the Qwen2.5-0.5B-Imse. model with PRM verification. ToolV consistently improves over PRM verification and majority voting.
Method
16
32
64
Majority
26.0
26.0
32.2
37.0
39.2
41.6
42,8
PRM Verifier
26.0
31.2
36.8
37.8
42.8
44.4
46.0
ToolV + PRM Verifier (Ours)
26.0
32.0
37.6
41.6
44.8
46.6
50.0
Table 10: Weighted Best-of-N accuracy MATH500 for the (amma-3.2-1B-Instrue model with PRM verification. ToolV provides clear gains over PRM based verification and majority voting methods.
Method
2
4
8
16
32
64
Majority
27.0
27.0
33.2
38.8
41.6
43.8
45.8
PRM Verifier
27.0
30.8
36.0
40.2
42.6
43.6
47.4
ToolV + PRM Verifier (Ours)
27.0
33.6
38.8
43.6
47.4
50.0
53.2
Table 11: Weighted Best-of-N accuracy MATH500 for the SmolLLM-3360M- -nntrrcctt model
with GenRM verification. ToolV provides consistent gains over verifier based and majority voting methods.
Method
2
4
8
16
32
64
Majority
3.2
3.2
3.2
3.6
4.8
5.6
5.8
Zero shot GenRM
3.2
3.2
3.2
3.6
4.8
5.6
5.8
Distilled GenRM
3.2
3.4
3.8
3.8
5.4
7.2
ToolV Distilled GenRM (Ours)
3.2
4.0
4.0
5.2
6.8
8.4
10.8
Table 12:
Weighted Best-of-N accuracy on MATH500 for the Qwen2.5-0.5P --nn2.5-005BBInstruue GenRM verification. The results show that ToolV provides clear gains verifier based and majority voting methods by filtering calculation errors that generative verification cannot resolve.
Instruct model with
Method
2
8
32
64
Majority
26.0
26.0
32.2
37.0
39.2
41.6
42,8
Zero shot GenRM
26.0
27.6
31.0
37.4
39.8
42.6
43.0
Distilled GenRM
26.0
31.6
34.4
38.4
40.8
43.8
44.6
ToolV Distilled GenRM (Ours)
26.0
32.0
36.8
40.8
44.0
45.0
47.2
28

---

Published as a conference paper at ICLR 2026
Table 13: Weighted Best-of-N accuracy on MATH500 for the Jlama-3.2-B-Insttuuc model with GenRM verification. The results show that ToolV provides clear gains over verifier based and majority voting methods by filtering calculation errors that generative verification cannot resolve.
Method
2
4
8
16
32
64
Majority
27.0
27.0
33.2
38.8
41.6
43.8
45.8
Zero-shot GenRM
27.0
28.0
33.8
38.8
40.6
44.4
46.0
Distilled GenRM
27.0
31.6
36.0
40.0
42.0
44.8
45.4
TooIV Distilled GenRM (Ours)
27.0
33.8
38.8
43.8
47.2
49.4
50.6
Table 14: Weighted Best-of-N accuracy on GSM8K for the 100-LMM-3360--nsse model with GenRM verification. ToolV provides clear gains over verifier based and majority voting methods.
Method
2
4
8
16
32
64
Majority Zero shot GenRM
10.92
10.92
13.57
18.35
23.43
26.69
28.51
10.92
10.92
13.57
18.35
23.43
26.69
28.51
Distilled GenRM
10.92
14.94
18.80
23.05
26.69
29.72
31.39
ToolV + Distilled GenRM (Ours)
10.92
15.31
20.92
26.54
32.75
36.85
39.88
Table 15: Weighted Best-of-N accuracy on GSMSK for the (wen2.55-0.5B-Inae model with GenRM verification. ToolV consistently outperforms verifier based and majority voting methods.
Method
8
16
32
64
Majority Zero shot GenRM
44.28
44.28
50.49
56.
59.14
60.80
61.71
44.28
43.44
50.11
56.63
59.82
61.18
62.62
Distilled GenRM
44.28
49.89
55.65
59.59
62.70
64.52
66.03
ToolV + Distilled GenRM (Ours)
44.28
50.87
56.56
62.62
64.90
66.57
68.31
Table 16: Weighted Best-of-N accuracy on GSM8K for the Llama-3.2-11 -- I-Istrutt model with GenRM verification. ToolV shows consistent improvements over verifier based and majority voting methods.
Method
2
4
8
16
32
64
Majority
49,96
49,96
58.15
64.44
67.17
68.46
69.45
Zero shot GenRM
49.96
51.18
58.76
64.75
67.48
68.99
70.58
Distilled GenRM
49.96
57.09
61.94
66.94
69.67
71.11
71.87
ToolV + Distilled GenRM (Ours)
49.96
58.15
64.29
68.23
71.04
73.62
74.60
29

---

Published as a conference paper at ICLR 2026
Example G.I: Success case of ToolV
Problem: Half the value of 3æ is 37. What the value of x??
Solution: ## Step Translate the given information into an equation
We are given that half the value of 3z 9 is + 37. This can be translated into the equation
3r-9 x: + 37.
## Step 2: Multiply both sides of the equation by 2 to clear the fraction
Multiplying both sides of the equation by to clear the fraction gives us 3r - 2x + 374. ## Step 3: Subtract 2r from both sides of the equation to isolate the variable term one side 374.
Subtracting 2x from both sides of the equation gives us x ## Step 4: Add 9 to both sides of the equation to further isolate the variable term Adding 9 to both sides of the equation gives us x 383.
The final answer is: 383
Let's verify the provided solution step by step. ## Step 1: Translate the given information into an equation
GenRM-CoT:
The solution states that half the value of 3z 9 is x + 37. The translation into the equation
3x-9 = x 37 is correct.
*eriiiication: This step is correct.
## Step 2: Multiply both sides of the equation by to clear the fraction
The solution claims that multiplying both sides by gives 3r 2x 374. This is accurate since multiplying both sides by eliminates the fraction.
*Verificatioon ** This step is correct.
## Step 3: Subtract 2r from both sides of the equation to isolate the variable term on one side The solution states that subtracting 2. from both sides results in x 374. This is also correct. ** *Verificattonn: This step is correct.
Step Add 9 to both sides of the equation to further isolate the variable term The solution states that adding to both sides gives 383. This is accurate well, as adding to x 9 gives x + 9 = 374 and thus x = 374 9. *eVeiiiication: ** This step is correct.
The final answer is given as 383 #### Final Verification
All steps have been verified to be correct, and the final answer is consistent with the calculations. Verification: Is the answer correct (Yes/No)? Yes
ToolV: import sympy sp
Define the variable
sp. symbols ( )
# Given equation
equation sp. Eq (3*x 9) / 2, x 37)
Solve the equation
solution sp. solve (equation, x) [0]
Verify the solution
is_correct sp. sp..immliff (solution) 383 print (is_correct) Interpreter output: False
Example G.2: Failure case of ToolV
Va + a
Problem: The operation & is defined for positive integers a and as a&ch
What is
Vab-
the value of 9&:2? Express your answer as common fraction in simplest radical form.
30

---

Published as a conference paper at ICLR 2026
Solution: Step 1: Substitute the given values of and into the definition of &c. We have 9 and b = 2, so we can substitute these values into the definition of & as follows:
(2) +9
9&:2
(()) -
## Step 2: Simplify the expressions inside the square roots. We can simplify the expressions
inside the square roots as follows:
V
V27
9&2
V18
V16
## Step 3: Further simplify the expressions by finding square roots of perfect squares inside the fraction. We can simplify the expressions by finding square roots of perfect squares inside the fraction as follows:
3v3
9&-2
3v3
The final answer is:
GenRM-CoT: Let's verify the solution step by step. Step Substituting values The solution correctly substitutes and b 2 into the expression for 9&:2:
(9) 9&22 (() -
This step is correct.
## Step 2: Simplifying inside the square roots The next part of the solution simplifies the expressions inside the square roots:
v18+9 99
227
9&:2
vI8
V16
This simplification is also correct. Step Finding square roots The solution proceeds find the square roots:
3v3 3
4
This simplification is correct as 33/ and V16 ## Final Answer The final answer given is:
3v3
4
This matches the expected answer provided in the verification section. In summary, each step in the solution executed correctly and led to the correct final answer. Verification: Is the answer correct (Yes/No)? Yes
Example G.3: Failure case of ToolV (cont.)
ToolV: import sympy as
Define the operation
def operation (a, b):
return sp.sqrt ( (a * b + a) / (a + b b))
Given values 9
31

---

Published as a conference paper at ICLR 2026
# Calculate the result
result operation (a, b)
Expected result
appected_reeult sp. Rational (3, 4) sp. (3)
Verify if the result matches the expected result is ss_correct sp.s sp.simpli fy (result) print ( fis_correct) Interpreter output: False
sp. sp.simplif (expected_resullt
32


Reviews:
[{"review_id": "UPU7YKv52u", "reviewer": "Reviewer_qnQz", "summary": "This paper proposes Tool-integrated Verification (T1), a two-stage framework that uses external tools (e.g., code interpreters) to filter candidate solutions before sLM-based verification, offloading memorization-heavy tasks like numerical calculations. The authors provide theoretical analysis showing tools reduce memorization requirements and prove the two-stage design improves test-time scaling performance. Experiments demonstrate that T1 enables a Llama-3.2 1B model to outperform Llama-3.1 8B on MATH benchmarks, improving verification accuracy for both PRMs and generative verifiers.", "questions": "1. What’s the exact performance of the proposed method and baselines in the chart? Could you provide some statistics at least for some key points?\n2. The MATH500 and GSM8K are fairly easy benchmarks. Could you try the more challenging benchmarks such as AIME 24, AIME 25 (also math domain)? Perhaps the method will shine better for challenging tasks.\n3. As mentioned in the weakness section, could you add the ablation and the cost-benefit analysis?\n4. Theoretically, the proposed method is not limited to the small LMs. Will the performance improvement vanish if the method is based on larger LMs?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper enables small language models to be reliable verifiers by combining it with the tools and boosts their performance by test-time scaling. The paper also provides solid theoretical proofs for the method, not only to well explain the method in the paper, but also to show a general analysis foundation for future research. The experiment design is also comprehensive, where the proposed T1 is compared with Majority Voting, Zero-shot GenRM, Distilled GenRM and Themis. The performance improvements are also remarkable on Math500 and GSM8K.", "weaknesses": "- Limited novelty and narrow focus: The core idea of using code interpreters for mathematical verification is well-established (PAL, Toolformer). The contribution is mainly combining existing techniques in the verification stage rather than introducing new methods. The paper focuses almost exclusively on mathematical reasoning (MATH, GSM8K), a narrow domain where code execution naturally provides ground truth verification. Even for MMLU-Pro tasks in the appendix, that retrieval helps verification is unsurprising and well-known from prior RAG literature.\n- Missing key ablation: The paper fails to evaluate ToolV alone as a verifier, only ToolV + RM combinations. This makes it hard to assess the individual contribution of each stage or determine if the two-stage design is necessary versus using ToolV alone.\n- Missing cost-benefit analysis: No computational overhead reported for generating and executing code or retrieval, unclear if performance gains justify additional inference cost versus using a larger model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "edG8Na7CA7", "reviewer": "Reviewer_sFSt", "summary": "The paper proposes Tool-integrated Verification (T1): a two-stage verifier for small LMs (sLMs) that (i) first filters candidate solutions using a tool-based verifier (ToolV) e.g., running generated Python or fact-checking with retrieved documents and (ii) then scores remaining candidates with a reward-model verifier (PRM or generative verifier). The core idea is to shift memorization-heavy verification to external tools so sLMs benefit more from test-time scaling (Best-of-N). The method is trained via multi-LoRA distillation from large teacher verifiers. Experiments on MATH500, GSM8K, and MMLU-Pro report that sLMs with T1 outperform or match much larger models, although some larger-model baselines are run with N = 1 while sLMs use N up to 64, which complicates fairness.", "questions": "* Cost/latency is missing for a method that leans on tools. Can you report tokens, wall-clock, and $ per instance for T1 vs PRM/GenRM-only under equal budgets?\n* Please show accuracy vs k code completions (e.g., 1–8), vs # retrieved docs (1–5), and vs verifier thresholds. These ablations would isolate where T1 helps most.\n* Which components of the theorems do you believe carry over to real math/RAG settings, and which do not? A brief “theory-to-practice” paragraph would calibrate reader expectations.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Clear theoretical decomposition of sub-optimality into OTC and a verifier-driven factor; three-regime picture is intuitive and actionable. (Theorem 3.6; Fig. 1.)\n* Precise AiC analysis with explicit coverage violation condition and sub-optimality formulas.\n* SMC construction via maximal coupling with matching complexity to SRS; neatly links transport optimality with compute.", "weaknesses": "* Plots lack error bars/CI; seeds aren’t disclosed. Given the sampling-heavy setup, dispersion matters. Report mean ± sd over ≥3 seeds, plus 95% CIs or stratified bootstrap CIs for all main tables/figures.\n* The binary-generator and simple addition check help intuition but are far from math word-problem or retrieval settings. The paper should tighten the stated scope and discuss where the guarantees plausibly transfer.\n* We don’t see how accuracy moves with # ToolV code completions (k), # retrieved docs, verifier thresholds, or retrieval quality (BM25 vs dense, top-k). These directly test your causal story; please add these sweeps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RBX0uwMxYN", "reviewer": "Reviewer_tsxd", "summary": "Tools can offload specific skills, such as arithmetic or fact retrieval, from language models to purpose-built systems. This paper extends that idea to verification: it introduces a tool-integrated verifier that allows smaller language models to act as effective verifiers. The method first applies a tool-based binary filter to discard incorrect candidates, and then uses a reward model to score the remaining answers. A simple theoretical model suggests that tool integration reduces the need for memorization, and experiments on GSM8K, MATH500 and MMLU-Pro show improved Best-of-N performance.", "questions": "Please provide the 2 suggested ablations above, and provide plots showing total cost in the x-axis, not only the number of generated solutions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Well written.\n* Consistent empirical gains across several model families and sizes.\n* Theoretical intuition connects tools to reduced memorization burden and to improved Best-of-N selection via reduced verifier noise.\n* The fact that ToolV and GenRM stages are just LoRA adapters of the same model (which is the same as the reasoner) makes the deployment in low-resource scenarios easier.", "weaknesses": "1. I understand that offloading to tools allows smaller models, but the design choice seems odd: why restrict tool use to the verifier? Qualitative examples (e.g., Example G.1) show the verifier merely calls an equation solver that could equally have been used by the generator. Giving tools to the reasoner feels like the more direct and widely studied approach (e.g., Toolformert, PAL, TORA), so the novelty here mainly comes from an asymmetric setup rather than a new tool integration method.\n\n2. It’s unclear if the improvements from Distilled GenRM → ToolV + Distilled GenRM arise from tool use or simply from adding another verification stage. A control experiment where the two-stage verifier is used but with no external tools would clarify whether the tools themselves drive the gains.\n3. The proposed approach spends roughly 2× verification compute compared to the baseline, yet this is not accounted for. If Figures 3 and 4 plotted accuracy against the number of generated solutions, the ToolV + Distilled PRM curve should arguably be shifted right to reflect the extra verification cost. This additional cost might erase scaling gains in several cases. Without such normalization, it is difficult to judge compute-efficiency trade-offs fairly. Reporting wall-time, total output tokens, or at least the number of model calls would provide a more balanced comparison than using only the number of generations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)