Token-Supervised Value Models for Enhancing Mathematical Problem-Solving Capabilities of Large Language Models
Authors: Jung Hyun Lee
June Yong Yang
Byeongho Heo
Dongyoon Han
Kyungsu Kim
Eunho Yang
Kang Min Yoo
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Mathematical Problem-Solving
Verifiers
Primary area: foundation or frontier models, including LLMs

Abstract: With the rapid advancement of test-time compute search strategies to improve the mathematical problem-solving capabilities of large language models (LLMs), the need for building robust verifiers has become increasingly important. However, all these inference strategies rely on existing verifiers originally designed for Best-of-N search, which makes them sub-optimal for tree search techniques at test time. During tree search, existing verifiers can only offer indirect and implicit assessments of partial solutions or under-value prospective intermediate steps, thus resulting in the premature pruning of promising intermediate steps. To overcome these limitations, we propose token-supervised value models (TVMs) -- a new class of verifiers that assign each token a probability that reflects the likelihood of reaching the correct final answer. This new token-level supervision enables TVMs to directly and explicitly evaluate partial solutions, effectively distinguishing between promising and incorrect intermediate steps during tree search at test time. Experimental results demonstrate that combining tree-search-based inference strategies with TVMs significantly improves the accuracy of LLMs in mathematical problem-solving tasks, surpassing the performance of existing verifiers.

Published as a conference paper at ICLR 2025
TOKEN-SUPERVISEN VALUE MODELS FOR ENHANC- ING MATHEMATICAL PROBLEM-SOLVING CAPABILI- TIES OF LARGE LANGUAGE MODELS
Jung Hyun Lee June Yong Yang 112, Byeongho Heo', Dongyoon Han',
Kyungsu Kim*, Eunho Yang" Kang Min Yoo
NAVER Cloud KAIST AI NAVER AI Lab SNU 5 AITRICS
fonliwad101, 1aaedi1,,,loooooooooooooooo laoconeth (eg.netthemmmm...mmmma com
dongyoon han, kangmin www.mm....oo/eaaaeeere...e...... yoo]enavercorp.com
kyskimesnu ac. kr, wwnhoy@kaist.ac. ac.
ABSTRACT
With the rapid advancement of test-time compute search strategies to improve the mathematical problem-solving capabilities of large language models (LLMss), the need for building robust verifiers has become increasingly important. How- ever, all these inference strategies rely on existing verifiers originally designed for Best-of-N search, which makes them sub-optimal for tree search techniques at test time. During tree search, existing verifiers can only offer indirect and implicit as- sessments of partial solutions under- under-value prospective intermediate steps, thus resulting in the premature pruning of promising intermediate steps. To overcome these limitations, we propose token-supervised value models (TVMs) new class of verifiers that assign each token probability that reflects the likelihood of reaching the correct final answer. This new token-level supervision enables TVMs to directly and explicitly evaluate partial solutions, effectively distinguish- ing between promising and incorrect intermediate steps during tree search at test time. Experimental results demonstrate that combining treesearcc--aasee infer- ence strategies with TVMs significantly improves the accuracy of L.LMs in math- ematical momblem--olving tasks, surpassing the performance of existing verifiers.
INTRODUCTION
Although recent large language models (LLMs) (Jiang et al. 2023 Dubey et al. 2024) have show- cased extensive capabilities across various domains, they still face challenges with complex multi- step reasoning tasks such as mathematical problem-solving. Considering that existing reasoning problems can often be solved by drawing inferences from pre-trained knowledge (Snell al., 2024), recent studies have focused on inference techniques that invest additional computational effort at test time better elicit the appropriate knowledge from these models. The simplest and most conven- tional inference strategy to scale up test-time compute is Best-of-N search (Lightman et al.) 2023), which selects one of the N' generated solutions based on specific criterion. For solving math word problems, since automated tools exist to verify the exact correctness of candidate solution at test time (as the ground truth answer is unavailable by definition), researchers have introduced the use of neural verifieers trained to assess the correctness of the candidate solution in Best-of-N search.
Existing verifiers for Best-of-N search in mathematical wroblem-solving tasks can be categorized into two types: supeeviee reward models (ORMs) and supervied reward models (PRMs). ORMs (Cobbe et al. 2021 Uesato et al. 2022) are trained to assess the correctness of a solution by labeling every token in solution as either correct incorrect based solely whether the final answer in the solution is correct. In contrast, PRMs (Uesato et al.] 2022 Lightman et al.] 2023 Wang et al. 2024b c) Chen et al. 2024 Luo et al.) 2024) are trained with step-level labels to assess the correctness of each intermediate solution step. Thanks to their finer-grained assessment, PRMs are better equipped to diagnose errors even when only few intermediate steps are incor-
"Equal contribution.

---

Published as a conference paper at ICLR 2025
<GSMBK PROBLEMD: Jon zuna telathion. Is takea hin 4D manutes for the ovix, an hour and 20 minates for the bise xide and so minutes for the run. compazed to Jon, Janes rinistes the ewin 10s faster but takes ninutes longer the bike, If Jon ninute,, how long did it take Janes to do the run? (Anawer: 59)
Answer
Step
PartialSolutton
Partial Solution
Solution
ORM
60+20-80
80-90=170
time was 170-10=1000 minutes.
(Score: 0.914)
60+20-80
60-90.170
firished 170 minutee (Score: 0.900)
160 121 Answer: 39
100 180
(Score: 0.175)
WRONG
the racein in 170+ 180 mindtes.
(Score: 0.160)
PRM
60+20=80
60-90e170
James took 170-10=160 minutes (Score:
60-20-80
80-90-170
total time
minutes.
160 121 Answer: 39
total time 10 minutes longer (Score: 0.657)
WRONG
total time is 10 minutes longer (Score: 0.646)
TVM
60+20-80
then look 170+10 minutes (Score: 0.777)
85-121
60+20=80
80-90:170
Rrished 10:00-011100000 160 minutes
(Score: 0.445)
180 121 Answer: 59
Brished 170 10:100
minutes
0.411)
CORRECT
have finished 170-10=160
(Score:: 0.354)
Illustration of ORM's, PRM's, and TVM's (ours) intermediate steps and their corresponding scores under Step-by-Step Beam Search for a test problem in GSM8K. The ORM incorrectly predicts wrong intermediate steps (colored red) with excessively high scores (0.914 or 0.906) while assigning very low scores (0.175 0.160) correct steps (colored blue). Although the PRM predicts correct steps with higher scores (0.657 or 0.6418 than the ORM, it still assigns comparable scores to correct steps with respect to scores of wrong steps (0.710 (10 or 0,661), causing the premature pruning of promising steps. Yet, the TVM accurately predicts a correct step with high score (0.777) and wrong steps with relatively low scores (0.448, 0,411, 0.354), thus improving the performance of neee-search-ae based inference algorithms over both ORM and PRM.
Figure I:
rect, thereby resulting in a lower false positive error (Lightman et al. 2023). As result, PRMs are generally considered more robust and preferable as verifiers than ORMs when using Best-of- N search. Nonetheless, the accuracy of Best-of-N search typically plateaus once N exceeds a few hundred (Brown et al., 2024).
To more effectively utilize additional test-time computation to enhance the mathematical problem- solving capabilities of LLMs, tree search algorithms such as Step-by-Step Beam Search and Monte Carlo Tree Search have been actively explored as alternatives (Yu al. 2024 Feng et al. 2024 Chen
et al. 2024 Wu et al.) 2024 Snell et al. 2024) to the Best-of-N search. While Best-of-N search only permits neural verifiers to score solutions after they are fully generated, rre--eeach--base inference methods enable verifiers intervene during the solution generation process. This results in improved performance compared to Best-of-N search while utilizing less inference-time computation. Yet, all test-time search strategies rely on existing verifiers (i.c., ORMs and PRMs), which were initially developed for Best-of-N search. As a result, it remains unclear whether both ORMs and PRMs are suited for -res-eeacc-baased inference techniques. For instance, when implementing tree-search- based inference strategies, Yu et al.) (2024) argued that ORMs are more suitable than PRMs while Wu et al. (2024); [nnel et al. (2024) favored PRMs instead of ORMs. this paper, we will elucidate our assertion that both ORMs and PRMs have drawbacks in utilizing additional computation at inference time with tree search algorithms.
One key property to maximize the accuracy of LLMS in solving math word problems through tree search at inference time is that poor intermediate solution steps must be preemptively filtered out, while promising ones should be preserved and further explored. In turn, the verifier needs to be trained to predict which partial solutions are on the right path toward a correct final answer. How- ever, both ORMs and PRMs possess limitations in achieving this. Specifically, given that ORMs are trained with every token labeled as either correct or incorrect based merely on whether the final answer is correct, they can only infer the potential correctness of partial solution implicitly and
2

---

Published as a conference paper at ICLR 2025
indirectly. Additionally, since PRMs are trained to determine entire intermediate step incor- rect even only the final few tokens are erroneous, they are prone to valuingg prospective intermediate steps by assigning scores comparable to those of incorrect steps shown in Figure !I
In this work, we reveal that PRMs lacking in intra-step supervision systematically exhibit a high false negative error in evaluating the correctness of intermediate steps. This phenomenon results in the under-valuing and premature pruning of promising intermediate steps during tree search at test time (see Figure [ negatively impacting task accuracy. To equip verifier with a more direct and explicit ability to evaluate partial solutions while minimizing false negative errors, we propose token laken-supervised value models (TVMs) new class of verifiers trained by supervising each token in solution with the probability of reaching the correct final answer. As TVMs are designed to di- rectly and explicitly predict the potential correctness of a partial solution, they are more effective at evaluating whether a partial solution is on promising path toward the correct answer than ORMs. In addition, unlike PRMs, TVMs benefit from token-level supervision with distinct correctness proba- bility scores. When labeling tokens in an incorrect intermediate step, only the last few tokens can be labeled as zero (i.e., incorrect) while the rest are assigned with positive values. This enables TVMs to separate promising intermediate steps from incorrect ones (see Figure I and thus attain lower false negative error than PRMs, while preserving a false positive error comparable to that of PRMs. Therefore, TVMs demonstrate improved performance in vree-search-basel inference strategies over both ORMs and PRMs.
Our contribution is threefold:
To the best of our knowledge, we are the first to disclose that PRMs produce a high false negative error, which we demonstrate to be detrimental to the performance of tree-search- based inference algorithms due to premature pruning of promising intermediate steps.
We propose the cuper-supervised Value Model (TVM) new type of verifiers that trained to directly and explicitly estimate the likelihood of reaching the correct final answer for each token in solution. The TVM achieves lower false negative error than the PRM, while maintaining a false positive error comparable to that of the PRM, which makes the TVM particularly suitable for tree-search-base based inference methods (see Figure D.
We provide theoretical insight that the value of each token equivalent the probability of reaching the correct final answer given until that token, which leads us to name the token-supervised value model, not a reward model.
2 TEST-TIME STRATEGIES FOR MATHEMATICAL. PROBLEM-SOLLVNG
This section briefly reviews test-time strategies to enhance the mathematical problem-solving capa- bilities of large language models (LLMs) by leveraging additional compute at inference time. Here, we discuss two main test-time search strategies: (i) Best-of-N Search and (ii) Tree Search.
Best-of-N Search. The most basic approach to test-time compute utilization is to sample N solu- tions in parallel and choose the one most likely to be correct, which known Best-of-N search. For tasks such as code generation and neural theorem proving, each of the N candidate solutions can be automatically identified as either correct or incorrect using unit tests or proof assistants (c.g.. Lean 4 (Moura & Ullrich, (021)), leading to improved pass rates as N increases (Brown et al.) 2024]. However, for math word problems, determining the correctness of a solution cannot be auto- mated without knowing the ground truth answer at inference time. To address this, neural verifier is employed to assess the correctness of the N sampled solutions. In this framework, is critical to control the false positive error of the verifier, as it has to identify the single most probable solution and discard the rest.
Tree Search. An advanced method for test-time compute utilization to alter the reasoning tra- jectory of an LLM by allowing the verifier to intervene in the intermediate steps of the reasoning process, which can prevent errors in earlier steps from propagating to subsequent steps. To this end, researchers have actively explored Tree Search more effective alternative to Best-of-N search. One sonyy-t-impleeens and well-studied tree-search- based inference strategy is Step-by-Step Beam
Search YYu et al. 2024 Chen al. 2024 Snell al. 2024). It operates as follows: (i) The L.M
3

---

Published as a conference paper at ICLR 2025
Reasoning Step
Reasoning Step
DXM
(Outcome connect)
OMM
[Dutcome cORnECT)
Path
6.5 (Proces Neuriee
[Process CONNECT)
CORRECT
TVM
0.
0.3
93
0.5
25
0.5
TVM
65
0.5
03
0.5
1+
Token NEUTIAAL
Tokon NEUTRAL
Tokos coNREc)
DMM
[Cuttame WROwG
CRM
(Cutcome WRONG
AinelAmower
Path 2
0.5 (Prooos NEUmMA]
[Prosos (emore))
WRONG
TVM
05
0.5
8.5
0.5
as
0.
TVM
25
0.5
6.5
0.5
0.5
Token NEUTRAL
Toben NEUTTAA
Taken IRROWG
Figure 2: Illustrative comparison of token-level supervision (TVM; ours) with outcome supervision (ORM) and process supervision (PRM). We provide two examples for each correct and wrong reasoning path. Outcome supervision employs homogeneous labels judged by the correctness of an entire reasoning path, uniformly labeling all tokens in every reasoning step as either correct or wrong. While process supervision uses percentage score (i.e., 0.5) before the reasoning step 4, it assigns the reasoning step 4 to zero due to the incor- rectness of only the final few tokens (see the bold box), which causes high false negative error. On the other hand, token-level supervision (ours) allows for labeling only the last few tokens as zero while giving positive values to the remaining tokens the reasoning step 4, thus achieving a lower false negative error than process supervision and distinguishing promising intermediate steps from incorrect ones as demonstrated in Figure
generates K initial reasoning steps in parallel. (ii) The verifier scores the K steps, pruning low- scoring ones and retaining only b steps in the beam. (iii) For each of these b candidates, the next K/b steps are generated in parallel, leading to 6 x K/b = K new subsequent steps. (iv) This pro- cess is repeated until stopping criterion is met. A variation of this technique, known as Reward Balanced Search (REBASE), introduced by Ww et al. (2024), balances the pruning and expansion width of the K steps at each depth. Additionally. Monte Carlo Tree Search and its variants have been explored, but multiple studies YYu et al. 2024; Chen et al. 2024 Snell et al., 2024 Wu et al. 2024) indicate that they generally underperform compared to Step-by-See bySetp Beam Search and REBASE.
3 PITFALLS OF OUTCOME AND PROCESS SUPERVISION
In Section 3.1 we outline the limitations of outcome-supervised reward models (ORMs) and process-supervised reward models (PRMs) for tree search strategies test time by using illustra- tive examples presented in Figure [ Section 3.2 provides a brief overview of the preliminary setup for training neural verifier enhance the mathematical problem-solving capabilities of large lan- guage models (LLMs). In Sections 3.3 and 3.4 we then analyze the issues arising from outcome supervision and process supervision, respectively.
3.1 PROBLEM STATEMENT
To maximize LLMs' problem-solving accuracy with tree search test time, crucial effec- tively prune poor intermediate steps while exploiting prospective ones. This requires a verifier that can accurately predict which partial solutions are on the right track toward the correct final answer. However, we hypothesize that existing supervision approaches (i.e., outcome and process supervi- sion) have their own inherent limitation when implementing tree search strategies at test time.
Figure T illustrates that ORMs assign excessively high scores (e.g., 0.914 or 0.906) to incorrect intermediate solution steps while giving totally low scores (e.g., 0.175 or 0.160) to correct ones. Although PRMs score correct intermediate steps (e.g.. 0.657 or 0.648) higher than ORMs, PRMs still assign comparable scores to correct steps with respect to scores of incorrect steps (e.g., 0.710 or 0.661). leading to the premature pruning of promising steps. In Sections 3.3 and 3.4 we further validate our hypothesis by demonstrating that ORMs generate less accurate value estimates, while PRMs under-value promising steps due to high false negative errors.
4

---

Published as a conference paper at ICLR 2025
Table 1: Root mean squared error (RMSE) between true value and a verifier's value estimation at the token level on the test set of GSM8K for Mistral 7B and Mistral 7B MetaMath, where verifier either ORM or TVM. We approximate true value of a token Eq. boyy sampling 256 reasoning paths per test problem, because it is intractable to obtain the ground truth value due to the infcasibility of calculating the expected returns analytically for all possible paths.
Method Mistral 7B
Mistral 7B MetaMath
ORM
0.2813
0.2471 0.2406
TVM
0.2575
3.2 TRAINING A NEURAL VERIFIER FOR MATIEMATICAL PROBLEM-SOLVING
Since LLMs are autoregressive models based on next-token prediction, they are incapable of retract- ing or modifying previously generated outputs. For mathematical prblem--ollving reward models can be employed as verifiers evaluate the correctness of the generated outputs. A verifier is trained via supervised learning on dataset obtained by sampling multiple solutions per training problem 2rr using the LLM. Given a training math word problem (tr as an input, the LLM first generates Nor solutions (or reasoning paths), where the n-th reasoning path consists of reasoning steps { [ and final answer On for = 1, ., .yr.. In token-level notation, the n-th reasoning path can also be expressed as a sequence of tokens, denoted by ( The final answer is correct if equal to the ground truth answer à, and incorrect otherwise. To train a verifier, supervision is traditionally given in two ways with respect to its gramularity: (i) outcome supervision in ORMs (Cobbe et al.
2021
Uesato al.
2022) and (ii) process supervision in PRMs (Uesato al. 2022
Lightman al.
2023
Wang et al.
2024b Chen et al. 2024
Luo et al. 2024
3.3
OUTCOME SUPERVISION
An ORM (Cobbe et al. 2021 Uesato al. 2022), fORM is verifier trained to model the outcome reward function (-). which is the correctness of a final answer:
1 i f
(a
To train fORM, outcome supervision is employed. Given Nor reasoning paths generated for train- ing problem Yer, as described in Figure outcome supervision labels every token in each reasoning path correct if its final answer is correct, which is precisely the outcome reward (Eq. T In turn, the ORM loss for a CORM is defined as:
Nic To
LORM= Σ
D e (ro(an); FORM (gtr, la,1, (a,1,,n,2, kn,2," " kn,k))
(2)
where the mean squared error is typically used as the loss function f(.). Note Cobbe al. (2021) demonstrated that a token-level verifier trained to judge the correctness for every token in solution improves over solution-level verifier trained to determine the correctness only for the final token.
Albeit designed as a reward model for Best-of-N search, ORMs can be alternatively described modeling the cumulative reward for each token, where all intermediate rewards are zero (i.e.,
r(bn,)) = 0 for every and k) and the discount factor 7 is set to 1 (Yu et al. 2024). The cumulative reward following an intermediate token ln.k. P((m,,)== Ei=n" ( (((,, is calculated as
+ 1=11= if an à 0 + 0 if a, + à,
0
R(f,,x) r(t,,t+1) + r(tn,T.) (a,))
(3)
which is equivalent to (a,) in Eq. This implies that an intermediate reasoning path is labeled as correct if the final answer is correct, and vice versa. In this sense, Yu et al. (2024) showed that ORMs indirectly and implicitly learn the potential correctness of an intermediate reasoning path.
However, such implicit supervision through homogeneous token labeling renders ORMs to still fall short in precisely evaluating whether an intermediate reasoning path is a promising track towards the correct final answer. This can be corroborated by not just Figure T but also Table which shows that the ORM yields less accurate value estimates compared to the TVM.
5

---

Published as a conference paper at ICLR 2025
TVM (Ours)
TVW (Ours)
PRM
30%-
PAM
42%
27%
69
Verifier's per solution
Verifier's per step
(a) Precision- Recall Curve
(b) Verifier's Scores Histogram for Correct Sampled Solutions/Steps
Figure 3: Comparative analysis of PRM and TVM (ours) by sampling 256 solutions per test problem in GSM8K gathering all sampled solutions (i.e., 1319- 337664 solutions). (a) The precision- recalll curve of PRM and TVM by adjusting the classification threshold hetween 0.4 and 0.6 increments 0.05. (b) Histogram of a verifier's scores for correct sampled solutions (left) and their corresponding steps (right).
3.4 PROCESS SUPERVISION
To outperform outcome supervision for Best-of-N search, process supervision enables step-wise assessments of reasoning path through explicit training on the correctness of each individual rea- soning step, which is finer-grained supervision than outcome supervision. The correctness of each step either labeled via human annotation (Uesato et al. 2022 Lightman et al., 2023) or automa- tion (Wang et al.) 2024b Chen al. 2024 Luo et al. 2024). Since acquiring human annotations is labor- iabor-intenssve and costly, we mainly focus on process supervision without human annotations.
Following Wang al. 22024b,, $n.j is annotated by sampling fixed number of reasoning paths con- ditioned on sequence of intermediate reasoning steps $p,1,' "", If all of the sampled reasoning paths reach wrong final answers, Sn,j is labeled incorrect with the process reward rp rp($n.)) 0. Otherwise, $n.j can be labeled as either (p(5))) = 1 (i.e., correct) or the probability of the sam- pled reasoning paths reaching the correct final answer. Using the per-step labels obtained through automation, a PRM is trained to provide a step-level assessment by minimizing the following loss:
8n.j.
Ner S.m
LPRM Σ
(rp(snas) frrm 8n,1, 8n,2. " 8n..))
where l denotes the binary cross entropy loss.
This form of step-level supervision improves the identification of errors even when only few in- termediate steps are erroneous, leading PRMs have low false positive errors (i.e., high precision), as discussed in Lightman et al. (2023). However, we observe that PRMs without human annotations suffer from high false negative errors (i.e., low recall), because process supervision labels an entire intermediate step incorrect even if only the final few tokens are wrong (see Figure D As a result, as shown in Figure 3(a) PRMs without human annotations exhibit significantly lower recall com- pared to proposed verifier, the TVM. Hereafter, refer to PRMs without human annotations simply as PRMs to keep the expression concise.
To further investigate the PRM's recall, we compare the scores assigned by the PRM and TVM to correct sampled solutions (the left side of Figure 3(b) and their corresponding steps (the right side of Figure bbb). Due the PRM's ' significantly lower recall, its overall solution scores naturally lower than those of the TVM. Surprisingly. the PRM also assigns lower scores to individual correct steps compared to the TVM, resulting in smaller proportion of steps being scored close one. This supports that the PRM tends to under-value promising intermediate solution steps, thus resulting in premature pruning during tree search at inference time.
METHOD
This section introduces proposed verifier, the Token-supervised Value Model (TVM), which is based on a new token-level supervision approach to directly and explicitly estimate the probability of reaching the final answer for each token along reasoning path. We first outline how to empirically compute por-token correctness probability scores from the N,, generated reasoning paths for token- level supervision. Then, we provide theoretical insight into our proposed verifier value model.

---

Published as a conference paper at ICLR 2025
Generated reasoning paths (N₂-
Answer
Toben
fL2
LLa-1
Path
Terry
took
Jerry
took
Linda
tock
and
Stary
look
656
X
Fex
Path 2
Terry
took
Jerry
took
and
Linda
t00k
com
Vn
84 X
Fax
Path
took
0.35
0.33
0:33
0.33
0.35
0.35
0.5
0.5
0.5
0.5
0.5
055
05
Figure 4: Illustration of token-level supervision with correctness probability scores using Eq. For a single training problem qt. Nrr rcasoning-ansee answer pairs are sampled using an LLM. Here, let Nor 3 for convenience. (1) All three sentences begin with the same tokens {fi, fim1 correct final answer (357). Accordingly, every token position, however, only one sentence starts with f1,a, which reaches an incorrect final answer (656). Thus, all tokens after fı,a are labeled as 0/1 = 0. (3) The remaining two sentences continue with the same tokens {t2a 16-1 only one of which is correct. Hence, every token of {tz.a 1b-1 is labeled as 1/2 = 0.5. (4) Finally, at the b-th position, which one is correct is pre-determined. As a result, all tokens after 22,6 are labeled as 0/1 = 0, whereas all tokens after 13,6 as 1/1
1a-1 and only one of them reaches the
[= labeled as 1/3 0.33. (2) At the a-th
4.1 TOKEN-LEVEL SUPERVISION WITH CORRECTNESS PROBABILITY SCORES
To supervise each token in a reasoning path according to its potential in deducing the correct final answer, we label each token as the probability of reaching the correct final answer conditioned on until that token. To be more concrete, we label an intermediate token t, as
(the final answer will be älgtr: fn,1," stink )= P( ",...,,,,n Ln. O the final answer will be algee)
(5)
IP( -. t
for k = 1,... Tn and practice, from Ner generated reasoning paths, Eq. [ can be empirically computed as the ratio of correct reasoning paths starting from (fn,ee J among Ner. respectively. Hence, the label of each token Ln can be assigned with
1, No.. Let ",.,, b be } interchangeably. In
among Ner total reasoning paths starting from
[the final answer will be älger; ( =2) EN U1(ftm 1 He naw = 2)/ ENZ, l((tt,kk H4= - (tw Je 1)
where II-) is the indicator function and Nor cancels out in the right hand side. Finally, the token- supervised value model (TVM), frVM M is trained by minimizing the following loss using Eq. 6
Now Th
LTVM-
(P final answer will àlg ftr: fn,1," (,,,,) frVM
(,,,), (7)
where E is the mean squared error.
Thanks to this new token-level supervision with Eq. 5 the TVM is trained to directly and explicitly evaluate whether an intermediate reasoning path (i.e., (l ))==)) is on promising track toward the correct final answer, thereby producing more accurate value estimates than the ORM as shown in Table II Not only that, illustrated in Figure 3(a) the TVM can also achieve lower false negative error than the PRM with a reduction ranging from 8 to 14%p, while maintaining a comparable false positive error (within 1%p of the PRM). The overall procedure and algorithm of calculating Eq. 6 are outlined in Figure and detailed in Appendix B respectively.
Although Eq. can be computed without sampling multiple roll-outs train PRMs. Eq. 10.55 is gener- ally calculated by sampling multiple roll-outs given for and (ln, } like Wang et al. (2024b ch; Chen et al. (2024): Luo et al. (2024) do for training PRMs. is worth noticing that the multiple roll- out process per token requires O n=1 E/" Tn -k) = 0 Ner T2)) token generation, as T,, -k tokens need to be generated per roll-out for each and k. However, Eq. 6 can be easily calculated
-Ne-
once N₂r reasoning paths are sampled, requiring only O O Tn)) token generation.
Nor
7

---

Published as a conference paper at ICLR 2025
4.2 THEORETICAL INSIGHT: PROBABILITY SCORING AS VALUE MODELING
For a token in.k in an intermediate reasoning path 1. 1n -n = the expected cumu- lative reward (i.e., value) is written
((tm,k) =
(t(m,,++))) Itr: t
(8)
1=1
where r(.) and y denote a reward function and the discount factor, respectively. For conventional settings of reinforcement learning with LLMs (Wang et al. 2024a) ( 1 and no intermediate rewards), under the specific outcome reward formulation of Eq. | the expected cumulative reward
(i.e., value in Eq. g can reduce to the probability of reaching the correct final answer conditioned } =K==1 which can be straightforwardly
on the question for and intermediate reasoning path (ln,,* computed from generated reasoning paths (Section (.1).
Let the reward function r(tn,k) be defined as Eq. which includes only the out- come reward and no intermediate reward (i.e., r(tr,x) = O except the final answer). Then, with the fa ability of reaching the correct final answer conditioned on qtr and {Ln,1,"' ",n,,))= (1m,e ===
Proposition 4.1.
discount factor y L, the expected cumulative reward (i.e.. value in Eq. (8) is equivalent to the prob-
E (--1r(tn (tm,,ttt Atr, fn.1," 1 i kn,k] (the final answer will be à ftr: 1m,1, 1 bnkk)-
In light of Proposition 4. we coin our proposed verifier as the token-supervised value model (TVM), not reward model. Not only that, given that tree search is fundamentally intended to be guided by value rather than reward, Proposition 4. is important, as Proposition 4.1 guarantees that TVMs allow tree search algorithms to be value-guided.
EXPERIMENTS
To demonstrate the effectiveness of proposed verifier, the token-supervised value model (TVM), in solving math word problems, we conduct experiments on the GSM8K (Cobbe et al.] 2021 and
MATH (Hendrycks al. [202]) benchmarks. To perform tree search test time, we use two differ- ent tree search methods, Step-by-Step Beam Search and REward BA 3Alanced SEarch (REBASE) (Wul al. 2024), which generally outperform other strategies such as Monte Carlo Tree Search or its variants, as observed in Yu et al. (2024); (Cen et al.) (2024); Snell et al. (2024); Wu et al. (2024).
Unless otherwise specified, we set K K4 for Step-by-Step Beam Search and REBASE, and 2
for self-consistency (Wang et al. 2023) and Best-of-N search following WWng et al. (2024b). Re-
gardless of which search strategy to use, we choose the single solution ranked highest by verifier
as our final solution.
Our experiments are based on the following LLMs: (1) Mistral 7B (Jiang et al.) 2023). ) Llama 8B (AI@ Meta 2024) and (2) those fine-tuned on MetaMATH (Yu et al. (023). We opt for LLMs under 10B parameters which is a more interesting setting for experimental research considering the observation that using more test-time compute with smaller language models can surpass using less test-time compute with larger language models (Wu et al.) 2024; Snell et al., 2024). For all experiments, verifier is of the same size and architecture TVMs, following Cobbe et al. (2021), a verifier is extended with a scalar head composed of a single gain parameter and single bias parameter. For PRMs without human annotations, we employ Math- Shepherd (Wang et al., 2024b). As all experimental results in Wang et al. (20245) are only based on LLMs fine-tuned on MetaMATH, we compare TVM with Math Sheh-Shepherd only for Mistral 7B MetaMath and Llama 3 8B MetaMath. For convenience, we call Math- Shth-Sheppeerd PRM hereafter. More experimental details are deferred Appendix
the LLM. In the
of ORMs and
5.1 GRADE SCHOOL MATHEMATICS (GSM8K)
Table 2 displays the comparison of TVM with both ORM and PRM under self-consistency, Best- of-N search, Step-by-Step Beam Search, and REBASE on the GSM8K benchmark for Mistral 7B, Mistral 7B MetaMath, Llama 3 8B, and Llama 3 8B MetaMath. For Step-by-Step Beam Search
8

---

Published as a conference paper at ICLR 2025
Table 2: Accuracy of Mistral 7B, Mistral 7B MetaMath, Llama 3 8B, and Llama 3 8B MetaMath on the GSM8K benchmark under self-consistency (N 256), Best-of-N search (N = 256), Step- by-Step Beam Search (K = 40, b 10), and REBASE (K" 40). A verifier shares the same model size and architecture as the LLM. A bold number means the best accuracy under the same test-time search strategy. A boxed number and underlined number represent the best accuracy and the second best accuracy respectively, irrespective of the choice a test-time search strategy. Three random trials conducted to compute the mean accuracy and standard deviation of TVM.
Search Strategy
Method
7B
Mistral 7B Meta
Ilama
Ilama Meta
Self-Consistency
79.23
83.90
80.97
85.44
ORM
85.52
87.41
87.79
89.77
Best-of-N
PRM
88.55
90.30
Search
TVM (Ours)
88, 17
89 01.
88. 70
90.35 37
Step-by-Step
ORM
86.73 73
87.79
88.10
89 60
Beam Search
PRM
86.66
88.93
TVM (Ours)
87.69+00
88. $8.70-0 0.16
100.06-00.07
90.35++00
ORM
86.81
88.40
87.49
89130
REBASE
PRM
86.28
88.70
TVM (Ours)
87.97:00.16
189.21=0.11
88 0.:00.00099
89.84 $9.84:0..21
Table 3: Accuracy of Mistral 7B MetaMath, and Llama 8B MetaMath on the MATH bench- mark under self-connistency (N = 256), Best-of-N search (N = 256), Step-by-Step Beam Search (K = 40, b 10), and REBASE (K 40). A verifier shares the same model size and architec- ture as the LLM. A bold number means the best accuracy under the same test-time search strategy. A boxed number and underlined number represent the best accuracy and the second best accu- racy respectively, irrespective of the choice of test-time search strategy. Three random trials are conducted to compute the mean accuracy and standard deviation of TVM.
Search Strategy
Method
Mistral 7B MetaMath
Llama MetaMath
Self- Caff-Consistency
35.10
42.40
ORM
36.40
43.80
Best-of-N
PRM
37.30
44.40
Search
TVM (Ours)
37.40
43.40
ORM
36,80
42.40
Step-by-Step
PRM
36.80
42.20
Beam Search
TVM (Ours)
39. 39.333+0.10
45. 05..00+0..55 0.59
ORM
37.20
42.20
REBASE
PRM
37.60
41.80
TVM (Ours)
38.73.10.09
43.60.10.26
and REBASE, TVM consistently outperforms other baseline verifiers. Notably, TVM considerably outperforms PRM, which implies that using the PRM negatively affects the performance of tree search at inference time. Surprisingly, TVM also improves over ORMs an PRMs even for Best-of-N search. It is worth noticing that using Step-by- Step Beam Search REBASE with the TVM can perform closely or even surpass to the level of using Best-of-N search with TVM while spending about 6x less FLOPs and 2x less execution time as indicated in Table
5.2 ADVANCED MATHEMATICS (MATH)
Following Lightman et al. (202)): : Wang et al. (2024b), we also use 500 test MATH problems for evaluation. which the same test dataset of Lightman et al. (2023), incorporating the remaining 4500 test problems into the training dataset of MATH.
In Table 3 consistency, ofst-of-N search, Step-by-Step Beam Search, and REBASE on the MATH benchmark for Mistral 7B MetaMath and Llama 3 8B MetaMath. Although the PRM surpasses the TVM by 1.0%p under Best-of-N search for Llama 8B MetaMath, it is noteworthy that using Step-by-Step Beam Search with the TVM outperforms the best accuracy under Best-of-N search for both Mistral
the experimental results of TVM are compared with those of ORM and PRM under self-

---

Published as a conference paper at ICLR 2025
Table 4: FLOPs and execution time of sampling N,, reasoning paths to train TVMs on the GSM8K
(Ne, = 100) and MATH (N,, = 25) benchmarks for Mistral 7B MetaMath without and with vLLM.
GSMSK FLOPs
GSM8K Time
MATH FLOPs
MATH Time
Sampling w/o vL.LM
130.4 x 10
8.2 hours
204.1 X 10
20.3 hours
Sampling w/ vLLM
130.4 x 10"h
4.6 hours
204.1 x 10
5.7 hours
Table 5: FLOPs and execution time of Best-of-N search (N 256) without and with vLLM, Step-
by-Step Beam Search (K = 40, b = 10), and REBASE (K = 40) on the GSM8K and MATH
benchmarks for Mistral 7B MetaMath.
Search Strategy
GSM8K FL.OPs
GSM8K Time
MATH FLOPs
MATH Time
Best-of-N search w/o vLLMM
589.3 x 10
6.5 hours
871.0 x 10
22.0 hours
Best-of-N search w/ vLLM
589.3 10
2.1 hours
871.0 10
2.4 hours
Step-hy-Step Beam Search
94.3 x 10
0.9 hours
142.9 x 10
1.1 hours
REBASE
94.3 10
1.3 hours
142.9 10
2.6 hours
7B MetaMath and Llama 3 8B MetaMath even with approximately 6x less FLOPs and 2x less execution time seen in T
5.3 COMPUTE ANALYSIS
Training Compute Analysis. To illustrate the computational cost for training TVMs, we estimate floating point operations (FLOPs) and measure the execution time required to sample Nor reasoning paths. These measurements were conducted using 8x NVVIIDIA A100-80GB GPUs for Mistral 7B MetaMath on GSM8K (N = 100) and MATH (Ner = 25), as reported in Table Since sampling Ntr reasoning paths has a linear complexity as explained in Section 4.1 the sampling process takes at most less than a day even without L.M serving engines such as vLLM (Kwon et al. 2023). With
vLLM, the sampling process can be accelerated by at least factor of two.
Inference Compute Analysis. To compare the inference computation between Best-of-N search, Step-by-Step Beam Search, and REBASE, for Mistral 7B MetaMath, we estimate floating point op- erations (FLOPs) following Wu et al. (002)); Snell et al. (2024) and measure the execution time when using x SXNVIDIA A100-80GB GPUs on GSMSK and MATH in Table 5 Since 2=256 is much larger than 40 Best-of-N search consumes much more FLOPs than Step-by-Step Beam Search and REBASE. As Step-by-Steg Step Beam Search uses the same K=40 as REBASE, the esti- mated FLOPs are the same for both Step-by-Step Beam Search and REBASE. Nonetheless, Step- by-Step Beam Search spends less execution time than REBASE due to the fact that the steps are uniformly expanded in parallel with K/ children as delineated in Section
CONCLUSION
In this work, we reveal inherent limitations in existing neural verifiers for mathematical problem- solving, namely outcome-supervised reward models (ORMs) and process-supervised reward models (PRMs), when applied to tree search algorithms and PRMs were originally designed for Best-of-N search. Consequently, ORMs can only infer the correctness of partial solutions indirectly and implicitly, while PRMs suffer from high false negative errors, leading them to under-value value promising intermediate steps. To overcome these issues, we propose super-super value models (TVMs), a new class of verifiers trained with token-level supervision, where each token is assigned a probability reflecting the likelihood of reaching the correct final answer. This new token-level supervision approach enables TVMs to more accurately assess which partial solutions than PRMs while maintaining comparable false positive errors. As a result, TVMs significantly enhance the performance of tree search algorithms test time over both ORMs and PRMs.
inference time. This is because both ORMs
a right track than ORMs. and to attain lower false negative errors
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) N00.R2021-11211344 Arti- ficial Intelligence Graduate School Program (Seoul National University), and No. 2022-0-00984, Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation].
REFERENCES
Aeta Llama 3 model card. 2024. URL https::/ggthub..co http://hb...ccomoto/eteeaeaa 11ama/
www.://maaii////aaaa /amaann//MDLL.. MODELL CARD md
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V. Le, Christopher Ré, and
Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024.
Guoxin Chen, Minpeng I. Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process, 2024. URL ltttps://axxiii/ org/ abs/2405. 03553
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika lyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathicu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajiwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong. Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel. Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly. Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini. Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong. Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur,
11

---

Published as a conference paper at ICLR 2025
Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisemman, Azadch Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang. Guna Lakshminarayanan, Hamid Shojanazeri, Ham Zou. Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk. Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov. Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- Saan-Baptiste Gaya, Jeff Marcus, Jeff Tang. Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill. Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang. Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A. Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo. Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng. Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy. Raghu Nayani. Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha. Shiva Shankar, Shugiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe. Soumith Chintala. Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury. Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vítor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian. Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL ptcttss://axxie
abs/ /aoos//2447 2407.21772222
Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training, 2024.
12

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anen Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, lyya Sutskever, and Karl Cobbe. Let's verify step by step, 2023.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL httpss////peeeeieewee net./ www.netfforrmt ?
Bkg6RiCqY7
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024. htthhttps://arxxi wwwp://w.rrr/bb//0000000
06592
Leonardo de Moura and Sebastian Ullrich. The lean 4 theorem prover and programming language. In Automated Deduction CADE 28: 28th International Conference Automated Deduction, Virtual Event, July 12-15, 2021, Proceedings, pp. 625-635, Berlin, Heidelberg, 2021. Springer- Verlag. ISBN 978 300-3-000007777---- doi: 10. 1007/977 3- 03--7-37777777-7777 URL httttss:://dol
9rc/10.1007/998- 030 79876 37
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL htttps://arxiv.orr
abs/ /4tp/248...o 03314
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1999-1958 2014. URL http:::/jill..rrppappppeeeeee srivastaval4a. 1attaaaa11a...m11111 html
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. 2022.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Secrets of rlhf in large language models part ii: Reward modeling, 2024a. URL
wwwpiiioorrgaaaaa org. abs 2101... 0680)
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd, Verify and reinforce Iims step-by-step by-step without human annotations, 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https:// /ptp://veeeeeeieeweeee net/ www.ummid-PPLIIIIMMM....
Zihan Wang. Yunxuan Li. Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision, 2024c. URL https:
org/ abs /2002 02658
13

---

Published as a conference paper at ICLR 2025
Yangzhen Wu, Zhiqing Sun, Shanda Li. Sean Welleck, and Yiming Yang. An empirical analysis of compute-optimall inference for problem-solving with language models, 2024. URL https: /arxiv.orr//as//200... 00724
Fei Yu, Anningzhe Gao, and Benyou Wang. Ovm, outcome-supervised value models for planning in mathematical reasoning, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.122 12284, 2023.
14

---

Published as a conference paper at ICLR 2025
A PROOF OF PROPOSITION 4.1
Let the reward function r((n,)) be defined as Eq. which includes only the outcome reward and no intermediate reward (i.e., r(fn,k)
except the final answer). Then, with the discount factor 7 1, >i=1" ( = 2t=1 (((,,e+)) becomes either or zero, depending whether the resulting final answer will be à or not, respectively. As a result, the expected cumulative reward (value in Eq. () can be written
E
[(m,k41) Seer bndls bnnk (:7=1) = 1)
r r(m,,441) = r gers "n (kmnk (:
Σ 1=1
(lm,,411) = 0 or 1)
(ln,44)) 1/tees bn -. bn,
1m1
P(ee final answer will be a dlger 1n,1 .. kn,
because Ei=1 r(In, = only if the resulting final answer will be
15

---

Published as a conference paper at ICLR 2025
B ALGORITHM FOR TOKEN-LEVEL SUPERVISION WITH CORRECTNESS PROBABILITY SCORES
Algorithm Token-level Supervision with Correctness Probability Scores
For question Qt., Nr reasoning paths, each consisting of (t 17t, and a final answer Ons the ground truth answer à and the outcome reward function ro(a,) in Bq | for n = 1, No.. Ensure:
Require:
H dictO)
for 1,..,, , do for & = 1.... To do if not H. keyttann bukk then H. ..iiseet([t..... kn,k), (1)
else
tn,e] Hiinser(( (fn.s fn.b (c ro (a(a)) +
end if end for end for
for
= 1, ..., No, do
To do for k = Hettt-..et( t..k
// c mcans the number correct reasoning paths starting from " Ln
// f indicates the number reasoning paths starting from bn,1," bn
((t,,)) =
Eq
end for
end for
16

---

Published as a conference paper at ICLR 2025
C IMPLEMENTATION DETAILS
In Section 5.1 following Cobbe et al. 2021), an LLM fine-tuned on the training dataset of
GSM8K for two epochs with batch size of 128 and learning rate of 1e-5. Then, we sample Ner = 100 reasoning paths per training problem. In Section 5.2 an LLM fine-tuned on MetaMath generates Nrr 25 reasoning paths for each training problem. We generate Nrr reasoning paths with temperature of 0.7, top-k of 50, and a top-p of 1.0.
We employ the same architecture as Cobbe et (2021), a language model extended with a scalar head composed of single gain parameter and single bias parameter, to output a score for each token reasoning path. In addition, following Cobbe et al. (202T), we use both language mod- eling objective and the verification objective, with 20% dropout (Srivastava et al.) 2014). We use the AdamW optimizer (Loshchilov & Hutter 2019) with a linear scheduler to train a verifier. Note that in all experiments, verifier shares the same model size and architecture as the LLM used to generate the Nr reasoning paths.
Table 6: Learning rate and batch size for training the TVM (ours) when using Mistral 7B, Mistral 7B MetaMath, Llama 3 8B, and Llama 8B MetaMath to generate N1, 100 reasoning paths per training problem in GSM8K in Section 5.I
Mistral-7B
Mistral-7B-MetaMail
Llama3-8B
Llama3-DD-MeuaMath
Learning rate
2e-6
2c-6
le-5
2c-6
Batch size
512
512
512
512
Table Learning rate and batch size for training the TVM (ours) when using Mistral 7B MetaMath and Llama 3 8B MetaMath to generate Nrr = 25 reasoning paths per training problem in MATH in Section 5.2
Mistral 7B-MctaMalt
Llama3- 1-13--B-MMan 3B-MetaMath
Learning rate Batch size
2c-6
2e-6
512
512
For Best-of-N search, Step-by-Step Beam Search, and REBASE, we use temperature of 0.7, top-k 50, and top-p of 1.0. The maximum new token length is set to 400 for GSM8K and 1024 for MATH, respectively.
17

---

Published as a conference paper at ICLR 2025
D DISCUSSION: TOKEN-SUPERVISED SIGNAL IN TVMS FOR THE LATTER PART OF REASONING PATHS
If reasoning paths sampled with small Nr., the likelihood of reasoning paths overlapping after the first couple dozen (e.g.. 15-20) tokens could be low. In such a case, the token-supervision sig- nal for the latter part of each reasoning path would be labeled uniformly either 0, depending on whether the final answer is correct.
However, with large enough Ntr (e.g., Nor 100 for GSM8K as Cobbe et al. (202 we can obtain reasoning paths that overlap significantly beyond the first 13-20 tokens, up to the later steps of reasoning. As demonstrated Figure 2
reasoning paths can overlap up the middle of the fourth step which consists of the first 93 tokens in terms of token count. Below, we provide the complete version of the two overlapping reasoning paths that were illustrated Figure 2 Bold text indicates the overlapped part between two reasoning paths in Figure 2, which is 93 tokens in total.
Reasoning path 1 in Figure 2
Terry took bushels * 14 ears of corn per bushel 112 ears of corn.
Jerry took bushels * 14 ears of corn per bushel 42 ears of corn.
Linda took 12 bushels Stacy took 21
ears corn per bushel 168 ears of corn.
Alzogether, Bob's friends tock 112 42 168 21 343 of corn.
Thus, Bob is left with 50 bushels 14 ears of corn bushel - 343 ears of corn 357 ears of corn
Reasoning path 2 in Figure 2
Terry took bushels 14 ears of corn per bushel 112 ears of corn. Jerry took bushels
of per bushel
Linda took 12 bushels
of per bushel 168 of Stacy took 21 ears of corn 168 ears of cern = 189 ears of corn.
Bob had 50 bushels 14 ears of corn per bushel - 700 ears of corn to start with.
So, Bob has 700 ears corn f 112 ears of corn 42 ears of corn
corn 189
189 of remaining.
18

---

Published as a conference paper at ICLR 2025
Additionally, two complete reasoning paths for MATH also provided below. The overlapping section between these two reasoning paths, totaling 129 tokens, is highlighted in bold.
Reasoning path I on MATH
(x+1) 224 (x+1) (3-x) + (3-x) 22
$= (x"2+2x+1) +2 (3-x) (x+1) + (9- 6x+x 22) $
$= x'2+2x+1+2 (3x+3-x + (9- 6x+x (
$= 2x
2$
$= (x"2-2x"2+x"2" (2x-xx+xxxxx 62-- (1+6+91
0+0+165
b166s
Reasoning path 2 on MATH
(x+1) 2+2 (x+1) (3-x) + (3-x) "2$
$= (x*2+2x+1) +2 (3-x) (x+1) (9-6x)) 6x+x $
$= ( 6xxx 2)
$=
$= [143s
19

---

Published as a conference paper at ICLR 2025
E ABLATION STUDY
Table 8: Mean accuracy and standard deviation for Mistral 7B and Mistral 7B MetaMath on the GSM8K benchmark according varying sizes of K and b when utilizing Step-by-Step Beam Search with the TVM. Three random trials are carried out.
K,
FLOPs
Mistral 7B
Mistral 7B MetaMath
20, 5
47.1 10
86.05
10.37
88.12 =0.25
40, 10
94.3 x 10
87.69
#0.22
88.70 =0.16
80, 20
188.5 10
87.89
+0.35
88.75 =0.20
100,
235.6 10
87.92 +0.13
88.80 0.07
Beam size study. To investigate whether the accuracy of using the TVM improves with larger values of K and Step-by-Step Beam Search, we conduct experiments using the TVM with varying sizes of K and b for Mistral 7B and Mistral 7B MetaMath on the GSM8K benchmark. Table g shows that the accuracy of using the TVM on GSM8K increases as both K and b grow from 20 and 5 40 and 10. However, the accuracy of using the TVM remains relatively stable with the rise K and from 40 and 10 to 100 and 25, while the inference computation (i.e., FLOPs) increases by 2.5 times. Hence, 40 and 10 would be an appropriate choice for and considering the trade-off' between FLOPs and the improved performance.
20


Reviews:
[{"review_id": "XNBDOY5LZL", "reviewer": "Reviewer_S2DY", "summary": "This paper proposes token-supervised value models (TVMs) as a superior way to verify whether tree-search based math reasoning models are on the right track or not. The authors compare TVMs to output-supervised reward models (ORMs) and process-supervised reward models (PRMs). They argue that both of these existing verification methods, having been designed for evaluation of full trajectories, are not suitable for verifying partial trajectories, especially at the token level. The conduct experiments with two math datasets, showing that the use of TVMs improves performance. Additional experiments are also provided.", "questions": "Please see the weaknesses section above. I am happy to consider increasing my score if I can get more clarity.\n\n====\nREVIEW RATING UPDATED AFTER AUTHOR RESPONSE\n====", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The authors make a clear argument about what current verifiers (ORMs and PRMs) lack, and why TVMs should be able to help. Limitations of ORMs and PRMs are justified with some empirical support. Logically thinking, the progression from ORMs to PRMs to TVMs seems natural.\n\n* Figures and examples (although small and hard to read) are useful.\n\n* The method is clearly described, along with a formulation of prior (ORM and PRM) methods.\n\n* Experiments show a notable improvement when using TVMs, especially for step-level beam search and for the MATH benchmark.\n\n* Additional experiments around FLOPs used and execution time support the idea that tree-based search methods are more efficient, especially if one can pair them with effective verifiers.", "weaknesses": "* One obvious disadvantage of TVM compared to PRM is that during *training*, the same N_{tr} sampled reasoning paths per training problem are expected to provide a lot more supervision in TVM. Specifically, suppose a typical reasoning path has 5 steps and 50 tokens. Then, in PRMs, it is easy to imagine several reasoning paths sharing similar partial paths  steps, leading to a good signal for supervising the 5 steps. However, with ~50 token long paths, the chances of partial sampled paths overlapping would decreases quickly and would be very slim after the first 15-20 tokens. So I'm not sure what kind of token-supervision signal can one actually get for the latter parts of the reasoning paths -- unless paths are sampled differently or N_{tr} is higher. The paper doesn't seem to address / discuss this.\n\n* The gains, when using TVM, are rather slim in step-level beam search for the GSM8k dataset, only about 1%. This is not very different from the gains when using best-of-N search. If fact, for Mistral, the gains from using TVM are slightly higher for best-of-N compared to step-level beam search. This goes against the motivational argument presented earlier, namely that ORMs and PRMs are suitable for best-of-N search but not tree search, and it's the latter than needs another verification method.\n\n* In general, TVM doesn't seem to help that much on GSM8k.\n\n* I don't understand what Proposition 4.1 is trying to say. Why is it not \"trivially true\" if the reward function is 0 at intermediate tokens and 1 at the output token iff is it the correct output? Is this proposition related to TVMs or also equally applicable to ORMs and PRMs (which seems to be the case)? I am not following the significance of this formal proposition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ri7jOMzG2x", "reviewer": "Reviewer_6Dfx", "summary": "The paper proposes Token-Supervised Value Models, to enhance the mathematical problem-solving capabilities of LLMs. Traditional methods lack effectiveness when applied to tree search algorithms, leading to premature pruning of promising steps. TVMs address this by assigning a probability to each token that reflects its likelihood of contributing to a correct answer, enabling direct evaluation of intermediate solutions. Experiments show that TVMs outperform ORMs and PRMs in accuracy, especially in tree-search-based strategies, by reducing false negatives and improving the assessment of solution paths​.", "questions": "1. How is $N_{tr}$ determined, given its direct impact on probability estimation? Is it chosen through parameter tuning, or is there additional insight guiding this choice?\n\n2. A follow-up question: During fine-tuning, do these $N_{tr}$ trials often share some prefix tokens, and how to control them? For example, in Figure 4, the three paths share the first $a-1$ tokens, allowing those tokens to have the value 0.33. If they don’t share any tokens, it seems the problem would reduce to a PVM.\n\n\n3. How to interpreted Figure 3(b)? The \"Verifier’s Scores Histogram for Correct Sampled Solutions\" shows that the weights are more concentrated for correct solutions, but what about the histogram on the right side?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. TVM is a novel approach. It assesses each token's contribution to the likelihood of reaching the correct answer, enabling more precise evaluations of intermediate solutions during tree search.\n\n2. The authors address that TVMs lower false negative rates by distinguishing between promising and incorrect intermediate steps, enhancing recall without sacrificing precision.\n\n3. TVMs show consistent improvements in accuracy across various benchmarks, including GSM8K and MATH, indicating their robustness and versatility.", "weaknesses": "1. TVMs require token-level supervision, which could add complexity to the training and fine-tuning process compared to traditional verifiers.\n \n2. The effectiveness of TVMs relies on reasonably accurate token-level probability estimation based on samples of successful and failed trials. For difficult problems, the sampling process controlled by the LLM can be challenging, as most samples may be incorrect, potentially reducing the effectiveness of TVMs to that of a PRM.\n\n3. Some sections need clarification; please see the questions below.\n\n*Typo: In several places, there is an extra \"p\" added after \"%\", e.g., \"14%p\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eoWl9r3agJ", "reviewer": "Reviewer_E5Q5", "summary": "This paper proposes token value models (TVMs), a new method for training verifiers used at test time to improve the problem-solving capabilities of LLMs. Existing training methods, outcome-supervised reward models (ORMs), and process-supervised reward models (PRMs) have drawbacks in that they can give only indirect supervision for intermediate steps. TVMs enable giving direct supervision over tokens by exploiting the probability that the token will read to a correct answer.  Experimental results show that TVM shows better predictive performance than ORMs and PRMs.", "questions": "1. (Figure 1) I need help understanding why ORM assigns low scores to correct steps and PRM assigns high scores to wrong intermediate steps based on this example. More convincing examples would help readers understand the superiority of the TVM.\n\n2. I cannot understand the explanation of TVM's complexity (line 377). TVM is said to be a tree-search-based method (line 132), but the explanation here says that TVM computes eq.(6) from a set of fixed reasoning paths and does not need a tree search. I'm happy if the authors address this inconsistency. \n\n3. Moreover, I think the roll-out process (line 374) is also beneficial for the TVM since we can estimate conditional probabilities eq.(6) more precisely for a token if we have more sample runs. Why does TVM avoid this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is simple and easy to understand how it works.\n2. Experiments are conducted with combinations of multiple benchmarks, LLMs, and search strategies. The experimental results support the proposed method's superiority.", "weaknesses": "Although the experimental results are interesting, the paper's presentation has some problems and thus does not explain the reasons why the TVM works well. Therefore, I think this paper needs a further revision.\n  1. The paper says that TVM is superior to ORM and PRM since it uses token-level direct supervision (line 113). I'm not sure why we can say TVM is direct since ORMs, PRMs (without human annotations), and TVMs all rely on supervision signals for outcomes. No direct supervision on intermediate steps is given. Therefore, all these methods share the same problem described as the disadvantages of the ORMs (line 267).\n  2. Moreover, the supervision of TVM (eq. (5)) based on conditional probability looks pretty similar to the reward for ORMs (eq. (3)). If we have $N_{tr}$ samples, the reward for a token $t_{n, k}$ in ORMs will be close to the success rate of the paths containing $t_{n, k}$. Therefore, the proposed supervision seems less novel.\n  3. The paper says that one of its main contributions is that it is the first to disclose that PRMs produce a high false negative error. However, this is not true, as the explanation (line 306, Figure 2) assumes that PRMs do not use human annotation. The false-negative problem would not occur if we had human annotation on intermediate steps. Therefore, the statement is incorrect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MsGkvQa4EA", "reviewer": "Reviewer_fQ6h", "summary": "The paper presents TVM -- token-supervised value models that can act as neural verifiers and provided tree-based LLM reasoning methods with a way to rank different parts of the search tree as more/less promising on reaching a solution in the domain of math problems. The authors state that existing SOTA like Outcome-based (ORMs) and Process-based (PRMs)  supervision models cannot rank paths effectively due to the way they are trained. ORMs only provide a reward if the answer is correct whereas PRMs provide the same reward to all steps. This causes issues in recall with PRMs since an entire step is marked as a failure even if only a few tokens are wrong.\n\nTo mitigate this drawback of PRMs, the authors introduce TVMs that utilizes different generated reasoning paths in the training set to provide different \"weights\" or rewards. For any substring of a correct solution, each token of that substring is allocated a weight based on the total substrings (or reasoning paths) that were generated. This can be thought of as a value function and the authors provide some soft theoretical analysis for it. \n\nThe authors then conduct an empirical evaluation that showcases the strengths of their approach.", "questions": "Overall I liked this paper and I think it will be a good fit in the program. I hope the authors are able to resolve my concerns.\nPlease respond to my comments on the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1. The paper is very well-written and organized although it seems a bit verbose with some aspects not really required.\n\nS2. The presented idea is very convincing and intuitive. The idea is novel and weighing up of tokens (or a partial order sequence) has been successfully explored as an idea in many different domains.\n\nS3. The baselines are good and the empirical setup overall is good.", "weaknesses": "W1. The paper was a bit verbose even if it was well-written. There are several aspects that could have been cut. For example, I do not understand why  Tables 4, 5 were included since it does not appear to be a major contribution of the paper. Table 3 already showcases all methods with different tree-search paradigms so I am not sure why this was necessary. Please comment. An analysis of training effort would have been better here.\n\nW2. The empirical evaluation is a bit lacking. The gains seem marginal in most cases and std. deviations are missing from these plots. Please provide them so that a better assessment can be made.  Also, it seems that this approach can be more general. I am not sure why the authors have limited to only math tasks. A more general evaluation would have improved the paper's impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)