Token-Supervised Value Models for Enhancing Mathematical Problem-Solving Capabilities of Large Language Models

## Abstract

Abstract Large Language Models (LLMs) have demonstrated impressive problem-solving capabilities in mathematics through step-by-step reasoning chains.
However, they are susceptible to reasoning errors that impact the quality of subsequent reasoning chains and the final answer due to language models’ autoregressive token-by-token generating nature.
Recent works have proposed adopting external verifiers to guide the generation of reasoning paths, but existing works utilize models that have been trained with step-by-step labels to assess the correctness of token-by-token reasoning chains.
Consequently, they struggle to recognize discriminative details of tokens within a reasoning path and lack the ability to evaluate whether an intermediate reasoning path is on a promising track toward the correct final answer.
To amend the lack of sound and token-grained math-verification signals, we devise a novel training scheme for verifiers that apply token-level supervision with the expected cumulative reward (i.e., value).
Furthermore, we propose a practical formulation of the cumulative reward by reducing it to finding the probability of future correctness of the final answer and thereby enabling the empirical estimation of the value.
Experimental results on mathematical reasoning benchmarks show that Token-Supervised Value Model (TVM) can outperform step-by-step verifiers on GSM8K and MATH with Mistral and Llama.

## 1 Introduction

Figure: Figure 1: Illustrative comparison of token-level supervision (TVM; ours) with outcome supervision (ORM) and process supervision (PRM). We provide two examples for each correct and wrong reasoning path. Inthe reasoning step 4 of each example, both ORM and PRM use uniform labels judged by the correctness of either an entire reasoning path or step, which poses challenges for recognizing discriminative details of tokens within a reasoning path. On the other hand, TVM is supervised with distinct per-token labels, thus enabling the distinction of the details of tokens within a reasoning path and leading to more precise outcomes (see Fig. 2).
Refer to caption: x1.png

Large language models (LLMs) pre-trained on massive data have achieved human-level performance across a wide range of tasks in natural language processing (Maslej et al., ). A notable exception to this trend is complex multi-step reasoning tasks such as mathematical problem solving, where current state-of-the-art LLMs still struggle to attain near-human performance.
Previous studies have been focused on enhancing the reasoning capabilities of LLMs through:
encouraging LLMs to generate step-by-step thought processes via few-shot or zero-shot prompting (Wei et al., ; Kojima et al., ); fine-tuning LLMs with question-solution pairs to generate intermediate reasoning steps before producing a final answer (Cobbe et al., ; Luo et al., ; Yu et al., ; Yuan et al., ); and employing aggregation techniques such as majority voting over final answers extracted from solutions generated by LLMs (Wang et al., ).

However, when LLMs are left to their own devices to solve given problems, they remain error-prone due to their autoregressive nature in generating reasoning paths. If an LLM, by chance, produces a single error during generation, the reasoning path can be easily steered towards a wrong answer. This would worsen for LLMs when they face more complex reasoning tasks such as advanced-level mathematical problems in the MATH dataset (Hendrycks et al., ).
To address this, researchers have focused on providing external aid to the LLM by training verifiers to assess the correctness of generated reasoning paths.

Existing verifiers can be categorized into two types: outcome-supervised reward models (ORMs) and process-supervised reward models (PRMs). ORMs (Cobbe et al., ; Uesato et al., ; Yu et al., ) are trained to assess the correctness of a reasoning path by labeling each token as either correct or incorrect solely based on whether the final answer in the reasoning path is correct.
PRMs (Lightman et al., ; Uesato et al., ; Wang et al., ) are trained with step-level labels to assess the correctness of each reasoning step, and they are generally preferred over ORMs due to the finer resolution of assessment in practice. Despite being proposed to assist LLMs, current verifiers may retain a fundamental misalignment with their per-token granularity.
Since ORMs and PRMs employ uniform labels according to the correctness of either a whole reasoning path or step, respectively (Fig. 1), we argue that they were not designed to (i) learn the discriminative details of tokens within a reasoning path or (ii) evaluate whether an intermediate reasoning path is on a promising track toward the correct final answer.

In this paper, we propose the Token-supervised Value Model (TVM), a novel verifier that supervises each token in a reasoning path with a distinctive label, training each token with the expected cumulative reward. Unlike ORMs and PRMs, our token-level supervision with distinct per-token value labels along a reasoning path (Fig. 1) equips TVMs with the ability to capture the discriminative details of tokens within a reasoning path (see Fig. 2).
Furthermore, providing a theoretical insight that the value of each token is equivalent to the probability of reaching the correct final answer from that token, we propose to label each token via empirical value estimation along sampled reasoning paths.TVM is trained to predict the probability of a per-token intermediate reasoning path being on a promising track toward the correct final answer. Therefore
TVM could choose among candidate reasoning paths most likely to reach the correct final answer, whether they are partial or complete.
Our contributions are threefold:

- •
We propose the Token-supervised Value Model (TVM), a new verifier capable of capturing token-wise details via direct supervision with the expected cumulative reward (i.e., value) for each token along a reasoning path.
- •
We generate per-token labels for verifier supervision via empirical value estimation, which allows TVM to predict the probability of an intermediate reasoning path reaching the correct final answer.
- •
We show that TVM achieves performance improvements on GSM8K and MATH benchmarks across LLMs under 10B parameters, compared to ORMs and PRMs.

## 2 Background

This section reviews existing verifier frameworks for enhancing the mathematical reasoning capabilities of LLMs. Sec. 2.1 outlines the preliminary setups for training verifiers in mathematical reasoning verification. The subsequent sections revisit two existing types of supervision for verifier training: outcome supervision (Sec. 2.2) and process supervision (Sec. 2.3).

### 2.1 Training Verifiers for Mathematical Reasoning

The mathematical reasoning capabilities of LLMs can be enhanced by employing reward models as external verifiers to assess the generated reasoning paths (Cobbe et al., ; Uesato et al., ; Lightman et al., ; Yu et al., ; Wang et al., ). The verifier is generally trained via supervised learning on a dataset obtained by sampling multiple reasoning paths per training problem using an LLM. Specifically, given a training problem $q_{tr}$ as an input, the LLM first generates $N_{tr}$ reasoning paths, where $n$-th reasoning path is comprised of reasoning steps $\{s_{n,j}\}_{j=1}^{S_{n}}$ and a final answer $a_{n}$ for $n=1,\cdots,N_{tr}$. In token-level notation, the $n$-th reasoning path can also be expressed as a sequence of tokens $\{t_{n,k}\}^{T_{n}}_{k=1}$. Hereafter, $\{s_{n,\cdot}\}_{1}^{j}$ and $\{t_{n,\cdot}\}_{1}^{k}$ means $\{s_{n,1},\cdots,s_{n,j}\}$ and $\{t_{n,1},\cdots,t_{n,k}\}$, respectively.
The final answer $a_{n}$ is correct if it is equal to the ground truth answer $\hat{a}$, and incorrect otherwise. Based on the correctness of the sampled reasoning paths, supervision is traditionally given in two ways: (i) outcome supervision (Cobbe et al., ; Uesato et al., ; Yu et al., ) and (ii) process supervision (Uesato et al., ; Lightman et al., ; Wang et al., ).

### 2.2 Outcome Supervision

Prior works (Cobbe et al., ; Uesato et al., ; Yu et al., ) employ outcome supervision to label an entire reasoning path as correct if its final answer is correct (Fig. 1). The outcome reward function $r_{o}(\cdot)$ is the correctness of the final answer:

$$ $\displaystyle r_{o}(a_{n})=\left\{\begin{array}[]{l}1\text{ if }a_{n}=\hat{a} \\ 0\text{ if }a_{n}\neq\hat{a}\end{array}\right.$ (3) $$

for $n=1,\cdots,N_{tr}$. An outcome-supervised reward model (ORM) $f_{ORM}$ is trained with every token in a reasoning path labeled as the outcome reward (Eq. 3). The ORM loss $\mathcal{L}_{ORM}$ is defined as

$$ $\displaystyle\mathcal{L}_{ORM}{=}\sum_{n,k}^{N_{tr},T_{n}}\leavevmode \resizebox{260.17464pt}{}{$\ell\left(r_{o}(a_{n}),f_{ORM}(q_{tr},\{t_{n,\cdot} \}_{1}^{k})\right)$.}$ (4) $$

The mean squared error is typically used as a loss function $\ell(\cdot)$ in Eq. 4. Cobbe et al. () demonstrated that a token-level verifier trained to judge the correctness after every token performs better than a solution-level verifier trained to determine the correctness only after the final token.

Interestingly, Yu et al. () showed that ORMs can be alternatively described as modeling the cumulative reward for each token, where all intermediate rewards are zero (i.e., $r(t_{n,k})=0$ for every $n$ and $k$) and the discount factor $\gamma$ is set to 1. The cumulative reward following an intermediate token $t_{n,k}$, $R(t_{n,k})$ is calculated as

$$ $\displaystyle R(t_{n,k})=r(t_{n,k+1})+\cdots+r(t_{n,T_{n}})+r_{o}(a_{n})$ $\displaystyle=\begin{cases}0+\cdots+0+1=1&\text{if }a_{n}=\hat{a}\\ 0+\cdots+0+0=0&\text{if }a_{n}\neq\hat{a},\end{cases}$ (5) $$

which is equivalent to $r_{o}(a_{n})$ in Eq. 3. This entails that an intermediate reasoning path is labeled as correct if the final answer is correct, and vice versa. In this sense, ORMs can indirectly and implicitly learn the potential correctness of an intermediate reasoning path (Yu et al., ).

### 2.3 Process Supervision

Process supervision enables a more accurate assessment of a reasoning path by explicitly training a verifier on the correctness of each step with step-level supervision Lightman et al. (). The correctness of each reasoning step is either labeled via human annotation (Uesato et al., ; Lightman et al., ) or automation (Wang et al., ). Since acquiring human annotations is labor-intensive and costly, we mainly focus on process supervision without human annotations.

Following Wang et al. (), an intermediate reasoning step $s_{n,j}$ can be labeled as correct if at least one of the reasoning paths starting from $s_{n,j}$ reaches the correct final answer $\hat{a}$ (Fig. 1). In practice, $s_{n,j}$ is annotated by sampling a fixed number of reasoning paths conditioned on a sequence of intermediate reasoning steps $\{s_{n,\cdot}\}_{1}^{j}=\{s_{n,1},\cdots,s_{n,j}\}$. If at least one of the sampled reasoning paths reaches the correct final answer, $s_{n,j}$ is labeled as correct with the process reward $r_{p}(s_{n,j})=1$. Otherwise, $s_{n,j}$ is labeled as incorrect and $r_{p}(s_{n,j})=0$. Using the per-step labels obtained through automation, a Process-supervised Reward Model (PRM) is trained to provide a step-level assessment by minimizing the following loss:

$$ $\displaystyle\mathcal{L}_{PRM}{=}\sum_{n,j}^{N_{tr},S_{n}}\leavevmode \resizebox{260.17464pt}{}{$\ell\left(r_{p}(s_{n,j}),f_{PRM}(q_{tr},\{s_{n, \cdot}\}_{1}^{j})\right)$},$ (6) $$

where $\ell$ denotes the binary cross entropy loss.

Figure: Figure 2: Illustration of reasoning paths ranked highest by ORM/PRM/TVM (ours) among $\mathbf{256}$ candidate reasoning paths for a test problem from GSM8K. We use Mistral-7B-MetaMath and illustrate practical failure cases of ORM and PRM compared to ours. (a) The reasoning step 4 begins with ‘‘So the total difference’’ but ends with a summation. Hence, as soon as step 4 is finished, the TVM score decreases dramatically. (b) The reasoning step 5 starts with ‘‘Thus, the total difference’’ but ends in subtracting a large number (‘‘120’’) from a small number (‘‘95’’), which is the exact opposite of the definition of difference. Thus, the TVM score declines. (c) The reasoning step 3 opens with ‘‘The total difference’’, ending in subtracting the small number (‘‘95’’) from the large number (‘‘120’’), which is finally correct. As a result, immediately after the token ‘‘=’’ emerges, the TVM score rises while the PRM score remains intact due to its step-wise assessment. Therefore, TVM can filter out (a) and (b) while selecting (c) with the highest score, enabling token-level discrimination within a reasoning path.
Refer to caption: x2.png

## 3 Method

In this section, we introduce our proposed method coined Token-supervised Model (TVM), a novel verifier trained with a token-level supervision strategy to directly estimate the expected cumulative reward (i.e., value) for each token along a reasoning path. We also describe how to empirically estimate per-token value labels from $N_{tr}$ generated reasoning paths for token-level supervision.

### 3.1 Motivation

As mentioned in Sec. 2, both outcome supervision (ORMs) and process supervision (PRMs) utilize homogeneous labels determined by the correctness of either the entire reasoning path or step (Fig. 1). Consequently, we hypothesize that they are likely to be neither explicitly nor directly trained to (i) learn the discriminative details of tokens within a reasoning path or (ii) evaluate whether an intermediate reasoning path is on a promising track toward the correct final answer.

We elucidate our assertion through cases observed in practice, as illustrated in Fig. 2. In the reasoning path ranked highest by ORM ((a) in Fig. 2), reasoning step 4 begins with ‘‘So the total difference’’ but ends with a summation, where a logical error occurs. However, ORM is unable to catch the error and maintains a score over $0.4$, the highest score among $256$ candidate reasoning paths. In the reasoning path ranked highest by PRM ((b) in Fig. 2), reasoning step 5 starts with ‘‘Thus, the total difference’’ but ends in subtracting a larger number (‘‘120’’) from a smaller number (‘‘95’’), which is the exact opposite of the definition of difference.
In the reasoning path, ‘‘120’’ appears in reasoning step 4 after ‘‘95’’ appears in reasoning step 3.
Since PRMs focus on assessing the correctness of the current reasoning step, the sequential appearance of numbers and the resulting subtraction are considered correct by the PRM even though the reasoning path is unlikely to lead to a correct answer.
The observed failures inspire the proposal of a token-level value supervision strategy for training verifiers.

Figure: Figure 3: Illustration of empirical value estimation using Eq. 11. For a single training problem, $N_{tr}$ reasoning-answer pairs are sampled using an LLM. Here, let $N_{tr}=3$ for convenience. (1) All three sentences begin with the same tokens $\{t_{1,k}\}_{k=1}^{a-1}$, and only one of them reaches the correct final answer ($357$). Accordingly, every token of $\{t_{1,k}\}_{k=1}^{a-1}$ is labeled as $1/3=0.33$. (2) At the $a$-th position, however, only one sentence starts with $t_{1,a}$, which reaches an incorrect final answer ($656$). Thus, all tokens after $t_{1,a}$ are labeled as $0/1=0$. (3) The remaining two sentences continue with the same tokens $\{t_{2,k}\}_{k=a}^{b-1}$, only one of which is correct. Hence, every token of $\{t_{2,k}\}_{k=a}^{b-1}$ is labeled as $1/2=0.5$. (4) Finally, at the $b$-th position, which one is correct is pre-determined. As a result, all tokens after $t_{2,b}$ are labeled as $0/1=0$, whereas all tokens after $t_{3,b}$ as $1/1=1$.
Refer to caption: x3.png

### 3.2 Token-level Value Supervision

To overcome the issues above, we propose a new verifier based on token-level supervision with distinctive token-wise labels according to the *potential* of tokens in deducing the correct final answer.
A natural choice to appropriately reflect the token-wise potential is prospective *value* modeling (Sutton and Barto, ), which is fine-grained and future-oriented compared to retrospective cumulative reward modeling (Eq. 5).
Accordingly, we construct a supervision scheme for token $t_{n,k}$ in a reasoning path $\{t_{n,\cdot}\}_{1}^{k}=\{t_{n,1},\cdots,t_{n,k}\}$ with the *expected* cumulative reward (i.e., value):

$$ $\displaystyle V(t_{n,k})=\mathbb{E}\big{[}\sum_{l=1}^{\infty}\leavevmode \resizebox{195.12767pt}{}{$\gamma^{l-1}r(t_{n,k+l})\big{|}q_{tr},\{t_{n,\cdot} \}_{1}^{k}$}\big{]},$ (7) $$

where $r(\cdot)$ and $\gamma$ denote a reward function and the discount factor, respectively.

The primary challenge in training value models as verifiers is estimating the value labels of a generated reasoning path Yu et al. ().
However, under the specific outcome reward formulation of Eq. 3 and no intermediate rewards, the expected cumulative reward (Eq. 7) reduces to the probability of reaching the correct final answer conditioned on the question $q_{tr}$ and intermediate reasoning path $\{t_{n,\cdot}\}_{1}^{k}$, which can be straightforwardly computed from generated reasoning paths and can indicate whether an intermediate reasoning path (i.e., $\{t_{n,\cdot}\}_{1}^{k}$) is on a promising track toward the correct final answer.

###### Proposition 3.1 .

Let the reward function $r(t_{n,k})$ be defined as Eq. 3, which includes only the outcome reward with the discount factor $\gamma=1$ and no intermediate reward (i.e., $r(t_{n,k})=0$ except the final answer). Then, the expected cumulative reward (Eq. 7) is equivalent to the probability of reaching the correct final answer conditioned on $q_{tr}$ and $\{t_{n,\cdot}\}_{1}^{k}=\{t_{n,1},\cdots,t_{n,k}\}$:

$$ $\displaystyle\mathbb{E}\big{[}\sum_{l=1}^{\infty}\gamma^{l-1}r(t_{n,k+l})\big{ |}q_{tr},\{t_{n,\cdot}\}_{1}^{k}\big{]}$ (8) $\displaystyle=\mathbb{P}(\textit{the final answer will be }\hat{a}|q_{tr},\{t_ {n,\cdot}\}_{1}^{k}).$ $$

The right-hand side of Eq. 8 can be empirically estimated from generated reasoning paths by calculating the proportion of correct reasoning paths starting from $\{t_{n,\cdot}\}_{1}^{k}$ among total reasoning paths starting from $\{t_{n,\cdot}\}_{1}^{k}$ (see Sec. 3.3).

Following Proposition 3.1, we train the *Token-supervised Value Model (TVM)* by supervising each token with a value label empirically estimated as the probability of reaching the correct final answer given until that token. The objective of TVM is

$$ $\displaystyle\mathcal{L}_{TVM}=\sum_{n,k}\leavevmode\resizebox{238.49231pt}{}{ $\ell\left(\mathbb{P}_{n,k},f_{TVM}(q_{tr},\{t_{n,\cdot}\}_{1}^{k})\right)$}$ (9) $$

for $n=1,\cdots,N_{tr}$ and $k=1,\cdots,T_{n}$, where $\mathbb{P}_{n,k}$ indicates the right-hand side of Eq. 8 and the loss function $\ell$ is the mean squared error.

**Table 1: Accuracy of Mistral-7B, Mistral-7B-MetaMath, Llama3-8B, and Llama3-8B-MetaMath on the GSM8K benchmark under best-of-N search ($N=256$) and verifier-guided step-level beam search ($K=40$, $b=10$). "BS" stands for beam search.**
| Search Strategy | Method | Mistral-7B | Mistral-7B-MetaMath | Llama3-8B | Llama3-8B-MetaMath |
| --- | --- | --- | --- | --- | --- |
|  | Self-Consistency | $79.23$ | $83.90$ | $80.97$ | $85.44$ |
| \hdashline<br>Best-of-N<br>Search | ORM | $85.52$ | $86.20$ | $87.79$ | $89.77$ |
| Math-Shepherd | - | $87.10$ | - | $89.23$ |  |
| TVM (Ours) | $\mathbf{88.17}$ | $\mathbf{88.86}$ | $\mathbf{88.70}$ | $\mathbf{90.37}$ |  |
| \hdashlineVerifier-guided | OVM | $86.73$ | $87.79$ | $88.10$ | $89.69$ |
| Step-level BS | TVM (Ours) | $\mathbf{87.72}$ | $\mathbf{88.78}$ | $\mathbf{89.01}$ | $\mathbf{90.30}$ |

Compared to existing verifiers, the resolution of assessment provided by the proposed token-level value supervision adequately matches the token-wise granularity of LLMs, thereby being able to capture the discriminative details of tokens within a reasoning path (Fig. 2).
In contrast to ORMs, TVM is trained to directly estimate the probability of an intermediate reasoning path being on a promising track toward the correct final answer (Proposition 3.1). As a result, TVM can choose the reasoning path most likely to reach the correct final answer among candidate reasoning paths, whether they are partial or complete.

During inference, TVM can be employed to either search the reasoning path most likely to be correct over complete reasoning paths generated from an LLM (Lightman et al., ) or distinguish prospective candidates likely to reach the correct final answer among partially generated reasoning paths.
For the latter, we conduct a detailed study in Sec. 4.3 in the setting of verifier-guided step-wise beam search (Yu et al., ).

###### Proposition 3.1 .

### 3.3 Empirical Value Estimation

As discussed in Sec. 3.2, Proposition 3.1 alleviates the practical challenges of value estimation (Eq. 7) by formulating the value as the ratio of correct reasoning paths to total reasoning paths.
Following Eq. 7 and Eq. 8, the estimated value for each token $t_{n,k}$ can be represented as

$$ $\displaystyle V(t_{n,k})=\mathbb{E}\big{[}\sum_{l=1}^{\infty}\gamma^{l-1}r(t_{ n,k+l})\big{|}q_{tr},\{t_{n,\cdot}\}_{1}^{k}\big{]}$ $\displaystyle=\mathbb{P}(\textit{the final answer will be }\hat{a}|q_{tr},\{t_ {n,\cdot}\}_{1}^{k})$ $\displaystyle=\frac{\mathbb{P}(\{t_{n,\cdot}\}_{1}^{k}\cap\textit{the final answer will be }\hat{a}|q_{tr})}{\mathbb{P}(\{t_{n,\cdot}\}_{1}^{k}|q_{tr})}.$ (10) $$

In practice, Eq. 10 can be empirically estimated from $N_{tr}$ generated reasoning paths as the ratio of correct reasoning paths starting from $\{t_{n,\cdot}\}_{1}^{k}$ among $N_{tr}$ and total reasoning paths starting from $\{t_{n,\cdot}\}_{1}^{k}$ among $N_{tr}$, respectively. The value label of each token $V(t_{n,k})$ is assigned as

$$ $\displaystyle\frac{\sum_{n^{\prime}=1}^{N_{tr}}\mathbb{I}(\{t_{n^{\prime}, \cdot}\}_{1}^{k}=\{t_{n,\cdot}\}_{1}^{k}\cap a_{n^{\prime}}=\hat{a})/N_{tr}}{ \sum_{n^{\prime}=1}^{N_{tr}}\mathbb{I}(\{t_{n^{\prime},\cdot}\}_{1}^{k}=\{t_{n ,\cdot}\}_{1}^{k})/N_{tr}}$ (11) $$

where $\mathbb{I}(\cdot)$ is the indicator function and $N_{tr}$ cancels out. The overall procedure of empirical value estimation is described in Figure 3. The overall algorithm is deferred to Appendix C.

## 4 Experiments

To demonstrate the efficacy of TVM in improving the mathematical reasoning capabilities of LLMs, we conduct extensive experiments on the GSM8K (Cobbe et al., ) and MATH (Hendrycks et al., ) benchmarks. Our experiments are based on the following large language models: 1) Mistral-7B (Jiang et al., ), Llama3-8B (AI@Meta, ); 2) those fine-tuned on MetaMATH (Yu et al., ) We use two existing verifier utilization strategies: (i) best-of-N search and (ii) step-by-step beam search.

**Table 2: Accuracy of Mistral-7B-MetaMath, and Llama3-8B-MetaMath on the MATH benchmark under best-of-N search ($N=256$) and verifier-guided step-level beam search ($K=40$, $b=10$). "BS" stands for beam search.**
| Search Strategy | Method | Mistral-7B-MetaMath | Llama3-8B-MetaMath |
| --- | --- | --- | --- |
|  | Self-Consistency | $35.10$ | $42.40$ |
| \hdashline<br>Best-of-N<br>Search | ORM | $36.40$ | $\mathbf{43.60}$ |
| Math-Shepherd | $37.30$ | $43.40$ |  |
| TVM (Ours) | $\mathbf{37.40}$ | $43.40$ |  |
| \hdashlineVerifier-guided | OVM | $36.60$ | $42.40$ |
| Step-level BS | TVM (Ours) | $\mathbf{39.20}$ | $\mathbf{45.20}$ |

#### Best-of-N search.

The best-of-N search strategy introduced in Lightman et al. () is a conventional experimental setting to evaluate the performance of a verifier. For every test problem, an LLM first generates $N$ complete reasoning paths. The reasoning path ranked highest by the verifier is chosen as the final candidate. For all experiments, we set $N=256$ following Wang et al. () unless specified otherwise.

#### Verifier-guided step-level beam search (BS).

To prevent errors in an intermediate reasoning step from propagating to subsequent steps, Yu et al. () proposed guided decoding during intermediate reasoning steps via a verifier, a search strategy we call verifier-guided step-level beam search. For a test problem, after an LLM partially generates $K$ reasoning paths each containing only the first intermediate reasoning step, the verifier-guided step-level beam search strategy alternates between the following two steps until all $K$ partially generated reasoning paths are complete: (i) a verifier selects the top-$b$ $(<K)$ ranked partially generated reasoning paths, and (ii) the LLM generates $K/b$ subsequent intermediate reasoning steps for each path chosen by the verifier. Among the $K$ complete reasoning paths, the one scored highest by the verifier is selected. Thanks to verifier intervention in generating each intermediate reasoning step, with $K$ much smaller than $N$, the performance of verifier-guided step-level beam search can be similar to that of best-of-N search in Table 1 and 2.

### 4.1 Grade School Mathematics (GSM8K)

#### Setups.

An LLM is fine-tuned on the training dataset of GSM8K for two epochs with a batch size of $128$ and a learning rate of $1e$-$5$. Then, we sample $N_{tr}=100$ reasoning paths per training problem with a temperature of $0.7$ from the fine-tuned LLM and label each token in a reasoning path as Eq. 11. Finally, TVM initialized from either the same LLM or the fine-tuned LLM is trained on this dataset for one epoch with a batch size of $512$ and a learning rate of either $2e$-$6$ or $1e$-$5$. More experimental details are deferred to Appendix E.

#### Results.

In the case of best-of-N search, we compare TVM with ORM (Cobbe et al., ) and Math-Shepherd (Wang et al., ), a PRM without human annotations, as explained in Sec. 2. As all experimental results in Wang et al. () are only based on LLMs fine-tuned on MetaMATH, we also evaluate Math-Shepherd only for Mistral-7B-MetaMath and Llama3-8B-MetaMath. Despite using large $N$, Table 1 shows that TVM surpasses ORM and Math-Shepherd with improvements ranging from 0.6 to 2.6%p as well as self-consistency from 4.9 to 8.9%p, across the board.

Under the verifier-guided step-level beam search strategy, we primarily compare TVM against OVM (Yu et al., ) because Yu et al. () confirmed that step-level beam search guided by a token-level verifier performs significantly better than that guided by a sentence-level value model (Feng et al., ) on GSM8K. Further comparison to Feng et al. () is presented in Appendix B. In Table 1, TVM also consistently outperforms OVM ranging from 0.6 to 1.0%p.

One might wonder why the accuracy of OVM ($K=40$) for Mistral-7B is much higher than that of OVM ($K=100$) reported in Yu et al. (). This discrepancy arises because, in our experiments, some tokens (e.g., $<<$, $>>$) are correctly converted to token IDs by the Mistral-7B tokenizer.

### 4.2 Advanced Mathematics (MATH)

#### Setups.

We employ fine-tuned LLMs on MetaMath (Mistral-7B-MetaMath and Llama3-8B-MetaMath) without any further fine-tuning on the training dataset of MATH in order to sample reasoning paths in a newline-delimited format. Following Lightman et al. (); Wang et al. (), we also use $500$ test MATH problems for evaluation, which is the same test dataset of Lightman et al. (), incorporating the remaining $4500$ test problems into the training dataset of MATH. For each training problem, a fine-tuned LLM on MetaMath generates $N_{tr}=25$ reasoning paths with a temperature of $0.7$, with each token labeled as Eq. 11. Then, we train TVM initialized from the same fine-tuned LLM for one epoch on this dataset with a batch size of $512$ and a learning rate of $2e$-$6$. Further experimental details are given in Appendix E.

#### Results.

Similar to Sec. 4.1, Table 2 compares (i) TVM’s best-of-N search performance with ORM and Math-Shepherd and (ii) TVM-guided step-level beam search to ORM-guided step-level beam search (i.e., OVM). In the former case, the performance of TVM is slightly superior or almost comparable to that of ORM and Math-Shepherd. This might be due to the fact that an LLM is extremely prone to producing errors in the process of generating $N$ reasoning paths for difficult MATH problems. However, when capitalizing on the verifier-guided step-level beam search strategy, not only does TVM outperform the OVM ranging from 2.6 to 2.8%p, but TVM-guided step-level beam search also exhibits much better performance than best-of-N search by any verifier even if $K=40$ is much smaller than $N=256$.

Figure: Figure 4: Illustration of OVM’s and TVM’s predictions under verifier-guided step-level beam search. In the third reasoning step, while OVM incorrectly predicts a wrong intermediate reasoning path with the highest score while assigning a low score to a correct path, TVM accurately predicts a correct intermediate path with the highest score and a wrong one with a low score.
Refer to caption: x4.png

### 4.3 Analyses on Verifier-guided Step-level BS

#### Case study.

To validate the superiority of TVM over OVM in predicting whether an intermediate reasoning path is on a promising track toward the correct answer, for a test problem in the GSM8K benchmark, we compare OVM’s and TVM’s predictions. As illustrated in Fig. 4, in the third reasoning step, OVM incorrectly predicts a wrong intermediate reasoning path with the highest score while assigning a low score to a correct path. This occurs because OVM is inherently identical to ORM trained to implicitly and indirectly learn the potential correctness of an intermediate reasoning path. In contrast, TVM accurately predicts a correct intermediate path with the highest score and a wrong one with a low score. As TVM is trained to directly and explicitly estimate the probability of reaching the correct final answer for each token along a reasoning path, TVM can effectively predict at inference whether an intermediate reasoning path is on a promising track toward the correct answer.

**Table 3: Mean and standard deviation of TVM’s accuracy for Mistral-7B and Mistral-7B-MetaMath on the GSM8K benchmark according to varying sizes of $K$ and $b$ when employing verifier-guided step-level beam search. Three random trials are carried out.**
| $K$, $b$ | Mistral-7B | Mistral-7B-MetaMath |
| --- | --- | --- |
| $40$, $10$ | 87.69 $\pm 0.22$ | 88.70 $\pm 0.16$ |
| $80$, $20$ | 87.89 $\pm 0.35$ | 88.75 $\pm 0.20$ |
| $100$, $25$ | 87.92 $\pm 0.13$ | 88.80 $\pm 0.07$ |

#### Beam size study.

To investigate whether the accuracy of TVM improves with larger values of $K$ and $b$ in verifier-guided step-level beam search, we conduct experiments using TVM with varying sizes of $K$ and $b$ for Mistral-7B and Mistral-7B-MetaMath on the GSM8K benchmark. Table 3 shows that the accuracy of TVM on GSM8K increases as both $K$ and $b$ grow, but reaches a saturation point when $K=100$ and $b=25$.

## 5 Related Work

#### Best-of-N search.

For $N$ complete reasoning paths, a verifier Cobbe et al. (); Uesato et al. (); Lightman et al. (); Wang et al. () ranks and picks the highest-scored reasoning path. Although best-of-N search using a verifier shows much superior performance compared to verifier-free strategies such as self-consistency (Wang et al., ), best-of-N search still possesses the same drawback as self-consistency as a large quantity of generated reasoning paths are required to solve challenging reasoning problems.

#### Step-level beam search.

In contrast to the selection among complete reasoning paths, several studies have focused on step-level beam searches for partial reasoning paths. Step-level beam search can be divided into (i) verifier-free step-level beam search and (ii) verifier-guided step-level beam search.

Under the verifier-free step-level beam search strategy, Yao et al. (); Hao et al. () allow value estimation by prompting LLMs to sample or simulate long-term outcomes during inference. Alternatively, Feng et al. (); Yu et al. () introduce step-level beam search guided by a sentence-level value model and an outcome-supervised reward model, respectively. Although Feng et al. (); Yu et al. () show that verifier-guided step-level beam search achieves significant accuracy improvements over verifier-free one, each approach has its own weakness. As delineated in Yu et al. (), a sentence-level value model is unsuitable for step-level beam search. In addition, Yu et al. () uses an outcome-supervised reward model, not a value model. As a result, there is still room for improvement in the performance of verifier-guided step-level beam search.

## 6 Conclusion

In this paper, we introduce a novel verifier termed the Token-supervised Value Model (TVM). This model uses per-token value labels to guide LLMs toward promising mathematical reasoning paths.
Unlike traditional verifiers, which lack token-level labels and thus cannot precisely evaluate intermediate steps in reasoning paths, TVM could estimate the expected cumulative reward for each token. This enables TVM to identify detailed token-level information and perform more precise reasoning at intermediate paths leading to the correct answer. Experimental results on benchmarks such as GSM8k and MATH have revealed that TVM outperforms previous verifiers across 7B-scale LLMs, including Mistral-7B and Llama3-8B, demonstrating its enhanced accuracy and effectiveness.

## Limitations

Our method has demonstrated significant improvements over previous competing methods, but resource constraints limited us from running further experiments. Our TVM was primarily evaluated using 7B-scale models for mathematical reasoning, but it can be applied to larger models and extended to other domains. Additionally, our model could be utilized as a value model in reinforcement learning, such as in Proximal Policy Optimization training (Schulman et al., ; Zheng et al., ), to supervise LLMs.

## Appendix A Proof of Proposition 3.1

Let the reward function $r(t_{n,k})$ be defined as Eq. 3, which includes only the outcome reward with the discount factor $\gamma=1$ and no intermediate reward (i.e., $r(t_{n,k})=0$ except the final answer). Then, $\sum_{l=1}^{\infty}\gamma^{l-1}r(t_{n,k+l})=\sum_{l=1}^{\infty}r(t_{n,k+l})$ becomes either one or zero, depending on whether the resulting final answer will be $\hat{a}$ or not, respectively. As a result, the expected cumulative reward (Eq. 7) is written as

$$ $\displaystyle\mathbb{E}\big{[}\sum_{l=1}^{\infty}\gamma^{l-1}r(t_{n,k+l})\big{ |}q_{tr},\{t_{n,\cdot}\}_{1}^{k}\big{]}$ $\displaystyle=\mathbb{E}\big{[}\sum_{l=1}^{\infty}r(t_{n,k+l})\big{|}q_{tr},\{ t_{n,\cdot}\}_{1}^{k}\big{]}\quad(\because\gamma=1)$ $\displaystyle=\sum_{r=0}^{1}r*\mathbb{P}\big{(}\sum_{l=1}^{\infty}r(t_{n,k+l}) =r\big{|}q_{tr},\{t_{n,\cdot}\}_{1}^{k}\big{)}\quad(\because\sum_{l=1}^{\infty }r(t_{n,k+l})=0\text{ or }1)$ $\displaystyle=\mathbb{P}\big{(}\sum_{l=1}^{\infty}r(t_{n,k+l})=1\big{|}q_{tr}, \{t_{n,\cdot}\}_{1}^{k}\big{)}$ $\displaystyle=\mathbb{P}(\textit{the final answer will be }\hat{a}|q_{tr},\{t_ {n,\cdot}\}_{1}^{k}),$ $$

because $\sum_{l=1}^{\infty}r(t_{n,k+l})=1$ only if the resulting final answer will be $\hat{a}$.

## Appendix B Additional Comparison of TVM with a sentence-level value model (Feng et al., 2024 ) as well as OVM

Although Yu et al. () corroborated that step-level beam search guided by a token-level verifier performs significantly better than that guided by a sentence-level value model (Feng et al., ) on GSM8K, we additionally compare our TVM with a sentence-level value model (Feng et al., ) as well as OVM for Llama2-7B.

**Table 4: Mean and standard deviation of accuracy of a sentence-level value model (Feng et al., ), OVM, and TVM for Llama2-7B (Touvron et al., ) on the GSM8K benchmark in the case of $K=10$ under the verifier-guided step-level beam search strategy. For TVM, three random trials are conducted.**
| Search Strategy | Method | Llama2-7B |
| --- | --- | --- |
| Verifier-guided | Feng et al. () | 52.20 $\pm 0.90$ |
| Step-level | OVM | 66.50 $\pm 0.20$ |
| Beam Search | TVM (Ours) | 66.82 $\pm 0.38$ |

As seen in Table 4, our TVM is superior to both a sentence-level value model (Feng et al., ) and OVM. As explained in Yu et al. (), under the verifier-guided step-level beam search strategy, outcome-supervised reward models can pretend to be a value model, but process-supervised reward models cannot. As a result, the accuracy of a sentence-level value model is worse than that of OVM and TVM.

## Appendix C Algorithm of Empirical Value Estimation

Figure: Algorithm 1 Empirical Value Estimation

## Appendix D Compute Analysis between Best-of-N Search and Verifier-guided Step-level Beam Search

**Table 5: Execution time of best-of-N search without and with vLLM (Kwon et al., ) and verifier-guided step-level beam search on the GSM8K and MATH benchmarks when using $8$ x NVIDIA A100-80GB GPUs and Mistral-7B-MetaMath.**
| Search Strategy | GSM8K | MATH |
| --- | --- | --- |
| Best-of-N search w/o vLLM | $6.5$ hours | $22$ hours |
| Best-of-N search w/ vLLM | $2.1$ hours | $2.4$ hours |
| Verifier-guided step-level beam search | $\mathbf{0.9}$ hours | $\mathbf{1.1}$ hours |

## Appendix E Implementation Details

In both Sec. 4.1 and Sec. 4.2, following Cobbe et al. (), we use both a language modeling objective and the verification objective in Eq. 9, with $20\%$ dropout (Srivastava et al., ). Additionally, we employ the same architecture as Cobbe et al. (), a language model extended with a scalar head composed of a single gain parameter and a single bias parameter, to output a score for each token in a reasoning path. We use the AdamW optimizer with a linear scheduler. We generate $N_{tr}$ reasoning paths with a temperature of $0.7$, a top-k of $50$, and a top-p of $1.0$. Note that for every experiment, a verifier has the same model size and architecture as an LLM used to generate $N_{tr}$ reasoning paths.

**Table 6: Learning rate, batch size, and verifier initialization for training TVM when using Mistral-7B, Mistral-7B-MetaMath, Llama3-8B, and Llama3-8B-MetaMath to generate $N_{tr}=100$ reasoning paths per training problem of GSM8K in Sec. 4.1. Fine-tuned Mistral-7B and fine-tuned Llama3-8B means that they are fine-tuned on the training dataset of GSM8K as described in Sec. 4.1.**
|  | Mistral-7B | Mistral-7B-MetaMath | Llama3-8B | Llama3-8B-MetaMath |
| --- | --- | --- | --- | --- |
| Learning rate | $2e$-$6$ | $2$e-$6$ | $1$e-$5$ | $2$e-$6$ |
| Batch size | $512$ | $512$ | $512$ | $512$ |
| Verifier initialization | fine-tuned Mistral-7B | Mistral-7B-MetaMath | fine-tuned Llama3-8B | Llama3-8B-MetaMath |

**Table 7: Learning rate, batch size, and verifier initialization for training TVM when using Mistral-7B-MetaMath and Llama3-8B-MetaMath to generate $N_{tr}=25$ reasoning paths per training problem of MATH in Sec. 4.2.**
|  | Mistral-7B-MetaMath | Llama3-8B-MetaMath |
| --- | --- | --- |
| Learning rate | $2e$-$6$ | $2$e-$6$ |
| Batch size | $512$ | $512$ |
| Verifier initialization | Mistral-7B-MetaMath | Llama3-8B-MetaMath |

For both best-of-N search and verifier-guided step-level beam search, we also use a temperature of $0.7$, a top-k of $50$, and a top-p of $1.0$. The maximum new token length is set to $400$ for GSM8K and $1024$ for MATH, respectively.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses section above. I am happy to consider increasing my score if I can get more clarity.

====
REVIEW RATING UPDATED AFTER AUTHOR RESPONSE
====
soundness: 3
strengths: * The authors make a clear argument about what current verifiers (ORMs and PRMs) lack, and why TVMs should be able to help. Limitations of ORMs and PRMs are justified with some empirical support. Logically thinking, the progression from ORMs to PRMs to TVMs seems natural.

* Figures and examples (although small and hard to read) are useful.

* The method is clearly described, along with a formulation of prior (ORM and PRM) methods.

* Experiments show a notable improvement when using TVMs, especially for step-level beam search and for the MATH benchmark.

* Additional experiments around FLOPs used and execution time support the idea that tree-based search methods are more efficient, especially if one can pair them with effective verifiers.
summary: This paper proposes token-supervised value models (TVMs) as a superior way to verify whether tree-search based math reasoning models are on the right track or not. The authors compare TVMs to output-supervised reward models (ORMs) and process-supervised reward models (PRMs). They argue that both of these existing verification methods, having been designed for evaluation of full trajectories, are not suitable for verifying partial trajectories, especially at the token level. The conduct experiments with two math datasets, showing that the use of TVMs improves performance. Additional experiments are also provided.
weaknesses: * One obvious disadvantage of TVM compared to PRM is that during *training*, the same N_{tr} sampled reasoning paths per training problem are expected to provide a lot more supervision in TVM. Specifically, suppose a typical reasoning path has 5 steps and 50 tokens. Then, in PRMs, it is easy to imagine several reasoning paths sharing similar partial paths  steps, leading to a good signal for supervising the 5 steps. However, with ~50 token long paths, the chances of partial sampled paths overlapping would decreases quickly and would be very slim after the first 15-20 tokens. So I'm not sure what kind of token-supervision signal can one actually get for the latter parts of the reasoning paths -- unless paths are sampled differently or N_{tr} is higher. The paper doesn't seem to address / discuss this.

* The gains, when using TVM, are rather slim in step-level beam search for the GSM8k dataset, only about 1%. This is not very different from the gains when using best-of-N search. If fact, for Mistral, the gains from using TVM are slightly higher for best-of-N compared to step-level beam search. This goes against the motivational argument presented earlier, namely that ORMs and PRMs are suitable for best-of-N search but not tree search, and it's the latter than needs another verification method.

* In general, TVM doesn't seem to help that much on GSM8k.

* I don't understand what Proposition 4.1 is trying to say. Why is it not "trivially true" if the reward function is 0 at intermediate tokens and 1 at the output token iff is it the correct output? Is this proposition related to TVMs or also equally applicable to ORMs and PRMs (which seems to be the case)? I am not following the significance of this formal proposition.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How is $N_{tr}$ determined, given its direct impact on probability estimation? Is it chosen through parameter tuning, or is there additional insight guiding this choice?

2. A follow-up question: During fine-tuning, do these $N_{tr}$ trials often share some prefix tokens, and how to control them? For example, in Figure 4, the three paths share the first $a-1$ tokens, allowing those tokens to have the value 0.33. If they don’t share any tokens, it seems the problem would reduce to a PVM.


3. How to interpreted Figure 3(b)? The "Verifier’s Scores Histogram for Correct Sampled Solutions" shows that the weights are more concentrated for correct solutions, but what about the histogram on the right side?
soundness: 3
strengths: 1. TVM is a novel approach. It assesses each token's contribution to the likelihood of reaching the correct answer, enabling more precise evaluations of intermediate solutions during tree search.

2. The authors address that TVMs lower false negative rates by distinguishing between promising and incorrect intermediate steps, enhancing recall without sacrificing precision.

3. TVMs show consistent improvements in accuracy across various benchmarks, including GSM8K and MATH, indicating their robustness and versatility.
summary: The paper proposes Token-Supervised Value Models, to enhance the mathematical problem-solving capabilities of LLMs. Traditional methods lack effectiveness when applied to tree search algorithms, leading to premature pruning of promising steps. TVMs address this by assigning a probability to each token that reflects its likelihood of contributing to a correct answer, enabling direct evaluation of intermediate solutions. Experiments show that TVMs outperform ORMs and PRMs in accuracy, especially in tree-search-based strategies, by reducing false negatives and improving the assessment of solution paths​.
weaknesses: 1. TVMs require token-level supervision, which could add complexity to the training and fine-tuning process compared to traditional verifiers.
 
2. The effectiveness of TVMs relies on reasonably accurate token-level probability estimation based on samples of successful and failed trials. For difficult problems, the sampling process controlled by the LLM can be challenging, as most samples may be incorrect, potentially reducing the effectiveness of TVMs to that of a PRM.

3. Some sections need clarification; please see the questions below.

*Typo: In several places, there is an extra "p" added after "%", e.g., "14%p".

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. (Figure 1) I need help understanding why ORM assigns low scores to correct steps and PRM assigns high scores to wrong intermediate steps based on this example. More convincing examples would help readers understand the superiority of the TVM.

2. I cannot understand the explanation of TVM's complexity (line 377). TVM is said to be a tree-search-based method (line 132), but the explanation here says that TVM computes eq.(6) from a set of fixed reasoning paths and does not need a tree search. I'm happy if the authors address this inconsistency. 

3. Moreover, I think the roll-out process (line 374) is also beneficial for the TVM since we can estimate conditional probabilities eq.(6) more precisely for a token if we have more sample runs. Why does TVM avoid this?
soundness: 3
strengths: 1. The proposed method is simple and easy to understand how it works.
2. Experiments are conducted with combinations of multiple benchmarks, LLMs, and search strategies. The experimental results support the proposed method's superiority.
summary: This paper proposes token value models (TVMs), a new method for training verifiers used at test time to improve the problem-solving capabilities of LLMs. Existing training methods, outcome-supervised reward models (ORMs), and process-supervised reward models (PRMs) have drawbacks in that they can give only indirect supervision for intermediate steps. TVMs enable giving direct supervision over tokens by exploiting the probability that the token will read to a correct answer.  Experimental results show that TVM shows better predictive performance than ORMs and PRMs.
weaknesses: Although the experimental results are interesting, the paper's presentation has some problems and thus does not explain the reasons why the TVM works well. Therefore, I think this paper needs a further revision.
  1. The paper says that TVM is superior to ORM and PRM since it uses token-level direct supervision (line 113). I'm not sure why we can say TVM is direct since ORMs, PRMs (without human annotations), and TVMs all rely on supervision signals for outcomes. No direct supervision on intermediate steps is given. Therefore, all these methods share the same problem described as the disadvantages of the ORMs (line 267).
  2. Moreover, the supervision of TVM (eq. (5)) based on conditional probability looks pretty similar to the reward for ORMs (eq. (3)). If we have $N_{tr}$ samples, the reward for a token $t_{n, k}$ in ORMs will be close to the success rate of the paths containing $t_{n, k}$. Therefore, the proposed supervision seems less novel.
  3. The paper says that one of its main contributions is that it is the first to disclose that PRMs produce a high false negative error. However, this is not true, as the explanation (line 306, Figure 2) assumes that PRMs do not use human annotation. The false-negative problem would not occur if we had human annotation on intermediate steps. Therefore, the statement is incorrect.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: Overall I liked this paper and I think it will be a good fit in the program. I hope the authors are able to resolve my concerns.
Please respond to my comments on the weaknesses.
soundness: 3
strengths: S1. The paper is very well-written and organized although it seems a bit verbose with some aspects not really required.

S2. The presented idea is very convincing and intuitive. The idea is novel and weighing up of tokens (or a partial order sequence) has been successfully explored as an idea in many different domains.

S3. The baselines are good and the empirical setup overall is good.
summary: The paper presents TVM -- token-supervised value models that can act as neural verifiers and provided tree-based LLM reasoning methods with a way to rank different parts of the search tree as more/less promising on reaching a solution in the domain of math problems. The authors state that existing SOTA like Outcome-based (ORMs) and Process-based (PRMs)  supervision models cannot rank paths effectively due to the way they are trained. ORMs only provide a reward if the answer is correct whereas PRMs provide the same reward to all steps. This causes issues in recall with PRMs since an entire step is marked as a failure even if only a few tokens are wrong.

To mitigate this drawback of PRMs, the authors introduce TVMs that utilizes different generated reasoning paths in the training set to provide different "weights" or rewards. For any substring of a correct solution, each token of that substring is allocated a weight based on the total substrings (or reasoning paths) that were generated. This can be thought of as a value function and the authors provide some soft theoretical analysis for it. 

The authors then conduct an empirical evaluation that showcases the strengths of their approach.
weaknesses: W1. The paper was a bit verbose even if it was well-written. There are several aspects that could have been cut. For example, I do not understand why  Tables 4, 5 were included since it does not appear to be a major contribution of the paper. Table 3 already showcases all methods with different tree-search paradigms so I am not sure why this was necessary. Please comment. An analysis of training effort would have been better here.

W2. The empirical evaluation is a bit lacking. The gains seem marginal in most cases and std. deviations are missing from these plots. Please provide them so that a better assessment can be made.  Also, it seems that this approach can be more general. I am not sure why the authors have limited to only math tasks. A more general evaluation would have improved the paper's impact.

Meta-review: This work introduces token-supervised value models (TVMs), which provide token-level supervision using verifiers. The verifiers assign a score to each token, indicating the probability of reaching a correct final answer. The authors show that TVMs achieve lower false negative errors than process-supervised reward models without human annotations, and provide interesting theoretical insights that their verifiers are equivalent to value functions, which can be used to guide tree search.

The reviewers suggested that the proposed idea is novel and convincing, the experiments show good performance, and the paper is well-written in general with clear descriptions. However, some reviewers also noted issues about training complexity, low performance gain on some tasks and verbosity in writing. The authors actively engaged with reviewers during the rebuttal and discussion phase.

Overall I think the strengths outweigh the weaknesses. All reviewers leaned towards acceptance and I concur.

Decision: Accept (Poster)